what's the ping to your destination from your wireless device? does it improve if you use a wired connection? Are you running spdMerlin to monitor ping and jitter from your router to your ISP or a server you connect to regularly (google, steam...)?
my router pings my ISP's servers every 10 mins or so, and that varies from upper 6ms ping+jitter to ~15ms (avg <8ms) on a FTTN DSL connection with native IPv6, but I can't tell you how many miles of fibre-optic cabling is between here and there even if I'm only 30ish miles as the crow flies from the datacenter.
distance from router and barriers between router/client would add latency. remove them by adding a copper connection - much less frustrating ultimately than endlessly tweaking and testing.
best would be to have a direct connection with the other computer you're communicating with, but those direct connections are a little expensive unless it's only to the next room. IPv6 might help cut a few nanoseconds off the ping if the other end is using it too, but unless you can choose servers for the fewest hops (either v4 or v6), it pretty much is what it is.
Another thing to have a look at is SQM/QoS - if your ISP has it enabled on their gateway/modem, it might be to help keep their network limping along smoothly for them. You can probably bridge their equipment, use your router to "sign in" to your ISP and use your own QoS to try to eliminate bufferbloat and packet loss, which may "speed up" your internet feel - buffering (like NAT) induces a slight (or sometimes significant) latency that can interfere with "real time/live action" internet activities.
so, while its a lot to look into and get familiar with and possibly change, I hope this helps. remember - you asked ;p lol
Ping is a really bad indicator of latency. For many years it has been de-prioritized and even before that, can really only be trusted to like 1-2msec granularity. Yes, it can give you an indication and is better than nothing, but not by much. UDP jitter is the best probe to use but finding a destination that supports that can be challenging (they do exist). TCP ping is the next best, again you need to find someone that supports that, but the results aren't as accurate as UDP since UDP measures two separate streams, one outbound, and a separate inbound, where TCP and ICMP measure round trip.
I did extensive testing with Cisco IPSLA (used for monitoring latency and jitter, as well as uptime etc) and the above is based on my findings. Even when ICMP was given the same priority as all other network traffic, it was nowhere near as accurate as the other two. ICMP could fluctuate nearly 1msec, TCP about 500uSec, and UDP was extremely consistent except when the monitoring box was heavily loaded with lots of monitors, even then it was a consistent increase, i.e. having 100 probes running would add 50uSec but that 50uSec didn't change from one test to the next.
IPv6 itself is not any faster or slower. Eliminating NAT can yield some latency improvements, but the much larger header on IPv6 actually adds latency, so it ends up being a wash. A lot of testing has been done and there has been no consistent result. Some found IPv6 slightly faster, some found it actually slower. Since the internet has fewer native IPV6 end-to-end paths than IPv4 (though they are nearing parity at this point) that can have a negative effect on latency also. Some ISPs are still using tunnels and 4to6/6to4 conversions which hurts you too. In reality it comes down to the path you take and the end server. The vast majority of people are still using IPv4 so companies still focus the majority of their capacity on that, putting in a few IPv6 servers to serve the users that are on that. There are a couple exceptions like Google and Facebook who have dedicated a lot of resources to their IPv6 deployment, however if your path to them is longer than the IPv4 route, again, could be worse in the long run.
Fiber distance to your ISPs central office or POP is pretty inconsequential. Around 3.3 uSec per kilometer or 8.xx microseconds per mile.