Thanks for the reply!.
I did some experimenting.
It seems that most of the problems whatever device the router was talking to on the WAN. I have no idea how it controls bandwidth, but it was very messy at lower bandwidths.
At 300Mbps (limited by my ISP), I was getting latency increases off the charts (>250ms) when I approached that 300Mbps limit (>270Mbps).
Cake did great job of reducing the latency bringing it back to <20ms or so.
But at >500Mbps, without Cake, the problems had largely vanished. Latency increases were nominal.
The poor latency performance seems to be down to whatever device was limiting my bandwidth. At 500Mbps and above it performed very well, but below that it was a nightmare.
I am not a network person so forgive me if none of that makes sense!!