Even a download only situation creates a lot of ack packets going back up the upload, and on way asymmetric links like are popular on Cable... it's easy to load it down.
Run SQM on the uplink only, if your router dosen't have a lot of CPU. That helps quite a lot. I used to have a TP-Link...