The router's speed is just how fast it can transfer data. (not sure if you thought the bottleneck was the router or not, but i'm sure it's not)
What's the speed on your isp's contract? Or is there a cap?
If the package you bought is a higher speed or there isn't any cap, then it may be that your isp is going over some rough spots with their hardware. The one we use, TDS, goes down every few weeks.
Or like Hopea said, the server's slow, like Toribash's forums are (a lot) :P Where are you downloading from?