For some more information, it comes down to the routing protocols used by the routers/switches/modems depending on the protocol and the network heirachy.
say we have a client at site A and a server/website at site B and C
client a wants to communicate with server C and all 3 are connected, depending on the protocol it may be routed through server B first even if the distance is shorter to server C with a more direct route.
When there are more servers for example B, C, D and E this is where VPN's and Tunneling programs come into play, let's say the default route is A<>B>C>D<>E and you just want to communicate to server D, a decent VPN/Tunnel would be server E that uses different protocols and is a lot closer to your destination giving you a better ping and faster connection.
OSPF has pro's and cons though, it may be the shortest distance but it may also be the slowest server, but it is automatic to build a routing table, unlike RIP that is manually created and gives full control of routing, basically majority of the internet gateways/servers/switches that are cisco use OSPF as it is faster than manally setting up the RIP protocol, QoS also comes into play when streaming video or gaming online, priority systematically gives larger bandwidth to those that require it.
In a small network environment its best to turn off QoS as it splits the internet connection into multiple threads so to speak.
say you have 10 computers all running from the same router they are all running 1gbit and the internet speed is 10gbit, this would be perfect as all computers would have 1gbit internet.
now to look at a more realistic setup, you have 10 computers 1gbit connection for each and internet speed is 100mbit this would give each 10 computers 10mbit connections with QoS enabled, even if only your computer is using the internet but the others are turned on you will only get a top speed of 10mbit on your 100mbit internet. this is where turning off QoS actually helps your internet speed, in doing this though it hinders everyone elses internet speed to practically nothing.
Hope this helps :D
http://www.cisco.com/public/technotes/tech_protocol.shtml
info on cisco protocols ^
one last thing that also changes internet speed dramatically is DNS servers, for example i had an 8mbit connection but i was only getting around 4.5mbit average, after testing multiple DNS servers using multiple dns finder tools i found on the internet i found a much faster responding DNS server and increased my speed to 7.2mbit using a completely different network providers dns server, this helps for ping and web page retrieval.