Aside from enabling QOS on your network, you've probably shown that your Internet Out / In can not handle the traffic your Mac requested. A big traffic source could be locally cached iCloud data:
You can immerse yourself in everything the Mac does by opening the Activity Monitor. Select the network and check the number and volume of the received and sent packets. Over 30 to 300 seconds should give you a good idea if you can or want to quit programs that request more data than the router can handle.
Also contact your ISP / Router Support. There are Quality of Service and other elements that prevent the buffer from bloating and allow ping / ICMP traffic to be unaffected by large downloads. You do not have to change anything on the Mac to keep the ping times exactly where you want – only most people and routers let each device use the entire network – whoever comes first, first served. You can change this on the network page.