First Thing: You could be using main server as a streaming server.... Main server have cron jobs running. php services running. python scripts running, lots of cods running and master nginx running.
Secong Thing: Where is your server location...?
Server Locations difference between your and your provider could be much more....