Anything interesting in the client logs on an effected system?
Do you have command polling enabled? What interval?
Are other things happening at the same time like @TheTick mentions? What about datacenter backup traffic?
Are all of your relays dedicated to that task? or do you have relays that are desktop systems that could be powered off in the evening? How many endpoints do you typically have per relay?
There are some download retry settings that can be tuned that may help:
_BESClient_Download_RetryMinutes=1
_BESClient_Download_CheckAvailabilitySeconds=120
I have seen cases where the client will wait 10 minutes between each download which was the cause of an issue like you are describing, but what is happening could be many things.