Sync time increases exponentially as new connections are accepted

[LilithSilver]

Using the following stress test: https://gist.github.com/LilithSilver/4ae742459d112e4d849bd2e45dca0917

I get an exponential slowdown in the sync_time logs.