We believe we found the culprit, its a BSD specific issue and should be fixed in a future build.
I’d like to thank @Pixaro for leading me in the right direction.
You should see something like this in dmesg or /var/log/messages:
sonewconn: pcb 0xfffff801865335b8: Listen queue overflow: 193 already in queue awaiting acceptance (1 occurrences)
This is actually the culprit, we are setting a very low limit (the default of 128 on freebsd) and this was done at compile time (In previous builds its probably higher) but in any case this should be handled in the app it self, we will have a patch soon.