Udp balancing fire and forget mode 100% cpu usage

[fedelway]

In my organization we are using gobetween to load balance Udp traffic to an ELK stack. Our app servers send udp messages with log lines to gobetween which forwards this traffic to several logstash instances.

We tried the roundrobin configuration but we detected the traffic was not evenly balanced (on a total packet basis). So we changed max_requests to 1 to enable fire and forget mode.

The result was the balacing was done right, with logstash instances receiving the same amount of traffic, but the CPU usage got to 100%. With 50-60% from system usage. Previously, before the fire and forget mode was activated, the CPU usage was around 15-20%.

I suspect this is related to the function electAndConnect that is used with the fireAndForget mode (in src/server/udp/server.go) that resolves the address and opens the udp socket on every packet that is received by gobetween. Reusing the sockets probably solves the issue.

[illarion]

@fedelway thank you for great bug report. I've checked your assumption and it's 100% correct. I am working on a fix right now.

[illarion]

@fedelway could you please test the fix in latest gobetween build from master branch?

[fedelway]

@illarion We're running the latest build and it is working perfectly. The CPU usage is stable at 20-30%. I'm going to be monitoring how it behaves over time in my environment.

[fedelway]

I left the balancer running for a whole day and it worked correctly. No errors were logged.