go-libp2p-secio using significant CPU resources

[prestonvanloon]

When profiling the running beacon chain node, we are seeing secio consume the majority of resources in the node.

Here are a few 30s samples of the running beacon node.

pprof.image.binary.samples.cpu.019.pb.gz
pprof.image.binary.samples.cpu.020.pb.gz
pprof.image.binary.samples.cpu.021.pb.gz

View these samples with the go pprof tool.

go tool pprof -http 0.0.0.0:9000 /path/to/pprof.image.binary.samples.cpu.021.pb.gz

Then view at http://0.0.0.0:9000

We have tried turning off secio with libp2p.NoSecurity but we faced issues and had to revert.
See: #2622 #2633

I don't recall the specific issue, but we couldn't get nodes to peer and communicate. Will follow up on that in another issue.

[prestonvanloon]

cc: @djrtwo

[djrtwo]

cc @raulk

[raulk]

Thanks for attaching CPU profiles! On mobile now and can’t open; will look tomorrow. We merged native OpenSSL RSA crypto support under the “openssl” build tag a few weeks ago: libp2p/go-libp2p-crypto#61. It’s 3x faster than the go sdk. I wonder if that’s your bottleneck though, as this only manifests when under heavy load (e.g. relay nodes piping traffic for some 100k peers).

[rauljordan]

We have not seen this in ages, closing unless it becomes noteworthy again