Limit the number of accepted connections #1391
Comments
mrBliss
added
networking
byron
|
To account how many resources are available for downstream peers, we need to estimate how many resources are taken by upstream ones. Currently we have a static configuration which decides how many upstream peers we will connect to, with the peer2peer governor this will change but still we should be able to have a policy which governs that (@dcoutts). |
|
I assigned myself to work on the network level interface. |
|
@coot This ticket is only about accepting a limited number of accepted clients. We can use a relatively low value for it that is surely safe for now, e.g., 20. Such a limit must be in place before the release, otherwise the node will crash when too many clients connect. After adding such a limit, we can do more precise accounting for resources, and increase the limit based on that, but that's not what this ticket is about. We have IntersectMBO/ouroboros-consensus#736 and IntersectMBO/ouroboros-consensus#735 for that. IMO, the network is the most logical place to have such a limit. @coot Do you think the network team will be able to add such a limit in this sprint? |
|
Blocked on #1473. |
|
This is blocked on networking improvements, and the networking team tells me that this won't be done this sprint. Moving to the next. |
|
This will be handled network side, so consensus is not involved. |
coot
changed the title
|
@coot I suspect giving the timing we will need to implement this for the existing connection manager, and merge the new p2p-governor based connection manager after, targeting the Shelley testnet. |
|
@dcoutts, ok - I am working now on Windows issues (CI, and finishing the re-factorization of |
|
PR #1831 |
1831: Policy for lmitting number of accepted connections r=coot a=coot This pr implements a policy which allows us to limit the number of accepted connections, addressing the issue #1391. There are two thresholds: - `acceptedConnectionsHardLimit` - `acceptedConnectionsSoftLimit` And a delay - `acceptedConnectionsDelay` If we are below the soft limit, we accept connections without any delay; Above the *soft limit*, we start rate limiting using a linear regression of the `acceptedConnectionsDelay`. If we hit the hard limit, then we wait until the number of connections drops (but not shorted than `acceptedConnectionsDelay`). (We could add a third threshold to drop below that, but we should be good for now with this policy). This is a standalone component, so it will be easy to bring it to `connection-manager`. Todo: - [x] add a trace Co-authored-by: Marcin Szamotulski <[email protected]>
At the moment, if a node runs out of file handles or other resources because too many clients have connected to it, it will shut down with an error. Clients will try the next node, which will also run out of resources and shut down, then they'll go to the next node and so on, leading to a cascading failure.
Instead, it should decline the new client so that it can keep on serving the connected clients. The client should wait or try the next server so that the network is unaffected.
To figure out the maximum number of clients, we should figure out a conservative maximum bound on the resources required to serve a client, see #1278.
Related: #1384 #1378 (comment)
The text was updated successfully, but these errors were encountered: