Verify and document per-connection resource requirements

[mrBliss]

Resources, e.g., open file handles or cached index files, should be per-client as opposed to global/shared, as it will make it easier to estimate how many resources we will need in practice. The goal is to reject new clients instead of running out of resources while trying to serve a new client. Another guiding principle: the worst case should be as good as the average case.

Document how many resources a client needs:

• The BlockFetchServer will use an ImmutableDB iterator, which will need a file handle for the epoch file.
• Same for the ChainSyncServer.

We can still have some bounded shared resources to optimise things.

Besides documenting the upper bound of resources each server will need, we should also verify this. We can use the ResourceRegistry for this: create a ResourceRegistry for each server (or, with support from the network, one registry per client), and limit the number of resources it can acquire at the same time.

[mrBliss]

See #618 for some thoughts about the current global cache in the ImmutableDB.

[nfrisby]

Just an update: the P2P work needs this in order to avoid having so many hot peers that we exhaust the number of open FDs the OS allows us.

cc: @coot @dcoutts

[coot]

That's right @nfrisby. For the default configuration for an SPO: 20 hot peers (3 or 4 fd's per connection), 50 warm peers (1 fd per connection), gives a sane number fd's. If we have a formula how to compute number of fd's per peer (which might be upstream, downstream or both), we could use it to:

• set the server hard limit accordingly (e.g. how many connections can be accepted)
• use it in cardano-node to verify that the configuration is sane (I think that a warning would be enough).

[dnadales]

Possibly related to #20