Continuous backlog population

[pwojcikdev]

Backlog population is a process in which a node scans all accounts in the ledger, with or without any confirmed blocks, and forwards (activates) those accounts which do not have all their blocks confirmed to election scheduler for prioritization and eventual queuing in proper bucket. It is necessary to do this periodically, because the amount of space in each bucket is limited (currently ~2000 entries) and number of accounts needing confirmations can be much higher than that, especially during bootstrap or network spam attack.

The problem with current implementation is that this process runs every 5 minutes and scans the whole ledger at once, leading to situations where we run out of accounts to prioritize before the next run has started. This is especially visible during bootstrapping, a graph showing such situation is included below. We can clearly see the bumps in AEC occupancy where prioritization queue is filled, followed by periods of idleness when priority queue is emptied:

This PR fixes that by modifying the way the ledger scan is done. Instead of 5 minute interval, we run the scan all the time (unless disabled by setting frontiers_confirmation = disabled node config setting), but we throttle the rate at which the scan is done to limit consumption of node resources. The rate and frequency is controlled by two new node-config.toml settings: backlog_scan_batch_size and backlog_scan_frequency. By default it scans 10000 accounts per second divided into 10 batches, so 1000 accounts per batch. This is rather conservative and should be later adjusted with feedback from beta node operators (before this PR we dit it in batches of 64k).

The result of this PR is the AEC that stays full almost all the time (except the initial phase of the bootstrap):

[dsiganos]

It looks like the unit test 'request_aggregator.cannot_vote' failed 4 times.

[pwojcikdev]

@dsiganos I see, there seems to be two tests that are still failing, I'm looking into that. Appears to only break on GH runners, so a bit annoying to debug.

[dsiganos]

It fails when the system is under heavy load. Starting a parallel build from scratch seems to make it crash.
I am currently fixing the qwahzi unit test and then I will come back to this.

I got this crash on my laptop:

[ RUN      ] request_aggregator.cannot_vote
/Users/ds/CLionProjects/nano-node/nano/core_test/request_aggregator.cpp:501: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:
zsh: segmentation fault  ./core_test --gtest_repeat=1000 --gtest_filter=request_aggregator.cannot_vote

[dsiganos]

I left a number of minor comments but it looks good to me overall.