Node telemetry (#2446)

* Node telemetry

* Add genesis block

* Don't checkif at the end of the stream with UDP to support compatibility with different message versions

* Add single request and optional blocking

* Formatting

* Use nano locks

* Fix clang build

* Out of date comments

* Update some names of classes/functions/variables

* Fix for disconnected peer channels & some cleanup

* Remove unnecessary lock requirement in active_transactions::lock. (#2475)

* Remove unnecessarry lock requirement in active_transactions::lock.
This also removes some code-smell around passing in a bool to determine if the lock should actually be acquired.

* Using a timeout counter in line with other test idioms.

* Confirmation solicitor revamp (#2472)

* Optional minimum version when querying representatives from crawler

* Revamping confirmation_solicitor to mimick previous active_transactions behavior

* Use a time-based approach to throttle confirmation requests and block flooding

* Addressing Wesley review

* Remove unusued node.hpp include (thanks wes)

* Simplify logic by using unordered_map::operator[] which calls the default constructor if not found

* Split solicitor add into broadcast+add and bring back the logic to  active_transactions

This brings back rate-limitting logic and modifying election variables to active_transactions only. Timings are also slightly adjusted:

- Only 2 requests required before starting to flood blocks
- Timings for test network

* Rename flag

* Only broadcast OR request confirmation in the same loop for the same election

* Enclose lambda in clang-format off

* Beta network reset #2 (#2476)

* Delete rep_weights_beta.bin

* New genesis

* Change header_magic_number to one never used before

* use v1.1 for actions-aws-cli (#2486)

* The start of CLI tests (#2403)

* The start of CLI tests

* Add needed header file after merge

* Serg review comments

* Websocket bootstrap subscription (#2471)

* Bootstrap attempt ID
* Websocket bootstrap subscription

* Change processed blocks factor for requeued pulls (#2437)

From `processed blocks / 10000` to `processed blocks / 4096` for better processing of largest chains in case of failures

* Upgrade confirmation height table to include cemented frontier (#2481)

* Upgrade confirmation height table to include cemented frontier

* Stein review comments

* Add //clang-format block around lambda

* Update comment (thanks Gui)

* Upgrade to v17 instead

* Update comments

* Request aggregator (#2485)

* Request aggregator

Adds a class that runs in a new thread to pool confirmation requests by endpoint.
This allows a reduction of bandwidth, vote generation, and moves some vote generation out of the I/O threads.

* Use a constant for confirm_ack_hashes_max

* Small code simplification

* Use const transaction

* Disable clang-format for lambdas

* Add missing deadlines and update deadline before poll block

* Use a scoped lock_guard pattern and initialize start in-class

* Misc. fixes and documentation

* use clang-format-8 (#2490)

There is some funkiness with clang-format-9 and lambdas that are not
honored

* Formatting fix so clangformat 8 applies cleanly (#2491)

* Support multiple work peers in the same host (#2477)

* Correct check for peers when creating work

* Support multiple work peers in the same address

* Send cancels despite errors to account for non-conforming implementations

* Add tests using a fake work peer, acting as good, malicious or slow

* Comment

* Formatting

* Fix multiple response error in RPC

* Formatting

* Add extra error handling when specifying port and not address with the RPC (and vice versa)

* Formatting

* Serg comments

* Formatting

* Mask not needed

* Missed file check in for mask removal

* Fix test failures

* Gui suggestions

* Formatting

* Fix assert with test on Windows and some cleanup

* Lower number of nodes in node_telemetry.many_nodes when using sanitizers

Co-authored-by: clemahieu <[email protected]>
Co-authored-by: Guilherme Lawless <[email protected]>
Co-authored-by: Russel Waters <[email protected]>
Co-authored-by: Sergey Kroshnin <[email protected]>
Co-authored-by: cryptocode <[email protected]>

nano/core_test/CMakeLists.txt

@@ -19,11 +19,12 @@ add_executable (core_test
 	ledger.cpp
 	locks.cpp
 	logger.cpp
-	network.cpp
-	node.cpp
 	message.cpp
 	message_parser.cpp
 	memory_pool.cpp
+	network.cpp
+	node.cpp
+	node_telemetry.cpp
 	processor_service.cpp
 	peer_container.cpp
 	request_aggregator.cpp

nano/core_test/message_parser.cpp

@@ -25,33 +25,37 @@ class test_visitor : public nano::message_visitor
 	}
 	void bulk_pull (nano::bulk_pull const &) override
 	{
-		++bulk_pull_count;
+		ASSERT_FALSE (true);
 	}
 	void bulk_pull_account (nano::bulk_pull_account const &) override
 	{
-		++bulk_pull_account_count;
+		ASSERT_FALSE (true);
 	}
 	void bulk_push (nano::bulk_push const &) override
 	{
-		++bulk_push_count;
+		ASSERT_FALSE (true);
 	}
 	void frontier_req (nano::frontier_req const &) override
 	{
-		++frontier_req_count;
+		ASSERT_FALSE (true);
 	}
 	void node_id_handshake (nano::node_id_handshake const &) override
 	{
-		++node_id_handshake_count;
+		ASSERT_FALSE (true);
 	}
+	void telemetry_req (nano::telemetry_req const &) override
+	{
+		ASSERT_FALSE (true);
+	}
+	void telemetry_ack (nano::telemetry_ack const &) override
+	{
+		ASSERT_FALSE (true);
+	}
+
 	uint64_t keepalive_count{ 0 };
 	uint64_t publish_count{ 0 };
 	uint64_t confirm_req_count{ 0 };
 	uint64_t confirm_ack_count{ 0 };
-	uint64_t bulk_pull_count{ 0 };
-	uint64_t bulk_pull_account_count{ 0 };
-	uint64_t bulk_push_count{ 0 };
-	uint64_t frontier_req_count{ 0 };
-	uint64_t node_id_handshake_count{ 0 };
 };
 }
 

nano/core_test/node.cpp

@@ -3433,7 +3433,7 @@ TEST (node, bidirectional_tcp)
 	// Test block confirmation from node 2
 	system.wallet (1)->insert_adhoc (nano::test_genesis_key.prv);
 	confirmed = false;
-	system.deadline_set (10s);
+	system.deadline_set (20s);
 	while (!confirmed)
 	{
 		auto transaction1 (node1->store.tx_begin_read ());