Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimise TransactionFetcher #6012

Merged
merged 68 commits into from
Jan 18, 2024
Merged
Show file tree
Hide file tree
Changes from 57 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
2871557
Limit tx requests on by peer basis
emhane Jan 8, 2024
e69907d
Remove dead code
emhane Jan 8, 2024
f7a6a74
Semantic clean up
emhane Jan 8, 2024
8b3a058
Buffer missing hashes for re-fetch
emhane Jan 8, 2024
8f2c8fd
Limit tx requests per peer
emhane Jan 9, 2024
7718fea
Semantic clean up
emhane Jan 9, 2024
056520b
Pack eth68 tx requests based on size element
emhane Jan 9, 2024
e850548
Implement retry logic on a by tx hash basis
emhane Jan 9, 2024
f8922a0
Store only fallback peers that may be used
emhane Jan 9, 2024
684c939
Store buffered hashes in lru cache
emhane Jan 9, 2024
f10a05d
Fix lru cache to match docs
emhane Jan 10, 2024
e957d65
Fix (retries, unknown hashes) invariant
emhane Jan 10, 2024
6d8d5e3
Add todo for packing eth68 request from buffer
emhane Jan 10, 2024
b2619a1
Store active peers in LruMap
emhane Jan 10, 2024
b0cd340
Fix (unknown-hashes, buffered-requests) invariant and packing requests
emhane Jan 10, 2024
9e75ddf
Add cache eviction feedback to keep in bounds
emhane Jan 10, 2024
ed54fda
Add wrapper for schnellru::LruMap that impl Debug and make evicition …
emhane Jan 10, 2024
c635548
Fix dependencies
emhane Jan 11, 2024
5255b32
Make eviction feedback sync
emhane Jan 11, 2024
fc06d71
Fix bug peer not idle
emhane Jan 11, 2024
717da52
Avoid filtering out to then include again
emhane Jan 11, 2024
fe271f6
Fix bug remove peer from fallback peers upon packing request from buffer
emhane Jan 11, 2024
353b45f
fixup! Fix dependencies
emhane Jan 11, 2024
6c20e76
Add test for lru map debug impl
emhane Jan 11, 2024
32d4024
Lint fix
emhane Jan 11, 2024
37f84f4
Fix merge conflicts
emhane Jan 12, 2024
63e44b5
Fix merge conflicts with schnellru-debug-wrapper
emhane Jan 12, 2024
3905331
* Add tests for transaction fetcher
emhane Jan 12, 2024
8e498e0
fixup! * Add tests for transaction fetcher * Debug request filling fr…
emhane Jan 13, 2024
60d7942
fixup! * Add tests for transaction fetcher * Debug request filling fr…
emhane Jan 13, 2024
e64bb04
Fix lint
emhane Jan 13, 2024
c35edf6
Store size metadata for eth68 announcements and debug request packing
emhane Jan 13, 2024
c28642b
Optimise unknown peers lookup with lru cache
emhane Jan 14, 2024
f0ce831
Merge branch 'main' into emhane/optimise-tx-retrieval
emhane Jan 14, 2024
14583d2
fixup! Optimise unknown peers lookup with lru cache
emhane Jan 14, 2024
5c2f996
Remove unused semaphore
emhane Jan 14, 2024
a2f3c92
Call closure directly to avoid mutable variable
emhane Jan 15, 2024
a933c5d
Remove ended sessions in fallback peers before storing new fallback peer
emhane Jan 15, 2024
72d66cb
Break loop early for full request for eth68 hashes too
emhane Jan 15, 2024
5c2b6c3
Move duplicate code to method
emhane Jan 15, 2024
ef1a256
Fix comment
emhane Jan 15, 2024
50dd2e7
Improve interface between NewPooledTransactionHashes and NewPooledTra…
emhane Jan 15, 2024
5db32be
Simplify pack_hashes signature
emhane Jan 15, 2024
0129b2d
Update docs
emhane Jan 15, 2024
180c3bf
Re-scope method get_idle_per_for
emhane Jan 15, 2024
4b6035e
Fix invariant check
emhane Jan 15, 2024
d8bac3a
improve usability of schnellru::LruMap and LruCache (LinkedHashSet ca…
emhane Jan 15, 2024
41999ab
Merge branch 'main' into emhane/optimise-tx-retrieval
emhane Jan 15, 2024
c19dc2e
fixup! Merge branch 'main' into emhane/optimise-tx-retrieval
emhane Jan 15, 2024
a8c5ef8
Fix lint
emhane Jan 15, 2024
6f42809
Standardise log output abbreviation
emhane Jan 16, 2024
50b623d
Sanatise rlp in decoding NewPooledTransactionHashes
emhane Jan 16, 2024
6cc7579
fixup! Sanatise rlp in decoding NewPooledTransactionHashes
emhane Jan 16, 2024
57d6337
Remove panic from library
emhane Jan 16, 2024
1e2e5ea
fixup! Standardise log output abbreviation
emhane Jan 16, 2024
2c135ef
Reset commit 50b623d88 and 6cc7579f8, out of scope to fix breaking tests
emhane Jan 16, 2024
c0b2686
Add module docs
emhane Jan 16, 2024
9dbdd20
Fix bug surplus hashes
emhane Jan 17, 2024
2b9b2c4
Extend fill request test to cover eth68 packing
emhane Jan 17, 2024
3940f74
Fix formatting
emhane Jan 17, 2024
639d955
Update docs
emhane Jan 17, 2024
fc489f6
fixup! Update docs
emhane Jan 17, 2024
6964716
chore(dep): replace unmaintained dependency tui with ratatui (#6114)
zerosnacks Jan 18, 2024
e3669f5
Fix semantics and docs
emhane Jan 18, 2024
86bf8fb
Downgrade log level from warn to debug
emhane Jan 18, 2024
c79e07f
Move TransactionFetcher to own file
emhane Jan 18, 2024
22027c3
Fix docs
emhane Jan 18, 2024
a99ade5
fixup! Fix docs
emhane Jan 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 24 additions & 0 deletions crates/net/eth-wire/src/types/broadcast.rs
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,14 @@ impl NewPooledTransactionHashes {
}
}

/// Returns a mutable reference to transaction hashes.
pub fn hashes_mut(&mut self) -> &mut Vec<B256> {
match self {
NewPooledTransactionHashes::Eth66(msg) => &mut msg.0,
NewPooledTransactionHashes::Eth68(msg) => &mut msg.hashes,
}
}

/// Consumes the type and returns all hashes
pub fn into_hashes(self) -> Vec<B256> {
match self {
Expand Down Expand Up @@ -188,6 +196,15 @@ impl NewPooledTransactionHashes {
NewPooledTransactionHashes::Eth68(msg) => msg.hashes.len(),
}
}

/// Returns an iterator over tx hashes zipped with corresponding eth68 metadata if this is
/// an eth68 message.
pub fn as_eth68(&self) -> Option<&NewPooledTransactionHashes68> {
match self {
NewPooledTransactionHashes::Eth66(_) => None,
NewPooledTransactionHashes::Eth68(msg) => Some(msg),
}
}
}

impl From<NewPooledTransactionHashes> for EthMessage {
Expand Down Expand Up @@ -265,6 +282,13 @@ pub struct NewPooledTransactionHashes68 {
pub hashes: Vec<B256>,
}

impl NewPooledTransactionHashes68 {
/// Returns an iterator over tx hashes zipped with corresponding metadata.
pub fn metadata_iter(&self) -> impl Iterator<Item = (&B256, (u8, usize))> {
self.hashes.iter().zip(self.types.iter().copied().zip(self.sizes.iter().copied()))
}
}

impl Encodable for NewPooledTransactionHashes68 {
fn encode(&self, out: &mut dyn bytes::BufMut) {
#[derive(RlpEncodable)]
Expand Down
1 change: 1 addition & 0 deletions crates/net/network/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ rand.workspace = true
secp256k1 = { workspace = true, features = ["global-context", "rand-std", "recovery"] }
derive_more.workspace = true
schnellru.workspace = true
itertools.workspace = true

enr = { workspace = true, features = ["rust-secp256k1"], optional = true }
tempfile = { workspace = true, optional = true }
Expand Down
8 changes: 2 additions & 6 deletions crates/net/network/src/cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,13 @@ impl<T: Hash + Eq> LruCache<T> {

/// Remove the least recently used entry and return it.
///
/// If the `LruCache` is empty this will return None.
/// If the `LruCache` is empty or if the eviction feedback is
/// configured, this will return None.
#[inline]
fn remove_lru(&mut self) -> Option<T> {
self.inner.pop_front()
}

#[allow(dead_code)]
/// Expels the given value. Returns true if the value existed.
pub fn remove(&mut self, value: &T) -> bool {
self.inner.remove(value)
Expand All @@ -80,14 +80,12 @@ impl<T: Hash + Eq> LruCache<T> {
}

/// Returns number of elements currently in cache.
#[cfg(test)]
#[allow(dead_code)]
pub fn len(&self) -> usize {
self.inner.len()
}

/// Returns `true` if there are currently no elements in the cache.
#[cfg(test)]
#[allow(dead_code)]
pub fn is_empty(&self) -> bool {
self.inner.is_empty()
Expand Down Expand Up @@ -136,7 +134,6 @@ where
K: Hash + PartialEq,
{
/// Returns a new cache with default limiter and hash builder.
#[allow(dead_code)]
pub fn new(max_length: u32) -> Self {
LruMap(schnellru::LruMap::new(ByLength::new(max_length)))
}
Expand All @@ -147,7 +144,6 @@ where
K: Hash + PartialEq,
{
/// Returns a new cache with [`Unlimited`] limiter and default hash builder.
#[allow(dead_code)]
pub fn new_unlimited() -> Self {
LruMap(schnellru::LruMap::new(Unlimited))
}
Expand Down
2 changes: 1 addition & 1 deletion crates/net/network/src/session/active.rs
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ impl ActiveSession {
// request was already timed out internally
self.update_request_timeout(req.timestamp, Instant::now());
}
};
}
} else {
// we received a response to a request we never sent
self.on_bad_message();
Expand Down
1 change: 1 addition & 0 deletions crates/net/network/src/swarm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ use std::{
sync::Arc,
task::{Context, Poll},
};

use tracing::trace;

#[cfg_attr(doc, aquamarine::aquamarine)]
Expand Down
Loading
Loading