Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support in memory lookups in BlockchainProvider2 #9999

Merged
merged 7 commits into from
Aug 2, 2024
Merged

feat: support in memory lookups in BlockchainProvider2 #9999

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b1a79bd
feat: support in memory lookups in BlockchainProvider2
fgimenez Aug 1, 2024
ede417d
add unit tests and fix canonical_chain implementation
fgimenez Aug 2, 2024
8bbae9e
clippy
fgimenez Aug 2, 2024
ee40f76
more comments
fgimenez Aug 2, 2024
77bbd85
introduce convert_range_bounds function
fgimenez Aug 2, 2024
71e6013
move in memory tx lookup related logic to CanonicalInMemoryState
fgimenez Aug 2, 2024
0b16a77
typo
fgimenez Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions crates/chain-state/src/in_memory.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,8 @@
let pending = self.inner.in_memory_state.pending.read().clone();
let head = self.inner.in_memory_state.head_state();

// this clone is cheap beacuse we only expect to keep in memory a few

Check failure on line 426 in crates/chain-state/src/in_memory.rs

View workflow job for this annotation

GitHub Actions / codespell 

beacuse ==> because
// blocks and all of them are Arcs.
let blocks = self.inner.in_memory_state.blocks.read().clone();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be okay because we only expect a few blocks and all of them are just arcs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, added a comment to make it more clear


std::iter::once(pending).filter_map(|p| p.map(Arc::new)).chain(std::iter::successors(
Expand Down
94 changes: 53 additions & 41 deletions crates/storage/provider/src/providers/blockchain_provider.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ use reth_stages_types::{StageCheckpoint, StageId};
use reth_storage_errors::provider::ProviderResult;
use revm::primitives::{BlockEnv, CfgEnvWithHandlerCfg};
use std::{
ops::{Bound, RangeBounds, RangeInclusive},
ops::{Add, Bound, RangeBounds, RangeInclusive, Sub},
sync::Arc,
time::Instant,
};
Expand Down Expand Up @@ -95,6 +95,30 @@ where
pub fn canonical_in_memory_state(&self) -> CanonicalInMemoryState {
self.canonical_in_memory_state.clone()
}

// Helper function to convert range bounds
fn convert_range_bounds<T>(
&self,
range: impl RangeBounds<T>,
end_unbounded: impl FnOnce() -> T,
) -> (T, T)
where
T: Copy + Add<Output = T> + Sub<Output = T> + From<u8>,
{
let start = match range.start_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n + T::from(1u8),
Bound::Unbounded => T::from(0u8),
};

let end = match range.end_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n - T::from(1u8),
Bound::Unbounded => end_unbounded(),
};

(start, end)
}
}

impl<DB> BlockchainProvider2<DB>
Expand Down Expand Up @@ -165,19 +189,14 @@ where

fn headers_range(&self, range: impl RangeBounds<BlockNumber>) -> ProviderResult<Vec<Header>> {
let mut headers = Vec::new();
let start = match range.start_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n + 1,
Bound::Unbounded => 0,
};
let end = match range.end_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n - 1,
Bound::Unbounded => self.canonical_in_memory_state.get_canonical_block_number(),
};
let (start, end) = self.convert_range_bounds(range, || {
self.canonical_in_memory_state.get_canonical_block_number()
});

for num in start..=end {
if let Some(block_state) = self.canonical_in_memory_state.state_by_number(num) {
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
Comment on lines +198 to +199
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it might be that we need to move this feature into the canonicalmemory type directly.

will track in a new issue

headers.push(block_state.block().block().header.header().clone());
} else {
let mut db_headers = self.database.headers_range(num..=end)?;
Expand All @@ -202,19 +221,14 @@ where
range: impl RangeBounds<BlockNumber>,
) -> ProviderResult<Vec<SealedHeader>> {
let mut sealed_headers = Vec::new();
let start = match range.start_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n + 1,
Bound::Unbounded => 0,
};
let end = match range.end_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n - 1,
Bound::Unbounded => self.canonical_in_memory_state.get_canonical_block_number(),
};
let (start, end) = self.convert_range_bounds(range, || {
self.canonical_in_memory_state.get_canonical_block_number()
});

for num in start..=end {
if let Some(block_state) = self.canonical_in_memory_state.state_by_number(num) {
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
sealed_headers.push(block_state.block().block().header.clone());
} else {
let mut db_headers = self.database.sealed_headers_range(num..=end)?;
Expand All @@ -232,16 +246,9 @@ where
mut predicate: impl FnMut(&SealedHeader) -> bool,
) -> ProviderResult<Vec<SealedHeader>> {
let mut headers = Vec::new();
let start = match range.start_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n + 1,
Bound::Unbounded => 0,
};
let end = match range.end_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n - 1,
Bound::Unbounded => self.canonical_in_memory_state.get_canonical_block_number(),
};
let (start, end) = self.convert_range_bounds(range, || {
self.canonical_in_memory_state.get_canonical_block_number()
});

for num in start..=end {
if let Some(block_state) = self.canonical_in_memory_state.state_by_number(num) {
Expand All @@ -252,6 +259,8 @@ where
headers.push(header);
} else {
let mut db_headers = self.database.sealed_headers_while(num..=end, predicate)?;
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
headers.append(&mut db_headers);
break;
}
Expand Down Expand Up @@ -284,6 +293,8 @@ where
hashes.push(block_state.hash());
} else {
let mut db_hashes = self.database.canonical_hashes_range(number, end)?;
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
hashes.append(&mut db_hashes);
break;
}
Expand Down Expand Up @@ -451,6 +462,8 @@ where

for num in range.clone() {
if let Some(block_state) = self.canonical_in_memory_state.state_by_number(num) {
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
blocks.push(block_state.block().block().clone().unseal());
} else {
let mut db_blocks = self.database.block_range(num..=*range.end())?;
Expand All @@ -472,6 +485,8 @@ where
if let Some(block_state) = self.canonical_in_memory_state.state_by_number(num) {
let block = block_state.block().block().clone();
let senders = block_state.block().senders().clone();
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
blocks.push(BlockWithSenders { block: block.unseal(), senders });
} else {
let mut db_blocks = self.database.block_with_senders_range(num..=*range.end())?;
Expand All @@ -493,6 +508,8 @@ where
if let Some(block_state) = self.canonical_in_memory_state.state_by_number(num) {
let block = block_state.block().block().clone();
let senders = block_state.block().senders().clone();
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
blocks.push(SealedBlockWithSenders { block, senders });
} else {
let mut db_blocks =
Expand Down Expand Up @@ -591,22 +608,17 @@ where
&self,
range: impl RangeBounds<BlockNumber>,
) -> ProviderResult<Vec<Vec<TransactionSigned>>> {
let start = match range.start_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n + 1,
Bound::Unbounded => 0,
};
let end = match range.end_bound() {
Bound::Included(&n) => n,
Bound::Excluded(&n) => n - 1,
Bound::Unbounded => self.canonical_in_memory_state.get_canonical_block_number(),
};
let (start, end) = self.convert_range_bounds(range, || {
self.canonical_in_memory_state.get_canonical_block_number()
});

let mut transactions = Vec::new();
let mut last_in_memory_block = None;

for number in start..=end {
if let Some(block_state) = self.canonical_in_memory_state.state_by_number(number) {
// TODO: there might be an update between loop iterations, we
// need to handle that situation.
transactions.push(block_state.block().block().body.clone());
last_in_memory_block = Some(number);
} else {
Expand Down
Loading