RPC provider panics when querying a specific block

[lgaroche]

Describe the bug

Reth node configured with no pruning of block receipts. See configuration below.
Bug: when querying the RPC provider for eth_getBlockByNumber with params ["0xb45a", false], the provider panicks and does not return the block. All previous block queries were executed correctly, also the next blocks.

Error (see panic.log.zip for full trace):

thread 'tokio-runtime-worker' panicked at /root/reth/crates/storage/provider/src/providers/database/provider.rs:1156:61:
stored block is valid

Running on locally built reth 0.1.0-alpha.13 (cf500610) using maxperf flags.

Command: RUST_BACKTRACE=full RUST_LOG=debug ./reth node --datadir /var/lib/reth/mainnet --authrpc.addr 127.0.0.1 --authrpc.port 8551 --http -vvvv > panic.log 2>&1

RPC query: curl -X POST http://localhost:8545 -d '{"method": "eth_getBlockByNumber", "params": ["0xb45a", false], "id": 1, "jsonrpc": "2.0"}' -H "Content-Type: application/json"

Steps to reproduce

1. Sync a node with no pruning of block receipts
2. Query RPC with eth_getBlockByNumber of block 0xb45a
3. Provider panicks

Node logs

thread 'tokio-runtime-worker' panicked at /root/reth/crates/storage/provider/src/providers/database/provider.rs:1156:61:
stored block is valid
stack backtrace:
   0:     0x563d9db14cc5 - std::backtrace_rs::backtrace::libunwind::trace::ha69d38c49f1bf263
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x563d9db14cc5 - std::backtrace_rs::backtrace::trace_unsynchronized::h93125d0b85fd543c
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x563d9db14cc5 - std::sys_common::backtrace::_print_fmt::h8d65f438e8343444
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x563d9db14cc5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h41751d2af6c8033a
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x563d9d038a1c - core::fmt::rt::Argument::fmt::h5db2f552d8a28f63
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/fmt/rt.rs:138:9
   5:     0x563d9d038a1c - core::fmt::write::h99465148a27e4883
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/fmt/mod.rs:1114:21
   6:     0x563d9dad98ed - std::io::Write::write_fmt::hee8dfd57bd179ab2
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/io/mod.rs:1763:15
   7:     0x563d9db1656e - std::sys_common::backtrace::_print::h019a3cee3e814da4
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x563d9db1656e - std::sys_common::backtrace::print::h55694121c2ddf918
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x563d9db16134 - std::panicking::default_hook::{{closure}}::h29cbe3da3891b0b0
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:272:22
  10:     0x563d9db1713f - std::panicking::default_hook::h881e76b2b8c74280
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:292:9
  11:     0x563d9db1713f - std::panicking::rust_panic_with_hook::hcc36e25b6e33969c
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:731:13
  12:     0x563d9db16c4c - std::panicking::begin_panic_handler::{{closure}}::ha415efb0f69f41f9
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:609:13
  13:     0x563d9db16ba6 - std::sys_common::backtrace::__rust_end_short_backtrace::h395fe90f99451e4e
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:170:18
  14:     0x563d9db16b91 - rust_begin_unwind
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:597:5
  15:     0x563d9befdb74 - core::panicking::panic_fmt::h452a83e54ecd764e
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:72:14
  16:     0x563d9befe192 - core::panicking::panic_display::h559b2f73c68a58f2
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:168:5
  17:     0x563d9befe192 - core::panicking::panic_str::h9521b3ee4fd35ecd
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:152:5
  18:     0x563d9befe192 - core::option::expect_failed::h92d9ca41185c3cd6
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/option.rs:1988:5
  19:     0x563d9d8c085a - reth_primitives::block::Block::with_senders::h64a12234460921e2
  20:     0x563d9c138e9a - <reth_provider::providers::BlockchainProvider<DB,Tree> as reth_provider::traits::block::BlockReader>::block_with_senders::h97ccd89aec3465ba
  21:     0x563d9c0d2718 - <reth_rpc::eth::cache::EthStateCacheService<Provider,Tasks> as core::future::future::Future>::poll::{{closure}}::hf080b32b1c53e309
  22:     0x563d9da629f6 - <tracing_futures::Instrumented<T> as core::future::future::Future>::poll::h61956c345f6778d3
  23:     0x563d9da617a0 - tokio::runtime::task::raw::poll::h63709f3cb780961b
  24:     0x563d9db2b7aa - std::sys_common::backtrace::__rust_begin_short_backtrace::hdbb50794f4ea7140
  25:     0x563d9db2b4a1 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h7290647c8a3962fd
  26:     0x563d9db18f25 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h51435299acd7166e
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2007:9
  27:     0x563d9db18f25 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3f833c0a4926bdd4
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/alloc/src/boxed.rs:2007:9
  28:     0x563d9db18f25 - std::sys::unix::thread::Thread::new::thread_start::h2c486d0230ab0d99
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys/unix/thread.rs:108:17
  29:     0x7fd6e249f802 - start_thread
  30:     0x7fd6e243f450 - __GI___clone3
  31:                0x0 - <unknown>

Platform(s)

Linux (x86)

What version/commit are you on?

reth 0.1.0-alpha.13 (cf50061)

What database version are you on?

Current database version: 1
Local database version: 1

What type of node are you running?

Pruned with custom reth.toml config

What prune config do you use, if any?

[prune]
block_interval = 5

[prune.parts]
sender_recovery = { distance = 65_536 }
transaction_lookup = { distance = 65_536 }
account_history = { distance = 65_536 }
storage_history = { distance = 65_536 }

If you've built Reth from source, provide the full command you used

cargo build --profile maxperf

Code of Conduct

• I agree to follow the Code of Conduct

[mattsse]

@shekhirin this looks like a full node/prune issue

reth/crates/primitives/src/block.rs

Lines 72 to 73 in f16a22d

	TransactionSigned::recover_signers(&self.body, self.body.len())
	.expect("stored block is valid")

panics if this returns None

reth/crates/primitives/src/transaction/mod.rs

Lines 1007 to 1011 in f16a22d

	if num_txes < *PARALLEL_SENDER_RECOVERY_THRESHOLD {
	txes.into_iter().map(|tx| tx.recover_signer()).collect()
	} else {
	txes.into_par_iter().map(|tx| tx.recover_signer()).collect()
	}

also, there's a bug here if the body.len == 0 not true

[mattsse]

oh I think I know what's happening, we don't have the senders in full node and therefor fallback to dynamic recovery and https://etherscan.io/tx/0x9e6e19637bb625a8ff3d052b7c2fe57dc78c55a15d258d77c43d5a9c160b0384 is pre EIP-2 with a higher s value so recovery fails. with the function above

@Rjected

[lgaroche]

Thank you for the quick analysis. On a side note, do you recommend keeping full senders recovery when syncing a node? I wasn't sure about it, does it speed up block receipt queries?

[mattsse]

I'm not sure how big the senders table is on mainnet, but likely not small (quite a few txs...)

but having them is faster than recovering them again dynamically, depending on the block (e.g. block with 200 txs would need to recover all txs)

[lgaroche]

Reth book says ~75GB.
I will resync from scratch with senders recovery, do you have everything you need to fix/test the issue or do you need me to keep the database a bit longer?

[mattsse]

I have everything I need,
adding the fix rn