Experimental engine errors with no transaction found for Number(XXX) while new blocks are being processed

[BowTiedDevil]

Describe the bug

When op-reth is actively synced and following the chain tip using the experimental engine, calls to eth_getLogs (and probably others) throw errors like: (code: -32603, message: no transaction found for Number(689728934), data: None) (1x)

Curiously, the errors only occur when the node is actively syncing. If I disable the consensus client (op-node), op-reth will paused at the last block height and the same RPC calls complete without any error. Resuming the sync by restarting op-node will result in the calls failing again.

Steps to reproduce

Run reth or op-reth with --engine.experimental and execute a large RPC call to eth_getLogs using cryo or similar.

Here is a simple bash script that executes a series of large calls against my op-reth node synced to Base mainnet:

fetch_base.sh

DATA_DIR=/home/btd/code/cryo_data/
CHAIN=base
RPC_URL=http://localhost:8543

CRYO_CHUNK_SIZE=100000
CRYO_REORG_BUFFER=0
CRYO_COMMON_OPTIONS="--rpc ${RPC_URL} \
        --reorg-buffer ${CRYO_REORG_BUFFER} \
        --u256-types binary \
        --chunk-size ${CRYO_CHUNK_SIZE}"

UNISWAP_V3_DEPLOYMENT_BLOCK=1_371_680
UNISWAP_V3_FACTORY_CONTRACT_ADDRESS=0x33128a8fC17869897dcE68Ed026d694621f6FDfD

POOLCREATED_EVENT_HASH="0x783cca1c0412dd0d695e784568c96da2e9c22ff989357a2e8b1d9b2b4e6b7118"
POOLCREATED_EVENT_SIGNATURE="PoolCreated(address indexed token0, address indexed token1, uint24 indexed fee, int24 tickSpacing, address pool)"

# UniswapV3 new pools
DIR="${DATA_DIR}/${CHAIN}/uniswap_v3_poolcreated_events"
if [ -d $DIR ]; then rm "${DIR}/$(ls ${DIR} | tail -1)"; fi
cryo logs \
        $CRYO_COMMON_OPTIONS \
        --blocks "${UNISWAP_V3_DEPLOYMENT_BLOCK}:${LAST_BLOCK}" \
        --contract "${UNISWAP_V3_FACTORY_CONTRACT_ADDRESS}" \
        --event "${POOLCREATED_EVENT_HASH}" \
        --event-signature "${POOLCREATED_EVENT_SIGNATURE}" \
        --output-dir $DIR

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

reth-optimism-cli Version: 1.0.7
Commit SHA: 9f23443
Build Timestamp: 2024-09-19T18:04:36.208737399Z
Build Features: asm_keccak,optimism
Build Profile: maxperf

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

base

What type of node are you running?

Pruned with custom reth.toml config

What prune config do you use, if any?

[prune]
block_interval = 5

[prune.segments]
sender_recovery = "full"
account_history = { distance = 10064 }
storage_history = { distance = 10064 }

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[joshieDo]

possible that we are skipping the in-memory state, and checking straight away on storage. which would explain why it works fine when its not actively syncing

[joshieDo]

or actually might be related to #10037 on block_state_by_tx_id

[emhane]

possibly related #10181