Westend node memory usage x3.5 compared to Polkadot/Kusama

[rvalle]

Hi

For Polkawatch decentralization analytics we run many nodes, usually on constrained resources (bandwidth and memory) with significant success, our Polkadot and Kusama nodes run on 4gb VMs with polkadot binary using 1.5g memory, we typically run with the following parameters:

docker run \
  -d \
  parity/polkadot \
      --chain=polkadot/kusama \
      --out-peers 2 \
      --in-peers 2 \
      --in-peers-light 1 \
      --max-parallel-downloads 1 \
      --no-mdns \
      --no-private-ipv4 

However, westend is consuming 3.5 times more memory when compared to polkadot or kusama nodes while syncing, reaching #18944395.

Is this normal?

[rvalle]

This is happening in the context of the SYNC process getting stuck.

[vishal-sys]

How can i start to contribute in it as i'm a jr rust developer ?
plz can u guide me as i'm eager to contribute

[bkchr]

@rvalle You don't run with any kind of special pruning flags?

[rvalle]

@bkchr not for this one, running as standard node, I am using it to test p2p DHT stuff.

I actually think syncing is failing. it is stock for days with same finalized block #

best: #18983076 (0xa13f…212e), finalized #18981888

I have never noticed so much difference between best and finalized.

[rvalle]

@bkchr since this node is only for testing, if sync is the issue, could not I just use one of the fast synching profiles? warp perhaps?

[rvalle]

@bkchr I am also failing to get it to sync using wrap:

2024-03-19 08:28:18 ⏩ Warping, Downloading finality proofs, 0.00 Mib (2 peers), best: #0 (0xe143…423e), finalized #0 (0xe143…423e), ⬇ 149.4kiB/s ⬆ 0.4kiB/s    
2024-03-19 08:28:19 Warp sync is complete, continuing with state sync.    
2024-03-19 08:28:23 ⚙️  State sync, Importing state, 0%, 1.61 Mib (3 peers), best: #0 (0xe143…423e), finalized #0 (0xe143…423e), ⬇ 325.6kiB/s ⬆ 4.0kiB/s    
2024-03-19 08:28:24 Failed to import target block with state: Other(ClientImport("Execution failed: Other: Exported method BabeApi_current_epoch is not found")).    
2024-03-19 08:28:24 State sync failed. Falling back to full sync.    
2024-03-19 08:28:25 💔 Verification failed for block 0xdb5568e8094fecd5a64f04f36ee76f9593276450949ecbbdaee996ee8e309704 received from (12D3KooWMoWNEVFC1EnQfMArbitTi8TkkbrSVpe1anHE2kHjyVHu): "Could not fetch epoch at 0xa4f8a8ce6014499c9b4b71490a2c1639373c4227d83bd25686612eeccd1b274b"    
2024-03-19 08:28:26 💔 Verification failed for block 0xdb5568e8094fecd5a64f04f36ee76f9593276450949ecbbdaee996ee8e309704 received from (12D3KooWP4vwqiPkZBmAknjeV9h8dtsPYtHy7ntwyhMea9T7tqeN): "Could not fetch epoch at 0xa4f8a8ce6014499c9b4b71490a2c1639373c4227d83bd25686612eeccd1b274b"    
2024-03-19 08:28:28 ⚙️  Syncing 87974.6 bps, target=#440473 (1 peers), best: #439873 (0xa4f8…274b), finalized #439873 (0xa4f8…274b), ⬇ 36.3kiB/s ⬆ 3.6kiB/s    

perhaps this is related to issue #60

[rvalle]

this may be related to: https://substrate.stackexchange.com/questions/9730/rococo-cant-warp-sync-stuck-at-16mb-finality-proof-download

mentions that rococo and westend and not wrapable

[rvalle]

I only managed to sync the node using the fast method. and now memory consumption seems normal.
I have the impression that there is something broken, but perhaps due to the issue mentioned in stackexchange.
@bkchr you can reopen the issue if you want to further investigate.

[bkchr]

Thank you for reporting back! If your node was stuck on a finalized block, this should explain the memory increase.