Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polkadot parachains not producing blocks until next session after v1.2 runtime enactement #4226

Closed
2 tasks done
alexggh opened this issue Apr 21, 2024 · 1 comment · Fixed by #4269
Closed
2 tasks done
Assignees

Comments

@alexggh
Copy link
Contributor

alexggh commented Apr 21, 2024

When runtime v1.2 got enacted all protocols for async backing got enabled, that halted parachains until next session. Parachains started producing blocks again after the new session.

This is identical with the kusama incident reported here: #3058, which should have been fixed.

Root cause

This is exactly the same issue as #3058 and the problem happen again because the fix in #3063 was not entirely correct. Because v2::handle_deactivate_leaves https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/node/network/statement-distribution/src/lib.rs#L344 gets called even for leafs that do not have prospective parachains enabled yet, we end up deleting the topology we saved for when leaves with async backing will come here https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/node/network/statement-distribution/src/v2/mod.rs#L829.

So the moment async backing gets enabled, we end-up in situation where unused_topologies is empty, so without any saved topology this code will never get executed https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/node/network/statement-distribution/src/v2/mod.rs#L634.

Hence why the next session fixed the issue.

TODO

  • Root cause
  • Write a proper postmortem.
@Polkadot-Forum
Copy link

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/2024-04-21-polkadot-parachains-stalled-until-next-session/7526/1

github-merge-queue bot pushed a commit that referenced this issue Apr 25, 2024
... a few sessions too late :(, this already happened on polkadot, so as
of now there are no known relay-chains without async backing enabled in
runtime, but let's fix it in case someone else wants to repeat our
steps.

Fixes: #4226

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
@github-project-automation github-project-automation bot moved this from In Progress to Completed in parachains team board Apr 25, 2024
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this issue Aug 2, 2024
…tytech#4269)

... a few sessions too late :(, this already happened on polkadot, so as
of now there are no known relay-chains without async backing enabled in
runtime, but let's fix it in case someone else wants to repeat our
steps.

Fixes: paritytech#4226

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Completed
Development

Successfully merging a pull request may close this issue.

2 participants