feat(proto, conductor, cd)!: changes for Forma migration #1843

ethanoroshiba · 2024-12-03T15:24:54Z

Summary

Makes the changes necessary in the monorepo to migrate Forma to Mainnet.

Background

Migrating Forma to mainnet necessitates introducing stop and start logic into the Conductor. This makes the proto and Conductor changes necessary for that. In addition to this, this will allow for future needs to stop and start EVMs on our networks. These changes will also need to be reflected in the GenesisInfo sent by rollups to the conductor.

The shutdown and restart logic for the Conductor should be as follows:

Firm Only Mode: Once the stop height is reached for the firm block stream, the firm block at this height is executed (and commitment state updated) before restarting the internal conductor, prompting the rollup for a new GenesisInfo with new start/stop heights (and potentially chain IDs).
Soft and Firm Mode: Once the stop height is reached for the soft block stream, no more soft blocks are executed (including the one at the stop height). The firm block is still executed for this height, and then Conductor restarts.
Soft Only Mode: Once the stop height is reached for the soft block stream, no more soft blocks are executed (including the one at the stop height). Conductor is then restarted.

The aforementioned rollup changes to be made in astria-geth necessitate changing our current charts config for the following. "Forks" now describes a set of instructions for when and how to stop and restart the EVM at different heights, such that changes (such as network migration) can be made.

config:
        # < non astria-prefixed items... >
        AstriaOverrideGenesisExtraData:
        AstriaRollupName:
        AstriaForks: []
                # - < fork name >:
                #         height: ""
                #         halt: ""
                #         snapshotChecksum: ""
                #         extraDataOverride: ""
                #         feeCollector: ""
                #         EIP1559Params: {}
                #         sequencer:
                #                 chainId: ""
                #                 addressPrefix: ""
                #                 startHeight: ""
                #                 stopHeight: ""
                #         celestia:
                #                 chainId: ""
                #                 startHeight: ""
                #                 heightVariance: ""
                #         bridgeAddresses: []

Changes

Changed sequencer_genesis_block_height to sequencer_start_block_height in GenesisInfo (in the execution API).
Added sequencer_stop_block_height, rollup_start_block_height, sequencer_chain_id, and celestia_chain_id, and halt_at_stop_height to GenesisInfo.
Changed sequencer to rollup height calculation to start from rollup_start_block_height in GenesisInfo, so that Conductor doesn't start from rollup height 1 again after restarting.
Removed sequencer and celestia chain ID environment vars from Conductor, and changed chain ID checks to rely on obtaining these from the initialized executor state.
Added astriaForks to EVM rollup config, to reflect changes in astria-geth: https://github.com/astriaorg/astria-geth/pull/59/files.
Added restart and halt logic for when the stop height is reached, where Conductor will shut down at the stop height if halt_at_stop_height is true, and restart otherwise.

Testing

Added restart logic to check_for_restart_ok unit test.
Added blackbox tests to ensure conductor correctly restarts in each mode, and continues to execute at the correct height after fetching updated genesis info and commitment state.
Added blackbox test in SoftAndFirm mode to ensure the conductor shuts down at the stop height if GenesisInfo.halt_at_stop_height is true.
Added smoke test which restarts EVM and Conductor mid-way through the regular smoke test.

Changelogs

Changelogs updated

Breaking Changelist

Proto changes are not technically breaking, since fields were only added, but the old genesis info shape will be rejected since it does not contain the chain IDs to validate against the networks the conductor is connecting to (and would fail even if this was the case, due to sequencer_stop_block_height being read from wire format as 0). Thus, rollups will need to provide an updated GenesisInfo to conductor (includingastria-geth).
Broke domain type GenesisInfo

Related Issues

closes #1841

ethanoroshiba · 2024-12-09T21:36:24Z

~~Opening this for review despite failing status checks so that review isn't blocking on Geth changes. Once a Geth image can be pulled, we'll be able to run the smoke tests.~~
Smoke tests passing now.

SuperFluffy · 2025-01-06T13:55:40Z

crates/astria-conductor/src/executor/mod.rs

OK, I thought about this code some more and I still think that regular operation should not be conflated with errors.

At the same time I think we are not being good citizens in that we keep hitting an endpoint (Sequencer) and then just throwing away the data as is currently the case in execute_soft. Instead, Conductor should actively shut down the channels to the sequencer and celestia tasks after having reached the stop-height.

Then you can have a fall-through case where the executor task just shuts down normally if both channels are closed.

Attempted to do so via dropping in 00b10a4. Let me know if this works!

crates/astria-conductor/src/executor/mod.rs

SuperFluffy · 2025-01-06T13:58:58Z

crates/astria-conductor/src/executor/mod.rs

+                self.state.sequencer_stop_block_height(),
+            ))
+            .wrap_err(
+                "soft height met or exceeded sequencer stop height, attempting restart with new \


Executor does not get to decide this. The decision on whether or not to restart lies with the conductor task one level up.

SuperFluffy · 2025-01-06T14:02:53Z

crates/astria-conductor/src/executor/mod.rs

@@ -457,19 +505,36 @@ impl Executor {
        err,
    ))]
    async fn execute_firm(&mut self, block: ReconstructedBlock) -> eyre::Result<()> {


This should be done in this PR. The current restart logic is with respect to an error encountered from an RPC. This new restart logic is due to regular operation.

I don't think a cancellation token is necessary here. I think it's sufficient to have the executor task actively close its channel to the celestia/sequencer tasks. Then you get those 2 tasks to shut down for free while maintaining a consistent view. Only need to remember to not kill the entire conductor if one of those goes down

SuperFluffy · 2025-01-06T14:03:42Z

proto/executionapis/astria/execution/v1/execution.proto

Should we add a field here to indicate to conductor whether or not it should stop or restart on having reached the stop height?

I don't think it's strictly necessary, but I wonder if it may be better UX. I'll confer with @mycodecrafting to see what he thinks

Surfacing offline discussion with @mycodecrafting that this would be better UX. Noting that the changes required in Geth to accomplish this involve peeking at the next fork in the Geth genesis to see whether halt: true, and then including that information in GenesisInfo

Introduced this field in 680d984 and implemented the logic for it in 02d04d7

ethanoroshiba · 2025-01-06T20:07:29Z

crates/astria-conductor/src/executor/mod.rs

+        &mut self,
+        block: ReconstructedBlock,
+    ) -> eyre::Result<Option<StopHeightExceded>> {
+        if self.is_firm_block_height_exceded(&block) {


Should we also close the firm block channel here, similar to how we close the soft block channel?

ethanoroshiba · 2025-01-28T18:43:36Z

Closing in favor of #1928.

proto changes, add stop/restart logic

d8b9778

github-actions bot added conductor pertaining to the astria-conductor crate proto pertaining to the Astria Protobuf spec cd labels Dec 3, 2024

ethanoroshiba temporarily deployed to BUF December 3, 2024 15:25 — with GitHub Actions Inactive

ethanoroshiba added the override-freeze label Dec 3, 2024

add soft and firm test, update change log and valus

fdbe302

ethanoroshiba temporarily deployed to BUF December 3, 2024 19:01 — with GitHub Actions Inactive

ethanoroshiba added the allow-breaking-proto label Dec 3, 2024

version updates

30bd915

ethanoroshiba temporarily deployed to BUF December 3, 2024 19:24 — with GitHub Actions Inactive

update chart.lock

ff9ad8c

ethanoroshiba temporarily deployed to BUF December 3, 2024 19:26 — with GitHub Actions Inactive

add rollup_start_block_height field to GenesisInfo

af132a1

ethanoroshiba temporarily deployed to BUF December 4, 2024 20:02 — with GitHub Actions Inactive

update height mapping calculation, flush out tests

59456a4

ethanoroshiba temporarily deployed to BUF December 4, 2024 21:54 — with GitHub Actions Inactive

ethanoroshiba added 2 commits December 5, 2024 12:30

geth devtag

afc2499

Merge branch 'main' into ENG-1031/forma_migration

f7c9682

ethanoroshiba temporarily deployed to BUF December 5, 2024 19:05 — with GitHub Actions Inactive

ethanoroshiba marked this pull request as ready for review December 9, 2024 21:36

ethanoroshiba requested review from a team as code owners December 9, 2024 21:36

ethanoroshiba requested review from Fraser999, SuperFluffy and aajimal December 9, 2024 21:36

fix height logic

90b1f11

ethanoroshiba temporarily deployed to BUF December 9, 2024 22:58 — with GitHub Actions Inactive

fix values.yaml

1b93dfb

ethanoroshiba temporarily deployed to BUF December 16, 2024 17:12 — with GitHub Actions Inactive

add values

2283672

ethanoroshiba temporarily deployed to BUF December 16, 2024 17:16 — with GitHub Actions Inactive

fix devTag

3a73489

ethanoroshiba temporarily deployed to BUF December 16, 2024 17:27 — with GitHub Actions Inactive

update specs

483aa39

ethanoroshiba requested a review from a team as a code owner January 3, 2025 14:58

github-actions bot added the documentation Improvements or additions to documentation label Jan 3, 2025

ethanoroshiba temporarily deployed to BUF January 3, 2025 14:58 — with GitHub Actions Inactive

SuperFluffy reviewed Jan 6, 2025

View reviewed changes

add preliminary to GenesisInfo

680d984

ethanoroshiba temporarily deployed to BUF January 6, 2025 16:26 — with GitHub Actions Inactive

ethanoroshiba added 2 commits January 6, 2025 13:19

use enum to propagate exit reason instead of errors

00b10a4

add halt logic

02d04d7

ethanoroshiba temporarily deployed to BUF January 6, 2025 19:44 — with GitHub Actions Inactive

rustfmt

af7ebd0

ethanoroshiba temporarily deployed to BUF January 6, 2025 19:46 — with GitHub Actions Inactive

ethanoroshiba commented Jan 6, 2025

View reviewed changes

Merge branch 'main' into ENG-1031/forma_migration

e890e64

ethanoroshiba temporarily deployed to BUF January 6, 2025 20:52 — with GitHub Actions Inactive

ethanoroshiba added 2 commits January 23, 2025 13:20

Merge branch 'main' into ENG-1031/forma_migration

b1fcb29

make stop block height inclusive

ffa43b6

ethanoroshiba temporarily deployed to BUF January 23, 2025 19:54 — with GitHub Actions Inactive

update chart.lock

b7d18d8

ethanoroshiba temporarily deployed to BUF January 23, 2025 19:59 — with GitHub Actions Inactive

bump chart version

d2031df

ethanoroshiba temporarily deployed to BUF January 23, 2025 20:03 — with GitHub Actions Inactive

ethanoroshiba closed this Jan 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(proto, conductor, cd)!: changes for Forma migration #1843

feat(proto, conductor, cd)!: changes for Forma migration #1843

ethanoroshiba commented Dec 3, 2024 •

edited

Loading

ethanoroshiba commented Dec 9, 2024 •

edited

Loading

SuperFluffy Jan 6, 2025

ethanoroshiba Jan 6, 2025

SuperFluffy Jan 6, 2025

SuperFluffy Jan 6, 2025

SuperFluffy Jan 6, 2025

ethanoroshiba Jan 6, 2025

ethanoroshiba Jan 6, 2025

ethanoroshiba Jan 6, 2025

ethanoroshiba Jan 6, 2025

ethanoroshiba commented Jan 28, 2025

feat(proto, conductor, cd)!: changes for Forma migration #1843

feat(proto, conductor, cd)!: changes for Forma migration #1843

Conversation

ethanoroshiba commented Dec 3, 2024 • edited Loading

Summary

Background

Changes

Testing

Changelogs

Breaking Changelist

Related Issues

ethanoroshiba commented Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ethanoroshiba commented Jan 28, 2025

ethanoroshiba commented Dec 3, 2024 •

edited

Loading

ethanoroshiba commented Dec 9, 2024 •

edited

Loading