feat(store/v2): Implement State Migration #19327
Conversation
|
Warning Rate Limit Exceeded@cool-develope has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 29 minutes and 14 seconds before requesting another review. How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. WalkthroughThis update introduces a comprehensive migration framework to transition the entire state from Changes
Assessment against linked issues
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
|
@cool-develope your pull request is missing a changelog! |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 7
Configuration used: .coderabbit.yml
Files selected for processing (7)
- store/go.mod (1 hunks)
- store/migration/manager.go (1 hunks)
- store/migration/manager_test.go (1 hunks)
- store/migration/stream.go (1 hunks)
- store/proof/proof.go (3 hunks)
- store/snapshots/manager.go (1 hunks)
- store/snapshots/stream.go (1 hunks)
Additional comments: 11
store/migration/manager.go (2)
- 18-25: The struct
Manageris well-defined with clear responsibilities for managing the migration process. The inclusion of both snapshot managers and snapshotter interfaces for storage and commit operations ensures modularity and separation of concerns.- 28-34: The constructor
NewManagercorrectly initializes theManagerstruct. It adheres to best practices by taking dependencies as arguments, which facilitates testing and decoupling.store/migration/stream.go (2)
- 15-21: The
MigrationStreamstruct is well-designed, encapsulating the necessary fields for managing the migration stream. The use of a channel for buffering and an atomic value for error handling is appropriate for concurrent operations.- 41-45: The
CloseWithErrormethod properly stores the error and closes the channel. This approach for error handling in a concurrent environment is appropriate.store/go.mod (1)
- 23-23: The addition of
golang.org/x/syncat versionv0.5.0is appropriate for the concurrent features used in the migration process. Ensure that this version is compatible with other dependencies to avoid conflicts.store/snapshots/stream.go (2)
- 22-27: The introduction of the
WriteCloserinterface with aCloseWithErrormethod is a good design choice. It extendsprotoio.WriteCloserand enhances error handling capabilities during the migration process.- 22-27: The
CloseWithErrormethod in theStreamWriterstruct correctly delegates error handling to thechunkWriter, ensuring that errors are propagated through the stream pipeline. This is a good practice for managing errors in complex I/O operations.store/proof/proof.go (4)
- 103-103: The replacement of
errorsmodwith the standarderrorspackage for error wrapping within theCommitmentOpfunction is a good simplification. This change streamlines error handling across the migration process.- 112-112: The use of
errors.Wrapffrom the standarderrorspackage for detailed error messages enhances the readability and debuggability of the code. This is a good practice for error handling.- 118-118: Consistent use of
errors.Wrapffor error handling in theCommitmentOpfunction maintains code readability and helps in pinpointing the source of errors during the migration process.- 122-122: The validation of arguments length in the
CommitmentOpfunction is crucial for ensuring the function is called with the correct data. The detailed error message provided helps in troubleshooting potential issues.
store/snapshots/manager.go
Outdated
| // CreateMigration creates a migration snapshot and writes it to the given writer. | ||
| // It is used to migrate the state from the original store to the store/v2. | ||
| func (m *Manager) CreateMigration(height uint64, protoWriter WriteCloser) error { | ||
| if m == nil { | ||
| return errorsmod.Wrap(store.ErrLogic, "Snapshot Manager is nil") | ||
| } | ||
|
|
||
| err := m.begin(opSnapshot) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer m.end() | ||
|
|
||
| go func() { | ||
| if err := m.commitSnapshotter.Snapshot(height, protoWriter); err != nil { | ||
| protoWriter.CloseWithError(err) | ||
| return | ||
| } | ||
| protoWriter.Close() | ||
| }() | ||
|
|
||
| return nil | ||
| } |
There was a problem hiding this comment.
The CreateMigration method is a significant addition, facilitating the creation of a migration snapshot for state migration. It correctly handles the migration process asynchronously, which is essential for performance. However, the method lacks detailed error handling and logging for the asynchronous operation. Consider adding logging for the start and completion of the migration process, as well as error logging.
+ m.logger.Info("Starting migration snapshot creation", "height", height)
+ m.logger.Error("Failed to create migration snapshot", "height", height, "error", err)
+ m.logger.Info("Completed migration snapshot creation", "height", height)Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| // CreateMigration creates a migration snapshot and writes it to the given writer. | |
| // It is used to migrate the state from the original store to the store/v2. | |
| func (m *Manager) CreateMigration(height uint64, protoWriter WriteCloser) error { | |
| if m == nil { | |
| return errorsmod.Wrap(store.ErrLogic, "Snapshot Manager is nil") | |
| } | |
| err := m.begin(opSnapshot) | |
| if err != nil { | |
| return err | |
| } | |
| defer m.end() | |
| go func() { | |
| if err := m.commitSnapshotter.Snapshot(height, protoWriter); err != nil { | |
| protoWriter.CloseWithError(err) | |
| return | |
| } | |
| protoWriter.Close() | |
| }() | |
| return nil | |
| } | |
| // CreateMigration creates a migration snapshot and writes it to the given writer. | |
| // It is used to migrate the state from the original store to the store/v2. | |
| func (m *Manager) CreateMigration(height uint64, protoWriter WriteCloser) error { | |
| if m == nil { | |
| return errorsmod.Wrap(store.ErrLogic, "Snapshot Manager is nil") | |
| } | |
| err := m.begin(opSnapshot) | |
| if err != nil { | |
| return err | |
| } | |
| defer m.end() | |
| go func() { | |
| m.logger.Info("Starting migration snapshot creation", "height", height) | |
| if err := m.commitSnapshotter.Snapshot(height, protoWriter); err != nil { | |
| protoWriter.CloseWithError(err) | |
| m.logger.Error("Failed to create migration snapshot", "height", height, "error", err) | |
| return | |
| } | |
| protoWriter.Close() | |
| m.logger.Info("Completed migration snapshot creation", "height", height) | |
| }() | |
| return nil | |
| } |
cool-develope
changed the title
alexanderbez
changed the title
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: .coderabbit.yml
Files selected for processing (3)
- store/migration/manager.go (1 hunks)
- store/migration/stream.go (1 hunks)
- store/snapshots/manager.go (1 hunks)
Files skipped from review as they are similar to previous changes (3)
- store/migration/manager.go
- store/migration/stream.go
- store/snapshots/manager.go
|
Some thoughts from reviewing on our team call this morning:
|
|
I do think we should evaluate resuming halted migrations, but should be done in a separate PR as to make review slightly easier. |
Co-authored-by: Aleksandr Bezobchuk <[email protected]>
Co-authored-by: Aleksandr Bezobchuk <[email protected]>
Co-authored-by: Aleksandr Bezobchuk <[email protected]>
Co-authored-by: Aleksandr Bezobchuk <[email protected]>
yeah, I've added them to TODOs here #18603 |
Description
Closes: #19326
CreateMigrationstaff in the snapshots manager for the state migrationAuthor Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...
!in the type prefix if API or client breaking changeCHANGELOG.mdReviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
I have...
Summary by CodeRabbit
store/v1tostore/v2, including in-memory message handling for migration without disk I/O.