-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: copy and prune data from database with move_to_static_files
, before a pipeline run/unwind
#8127
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that makes sense to me,
the clone is a bit meh, not a big deal, but what if we Arc the DB in the producer instead?
This reverts commit 2672ab5.
move_to_static_files
pre-pipeline copies and prunes data from databasemove_to_static_files
copies and prunes data from database
move_to_static_files
copies and prunes data from databasemove_to_static_files
copies and prunes data from database
move_to_static_files
copies and prunes data from databasemove_to_static_files
, before a pipeline run/unwind
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change LGTM, only need to set the pruner timeout because IIUC otherwise we'll use the default 100ms one
tested with |
@shekhirin 8b5444e approve? |
…efore a pipeline run/unwind (#8127)
…efore a pipeline run/unwind (paradigmxyz#8127)
Before we were only copying data from database to static files with the expectation that during live sync the pruner would gradually delete it.
However, if there is an unwind that fails/crashes/panics for whatever reason, we end up in a situation where the database has supposedly unwounded data (potentially invalid), even though static file data has been unwound.
This PR makes sure that we delete whatever data we copied pre-pipeline run/unwind. So, in case of a failed unwind, there's no leftover database data