Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not write ledger snapshot without copying blocks to imm DB #1456

Closed
edsko opened this issue Jan 14, 2020 · 0 comments · Fixed by #1518
Closed

Do not write ledger snapshot without copying blocks to imm DB #1456

edsko opened this issue Jan 14, 2020 · 0 comments · Fixed by #1518
Assignees
Labels
bug Something isn't working chain db consensus issues related to ouroboros-consensus
Milestone

Comments

@edsko
Copy link
Contributor

edsko commented Jan 14, 2020

At the moment there are two processes happening in the node that are unaware of each other:

  1. There is a thread moving blocks from the vol DB to the imm DB.
  2. There is a thread taking period on-disk snapshots of the ledger DB.

The problem is that if the latter thread takes a snapshot, it might consider a block to be immutable (and hence use it for the snapshot) which hasn't yet been copied into the imm DB by the former thread. When this happens, on start up the node will be unable to use this ledger snapshot and will have to default to an older one (#1454).

The problem is exacerbated by the fact that the node now takes a ledger snapshot on shutdown, without also doing a final volatile to immutable DB copy. It's therefore not all that unlikely to happen. What I think needs to happen is

  1. Only take a ledger DB snapshot after copying blocks, making sure that there aren't any race conditions (in other words, decide which ledger snapshot to take, then copy to imm DB, then write the snapshot).
  2. Don't take the final snapshot on node shutdown, and instead adjust the snapshot policy (Revise ledger DB disk snapshot taking policy #1264).
@edsko edsko added bug Something isn't working consensus issues related to ouroboros-consensus labels Jan 14, 2020
@edsko edsko added this to the S5 2020-01-30 milestone Jan 14, 2020
edsko added a commit that referenced this issue Jan 28, 2020
edsko added a commit that referenced this issue Jan 28, 2020
edsko added a commit that referenced this issue Jan 28, 2020
iohk-bors bot added a commit that referenced this issue Jan 29, 2020
1518: Better ledger DB policy r=mrBliss a=edsko

Closes #1456
Closes #1264

Co-authored-by: Edsko de Vries <[email protected]>
@iohk-bors iohk-bors bot closed this as completed in 67ec998 Jan 29, 2020
@iohk-bors iohk-bors bot closed this as completed in #1518 Jan 29, 2020
redxaxder pushed a commit that referenced this issue Feb 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working chain db consensus issues related to ouroboros-consensus
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant