Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: events: Improve performance of event migration from V1 to V2 #11064

Merged
merged 1 commit into from
Jul 11, 2023

Conversation

fridrik01
Copy link
Contributor

@fridrik01 fridrik01 commented Jul 11, 2023

Related Issues

Fixes: #11056

Proposed Changes

Added a few temporary indices to help speed up the migration.

Also, after the migration finishes successfully, I perform some manually DB administration to free up disk space (reducing size on disk from 3GB to 0.5GB).

Additional Info

Test plan

I used the large events DB that @TippyFlitsUK had generated and shared with me (link) as a benchmark when testing out this PR. Ths db contained 5.9mio entries and 21.6mio event entries.

Using this events DB and starting lotus resulted in migration from V1 to V2 in about 55 seconds:

./lotus daemon
...
2023-07-11T10:17:53.644Z        INFO    filter  filter/index.go:166     Migrating events from head to 2683360
2023-07-11T10:17:53.652Z        INFO    filter  filter/index.go:172     Migrating height 3022000 (remaining 338640)
2023-07-11T10:17:53.734Z        INFO    filter  filter/index.go:172     Migrating height 3021000 (remaining 337640)
2023-07-11T10:17:53.812Z        INFO    filter  filter/index.go:172     Migrating height 3020000 (remaining 336640)
2023-07-11T10:17:53.890Z        INFO    filter  filter/index.go:172     Migrating height 3019000 (remaining 335640)
...
2023-07-11T10:18:32.813Z        INFO    filter  filter/index.go:172     Migrating height 2688000 (remaining 4640)
2023-07-11T10:18:32.979Z        INFO    filter  filter/index.go:172     Migrating height 2687000 (remaining 3640)
2023-07-11T10:18:33.269Z        INFO    filter  filter/index.go:172     Migrating height 2686000 (remaining 2640)
2023-07-11T10:18:33.380Z        INFO    filter  filter/index.go:172     Migrating height 2685000 (remaining 1640)
2023-07-11T10:18:33.459Z        INFO    filter  filter/index.go:172     Migrating height 2684000 (remaining 640)
2023-07-11T10:18:39.387Z        INFO    filter  filter/index.go:232     cleaned up 18953979 entries that had deleted events
2023-07-11T10:18:41.799Z        INFO    filter  filter/index.go:252     Performing DB vacuum and wal checkpointing to free up space after the migration
2023-07-11T10:18:43.082Z        INFO    filter  filter/index.go:262     Successfully migrated events to version 2 in 55.423608953s

After the migration, the events DB contained 0,7mio (737494) entries and 2.7mio (2705234) event entries and the size on disk went from 3GB to 0.5GB.

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@fridrik01 fridrik01 force-pushed the 11056-speedup-event-migration branch 2 times, most recently from 0fdd47f to 599f5b8 Compare July 11, 2023 10:00
After the migration we also perform db administration on the
events DB to reduce its file size on disk.
@fridrik01 fridrik01 force-pushed the 11056-speedup-event-migration branch from 599f5b8 to 0d7621b Compare July 11, 2023 10:17
@fridrik01 fridrik01 marked this pull request as ready for review July 11, 2023 10:35
@fridrik01 fridrik01 requested a review from a team as a code owner July 11, 2023 10:35
@fridrik01 fridrik01 changed the title fix: Add tmp indices to events table while performing migration to V2 fix: events: Improve performance of event migration from V1 to V2 Jul 11, 2023
@arajasek
Copy link
Contributor

Fantastic, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Events: Migrating from v1 to v2 is slow
2 participants