feat:profiling:state summary and visualization #11012
Conversation
| closer func() | ||
| } | ||
|
|
||
| func loadChainStore(ctx context.Context, repoPath string) (*StoreHandle, error) { |
There was a problem hiding this comment.
Probably could have a mode where this loads a snapshot to a memory-map blockstore, would need a decent amount of RAM, but should be much faster
|
Having this in two branches is causing an issue on the deployment infra. I think this code is good to merge, and all of the comments above are good suggestions that we should open a second issue for. |
yup, I just need a green check mark, can you approve @snissn ? |
| type StoreHandle struct { | ||
| bs blockstore.Blockstore | ||
| cs *store.ChainStore | ||
| sm *stmgr.StateManager |
There was a problem hiding this comment.
Can we drop the StateManager here? Conceptually, there isn't any state management you're doing, just a lot of reading of the StateTree. I suspect you'll be able to easily replace calls to sm.StateTree or sm.LoadActor with equivalent operations on the StateTree itslef.
There was a problem hiding this comment.
Motivation here is that it tends to make things a little more future-proof, and drops a lot of unrelated logic that the StateManager has (migrations, drand, network versions, etc.)
| }, | ||
| &cli.BoolFlag{ | ||
| Name: "pretty", | ||
| Usage: "print formated output instead of ldjson", |
There was a problem hiding this comment.
| Usage: "print formated output instead of ldjson", | |
| Usage: "print formatted output instead of ldjson", |
Related Issues
#10884
Proposed Changes
Additional Info
Testplan
I've done many manual runs on calibration and mainnet repos and both lotus-shed and python script generate plausible output. Ive cross checked output data and it appears legit matching lotus chain store size + providing an explanation for pre nv19 snapshot bloat. I think this is good enough for running a monitoring service. If we find problems we can fix them as they show up.
Speed
While reviewing please keep an eye out for optimization opportunities in the shed command, today it takes about 5 hours to run so a 20% improvement is an hour saved.
Halp
@jennijuju where should documentation for this go? I have lotus-shed command help but probably we want something a little more thorough. Is https://github.com/filecoin-project/lotus/discussions/categories/tutorials enough?
Extension ideas
I am hoping to tweak the python script to take in visualization chart type information so we can visualize same data in a few different ways.
State summary shed command is readily extensible by using a path identifier for protocol areas. We could extend this to breakdown message usage by message type for example.
We could use similar visualization on gas traces. In particular cron gas traces would be useful to keep track of in the same way.
Checklist
Before you mark the PR ready for review, please make sure that:
<PR type>: <area>: <change being made>fix: mempool: Introduce a cache for valid signaturesPR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, testarea, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps