Add keepInterval to PruningOptions to support more flexible pruning strategy

[adu-crypto]

Summary

We could introduce a KeepInterval field in PruningOptions to keep old version states with specific interval.

Problem Definition

the current pruning strategy is determined by PruningOption:

// PruningOptions defines the pruning strategy used when determining which
// heights are removed from disk when committing state.
type PruningOptions struct {
	// KeepRecent defines how many recent heights to keep on disk.
	KeepRecent uint64

	// Interval defines when the pruned heights are removed from disk.
	Interval uint64

	// Strategy defines the kind of pruning strategy. See below for more information on each.
	Strategy PruningStrategy
}

where Interval actually means at which height the pruning actions would happen.
Sometimes if we could prune the old versions in range with interval, we could save disk storage as well as serving the request for historical data by replaying blocks from the nearest version state.
We could do this by introducing a KeepInterval field in PruningOptions to keep old version states with specific interval.

if we could prune the old versions in range with interval, we could save disk storage as well as serving the request for historical data by replaying blocks from the nearest kept block.

Proposal

We could introduce a KeepInterval field in PruningOptions to keep old version states with specific interval.

[adu-crypto]

@p0mvn I'm not 100% sure about the current pruning logic, could you help confirm and give some advice?

[alexanderbez]

I'm not really following the proposal TBH. What does "range with interval" mean?

[p0mvn]

for historical data by replaying blocks from the nearest version state.
We could do this by introducing a KeepInterval field in PruningOptions

How would we replay if only some heights are kept while others are deleted? For example, if KeepInterval is 3. Then, we keep heights 3, 6, 9. but everything in between is removed and unavailable for replay.

To explain the reasons for having KeepInterval removed, it happened so because there was no known use case for it. Originally, it had been used to make sure that we don't accidentally prune a height, at which a snapshot is being concurrently taken. So the requirement was for snapshot-interval to be a multiple of pruning-keep-interval.

To mitigate this, we now auto-prune heights that are multiples of snapshot-interval after a snapshot is complete so that it does not pollute the database.

As a result, the need for pruning-keep-interval was eliminated, and we removed it. I don't know of any other prior use case for pruning-keep-interval.

Please let me know if that helps

[adu-crypto]

How would we replay if only some heights are kept while others are deleted? For example, if KeepInterval is 3. Then, we keep heights 3, 6, 9. but everything in between is removed and unavailable for replay.

Supposing we have old states kept at heights 3, 6, 9 and our purpose is to serve the request that queries Alice's balance at height 5, we could LoadVersion(3) to fetch Alice's balance at height 3 and apply txs in block 4 and 5 to find the balance change to return the correct balance, right?

[adu-crypto]

I'm not really following the proposal TBH. What does "range with interval" mean?

my words could be misleading, I actually means the pruning-keep-interval that keeps old version states at every Nth height.

[tac0turtle]

@adu-crypto could you explain the use case in which the intermediate states would be useful?

Looking at nodes it seems it would be more useful to use an archive node instead of intervals of state

[alexanderbez]

@p0mvn is correct -- there was no use case for keep-interval.

[adu-crypto]

Looking at nodes it seems it would be more useful to use an archive node instead of intervals of state

yes, archive node could be used to serve the historical data request, but archive node is extremely big in size and pruned full nodes could save a lot of the disk space.
We need some checkpoints as starting point to replay from.
Supposing we keep 1/2 of the states, we only need to replay 1/2 block on average as expectation to reach the desired state while saving us a lot of the disk space(no sure how much).

[adu-crypto]

or we could choose not to move persisted snapshot heights that are older than keep-recent to pruneHeights so we could use persisted snapshot heights as checkpoints.

[ValarDragon]

I would strongly prefer the architecture of making checkpoints from snapshots.

I think the right end-state of LSM-backed DB data layouts is that every state entry stores a 'version diff' (which consequently makes such checkpoints not that well suited in the same DB, and they should exist in a separate DB)

But what you could do if you had both (and wanted light client proofs) is:
store diff log in main LSM DB, keep checkpoints separately, very quickly apply diffs retrieved from main LSM DB, to the checkpoint, to get the desired intermittent state.

(Granted all of this is probably better architected more directly)

[alexanderbez]

I'm still not following what this issue is proposing exactly nor what is currently insufficient with the current pruning options.

[adu-crypto]

@alexanderbez This issue is trying to add a new pruning option to make checkpoints from history versions, because the current pruning just moves any old version states older than keep-recent to pruneHeights to be pruned even if it was a snapshot height.
But on the other hand, we could make checkpoints from persisted snapshot heights. So this way we don't need to add new pruning options but just making use of snapshot-interval option.

Anyway, as the latter solution should be better I would close this issue and try to propose a replaying mechanism for sdk states management to tradeoff between disk space and query performance.

what's your suggestion?

[adu-crypto]

I think the right end-state of LSM-backed DB data layouts is that every state entry stores a 'version diff' (which consequently makes such checkpoints not that well suited in the same DB, and they should exist in a separate DB)

@ValarDragon as for version diff db layouts, maybe we can get some inspirations from ethereum Erigon client storage design, but this could be a big change for store architecture.

1. Store state in plain kv db.
2. Store historical changesets explicitly to support versioning.
3. Rebuild ethereum compatible merkle trie root hashes on the fly [2].

[alexanderbez]

I recall @p0mvn doing all this work to ensure snapshot heights were never pruned until it was safe to do so...?

[tac0turtle]

this work is being captured in the store v2 design document. We decided to go with iavl snapshots to solve the itermediate states design