Create nmt inclusion proofs for transactions in a given block

[evan-forbes]

Proving transactions are included in a block using nmt proofs

How this is currently done

celestia-core/rpc/core/tx.go

Lines 39 to 42 in 863a00a

	if prove {
	block := env.BlockStore.LoadBlock(height)
	proof = block.Data.Txs.Proof(int(index)) // XXX: overflow on 32-bit machines
	}

What we changed

celestia-core/rpc/core/tx.go

Lines 40 to 47 in 4c7092e

	var txProof types.TxProof
	if proveTx {
	block := env.BlockStore.LoadBlock(height)
	txProof, err = prove.ProveTxInclusion(consts.DefaultCodec(), block.Data, int(block.Data.OriginalSquareSize), int(r.Index))
	if err != nil {
	return nil, err
	}
	}

celestia-core doesn't store block data in the format that we commit to, so we have to regenerate that data if we want to use it to generate a proof. Since the transactions are always in the beginning of the square, we don't have to regenerate the entire extended data square, we only have to regenerate the row that the transaction is in. We do this by calculating the position of the transaction in the square, and then progressively filling in the rest of the row by using the other block data.

celestia-core/pkg/prove/proof.go

Lines 137 to 178 in 4c7092e

	// genOrigRowShares progressively generates data square rows for the original
	// data square, meaning the rows only half the full square length, as there is
	// not erasure data
	func genOrigRowShares(data types.Data, originalSquareSize, startRow, endRow int) [][]byte {
	wantLen := (endRow + 1) * originalSquareSize
	startPos := startRow * originalSquareSize
	shares := data.Txs.SplitIntoShares()
	// return if we have enough shares
	if len(shares) >= wantLen {
	return shares[startPos:wantLen].RawShares()
	}
	shares = append(shares, data.IntermediateStateRoots.SplitIntoShares()...)
	if len(shares) >= wantLen {
	return shares[startPos:wantLen].RawShares()
	}
	shares = append(shares, data.Evidence.SplitIntoShares()...)
	if len(shares) >= wantLen {
	return shares[startPos:wantLen].RawShares()
	}
	for _, m := range data.Messages.MessagesList {
	rawData, err := m.MarshalDelimited()
	if err != nil {
	panic(fmt.Sprintf("app accepted a Message that can not be encoded %#v", m))
	}
	shares = types.AppendToShares(shares, m.NamespaceID, rawData)
	// return if we have enough shares
	if len(shares) >= wantLen {
	return shares[startPos:wantLen].RawShares()
	}
	}
	tailShares := types.TailPaddingShares(wantLen - len(shares))
	shares = append(shares, tailShares...)
	return shares[startPos:wantLen].RawShares()
	}

we also switched to using nmt proofs

celestia-core/proto/tendermint/types/types.pb.go

Lines 1606 to 1624 in 62e497a

	// Proof represents proof of a namespace.ID in an NMT.
	// In case this proof proves the absence of a namespace.ID
	// in a tree it also contains the leaf hashes of the range
	// where that namespace would be.
	type NMTProof struct {
	// start index of this proof.
	Start int32 `protobuf:"varint,1,opt,name=start,proto3" json:"start,omitempty"`
	// end index of this proof.
	End int32 `protobuf:"varint,2,opt,name=end,proto3" json:"end,omitempty"`
	// Nodes that together with the corresponding leaf values
	// can be used to recompute the root and verify this proof.
	Nodes [][]byte `protobuf:"bytes,3,rep,name=nodes,proto3" json:"nodes,omitempty"`
	// leafHash are nil if the namespace is present in the NMT.
	// In case the namespace to be proved is in the min/max range of
	// the tree but absent, this will contain the leaf hash
	// necessary to verify the proof of absence.
	LeafHash []byte `protobuf:"bytes,4,opt,name=leaf_hash,json=leafHash,proto3" json:"leaf_hash,omitempty"`
	}

celestia-core/types/tx.go

Lines 111 to 130 in 62e497a

	func (tp *TxProof) VerifyProof() bool {
	for i, proof := range tp.Proofs {
	nmtProof := nmt.NewInclusionProof(
	int(proof.Start),
	int(proof.End),
	proof.Nodes,
	true,
	)
	valid := nmtProof.VerifyInclusion(
	consts.NewBaseHashFunc(),
	consts.TxNamespaceID,
	tp.Data[i],
	tp.RowRoots[i],
	)
	if !valid {
	return false
	}
	}
	return true
	}

Closes #416

[lgtm-com]

This pull request introduces 1 alert when merging 72273fd into 863a00a - view on LGTM.com

new alerts:

• 1 for Off-by-one comparison against length

[Wondertan]

Typo in file name. Currently ‘postion.go’, but I guess should be ‘position.go’

[evan-forbes]

renamed for clarity

[evan-forbes]

Since we don't store the data in the format that we commit to or the square size for each block, we cash the original square size in the data after we generate it. This is then saved and loaded when loading the block to serve a TxProof request.

We can change this in the future by saving adding things to BlockMeta or something similar.

[evan-forbes]

It's possible for a transaction to span multiple rows, and therefore we now need to potentially prove two or more proofs

[evan-forbes]

must have forgot to pull this from #546, which is very important to stop the node from panicking when attempting to generate the data availability header. Otherwise, it's possible to to order message so that their namespaces are not in lexicographical order. tbh, idek how devnet is still alive

[evan-forbes]

deleted this because I also deleted the old way to create proofs for txs. We can keep both though if we want.

[evan-forbes]

idk how this was 15,098 loc. looking into this

[evan-forbes]

will change this to take a slice of row roots as args, and not use the roots already stored in the TxProof