Abstract database to work with raw bytes instead of types #1548

xgreenx · 2023-12-14T16:39:15Z

Overview

With each Fuel node, the data from the blockchain is stored in a database. The tables on the database expect specific types, so as the blockchain evolves and those types change, we need the database to accept updated and new types.

The most general way of doing this is to always store the data as raw bytes, rather than strong types, and as those types change shape they can just be serialized accordingly.

Design Thoughts/Questions

What happens if a network update requires new tables added to the db, not just changes in the shape stored on the tables?
What is stored on db is rawbytes, but the interface for the DB should just be a non_exhaustive enum right? We don't want to be stringly typed at our interface (port).

The text was updated successfully, but these errors were encountered:

MitchTurner · 2023-12-19T00:10:43Z

Could the database types not just be an non_exhaustive enum instead of bytes? Maybe there's not a meaningful difference between that and what is proposed above since it all gets serialized anyway?

Preparation before start work on #1548. The `KeyValueStore` trait has some duplicated logic. This PR removes it, minimizing the number of methods that we need to implement. Also I applied the original ordering of the method as in the trait.

Related work to the #1548. The changes move `KeyValueStore` to the `fuel-core-storage` crate. It requires updating the trait to use `StorageResult` instead of `DatabaseResult`, causing according to changes in the downstream crates. Also extracted `iter_all` functionality into a separate trait, because it is not used by the state transition logic and more fancy stuff for API.

…1576) ## Overview Closes #1548 Closes #430 The change moves the implementation of the storage traits for required tables from `fuel-core` to `fuel-core-storage` crate. The change also adds a more flexible configuration of the encoding/decoding per the table and allows the implementation of specific behaviors for the table in a much easier way. It unifies the encoding between database, SMTs, and iteration, preventing mismatching bytes representation on the Rust type system level. Plus, it increases the re-usage of the code by applying the same structure to other tables. It is a breaking PR because it changes database encoding/decoding for some tables. ### StructuredStorage The change adds a new type `StructuredStorage`. It is a wrapper around the key-value storage that implements the storage traits(`StorageInspect`, `StorageMutate`, `StorageRead`, etc) for the tables with structure. This structure works in tandem with the `TableWithStructure` trait. The table may implement `TableWithStructure` specifying the structure, as an example: ```rust impl TableWithStructure for ContractsRawCode { type Structure = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } ``` It is a definition of the structure for the `ContractsRawCode` table. It has a plain structure meaning it simply encodes/decodes bytes and stores/loads them into/from the storage. As a key codec and value codec, it uses a `Raw` encoding/decoding that simplifies writing bytes and loads them back into the memory without applying any serialization or deserialization algorithm. If the table implements `TableWithStructure` and the selected codec satisfies all structure requirements, the corresponding storage traits for that table are implemented on the `StructuredStorage` type. ### Codecs Each structure allows customizing the key and value codecs. It allows the use of different codecs for different tables, taking into account the complexity and weight of the data and providing a way of more optimal implementation. That property may be very useful to perform migration in a more easier way. Plus, it also can be a `no_std` migration potentially allowing its fraud proving. An example of migration: ```rust /// Define the table for V1 value encoding/decoding. impl TableWithStructure for ContractsRawCodeV1 { type Structure = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } /// Define the table for V2 value encoding/decoding. /// It uses `Postcard` codec for the value instead of `Raw` codec. /// /// # Dev-note: The columns is the same. impl TableWithStructure for ContractsRawCodeV2 { type Structure = Plain<Raw, Postcard>; fn column() -> Column { Column::ContractsRawCode } } fn migration(storage: &mut Database) { let mut iter = storage.iter_all::<ContractsRawCodeV1>(None); while let Ok((key, value)) = iter.next() { // Insert into the same table but with another codec. storage.storage::<ContractsRawCodeV2>().insert(key, value); } } ``` ### Structures The structure of the table defines its behavior. As an example, a `Plain` structure simply encodes/decodes bytes and stores/loads them into/from the storage. The `SMT` structure builds a sparse merkle tree on top of the key-value pairs. Implementing a structure one time, we can apply it to any table satisfying the requirements of this structure. It increases the re-usage of the code and minimizes duplication. It can be useful if we decide to create global roots for all required tables that are used in fraud proving. ```rust impl TableWithStructure for SpentMessages { type Structure = Plain<Raw, Postcard>; fn column() -> Column { Column::SpentMessages } } | | \|/ impl TableWithStructure for SpentMessages { type Structure = Sparse<Raw, Postcard, SpentMessagesMerkleMetadata, SpentMessagesMerkleNodes>; fn column() -> Column { Column::SpentMessages } } ``` ### Side changes #### `iter_all` The `iter_all` functionality now accepts the table instead of `K` and `V` generics. It is done to use the correct codec during deserialization. Also, the table definition provides the column. <img width="1234" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/74595ee2-bbd2-48a1-b0da-edf47abd7a4f"> #### Duplicated unit tests The `fuel-core-storage` crate provides macros that generate unit tests. Almost all tables had the same test like `get`, `insert`, `remove`, `exist`. All duplicated tests were moved to macros. The unique one still stays at the same place where it was before. <img width="679" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/a4eb1fd9-c008-4ab0-902a-ab1fdbc855a8"> #### `StorageBatchMutate` Added a new `StorageBatchMutate` trait that we can move to `fuel-storage` crate later. It allows batch operations on the storage. It may be more performant in some cases. ```rust /// The traits allow work with the storage in batches. /// Some implementations can perform batch operations faster than one by one. pub trait StorageBatchMutate<Type: Mappable>: StorageMutate<Type> { /// Initialize the storage with batch insertion. This method is more performant than /// [`Self::insert_batch`] in some case. /// /// # Errors /// /// Returns an error if the storage is already initialized. fn init_storage( &mut self, set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>, ) -> Result<()>; /// Inserts the key-value pair into the storage in batch. fn insert_batch( &mut self, set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>, ) -> Result<()>; /// Removes the key-value pairs from the storage in batch. fn remove_batch(&mut self, set: &mut dyn Iterator<Item = &Type::Key>) -> Result<()>; } ``` ### Follow-up It is one of the changes in the direction of the forkless upgrades for state transition functions and fraud proofs. The idea behind this is that the `fuel_core_executor::Executor` will work directly with the `StructuredStorage` instead of the `Database`. It will perform only state transition-related modifications to the storage, while all outside modifications like updating of receipts, transition status, block insertions, messages removing, and transaction storing will be a part of another service/process.

Preparation before start work on FuelLabs/fuel-core#1548. The `KeyValueStore` trait has some duplicated logic. This PR removes it, minimizing the number of methods that we need to implement. Also I applied the original ordering of the method as in the trait.

Related work to the FuelLabs/fuel-core#1548. The changes move `KeyValueStore` to the `fuel-core-storage` crate. It requires updating the trait to use `StorageResult` instead of `DatabaseResult`, causing according to changes in the downstream crates. Also extracted `iter_all` functionality into a separate trait, because it is not used by the state transition logic and more fancy stuff for API.

…#1576) ## Overview Closes FuelLabs/fuel-core#1548 Closes FuelLabs/fuel-core#430 The change moves the implementation of the storage traits for required tables from `fuel-core` to `fuel-core-storage` crate. The change also adds a more flexible configuration of the encoding/decoding per the table and allows the implementation of specific behaviors for the table in a much easier way. It unifies the encoding between database, SMTs, and iteration, preventing mismatching bytes representation on the Rust type system level. Plus, it increases the re-usage of the code by applying the same structure to other tables. It is a breaking PR because it changes database encoding/decoding for some tables. ### StructuredStorage The change adds a new type `StructuredStorage`. It is a wrapper around the key-value storage that implements the storage traits(`StorageInspect`, `StorageMutate`, `StorageRead`, etc) for the tables with structure. This structure works in tandem with the `TableWithStructure` trait. The table may implement `TableWithStructure` specifying the structure, as an example: ```rust impl TableWithStructure for ContractsRawCode { type Structure = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } ``` It is a definition of the structure for the `ContractsRawCode` table. It has a plain structure meaning it simply encodes/decodes bytes and stores/loads them into/from the storage. As a key codec and value codec, it uses a `Raw` encoding/decoding that simplifies writing bytes and loads them back into the memory without applying any serialization or deserialization algorithm. If the table implements `TableWithStructure` and the selected codec satisfies all structure requirements, the corresponding storage traits for that table are implemented on the `StructuredStorage` type. ### Codecs Each structure allows customizing the key and value codecs. It allows the use of different codecs for different tables, taking into account the complexity and weight of the data and providing a way of more optimal implementation. That property may be very useful to perform migration in a more easier way. Plus, it also can be a `no_std` migration potentially allowing its fraud proving. An example of migration: ```rust /// Define the table for V1 value encoding/decoding. impl TableWithStructure for ContractsRawCodeV1 { type Structure = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } /// Define the table for V2 value encoding/decoding. /// It uses `Postcard` codec for the value instead of `Raw` codec. /// /// # Dev-note: The columns is the same. impl TableWithStructure for ContractsRawCodeV2 { type Structure = Plain<Raw, Postcard>; fn column() -> Column { Column::ContractsRawCode } } fn migration(storage: &mut Database) { let mut iter = storage.iter_all::<ContractsRawCodeV1>(None); while let Ok((key, value)) = iter.next() { // Insert into the same table but with another codec. storage.storage::<ContractsRawCodeV2>().insert(key, value); } } ``` ### Structures The structure of the table defines its behavior. As an example, a `Plain` structure simply encodes/decodes bytes and stores/loads them into/from the storage. The `SMT` structure builds a sparse merkle tree on top of the key-value pairs. Implementing a structure one time, we can apply it to any table satisfying the requirements of this structure. It increases the re-usage of the code and minimizes duplication. It can be useful if we decide to create global roots for all required tables that are used in fraud proving. ```rust impl TableWithStructure for SpentMessages { type Structure = Plain<Raw, Postcard>; fn column() -> Column { Column::SpentMessages } } | | \|/ impl TableWithStructure for SpentMessages { type Structure = Sparse<Raw, Postcard, SpentMessagesMerkleMetadata, SpentMessagesMerkleNodes>; fn column() -> Column { Column::SpentMessages } } ``` ### Side changes #### `iter_all` The `iter_all` functionality now accepts the table instead of `K` and `V` generics. It is done to use the correct codec during deserialization. Also, the table definition provides the column. <img width="1234" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/74595ee2-bbd2-48a1-b0da-edf47abd7a4f"> #### Duplicated unit tests The `fuel-core-storage` crate provides macros that generate unit tests. Almost all tables had the same test like `get`, `insert`, `remove`, `exist`. All duplicated tests were moved to macros. The unique one still stays at the same place where it was before. <img width="679" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/a4eb1fd9-c008-4ab0-902a-ab1fdbc855a8"> #### `StorageBatchMutate` Added a new `StorageBatchMutate` trait that we can move to `fuel-storage` crate later. It allows batch operations on the storage. It may be more performant in some cases. ```rust /// The traits allow work with the storage in batches. /// Some implementations can perform batch operations faster than one by one. pub trait StorageBatchMutate<Type: Mappable>: StorageMutate<Type> { /// Initialize the storage with batch insertion. This method is more performant than /// [`Self::insert_batch`] in some case. /// /// # Errors /// /// Returns an error if the storage is already initialized. fn init_storage( &mut self, set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>, ) -> Result<()>; /// Inserts the key-value pair into the storage in batch. fn insert_batch( &mut self, set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>, ) -> Result<()>; /// Removes the key-value pairs from the storage in batch. fn remove_batch(&mut self, set: &mut dyn Iterator<Item = &Type::Key>) -> Result<()>; } ``` ### Follow-up It is one of the changes in the direction of the forkless upgrades for state transition functions and fraud proofs. The idea behind this is that the `fuel_core_executor::Executor` will work directly with the `StructuredStorage` instead of the `Database`. It will perform only state transition-related modifications to the storage, while all outside modifications like updating of receipts, transition status, block insertions, messages removing, and transaction storing will be a part of another service/process.

Preparation before start work on FuelLabs/fuel-core#1548. The `KeyValueStore` trait has some duplicated logic. This PR removes it, minimizing the number of methods that we need to implement. Also I applied the original ordering of the method as in the trait.

Related work to the FuelLabs/fuel-core#1548. The changes move `KeyValueStore` to the `fuel-core-storage` crate. It requires updating the trait to use `StorageResult` instead of `DatabaseResult`, causing according to changes in the downstream crates. Also extracted `iter_all` functionality into a separate trait, because it is not used by the state transition logic and more fancy stuff for API.

…#1576) ## Overview Closes FuelLabs/fuel-core#1548 Closes FuelLabs/fuel-core#430 The change moves the implementation of the storage traits for required tables from `fuel-core` to `fuel-core-storage` crate. The change also adds a more flexible configuration of the encoding/decoding per the table and allows the implementation of specific behaviors for the table in a much easier way. It unifies the encoding between database, SMTs, and iteration, preventing mismatching bytes representation on the Rust type system level. Plus, it increases the re-usage of the code by applying the same structure to other tables. It is a breaking PR because it changes database encoding/decoding for some tables. ### StructuredStorage The change adds a new type `StructuredStorage`. It is a wrapper around the key-value storage that implements the storage traits(`StorageInspect`, `StorageMutate`, `StorageRead`, etc) for the tables with structure. This structure works in tandem with the `TableWithStructure` trait. The table may implement `TableWithStructure` specifying the structure, as an example: ```rust impl TableWithStructure for ContractsRawCode { type Structure = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } ``` It is a definition of the structure for the `ContractsRawCode` table. It has a plain structure meaning it simply encodes/decodes bytes and stores/loads them into/from the storage. As a key codec and value codec, it uses a `Raw` encoding/decoding that simplifies writing bytes and loads them back into the memory without applying any serialization or deserialization algorithm. If the table implements `TableWithStructure` and the selected codec satisfies all structure requirements, the corresponding storage traits for that table are implemented on the `StructuredStorage` type. ### Codecs Each structure allows customizing the key and value codecs. It allows the use of different codecs for different tables, taking into account the complexity and weight of the data and providing a way of more optimal implementation. That property may be very useful to perform migration in a more easier way. Plus, it also can be a `no_std` migration potentially allowing its fraud proving. An example of migration: ```rust /// Define the table for V1 value encoding/decoding. impl TableWithStructure for ContractsRawCodeV1 { type Structure = Plain<Raw, Raw>; fn column() -> Column { Column::ContractsRawCode } } /// Define the table for V2 value encoding/decoding. /// It uses `Postcard` codec for the value instead of `Raw` codec. /// /// # Dev-note: The columns is the same. impl TableWithStructure for ContractsRawCodeV2 { type Structure = Plain<Raw, Postcard>; fn column() -> Column { Column::ContractsRawCode } } fn migration(storage: &mut Database) { let mut iter = storage.iter_all::<ContractsRawCodeV1>(None); while let Ok((key, value)) = iter.next() { // Insert into the same table but with another codec. storage.storage::<ContractsRawCodeV2>().insert(key, value); } } ``` ### Structures The structure of the table defines its behavior. As an example, a `Plain` structure simply encodes/decodes bytes and stores/loads them into/from the storage. The `SMT` structure builds a sparse merkle tree on top of the key-value pairs. Implementing a structure one time, we can apply it to any table satisfying the requirements of this structure. It increases the re-usage of the code and minimizes duplication. It can be useful if we decide to create global roots for all required tables that are used in fraud proving. ```rust impl TableWithStructure for SpentMessages { type Structure = Plain<Raw, Postcard>; fn column() -> Column { Column::SpentMessages } } | | \|/ impl TableWithStructure for SpentMessages { type Structure = Sparse<Raw, Postcard, SpentMessagesMerkleMetadata, SpentMessagesMerkleNodes>; fn column() -> Column { Column::SpentMessages } } ``` ### Side changes #### `iter_all` The `iter_all` functionality now accepts the table instead of `K` and `V` generics. It is done to use the correct codec during deserialization. Also, the table definition provides the column. <img width="1234" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/74595ee2-bbd2-48a1-b0da-edf47abd7a4f"> #### Duplicated unit tests The `fuel-core-storage` crate provides macros that generate unit tests. Almost all tables had the same test like `get`, `insert`, `remove`, `exist`. All duplicated tests were moved to macros. The unique one still stays at the same place where it was before. <img width="679" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/a4eb1fd9-c008-4ab0-902a-ab1fdbc855a8"> #### `StorageBatchMutate` Added a new `StorageBatchMutate` trait that we can move to `fuel-storage` crate later. It allows batch operations on the storage. It may be more performant in some cases. ```rust /// The traits allow work with the storage in batches. /// Some implementations can perform batch operations faster than one by one. pub trait StorageBatchMutate<Type: Mappable>: StorageMutate<Type> { /// Initialize the storage with batch insertion. This method is more performant than /// [`Self::insert_batch`] in some case. /// /// # Errors /// /// Returns an error if the storage is already initialized. fn init_storage( &mut self, set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>, ) -> Result<()>; /// Inserts the key-value pair into the storage in batch. fn insert_batch( &mut self, set: &mut dyn Iterator<Item = (&Type::Key, &Type::Value)>, ) -> Result<()>; /// Removes the key-value pairs from the storage in batch. fn remove_batch(&mut self, set: &mut dyn Iterator<Item = &Type::Key>) -> Result<()>; } ``` ### Follow-up It is one of the changes in the direction of the forkless upgrades for state transition functions and fraud proofs. The idea behind this is that the `fuel_core_executor::Executor` will work directly with the `StructuredStorage` instead of the `Database`. It will perform only state transition-related modifications to the storage, while all outside modifications like updating of receipts, transition status, block insertions, messages removing, and transaction storing will be a part of another service/process.

xgreenx added upgradability SDK team The issue is ready to be addressed by SDK team labels Dec 14, 2023

xgreenx assigned MitchTurner Dec 14, 2023

xgreenx mentioned this issue Dec 18, 2023

Remove duplicating logic in the KeyValueStore trait #1559

Merged

xgreenx mentioned this issue Dec 19, 2023

Move KeyValueStore to the fuel-core-storage crate #1566

Merged

xgreenx assigned xgreenx and unassigned MitchTurner Dec 21, 2023

xgreenx mentioned this issue Dec 24, 2023

Move storage traits implementation to the fuel-core-storage crate #1576

Merged

xgreenx assigned xgreenx and unassigned xgreenx Jan 8, 2024

xgreenx closed this as completed in #1576 Jan 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Abstract database to work with raw bytes instead of types #1548

Abstract database to work with raw bytes instead of types #1548

xgreenx commented Dec 14, 2023 •

edited by MitchTurner

Loading

MitchTurner commented Dec 19, 2023

Abstract database to work with raw bytes instead of types #1548

Abstract database to work with raw bytes instead of types #1548

Comments

xgreenx commented Dec 14, 2023 • edited by MitchTurner Loading

Overview

Design Thoughts/Questions

MitchTurner commented Dec 19, 2023

xgreenx commented Dec 14, 2023 •

edited by MitchTurner

Loading