Add option to use RIBS as Motion blob store #26
Conversation
| return nil, err | ||
| } | ||
| } | ||
| if err := batch.Flush(ctx); err != nil { |
There was a problem hiding this comment.
I notice you're not storing the UnixFS intermediate tree (that would produce a RootCID for the whole file). I dunno if that matters.
There was a problem hiding this comment.
I don't believe it matters until we start picking apart what RIBS does internally. Right now at the level of abstraction we are working with in this PR there are just chunks in form of IPFS blocks and their order.
Implement an experimental option to use RIBS as Motion `blob.Store`. The implementation uses a flat-file directory to store an index of Motion `blob.ID` to IPFS blocks stored by RIBS. RIBS would then accumulate the stored information and makes deals in the background. The SP selection and wallet configuration is hard coded in RIBS. The FileCoin wallet is instantiated automatically if it does not exist at `~/.ribswallet`. Making this configurable requires changes in RIBS. Retrieval API implements an `io.ReadSeekCloser` which allows range reads over a list of CIDs. Offloading onto S3 compatible APIS, like AWS S3 and CloudFlare R2 is supported by RIBS and activated if specific environment variables are set. See: https://github.com/FILCAT/ribs/blob/0d5f0580effc112b01fe676f080bdbf1c2c7568a/rbdeal/car_offload_s3.go#L28 The RIBS storage is disabled by default. It is enabled by setting the option `--experimentalRibsStore`. This implementation is experimental: it should not be used in Production environment. It is implemented to facilitate experimentation and scoping of the Motion project. Disable 32-bit tests due to RIBS dependencies.
masih
force-pushed
the
masih/smoking_ribs
branch
from
3d3c35b to
08fa06e
Compare
|
may be worth making the wallet extensibility change to ribs. i suspect it will be accepted, but we'd probably want to set that through our own API rather than having it change over time |
Yup I just got write access to ribs repo and doing that now. |
|
lets still default to merging quickly, and happy to see this merged. |
The flag that enables it (disabled by default) is |
TorfinnOlsen
added
M2.0
Implement an experimental option to use Singularity as Motion blob.Store. A singularity blob store for now is a local store + a singularity connector For a put operation, the file is saved to the local blob store, then the local file is added to Singularity for scanning (as an item). The returned ID is the ID generated by singularity, rather than the UUID of the file (note: this changes the ID type to a raw string -- currently singularity uses integer indexes for each item -- @xinaxu maybe has an opinion about whether Singularity might either convert IDs to UUIDs or add a UUID at least for Item -- alternatively, we can make a single motion ID -> singularity primary key mapping in a file held in memory). For a get operation, the id is sent to Singularity to retrieve metadata about the file, and then the file is retrieved from the local store. (for real retrieval with Filecoin, we could have the GetItem call to Singularity also hydrate the local store, though this may not be the most performant option) Note that all this really does so far is start creating entities within Singularities database and scan files into Items & ItemParts. I haven't attempted to start data prep workers or make filecoin deals. Also, currently the singularity connector is implemented with a direct connection to an SQLite DB setup for Singularity, but could also be implemented over the singularity API. Per the thinking in #24, what may make most sense is to get #26 fully working to allow partners to try storing deals on Filecoin, while we continue to build this solution out and harden/productionize Singularity's deal making.
Implement an experimental option to use RIBS as Motion
blob.Store. The implementation uses a flat-file directory to store an index of Motionblob.IDto IPFS blocks stored by RIBS.RIBS would then accumulate the stored information and makes deals in the background. The SP selection and wallet configuration is hard coded in RIBS. The FileCoin wallet is instantiated automatically if it does not exist at
~/.ribswallet. Making this configurable requires changes in RIBS.Retrieval API implements an
io.ReadSeekCloserwhich allows range reads over a list of CIDs.Offloading onto S3 compatible APIS, like AWS S3 and CloudFlare R2 is supported by RIBS and activated if specific environment variables are set. See: https://github.com/FILCAT/ribs/blob/0d5f0580effc112b01fe676f080bdbf1c2c7568a/rbdeal/car_offload_s3.go#L28
The RIBS storage is disabled by default. It is enabled by setting the option
--experimentalRibsStore.This implementation is experimental: it should not be used in Production environment. It is implemented to facilitate experimentation and scoping of the Motion project.