[FR] - More flexible database storage configuration options

[gitmachtl]

The database on disc is constantly growing, at the time of writing its around 190GB.

Many SPOs using VPS instances with a fixed size base volume, new volumes can be added most of the time.

We have different kinds of data in the database: There are the volatile ones and the ledger state which require fast access times. On the other hand most of the database is made up of the immutable data, which is accessed far less often. These data could be stored on a cheaper tiered volume.

There might be some tricks around currently to achieve the same result, by using symbolic links that are generated dynamically via extra scripts. But all in all, thats not a very managable solution for the regular SPO imo.

So, as the database grows and grows - especially the immutable - i would like to request a new feature to do a more specific storage configuration. Also with multiple storage locations for the immutable database, including an optional maximum size cap.

A new config section in the config.json would be nice, kind of like:

{
  "database": {
    "ledger": "/home/user/faststorage/db",
    "volatile": "/home/user/faststorage/db",
    "immutable": [
      {
        "path": "/home/user/slowstorage/db",
        "maxSizeGB": 100
      },
      {
        "path": "/home/user/addedstorage/db2",
        "maxSizeGB": 60
      },
      {
        "path": "/home/user/anotherstorage/db",
        "maxSizeGB": -1
      }
    ]
  }
}

To be able to specify fast storage locations for the ledger and volatile database. And to be able to define a set of storages that is filled up with the immutable data. A size cap would be nice, so you can specify that the first storage is filled up with 100GB in that example, after that is full the node uses the next storage path and fills it up until 60GB are reached. And after that it fills up the next storage location with a disabled size cap.

Also the question of being able to share a common immutable database comes up quite often. I know that this is a difficult task, because who decides to write to the database, and who is just only reading it. But that would be another task imo. This FR is for a single node configuration.

Best regards,
Martin

[gitmachtl]

After some chat with @disassembler , there might be an option with the

--ssd-database-dir FILEPATH
                           Directory where the LMDB is stored.

in the upcoming utxohd branch, to set at least a different path (if needed) for the immutable and the volatile/ledger directory.

[jasagredo]

Just wanted to mention that the utxohd flags --ssd-*-dir will move where the LMDB directory is located, or where we store snapshots from the ledger directory. It is not exactly the same as you are requesting.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[jasagredo]

The PR #5944 merged this functionality into the node.