Nexus: Account for all storage on U.2s

[smklein]

In the context of the "Support Bundles" project, it is currently difficult for Nexus to answer the question "will a particular U.2 have enough space to store a support bundle of a particular size?".

More generally, Nexus may ask "If I provision a resource to a U.2, and I want it to have some minimum / maximum space, will it fit?

We need the following tools to answer this question:

1. Nexus wants Sled Agent: API to manage datasets independently of zones #6042 to request datasets explicitly, rather than implicitly. This should apply to all datasets (see: Cleanup: Use explicit dataset request API to request zone datasets #6043)
2. Nexus needs Sled Agent: Allow the setting of quotas and reservations on datasets #6107 , to know that provisioned datasets won't overflow their expected capacities.
3. Nexus would like to ensure that all storage usage within U.2s is tracked, to some degree. This depends on e.g. Dataset accounting: VMM zone filesystems should be explicitly requested #6108, Dataset accounting: Probe zone filesystems should be explicitly requested #6109
4. Nexus needs to keep track of the expected quotas/reservations given to each U.2, and compare the usages of "all datasets on a zpool, so far" when making adjustments to the allocations.

It would also be extremely nice to have commands through omdb that can display these numbers, and give context on what they mean.

As prior art, the "region allocation" query for crucible accounts for some of this information: https://github.com/oxidecomputer/omicron/blob/main/nexus/db-queries/src/db/queries/region_allocation.rs

• It considers the "zpool size usage" to be "the sum of all datasets on that pool". This currently only considers other Crucibles, so it's not a true picture of the world, but an approximate one:
  

  omicron/nexus/db-queries/src/db/queries/region_allocation.rs

  Lines 121 to 125 in 89b998c

  	old_zpool_usage AS (
  	SELECT
  	dataset.pool_id,
  	sum(dataset.size_used) AS size_used
  	FROM dataset WHERE ((dataset.size_used IS NOT NULL) AND (dataset.time_deleted IS NULL)) GROUP BY dataset.pool_id),");

• It considers the "zpool capacity size" based on output from inventory, and checks that "old usage + proposed new usage <= total usage":
  

  omicron/nexus/db-queries/src/db/queries/region_allocation.rs

  Lines 175 to 178 in 89b998c

  	(old_zpool_usage.size_used + ").param().sql(" ) <=
  	(SELECT total_size FROM omicron.public.inv_zpool WHERE
  	inv_zpool.id = old_zpool_usage.pool_id
  	ORDER BY inv_zpool.time_collected DESC LIMIT 1)