Direct VMDK manipulation investigation

[hickeng]

Investigate how we can directly manipulate VMDKs on ESX. The following high level tasks are of specific interest:

• Extract tar archive onto VMDK - presuming ext4 as the filesystem format should be suitable for linux rootfs
• Expand VMDK capacity, followed by resize filesystem - this should be a "rare" operation so could be handled via a block device attach to a service VM, but we should know if it's viable via direct operation.

[gigawhitlocks]

@hickeng, I had a discussion w/ the Lifecycle team regarding this, and received this response "you probably want to directly consume bits of disklib, ala diskTool. Writing useful state into the VMDK requires some tool that understands ext4. I vaguely recall the tools team having some code that inspects VMDK content invasively, but don't know the details"

Further asking about ext4 seems to indicate that there's no ext4 partitioning available on ESX, however this sort of thing could be done from inside of a Linux VM (I'm sure this is not surprising). Further discussion with Mike Hall seems to indicate that we might be out of luck as far as doing this in an application running directly on ESX but a service VM (or maybe just a process running on the VCH) should be able to perform these actions in a way that's more efficient/performant than our current approach, through the use of loopback block devices.

I tend to agree that performing these operations on the host directly would at the very least be considerably more challenging than performing them inside of a VM.

[hickeng]

We're currently performing the manipulation inside a VM, and that introduces either bottlenecks or significant throughput degradation; the latter could just require optimization, that's TBD.

Presenting the disk as a true block device in the VM requires attach/detach operations, which are serial and expensive as they entail a fast-suspend-resume, and you are limited to 4 scsi controllers and 15 disks per controller. You could partially address the serial element with clever operation batching, but the latency isn't going to improve.

If you mount the datastore directly (via 9p for example) then you bypass the need for attach/detach (see BON-26) however throughput was significantly lower. If the throughput performance could be improved and the security implications addressed then this latter approach becomes viable.

[gigawhitlocks]

Yeah, @mlh78750 and I discussed the current drawbacks with the current 'true block device' approach, @mlh78750 can you weigh in on this discussion?

[mlh78750]

The thought I had was could we use something disklib and create a VMDK on a normal filesystem on a guest. Then loopback mount it as a block device and use the normal linux tools to format and lay the archives on that filesystem. Then unmount and move that VMDK (file) to the datastore.

[fdawg4l]

@mlh78750 loopback to what? You create the vmdk in the hypervisor or in the guest? If in the guest, sure, but is there a driver for that which doesn't rely on FUSE?

[mlh78750]

I was under the assumption we had no support to format ext4 and mount it on the host. If that is not the case and we can, then that is a better solution. If we cannot, then yes, I was thinking of something using FUSE to mount that image.

[fdawg4l]

@hickeng do you have a pointer to the bonneville 9p code? I recall there are bits that live in the daemon (but could be wrong) and bits that live in esx (but could be wrong). A pointer to both/either would be appreciated.

[fdawg4l]

This looks mostly like BON-147, BON-148, and BON-149. FYI in case we want to close those jira issues as this should probably subsume them.

[dougm]

Parent issue: #38

[gigawhitlocks]

Is this still relevant? @hickeng

[hickeng]

@gigawhitlocks still relevant, just not in plan for the near future - this requires an ESX agent and should be deferred until we have got the VMCI/vSocket communication work in place (that will supply the necessary basics and is higher priority).