-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Xen core platform stack #23
Conversation
Add Xen public API headers (xen/include/public/*) as of RELEASE-4.13.1 (upstream commit: 6278553325a9f76d37811923221b21db3882e017).
This is a rewrite of the MirageOS Xen platform stack, with the overall goal of replacing Mini-OS and moving to a minimal, legacy-free codebase which builds MirageOS unikernels as Xen PVHv2 domUs only. At the same time, this change aligns the Xen backend with existing Solo5 backends as much as is practical and replaces the OCaml runtime with that provided by ocaml-freestanding. This is a breaking change. MirageOS libraries and applications that make use of _Xen-specific_ APIs will need to be updated. See below for a list of the interfaces that have changed. Security posture improvements: With the move to a Solo5/ocaml-freestanding base MirageOS gains several notable improvements to security posture for unikernels on Xen: - Stack smashing protection is enabled unconditionally for all C code. - W^X is enforced throughout, i.e. `.text` is read-execute, `.rodata` is read-only, non-executable and `.data`, heap and stack are read-write and non-executable. - The memory allocator used by the OCaml runtime is now dlmalloc (provided by ocaml-freestanding), which is a big improvement over the Mini-OS malloc. Requirements changes: - Xen version 4.10 or later is required, as this is the earliest version which provides production-quality and stable PVHv2 APIs. - Only `x86_64` hosts with HAP (EPT) are supported. Support for ARM32 has been removed. Support for ARM64 may be added at a later date if there is interest. Interface changes: - The `Eventchn` and `Generation` modules, previously provided by `xen-evtchn` have been folded into `mirage-xen` as `OS.Eventchn` and `OS.Generation` respectively. Applications and libraries should replace any dependency on `xen-evtchn` with one on `mirage-xen {>= "6.0.0"}` and use the corresponding submodules in `OS` directly. - `OS.Lifecycle.await_shutdown_request` is a stub, as for the other Solo5 backends. - `OS.Sched` has been removed. These interfaces (notably domain suspend/resume) were present but never fully implemented in the old Xen stack. They are not required for PVH domains or full-system suspend/resume and applications/libraries should remove any calls to them. - `OS.Start_info` has been removed. These interfaces were specific to Xen PV, and don't have a direct equivalent for PVH domains. - `OS.Xenctrl` has been removed. There are no known users of these interfaces, which appear to be a left-over from earlier versions of the Xen stack. Dependency changes: - Now depends on `solo5-bindings-xen` (for the low-level VCPU initialisation and startup) and `ocaml-freestanding` (for the OCaml runtime). It follows that `-freestanding` variants of existing packages such as `zarith` and `gmp` are used instead of `-xen` variants. - The `io-page-xen` package is no longer used. Applications/libraries depending on it should replace the dependency with one on `io-page`. - As mentioned above, the `xen-evtchn` package is no longer used and applications/libraries should replace their dependency with one on `mirage-xen {>= "6.0.0"}`. C stubs changes: - `alloc_pages_stubs.c`: added, provides stubs for `Io_page`, as in `mirage-solo5`. - `checksum_stubs.c`: added, provides stubs for `Tcpip`, as in `mirage-solo5`. - `exit_stubs.c`: removed, equivalent provided by `ocaml-freestanding`. - `sched_stubs.c`: removed, no longer required. - `start_info_stubs.c`: removed, no longer required. - `xb_stubs.c`: removed, no users and no longer required. Apart from the above, the majority of C stubs which are used by `mirage-xen` have been renamed to follow the `mirage_xen_XXX` rather than `stub_XXX` convention. The low-level C interfaces to Xen event channels (`evtchn.c`) and grant tables (`gnttab.c) have been re-written from scratch. Some additional interfaces have been added to `main.c` to replace functionality that was previously part of `start_info_stubs.c` (obtaining initial PV console and Xenstore ports and I/O rings from Xen). Relationship with the Solo5 Xen bindings: For Xen, Solo5 only provides the initial PVH boot, VCPU setup/traps and C-side PV console code. Some Mirage-internal private APIs are provided to allow `mirage-xen` to interact with the hypervisor -- this is fully explained in Solo5 commit Solo5/solo5@f4b47d1.
Update mirage-block-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.
Update mirage-console-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.
Update mirage-net-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.
Update vchan-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.
Some notes on what needs review or further discussion:
|
it is actually used by the (unreleased) framebuffer library https://github.com/cfcs/mirage-framebuffer/blob/f63d54dd86f4ab723cc471fbf127d5fc105aa656/qubes/framebuffer_qubes.ml#L122-L123 /cc @cfcs |
This is needed by mirage-firewall, otherwise Qubes won't be able to shut down the VM. |
This is actually used and functional, so reinstating it.
Sorry about that, I was a bit too over-zealous in removing code there. I've reinstated it in 528d6c2. It'd be good if you could confirm that it (still) works as advertised. |
Thanks! I'll test it once the new Qubes is out. But since it hasn't changed from the current code, I don't see why it shouldn't work. |
The initial implementation of mirage_xen_gnttab_{map,mapv,unmap} was wrong in using heap-allocated memory for grant mappings imported from other Xen domains. This is not possible, since once a grant is unmapped the associated guest pages are mapped by Xen to a "dead" page (writes are a no-op, reads return 0xff) and can no longer be re-used as "normal" memory. This change introduces a separate virtual memory range used for such mappings; a private ABI from Solo5 provides a way to determine which memory range to use and a simple bitmap allocator is used to externally manage usage of pages in this memory range.
The total number of heap pages is computed from the Solo5 heap size (si->heap_size) and the number of used pages is computed from the result returned by dlmalloc's malloc_footprint(). Note that mallinfo() is deliberately not used in the latter case as that call is expensive and existing unikernels rely on this interface being cheap to call.
Required for solo5__xen_get_gntmap_area().
I have pushed the following additional changes to this branch/PR:
94372da is probably of most interest to any lurkers, from the commit log:
|
Now requires xenial due to Solo5 GCC requirements.
The new PVH Xen stack is currently x86_64-specific; at the same time relax "os" to match solo5-bindings-xen.
The expressions in question are actually constant at compile-time, but older compilers don't seem to pick that up, so use a traditional #define instead.
I reviewed 94372da and am confident it works as advertised. |
yes indeed |
executing
this is with dune 2.7.1, my suspicion is that this magically adds |
Can you upload the entire output of |
relevant PR ocaml/dune#3565 (part of dune 2.7.0) and indeed in my
|
the requested log.txt |
The forward declaration and typedef in bindings.h are sufficient, do not re-declare the typedef here.
The C code in bindings/ *must* be built using exact C flags and linked only statically. Until there is a way to do this in an OCaml + dune + Mirage environment, we need to use Makefile here.
b00b76b
to
9fe3875
Compare
Travis is green, so this is good to merge, finally! There is some follow-up work to do before release but that can be done in separate PRs. |
As discussed in mirage/mirage-xen#23, using dune's C compilation support leads to unexpected CFLAGS, especially -fPIC and -fno-strict-aliasing in our setup. To regain control of CFLAGS, instead use a Makefile. There's no need for the OCaml mirage-crypto.freestanding / mirage-crypto.xen sublibraries anymore. Failure of compilation are allowed if the respective opam packages (mirage-xen-posix, ocaml-freestanding) are not installed. For a successful cold `dune build @install` run, (alias install) is added to the (copy-files# ..) stanze in xen/dune and freestanding/dune. The libmirage_crypto_{xen,freestanding}_stubs.a are installed directly in the lib/mirage-crypto/ directory.
As discussed in mirage/mirage-xen#23, using dune's C compilation support leads to unexpected CFLAGS, especially -fPIC and -fno-strict-aliasing in our setup. To regain control of CFLAGS, instead use a Makefile. There's no need for the OCaml mirage-crypto.freestanding / mirage-crypto.xen sublibraries anymore. Failure of compilation are allowed if the respective opam packages (mirage-xen-posix, ocaml-freestanding) are not installed. For a successful cold `dune build @install` run, (alias install) is added to the (copy-files# ..) stanze in xen/dune and freestanding/dune. The libmirage_crypto_{xen,freestanding}_stubs.a are installed directly in the lib/mirage-crypto/ directory.
The dune C compilation support adds CFLAGS (specifically -fPIC and -fno-strict-aliasing) which are not necessary and should be avoided when compiling libmirage-solo5_bindings.a. This follows from the discussion and solution in mirage/mirage-xen#23.
As discussed in mirage/mirage-xen#23, using dune's C compilation support leads to unexpected CFLAGS, especially -fPIC and -fno-strict-aliasing in our setup. To regain control of CFLAGS, instead use a Makefile. There's no need for the OCaml mirage-crypto.freestanding / mirage-crypto.xen sublibraries anymore. Failure of compilation are allowed if the respective opam packages (mirage-xen-posix, ocaml-freestanding) are not installed. For a successful cold `dune build @install` run, (alias install) is added to the (copy-files# ..) stanze in xen/dune and freestanding/dune. The libmirage_crypto_{xen,freestanding}_stubs.a are installed directly in the lib/mirage-crypto/ directory.
CHANGES: Version 6.0.0 is a re-write of the MirageOS Xen platform stack, with the overall goal of replacing Mini-OS and moving to a minimal, legacy-free codebase which builds MirageOS unikernels as Xen PVHv2 domUs only. At the same time, this change aligns the Xen backend with existing Solo5 backends as much as is practical and replaces the OCaml runtime with that provided by ocaml-freestanding. This is a breaking change. MirageOS libraries and applications that make use of Xen-specific APIs will need to be updated. See below for a list of the interfaces that have changed. Requirements changes: * OCaml 4.08 or later and Dune 2.6 or later are required. * Xen version 4.10 or later is required, as this is the earliest version which provides production-quality and stable PVHv2 APIs. * Additionally, the custom Xen version 4.8 used by Qubes OS 4.0 is supported, as this has enough of PVHv2 backported to it. * Only x86\_64 hosts with HAP (EPT) are supported. Support for ARM32 has been removed. Support for ARM64 may be added at a later date if there is interest. Security posture improvements: With the move to a Solo5 and ocaml-freestanding base MirageOS gains several notable improvements to security posture for unikernels on Xen: * Stack smashing protection is enabled unconditionally for all C code. * W^X is enforced throughout, i.e. `.text` is read-execute, `.rodata` is read-only, non-executable and `.data`, heap and stack are read-write and non-executable. * The memory allocator used by the OCaml runtime is now dlmalloc (provided by ocaml-freestanding), which is a big improvement over the Mini-OS malloc, and incorporates features such as heap canaries. Interface changes: * The `Eventchn` and `Generation` modules, previously provided by `xen-evtchn` have been folded into `mirage-xen` as `OS.Eventchn` and `OS.Generation` respectively. Applications and libraries should replace any dependency on `xen-evtchn` with one on `mirage-xen {>= "6.0.0"}` and use the corresponding submodules in `OS` directly. * The `OS.MM` module has been replaced with a new `OS.Memory` module, with a statistics interface modeled after stdlib's `Gc` stat type. All units use the system word size rather than pages, which were ill-defined. * `OS.Sched` has been removed. These interfaces (notably domain suspend/resume) were present but never fully implemented in the old Xen stack. They are not required for PVH domains or full-system suspend/resume and applications/libraries should remove any calls to them. * `OS.Start_info` has been removed. These interfaces were specific to Xen PV, and don't have a direct equivalent for PVH domains. * `OS.Xenctrl` has been removed. There are no known users of these interfaces, which appear to be a left-over from earlier versions of the Xen stack. * `OS.Xen.virt_to_mfn` has been removed. This is a little-used low-level vestige of Mini-OS, and in the Solo5-based virtual memory arrangement should be replaced with the equivalent computation (`addr >> 12`) in application code directly. Dependency changes: * Now depends on `solo5-bindings-xen` (for the low-level VCPU initialisation and startup) and `ocaml-freestanding` (for the OCaml runtime). It follows that `-freestanding` variants of existing packages such as `zarith` and `gmp` are used instead of `-xen` variants. * The `io-page-xen` package is no longer used. Applications/libraries depending on it should replace the dependency with one on `io-page`. * As mentioned above, the `xen-evtchn` package is no longer used and applications/libraries should replace their dependency with one on `mirage-xen {>= "6.0.0"}`. C stubs changes: * `alloc_pages_stubs.c`: added, provides stubs for `Io_page`, as in `mirage-solo5`. * `checksum_stubs.c`: added, provides stubs for `Tcpip`, as in `mirage-solo5`. * `exit_stubs.c`: removed, equivalent provided by `ocaml-freestanding`. * `sched_stubs.c`: removed, no longer required. * `start_info_stubs.c`: removed, no longer required. * `xb_stubs.c`: removed, no users and no longer required. Apart from the above, the majority of C stubs which are used by `mirage-xen` have been renamed to follow the `mirage_xen_XXX` rather than `stub_XXX` convention. The low-level C interfaces to Xen event channels (`evtchn.c`) and grant tables (`gnttab.c`) have been re-written from scratch. Some additional interfaces have been added to `main.c` to replace functionality that was previously part of `start_info_stubs.c` (obtaining initial PV console and Xenstore ports and I/O rings from Xen). Due to issues with mixing PIC and non-PIC code when compiling the C stubs with Dune, the build system has been adapted to use a foreign Makefile for `libmirage-xen_bindings.a`. This is the recommended method to use when building libraries containing C code for use with MirageOS. See conversation in mirage/mirage-xen#23 for details. Relationship with the Solo5 Xen bindings: For Xen, Solo5 only provides the initial PVH boot, VCPU setup/traps and C-side PV console code. Some Mirage-internal private APIs are provided to allow `mirage-xen` to interact with the hypervisor -- this is fully explained in Solo5 commit Solo5/solo5@f4b47d1 and follow-ups.
The dune C compilation support adds CFLAGS (specifically -fPIC and -fno-strict-aliasing) which are not necessary and should be avoided when compiling libmirage-solo5_bindings.a. This follows from the discussion and solution in mirage/mirage-xen#23. Co-authored-by: Martin Lucina <[email protected]>
sorry to be super late to the party. |
…-xen 6.0.0) also use a Makefile for freestanding builds (for tight control of CFLAGS), see mirage/mirage-xen#23 for motivation and further details
…-xen 6.0.0) also use a Makefile for freestanding builds (for tight control of CFLAGS), see mirage/mirage-xen#23 for motivation and further details
…-xen 6.0.0) also use a Makefile for freestanding builds (for tight control of CFLAGS), see mirage/mirage-xen#23 for motivation and further details update lower bound to dune 2.6
Part of mirage/mirage#1159. The interesting commits start from 5bb9bc5 onwards.
This is a rewrite of the MirageOS Xen platform stack, with the overall goal of replacing Mini-OS and moving to a minimal, legacy-free codebase which builds MirageOS unikernels as Xen PVHv2 domUs only. At the same
time, this change aligns the Xen backend with existing Solo5 backends as much as is practical and replaces the OCaml runtime with that providedby ocaml-freestanding.
This is a breaking change. MirageOS libraries and applications that make use of Xen-specific APIs will need to be updated. See below for a list of the interfaces that have changed.
Security posture improvements:
With the move to a Solo5/ocaml-freestanding base MirageOS gains several notable improvements to security posture for unikernels on Xen:
.text
is read-execute,.rodata
is read-only, non-executable and.data
, heap and stack are read-write and non-executable.Requirements changes:
x86_64
hosts with HAP (EPT) are supported. Support for ARM32 has been removed. Support for ARM64 may be added at a later date if there is interest.Interface changes:
Eventchn
andGeneration
modules, previously provided byxen-evtchn
have been folded intomirage-xen
asOS.Eventchn
andOS.Generation
respectively. Applications and libraries should replace any dependency onxen-evtchn
with one onmirage-xen {>= "6.0.0"}
and use the corresponding submodules inOS
directly.OS.Lifecycle.await_shutdown_request
is a stub, as for the other Solo5 backends.OS.Sched
has been removed. These interfaces (notably domain suspend/resume) were present but never fully implemented in the old Xen stack. They are not required for PVH domains or full-system suspend/resume and applications/libraries should remove any calls to them.OS.Start_info
has been removed. These interfaces were specific to Xen PV, and don't have a direct equivalent for PVH domains.OS.Xenctrl
has been removed. There are no known users of these interfaces, which appear to be a left-over from earlier versions of the Xen stack.Dependency changes:
solo5-bindings-xen
(for the low-level VCPU initialisation and startup) andocaml-freestanding
(for the OCaml runtime). It follows that-freestanding
variants of existing packages such aszarith
andgmp
are used instead of-xen
variants.io-page-xen
package is no longer used. Applications/libraries depending on it should replace the dependency with one onio-page
.xen-evtchn
package is no longer used and applications/libraries should replace their dependency with one onmirage-xen {>= "6.0.0"}
.C stubs changes:
alloc_pages_stubs.c
: added, provides stubs forIo_page
, as inmirage-solo5
.checksum_stubs.c
: added, provides stubs forTcpip
, as inmirage-solo5
.exit_stubs.c
: removed, equivalent provided byocaml-freestanding
.sched_stubs.c
: removed, no longer required.start_info_stubs.c
: removed, no longer required.xb_stubs.c
: removed, no users and no longer required.Apart from the above, the majority of C stubs which are used by
mirage-xen
have been renamed to follow themirage_xen_XXX
rather thanstub_XXX
convention.The low-level C interfaces to Xen event channels (
evtchn.c
) and grant tables (gnttab.c
) have been re-written from scratch. Some additional interfaces have been added tomain.c
to replace functionality that was previously part ofstart_info_stubs.c
(obtaining initial PV console and Xenstore ports and I/O rings from Xen).Relationship with the Solo5 Xen bindings:
For Xen, Solo5 only provides the initial PVH boot, VCPU setup/traps and C-side PV console code. Some Mirage-internal private APIs are provided to allow
mirage-xen
to interact with the hypervisor -- this is fully explained in Solo5 commitSolo5/solo5@f4b47d1.