Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate the creation of Linux environments for Dangerzone testing #286

Closed
apyrgio opened this issue Dec 9, 2022 · 2 comments · Fixed by #289
Closed

Automate the creation of Linux environments for Dangerzone testing #286

apyrgio opened this issue Dec 9, 2022 · 2 comments · Fixed by #289
Assignees
Labels
Milestone

Comments

@apyrgio
Copy link
Contributor

apyrgio commented Dec 9, 2022

Dangerzone is a multi-platform application, currently available on Windows, MacOS, and several Linux flavors. In order to test new Dangerzone features on these environments, we need to have access to them. Typically, this requires dedicated hardware, and some time to provision (and possibly deprovision) these environments.

This issue focuses on the Linux environments that we need to support. Windows and MacOS environments are still a hard problem, mainly because they require nested virtualization, if the user does not have dedicated hardware.

As of writing this, we are supporting 8 Linux environments (source):

  • Ubuntu 22.10 (kinetic)
  • Ubuntu 22.04 (jammy)
  • Ubuntu 20.04 (focal)
  • Debian 12 (bookworm)
  • Debian 11 (bullseye)
  • Fedora 37
  • Fedora 36
  • Fedora 35

Our QA requirements for adequately testing these environments are the following:

  1. Create a development environment. That is, being able to run the corresponding section in our BUILD.md, which includes installing build tools, Poetry, and installing the project's dependencies.
  2. Build the container image for Dangerzone. This step requires running Podman in the Linux environment.
  3. Run the Dangerzone tests. This step also requires Podman.
  4. Build the .deb / .rpm package for Dangerzone. This step requires a dev environment (see 1).
  5. Test QA scenarios on the Linux environment. This step requires Podman and GUI support.
@apyrgio apyrgio self-assigned this Dec 9, 2022
@apyrgio apyrgio added enhancement New feature or request P:linux container labels Dec 9, 2022
@apyrgio
Copy link
Contributor Author

apyrgio commented Dec 9, 2022

A way to implement this feature could be to run these environments in separate containers. This approach has the following benefits:

  1. Containers need few resources.
  2. Performance is on par with the user's host.
  3. Container creation is declarative, which aids in caching and reproducibility.

The drawbacks in this approach are the following:

  1. Nested containerization is difficult. The first containerization layer is usually very restrictive, and prevents users from creating more layers.
  2. GUI support from within containers is not straightforward.

For (1), we can make the first containerization layer as privileged as the user in the host. Security-wise this should not be much of an issue, since it's an improvement from building Dangerzone directly on the host. For (2), we need to mount the X11 / Wayland sockets to the container, and we can document how it works.

@apyrgio
Copy link
Contributor Author

apyrgio commented Dec 12, 2022

Some notes on running Podman within another container:

  1. The /usr/bin/new[gu]idmap executables may not have the suid bit set. The Podman authors have acknowledged this issue, and suggest either reinstalling the corresponding package, or setting the bit manually. We choose the first option, since it affects only old versions of Fedora. Ideally, we should drop this workaround after they fix it upstream.
  2. The default range in /etc/sub[gu]id is user:100000:65536. However, if one runs Podman in Podman, this range is not acceptable for the innner container. The reason is that the outer container will be granted 65536 UIDs/GIDs, and the inner container will try to reserve UID/GID 100000, which is higher than 65536. For this reason, we reserve a lower UID and a smaller range, to make sure that the inner container will work.

apyrgio added a commit that referenced this issue Dec 12, 2022
Introduce `dev_scripts/env.py`, which is a script for building
Dangerzone environments for various Linux distros, and running commands
in them.

Closes #286
apyrgio added a commit that referenced this issue Dec 12, 2022
Add a design document for `dev_scripts/env.py`, which is a script that
creates Dangerzone environments for various Linux distros. In this
design document, we explain various architectural decisions that we have
taken for this script, as well as how it works under the hood, what are
its shortcomings, etc.

Refs #286
apyrgio added a commit that referenced this issue Dec 12, 2022
Ignore the `dev_scripts/envs` folder when running tests or linting code,
as it may contain files that are not owned by the current user. In this
case, we've seen that pytest/black etc. fail.

This typically happens when the user has run Dangerzone in a
containerized environment (see #286), and Podman created a directory
with files owned by the user in the nested container.
@apyrgio apyrgio added this to the 0.4.1 milestone Dec 13, 2022
apyrgio added a commit that referenced this issue Jan 4, 2023
Introduce `dev_scripts/env.py`, which is a script for building
Dangerzone environments for various Linux distros, and running commands
in them.

Closes #286
apyrgio added a commit that referenced this issue Jan 4, 2023
Add a design document for `dev_scripts/env.py`, which is a script that
creates Dangerzone environments for various Linux distros. In this
design document, we explain various architectural decisions that we have
taken for this script, as well as how it works under the hood, what are
its shortcomings, etc.

Refs #286
apyrgio added a commit that referenced this issue Jan 4, 2023
Ignore the `dev_scripts/envs` folder when running tests or linting code,
as it may contain files that are not owned by the current user. In this
case, we've seen that pytest/black etc. fail.

This typically happens when the user has run Dangerzone in a
containerized environment (see #286), and Podman created a directory
with files owned by the user in the nested container.
apyrgio added a commit that referenced this issue Jan 10, 2023
Introduce `dev_scripts/env.py`, which is a script for building
Dangerzone environments for various Linux distros, and running commands
in them.

Closes #286
apyrgio added a commit that referenced this issue Jan 10, 2023
Add a design document for `dev_scripts/env.py`, which is a script that
creates Dangerzone environments for various Linux distros. In this
design document, we explain various architectural decisions that we have
taken for this script, as well as how it works under the hood, what are
its shortcomings, etc.

Refs #286
apyrgio added a commit that referenced this issue Jan 10, 2023
Ignore the `dev_scripts/envs` folder when running tests or linting code,
as it may contain files that are not owned by the current user. In this
case, we've seen that pytest/black etc. fail.

This typically happens when the user has run Dangerzone in a
containerized environment (see #286), and Podman created a directory
with files owned by the user in the nested container.
apyrgio added a commit that referenced this issue Jan 16, 2023
Introduce `dev_scripts/env.py`, which is a script for building
Dangerzone environments for various Linux distros, and running commands
in them.

Closes #286
apyrgio added a commit that referenced this issue Jan 16, 2023
Add a design document for `dev_scripts/env.py`, which is a script that
creates Dangerzone environments for various Linux distros. In this
design document, we explain various architectural decisions that we have
taken for this script, as well as how it works under the hood, what are
its shortcomings, etc.

Refs #286
apyrgio added a commit that referenced this issue Jan 16, 2023
Ignore the `dev_scripts/envs` folder when running tests or linting code,
as it may contain files that are not owned by the current user. In this
case, we've seen that pytest/black etc. fail.

This typically happens when the user has run Dangerzone in a
containerized environment (see #286), and Podman created a directory
with files owned by the user in the nested container.
apyrgio added a commit that referenced this issue Jan 16, 2023
Add a design document for `dev_scripts/env.py`, which is a script that
creates Dangerzone environments for various Linux distros. In this
design document, we explain various architectural decisions that we have
taken for this script, as well as how it works under the hood, what are
its shortcomings, etc.

Refs #286
apyrgio added a commit that referenced this issue Jan 16, 2023
Ignore the `dev_scripts/envs` folder when running tests or linting code,
as it may contain files that are not owned by the current user. In this
case, we've seen that pytest/black etc. fail.

This typically happens when the user has run Dangerzone in a
containerized environment (see #286), and Podman created a directory
with files owned by the user in the nested container.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant