Use matrix to add Bookworm jobs to CI

[legoktm]

Description

🔮 Reviewers: Taking a look at the CI pipelines for each commit should give you context on the next commit. 🙂

Add bookworm jobs to CI. These jobs will not be marked as required, as it is expected that they will sometimes fail.

Closes #1545

Test plan

• CI looks reasonable (Bookworm jobs are not necessarily passing, but Bullseye jobs must be passing)
• It is OK to update PyQt5-sip from 12.8.1 to 12.9.0 (Do we need to match the system version?)
• It is OK to update alembic from 1.0.2 to 1.1.0

[legoktm]

Most of the bookworm failures are because we are currently pinned to an old version of pyqt5-sip that did not upload Python 3.10 wheels: https://pypi.org/project/PyQt5-sip/12.8.1/#files

Either we install a build toolchain in each job (build-essential python3-dev) so we can build this from source, or we update to a newer version of pyqt5-sip that has 3.10 wheels. The very next version, 12.9.0, looks to be the first that will work.

[gonzalo-bulnes]

Thank you for troubleshooting @legoktm!

My personal inclination would be to upgrade dependencies (in this case pyqt5-sip) in the spirit of reducing complexity and more generally following upstream updates.

That being said, I think @creviera may have thoughts on this and pyqt5-sip in particular? (Maybe I'm wrong!)

[gonzalo-bulnes]

The CI config is looking great! Thank you for also DRYing the nightlies definition significantly.

I take note that blocks can be named inline where they're used for the first time! 💡

In the package upgrade, I'm thinking of performing the necessary upgrades in a separate PR once we've got them identified, so that this PR doesn't get cluttered by package changes. I'm happy to look into that from the pyqt5-sip pointer next week. : )

[gonzalo-bulnes]

Heads up @legoktm, I've added one job in #1556. It stands alone, so shouldn't b difficult to integrate to the matrix.

[legoktm]

Ack, rebasing on top of that now...

[gonzalo-bulnes]

I started looking into which dependencies need upgrade for Bookworm jobs to pass.

• Using pyqt5-sip==12.9.0 makes some good progress...
• ...but Alembic migrations don't seem happy for some reason in Bookworm. (example in CircleCI)

---

Edited to add:

• Using alembic==1.1.0 clears the remaining test suite errors 👍 (example in CircleCI)
• It requires building wheels (both in Bullseye and Bookworm)
• I haven't checked the change log or assessed suitability for the Client ⚠️

---

Back to pyqt5-sip: the current version was chosen to match the one shipped with Bullseye. I'm not sure if that is a convenience choice or a actual requirement. I believe we rely on system packages, and that it's the latter, in which case we might want to create separate requirement files for Bookworm. 🌵

[legoktm]

Re: alembic, we probably need this fix: https://alembic.sqlalchemy.org/en/latest/changelog.html#change-5681bfa8b70eb750d3060b4541fa1662

[gonzalo-bulnes]

The testing requirements in Bookworm has to do with differing Python versions. I've added make targets for Bookworm... but I realize that goes against the idea of using a matrix to run the same list of jobs under different Python versions : / (That means maybe we'll want to remove the dev-bookworm-requirements target from the Makefile.)

With that in mind, and looking at the CI failure: maybe we should ignore the comments when comparing requirements files, what do you think @legoktm ?

[gonzalo-bulnes]

Re-running the CI pipeline after freedomofpress/securedrop-builder#379 has been merged should make it green 🍏

I think our automated testing coverage of the database aspects of the code should be enough to upgrade alembic with confidence. With that being said, we should still perform a diff review on the package shouldn't we @creviera @rocodes ?

[gonzalo-bulnes]

@legoktm Let's clean up this branch before marking it as ready for review:

• we've got your commit labelled as WIP
• and a few of mine can be squashed (e.g. I added then removed Bookworm-specific files)

[gonzalo-bulnes]

I'll take on the opportunity of updating the PR to do the cleaning.

[gonzalo-bulnes]

A couple of question for you @creviera (and maybe @rocodes too!):

• We're updating PyQt5-sip from 12.8.1 to 12.9.0 - which is the first to support Python 3.10 but is not the system version under Debian Bullseye. Any thoughts?
• Should we perform a diff review for alembic that we upgraded too? (From 1.0.2 to 1.1.0, which doesn't worry me functionally, I think our automated test suite covers that well.)

Subsidiary question: what stops us from shipping the Qt dependencies like any other? Why do we use system packages? (I know you've told me but I forgot! 😬) Related to #1562

[gonzalo-bulnes]

I had a chat with @creviera.

We use system PyQt because:

• PyQt version needs to be very closely matched with the underlying Qt (C++) version
• the underlying Qt (C++) library is large, and wasn't easy to compile, which made shipping it with the Client unappealiing

With that in mind @legoktm, I think we've got a few options:

1. Ship this PR without the PyQt5-sip upgrade, and have the Bookworm jobs fail for the foreseeable future 💀
2. Ship custom requirements files for Bookworm like we used to for Buster (and loose the matrix config improvements 💀)
3. Ship distinct requirements files for Bookworm/Bullseye and make the specific jobs that require them aware of the distinction, like you suggested yesterday @legoktm. That's my favorite option. (We could even base the Bookworm file on the Bullseye ones to avoid duplication where possible.)

Any of these options would allow us to ship this PR with a green pipeline. 🍏

---

We also should have performed a diff review of alembic 1.1.0 before adding the wheel! (I'm following up on that in the relevant repo.)

[legoktm]

All makes sense. In your list 3 is my preferred option as well, but I'd also be okay with 1 in the short-term while we work on 3.

[gonzalo-bulnes]

I'll remove the commit that upgrades PyQt5-sip so that @creviera can review this PR. And we can manage the Bookworm failures in a separate PR 🙂

[gonzalo-bulnes]

(removed) To be clear: the CI pipeline will be read, but all the required jobs (a.k.a the "Bullseye"" jobs) will be green. 🍏

[sssoleileraaa]

Accepting the failing tests temporarily as you work on a separate bookworm requirements PR works for me too. I see you did a diff review for alembic (freedomofpress/securedrop-builder#379 (comment)) ✔️, code changes look good to me ✔️, and the config file looks much cleaner now 🎉. Nice teamwork @gonzalo-bulnes and @legoktm :) 🚀