Hackage revision policy

[hasufell]

@Mikolaj said in Matrix that cabal team decided to avoid revisions for "obvious reasons". I don't know what those are and didn't get a reference to the relevant discussion when I asked (supposedly that happened on matrix long time ago).

Can I get a clarification here?

CCing some hackage trustees: @Bodigrim @andreasabel @phadej @andreabedini

Context: #9523

[Mikolaj]

Thank you @hasufell. That's as a good opportunity as any to actually forge our policy on revisions.

To clarify, let me copy what I said on Matrix:

I think, we try to be conservative with revisions for the obvious reasons
(two conflicting views on the same package)
...
yes, we try to avoid revisions, I think, because they cause the bounds on Hackage 
to be different than on github and in other places which can be confusing in various ways
> So?
> So you rather have failing build plans?
that depends on what build plans they are and what is exactly the cause of the failure
I'm not trying to be difficult
but just make sure it's really needed
...
yeah, I'm not sure how widely it's shared --- it was a discussion on this channel some time ago
*a long time ago

[Bodigrim]

I’d expect Cabal to rely on revisions even more than an average Haskell package, because of sparse release cycles and ripples they cause. In the context of #9523 a revision seems a no-brainer to me, I don’t see who is to win from postponing it and forcing a new Cabal release instead.

[gbaz]

I don't recall any policy about avoiding hackage revisions for cabal.There's no considerations I can think of that would affect cabal different than any other hackage package regarding this. We just should try to also ensure that the bounds are changed in-repo as well, but this also holds for packages on hackage in general.

[ulysses4ever]

I feel like revisions unblock more people than they trouble, so I'm in support of revisions in the Cabal project.

[andreabedini]

If a build plan works (for some definition of works) it should be allowed by the version bounds.
I don't see where the problem is, haven't we done revision many times before?

GitHub can be updated to reflect what is on Hackage (like everybody does).

[hasufell]

Can I expect cabal developers to take care of this or should this be redirected to hackage trustees now and in the future?

[andreabedini]

I see #9523 and backports have been merged. I don't know if there have been discussions in other channels. Are we all on the same page now? @Mikolaj would you like to comment here?

[Mikolaj]

Thank you for the discussion. It seems we have a concrete policy emerging. Let's also mention this on Thursday's cabal dev's meeting.

Can we try to make the policy more precise? What are the pre-conditions for a revision?

1. A PR with the same bound change?
2. A merged PR only?
3. Somebody saying the new bound works?
4. Somebody with a good track record convincingly arguing the new bound works?
5. Somebody with a good track record reporting a successful local test run with the new bounds?
6. A successful cabal CI run with the new bounds (but note that this extra work (which we did in the past), on a branch, because normal CI would probably not pick the newest version, for whatever reason (e.g., other packages not permitting it yet))?

edit:
7. Are revisions of multiple versions of cabal acceptable, as a rule?
8. Only minor versions or major versions as well? How many minor/major versions back?
9. If we mandate test runs, as above, are these required also for the old revisions (keep in mind that CIs on branches older than the newest release branch usually don't work, since they are not patched to work around various internal and external problems)?
10. If test runs are required for old versions, does it include minor versions (from tags) or only major versions (from branches)?

[Mikolaj]

I don’t see who is to win from postponing it and forcing a new Cabal release instead.

To clarify, both cabal 3.10.3 and cabal 3.12.1 releases are imminent, from what I understand.

This reminds me, we also need to clarify if revising many subsequent versions of cabal (as is proposed by @hasufell in this case) is something routine or exceptional. And if we expect CI test runs for all the versions or only major ones. Let me add a point above.

[ffaf1]

If a bug pops up becuase of a bound change, there is the possibility that no commit in the repository matches what users downloaded from Hackage, making bug-fixing a potential nightmare.

---

edit: as a simple example, take this git history

9054084c8 Merge #9515
4c1d382a4 Refactor 
2759cab02 Relax bytestring bounds
5a345672d Add flair
729c659d2 Use -XRecordWildCards
eb46bdca4 (haskell/3.10.8) Release 3.10.8 

Now, if we revise Cabal 3.10.8 to loosen bytestring bounds, when a user types cabal get Cabal, such user will get a collection of files which does not correspond to any our of commits/snapshots (the user would get eb46bdca4 plus a cherry-picked Relax bytestring bounds).

So if we introduced a bug, it might be a pain to debug, especially so if the snapshot at 2759cab02 Relax bytestring bounds runs fine but the “eb46bdca4 (haskell/3.10.8) Release 3.10.8 plus cherrypick” does not.

If cabal decides to approve revisions, it could be good to think about ways to mitigate such risks.

[Mikolaj]

Yes, at least a warning in our docs somewhere could be beneficial, perhaps recommending --allow-newer (any other tricks?) if a bug report can't be reproduced with github code. Or recommending to try to repro with the code downloaded from Hackage, I guess?

I'm sure this has been discussed to death for other projects and in flamewars on the internets and trustees are dead tired of this. Dear trustees, is there a FAQ somewhere that you distilled from such discussion? Tips and tricks, what can go wrong, how to best warn about the discrepancy, what's the answer to folks worried about signed/unsigned blobs and possible attacks, etc.? (I'm just making up problems --- I haven't participated in the flame wars and I'm generally pro-revisions, especially for packages that don't release often.)

[gbaz]

The typical policy for most packages is to bump upper bounds only on the most current version of a given package (or perhaps on the most current minor release among a all of a few major release series, if multiple major releases are being supported). In the unfortunate event that bounds need to be restricted, that needs to happen on all affected packages, not just the most current.

In conjunction with bounds bumps or restrictions, typically only the head of the repository (or the branch for each major release) has its cabal file edited in github.

I do not believe there are any common standards on what is considered "sufficient evidence" for a bounds change, since project structures are all so different.

there's a faq here, but i don't know if it discusses most of what you would be asking for. as long as you just change bounds to be more accurate through revisions, there's no special tips or tricks needed, nothing particular that can go wrong, no need to warn about discrepancies, and no issue arises with signing or possible attacks: https://github.com/haskell-infra/hackage-trustees/blob/master/revisions-information.md

again: we've never had a policy that i recall against revisions. hecate made some revision to 3.10.1.0 just five months ago (https://hackage.haskell.org/package/cabal-install-3.10.1.0/revisions/)

i think this is memory playing tricks.

[Bodigrim]

If a bug pops up becuase of a bound change, there is the possibility that no commit in the repository matches what users downloaded from Hackage, making bug-fixing a potential nightmare.

There seems to be an underlying assumption that package builds are set in stone and it's only revisions which could get in the way of reproducibility. But usually a Hackage package can be built with a multitude of compilers and dependency versions; revisions do not really bring much added complexity to the equation.

there's no special tips or tricks needed, nothing particular that can go wrong, no need to warn about discrepancies, and no issue arises with signing or possible attacks

That's my experience as well.

[Mikolaj]

we've never had a policy that i recall against revisions

@gbaz: I've never said we had a policy nor policy against. You are probably reading Julian's paraphrase instead of the transcript. I, too, revised cabal on Hackage and related tools numerous times.

But we can have a policy now. Thank you for your input. That covers points from 7 upwards. How about the "pre-conditions for a revision" points? Dear trustees, when you relax bounds, what are your preconditions? Dear cabal maintainers, what is our preference?

[Bodigrim]

Dear trustees, when you relax bounds, what are your preconditions?

Fundamentally Hackage trustees can relax bounds as they see fit, following the policy. Practice varies from case to case and from trustee to trustee, depending on perceived urgency and effect, but usually the presence of relaxed bound in HEAD (!) and a successful run of cabal test on Hackage package (!) are sufficient grounds to revise the latest release without further consultations, unless maintainers do it first.

[hasufell]

I've never said we had a policy nor policy against. You are probably reading Julian's paraphrase instead of the transcript.

I didn't say you have a policy either if you read my post carefully.

But I think following this thread, it seems clear now that there was no discussion amongst cabal developers agreeing to avoid revisions. That was your claim and that is why I got resistance from you when asking to publish revisions. Which is rather weird, because I provided a PR and asked you to backport before doing the revision. So your point about revisions and branches diverging doesn't make sense.

The rest of the discussion is bikeshedding with little value to me.

I'm disappointed in you that such a simple request caused this amount of resistance and bikeshedding.

Look at my position: I'm not dealing with just Cabal. I'm dealing with a shitton of packages that potentially need laxer upper bounds, so that I can promote the AFPP proposal and the new OsPath API without users being locked out due to unsolvable build plans.

If every upstream would react this way, I'd have no energy left.

However, I'm happy that this discussion at least helped to clear up some questions for you.

Next time, I'll probably approach hackage trustees first, though, and follow whatever requests they have.

[Bodigrim]

Next time, I'll probably approach hackage trustees first, though, and follow whatever requests they have.

It is recommended to ask respective maintainers first by publicly raising an issue or leaving a comment under a relevant PR. If they agree, case solved, no need to spend sparse resources of Hackage trustees. If they disagree, they can probably provide a reason. If they ignore, forget or are busy, wait a week or two and file an issue at https://github.com/haskell-infra/hackage-trustees.

We do not really want to supplant maintainers or interfere with their line of work out of the blue.

[andreasabel]

Answering to the question how revisions could be tracked in the git repo (Q by @ffaf1):
There is a number of packages I am stewarding for Herbert where I am tagging also revisions in the git repo, see e.g. https://github.com/haskell-hvr/http-io-streams/tags

A workflow to create revision r of x.y.z.w would be:

1. Check out tag x.y.z.w (or x.y.z.w~q if q=r-1).
2. Create branch release-x.y.z.w.
3. Apply the version bumps and the x-revision bump.
4. Push branch so that CI runs.
5. Push the revision to hackage using hackage-cli push-cabal ....
6. Push tag x.y.z.w~r.

Point 4. could be problematic since CI cannot really be versioned; GHA does not support any versioning of their runners. That would mean that either CI changes need to be backported (maybe unrealistic for a complex project like Cabal), or testing is only done locally, maybe be more than one individual.

[andreasabel]

@Mikolaj,

Let's also mention this on Thursday's cabal dev's meeting.

could you add me to the google calendar again? I have received any invitation in months, so I suspect that I fell out somehow.

[Mikolaj]

could you add me to the google calendar again? I have received any invitation in months, so I suspect that I fell out somehow.

Hi Andreas. You are listed under your gmail address. Is it still valid? I've sent you a past invitation with your address in it. Let's sort it out via email.

[Mikolaj]

@hasufell: I don't have spare "energy" to dispute your claims, but let me state two things:

• not granting your wishes unquestionably is not "resistance", neither is not bowing to threats of escalation, actual public escalation and public drama, for which apparently you have all the energy needed (but I don't)
• we still don't have a policy, but AFAICS your request still does not meet the example cabal test pre-requisite mentioned by @Bodigrim and, given that it's IMHO on the modest side of the possible pre-requisites and that it matches how we handled some previous revisions (I remember helping to perform relevant tests on CI, constraining the versions), I'm not sure we can proceed; let me know if your request is important and urgent, then I guess we can make an exception (again I'm unapologetically publicly guessing a policy we don't have), especially that we managed to fast-track both the PRs already

[Bodigrim]

Dear all. I appreciate that everyone is extremely passionate about their work in this community, but please let's be patient and charitable to each other. We are in for the long haul.

If I may, I'd suggest to dedicate this topic to a general policy and discuss individual cases in relevant issues / PRs.

[Mikolaj]

Edit: The new authoritative location for the document is now https://github.com/haskell/cabal/wiki/Cabal-Packages-Hackage-Revision-Policy

---

In today's cabal devs video chat we've agreed on a policy (but we are still open to feedback and amendments (please comment below), especially from people absent in the latter part of the meeting; minor edits to this policy will be made here without versioning nor notifications). Here goes version 1.0:

tl;dr

1. revisions are good
2. only for the newest published cabal version
3. merged PR first
4. build and test locally (but ideally in CI)
5. leeway for exceptional circumstances

Full text:

1. We are lax with respect to cabal package revisions on Hackage. Any minor or major benefit (e.g., less build plans failing) is enough to justify applying any relaxation or restriction of bounds that matches reality.
2. By default, we only accept revision change requests for the newest minor version of the newest major version of cabal on Hackage.
3. The crucial pre-requisite we expect from a person requesting a revision is a PR modifying the deps merged to the branch that (loosely) corresponds to the revised cabal version (e.g., the common release branch for the whole major version).
4. The secondary pre-requisite is a local build (one architecture and one GHC version) of all cabal packages and a local cabal-testuite run (as described in https://github.com/haskell/cabal/blob/master/cabal-testsuite/README.md) that shows no regressions and where --with-cabal is pointing to the just built cabal binary (loosely) corresponding to the state after revision. Ideally, the whole cabal CI matrix of tests would be run in a dedicated new PR (afterwards closed and preserved) that adds constraints and --allow-newer to force the dependency versions of interest.
5. Exceptional circumstances may waive any of the above, in particular it may sometimes be prudent to revise before the pre-requisites are met. Transparency and paper trail are especially important in such cases (publicly state and document each decision, ask for objections).

Edit: Addendum for old cabal versions:

1. Ideally, we'd have one PR per cabal version affected, with full testing, etc., see above. This is, however, not possible when CI is dead on the particular release branch and bringing CI back to life is unlikely to be worthwhile.
2. For older cabal versions, in particular those with unmaintained release branches, we still welcome and encourage a PR to the relevant release branch. Upon merging, it would ensure the version bounds on the branch match the version bounds as revised Hackage. If possible, the CI would show the new bounds build fine, even if tests fail due to CI bit-rot. If CI can't show a successful build (and the build errors don't point to an actual problem with the proposed new bounds), a local build reported in the PR, with or without a local test run, would be a help to reviewers and the PR can be merged manually regardless of CI.
3. For really old versions or when the release branch is out of sync with Hackage already, we welcome Hackage trustees and cabal maintainers to scale back the level of formality accordingly. We certainly don't expect bringing the bounds on a long-abandoned branch back in sync with Haskell. A ping on the Matrix channel wouldn't hurt in any case, so that we are aware which branches are out of sync (possibly by accident) and, generally, to share the hacking joy and gratitude for anybody that cares about cabal.

[Mikolaj]

Thank you everybody for contributing to this process.

Loose ends:

• DONE (the document now resides at https://github.com/haskell/cabal/wiki/Cabal-Packages-Hackage-Revision-Policy and the bounds-forcing CI trick is at https://github.com/haskell/cabal/blob/master/CONTRIBUTING.md#hackage-revisions). After a shake-up period, we may want to move the policy text to the wiki or link to it from somewhere central. In the same place, @ulysses4ever would like to place the link to https://github.com/haskell/ghcup-metadata/blob/develop/.github/workflows/bindists.yaml and instructions for how to use a similar mechanism for cabal, e.g., to set package constraints for a CI build in order to obtain the "ideal" variant of point 4. That might also be the place to put the results of cabal-version semantics #9552 in. Any ideas for such a central page?
• DONE. The filepath revisions that started this discussion are now done, see Relax upper bounds on filepath #9523, using the "exceptional circumstances" clause from point 5 to skip a portion of point 4 (but not of point 2; @hasufell, please help us understand if that's a deal-breaker).
• I think @ffaf1 concerns about reproducing user problems with revised packages are valid. The branching scheme @andreasabel outlined above is a good solution, but it adds an overhead, for which the maintainers and the revision requesters may not be ready, and it requires documentation to be useful. Perhaps we could start with a modest documentation (first in a new ticket somewhere or in the wiki?) that links to the revisions FAQ and mentions cabal get as a way to get the exact revised package, --allow-newer to try to imitate the revisions (if they are relaxations of bounds) and the CI constraints trick from above.
• I still owe loyalty to that (imagined?) nameless cabal user or dev bitten by revisions who --- when I was the lone cabal maintainer (except for the solver co-maintained by @grayjay) aided by the sage Hackage maintainer --- objected to a revision, but I made it, but I agreed to be conservative with future revisions. So, the policy has changed. The person did not chime in during the process. We are still open to feedback. The least I can do is to help documenting how to debug in the presence of revisions in hope that alleviates any problems that user/dev has had with revisions in the past (which I forgot or they are entirely made up by me --- a lesson to put user stories and policies in writing).
• Anything else?

[hasufell]

We do not really want to supplant maintainers or interfere with their line of work out of the blue.

Just to be clear. I did not suggest to overrule or circumvent cabal mantainers.

My opinion on keeping project boundaries is very strong.

My expression was based on @Mikolaj indicating he trusts hackage trustees (conversation on matrix):

BTW, feel free to ask a Hackage trustee to do the revision --- I trust them and they'd take responsiblity

up to them if via testing or verifying your proof

I have nothing further to add.

[juhp]

I really wish to see movement in the opposite direction: a lot of distros are still on 3.8 or earlier, or just adopting it now.

The ecosystem and community would be better served by maintaining more than one stable branch and ideally their minor releases too.

eg I feel that

• [cabal-3.8 backport] stop cabal update generating dist-newstyle/ #9591

probably really warrants a new 3.8 release.

Also the fact that 3.8 couldn't/can't be built with ghc-9.4 (without revision) is kinda ridiculous... and this keeps happening.

[Mikolaj]

@juhp: thank you for your comment. BTW, you posted it twice; feel free to remove one of the copies. I think the topic deserves its own ticket, because not revising many releases (the intersection of your suggestions with this policy) is only a small aspect of the issue of active maintenance of many branches at once. Please open the ticket and move the discussion there, if you'd like to continue the thread.

My seed input for the new ticket is the following: I think, to be able to manage many active branches, we need a better release process, for which we need more release managers to use it and simplify it (maybe even release managers responsible for a single branch until it's officially retired?), for which we need a deeper developer pool to elect the release managers from. The single biggest practical problem with many active branches is the fragility of our GHA CI, which makes it necessary to produce many hacks and patches, some of them conflicting with normal commits that are definitely not getting backported. So this is real work, for which we need volunteers. Let me ask in the next dev's meeting if anybody would like to volunteer to indefinitely release-manage an old branch (e.g., 3.10 after 3.10.3, which finally stabilizes it, I hope).

[juhp]

@Mikolaj I agree with you, particularly working CI seems a prerequisite.

Okay I will try to open an issue soon to continue the discussion, thanks

[andreabedini]

This might have already been discussed but I don't see the answer explicitly articulated in this thread. How would a cabal policy relate to the work of hackage trustees? As a hackage trustee, I am mostly interested in keeping old versions of cabal-install (as uploaded on hackage) alive by making sure they still build correctly. Apologies if I bring up an old discussion; I just got back from holidays :)

[Mikolaj]

This might have already been discussed but I don't see the answer explicitly articulated in this thread. How would a cabal policy relate to the work of hackage trustees? As a hackage trustee, I am mostly interested in keeping old versions of cabal-install (as uploaded on hackage) alive by making sure they still build correctly. Apologies if I bring up an old discussion; I just got back from holidays :)

Indeed, this discussion and policy was, implicitly, only about bound bumps needed for recent cabal versions to support new versions of packages. We are just waking up to the other case, which is bumps of only old versions of cabal. Dear Hackage trustees, please do your thing as usual, while we discuss whether to reflect these bound changes on the old branches (with broken CI) or not here: #9616