Change in --id-prefix behavior for markdown leads to unexpected output

[jjallaire]

In Pandoc 3.1.4 a change to --id-prefix was made (5615101#commitcomment-124945528) to have its behavior affect IDs in markdown output. Previously the documentation stated that it only affected footnotes for markdown:

Specify a prefix to be added to all identifiers and internal links in HTML and DocBook output, and to footnote numbers in Markdown and Haddock output. This is useful for preventing duplicate identifiers when generating fragments to be included in other pages.

With this change, anyone previously relying on the footnotes-only behavior for markdown will now have all of internal IDs munged with the same prefix. This will likely be surprising / undesirable for these existing usages.

For example, in our visual editor that maps to Pandoc markdown we have previously used the --id-prefix argument to do prefixing of footnotes for book chapters (so when the chapters are combined the footnotes don't conflict). With this change the prefix is now applied to all IDs in the document, which means that user authored IDs are overwritten in a surprising fashion.

Perhaps this behavior should be rolled back or perhaps it should be maintained but a new --footnote-prefix option be added to target the footnote-only case. It seems like footnotes and ids are distinct enough in their role/behavior that this would be warranted.

Separately, the id prefixing behavior seems to stack, so for example if we consider a file test.md:

# Introduction {#intro}

Run this command:

$ pandoc test.md --output test2.md --to markdown --id-prefix foo

Which yields test2.md:

# Introduction {#foointro}

Then run this command:

$ pandoc test2.md --to markdown --id-prefix foo 
# Introduction {#foofoointro}

[patbakdev]

My use case for the original change was to build a box set consisting of two or more books written in markdown with individual, but conflicting chapter ids:

• book1.md:
  • chapter-one
  • chapter-two
• book2.md:
  • chapter-one
  • chapter-two

I still think this is a valid use case and I believe would have indeed worked without the change if the source files were html and not markdown. Adding the capability to markdown seemed logical given the point of the translation between the two.

I read the docs, but must have completely missed the implication around the footnotes.

It seems to me that there are at least two scopes here - file level things (section/div/h1 type ids) and local things (footnotes). I pass in a book for each each prefix and you pass in a chapter for each prefix. If the only "local" scope are footnotes then I think having both an id-prefix and a --footnote-prefix make the most sense. If there are other items in the "local" scope, that sounds like more work to identify and deal with. Figures? Captions?

However, I assume in practice there is no differentiation between global and local? ie. you are using a file per chapter approach??? and I am using a file per book approach. So finding a generic solution may be difficult.

As far as the stacking behavior, that is how I would expect it to work.

That being said, I did have a work-around (passing in the id-prefix as metadata and then using a filter to do the work. I would rather not go back to that since I think this has value for others, but its an escape hatch for me at least if we can't find a solution that works for footnotes beyond rolling back the change.

[jjallaire]

As far as the stacking behavior, that is how I would expect it to work.

The only issue w/ stacking is that it breaks round-tripping (so if you are using pandoc to write markdown from a visual editor it just keeps adding the same prefix over and over). Our issue is that the only way to affect footnote behavior is --id-prefix (footnote IDs are not part of the AST) so we have to call it repeatedly when the user saves (so get the id prefixes stacking on each other).

I do agree that IDs and footnotes are distinct enough in their semantics and usage that they shouldn't be controlled by the same option. That also suggests --footnote-prefix. That said, this change is going to break the output of anyone relying on --id-prefix for footnotes (not sure how many cases of the there are though?). So another way to go would be to leave --id-prefix alone and just introduce a new more flexible mechanism that can handle various scopes (IDs are indeed used for cross referencing by pandoc-crossref, Quarto, and perhaps others?).

[patbakdev]

I see. Stacking is actually what I want in my case since my build system is designed to take an arbitrary hierarchy of markdown files and "roll up" the tree so that there is only one final file which is then processed. I'm not sure if round-tripping (input=output) is or should be an expectation, but I realize there are many ways to lay things out and we all do so differently.

You now have me a little worried that this change may have broken things other than footnotes. Although I would think that would have always been the case for HTML sources given the documentation. (I don't use those other features so I can't say one way or the other.) I am also a strong believer in not breaking backward compatibility without a very good reason (and I do have a work-around in my case).

I think that leaves us with three options:

1. Rollback the Change
2. Add --footnote-prefix (Assuming this is a reasonable fix for this userbase, and wait and to see if there are any other complaints.)
3. Do a more thorough investigation around all uses of ids (unless someone else already has this information) and implement a more thorough change.

I would prefer #2, but I can easily see a good argument and can live with #1 (I really think we should do #3, but since I don't have the time or experise to do the work myself, I can't in good conscious push for it).

[jjallaire]

@jgm We've temporarily rolled back to Pandoc v3.1.2 to mitigate the new behavior. What are your thoughts on changes here? (if no changes are contemplated we can look for more elaborate hacks as workarounds but if some change is anticipated we'll certainly wait for that).