Make it easier for extensions to extend the TocTree directive

[nwf]

Rather than the approach taken by #2354,
which would require answering some tricky questions about the entire set
of Builders in the known universe, just expose just enough through hooks or
overridable routines so that individual instances of Sphinx can splice
in their own implementation of this functionality if it is desired.

[tk0miya]

Could you describe this to me, please? I don't understand what will be changed by this PR.

[nwf]

OK, I have finally seen this approach through and it's resulted in some things being simpler than I expected, so I think there are now fewer changes to the sphinx core. In any case, with these deltas applied, I can do what I want with a pretty small extension overriding the toctree directive:

import sphinx
from docutils import nodes
from docutils.statemachine import ViewList
from sphinx import addnodes
from sphinx.directives.other import TocTree

# Override TocTree directive to support inline explicit markup
class TocTree2(TocTree) :

    def _parse_one_entry(self, all_docnames, toctree, ret, entry):
        if entry.startswith("_ "):

            # Parse the inline syntax
            node = nodes.paragraph()
            self.state.nested_parse(ViewList([entry[2:]]), 0, node)

            # Put the parsed node as a child of the toctree so that it gets
            # crawled over by traversals (notably, including reference
            # resolution).  Because toctree nodes are handled (very) specially,
            # this won't get emitted into the resulting document
            toctree.children.append(node)

            # Dress up the node like the other toctree entries will be
            para = addnodes.compact_paragraph('', '')
            para.children = node.children[0].children
            item = nodes.list_item('', para)
            toc = nodes.bullet_list('', item)

            # Put the dressed up thing in the toctree entries
            toctree['entries'].append((None, toc))
        else:
            return super()._parse_one_entry(all_docnames, toctree, ret, entry)

def setup(app):
    app.add_directive("toctree", TocTree2, override=True)

[tk0miya]

Thank you for your explanation. Now I understand your idea. I can accept the refactoring of TocTree directive (But I can't promise that will be kept as a public interface in the future). On the other hand, I'm hesitating to change the internal structure of the toctree.

Note: I found you could hack the toctree via filling env.toc using dummy docname. It will be copied into the rendered toctree on TocTree.resolve().

[nwf]

Note: I found you could hack the toctree via filling env.toc using dummy docname. It will be copied into the rendered toctree on TocTree.resolve().

I'm afraid it's my turn to ask for clarification or a worked example.

[tk0miya]

I just made a hack code to do that.

diff --git a/sphinx/directives/other.py b/sphinx/directives/other.py
index e4fcc0f5c..3a3c97f19 100644
--- a/sphinx/directives/other.py
+++ b/sphinx/directives/other.py
@@ -16,6 +16,7 @@ from docutils.parsers.rst import directives
 from docutils.parsers.rst.directives.admonitions import BaseAdmonition
 from docutils.parsers.rst.directives.misc import Class
 from docutils.parsers.rst.directives.misc import Include as BaseInclude
+from docutils.statemachine import StringList

 from sphinx import addnodes
 from sphinx.deprecation import RemovedInSphinx40Warning, deprecated_alias
@@ -116,6 +117,25 @@ class TocTree(SphinxDirective):
                     ref = explicit.group(2)
                     title = explicit.group(1)
                     docname = ref
+                elif entry.startswith('_ '):
+                    # Generate content from given content (ex. "_ :ref:`blah blah`")
+                    node = nodes.Element()
+                    self.state.nested_parse(StringList([entry[2:]]), 0, node)
+                    toctree['entries'].append((None, entry))
+
+                    para = addnodes.compact_paragraph('', '', *node[0].children)
+                    list_item = nodes.list_item('', para)
+
+                    # Put generated content to env.tocs as a toctree of pseudo document.
+                    # To be precise, `entry` is not a not docname. So this assignment
+                    # is invalid.  This is only used to hack toctree resolver.
+                    #
+                    # Note:
+                    # * This will not work in parallel build.
+                    # * I noticed toctree resolver does not support pending_xref node.
+                    #   So :ref: will not work (the idea of #7913 is also).
+                    self.env.tocs[entry] = nodes.bullet_list('', list_item)
+                    continue
                 else:
                     ref = docname = entry
                     title = None

I believe this might work as your example. But, as commented in above code, I noticed both codes would not work because toctree resolver does not support resolving "pending_xref" node in the toctree.

[nwf]

I believe this might work as your example. But, as commented in above code, I noticed both codes would not work because toctree resolver does not support resolving "pending_xref" node in the toctree.

I'm not sure what you mean by "the toctree resolver" here, because AIUI pending_xrefs are resolved by the ReferencesResolver in

sphinx/sphinx/transforms/post_transforms/__init__.py

Lines 63 to 71 in e4c3e24

	class ReferencesResolver(SphinxPostTransform):
	"""
	Resolves cross-references on doctrees.
	"""
	default_priority = 10
	def run(self, **kwargs: Any) -> None:
	for node in self.document.traverse(addnodes.pending_xref):

. That is why the code I gave in #7913 (comment) above was careful to add the parsed node to the children of the toctree node, so that the .traverse() in the ReferencesResolver would visit it.

[nwf]

(Updated commit to try to appease flake8 and mypy. I cannot, sadly, make mypy work locally, so am reduced to guessing and trying CI. Apologies for the noise.)

[tk0miya]

I'm not sure what you mean by "the toctree resolver" here

Sorry for wrong wording. I meant "TocTree.resolve()".

That is why the code I gave in #7913 (comment) above was careful to add the parsed node to the children of the toctree node, so that the .traverse() in the ReferencesResolver would visit it.

I still don't try your code yet. But I think ReferencesResolver does not work for the toctree['entries']. I'll try to check yours later (if I have time).

[nwf]

That matches my understanding: ReferencesResolver uses the docutils Node traverse-al and so does not look for nodes within attributes, of which 'entries' is an example. That is, again, why the code I posted is careful to add the node which may contain a pending_xref to the toctree's children, so that ReferencesResolver does indeed find those pending_xrefs and cause them to replace themselves with their resolution. The toctree is simply the most convenient place to hang this node: its children are traversed but are not used when rendering; however, other nodes could serve as parents if desired.

I'm not terribly enthusiastic about munging with env.tocs as in your proposal, since it's abusing the semantics of shared state and, while

sphinx/sphinx/environment/adapters/toctree.py

Line 159 in 8cc0b89

toc = self.env.tocs[ref].deepcopy()

looks like it might do the right thing, surely there's going to be a problem that env.metadata won't have a similar key, so

sphinx/sphinx/environment/adapters/toctree.py

Line 160 in 8cc0b89

maxdepth = self.env.metadata[ref].get('tocdepth', 0)

will throw a KeyError? Moreover,

sphinx/sphinx/environment/adapters/toctree.py

Line 158 in 8cc0b89

refdoc = ref

means that this fake entry is going to propagate through recursion of _entries_from_toctree. That might be OK, but it's an extrusion of new state.

[tk0miya]

oh, I've missed the code that inserts a node to under the toctree node. Okay, I understand.

I'm not terribly enthusiastic about munging with env.tocs as in your proposal, since it's abusing the semantics of shared state

Yes, my idea is just a hack. Not a good way. But, as I commented above, I'd not like to change the internal structure of the toctree. So your TocTree2 directive will not work.

[nwf]

Closing, since I think this approach requires cooperation from the TocTree.