Add Python API documentation generation extension

[jbms]

Fixes #91

[jbms]

Note: This still needs some revision, especially the documentation, but I'd welcome feedback.

[jbms]

Also, will probably want to rename this extension along with many of the others, but I figured I'd wait to do that until #98 is merged to avoid conflicts.

[2bndy5]

yeah (thank you) - Refactoring should definitely get a separate PR.

[2bndy5]

I still find it offsetting when I click on a class name in a sig and find my self on a nested page. The number 1 reason I love this theme is the hamburger menu. And, when I'm transported to a nested page (for class summary), the hamburger menu isn't exactly showing the correct items.

For Example, the class overview for Dim only shows the public members in the hamburger menu. When I click on hull() (from the menu - not the page body), I find my self on another page and the hamburger menu is correct (but didn't change).

So, I think the global toc is structured strange. Notice after clicking on hull(), you'd have to back up 5 menus to get back to the overview page for Dim. This seems like each menu for the different sections of the class overview are stacked, but the left-side toc (when viewport is sufficiently large), the 5 sections are displayed at the same level (all within the class' menu).

Sorry for the long post, I'm not sure if I'm finding the right words here.

[2bndy5]

Some of the docstrings have unusual syntax because they match the output from
pybind11

Is this demo pre-compiled? I noticed that the tensorstore also uses 0-level indentation for docstrings. FYI, pybind11 will auto-dedent raw doctrings, so you can maintain the same level indentation in the C++ src. I can understand if it was done that way for easy docstring composition (when writing or copy-paste the docs srcs).

[jbms]

I actually changed how the TOC works compared to in the tensorstore docs. Previously the logic for splitting the toc into a global and local toc was not quite right (see first commit in this PR). Now, when in the "desktop" mode the links to members are in the left-side "global toc" and take you to a different page, while the right-side "local toc" (which contains sections, but also parameter names) takes you to links within the same page.

When in the "mobile" / "tablet" mode, though, I agree that the menu is behaving rather weirdly, with it starting out nested deep in a menu for some reason, and the back button moving to the previous sub-menu rather than taking you up a level. I will have to look into what is going on there.

[jbms]

Yes, the tensorstore.py file was generated using a modified version of https://github.com/sizmailov/pybind11-stubgen plus a bunch of manual editing. I didn't want to introduce the complexity of building an actual pybind11 extension as part of the doc build.

The reason there is no indentation is that the logic for parsing pybind11 overloaded function signatures expects the precise formatting used by pybind11 and does not work if the docstring is indented as is customary in Python code. I think it should work if the individual docstrings specified to pybind11 in the C++ code are indented, since as you say pybind11 will dedent them, and additionally, the additional syntax used by pybind11 to indicate overloaded functions would not be indented in any case.

The reason the tensorstore C++ code uses no indentation in its docstrings is unrelated:

• there is some programmatic composition of the docstrings for which differing indentation would be problematic
• they would be wrapped to inconsistent lengths

[mhostetter]

@jbms this is amazing!! ❤️ 🎉 This is the feature I've been dreaming of since I saw your PR in the sphinx-material repo and discovered your Tensorstore. I thought I'd test this on my library and provide feedback.

When trying to use the new python-apigen feature, the build crashed. I'm still investigating that and trying to provide a useful/reproducible error output.

But before sharing that, I discovered another issue. Using this branch without adding the "sphinx_immaterial.python_apigen" extension, the build no longer documents or displays properties. I traced it back to the last commit f2d5349.

I built the latest from my library using these two adjacent commits.

55a6bb7

f2d5349

[2bndy5]

This isn't the first time we've encountered regressions to conf.py usage. We could try using pytest with tox to run tests... Note sure what the best way to verify the test output is - I think sphinx-design uses pseudo XML comparison.

[mhostetter]

Regarding the modified TOC, here are my two cents (if they're worth anything).

I find all the class methods and attributes in the left side TOC a little confusing and cluttered. The left-side TOC is already displaying so much and to put all the sub-contents on the left as well seems, to me, to be a bit much. The right-side TOC doesn't really have a purpose anymore (for API documentation).

Perhaps both the left and right TOC can contain the contents, but the left-side TOC is not expanded by default? See below.

Current class

Expected/desired class (made in paint)

And then when you click on a method or property, it takes you to that page, but you can still see where you are in the hierarchy on the left. (The below screenshot is the current and desired behavior. No change requested.)

[2bndy5]

@mhostetter It would be difficult to please that request because you seem to want right side toc used for all but the deepest level. If I'm understanding the mock results and desired behavior, then the entities would end up duplicated for different locations/levels of the TOCs.

This could be a preference thing also. I kinda like the current intention - just need to fix the unexpected nesting.

It would be cool if the right side ToC is auto-hidden for pages with no local ToC to show; maybe it does that and I missed it - in any case, that'd be a separate issue.

[jbms]

Thanks @mhostetter and @2bndy5 for your feedback. I have struggled to figure out the best behavior for the TOC.

Originally when first designing this functionality for tensorstore my vision was exactly as @mhostetter described, with the right-side TOC serving as a "class outline".

The way this theme worked prior to the "Fix separation of local and global tocs" commit in this PR, is that the branch of the global TOC corresponding to the current page is simply cut off and moved to the right side (and then pruned so that it doesn't include multiple levels of headings from other pages). When paired with the python_apigen extension, that does result in the right-side TOC containing a "class outline" when on the page that documents a class. However, there are two downsides to this approach:

1. As @2bndy5 pointed out, it is then not clear which links are to the current page and which links take you to a different page.
2. When trying to use the TOCs to "drill down" repeatedly several levels of the docs, it is a bit jarring because entries are constantly moving between the left and right sides.

In the tensorstore docs, I tried to mitigate the second issue by using a special option on the API documentation pages for classes to duplicate the contents of the local TOC on the left side as well (rather than cutting it out). That allows you to continue to drill down on the left side without having to switch back and forth between the left and right side, while preserving the "class outline" on the right side. But it does not solve the first issue, and also duplicating the TOC doesn't seem very elegant.

Potentially some of the issues could be mitigated by the solutions outlined in #58.

Collapsing by default certain parts of the left side TOC is also an interesting option, but I am worried it would also make it less convenient to "drill down" multiple levels using the left side TOC.

With the current design in this PR, I agree that the right side TOC is not too useful for most classes, since it just contains the section headings (which are also in the left side TOC). For functions it is still useful since it shows parameters:

https://sphinx-immaterial--99.org.readthedocs.build/en/99/python_apigen_generated/tensorstore.Dim.__init__-unbounded.html

As far as auto-hiding the right-side TOC if it is empty, I think that actually used to be the behavior of upstream mkdocs-material but was changed a while back. I'm not sure what the motivation for that change was, but perhaps it was to ensure a consistent width for the main content regardless of whether there is a TOC, to keep the layout structure more consistent as you navigate between pages.

[jbms]

I will have to look into why properties are no longer being included in the TOC.

[mhostetter]

I will have to look into why properties are no longer being included in the TOC.

@jbms The properties are not documented on the page at all, with the exception of the broken autosummary table links.

1. As @2bndy5 pointed out, it is then not clear which links are to the current page and which links take you to a different page.

Here's an idea -- what about prepending a "->" (or some equivalent symbol) before the "M" and "P" colored icons in the right-side TOC to indicate this link is taking you to another page. I imagine this is only useful for API documentation.

This is a quick-and-dirty paint example.

[2bndy5]

Here's an idea -- what about prepending a "->" (or some equivalent symbol) before the "M" and "P" colored icons in the right-side TOC to indicate this link is taking you to another page

I think material design has an icon specifically for that

[mhostetter]

A documentation suggestion: In your usage section of the docs, you have:

python_apigen_modules = {
      "my_module": "api",
}

I would suggest you add that a user should create a page with this, for example:

my_module
=========

.. python-apigen::
   :fullname: my_module
   :objtype: module

It wasn't obvious to me how to invoke the API generation. I had to look at the python_apigen_demo.rst page to figure it out.

[jbms]

Thanks @mhostetter.

To be clear, all of the individual documentation pages are generated regardless of that directive, but they are not linked from anywhere else. The directive just inserts a summary of the module.

One change I have been considering is to change the directive so that it inserts a summary of a single group rather than the entire module.

For example, currently if we have multiple groups can do:

.. python-apigen::
   :fullname: my_module
   :objtype: module

Core
====

This is documentation for core stuff.

Indexing
=========

This is documentation for indexing stuff.

Instead, it would become:

Core
====

This is documentation for core stuff.

.. python-apigen:: core

Indexing
=========

This is documentation for indexing stuff.

.. python-apigen:: indexing

This would allow you to document entities from multiple different modules together, if they happen to be assigned the same group identifier.

This also matches what I've done for the C++ autosummary (not yet made into a PR).

[mhostetter]

I like that idea. It gives more control over the presentation of the API.

Also @jbms , I determined why the current state of this PR fails to build my docs. It is related to this chunk of code:

sphinx-immaterial/sphinx_immaterial/python_apigen.py

Lines 676 to 689 in f2d5349

	for entry, entry_node in group_members:
	# TODO(jbms): Currently sphinx does not have particularly good support
	# for a page occurring in multiple places in the TOC, which occurs with
	# inherited members. Nonetheless it mostly works. The next/prev ordering
	# of documents is based on first occurrence of a document within pre-order
	# traversal, which gives the correct result as long as the base class
	# occurs before the derived class. The parent relationship for TOC
	# collapsing, however, appears to be based on the last occurrence.
	# Ideally we would be able to explicitly control the parent relationship
	# for TOC collapsing, so that a member is listed under its "real" classs.
	docname = object_docnames[entry.object_name]
	toc_entries.append((entry.toc_title, docname))
	section += entry_node

I was able to reproduce the error I get in my docs with this simple foo module.

The foo module

# foo/__init__.py
from ._bar import *

# foo/_bar.py
def function_1(arg_1: str, arg_2: int) -> str:
    """
    This is a docstring.
    """
    return arg_1 * arg_2


class Charlie:
    def my_method(self, arg_1: str) -> str:
        """
        This is a docstring of a method.
        """
        return arg_1

    @property
    def my_property(self) -> int:
        """
        This is a docstring of a property.
        """
        return 1


class Delta(Charlie):
    @classmethod
    def my_classmethod(cls, arg_1: str) -> str:
        """
        This is a docstring of a method.
        """
        return arg_1

Error output

Running Sphinx v4.5.0
loading translations [en]... locale_dir /mnt/c/Users/matth/repos/galois/docs/locales/en/LC_MESSAGES does not exists
locale_dir /mnt/c/Users/matth/repos/galois/docs/locales/en/LC_MESSAGES does not exists
done
making output directory... done
locale_dir /mnt/c/Users/matth/repos/galois/docs/locales/en/LC_MESSAGES does not exists
[autosummary] generating autosummary for: api/foo.rst, index.rst
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://numpy.org/doc/stable/objects.inv...
myst v0.17.2: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=[], linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, footnote_transition=True, sub_delimiters=('{', '}'), words_per_minute=200)
locale_dir /mnt/c/Users/matth/repos/galois/docs/locales/en/LC_MESSAGES does not exists
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 2 source files that are out of date
updating environment: locale_dir /mnt/c/Users/matth/repos/galois/docs/locales/en/LC_MESSAGES does not exists
[new config] 8 added, 0 changed, 0 removed
reading sources... [ 12%] api/foo
reading sources... [ 25%] api/foo.Charlie
reading sources... [ 37%] api/foo.Charlie.my_method
reading sources... [ 50%] api/foo.Charlie.my_property
reading sources... [ 62%] api/foo.Delta

Traceback (most recent call last):
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/events.py", line 94, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 604, in object_description_transform
    self._merge_summary_nodes_into(contentnode)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 731, in _merge_summary_nodes_into
    self._add_group_summary(
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 687, in _add_group_summary
    docname = object_docnames[entry.object_name]
KeyError: 'foo._bar.Charlie.my_method'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/application.py", line 330, in build
    self.builder.build_update()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 286, in build_update
    self.build(to_build,
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 300, in build
    updated_docnames = set(self.read())
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 407, in read
    self._read_serial(docnames)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 428, in _read_serial
    self.read_doc(docname)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 468, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/io.py", line 181, in read_doc
    pub.publish()
  File "/usr/lib/python3/dist-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/io.py", line 101, in read
    self.parse()
  File "/usr/lib/python3/dist-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/parsers.py", line 89, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 171, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2343, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2355, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2097, in directive
    return self.run_directive(
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2147, in run_directive
    result = directive_instance.run()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 769, in run
    return self._make_object_description()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 748, in _make_object_description
    objdesc, _ = self._generate_autodoc(self._entry)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 613, in _generate_autodoc
    for x in sphinx.ext.autodoc.directive.parse_generated_content(
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/ext/autodoc/directive.py", line 108, in parse_generated_content
    nested_parse_with_titles(state, content, node)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/util/nodes.py", line 340, in nested_parse_with_titles
    return state.nested_parse(content, 0, node, match_titles=1)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 197, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2343, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2355, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2097, in directive
    return self.run_directive(
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2147, in run_directive
    result = directive_instance.run()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/domains/__init__.py", line 281, in run
    return super().run()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/apidoc_formatting.py", line 178, in run
    nodes = orig_run(self)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/directives/__init__.py", line 202, in run
    self.env.app.emit('object-description-transform',
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/application.py", line 440, in emit
    return self.events.emit(event, *args, allowed_exceptions=allowed_exceptions)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/events.py", line 102, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function PythonApigen._generate_autodoc.<locals>.object_description_transform at 0x7f76fe6f7160> for event 'object-description-transform' threw an exception (exception: 'foo._bar.Charlie.my_method')

Extension error (sphinx_immaterial.python_apigen):
Handler <function PythonApigen._generate_autodoc.<locals>.object_description_transform at 0x7f76fe6f7160> for event 'object-description-transform' threw an exception (exception: 'foo._bar.Charlie.my_method')

I believe the error is related to the fact that Charlie is declared in foo._bar.Charlie but is imported in the foo namespace? I can confirm if the contents of foo/_bar.py are placed in foo/__init__.py the docs build. Perhaps there needs to be a distinction between where the item is declared (eg foo._bar.Charlie) and how it is imported (eg foo.Charlie) in the object_docnames dictionary?

Thanks again for the PR and taking the time.

[jbms]

@mhostetter I'm still not sure about the cause of the attributes not displaying ---- when trying to build the documentation for your library, I was getting various unrelated build errors. However, it probably has to do with the changes to autodoc_property_type.py --- you might try adding some additional debugging output there.

[mhostetter]

@jbms as of today (I noticed some recent force pushes), the properties appear now as they did before and my docs build on this branch without modifying conf.py (i.e., not using python-apigen). This issue #99 (comment) is no longer applicable. 🎉

When building with python-apigen, I have noticed some build errors. I created an example project to demonstrate the issue. When using class inheritance, I get a build error. From my example, the class Delta is documented fine, however when Delta inherits from Charlie the builds fails with this output. I believe this has the same root cause as what I reported here #99 (comment).

Build output

['/home/matt/.local/bin/sphinx-build', '-b', 'html', '-v', 'docs/', 'docs/build/']
Running Sphinx v4.5.0
loading translations [en]... locale_dir /mnt/c/Users/matth/repos/foo/docs/locales/en/LC_MESSAGES does not exists
locale_dir /mnt/c/Users/matth/repos/foo/docs/locales/en/LC_MESSAGES does not exists
done
making output directory... done
locale_dir /mnt/c/Users/matth/repos/foo/docs/locales/en/LC_MESSAGES does not exists
[autosummary] generating autosummary for: api/foo.Charlie.my_method.rst, api/foo.Charlie.my_property.rst, api/foo.Charlie.rst, api/foo.Delta.my_classmethod.rst, api/foo.Delta.rst, api/foo.function_1.rst, api/foo.rst, index.rst
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://numpy.org/doc/stable/objects.inv...
myst v0.17.2: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=[], linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, footnote_transition=True, sub_delimiters=('{', '}'), words_per_minute=200)
locale_dir /mnt/c/Users/matth/repos/foo/docs/locales/en/LC_MESSAGES does not exists
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 8 source files that are out of date
updating environment: locale_dir /mnt/c/Users/matth/repos/foo/docs/locales/en/LC_MESSAGES does not exists
[new config] 8 added, 0 changed, 0 removed
reading sources... [ 12%] api/foo
reading sources... [ 25%] api/foo.Charlie
reading sources... [ 37%] api/foo.Charlie.my_method
reading sources... [ 50%] api/foo.Charlie.my_property
reading sources... [ 62%] api/foo.Delta

Traceback (most recent call last):
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/events.py", line 94, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 608, in object_description_transform
    self._merge_summary_nodes_into(contentnode)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 737, in _merge_summary_nodes_into
    self._add_group_summary(
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 693, in _add_group_summary
    docname = object_docnames[entry.object_name]
KeyError: 'foo._bar.Charlie.my_method'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/application.py", line 330, in build
    self.builder.build_update()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 286, in build_update
    self.build(to_build,
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 300, in build
    updated_docnames = set(self.read())
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 407, in read
    self._read_serial(docnames)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 428, in _read_serial
    self.read_doc(docname)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 468, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/io.py", line 181, in read_doc
    pub.publish()
  File "/usr/lib/python3/dist-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/io.py", line 101, in read
    self.parse()
  File "/usr/lib/python3/dist-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/parsers.py", line 89, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 171, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2343, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2355, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2097, in directive
    return self.run_directive(
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2147, in run_directive
    result = directive_instance.run()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 775, in run
    return self._make_object_description()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 754, in _make_object_description
    objdesc, _ = self._generate_autodoc(self._entry)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/python_apigen.py", line 617, in _generate_autodoc
    for x in sphinx.ext.autodoc.directive.parse_generated_content(
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/ext/autodoc/directive.py", line 108, in parse_generated_content
    nested_parse_with_titles(state, content, node)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/util/nodes.py", line 340, in nested_parse_with_titles
    return state.nested_parse(content, 0, node, match_titles=1)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 197, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2343, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2355, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2097, in directive
    return self.run_directive(
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2147, in run_directive
    result = directive_instance.run()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/domains/__init__.py", line 281, in run
    return super().run()
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx_immaterial/apidoc_formatting.py", line 198, in run
    nodes = orig_run(self)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/directives/__init__.py", line 202, in run
    self.env.app.emit('object-description-transform',
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/application.py", line 440, in emit
    return self.events.emit(event, *args, allowed_exceptions=allowed_exceptions)
  File "/home/matt/.local/lib/python3.8/site-packages/sphinx/events.py", line 102, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function PythonApigen._generate_autodoc.<locals>.object_description_transform at 0x7fb04274d550> for event 'object-description-transform' threw an exception (exception: 'foo._bar.Charlie.my_method')

Extension error (sphinx_immaterial.python_apigen):
Handler <function PythonApigen._generate_autodoc.<locals>.object_description_transform at 0x7fb04274d550> for event 'object-description-transform' threw an exception (exception: 'foo._bar.Charlie.my_method')

To reproduce, run sphinx-build -b html -v docs/ docs/build/ in the foo/ directory of this zip (which has foo/ and docs/ folders in it).

foo_example.zip

As always, thanks for your time and effort!

[2bndy5]

well it looks like sphinx recently moved to 5.0.1 as latest. I get a completely different error when using v5.0.1 (please note the deprecation warnings):

path\to\foo\venv\lib\site-packages\sphinx\ext\autodoc\__init__.py:618: RemovedInSphinx60Warning: The implementation of Documenter.get_object_members() will be removed from Sphinx-6.0.  
  warnings.warn('The implementation of Documenter.get_object_members() will be '
path\to\foo\venv\lib\site-packages\sphinx\ext\autodoc\__init__.py:618: RemovedInSphinx60Warning: The implementation of Documenter.get_object_members() will be removed from Sphinx-6.0.  
  warnings.warn('The implementation of Documenter.get_object_members() will be '
path\to\foo\venv\lib\site-packages\sphinx\ext\autodoc\__init__.py:618: RemovedInSphinx60Warning: The implementation of Documenter.get_object_members() will be removed from Sphinx-6.0.  
  warnings.warn('The implementation of Documenter.get_object_members() will be '
path\to\foo\venv\lib\site-packages\sphinx\ext\autodoc\__init__.py:618: RemovedInSphinx60Warning: The implementation of Documenter.get_object_members() will be removed from Sphinx-6.0.  
  warnings.warn('The implementation of Documenter.get_object_members() will be '

Extension error (sphinx_immaterial.python_apigen):
Handler <function _builder_inited at 0x00000222FA21F370> for event 'builder-inited' threw an exception (exception: AttributeDocumenter.get_doc() takes 1 positional argument but 2 were given)

If I switch back to v4.5.0, then I can reproduce the error reported by @mhostetter , but I also get warnings about duplicate object descriptions:

foo.Charlie:1: WARNING: duplicate object description of foo.Charlie, other instance in api/foo.Charlie, use :noindex: for one of them
foo.Charlie:1: WARNING: duplicate object description of foo._bar.Charlie, other instance in api/foo.Charlie, use :noindex: for one of them

Extension error (sphinx_immaterial.python_apigen):
Handler <function PythonApigen._generate_autodoc.<locals>.object_description_transform at 0x00000289FE26A680> for event 'object-description-transform' threw an exception (exception: 'foo._bar.Charlie.my_method')

[mhostetter]

FWIW, @2bndy5 I do not get those duplicate object warnings with Sphinx v4.5.0...

[2bndy5]

Oh we can't use -W yet; there's a couple warnings that would make the CI fail if we do. The best we can do is either no special args or -n, but the later might reveal warnings that we shouldn't be concerned with.

[jbms]

Reviewed apigen.py and noticed new build warnings. I would prefer we have -W enabled (in the CI) for the html builder, but we'll have to wait until #112 is addressed (specifically for our example usage of python optional syntax).

Actually there was a bug a in external_cpp_references that led to the warnings about nlohmann::json in some cases. That is now fixed.

I also fixed all of the warnings regarding missing references, and enabled -W for the CI build.

[jbms]

Thanks for the review!

[2bndy5]

tagged as v0.8.0