Doctest directives and comments missing from code samples

[stevendaprano]

BPO	36675
Nosy	@terryjreedy, @gpshead, @ezio-melotti, @merwok, @stevendaprano, @jdemeyer, @willingc, @asottile, @JulienPalard, @JDLH, @csabella, @miss-islington
PRs	bpo-36675: Fix doctest directives comments disappear bug #13125 bpo-36675: Remove obsolete code. #16024 [3.8] bpo-36675: Remove obsolete code. (GH-16024) #16030 bpo-36675: Doc: Reveal doctest directives #23620
Files	missing_directives.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-04-20.10:43:50.531>
labels = ['3.8', 'type-bug', '3.7', 'docs']
title = 'Doctest directives and comments missing from code samples'
updated_at = <Date 2021-10-26.04:00:41.470>
user = 'https://github.com/stevendaprano'

bugs.python.org fields:

activity = <Date 2021-10-26.04:00:41.470>
actor = 'minghua'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2019-04-20.10:43:50.531>
creator = 'steven.daprano'
dependencies = []
files = ['48277']
hgrepos = []
issue_num = 36675
keywords = ['patch']
message_count = 17.0
messages = ['340570', '340959', '342648', '342651', '342652', '342655', '342664', '344714', '352093', '352120', '352128', '367173', '383075', '383732', '384361', '385632', '385661']
nosy_count = 14.0
nosy_names = ['terry.reedy', 'gregory.p.smith', 'ezio.melotti', 'eric.araujo', 'steven.daprano', 'docs@python', 'jdemeyer', 'willingc', 'Anthony Sottile', 'mdk', 'JDLH', 'cheryl.sabella', 'miss-islington', 'minghua']
pr_nums = ['13125', '16024', '16030', '23620']
priority = 'normal'
resolution = 'later'
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36675'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

[stevendaprano]

(Apologies if this is the wrong place for reporting website bugs.)

The website is not rendering doctest directives or comments, either that or the comments have been stripped from the examples.

On the doctest page itself, all the comments are missing:

https://docs.python.org/3/library/doctest.html#directives

The first example says:

    >>> print(list(range(20))) 
    [0,   1,  2,  3,  4,  5,  6,  7,  8,  9,
    10,  11, 12, 13, 14, 15, 16, 17, 18, 19]

but without the directive, the test would fail. Screen shot attached.

Doctest directives are also missing from here:

https://docs.python.org/3/library/ctypes.html

My browser: Firefox 45.1.1

Also checked with text browser "lynx".

[terryjreedy]

I verified that the line in Doc/library/doctest.rst has the comment.
"
For example, this test passes::

   >>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE
   [0,   1,  2,  3,  4,  5,  6,  7,  8,  9,
   10,  11, 12, 13, 14, 15, 16, 17, 18, 19]
"
However, the comment is omitted from the .html built on Windows by Sphinx 1.8.1.
"
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="nb">print</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">20</span><span class="p">)))</span> 
   ***html for comment should be here***
<span class="go">[0,   1,  2,  3,  4,  5,  6,  7,  8,  9,</span>
<span class="go">10,  11, 12, 13, 14, 15, 16, 17, 18, 19]</span>
</pre></div>
</div>
"
To me, this is a bug with building the .html for doctest.rst.  Comments are preserved elsewhere.  For instance, the code example for https://docs.python.org/3/library/functions.html#dir.  The .rst file has
"
   The resulting list is sorted alphabetically.  For example:

      >>> import struct
      >>> dir()   # show the names in the module namespace  # doctest: +SKIP
"
Both use 3-space colon + indents to mean 'example block'.  The only difference is '::' versus ':'.  https://devguide.python.org/documenting/#source-code says :: is required.  idle.rst also has a code example with comments displayed (I just submitted a PR to not suppress color highlighting.)

I leave it to the doc experts to discover why the comments are not included in doctest.html.

[jdemeyer]

Isn't it a *feature* that those doctest directives are not shown? Those directives are meant for the doctest module only, not for the reader of the rendered documentation.

[merwok]

OP is about the documentation page for doctest itself!

[terryjreedy]

Doctest directives in code examples should be suppressed everywhere *except* in the doctest.html examples showing how to use directives. The patch only exposes them for doctest.html and not for ctypes or anywhere else.

They really should not be in the dir example code that I linked to.
https://docs.python.org/3/library/functions.html#dir
The problem there are the double comments with both a real comment and a directive.

https://devguide.python.org/documenting/#source-code does not say anything about '::' causing suppression of comments and ':' leaving them. It is misleading in implying the '::' is required for a code block.

[jdemeyer]

Doctest directives in code examples should be suppressed everywhere *except* in the doctest.html examples showing how to use directives.

Thanks for clarifying. I missed that.

[stevendaprano]

Doctest directives in code examples should be suppressed everywhere
*except* in the doctest.html examples showing how to use directives.
The patch only exposes them for doctest.html and not for ctypes or
anywhere else.

Thanks for the patch, and the extra information, but I disagree with the
decision to suppress the directives.

The reason I found this problem in the first case was that I started
with the ctypes documentation, where it says:

"Since some code samples behave differently under Linux, Windows, or Mac
OS X, they contain doctest directives in comments."

and I was very keen to see those directives so I could learn the right
way to deal with platform-dependent doctests. I was very confused that
they weren't visible.

https://docs.python.org/3/library/ctypes.html

I think that doctest directives are as much a part of documenting
correct usage as any other part of the example code, and they are
(semi-)human readable and (almost) self-documenting.

Consider this example from ctypes:

    >>> c_wchar_p("Hello, World")
    c_wchar_p(140018365411392)

In the absence of a directive, but knowing that it may have been
surpressed, I don't know how to interpret that. Is the output some
arbitrarily chosen value that doctest ought to skip? Or is that the
actual output that c_wchar_p("Hello, World") will return every single
time without fail?

If I was a ctypes expert, it might be blindingly obvious to me, but I'm
not, so I'm left in the dark. I don't know whether I should expect that
precise output each and every time, or something platform and
implementation specific.

If the directive #doctest:+SKIP was visible, I would know that it was an
arbitrarily chosen example.

My preference would be:

• keep the doctest directives visible, everywhere;

• make them a clickable link to the appropriate section
  in the doctest documentation;

• and, if possible, on mouse-over, they should display a
  tooltip with a message like

  "The output of this example is arbitrary."

Or similar wording.

They really should not be in the dir example code that I linked to.
https://docs.python.org/3/library/functions.html#dir

On the contrary: I think that the presence of the +SKIP directive helps
demonstrate that the output shown is a made-up example, not normative.

(Of course it helps that I know doctest, but even if I didn't, the
tooltip message would help.)

[csabella]

;tldr; There is a global configuration flag to show all the doctest directives for all the docs that are built. The default is to suppress the doctest directives. If a global setting isn't desired, then each doctest example will have to be changed individually.

I've looked at the Sphinx documentation for this and there are a few points to reference.

• `::` defines a Literal block in reST. [1] A literal block doesn't do anything special except from the last paragraph is this section of the docs:
  "Code highlighting can be enabled for these literal blocks on a document-wide basis using the `highlight` directive and on a project-wide basis using the `highlight_language` configuration option. The code-block directive can be used to set highlighting on a block-by-block basis. These directives are discussed later." CPython has the `highlight-language` set to python3.

• A doctest block does not need to be in a Literal Block. [2] They are identified automatically. [3] According to the doc, doctest directives are suppressed from presentation:

  Also, you can give inline doctest options, like in doctest:

    >>> datetime.date.now()   # doctest: +SKIP
    datetime.date(2008, 1, 1)

They will be respected when the test is run, but stripped from presentation output.

• Any section (doesn't need the single colon (`:`) can have the `code-block` directive on it. [4] The `code-block` directive can define the language for highlighting.

Note that in the Sphinx doctest [4] documentation, there is a config option trim_doctest_flags, which is True by default. Setting this to False will show all the doctest directives for all the doctests in the documentation.

The nice thing about the doc build recognizing a section as a doctest is that it adds the toggle in the upper right of the block to 'Hide the toggle and output' so that the example can be copy and pasted more easily. Changing it into something other than a doctest (like using a code-block directive, takes away this option. Using trim_doctest_flags=False retains the hide button.

Options:

1. Change global setting to false to show all doctest directives.
2. Change only the blocks where the doctest directives need to be shown, but this makes them lose the 'Hide prompt and output' button.
3. Add a new option under 'code-block' to control the displaying of the doctest directives at the code-block level. This would override the global setting. I think this option could be added locally, but may need to be done upstream in Sphinx.

[1] http://www.sphinx-doc.org/en/stable/usage/restructuredtext/basics.html#literal-blocks
[2] http://www.sphinx-doc.org/en/stable/usage/restructuredtext/basics.html#doctest-blocks
[3] http://www.sphinx-doc.org/en/stable/usage/extensions/doctest.html
[4] http://www.sphinx-doc.org/en/stable/usage/restructuredtext/directives.html#showing-code-examples

[JulienPalard]

I opened an issue on the sphinx-doc repo 1 to check if it would be possible to have an option in doctest blocks to not trim them.

We previously had a hack in Doc/tools/extensions/pyspecific.py to patch sphinx to not trim them for the doctest.rst file. But sphinx deprecated the hack we used :(

[gpshead]

New changeset 2c910c1 by Gregory P. Smith (Julien Palard) in branch 'master':
bpo-36675: Remove obsolete code. (GH-16024)
2c910c1

[miss-islington]

New changeset 94a6847 by Miss Islington (bot) in branch '3.8':
bpo-36675: Remove obsolete code. (GH-16024)
94a6847

[jdlh]

We discovered and fixed this same problem back in 2011-2012 with bpo-12947 .

That was apparently the source of the monkeypatch that was removed as "obselete code" on 2019-09-12. That old issue commentary has some suggestions about other workarounds. This includes adding explanatory text about the fact that doctest directives are missing from the examples which should be showing them. Maybe some of those workarounds would be effective for us now.

[JulienPalard]

New changeset c8a10d2 by Julien Palard in branch 'master':
bpo-36675: Doc: Reveal doctest directives (GH-23620)
c8a10d2

[JulienPalard]

Happy Christmas, everybody involved in this issue! I'm happy to announce this issue is resolved since a few days \o/

[asottile]

should the minimum sphinx version be bumped and/or this reverted:

cpython/Doc/conf.py

Line 48 in f7f0ed5

needs_sphinx = '1.8'

this change breaks, for example, sphinx 1.8.5 while building for ubuntu 20.04

The directive used here requires sphinx>=3.2.0

I notice some other attempts have been made to make the docs more compatible with sphinx 1.x in this release as well so there might be conflicting directions here: b63a620

[JulienPalard]

Due to #24282 this is sadly un-fixed.

Either we find another way to fix this, either we wait 3 releases and we re-commit #23620.

[jdlh]

My goodness, things get complex sometimes.

If we cannot make Sphinx preserve doctest directives and comments, perhaps we should go back to the historical bug discussion to look at workarounds which we considered earlier. For instance, maybe we should modify the text surrounding the examples to explain that doctests directives should appear there, but that our tool chain currently removes them.

At the moment, I see doctest directives in the doctest source code at:
https://github.com/python/cpython/blob/master/Doc/library/doctest.rst#directives
which do not appear in the corresponding HTML output at:
https://docs.python.org/3/library/doctest.html#directives

How about rewording the text before each of those examples? Or maybe finding some way to show those examples as literal text which Sphinx won't modify, even if it is not formatted like Python code examples?

By the way, the discussion of this same problem back in 2011-2012 is at bpo-12947 . At that time, we applied a "monkey patch" to the Sphinx code. I haven't read the discussion closely enough to figure out if such a patch would help now.

[sorcio]

Due to #24282 this is sadly un-fixed.

Either we find another way to fix this, either we wait 3 releases and we re-commit #23620.

Since docs are now built on Sphinx 4.5.0, it might be a good time to re-fix this.

I applied a cherry-picked eb7fcb8 locally and it seems to work fine.

Made this into a new PR #92318

[ezio-melotti]

This is now fixed on 3.10/3.11/main.
Thanks for the PR!