Added doctest encoding command line option #2101

wheerd · 2016-11-29T11:33:29Z

I added a command line option to specify the encoding of doctest (.txt/.rst) files.
This should fix the error described in #540 which I am also having on my windows machine.

…es. Defaults to UTF-8.

…cify encoding for internal testdir._makefile() for the tests.

RonnyPfannschmidt · 2016-11-29T11:38:22Z

testing/test_doctest.py

+    def test_encoding_latin1(self, testdir):
+        """Test support for --doctest-encoding option.
+        """
+        testdir._makefile(".txt", ["""


at first glance i wonder if this shouldn't use unicode marked strings like u"..."

i believe the actual encoding can be a parametrize parameter, and we might want to add an exotic one just to be on the paranoid side

That would break the code for Python 3.0 I guess because iirc it didn't have the unicode prefix anymore.

@wheerd python 3.3 added it back for more pleasant dual codebases

coveralls · 2016-11-29T11:42:35Z

Coverage increased (+0.0008%) to 92.713% when pulling d254c6b on wheerd:doctest-encoding into 36eb5b3 on pytest-dev:features.

coveralls · 2016-11-29T14:02:36Z

Coverage increased (+0.0008%) to 92.713% when pulling 929912d on wheerd:doctest-encoding into 36eb5b3 on pytest-dev:features.

RonnyPfannschmidt · 2016-11-29T15:52:45Z

testing/test_doctest.py

-            'üäö'
+        testdir._makefile(".txt", [u"""
+            >>> len(u'üäö')
+            3


i propose making those a format string adding u in front of it for py2, and leaving it out for py3
that way the output can be tested

In Python 2, they will be formatted as escape sequences: u'\x84\x94\x81'. Probably I can just repr(value) to the files.

nicoddemus · 2016-11-29T20:16:32Z

_pytest/doctest.py

@@ -43,6 +43,10 @@ def pytest_addoption(parser):
        action="store_true", default=False,
        help="ignore doctest ImportErrors",
        dest="doctest_ignore_import_errors")
+    group.addoption("--doctest-encoding",


I think this would be better off being a ini option instead:

If one has to use this option, then usually all doctest.txt files will have the same encoding (if the developers want to keep their sanity that is) so it makes sense to configure that in pytest.ini;

If really desired, one can use the -o option to override ini options in the command-line.

It is usually not a good idea to require users to pass certain command-line options to make the test suite pass, and in which case the users don't pass the correct option the test-suite will fail in mysterious ways;

nicoddemus · 2016-11-29T20:23:32Z

testing/test_doctest.py

@@ -129,6 +129,52 @@ def test_multiple_patterns(self, testdir):
            '*1 passed*',
        ])

+    def test_encoding_ascii(self, testdir):


I think you could use @pytest.mark.parametrize on the tests to avoid repeating yourself. Something like (already incorporating my request to change to an ini option):

import pytest @pytest.mark.parametrize('string, encoding', [ (u'foo', 'ascii'), (u'üäö', 'latin1'), (u'üäö', 'utf-8'), ]) def test_encoding_latin1(testdir, string, encoding): """Test support for doctest_encoding option. """ testdir.makeini(""" [pytest] doctest_encoding={0} """.format(encoding)) testdir._makefile(".txt", u""" >>> len(u'{0}') 3 """.format(string), encoding=encoding) result = testdir.runpytest() result.stdout.fnmatch_lines([ '*1 passed*', ])

EDIT: fixed a comment left by accident

nicoddemus

Other than my comments, please add a CHANGELOG entry under New Features and add yourself to the CHANGELOG. 😁

Parametrized the tests for it.

coveralls · 2016-11-30T11:02:35Z

Coverage decreased (-0.03%) to 92.68% when pulling 1f62e5b on wheerd:doctest-encoding into 36eb5b3 on pytest-dev:features.

nicoddemus · 2016-11-30T11:15:13Z

doc/en/doctest.rst

@@ -11,6 +11,13 @@ can change the pattern by issuing::
 on the command line. Since version ``2.9``, ``--doctest-glob``
 can be given multiple times in the command-line.

+You can specify the encoding that will be used for those doctest files
+using the ``--doctest-encoding`` command line option::


Please update the docs to mention the doctest_encoding ini option instead.

Sorry, forgot about that.

nicoddemus · 2016-11-30T11:16:38Z

testing/test_doctest.py

+            doctest_encoding={0}
+        """.format(encoding))
+        doctest = u"""
+            >>> u"{}"


The use of {} is not supported by py26 and is breaking the CI, please change that to {0}, {1}, etc.

nicoddemus · 2016-11-30T11:17:31Z

I think we're almost there, thanks again for the PR!

nicoddemus · 2016-11-30T13:26:32Z

doc/en/doctest.rst

@@ -11,6 +11,17 @@ can change the pattern by issuing::
 on the command line. Since version ``2.9``, ``--doctest-glob``
 can be given multiple times in the command-line.

+You can specify the encoding that will be used for those doctest files
+using the ``doctest_encoding`` ini option:


One last thing, please add this:

You can specify the encoding that will be used for those doctest files using the ``doctest_encoding`` ini option: .. versionadded:: 3.1 .. code-block:: ini

To convey to the users that this is new and was added in 3.1.

coveralls · 2016-11-30T13:30:03Z

Coverage decreased (-0.03%) to 92.68% when pulling f8fef07 on wheerd:doctest-encoding into 36eb5b3 on pytest-dev:features.

coveralls · 2016-11-30T13:39:47Z

Coverage decreased (-0.03%) to 92.68% when pulling f8fef07 on wheerd:doctest-encoding into 36eb5b3 on pytest-dev:features.

RonnyPfannschmidt · 2016-11-30T13:49:27Z

lovely - this is shaping up, good work

coveralls · 2016-11-30T18:09:54Z

Coverage decreased (-0.03%) to 92.68% when pulling 2edfc80 on wheerd:doctest-encoding into 36eb5b3 on pytest-dev:features.

nicoddemus · 2016-11-30T19:43:32Z

The py27-expect failure in AppVeyor is unrelated so this is good to merge.

Thanks again @wheerd!

Manuel Krebber added 2 commits November 29, 2016 12:29

Added a console option to specify the encoding to use for doctest fil…

ed97751

…es. Defaults to UTF-8.

Added some tests for --docstring-encoding option. Added option to spe…

d254c6b

…cify encoding for internal testdir._makefile() for the tests.

RonnyPfannschmidt reviewed Nov 29, 2016

View reviewed changes

Changed the tests to pass on python 2 as well.

929912d

RonnyPfannschmidt reviewed Nov 29, 2016

View reviewed changes

nicoddemus reviewed Nov 29, 2016

View reviewed changes

nicoddemus requested changes Nov 29, 2016

View reviewed changes

Manuel Krebber added 2 commits November 30, 2016 11:43

Changed the doctest_encoding option to an ini option.

c043bbb

Parametrized the tests for it.

Added CHANGELOG entry and myself to AUTHORS.

1f62e5b

nicoddemus reviewed Nov 30, 2016

View reviewed changes

Manuel Krebber added 2 commits November 30, 2016 14:17

Fixed the documentation for doctest_encoding.

b7fb9fa

Fixed the tests for python 2.6

f8fef07

nicoddemus reviewed Nov 30, 2016

View reviewed changes

Added "versionadded" for doctest_encoding ini option to docs.

2edfc80

nicoddemus approved these changes Nov 30, 2016

View reviewed changes

nicoddemus merged commit 669332b into pytest-dev:features Nov 30, 2016

decentral1se mentioned this pull request Dec 2, 2016

Encoding failure in doctest error handling on py3/windows #540

Closed

This was referenced May 31, 2017

Update pytest to 3.1.1 abkfenris/sugarloaf#88

Closed

Update pytest to 3.1.1 aio-libs-abandoned/aioes#154

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added doctest encoding command line option #2101

Added doctest encoding command line option #2101

wheerd commented Nov 29, 2016

RonnyPfannschmidt Nov 29, 2016

wheerd Nov 29, 2016

RonnyPfannschmidt Nov 29, 2016

coveralls commented Nov 29, 2016

coveralls commented Nov 29, 2016

RonnyPfannschmidt Nov 29, 2016

wheerd Nov 30, 2016

nicoddemus Nov 29, 2016 •

edited

Loading

nicoddemus Nov 29, 2016 •

edited

Loading

nicoddemus left a comment

coveralls commented Nov 30, 2016

nicoddemus Nov 30, 2016

wheerd Nov 30, 2016

nicoddemus Nov 30, 2016

nicoddemus commented Nov 30, 2016

nicoddemus Nov 30, 2016

coveralls commented Nov 30, 2016

coveralls commented Nov 30, 2016

RonnyPfannschmidt commented Nov 30, 2016

coveralls commented Nov 30, 2016

nicoddemus commented Nov 30, 2016

Added doctest encoding command line option #2101

Added doctest encoding command line option #2101

Conversation

wheerd commented Nov 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Nov 29, 2016

coveralls commented Nov 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicoddemus Nov 29, 2016 • edited Loading

Choose a reason for hiding this comment

nicoddemus Nov 29, 2016 • edited Loading

Choose a reason for hiding this comment

nicoddemus left a comment

Choose a reason for hiding this comment

coveralls commented Nov 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicoddemus commented Nov 30, 2016

Choose a reason for hiding this comment

coveralls commented Nov 30, 2016

coveralls commented Nov 30, 2016

RonnyPfannschmidt commented Nov 30, 2016

coveralls commented Nov 30, 2016

nicoddemus commented Nov 30, 2016

nicoddemus Nov 29, 2016 •

edited

Loading

nicoddemus Nov 29, 2016 •

edited

Loading