Add pytest functionality

[asteppke]

Combining Jupyter notebooks with test driven development feels great, treon is really helpful for CI pipelines. Is it possible besides doctest and unittest to also include the pytest framework?

It seems that pytest does not have a drop-in function such as the unittest.main() which executes the current module but requires a filename. Still this would be a nice addition and removes a lot of boilerplate one needs for the unittest framework.

[amit1rrr]

I debated for a while between supporting unittest vs pytest at the beginning & mainly chose unittest because,

• It's easier for non-developers to get started with (a lot of Jupyter users are non developers)
• Slightly easier to implement since it can be triggered as one of the code cell itself

Anyhow, thinking more about it, there's no reason why treon shouldn't support pytest as well. So I'm going to work on it sometime in the next few weeks (unless someone else picks it up before that).

Here's the rough implementation outline,

1. Convert all the notebooks under test into .py temporary files
2. Run py.test filename for each of those
3. Capture+Modify the stdout output to reflect correct notebook name instead of .py name
4. Include the result (success/failure) in overall result summary

Let me know if there's a better way.

[asteppke]

I agree that unittest should definitely be supported for the reasons you gave and also because it is part of the standard library.

Pytest seems to develop into somewhat of a successor to unittest, nose and other python test libraries. Additionally the syntax is easier to learn, e.g.

def test_example():
    assert [1, 2, 3] == [1, 2, 3]

is just much simpler than what unittest offers. To implement it I think think your route sounds good. There are also the ipytest and nbval projects which slightly overlap in functionality. I don't know if including either of these would make the integration of pytest into treon easier though.

[amit1rrr]

There are also the ipytest and nbval projects which slightly overlap in functionality. I don't know if including either of these would make the integration of pytest into treon easier though.

I had looked at ipytest earlier and it's mainly geared for running pytest as a cell magic from the Jupyter environment itself. Nothing wrong with it just wouldn't integrate well with treon's command line approach. Probably best to directly use nbconvert (for ipynb to py) and py.test so we can access the test result (success/failure) and integrate it into treon result summary etc.

Aside:
nbval is pretty nice actually but it doesn't run pytests, instead it's a pytest plugin that validates the notebook (i.e. see if cell output after execution matches the cell output already present in the file and such). Very handy. I have been thinking of something along the same lines, parameterised tests with papermill. Anyway, that's worthy of a separate issue.

[stas00]

I debated for a while between supporting unittest vs pytest at the beginning & mainly chose unittest because,
[...]
Let me know if there's a better way.

There is. You parse the cells - and then run them.

There is an old way that uses runipy - you can see here: https://github.com/stas00/pytest-ipynb (it's a fork of no longer maintained pytest module, that I fixed up just enough to satisfy my needs). You can see it in action by running make test in https://github.com/stas00/ipyexperiments after you installed my fork of pytest-ipynb.

And there is the modern approach, https://www.blog.pythonlibrary.org/2018/10/16/testing-jupyter-notebooks/

May be you could take over pytest-ipynb and integrate treon into it (or the other way around), I am not attached.

I've been trying to summarize the existing approaches here: https://github.com/stas00/pytest-ipynb/blob/master/ipynb-testing-ways.md - please send me a PM to add treon.

p.s. Kudos on all the great work with reviewnb and now treon, Amit!

[westurner]

I've been trying to summarize the existing approaches here: https://github.com/stas00/pytest-ipynb/blob/master/ipynb-testing-ways.md - please send me a PM to add treon.

Just learned of treon from https://github.com/markusschanta/awesome-jupyter/blob/master/README.md#testing ; which currently lists:

Testing

• ipytest - Test runner for running unit tests from within a notebook.
• nbval - Py.test plugin for validating Jupyter notebooks.
• sphinxcontrib-jupyter - Sphinx Extension for Generating Jupyter Notebooks.
• nosebook - Nose plugin for finding and running IPython notebooks as nose tests.
• treebeard - GitHub Action for testing/scheduling Jupyter notebooks.
• treon - Easy-to-use test framework for Jupyter Notebooks