Add a test to check that Evaluator evaluations match transformers examples

[fxmarty]

Basically just to check that Trainer and Evaluator do return the same evaluations. This was useful for me working on Optimum to debug slight differences between evaluations with pipelines vs. Trainer (https://github.com/fxmarty/optimum/tree/tests-benchmark/tests/benchmark/onnxruntime), where depending on the task you need to be careful with the kwargs passed to the pipeline to match the output.

Let me know if you think such tests for each tasks are useful or not. If not, I will just add TokenClassificationEvaluator, QuestionAnsweringEvaluator and ImageProcessingEvaluator without the tests.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lvwerra]

Hi @fxmarty thanks for working on this! That does indeed look useful, kind of a compatibility test between Trainer and evaluator. I am curious to hear what @LysandreJik thinks about this and how this is handled in other libraries (e.g. Trainer vs. accelerate)? The evaluate library is generally not tightly coupled to transformers so I am not sure how much we should test against it.

[lvwerra]

I have been thinking about this some more: I think although this is a great tool for debugging the evaluator at the point the PR is merged we know the expected value and can simply test against it, no? There is no reason to run the Trainer to get the value to compare against. What do you think?

[fxmarty]

Agreed it's probably a bit overkill, although checking against PyTorch examples makes sure that in the future we continue to match Trainer / PyTorch scripts examples behavior.

If you would prefer to have a fixed value in the test, let me know and I will edit accordingly!

[lvwerra]

After discussing with @LysandreJik I think we can add it. It will be an interesting signal if this fails. Thanks a lot for adding this! I left a few minor comments, let me know if you have any questions.

I don't think we need subfolders at this point and you can just create a new file in tests (e.g. test_trainer_evaluator_parity.py).

[fxmarty]

Thank you for the review @lvwerra , I did changes according to your comments! I think CircleCI is not running here so if you can trigger it it's nice.

Once this one is merged and we have agreed a solution in #167 I will edit the test for token-classification accordingly.

[lvwerra]

LGTM! 🚀