Primer acceptance tests like in black or mypy

[Pierre-Sassoulas]

Created this ticket following a discord and #4390 discussion with @cdce8p

[Pierre-Sassoulas]

Unfortunately, it's easier said than done. Mypy has mypy_primer, black has black_primer. The difference is that both are focused on maintaining stability whereas we constantly add new things as well. I see it with Home Assistant where I do something similar just manually. It's a lot of effort just keeping the repos in sync, comparing new error messages, checking if they are intended or false-positives, updating the existing code and pushing the changes back to Home Assistant.

What I'm trying to say: I would love it if we could automate it, just not sure it's actually possible / useful for our development cycle.

Originally posted by @cdce8p in pylint-dev/astroid#1211 (comment)

[Pierre-Sassoulas]

The difference is that both are focused on maintaining stability whereas we constantly add new things as well.

Ha, very true. But we can check that pylint is "not crashing". And I think we could check for the presence of error. There's a very slim possibility that there is a real error in a release of those very big libraries but I would bet on a pylint false positive or regression every time. And in particular if suddenly there's a ton of import-error or no-module-error because namespace packages are broken we would detect it.

Originally posted by @Pierre-Sassoulas in pylint-dev/astroid#1211 (comment)

[cdce8p]

This issue might be something we should prioritize. Both #5162 and #5170 could have been prevented if detected earlier.

[Pierre-Sassoulas]

I was going to say exactly that. Let's do it.

[Pierre-Sassoulas]

I'm already finding crash which is encouraging (#5172). However I think we'll need to install the environment of the project for this to work pylint need to be able to import dependencies. Also we can't easily add project because pylint . do not work without an __init__.py at the root level.Maybe we could install with pip and lint the dependencies installed in the tests ? What pip install is a package and is supposed to work, so we would not have this issue. I'm going to create a draft so we can discuss.

[cdce8p]

I'm already finding crash which is encouraging (#5172). However I think we'll need to install the environment of the project for this to work pylint need to be able to import dependencies. Also we can't easily add project because pylint . do not work without an __init__.py at the root level.

I haven't looked at it closely yet, but from what I think, you're right: We'll need to install all import dependencies for each project. That's different to what black_primer does. Maybe it's an idea to look how mypy_primer does it instead? They should also need to install dependencies.

Maybe we could install with pip and lint the dependencies installed in the tests ? What pip install is a package and is supposed to work, so we would not have this issue.

Not sure about that. git clone --depth 1 is probably better.

I'm going to create a draft so we can discuss.

Thanks for doing this 🚀 I'm looking forward to it!

[Pierre-Sassoulas]

Seeing mypy_primer, it's very evolved and it makes me want to fork it and adapt for pylint. There's a lot of project with a pylint configuration on github.

[cdce8p]

If adapting mypy_primer works, I think that would be a great idea. Managing that in a separate repo / fork could also be beneficial.