Create dev_requirements.txt from extras_require section of setup.cfg file using pip-compile

[ckaldemeyer]

What's the problem this feature will solve?

NOTE: This is pasted and adapted from my question on stackoverflow because I am not sure if this feature already exists: https://stackoverflow.com/questions/68712892/how-to-create-dev-requirements-txt-from-extras-require-section-of-setup-cfg

I use pip-tools to manage my dependencies and environments which perfectly generates a requirements.txt file for my package that consists of a setup.py that looks like this:

#! /usr/bin/env python
import os
from setuptools import setup


if "CI_COMMIT_TAG" in os.environ:
    VERSION = os.environ["CI_COMMIT_TAG"]
else:
    VERSION = "0.0.0"

setup(version=VERSION)

and a setup.cfg like this:

...
[options]
python_requires = >=3.7
zip_safe = False
packages = find:
include_package_data = True
install_requires =
    PyYAML
    Cython
    numpy==1.17.5
    pandas==0.25.3
    ...
package_dir=
    foo=bar

[options.extras_require]
testing =
    tox>=3.1.2
    pytest>=3.4.0
coverage =
    coverage
    pytest-cov>=2.5.1
other =
    anybadge
...

Running $ pip-compile --index-url https://foo@[email protected]/api/v4/projects/236/packages/pypi/simple --no-header --allow-unsafe yields my package requirements:

...
async-timeout==3.0.1
    # via aiohttp
attrs==21.2.0
    # via aiohttp
bcrypt==3.2.0
...

But this only includes all the packages from the install_requires section of my setup.cfg file and not the requirements from extras_require. It should work with a dev_requirements.in file as described here but I would rather only use one configuration file.

Describe the solution you'd like

Create a separate dev_requirements.txt from this extras_require section of my setup.cfg file using pip-compile without having to create a dev_requirements.in file.

Alternative Solutions

I am also open for other solutions to achieve my goal.

Additional context

Thanks in advance for your help!

[ckaldemeyer]

After digging a bit more, I found my answer in another issue: #625 (comment) Thanks for maintaining this cool package!

[webknjaz]

After digging a bit more, I found my answer in another issue: #625 (comment) Thanks for maintaining this cool package!

I just want to add that personally I prefer a separate file because it usually describes some env, not the package itself. So it doesn't really belong in the package metadata, nor is useful to the final consumers... But yeah, this question is considered controversial in the packaging community and the opinions are polarized.

[ckaldemeyer]

Yes, I see your point. Nevertheless: is there also an option to pass multiple envs/sections using --extra?

Let's say I have:

[options.extras_require]
dev =
    black==20.8b1
other =
    coverage
    mypy
    anybadge

How can I add both sections?

[atugushev]

pip-compile --extra=dev --extra=other should work

[ckaldemeyer]

You are my heroes of the day. Thanks a lot for your help and this wonderful package!

[webknjaz]

I think having multiple options should be enough: --extra dev --extra other. Here are some test cases that check this scenario in the original PR: https://github.com/jazzband/pip-tools/pull/1363/files#diff-99b06d953a9978ee564df2e67547b8a76f3b1ab116ab7c89a9005b9239f2a904R1803-R1807.