[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to `site-packages` instead of `dist-package` when using `pip install --no-cache-dir` #3457

squahtx · 2022-07-19T14:22:13Z

setuptools version

setuptools>=48.0.0

Python version

Python 3.7

OS

Debian 10

Additional environment information

Occurs in the debian:buster Docker container, where packages go into .../dist-packages:

$ sudo docker run -it debian:buster
...
root@721aec8b79f0:/# apt update && apt install python3 python3-pip -y
...
root@721aec8b79f0:/# find / | grep site-packages
root@721aec8b79f0:/# find / | grep dist-packages
/usr/lib/python3/dist-packages
/usr/local/lib/python3.7/dist-packages

Description

pip install --no-cache-dir msgpack==1.0.3 with setuptools>=48.0.0 installs msgpack to /usr/lib/python3.7/site-packages/, where Python can't find it.

When the --no-cache-dir option is removed or setuptools==47.3.2 is used, msgpack gets installed to /usr/local/lib/python3.7/dist-packages/ as expected.

Expected behavior

pip install --no-cache-dir msgpack==1.0.3 with setuptools>=48.0.0 installs msgpack to/usr/local/lib/python3.7/dist-packages/.

How to Reproduce

Start a debian:buster container, install Python and note that packages go into dist-packages.

$ sudo docker run -it debian:buster
...
root@721aec8b79f0:/# apt update && apt install python3 python3-pip -y
...
root@721aec8b79f0:/# find / | grep site-packages
root@721aec8b79f0:/# find / | grep dist-packages
/usr/lib/python3/dist-packages
/usr/local/lib/python3.7/dist-packages

Install setuptools==48.0.0:

root@721aec8b79f0:/# python3 -m pip install setuptools==48.0.0
Collecting setuptools==48.0.0
  Downloading https://files.pythonhosted.org/packages/52/b4/2b8decca516c7139f2d262b6a0c5dcd6388d0a2d2c760c6bbc830e49347e/setuptools-48.0.0-py3-none-any.whl (786kB)
    100% |████████████████████████████████| 788kB 1.4MB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 40.8.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed setuptools-48.0.0
root@721aec8b79f0:/# python3 -m pip list | grep setuptools
setuptools    48.0.0

Install msgpack==1.0.3. Don't use >=1.0.4, since they start specifying build dependencies.

root@721aec8b79f0:/# python3 -m pip install --no-cache-dir msgpack==1.0.3
Collecting msgpack==1.0.3
  Downloading https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz (123kB)
    100% |████████████████████████████████| 133kB 5.1MB/s
Installing collected packages: msgpack
  Running setup.py install for msgpack ... done
Successfully installed msgpack

pip list does not pick up msgpack

root@721aec8b79f0:/# python3 -m pip list | grep msgpack

msgpack has gone into /usr/lib/python3.7/site-packages/msgpack:

root@721aec8b79f0:/# find / | grep msgpack
/usr/lib/python3.7/site-packages/msgpack

Try again with setuptools==47.3.2, which works as expected:

root@721aec8b79f0:/# python3 -m pip install setuptools==47.3.2
Collecting setuptools==47.3.2
  Downloading https://files.pythonhosted.org/packages/90/0f/1e97de8eebf9d3c61b92202e69dbd40a8c0022ae25db5f64e0f28d501dd8/setuptools-47.3.2-py3-none-any.whl (582kB)
    100% |████████████████████████████████| 583kB 1.9MB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 48.0.0
    Uninstalling setuptools-48.0.0:
      Successfully uninstalled setuptools-48.0.0
Successfully installed setuptools-47.3.2
root@721aec8b79f0:/# python3 -m pip install --no-cache-dir msgpack==1.0.3
Collecting msgpack==1.0.3
  Downloading https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz (123kB)
    100% |████████████████████████████████| 133kB 5.1MB/s
Installing collected packages: msgpack
  Running setup.py install for msgpack ... done
Successfully installed msgpack-1.0.3
root@721aec8b79f0:/# python3 -m pip list | grep msgpack
msgpack       1.0.3

Output

verbose output, with setuptools==48.0.0:

root@721aec8b79f0:/# python3 -m pip install --no-cache-dir msgpack==1.0.3 -v
Created temporary directory: /tmp/pip-ephem-wheel-cache-_kdxzi0r
Created temporary directory: /tmp/pip-req-tracker-axqlt4bp
Created requirements tracker '/tmp/pip-req-tracker-axqlt4bp'
Created temporary directory: /tmp/pip-install-4jxlv1iz
Collecting msgpack==1.0.3
  1 location(s) to search for versions of msgpack:
  * https://pypi.org/simple/msgpack/
  Getting page https://pypi.org/simple/msgpack/
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/msgpack/ HTTP/1.1" 200 37762
  Analyzing links from page https://pypi.org/simple/msgpack/
...
  Using version 1.0.3 (newest of versions: 1.0.3)
  Created temporary directory: /tmp/pip-unpack-zyvft96v
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz HTTP/1.1" 200 123830
  Downloading https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz (123kB)
  Downloading from URL https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz#sha256=51fdc7fb93615286428ee7758cecc2f374d5ff363bdd884c7ea622a7a327a81e (from https://pypi.org/simple/msgpack/)
    100% |████████████████████████████████| 133kB 5.7MB/s
  Added msgpack==1.0.3 from https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz#sha256=51fdc7fb93615286428ee7758cecc2f374d5ff363bdd884c7ea622a7a327a81e to build tracker '/tmp/pip-req-tracker-axqlt4bp'
  Running setup.py (path:/tmp/pip-install-4jxlv1iz/msgpack/setup.py) egg_info for package msgpack
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/msgpack.egg-info
    writing pip-egg-info/msgpack.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/msgpack.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/msgpack.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/msgpack.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/msgpack.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/msgpack.egg-info/SOURCES.txt'
  Source in /tmp/pip-install-4jxlv1iz/msgpack has version 1.0.3, which satisfies requirement msgpack==1.0.3 from https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz#sha256=51fdc7fb93615286428ee7758cecc2f374d5ff363bdd884c7ea622a7a327a81e
  Removed msgpack==1.0.3 from https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz#sha256=51fdc7fb93615286428ee7758cecc2f374d5ff363bdd884c7ea622a7a327a81e from build tracker '/tmp/pip-req-tracker-axqlt4bp'
Installing collected packages: msgpack                                                                                                                                                                                                         Created temporary directory: /tmp/pip-record-fxx2yv1m                                                                                                                                                                                        Running setup.py install for msgpack ...     Running command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-4jxlv1iz/msgpack/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-fxx2yv1m/install-record.txt --single-version-externally-managed --compile                                                                          running install                                                                                                                                                                                                                              running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/msgpack
    copying msgpack/fallback.py -> build/lib.linux-x86_64-3.7/msgpack
    copying msgpack/_version.py -> build/lib.linux-x86_64-3.7/msgpack
    copying msgpack/__init__.py -> build/lib.linux-x86_64-3.7/msgpack
    copying msgpack/ext.py -> build/lib.linux-x86_64-3.7/msgpack
    copying msgpack/exceptions.py -> build/lib.linux-x86_64-3.7/msgpack
    running build_ext
    building 'msgpack._cmsgpack' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/msgpack
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -flto -fuse-linker-plugin -ffat-lto-objects -fPIC -D__LITTLE_ENDIAN__=1 -I. -I/usr/include/python3.7m -c msgpack/_cmsgpack.cpp -o build/temp.linux-x86_64-3.7/msgpack/_cmsgpack.o
    msgpack/_cmsgpack.cpp: In function ‘__pyx_f_7msgpack_9_cmsgpack_6Packer__pack(__pyx_obj_7msgpack_9_cmsgpack_Packer*, _object*, __pyx_opt_args_7msgpack_9_cmsgpack_6Packer__pack*)’:
    msgpack/_cmsgpack.cpp:4534:10: warning: ‘__pyx_r’ may be used uninitialized in this function [-Wmaybe-uninitialized]
       return __pyx_r;
              ^~~~~~~
    x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-3.7/msgpack/_cmsgpack.o -L/usr/lib -o build/lib.linux-x86_64-3.7/msgpack/_cmsgpack.cpython-37m-x86_64-linux-gnu.so
    running install_lib
    copying build/lib.linux-x86_64-3.7/msgpack/_cmsgpack.cpython-37m-x86_64-linux-gnu.so -> /usr/lib/python3.7/site-packages/msgpack
    running install_egg_info
    running egg_info
    creating msgpack.egg-info
    writing msgpack.egg-info/PKG-INFO
    writing dependency_links to msgpack.egg-info/dependency_links.txt
    writing top-level names to msgpack.egg-info/top_level.txt
    writing manifest file 'msgpack.egg-info/SOURCES.txt'
    reading manifest file 'msgpack.egg-info/SOURCES.txt'
    writing manifest file 'msgpack.egg-info/SOURCES.txt'
    removing '/usr/lib/python3.7/site-packages/msgpack-1.0.3-py3.7.egg-info' (and everything under it)
    Copying msgpack.egg-info to /usr/lib/python3.7/site-packages/msgpack-1.0.3-py3.7.egg-info
    running install_scripts
    writing list of installed files to '/tmp/pip-record-fxx2yv1m/install-record.txt'
done
  Removing source in /tmp/pip-install-4jxlv1iz/msgpack
Successfully installed msgpack
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-axqlt4bp'

The text was updated successfully, but these errors were encountered:

abravalheri · 2022-07-20T10:51:36Z

Hi @squahtx, thank you very much for opening the issue.
There are several aspects to unpack here... I tried to cover them in my comments below:

Please note that setuptools release process addresses problems by adding bug fixes to the latest version (in a linear style). So I recommend also testing this against the latest version of setuptools.
In the latest versions of setuptools and pip, setuptools is not responsible for installing the package to its final location. Setuptools will only build the wheel, and the installer (pip) will place the files in the correct locations.
Starting in v59.2.0, setuptools expect Debian-based systems to implement the _distutils_system_mod module to customize file locations. It seems that the buster version does not fulfil this requirement.
When installing python3-pip in buster, you get pip==18.1. This version of pip does not seem equipped to handle PEP 517 (implemented in pip 19.0). This means that it may delegate to Setuptools' install command when --no-cache-dir is given, which in turn depends on _distutils_system_mod (which is not implemented in Python 3.7 in buster).

With all of this in mind, There are a few things you can do to workaround this limitation in Debian systems.

Update pip before installing msgpack with python3 -m pip install -U pip, or
Set the environment variable SETUPTOOLS_USE_DISTUTILS=stdlib.

It might also be worthwhile to contact the maintainers of the Python packages in Debian and ask if it is possible to add _distutils_system_mod to Python 3.7 and other that still don't have it.

squahtx · 2022-07-20T13:52:56Z

Please note that setuptools release process addresses problems by adding bug fixes to the latest version (in a linear style). So I recommend also testing this against the latest version of setuptools.

Testing on the latest setuptools version produces the same unwanted result:

Output

root@20524aae547d:/# python3 -m pip install setuptools --upgrade
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/a4/53/bfc6409447ca024558b8b19d055de94c813c3e32c0296c48a0873a161cf5/setuptools-63.2.0-py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 1.1MB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 40.8.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed setuptools-63.2.0
root@20524aae547d:/# python3 -m pip list
Package       Version
------------- -------
asn1crypto    0.24.0
cryptography  2.6.1
entrypoints   0.3
keyring       17.1.1
keyrings.alt  3.1.1
pip           18.1
pycrypto      2.6.1
PyGObject     3.30.4
pyxdg         0.25
SecretStorage 2.3.1
setuptools    63.2.0
six           1.12.0
wheel         0.32.3
root@20524aae547d:/# python3 -m pip install --no-cache-dir msgpack==1.0.3
Collecting msgpack==1.0.3
  Downloading https://files.pythonhosted.org/packages/61/3c/2206f39880d38ca7ad8ac1b28d2d5ca81632d163b2d68ef90e46409ca057/msgpack-1.0.3.tar.gz (123kB)
    100% |████████████████████████████████| 133kB 5.2MB/s
Installing collected packages: msgpack
  Running setup.py install for msgpack ... done
Successfully installed msgpack
root@20524aae547d:/# python3 -m pip list | grep msgpack
root@20524aae547d:/#

Update pip before installing msgpack with python3 -m pip install -U pip

That fixes things, thank you!

Output

root@35dc4f39e7f4:/# python3 -m pip install pip setuptools --upgrade
Collecting pip
  Downloading https://files.pythonhosted.org/packages/96/2f/caec18213f6a67852f6997fb0673ae08d2e93d1b81573edb93ba4ef06970/pip-22.1.2-py3-none-any.whl (2.1MB)
    100% |████████████████████████████████| 2.2MB 693kB/s
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/a4/53/bfc6409447ca024558b8b19d055de94c813c3e32c0296c48a0873a161cf5/setuptools-63.2.0-py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 1.1MB/s
Installing collected packages: pip, setuptools
  Found existing installation: pip 18.1
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pip'. No files were found to uninstall.
  Found existing installation: setuptools 40.8.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed pip-22.1.2 setuptools-63.2.0
root@35dc4f39e7f4:/# python3 -m pip install --no-cache-dir msgpack==1.0.3
Collecting msgpack==1.0.3
  Downloading msgpack-1.0.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (299 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 299.4/299.4 kB 5.5 MB/s eta 0:00:00
Installing collected packages: msgpack
Successfully installed msgpack-1.0.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@35dc4f39e7f4:/# python3 -m pip list | grep msgpack
msgpack       1.0.3

squahtx added bug Needs Triage Issues that need to be evaluated for severity and status. labels Jul 19, 2022

squahtx mentioned this issue Jul 19, 2022

Unexplained sytest poetry build failure weirdness matrix-org/sytest#1269

Closed

squahtx changed the title ~~[BUG] setuptools>=48.0.0 installing to site-packages instead of dist-packages~~ [BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to site-packages instead of dist-packages Jul 19, 2022

abravalheri added the Waiting User Feedback label Jul 20, 2022

squahtx changed the title ~~[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to site-packages instead of dist-packages~~ [BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to site-packages instead of dist-packages when using pip install --no-cache-dir Jul 20, 2022

squahtx closed this as completed Jul 20, 2022

squahtx mentioned this issue Jul 20, 2022

Upgrade pip instead of using --cache-dir workaround in Synapse image matrix-org/sytest#1274

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to `site-packages` instead of `dist-package` when using `pip install --no-cache-dir` #3457

[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to `site-packages` instead of `dist-package` when using `pip install --no-cache-dir` #3457

squahtx commented Jul 19, 2022 •

edited

Loading

abravalheri commented Jul 20, 2022 •

edited

Loading

squahtx commented Jul 20, 2022

[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to site-packages instead of dist-package when using pip install --no-cache-dir #3457

[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to site-packages instead of dist-package when using pip install --no-cache-dir #3457

Comments

squahtx commented Jul 19, 2022 • edited Loading

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

abravalheri commented Jul 20, 2022 • edited Loading

squahtx commented Jul 20, 2022

[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to `site-packages` instead of `dist-package` when using `pip install --no-cache-dir` #3457

[BUG] setuptools>=48.0.0 installs msgpack==1.0.3 to `site-packages` instead of `dist-package` when using `pip install --no-cache-dir` #3457

squahtx commented Jul 19, 2022 •

edited

Loading

abravalheri commented Jul 20, 2022 •

edited

Loading