scandir doesn't gracefully handle non-available compiler

[AvdN]

The following Dockerfile ( getpip.py in directory pip coming from https://pip.pypa.io/en/stable/installing/ ):

FROM ubuntu:16.04

MAINTAINER Anthon van der Neut <[email protected]>

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get install -y \
  python \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

# this gets you the latest pip
COPY pip/get-pip.py /tmp/get-pip.py
RUN python /tmp/get-pip.py

ADD https://pypi.python.org/packages/95/40/ddbcd295ee58d5c1126645890bcf87853e407
5547308884e4f8ada27f195/scandir-1.4.tar.gz scandir.tar.gz

RUN pip install scandir.tar.gz

gives a compile error that breaks any packages depending on scandir

Step 7 : ADD https://pypi.python.org/packages/95/40/ddbcd295ee58d5c1126645890bcf87853e4075547308884e4f8ada27f195/scandir-1.4.tar.gz scandir.tar.gz
Downloading 28.78 kB/28.78 kB

 ---> 60f05d4cc682
Removing intermediate container 06cdebd2a676
Step 8 : RUN pip install scandir.tar.gz
 ---> Running in f3d0f0161a0d
Processing /scandir.tar.gz
Installing collected packages: scandir
  Running setup.py install for scandir: started
    Running setup.py install for scandir: finished with status 'error'
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-44UuK4-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-rCaaMd-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    copying scandir.py -> build/lib.linux-x86_64-2.7
    running build_ext
    building '_scandir' extension
    creating build/temp.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-44UuK4-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-rCaaMd-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-44UuK4-build/
The command '/bin/sh -c pip install scandir.tar.gz' returned a non-zero code: 1

AFAIK code using scandir will still work with the default os.walk() so transparent fall-back seems to be in order in case compilation cannot work.
The alternative would be to provide version specific whl files (e.g. using manylinux) with some noop wheels for 3.5 and higher.

[benhoyt]

How do other packages handle this? I'm not particularly interested (and don't have the time!) to learn how to create binary builds and wheels for different platforms. What's the actual cause of the gcc problem here?

[AvdN]

I don't know about other packages, but what I do for ruamel.yaml is try and compile a small program (which also tests if the library libyaml is installed), and if that doesn't work ext_modules in setup.py is set to None.

It don't try to do this test-compile if '--version' is in sys.argv, or '--restructuredtext' or 'sdist' as those indicate invocations by pip that are done during my build process that don't involve actual installation (you probably only have to compile when pip calls setup.py with bdist in sys.argv).
The actual compilation is done by initialising and using the distutils/setuptools compiler, so I'm using what will be used later on for the actual compile.

Testing what happens when the compiler is not there can easily be faked by temporarily setting the environment variables CC and CXX to non point to non-existing scripts.

[benhoyt]

I think I'm going to close this -- I think it should fail hard on a missing compiler, as the main purpose of scandir is to speed things up. You lose a lot of that speed benefit if it falls back to the pure Python implementation, which is really not "scandir proper".

[lorengordon]

I just ran into this trying to install pipenv on python 2.7 (CentOS 7). It has a dependency on pathlib2, which pulls in scandir.

I don't have a good answer for creating cross-platform wheels, though. I've been lucky enough to be able to rely on universal wheels for my own python packages.

[lorengordon]

Fwiw, pipenv just removed the pathlib2 dep so this is no longer a problem there.

[lorengordon]

@AvdN #106 ought to handle the absence of a compiler more gracefully...