Audit Python and package versions

[jason-upchurch]

Summary
As a developer, I would like to know:

• if there are backwards incompatibilities between package versions in use compared to most-recently available package versions
• the dependency tree for such packages (how do these packages affect other packages we use?)

Why this matters
Security compliance effort:

• implementing required package upgrades may be simpler if packages in use are mostly current relative to those available
• potentially avoid needing to upgrade through several major versions of a package to get to necessary version, where each version may contain backwards incompatibility
• This concern applies also to the dependent package(s) which may need to be upgraded along with the target package (the dependent package(s) may need to go through several major versions)

Potential benefits

• possibly stay ahead of vulnerabilities in some cases, e.g., the very old package update case
• lower instantaneous effort for security compliance (but will benefit from disciplined ongoing/routine maintenance to avoid large jumps in package versions)
• An audit may allow for explicit decisions on which packages should be upgraded and which can safely be ignored with a plan developed accordingly

Files to audit in fec-eregs

• requirements.txt
• requirements_dev.txt
• requirements_test.txt
• runtime.txt (Python version)

[jason-upchurch]

package	location	current	newest	has_bkwrd_incompatible
python	circleci container/ runtime.txt	3.6.3	3.7.4	true
Django	requirements.txt	1.11.23

TODO:
| amqp==2.5.1 | anyjson==0.3.3
| appnope==0.1.0 | asn1crypto==0.24.0
| attrs==19.1.0 | backcall==0.1.0
| billiard==3.5.0.5 | boto3==1.5.13
| botocore==1.8.50 | cached-property==1.3.1
| celery==4.1.0 | certifi==2019.6.16
| cfenv==0.5.3 | cffi==1.12.3
| chardet==3.0.4 | Click==7.0
| coloredlogs==10.0 | coverage==4.5.4
| coveralls==1.8.2 | cryptography==2.7
| decorator==4.4.0 | dj-database-url==0.4.2
| django-click==2.1.0 | django-debug-toolbar==1.9.1
| django-haystack==2.4.1 | django-mptt==0.8.7
| django-nose==1.4.6 | django-overextends==0.4.3
| django-rq==2.1.0 | djangorestframework==3.10.2
| docopt==0.6.2 | docutils==0.15.2
| elasticsearch==1.9.0 | entrypoints==0.3
| enum34==1.1.6 | -e git+https://github.com/fecgov/fec-eregs.git@fc6c354dfa3704511dbcc1ca0e72e27a05a3d53c#egg=fec_regparser&subdirectory=er| egs_extensions | flake8==3.7.8
| furl==2.0.0 | futures==3.1.1
| gevent==1.2.2 | gitdb2==2.0.5
| GitPython==2.1.8 | greenlet==0.4.15
| gunicorn==19.7.1 | httpretty==0.9.6
| humanfriendly==4.18 | idna==2.8
| importlib-metadata==0.19 | inflection==0.3.1
| invoke==0.22.0 | ipdb==0.10.3
| ipython==6.5.0 | ipython-genutils==0.2.0
| jedi==0.15.1 | jmespath==0.9.4
| json-delta==2.0 | jsonschema==2.5.1
| kombu==4.6.4 | lxml==4.4.1
| marshmallow==3.0.1 | mccabe==0.6.1
| mock==3.0.5 | more-itertools==7.2.0
| networkx==2.3 | newrelic==2.100.0.84
| nose==1.3.7 | nose-exclude==0.5.0
| orderedmultidict==1.0.1 | parso==0.5.1
| pbr==5.4.2 | pep8==1.5.7
| pexpect==4.7.0 | pickleshare==0.7.5
| pipdeptree==0.13.2 | prompt-toolkit==1.0.16
| psycopg2==2.7.3.2 | ptyprocess==0.6.0
| pycodestyle==2.5.0 | pycparser==2.19
| pyelasticsearch==1.4 | pyflakes==2.1.1
| Pygments==2.4.2 | pyOpenSSL==19.0.0
| pyparsing==2.4.2 | python-constraint==1.4.0
| python-dateutil==2.8.0 | pytz==2019.2
| redis==3.3.8 | regcore==4.2.0
| regparser==4.3.1 | regulations==8.4.2
| requests==2.21.0 | requests-cache==0.5.2
| requests-toolbelt==0.8.0 | roman==3.2
| rq==1.1.0 | s3transfer==0.1.13
| simplegeneric==0.8.1 | simplejson==3.16.0
| six==1.11.0 | smmap2==2.0.5
| sqlparse==0.3.0 | stevedore==1.30.1
| traitlets==4.3.2 | urllib3==1.24.3
| vine==1.3.0 | wcwidth==0.1.7
| webargs==5.1.3 | whitenoise==3.3.1
| `zipp==0.6.0

[pkfec]

Vulnerable packages are audited during weekly log process using SNYK tool. New Issues to handle package upgrades are submitted based on the severity. Eregs parsing requirements are monitored/fixed on monthly basis.