[High Priority] Fix Snyk vulnerability: Upgrade celery, vine, click

[johnnyporkchops]

Summary

This is due 03/29/22 so we need to get it in upcoming release

Upgrade to Celery 5.x has breaking changes:

• command to start the celery app has changed:

 - celery --app webservices.tasks  beat 
 - celery --app webservices.tasks worker

• minimum required kombu version is 5.0.0

• Resolves [Snyk: Medium] Celery - Stored Command Injection (03/29/2022) #5017

• Reference pr [WIP] update celery and related pkgs #4895

Upgrading Celery for Snyk recommended remediation path caused compatibility errors when installing requirements.

So I updated Click and Vine as well, Now requirements can be installed without error and Pytest passes,

1. Update click python package in: requirements-dev.txt
2. Update celery, kombu, redis python packages in: requirements.txt
3. Update the celery startup command in dev/feature/stage and prod manifest files

Required reviewers

Two backend developers must. @johnnyporkchops git didn't let me add you to the reviewers list. please feel free to test this PR if your time permits.

Impacted areas of the application

Celery tasks, Export results (download results set in fec.gov)

How to test

• git checkout 5017-snyk-medium-Celery-stored-command-injection
• pyenv virtualenv 3.7.12 venv-api3712
• pyenv activate venv-api3712
• pip install -r requirements.txt
• pip install -r requirements-dev.txt
• rm -rf node_modules (optional step)
• npm i (optional step)
• npm run build (optional step)
• pytest(all tests should pass)
• test downloads and celery tasks on local.Follow the wiki instructions
• use this command to start celery worker on cli: celery --app webservices.tasks worker --loglevel INFO

OR

Deploy this PR to Dev space:

• In tasks.py update:

DEPLOY_RULES = (

('dev', lambda _, branch: branch == '5017-snyk-medium-Celery-stored-command-injection'),

)

• test export/downloads
• Example: https://dev.fec.gov/data/receipts/?data_type=processed&two_year_transaction_period=2022&min_date=01%2F01%2F2021&max_date=01%2F01%2F2021&contributor_state=CA&recipient_committee_type=S

Note: I will the update the wiki after this PR is merged

[fec-jli]

The current celery version in requirement.txt, we should check kombu version:
celery==4.3.0 # if celery version is updated, need to verify compatibility with kombu and ensure correct version of kombu is pinned above

openFEC/requirements.txt

Line 42 in f24487c

celery==4.3.0 # if celery version is updated, need to verify compatibility with kombu and ensure correct version of kombu is pinned above

[johnnyporkchops]

The current celery version in requirement.txt, we should check kombu version:
celery==4.3.0 # if celery version is updated, need to verify compatibility with kombu and ensure correct version of kombu is pinned above

@jun, Kombu is updated to 5.24

[pkfec]

@johnnyporkchops Deployed a test branch to Dev space. Downloads failed with the following error.

   2022-03-29T10:23:35.80-0400 [APP/PROC/WEB/0] ERR An API error occurred with the status code of 500 (__init__() got an unexpected keyword argument 'username').
   2022-03-29T10:23:35.80-0400 [APP/PROC/WEB/0] OUT ERROR:webservices.rest:An API error occurred with the status code of 500 (__init__() got an unexpected keyword argument 'username').
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR [2022-03-29 14:23:35 +0000] [129] [ERROR] Error handling request /v1/download/totals/pac-party/?api_key=XXXXX&api_key=&sort_hide_null=false&sort_nulls_last=true&cycle=2022&sort=-receipts
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR Traceback (most recent call last):
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR File "/home/vcap/deps/0/python/lib/python3.7/site-packages/kombu/utils/functional.py", line 30, in __call__
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR return self.__value__
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR AttributeError: 'ChannelPromise' object has no attribute '__value__'
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR During handling of the above exception, another exception occurred:
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR Traceback (most recent call last):
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR File "/home/vcap/deps/0/python/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 925, in create_channel
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR return self._avail_channels.pop()
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR IndexError: pop from empty list
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR During handling of the above exception, another exception occurred:
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR Traceback (most recent call last):
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR File "/home/vcap/deps/0/python/lib/python3.7/site-packages/redis/connection.py", line 988, in get_connection
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR connection = self._available_connections.pop()
   2022-03-29T10:23:35.81-0400 [APP/PROC/WEB/0] ERR IndexError: pop from empty list

[pkfec]

@johnnyporkchops There are breaking changes in celery 5.X.X.
I just found out the way we start the celery worker has changed:

celery v4.3.0:
5088-venv) macadmins-mbp-5:openFEC pkasireddy$ celery worker --app webservices.tasks --loglevel INFO

celery v5.X.X: (5088-venv) macadmins-mbp-5:openFEC pkasireddy$ celery --app webservices.tasks worker --loglevel INFO

[codecov-commenter]

Codecov Report

Merging #5088 (e085d49) into develop (8e99aca) will not change coverage.
The diff coverage is n/a.

❗ Current head e085d49 differs from pull request most recent head 6132acd. Consider uploading reports for the commit 6132acd to get more accurate results

@@           Coverage Diff            @@
##           develop    #5088   +/-   ##
========================================
  Coverage    85.75%   85.75%           
========================================
  Files           81       81           
  Lines         7721     7721           
========================================
  Hits          6621     6621           
  Misses        1100     1100           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e99aca...6132acd. Read the comment docs.

[johnnyporkchops]

@pkfec Everything looks good! Great job

[fec-jli]

We deploy on dev. works as expected.
Thanks. good job.

[cnlucas]

Everything worked as expected and thank you for pairing with the test call @fec-jli and @pkfec