Get Celery working with new redis service

[lbeaufort]

Cloud.gov is transitioning to a managed redis service, which will require some code changes. We should test this service while it's still in beta, and give ourselves plenty of time before they retire the old service.

Completion criteria

• Get Celery (downloads) working with new redis service
• Make sure nothing explodes with new service in place ➡️ downloads don't work
• Ask cloud.gov when they plan to deprecate redis32: how much time do we have to get this working? ➡️ around October
• Is anyone else using Redis as a celery backend? ➡️ not that they know
• Let cloud.gov know that deleting isn't working

Technical steps

• Create service cf create-service SERVICE_NAME PLAN_NAME INSTANCE_NAME
cf create-service redis BETA-redis-5node fec-redis-beta
• Test delete - fails

cf delete-service fec-redis-beta
Really delete the service fec-redis-beta?> y
Deleting service fec-redis-beta in org fec-beta-fec / space dev as [email protected]...
FAILED
Server error, status code: 502, error code: 10001, message: Service instance fec-redis-beta: Service broker error: There was an error deleting the instance.

• Test manual deploy - downloads don't work (PR used for testing: Branch for manual deploys #4390)

Celery is the app connecting to redis, so we need to look at how those settings need to change. Maybe https://docs.celeryproject.org/en/stable/userguide/configuration.html#broker-use-ssl is what we need. Compare to code examples below, which are pure Redis client examples

app = celery.Celery('openfec')
app.conf.update(
    broker_url=redis_url(),
    imports=(
        'webservices.tasks.refresh',
        'webservices.tasks.download',
        'webservices.tasks.legal_docs',
    ),
    beat_schedule=schedule,
    broker_connection_timeout=30,  # in seconds
    broker_connection_max_retries=0,  # for unlimited retries
    task_acks_late=False
)

• Add better logging when celery can't connect to Redis? Can't see anything in the logs, downloads just spin. Celery logs will fail if it can't connect in some circumstances
• Test different service levels?
• Test passing 'none' to celery settings instead of importing ssl

---

Email from cloud.gov
The cloud.gov team is pleased to announce the launch of a beta of a new Redis broker using AWS Elasticache. This work is an important step in moving cloud.gov towards a more stable and reliable platform by moving away from a bespoke Kubernetes instances to AWS-hosted services.

The intent of the Redis broker beta is to test out the service with a limited number of customers before making it available to all cloud.gov customers. We’ve outlined below a few important things to keep in mind before and while you are using the beta.

The beta can only be used in non-production environments. cloud.gov expects this new service to be in beta for at least two weeks but it could be extended based on artifacts found and customer feedback.

For this beta, the cloud.gov operators will be limiting access to this new broker by specific organizations only. Due to limitations of Cloud Foundry, we can not limit access to specific spaces under an org. We ask that interested customers email [email protected] to request access to the beta and specify the org(s) in which they would like to enable the beta service.

While we took every measure to test the broker before releasing it to beta, testing for performance is one area that is difficult to assess pre-release. Therefore, we will be closely monitoring performance once in beta but we can’t say with certainty that it will be performant as expected.

The cloud.gov team will use the weekly office hours to solicit feedback from you about the beta.

The cloud.gov team reserves the right to update and change the broker and plan details while still in beta testing based on testing and feedback results.

The services instances created in beta testing will not be carried over into production once the service goes General Availability (GA). Once GA is announced, the beta plans will be disabled from creating new instances and deleted a short time later.

Please submit any feedback or support requests about the Redis broker beta to [email protected].

If you would like to participate in the beta, please contact [email protected] to request access to the beta and specify the org(s) in which you would like to enable the beta service.

Technical Notes for the beta release:

• The service name will simply be called redis

• The Redis offering will come in three plans:

name	nodes	memory	maintenance
BETA-redis-dev	Single node instance of Redis	512 MB of memory	AWS minor update maintenance is enabled weekly after hours
BETA-redis-3node	Three node instance of Redis in Primary/Replica mode with auto failover across two availability zones	512 MB of memory	AWS minor update maintenance is enabled weekly after hours
BETA-redis-5node	Five node instance of Redis in Primary/Replica mode with auto failover across two availability zones	512 MB of memory	AWS minor update maintenance is enabled weekly after hours

The current Redis version offered will be 5.0.6

Standard Redis password AUTH will be enabled like our current Redis offering

AWS does require TLS based communications with the cluster which is not native in the Redis client so customers will need to make changes to their applications to use this new service

Python examples:

• https://redislabs.com/lp/python-redis/
• https://github.com/andymccurdy/redis-py#ssl-connections
• https://github.com/cloud-gov/aws-redis-example/tree/master/python

If customers have successful code changes in various languages that they would like to share examples of with other cloud.gov customers, we encourage PRs to our example repo: https://github.com/cloud-gov/aws-redis-example

Thank you.

[lbeaufort]

In order to get this to work, I needed to add both

    broker_use_ssl={
        'ssl_cert_reqs': ssl.CERT_NONE,
    },
    redis_backend_use_ssl={
        'ssl_cert_reqs': ssl.CERT_NONE,
    },

to the celery conf (see celery/celery#5371)
and

    'settings': {
        'url': redis_url() + "?ssl=True",

to the celery-once conf. (see https://github.com/cameronmaske/celery-once/blob/4f68b6b3c8503b9360179a246521b07315cdf577/README.rst#redis-backend)

Alternately, it looks like celery and celery-once both support passing rediss:// to indicate TLS, but I can't get it to work.

def redis_url(use_ssl=True, use_cert_reqs=False):
    """
    Retrieve the URL needed to connect to a Redis instance, depending on environment.

    When running in a cloud.gov environment, retrieve the uri credential for the 'redis32' service.
    """

    # Is the app running in a cloud.gov environment
    if env.space is not None:
        redis_env = env.get_service(label='redis')
        redis_url = redis_env.credentials.get('uri')
        if use_ssl:
            redis_url = redis_url.replace("redis", "rediss")
        if use_cert_reqs:
            redis_url += "?ssl_cert_reqs=none"

        return redis_url

    return env.get_credential('FEC_REDIS_URL', 'redis://localhost:6379/0')


app = celery.Celery('openfec')
app.conf.update(
    broker_url=redis_url(),
    # TODO: If we move back to 4.3 we might need this to be 'ssl.CERT_NONE' in quotes
    # See https://github.com/celery/celery/issues/5371
    # cgeck kombu version
    # broker_use_ssl={
    #     'ssl_cert_reqs': ssl.CERT_NONE,
    # },
    # redis_backend_use_ssl={
    #     'ssl_cert_reqs': ssl.CERT_NONE,
    # },
    imports=(
        'webservices.tasks.refresh',
        'webservices.tasks.download',
        'webservices.tasks.legal_docs',
    ),
    beat_schedule=schedule,
    broker_connection_timeout=30,  # in seconds
    broker_connection_max_retries=0,  # for unlimited retries
    task_acks_late=False
)

app.conf.ONCE = {
    'backend': 'celery_once.backends.Redis',
    'settings': {
        'url': redis_url(use_cert_reqs=True),
        'default_timeout': 60 * 60
    }
}

[lbeaufort]

Some questions for cloud.gov:

• Would you expect connecting to the existing (non-beta) Redis service with SSL to fail?
• When I tested a few variations of the above changes, no errors were indicated in our logs, which is a bit alarming that there was a “silent” error. I did see errors when I tried to connect with rediss (see Get Celery working with new redis service #4358 (comment) for more info) but not when I was missing one of the 3 changes in my WIP PR

[lbeaufort]

Follow-up ticket here: #4393