🐛Webserver: exclusive/non-exclusive RabbitMQ consumers are deleting each other, and also probably replacing each other

[sanderegg]

What do these changes do?

After some Graylog analysis it came up that the following issue callback would happen quite often. It was found that both _rabbitmq_nonexclusive_queue_consumers and _rabbitmq_exclusive_queue_consumers were using the same app[APP_RABBITMQ_CONSUMERS_KEY] to store their respective consumers. E.G. on initialization the last one would replace the first one, and on deletion as well, thus the errors shown in Graylog. Not sure if this could also have some influence on the lost logs issues.

Another issue was also identified when a user is logging out (also visible in e2e tests), and for this an additional log output was added.

Unhandled exception:
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/channel.py", line 182, in rpc
    result = await countdown(self.rpc_frames.get())
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/tools.py", line 95, in __call__
    return await coro
  File "/usr/local/lib/python3.10/asyncio/queues.py", line 159, in get
    await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/abc.py", line 44, in __inner
    return await self.task
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.10/site-packages/servicelib/logging_utils.py", line 232, in log_catch
    yield
  File "/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/notifications/_rabbitmq_exclusive_queue_consumers.py", line 204, in _unsubscribe_from_rabbitmq
    await logged_gather(
  File "/home/scu/.venv/lib/python3.10/site-packages/servicelib/utils.py", line 141, in logged_gather
    raise error
  File "/home/scu/.venv/lib/python3.10/site-packages/servicelib/rabbitmq/_client.py", line 298, in unsubscribe
    queue = await channel.get_queue(queue_name)
  File "/home/scu/.venv/lib/python3.10/site-packages/aio_pika/channel.py", line 375, in get_queue
    return await self.declare_queue(name=name, passive=True)
  File "/home/scu/.venv/lib/python3.10/site-packages/aio_pika/robust_channel.py", line 233, in declare_queue
    queue: RobustQueue = await super().declare_queue(   # type: ignore
  File "/home/scu/.venv/lib/python3.10/site-packages/aio_pika/channel.py", line 350, in declare_queue
    await queue.declare(timeout=timeout)
  File "/home/scu/.venv/lib/python3.10/site-packages/aio_pika/queue.py", line 86, in declare
    self.declaration_result = await channel.queue_declare(
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/channel.py", line 858, in queue_declare
    return await self.rpc(
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/base.py", line 164, in wrap
    return await self.create_task(func(self, *args, **kwargs))
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/abc.py", line 46, in __inner
    raise self._exception from e
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/abc.py", line 46, in __inner
    raise self._exception from e
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/channel.py", line 441, in _reader
    await hook(frame)
  File "/home/scu/.venv/lib/python3.10/site-packages/aiormq/channel.py", line 410, in _on_close_frame
    raise exc
aiormq.exceptions.ChannelNotFoundEntity: NOT_FOUND - no queue 'simcore.services.instrumentation' in vhost '/'

Related issue/s

How to test

Dev Checklist

• No ENV changes or I properly updated ENV (read the instruction)

DevOps Checklist

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[pcrespov]

great catch!

[matusdrobuliak66]

Thanks! this might be also connected with sometimes missing logs?