Upgrade localstack version and remove pinned dependencies

[guille-sage]

Upgrade localstack version and remove pinned dependencies

In commit bdfb0d6 we pinned AWS dependencies, since we weren't sure if eventq was ready for upgrading them. Also they dropped Ruby 2.5 support. Since we have moved to have no pinned dependencies if possible, each environment can choose the best and most up to date dependencies available, so we can keep the Ruby 2.5 support and also ensure using the latest AWS libraries in Ruby 3.

Also we removed the fixed localstack version, as in version 1.4.0 they addressed the multiregion support.

Refactor AWS integration specs

Upgrading our localstack version showed how brittle our integration tests are, because newer localstack versions react a bit slower (read: 100+ ms to pick up a message instead of 10+ ms). Since we used sleep statements a ton to wait "just the right amount of time" to exceed the TTL, specs became flaky or consistently failed because the worker was picking them up later than intended.

Looking deeper into it showed that our approach was flawed from the get go. While we name the variable in specs retry_attempt_count it is actually a call_count. Therefore all sleeps were off by one iteration. In the past this created the following patterns in specs:

• 0.0s Boot worker
• 0.1s sleep 1 to let worker pick up message
• 0.2s Worker processes message
• 1.1s Check message has been processed
• 1.1s sleep 1 to wait for retry
• 1.2s Worker processes retry

Sometimes there was just 10 ms between assertion and worker processing the message. Since the worker is on a thread, this could have failed with older localstack versions as well. With the newer and slower localstack version most specs looked like this:

• 0.0s Boot worker
• 0.1s sleep 1 to let worker pick up message
• 0.8s Worker processes message
• 1.1s Check message has been processed
• 1.1s sleep 1 to wait for retry
• 2.0s Worker processes retry
• 2.1s sleep 1 to wait for 2nd retry
• 3.1s Check for 2nd retry fails as worker has not picked up 2nd retry yet

The flaw here is that visibility_timeout does not tell when the message is picked up again, but for how long it is NOT PICKED UP again. So while we need to at least wait for the visibility_timeout in specs, we cannot know when the worker actually comes around to pick the message up again. If we were using bigger numbers, like minutes for the visibility_timeout rather than seconds, the 0.1 - 1.9 seconds sometimes needed to pick up a message would not matter, but also would slow down specs to a halt.

The second flaw is that we are basically testing libraries here. We should trust AWS to ensure messages are not picked up for the visibility_timeout we set. Instead we should only test if the right visibility_timeout numbers are calculated for the libraries to use.

As a result we have adjusted the specs to use a visibility_timeout of zero whenever possible to speed up specs. However, we note down the calculated number to ensure the right one is normally passed on to AWS. This allows us to use the exponential backoff strategy, without waiting expontentially long.

In order to remove the need for sleep in specs, as the operation to test is run asynchroneously, we introduced a thread based queue to wait for results on. This way, as soon as a result is pushed to the queue in another thread, the thread waiting for results immediately picks it up and continues running the spec and assertions. By adding a decorator around a method that is invoked after each time a message was processed, we can simply say wait_for_message_processed to ensure the spec wait just long enough to continue.

To test visibility_timeout we can now use aws_visibility_timeout_queue, which gets pushed the visibility_timeout calculated for each call, e.g.

expect(aws_visibility_timeout_queue.pop).to eq(call: 5, visibility_timeout: 5)

Using these techniques we sped up specs considerably, made them much more reliable and still kept the integration part as real AWS libraries are used. Also we can still integrate with a real AWS account instead of using localstack.

Set log level to FATAL during spec runs

Since the logger is printing messages to STDOUT, all DEBUG, INFO and ERROR logs produced during tests are included in the test output. Since we rely heavily on failing to process a message on purpose, we have littered the output with ERROR logs and traces as well. By setting the log level to FATAL during spec runs, we ensure that we only see the spec output during test runs.

The base branch was changed.