Ensure Autoscaler never reduces processes to less than 1

[LasseRafn]

Last week, we started noticing some of our queues having 0 Processes assigned, despite this being against how Horizon is designed to work. Processes must always be positive, even if you specifically set minProcesses = 0.

After much digging, I eventually went through and added Log::debug statements all over the AutoScaler:

[2022-11-25 14:40:20]: Horizon maxProcesses: 15  
[2022-11-25 14:40:20]: Horizon processPools: 2  
[2022-11-25 14:40:20]: Horizon minProcesses: 1  
[2022-11-25 14:40:20]: Horizon $totalProcessCount: 1            <---- 
[2022-11-25 14:40:20]: Horizon totalProcessCountMethod: 16  
[2022-11-25 14:40:20]: Horizon $maxUpShift: -1                  <----
[2022-11-25 14:40:20]: Horizon desiredProcessCount: 15  
[2022-11-25 14:40:20]: Horizon pool->scale: 0  

Somehow Horizon would spawn 1 more process than was allowed (another issue for another day) and when then subtracting maxProcesses with totalProcessCount(), we would get -1.

Later down in the code it then picks which value to use for scaling:

min(
    $totalProcessCount + $maxUpShift,
    $supervisor->options->maxProcesses - (($supervisor->processPools->count() - 1) * $supervisor->options->minProcesses),
    $desiredProcessCount
)

Conclusion

As we saw above, $totalProcessCount = 1 and $maxUpShift = -1, which means this will return 0, setting processes to zero!

So when it tries to scale UP, it actually ends up scaling DOWN. Once it then hits 0, it seems to never be able to recover.

Solution

It is luckily not a difficult problem to solve. Wrapping max(0, ...) around the code that calculates $maxUpShift will prevent it from ever being negative, thus solving our problem.

I am aware that this technically fixes a problem that you might think is not there (why is processes greater than maxProcesses?) and whilst I agree, I also don't see the harm in covering all the bases.

Similar issue(s)

• Queues with 0 processes #981

My abbreviated config for reference

[
    "waits" => [
      "redis:default" => 60,
    ],
    "fast_termination" => false,
    "memory_limit" => 512,
    "defaults" => [
      "app" => [
        "connection" => "redis",
        "queue" => [
          "default",
          "test",
        ],
        "balance" => "auto",
        "maxProcesses" => 1,
        "memory" => 512,
        "tries" => 2,
        "nice" => 5,
        "timeout" => 1800,
      ],
    ],
    "environments" => [
      "production" => [
        "app" => [
          "maxProcesses" => 15,
          "balance" => "auto",
          "tries" => 10,
          "backoff" => [
            60,
            300,
            600,
            900,
          ],
        ],
      ],
    ],
  ]