feat(queue): Add failed job backoff configuration

[deer-wmde]

Currently failed jobs will be retried 5 times[1]. This adds job backoff configuration as suggested here, so that failed retries will happen with incremental delays:

• 10 seconds
• 1 minute
• 5 minutes
• 15 minutes
• 30 minutes

[1] https://github.com/wbstack/api/blob/main/start.sh#L20

Draft status because I couldn't make this work on my local cluster yet.

[m90]

When trying to test this, do you make jobs fail by calling $job->fail() or by throwing an exception? Not sure if it makes a difference, but maybe it's a first idea where to start debugging.

[deer-wmde]

When trying to test this, do you make jobs fail by calling $job->fail() or by throwing an exception? Not sure if it makes a difference, but maybe it's a first idea where to start debugging.

I tried testing without changing code, I tried to create wikis while no mediawiki pods are around (so I guess this is equivalent to the exception throwing). Will try those two ways explicitly, I didn't know about the $job->fail method

[deer-wmde]

When trying to test this, do you make jobs fail by calling $job->fail() or by throwing an exception? Not sure if it makes a difference, but maybe it's a first idea where to start debugging.

Turns out this makes ALL the difference. Calling $job->fail() (or rather $this->fail() inside a job) will tell Laravel to not retry that job.

While I'm now confident that this PR works as intended, this means that we have to refactor basically every Job class, that we want to respect the retry config (not just the one in this PR, but in general).

[m90]

If someone would set QUEUE_BACKOFF to the empty string, would this use the default? I'm wondering if we should have a null default so that we can turn this behavior on and off by means of configuration.

[deer-wmde]

Good thinking, I removed the values