Releases: oban-bg/oban
v2.18.3
Enhancements
-
[Basic] Use the shared concat operator when appending errors.
The standard
push
operation for updates is designed for arrays and usesarray_append
internally. This replaces all use ofpush
with a fragment that uses the||
operator instead, which works for both arrays and jsonb.CockroachDB doesn't support arrays of jsonb, but they do support simple jsonb columns. Now we can append to the errors column in either format for CRDB compatibility.
Bug Fixes
-
[Queue] Link the dynamic queue supervisor and
Midwife
for automatic restarts.When a producer crashes it brings the queue's supervisor down with it. With enough database errors, the producer may crash repeatedly enough to exhaust restarts and bring down the DynamicSupervisor in charge of all queues.
Now the supervisor is linked to the midwife to ensure that the midwife restarts as well, and it restarts all of the queues.
-
[Testing] Handle
insert_all/3
with streams for the:inline
testing engine.The inline engine's
insert_all_jobs
callback incorrectly expected changesets to always be a list rather and couldn't handle streams.
v2.18.2
Bug Fixes
-
[Repo] Prevent debug noise by ensuring default opts for standard transactions.
Without default opts each transaction is logged. Many standard operations execute each second, which makes for noisy logs. Now transaction opts are passed as a third argument to ensure defaults are applied.
-
[Repo] Increase transaction retry delay and increase with each attempt.
Bump the base transaction retry from 100ms to 500ms, and increase linearly between each successive attempt to provide deeper backoff. This alleviates pressure on smaller connection pools and gives more time to recover from contentions failures.
v2.18.1
Enhancements
-
[Repo] Automatically retry all transactions with backoff.
Avoid both expected an unexpected database errors by automatically retrying transactions. Some operations, such as serialization and lock not available errors, are likely to occur during standard use depending on how a database is configured. Other errors happen infrequently due to pool contention or flickering connections, and those should also be retried for increased safety.
This change is applied to
Oban.Repo.transaction/3
itself, so it will apply to every location that uses transactions. -
[Migration] Declare
tags
as an array oftext
rather thanvarchar
.We don't provide a limit on the size of tags and they could conceivably be larger than 256 characters. Externally the types are interchangeable, but internally there are minor advantages to using the text type.
There isn't a new migration; this change is only for new tables.
Bug Fixes
- [Repo] Correctly dispatch
query!/4
toquery!
rather thanquery
without a bang.
v2.18.0
🔭 Queue Shutdown Telemetry
A new queue shutdown event, [:oban, :queue, :shutdown]
, is emitted by each queue when it terminates. The event originates from the watchman
process, which tracks the total ellapsed time from when termination starts to when all jobs complete or the allotted period is exhausted.
Any jobs that take longer than the :shutdown_grace_period
(by default 15 seconds) are brutally killed and left as orphans. The ids of jobs left in an executing state are listed in the event's orphaned
meta.
This also adds queue:shutdown
logging to the default logger. Only queues that shutdown with orphaned jobs are logged, which makes it easier to detect orphaned jobs and which jobs were affected:
[
message: "jobs were orphaned because they didn't finish executing in the allotted time",
queue: "alpha",
source: "oban",
event: "queue:shutdown",
ellapsed: 500,
orphaned: [101, 102, 103]
]
🚚 Distributed PostgreSQL Support
It's now possible to run Oban in distributed PostgreSQL databases such as Yugabyte. This is made possible by a few simple changes to the Basic
engine, and a new unlogged
migration option.
Some PostgreSQL compatible databases don't support unlogged tables. Making oban_peers
unlogged isn't a requirement for Oban to operate, so it can be disabled with a migration flag:
defmodule MyApp.Repo.Migrations.AddObanTables do
use Ecto.Migration
def up do
Oban.Migration.up(version: 12, unlogged: false)
end
end
🧠 Job Observability
Job stop
and exception
telemetry now includes the reported memory and total reductions from the job's process. Values are pulled with Process.info/2
after the job executes and safely fall back to 0
in the event the process has crashed. Reductions are a rough proxy for CPU load, and the new measurements will make it easier to identify computationally expensive or memory hungry jobs.
In addition, thanks to the addition of Process.set_label
in recent Elixir versions, the worker name is set as the job's process label. That makes it possible to identify which job is running in a pid
via observer or live dashboard.
v2.18.0 — 2024-07-26
Enhancements
-
[Job] Support simple
unique: true
andunique: false
declarationsUniqueness can now be enabled with
unique: true
and disabled withunique: false
from job options or a worker definition. Theunique: true
option uses all the standard defaults, but sets the period to:infinity
for compatibility with Oban Pro's newsimple
unique mode. -
[Cron] Remove forced uniqueness when inserting scheduled jobs.
Using uniqueness by default prevents being able to use the Cron plugin with databases that don't support uniqueness because of advisory locks. Luckily, uniqueness hasn't been necessary for safe cron insertion since leadership was introduced and scheduling changed to top-of-the-minute many versions ago.
-
[Engine] Introduce
check_available/1
engine callbackThe
check_available/1
callback allows engines to customize the query used to find jobs in theavailable
state. That makes it possible for alternative engines, such Oban Pro's Smart engine, to check for available jobs in a fraction of the time with large queues. -
[Peer] Add
Oban.Peer.get_leader/2
for checking leadershipThe
get_leader/2
function makes it possible to check which node is currently the leader regardless of the Peer implementation, and without having to query the database. -
[Producer] Log a warning for unhandled producer messages.
Some messages are falling through to the catch-all
handle_info/2
clause. Previously, they were silently ignored and it degraded producer functionality because inactive jobs with dead pids were still tracked asrunning
in the producer. -
[Oban] Use structured messages for most logger warnings.
A standard structure for warning logs makes it easier to search for errors or unhandled messages from Oban or a particular module.
Bug Fixes
-
[Job] Include all fields in the unique section of
Job.t/0
.The unique spec lacked types for both
keys
andtimestamp
keys. -
[Basic] Remove
materialized
option fromfetch_jobs/3
.The
MATERIALIZED
clause for CTEs didn't make a meaningful difference in job fetching accuracy. In some situations it caused a performance regression (which is why it was removed from Pro's Smart engine a while ago).
v2.17.11
Bug Fixes
-
[Oban] Handle deprecation warnings from Elixir 1.17
-
[Notifier] Prevent noisy logging about switching between modes.
There's an apparent race condition in Sonar between pruning stale nodes on
:ping
and updating the status after a notification. This primarily happens in development for two reasons:- Development laptops are most prone to time warp because of system sleep.
- Apps only run a single node in development.
Using
monotonic_time/1
instead ofsystem_time/1
guards against clock drift/time warp effects. -
[Stager] Prevent notification status timeouts from bubbling into the Stager.
A clogged Ecto pool could cause cascading errors on startup due to a sequence of calls between the
Notifier
,Sonar
, andStager
.Sonar
sends a notification inhandle_continue
on startup.- The notification is blocked while the
Notifier
waits for a connection from the Ecto pool. Stager
checks for the connection status on startup, which would eventually time out because theSonar
hadn't finished initializing.- The
Stager
crashes from the timeout error.
This makes the following changes to prevent this sequence of events:
- The
Stager
no longer gets the sonar status during startup. - The
Notifier
catches timeout errors fromSonar
checks, warns about it, then returns an:unknown
status.
-
[Engine] Defensively check the process dictionary during inline testing.
Not all processes are guaranteed to return a value for the process dictionary. Sometimes a value was missing during inline testing, which would crash the test.
-
[Basic] Set
conflict?
flag when encountering a unique advisory lock.The
conflict?
flag wasn't set when inserting a unique job was blocked by an advisory lock. Now the flag is set on either a fetched duplicate, or when the advisory lock is set. -
[Job] Correct
replace_by_state_option
type by switching from keyword to tuples. -
[Config] Correctly type
shutdown_grace_period
as aninteger
rather than atimeout
.
v2.17.10
Enhancements
-
[Oban] Make all generated functions from
use Oban
overridable.Now the functions generated by
use Oban
are all marked withdefoverridable
for extensibility.
Bug Fixes
-
[Testing] Use
$callers
rather than$ancestors
for ancestry tree check.We care about Tasks for inline testing checks, not normal supervision tree ancestry. The
$callers
entry is the appropriate mechanism to find the trail of calling processes:
v2.17.9
Enhancements
-
[Testing] Check process ancestry tree for
with_testing_mode
override.Cascade the
with_testing_mode
block to nested processes that make use of:$ancestry
in the process dictionary, i.e. tasks. Now enqueuing a job within spawned processes likeTask.async
orTask.async_stream
will honor the testing mode specified inwith_testing_mode/2
. -
[PG] Support alternative namespacing in
PG
notifierBy default, all Oban instances using the same
prefix
option would receive notifications from each other. Now you can use thenamespace
option to separate instances that are in the same cluster without changing theprefix
.
Bug Fixes
-
[Oban] Restore zero arity version of
pause_all_queues/0
Both pause and resume variants lost their default argument in a refactor that shifted around guard clauses.
-
[Oban] Add
:oban_draining
to process dict while drainingThe flag marks the test process while draining to give hints to the executor and engines. It fixes an incompatibility between
Oban.drain_queue/2
and Pro'sTesting.drain_jobs/2
.
v2.17.8
Enhancements
-
[Backoff] Backoff retry on DBConnection and Postgrex errors from GenServer calls.
GenServer calls that result in a
ConnectionError
orPostgrex.Error
should also be caught and retried rather than crashing on the first attempt.
Bug Fixes
-
[Notifier] Check for a live notifier process and propagate notify errors.
The
Notifier.notify/1
spec showed it would always return:ok
, but that wasn't the case when the notifier was disconnected or the process was no longer running. Now an error tuple is returned when a notifier process isn't running.This situation happened most frequently during shutdown, particularly from external usage of the Notifier like an application or the
oban_met
package.In addition, the errors bubble up through top level
Oban
functions likescale_queue/1
,pause_queue/1
, etc. to indicate that the operation can't actually succeed. -
[Peers.Postgres] Rescue
DBConnection.ConnectionError
in peer leadership check.Previously, only
Postgrex.Error
exceptions were rescued and other standard connection errors were ignored, crashing the Peer. Because leadership is checked immediately after the peer initializes, any connection issues would trigger a crash loop that could bring down the rest of the supervision tree.
v2.17.7
Bug Fixes
-
[Notifier] Prevent Sonar from running in
:testing
modes.Sonar has no purpose during tests, and it can cause sandbox issues when tests run with the Postgres notifier.
-
[Oban] Correctly handle pause and resume all with opts.
The primary clause had two default arguments and it was impossible to call
pause_all_queues/1
orresume_all_queues/1
with opts and no name.
v2.17.6
Enhancements
-
[Cron] Include cron indicator and original cron expression in meta.
When the Cron plugin inserts jobs the original, unnormalized cron expression is now stored in a job's meta under the
cron_expr
key. For compatibility with Pro'sDynamicCron
, meta also hascron: true
injected. -
[Worker] Change
backoff/1
spec to allow immediate rescheduling by returning0
The callback now specifies a
non_neg_integer
to allow retrying 0 seconds into the future. This matches the abilitiy to use{:snooze, 0}
.
Bug Fixes
-
[Notifier] Revert using single connection to deliver notices.
Production use of the single notifier connection in various environments showed timeouts from bottlenecks in the single connection could crash the connection and start new ones, leaving extra idle processes. In addition, Use interpolated NOTIFY instead of pg_notify with a JSON argument added parsing load because it could no longer use prepared statements.
-
[Worker] Apply custom backoff on timeout and unhandled exit
A worker's custom
backoff/1
wasn't applied after aTimeoutError
or unhandled exit. That's because the producer stores the executor struct before resolving the worker module. Now the module is resolved again to ensure custombackoff/1
are used. -
[Job] Revert setting the default priority key in Job schema.
The Ecto default was removed to allow overriding the default in the database and it was purposefully removed.