Releases: sorentwo/oban
v2.17.2
Enhancements
-
[Oban] Support passing changeset streams to
insert_all
.Accepting streams makes
Oban.insert_all
more flexible and may, in some circumstances, make it possible to reduce memory usage for streams of large resources.
Bug Fixes
-
[Config] Validate
:repo
option without checking forEcto.Repo
behaviour.Repo wrappers that don't implement all functions of the
Ecto.Repo
behaviour are still viable and shouldn't be validated with a behaviour check. This changes repo validation back to the way it was done in older versions, by checking that it's a valid module that exportsconfig/0
. -
[Peer] Handle rollback during
Oban.Peers.Postgres
peer electionInfrequently, the postgres peer election transaction returns
{:error, :rollback}
. Now that return value is handled to prevent a match error.The peer maintains its current
leader?
status on rollback—this may cause inconsistency if the leader encounters an error and multiple rollbacks happen in sequence. That tradeoff is acceptable because the situation is unlikely and less of an issue than crashing the peer. -
[Oban] Skip queue existence check for
pause_all_queues
andresume_all_queues
when thelocal_only
option is passed.
v2.17.1
v2.17.0
This release includes an optional database migration to disable triggers and relax priority checks. See the v2.17 upgrade guide for step-by-step instructions.
📟 Universal Insert Notifications
Historically, Oban used database triggers to emit a notification after a job is inserted. That allowed jobs to execute sooner, without waiting up to a second until the next poll event. Those triggers and subsequent notifications added some overhead to database operations bulk inserts into the same queue, despite deduplication logic in the trigger. Even worse, trigger notifications didn't work behind connection poolers and were restricted to the Postgres notifier.
Now insert notifications have moved out of the database and into application code, so it's possible to disable triggers without running database migrations, and they work for any notifier, not just Postgres.
Disable notifications with the insert_trigger
option if sub-second job execution isn't important or you'd like to reduce PubSub chatter:
config :my_app, Oban,
insert_trigger: false,
...
🧑🏭 Worker Conveniences
Workers received a few quality of life improvements to make defining unique
behaviour more expressive and intuitive.
First, it's now possible to define a job's unique period with time units like {1, :minute}
or {2, :hours}
, just like a job's :schedule_in
option:
use Oban.Worker, unique: [period: {5, :minutes}]
Second, you can set the replace
option in use Oban.Worker
rather than in an overridden new/2
or as a runtime option. For example, to enable updating a job's scheduled_at
value on unique conflict:
use Oban.Worker, unique: [period: 60], replace: [scheduled: [:scheduled_at]]
🐦🔥 Oban Phoenix Notifier
The new oban_notifiers_phoenix
package allows Oban to share a Phoenix application's PubSub for notifications. In addition to centralizing PubSub communications, it opens up the possible transports to all PubSub adapters. As Oban already provides Postgres
and PG
(Distributed Erlang) notifiers, the new package primarily enables Redis notifications.
config :my_app, Oban,
notifier: {Oban.Notifiers.Phoenix, pubsub: MyApp.PubSub},
...
🎚️ Ten Levels of Job Priority
Job priority may now be set to values between 0 (highest) and 9 (lowest). This increases the range from 4 to 10 possible priorities, giving applications much finer control over execution order.
args
|> MyApp.PrioritizedWorker.new(priority: 9)
|> Oban.insert()
Enhancements
-
[Oban] Add
Oban.pause_all_queues/2
andOban.resume_all_queues/2
.Pause and resume all queues with a single function call and a single notification signal, rather than manually looping through all queues and issuing separate calls.
-
[Cron] Add non-raising
Expression.parse/2
for use inCron.parse/2
and shared validations.Multiple locations used
parse!
and converted a raised exception into an error tuple. That was inefficient, repetitive, and violated the common practice of avoiding exceptions for flow control. -
[Validation] Use schema based validation for workers, plugins, and config.
Validations are now simpler and more consistent, and behaviour based notifiers such as Engine, Repo, and Peer are more descriptive.
-
[Engine] Expand telemetry meta for all engine callbacks events.
All callbacks now include every argument in telemetry event metadata. In some situations, e.g.
:init
, this simplifies testing and can be used to eliminate the need to poll a supervision tree to see which queues started. -
[Notifier] Add
Isolated
notifier for local use and simplified testing.Using PG for async tests has occasional flakes due to its eventually consistent nature. In tests and single node systems, we don't need to broadcast messages between instances or nodes, and a simplified "isolated" mechanism is ideal.
-
[Repo] Add
Repo.query!/4
forEcto.Repo
parity -
[Migration] Configure a third-party engine's migrator using the repo's
config
map.
Bug Fixes
-
[Cron] Guard against invalid cron range expressions where the left side is greater than the right, e.g.
SAT-FRI
. -
[Testing] Disable the
prefix
by default in generated testing helpers.A prefix is only necessary when it's not the standard "public" prefix, which is rarely the case in testing helpers. This makes it easier to use testing helpers with the
Lite
engine. -
[Testing] Remove
prefix
segment fromassert_enqueued
error messages.Not all engines support a prefix and the assert/refute message in testing helpers is confusing when the prefix is
nil
.
Deprecations
- [Gossip] The Gossip plugin is no longer needed, and shouldn't be used, by applications running Oban Web v2.10 or above.
v2.16.3
Bug Fixes
-
[Oban] Start
Peer
andStager
afterQueue
supervisorThe queue supervisor blocks shutdown to give jobs time to finish gracefully. During that time, the Peer could obtain or retain leadership despite all the plugins having stopped. Now, the Peer and Stager (only active on the leader) stop before the queue supervisor.
-
[Testing] Cast timestamp to utc_datetime in testing queries
Timestamps with a timezone are now cast to
:utc_datetime
via a changeset before runningOban.Testing
queries.
v2.16.2
Bug Fixes
-
[Testing] Match args/meta patterns in Elixir rather than the database
The containment operators,
@>
and<@
, used for pattern matching in tests are only available in Postgres and have some quirks. Most notably, containment considers matching any value in a list a successful match, which isn't intuitive or desirable.The other issue with using a containment operator in tests is that SQLite doesn't have those operators available and test helpers are shared between all engines.
Enhancements
-
[Testing] Support wildcard matcher in patterns for args/meta
Now that we match in Elixir, it's simple to support wildcard matching with a
:_
to assert that a key is present in a json field without specifying an exact value.assert_enqueued args: %{batch_id: :_, callback: true}
v2.16.1
v2.16.0
🐑 Oban Instance Module
New facade modules allow you to call Oban
functions on instances with custom names, e.g. not Oban
, without passing a t:Oban.name/0
as the first argument.
For example, rather than calling Oban.config/1
, you'd call MyOban.config/0
:
MyOban.config()
It also makes piping into Oban functions far more convenient:
%{some: :args}
|> MyWorker.new()
|> MyOban.insert()
🧩 Partial Matches in Testing Assertions
It's now possible to match a subset of fields on args or meta with all_enqueued
, assert_enqueued
, and refute_enqueued
. For example, the following assertion will now pass:
# Given a job with these args: %{id: 123, mode: "active"}
assert_enqueued args: %{id: 123} #=> true
assert_enqueued args: %{mode: "active"} #=> true
assert_enqueued args: %{id: 321, mode: "active"} #=> false
The change applies to args
and meta
queries for all_enqueued/2
, assert_enqueued/2
and refute_enqueued/2
helpers.
⏲️ Unique Timestamp Option
Jobs are frequently scheduled for a time far in the future and it's often desirable for to consider scheduled
jobs for uniqueness, but unique jobs only checked the :inserted_at
timestamp.
Now unique
has a timestamp
option that allows checking the :scheduled_at
timestamp instead:
use Oban.Worker, unique: [period: 120, timestamp: :scheduled_at]
Bug Fixes
-
[Reindexer] Correct relname match for reindexer plugin
We can safely assume all indexes start with
oban_jobs
. The previous pattern was based on an outdated index format from older migrations. -
[Testing] Support
repo
,prefix
, andlog
query options inuse Oban.Testing
v2.15.3
Enhancements
-
[Pruner] Prune jobs using the
scheduled_at
timestamp regardless of state.The previous pruning query checked a different timestamp field for each prunable state, e.g.
cancelled
usedcancelled_at
. There aren't any indexes for those timestamps, let alone the combination of each state and timestamp, which led to slow pruning queries in larger databases.In a database with a mixture of ~1.2m prunable jobs the updated query is 130x faster, reducing the query time from 177ms down to 1.3ms.
-
[Lite] Avoid unnecessary transactions during staging and pruning operations
Contention between SQLite3 transactions causes deadlocks that lead to period errors. Avoiding transactions when there isn't anything to write minimizes contention.
Bug Fixes
-
[Foreman] Explicitly pause queues when shutdown begins.
A call to
Producer.shutdown/1
was erroneously removed during theDynamicSupervisor
queue refactor. -
[Job] Preserve explicit state set along with
scheduled_in
time.The presence of a
scheduled_in
timestamp would always set the state toscheduled
, even when an explicit state was passed.
v2.15.2
Enhancements
-
[Repo] Pass
oban: true
option to all queries.Telemetry options are exposed in Ecto instrumentation but aren't obviously available in the opts passed to
prepare_query
. Now all queries have anoban: true
option so users can ignore them in multi-tenancy setups. -
[Engine] Generate a UUID for all
Basic
andLite
queue instances to aid in identifying orphaned jobs or churning queue producers. -
[Oban] Use
Logger.warning/2
and replace deprecated use of:warn
level with:warning
across all modules.
Bug Fixes
-
[Job] Validate changesets during
Job.to_map/1
conversionThe
Job.to_map/1
function converts jobs to a map "entry" suitable for use ininsert_all
. Previously, that function didn't perform any validation and would allow inserting (or attempting to insert) invalid jobs duringinsert_all
. Aside from inconsistency withinsert
andinsert!
,insert_all
could insert invalid jobs that would never run successfully.Now
to_map/1
usesapply_action!/2
to apply the changeset with validation and raises an exception identical toinsert!
, but before calling the database. -
[Notifier] Store PG notifier state in the registry for non-blocking lookup
We pull the state from the notifier's registry metadata to avoid
GenServer.call
timeouts when the system is under high load. -
[Migration] Add
primary_key
explicitly during SQLite3 migrationsIf a user has configured Ecto's
:migration_primary_key
to something other thanbigserial
the schema is incompatible with Oban's job schema.
v2.15.1
Enhancements
-
[Telemetry] Add
[:oban, :stager, :switch]
telemetry event and use it for logging changes.Aside from an instrumentable event, the new logs are structured for consistent parsing on external aggregators.
-
[Job] Remove default priority from job schema to allow changing defaults through the database
Bug Fixes
-
[Basic] Restore
attempt < max_attempts
condition when fetching jobsIn some situations, a condition to ensure the attempts don't exceed max attempts is still necessary. By checking the attempts outside the CTE we maintain optimal query performance for the large scan that finds jobs, and only apply the check in the small outer query.
-
[Pruner] Add missing
:interval
tot:Oban.Plugins.Pruner.option/0