Priority Queues

Jobs differ from one-another in terms of:

• Rate of ingestion
• Time of execution
• Failure rates

To handle different types of jobs, Goose allows configuration of custom queues via :queue key in client & worker opts.

Nuances

• A single worker process only executes job from 1 queue
  • Spawn multiple worker processes to execute jobs from multiple queues
• Queue name cannot be scheduled-jobs, cron-entries, cron-schedules or dead-jobs as Goose uses them internally
• Failed jobs can be routed to a different retry queue

Usage

(ns priority-queues
  (:require
    [goose.client :as c]
    [goose.worker :as w]))

(def high-priority-opts (assoc client-opts :queue "high"))
(c/perform-async high-priority-opts `high-priority-fn :foo)

(def low-priority-opts (assoc client-opts :queue "low"))
(c/perform-async low-priority-opts `low-priority-fn :bar)

(w/start (assoc worker-opts :queue "high" :threads 10))
(w/start (assoc worker-opts :queue "low" :threads 2))