Priority Queues

Jobs differ from one-another in terms of:

• Rate of ingestion
• Time of execution
• Failure rates

To handle different types of jobs, Goose allows configuration of custom queues. By default, all jobs go to default queue. To change this, use :queue option.

Nuances

• A single worker process only executes job from 1 queue
  • Spawn multiple worker processes to execute jobs from multiple queues
• Custom queue name cannot be scheduled-jobs or dead-jobs as Goose uses them internally.

Usage

(ns my-ns
  (:require [goose.client :as c]
            [goose.worker :as w]))

(def high-priority-opts (assoc c/default-opts :queue "high"))
(c/perform-async high-priority-opts `high-priority-fn :foo)

(def low-priority-opts (assoc c/default-opts :queue "low"))
(c/perform-async low-priority-opts `low-priority-fn :foo)

(w/start (assoc w/default-opts :queue "high" :threads 10))
(w/start (assoc w/default-opts :queue "low" :threads 2))