Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global hook to determine if Workers are allowed to fetch new jobs from Redis for execution #1103

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Commits on Aug 19, 2017

  1. Global hook to determine if Workers are allowed to fetch new jobs fro…

    …m Redis for execution. Hook function can be set up with kue.createQueue (through the options object):
    
    let kuePaused = true;
    const kuePostponeJobsPickupForExecutionStrategy = function(jobType: string) {
        const pause = kuePaused || memoryUsageSummaryMB >= (MEMORY_AVAILABLE_MB - 60);
        console.log(
            'Kue strategy to postpone new jobs pickup for execution, if memory is low.',
            'Memory usage (MB)', memoryUsageSummaryMB, 'out of', MEMORY_AVAILABLE_MB,
            pause ? 'KUE PAUSED' : ''
        );
        return pause ? 5000 : 0;
    };
    
    const q = kue.createQueue({
    ...
        postponeWorker: kuePostponeJobsPickupForExecutionStrategy,
    ...
    });
    
    Hook function should return a number: 0 if fetching a job is allowed immediately (with zero delay), or a positive integer representing a timeout in milliseconds to postpone this Worker's attempts to fetch new jobs for execution by this timeout. After the timeout, Worker is going to make an attempt to fetch a new job again, and will request postponeWorker hook function again, to determine if this time it is allowed to pick up a job for execution, or should delay again. And so on and on.
    
    Possible use case for this is to globally prevent Kue from uncontrollably grabbing new jobs for execution (for all job types or for a certain job type), if JS node is hitting upper memory limit, or if it's too much CPU used. With this feature, adaptive dynamic balancers can be implemented for better control over stability and scalability of Kue-based applications between many nodes (there is a good usage for this feature for single-noded applications as well, to control if the application is staying within host's quotas).
    max-loginov committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    112b094 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2018

  1. Merge pull request #1 from Automattic/master

    Update
    maxloginov authored Jun 6, 2018
    Configuration menu
    Copy the full SHA
    10567d2 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'master' into feature_postpone_worker

    * master:
      Trying to clarify the wording around pausing.
      Update .travis.yml - no need v4 and v7
      Update .travis.yaml - cover node v8 as well
      Update .travis.yaml - test platforms by node LTS
    max-loginov committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    623de37 View commit details
    Browse the repository at this point in the history