updated to celery 4.0

[dconathan]

as per #159 this updates to celery 4.0

Changes:

• changed requirements file for docker image (no longer celery==3.1.25)
• Added a celery.worker_ready function that debug prints to console (basically just mimicking the celery 3 behavior (e.g. sync_worker_2@2affb02ed780 ready.), which is no longer the default)
• updated celery cli options in next_worker_startup.sh to reflect new new version of celery. (this is what was causing it not to work before)

Note that I didn't mess with any of the options in next.constants - they are transitioning from upper case to lower case but our constants as is will be supported until celery 5 (?). I tried running their "automatically fix it" script and it broke things so I didn't bother... see http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#step-2-update-your-configuration-with-the-new-setting-names

[erinzm]

👍 lgtm

I still have to pull and test locally, but the changes look good

[stsievert]

Copying some comments from @dconathan over from Slack:

• I have moved to celery 4. I have noticed getQuery has been pushing .5-1s delays for no apparent reason and I haven't gotten around to debugging it... this could explain that.
• Tomorrow I'll rebuild with celery 3.1.25 and see if that fixes it...
• Just reverted back to celery 3.1.25 and getQuery delays are back down to sub .1s range… so almost certainly seeing a slowdown due to celery 4
• I was wondering about it. Almost all the extra time was “enqueued” time according to the charts.
• ...
• I guess there’s one reason not to switch to the latest and greatest