Free software: GNU Affero General Public License v3+
poetry add celery-aio-pool
Using pip
& PyPI.org
pip install celery-aio-pool
Using pip
& GitHub
pip install git+https://github.com/the-wondersmith/celery-aio-pool.git
git clone https://github.com/the-wondersmith/celery-aio-pool.git
cd celery-aio-pool
pip install -e "$(pwd)"
Celery now supports the configuration of out-of-tree worker pool classes, allowing you to configure custom pools like celery-aio-pool
directly:
-
Set the environment variable
CELERY_CUSTOM_WORKER_POOL
to the name of your desired worker pool implementation.- NOTE: The value must be formatted as
package:class
-
export CELERY_CUSTOM_WORKER_POOL='celery_aio_pool.pool:AsyncIOPool'
- NOTE: The value must be formatted as
-
Start Celery with
--pool=custom
to use the configured pool:-
celery worker --pool=custom --loglevel=INFO --logfile="$(pwd)/worker.log"
-
To verify the pool configuration, use celery inspect stats
:
celery --app=your_celery_project inspect stats
-> celery@freenas: OK
{
...
"pool": {
...
"implementation": "celery_aio_pool.pool:AsyncIOPool",
...
- Import
celery_aio_pool
in the same module where your Celery "app" is defined - Ensure that the
patch_celery_tracer
utility is called before any other Celery code is called
"""My super awesome Celery app."""
# ...
from celery import Celery
# add the following import
import celery_aio_pool as aio_pool
# ensure the patcher is called *before*
# your Celery app is defined
assert aio_pool.patch_celery_tracer() is True
app = Celery(
"my-super-awesome-celery-app",
broker="amqp://guest@localhost//",
# add the following keyword argument
worker_pool=aio_pool.pool.AsyncIOPool,
)
NOTE: Our preferred packaging and dependency manager is Poetry. Installation instructions can be found here.
Clone the repo and install the dependencies
$ git clone https://github.com/the-wondersmith/celery-aio-pool.git \
&& cd celery-aio-pool \
&& poetry install --sync
Alternatively, if you prefer not to use Poetry, celery-aio-pool
is fully PEP-517
compliant and can be installed directly by any PEP-517-compliant package manager.
$ cd celery-aio-pool \
&& pip install -e "$(pwd)"
TODO: Coming Soon™
To run the test suite:
$ poetry run pytest tests/
TODO: Coming Soon™