Cilantro is a task runner designed to manage long running distributed jobs that operate on file system objects. Its backend is written in Python (3.6+) and uses Celery and Flask. Its frontend is written in JavaScript and uses VueJS.
- Docker Community Edition
- NodeJS
Since the following process relies on make and uses bash related commands it can't be used in a Windows-Shell. The easiest way to circumvent this problem is to make use of the WSL-System of Windows 10 Pro. If you don't have Windows 10 Pro, go get Windows 10 Pro, also Docker dosn't work flawlessly without it.
Expecting that you have Win10 Pro, you need to activate the Linux Subsystem Feature and Install Ubuntu 18.04 or what ever Version is available by the time you read this. If your Ubuntu-Subsystem is up and running, open it and start installing the requierments. Begin with nodejs and pip3 for python
sudo apt update
sudo apt install nodejs python3-pip
After that start following this instruction very carefully, since you are going to setup a linux in a virtual environment communication with a docker in another environment running an linux serving an Webserver there are a lot of possibilities of mistakes.
https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly
After you done with that you are good to go on with the rest of this instructions.
Also, if you are developing with VSCode, there is a WSL Extension for that IDE available which works very well, gives you the possibilitie to use the linux subsystem from within VSCode, see: https://code.visualstudio.com/docs/remote/wsl
Run this command after first checking out the code:
make init
Then generate an access token at https://github.com/settings/tokens/new (you don't need to tick any of the scope boxes), and add the token to your local .env file.
Edit the .env file and replace the UID placeholders with your UID and GID. You can get these with the command id -u && id -g
Run all docker containers with backend services:
make run
Start the frontend locally:
make run-frontend
Code changes should be immediatly reflected in the corresponding containers. The frontend can be accessed under http://localhost:8080/.
To stop the application run:
make stop
If you have old container relics and encounter errors run:
docker-compose build
For now users are managed via the file config/users.yml. Every user is able to
login and start new jobs. Passwords in this file are encrypted with bcrypt.
You can generate a bcryt-hash here.
After changing the config/users.yml the service has currently to be restarted in order to reload the file.
The documentation is automatically generated when a commit is pushed to the
master branch.
The documentation files are held on the special branch gh-pages.
The generated HTML can be viewed via the following URL:
https://dainst.github.io/cilantro/
For the in-house CI Jenkins, the generated docu can be found under:
http://oneeyedjacks02.dai-cloud.uni-koeln.de/cilantro-docu/index.html
Dockerfiles for the different services and their dependencies are stored in
the subdirectory docker/. The complete stack defined for different
environments is configured with docker-compose files.
To build and publish the images follow the instructions provided in the docker README.
Published docker images can be found at dockerhub.
To start all containers and run all tests call:
make test
When the application is started with make run backend tests can be run
separately with:
make test-backend
Analogous frontend tests can be started with
make test-frontend
Similarly end-to-end tests that test the whole application with protractor can be run with:
make test-e2e
Single tests can be run with:
docker exec cilantro_test python -m unittest module.path.to.TestCase
e.g.
docker exec cilantro_test python -m unittest test.unit.worker.convert.test_cut_pdf.CutPdfTest
To add or update a python dependency in one of the images, you can first add or update the pyproject.toml file for the service in docker/<service/.
Then generate a fitting poetry.lock file with a run command:
docker-compose run --rm --entrypoint '' -u root -w "/poetry" -v "$(pwd)/docker/cilantro-nlp-worker:/poetry" nlp-worker poetry update(Change nlp-worker in both the volume definition and the container name to the service you need.)
This will use a temporary container to change the poetry.lock in the repo. You can now do a docker-compose build to update your container.
Change poetry update to e.g. poetry add <dependency> to change both the pyproject.toml as well as the poetry.lock.
- change promisesDelay-attribute in `frontend/test/e2e/protractor.conf to slow tests down if you wanna watch them (eg to 150)
Flower is included in the docker config and is available for debugging under http://localhost:5555.
OJS monitoring under http://localhost:4444
If you have checked out omp-docker and want to develop against its running instance (instead of the default mock server), you can do so by using the provided docker-compose.local_omp.yml by running make run-local-omp.
You need to setup your credentials according to https://github.com/dainst/ojs-cilantro-plugin.
If you have checked out ojs-docker and want to develop against its running instance (instead of the default mock server), you can do so by using the provided docker-compose.local_ojs.yml by running make run-local-ojs.
You need to setup your credentials according to https://github.com/dainst/ojs-cilantro-plugin.
New images for test or production can be build by calling the docker_image_build.sh with one of the service names in the docker-compose.build.yml (also a dir name below /docker).
You need to be logged in to DockerHub with an account that can access the /dainst organization, e.g.:
docker login
./docker_image_build.sh cilantro-convert-worker
or to build without using the cache:
./docker_image_build.sh cilantro-convert-worker no-cache
This will increment the version number of the published image (the version no. is tagged on DockerHub and kept in the VERSION file below the directories in docker).
To publish to the test/prod[portainer.idai.world] portainers:
- Adjust version numbers in
docker-compose.test.yml,docker-compose.prod.yml - Paste the contents of these files in the Portainers Stack editor (Stacks > workbench > Editor > Paste > Update the Stack)
To redeploy, you can simply remove all service and add them again:
- Portainer > Stacks > workbench > Mark all services > Remove
- Editor (top of sampe page) > Update the stack
On an empty portainer, one volume would have to be created inside the Portainer for each volume marked as external: true in the docker.compose.<env>.yml.
- Portainer > Volumes > Add volume
Most of these volumes can use an external volume configured with e.g. these options (example for the staging folder in prod):
device :/volume3/bcloud03/idaiworld-scans/idaiworkbench/staging
type nfs
o addr=10.201.0.95,rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14
However the volume with the mongo database files for the job-db service should not use an NFS volume (Keep the default options for a local volume) as that led to performance problems.
-
On Linux hosts the tests will fail because the data directory created by docker does not have the right permissions and the user account that runs the tests can not access it. The easiest way to fix that is just to change the owner on the whole directory and subfolders. The Makefile offers a short command for this:
make fix-permissions
After that re-run the tests and they may succeed.
-
In case of duplicate entries in the database clean your test containers with:
docker-compose down
-
If you run into an error using
make run-frontendon mac, runsoftwareupdate --install -ato update the whole system to the newest state. If the error remains, check if it is this Error-Code:getaddrinfo ENOTFOUND x86_64-apple-darwin13.4.0, if so use the commandunset HOSTand retry the make call. -
On Mac's on can encounter an error after running
make run-frontendof that kindgetaddrinfo ENOTFOUND x86_64-apple-darwin13.4.0
to fix this one needs to run the command
unset HOSTin the Terminal and re-run themake run-frontendcommand.
Cilantro generally uses the PEP 8 style guide.
Additionally parameters in method docstrings should be given as follows:
:param param_type param_name: parameter description
:raises ErrorType: Exception throw-condition description
- indentation: 4 spaces instead of tab
- idea: settings->editor->javascript
- atom: settings->editor
- names
- for js-variables: camelCase
- for members of datamodel (dataset, article): under_score
- in css: snake-case
- filenames and module names: under_score, eg: myController in my_controller.js
- ES6
let/constinstead ofvarwhere it makes sense: http://es6-features.org/#BlockScopedVariables- arrow function whenever function is not local and [this]-scope is not needed
- more
===instead of==- line endings with
;even after} - if without {} only in very simple one liners