cicd-sample-project

This is a sample project for Databricks, generated via cookiecutter.

While using this project, you need Python 3.X and pip or conda for package management.

Installing project requirements

pip install -r unit-requirements.txt

Install project package in a developer mode

pip install -e .

Testing

For local unit testing, please use pytest:

pytest tests/unit --cov

For an integration test on interactive cluster, use the following command:

dbx execute --cluster-name=<name of interactive cluster> --job=cicd-sample-project-sample-integration-test

For a test on an automated job cluster, deploy the job files and then launch:

dbx deploy --jobs=cicd-sample-project-sample-integration-test --files-only
dbx launch --job=cicd-sample-project-sample-integration-test --as-run-submit --trace

Interactive execution and development

1. dbx expects that cluster for interactive execution supports %pip and %conda magic commands.
2. Please configure your job in conf/deployment.yml file.
3. To execute the code interactively, provide either --cluster-id or --cluster-name.

dbx execute \
    --cluster-name="<some-cluster-name>" \
    --job=job-name

Multiple users also can use the same cluster for development. Libraries will be isolated per each execution context.

Preparing deployment file

Next step would be to configure your deployment objects. To make this process easy and flexible, we're using YAML for configuration.

By default, deployment configuration is stored in conf/deployment.yml.

Deployment for Run Submit API

To deploy only the files and not to override the job definitions, do the following:

dbx deploy --files-only

To launch the file-based deployment:

dbx launch --as-run-submit --trace

This type of deployment is handy for working in different branches, not to affect the main job definition.

Deployment for Run Now API

To deploy files and update the job definitions:

dbx deploy

To launch the file-based deployment:

dbx launch --job=<job-name>

This type of deployment shall be mainly used from the CI pipeline in automated way during new release.

CICD pipeline settings

Please set the following secrets or environment variables for your CI provider:

• DATABRICKS_HOST
• DATABRICKS_TOKEN

Testing and releasing via CI pipeline

• To trigger the CI pipeline, simply push your code to the repository. If CI provider is correctly set, it shall trigger the general testing pipeline
• To trigger the release pipeline, get the current version from the cicd_sample_project/__init__.py file and tag the current code version:

git tag -a v<your-project-version> -m "Release tag for version <your-project-version>"
git push origin --tags