payclip-people-dbt: Payclip People Data Pipelines

Built with databricks and dbt.

Initial Setup

1. Install dependencies with Poetry

poetry install
poetry install --with dbt,dev

2. Create a .env file using .env.example, fill the variables accordingly and source it with:

source .env

3. Install dbt packages.

poetry run dbt deps --project-dir "$(pwd)/payclip_people" --profiles-dir "$(pwd)/payclip_people"

Environments

Targets in dbt allow you to change between environments. These are defined within the profiles.yml file.

Dev env

To develop and test in dev environment, the --target dev flag should be used in dbt commands.

Mock data is available in the seeds folder, to use it just run the dbt seed command.

First, initialize the seeds:

poetry run dbt seed --project-dir "$(pwd)/payclip_people" --profiles-dir "$(pwd)/payclip_people" --target dev

To execute the bronze example model, use the dbt run command:

poetry run dbt run --project-dir "$(pwd)/payclip_people" --profiles-dir "$(pwd)/payclip_people" --target dev --model countries