This repository has been archived by the owner on Jul 24, 2024. It is now read-only.
Where to reference/install dbt packages with MWAA? #73
Comments
|
I was able to have custom modules by having the modules in the folder And finally, when using the dbt operator, I use the Hopefully you can replicate the same idea for your dbt packages. |
|
bit of a different question but related to the using Ithink the folder where the code runs in MWAA is non-writable?. see AWS MWAA docs on using I can get my dag to run locally using the following setup and code: Code:...
default_args = dict(
dbt_bin="dbt"",
profiles_dir="/opt/airflow/dags/dbt",
dir="/opt/airflow/dags/dbt"
)
with DAG(
start_date=datetime(2022, 3, 14),
schedule_interval="@once",
dag_id="dbt_test",
default_args=default_args,
tags=["dbt", "development"],
) as dag:
dbt_run = DbtRunOperator(
task_id="dbt_run_royalty",
models="+marts-report_team-royalty",
target="dev-sandervd",
)but if i run the similar setup in MWAA i get an error regarding writing logs. I use different default_args like so (required for MWAA) default_args = dict(
dbt_bin='/usr/local/airflow/.local/bin/dbt',
profiles_dir='/usr/local/airflow/dags/dbt/',
dir='/usr/local/airflow/dags/dbt/'
)I get the following error I can get it to run in MWAA if i run it in the cli_command = BashOperator(
task_id="dbt_run_mwaa",
bash_command=f"cp -R /usr/local/airflow/dags/dbt/ /tmp;\
cd /tmp/dbt;\
/usr/local/airflow/.local/bin/dbt run --project-dir /tmp/dbt/ -s incoming-royalty_etl-alibaba;\
",
) |
|
@sandervandorsten it looks like it might be an issue with your In order to get around
|
Thanks! I indeed changed my [other stuff]
...
# Changed folders that dbt writes to during runtime
# because with AWS MWAA the worker node's directory is non-writable
# # https://github.com/gocardless/airflow-dbt/issues/33
packages-install-path: "/tmp/dbt/dbt_packages"
log-path: "/tmp/dbt/logs"
target-path: "/tmp/dbt/target"
...
[other stuff] |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Where should I be installing packages (or referencing if deployed during a CI step) dbt packages while using MWAA? I'm currently installing to:
/tmp/dbt/packagesbut am experiencing about a 50/50 success rate when running the test operator. Here is what the graph view of my dags look like:This is the error log for when the runs fail:
The deps operator works great, then dbt run, but the test operator fails about 50% of the time. Should I be deploying package files to a folder within the MWAA S3 folder and then updating
profiles.ymlto reference that location?Environment context:
MWAA v2.0.2
dbt-core>=1.0
dbt-postgres>=1.0
airflow-dbt==0.4.0
The text was updated successfully, but these errors were encountered: