dbt & Python: doldrums of DAG-debugging

[dataders]

Highlights of when building a new data pipeline: getting the local PoC working, and getting it working in production. Everything in between? Pain and suffering. This is a fundamental challenge for all data orchestration activities because moving between compute targets and frameworks can be daunting. Adapting local code to run in a remote orchestration framework and debugging why the task failed is very painful, not only because it is “hard to scrut” but also because the errors you encounter are orthogonal to the programming problem at hand.

dbt Labs (and many others) recognize this pain point and understand the opportunity in making the experience more interactive, transparent, and therefore debug-able. Possible improvements in this realm: Providing suggestions and feedback to users during the Python script development process could save us data professionals oodles of time. Being able to jump right into the node in the graph that failed and tweak it in real time.

Contribute below with any questions, comments, complaints, or ideas below!