Skip to content

Add skill: migrating-dagster-to-airflow#243

Merged
jlaneve merged 2 commits into
mainfrom
add-migrating-dagster-to-airflow
Jul 13, 2026
Merged

Add skill: migrating-dagster-to-airflow#243
jlaneve merged 2 commits into
mainfrom
add-migrating-dagster-to-airflow

Conversation

@jlaneve

@jlaneve jlaneve commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What

A skill for migrating Dagster projects to Apache Airflow 3 on Astro. It includes:

  • A concept map covering how every Dagster feature translates to Airflow 3: assets, partitions, schedules, sensors, automation, IO managers, resources, ops/jobs, dbt, Pipes, Components, and Dagster+ config
  • Reference docs for the parts that are genuinely hard to translate
  • A step-by-step workflow: scan the repo (read-only), plan, migrate piece by piece, check each piece (does it parse, does it run, does it produce the same data as before), then cut over gradually with rollback one step away
  • Three small Python helpers: one lists everything in the Dagster repo so nothing gets missed, one checks the migrated DAGs, and one tracks progress so a migration can't be called done while anything is unaccounted for. They ship with their own tests, wired into CI in this PR.

How it's been tested

Seven full migrations before this PR: Dagster's OSS examples, two community projects (253 and 296 definitions), Dagster Labs' own open-source data platform, a synthetic project that uses every Dagster feature, and one run against a real production Snowflake. Wherever a project could actually run, the migrated pipelines produced the same output data as the original, checked table by table (93/93, 16/16, 13/13, 5/5), and we rehearsed a live cutover and rollback. The docs were also checked against the official Airflow, Dagster, and Cosmos documentation and put through three review passes; the troubleshooting file lists 18 real failures we hit along the way and how to fix each one.

Fit with this repo

  • Named and laid out like the other skills (it sits next to migrating-airflow-2-to-3), with the same style of edit hook, easy to drop if you'd rather not have it
  • Built not to go stale: the repo scanner asks Dagster itself what's in a project rather than keeping its own list of Dagster APIs, and anything version-dependent says which version it was checked on (Airflow 3.3 / Astro Runtime 3.3-2 / Cosmos 1.15 / Dagster 1.13)
  • scripts/ follows the analyzing-data pattern; the CI job added here is the only change outside the skill folder

🤖 Generated with Claude Code

https://claude.ai/code/session_01GGYcYRpwF7vMhgSSG7v3F8

A skill for migrating Dagster projects to Apache Airflow 3 on Astro:
concept map, deep-dive references, six-gate validation ladder, and
stdlib helper scripts with a self-contained pytest suite (wired into CI).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GGYcYRpwF7vMhgSSG7v3F8
@jlaneve
jlaneve requested a review from a team as a code owner July 13, 2026 12:47
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GGYcYRpwF7vMhgSSG7v3F8
@jlaneve
jlaneve merged commit 0a9f7c9 into main Jul 13, 2026
5 checks passed
@jlaneve
jlaneve deleted the add-migrating-dagster-to-airflow branch July 13, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant