Skip to content

Conversation

@thekaveman
Copy link
Member

Description

Caused by a bug in our Docker build process, see cal-itp/benefits#3314 for the fix at the source.

This PR adjusts versions for historical events going back to August 2025, when the bug was introduced.

Resolves cal-itp/benefits#3319

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Include commands/logs/screenshots as relevant.

If making changes to dbt models, please run the command poetry run dbt run -s CHANGED_MODEL and poetry run dbt test -s CHANGED_MODEL, then include the output in this section of the PR.

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

  • No action required
  • Actions required (specified below)

@github-actions
Copy link

github-actions bot commented Nov 18, 2025

Terraform plan in iac/cal-itp-data-infra-staging/composer/us

Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_composer_environment.calitp-staging-composer will be updated in-place
!~  resource "google_composer_environment" "calitp-staging-composer" {
        id               = "projects/cal-itp-data-infra-staging/locations/us-west2/environments/calitp-staging-composer"
        name             = "calitp-staging-composer"
#        (5 unchanged attributes hidden)

!~      config {
#            (8 unchanged attributes hidden)

!~          software_config {
!~              pypi_packages            = {
+                   "calitp-data-infra"      = "==2025.6.5"
+                   "pydantic"               = ">=1.9,<2.0"
#                    (11 unchanged elements hidden)
                }
#                (6 unchanged attributes hidden)

#                (1 unchanged block hidden)
            }

#            (8 unchanged blocks hidden)
        }

#        (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1074

@github-actions
Copy link

github-actions bot commented Nov 18, 2025

Terraform plan in iac/cal-itp-data-infra/composer/us

No changes. Your infrastructure matches the configuration.
No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1074

@thekaveman thekaveman force-pushed the fix/benefits-versions branch from b13a4b8 to 42aa933 Compare November 18, 2025 23:40
@github-actions
Copy link

github-actions bot commented Nov 18, 2025

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 18 to change, 3 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place
-   destroy

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/airtable_loader_v2/generate_gtfs_download_configs.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "MijWlA==" -> (known after apply)
!~      detect_md5hash      = "s632w01yc8uo408y4VdAyw==" -> "different hash"
!~      generation          = 1763690417635115 -> (known after apply)
        id                  = "calitp-staging-composer-dags/airtable_loader_v2/generate_gtfs_download_configs.py"
!~      md5hash             = "s632w01yc8uo408y4VdAyw==" -> (known after apply)
        name                = "dags/airtable_loader_v2/generate_gtfs_download_configs.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["dags/download_gtfs_schedule_v2/download_schedule_feeds.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "8qLecA==" -> (known after apply)
!~      detect_md5hash      = "iGapm0xJ3U0wowUUkId1eQ==" -> "different hash"
!~      generation          = 1763690417218909 -> (known after apply)
        id                  = "calitp-staging-composer-dags/download_gtfs_schedule_v2/download_schedule_feeds.py"
!~      md5hash             = "iGapm0xJ3U0wowUUkId1eQ==" -> (known after apply)
        name                = "dags/download_gtfs_schedule_v2/download_schedule_feeds.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "g2TBuw==" -> (known after apply)
!~      detect_md5hash      = "PSes9rK7j0FP6JcNRQhPLg==" -> "different hash"
!~      generation          = 1763690417804807 -> (known after apply)
        id                  = "calitp-staging-composer-dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"
!~      md5hash             = "PSes9rK7j0FP6JcNRQhPLg==" -> (known after apply)
        name                = "dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/calitp_data_infra/__init__.py"] will be destroyed
  # (because key ["plugins/calitp_data_infra/__init__.py"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "AAAAAA==" -> null
-       detect_md5hash      = "1B2M2Y8AsgTpgAmY7PhCfg==" -> null
-       event_based_hold    = false -> null
-       generation          = 1763690417199832 -> null
-       id                  = "calitp-staging-composer-plugins/calitp_data_infra/__init__.py" -> null
-       md5hash             = "1B2M2Y8AsgTpgAmY7PhCfg==" -> null
-       md5hexhash          = "d41d8cd98f00b204e9800998ecf8427e" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2F__init__.py?generation=1763690417199832&alt=media" -> null
-       metadata            = {} -> null
-       name                = "plugins/calitp_data_infra/__init__.py" -> null
-       output_name         = "plugins/calitp_data_infra/__init__.py" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2F__init__.py" -> null
-       source              = "../../../../airflow/plugins/calitp_data_infra/__init__.py" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/calitp_data_infra/auth.py"] will be destroyed
  # (because key ["plugins/calitp_data_infra/auth.py"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "6lsUtA==" -> null
-       detect_md5hash      = "+/KTbwc3sd3B4wBkY+HoUw==" -> null
-       event_based_hold    = false -> null
-       generation          = 1763690417207979 -> null
-       id                  = "calitp-staging-composer-plugins/calitp_data_infra/auth.py" -> null
-       md5hash             = "+/KTbwc3sd3B4wBkY+HoUw==" -> null
-       md5hexhash          = "fbf2936f0737b1ddc1e3006463e1e853" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fauth.py?generation=1763690417207979&alt=media" -> null
-       metadata            = {} -> null
-       name                = "plugins/calitp_data_infra/auth.py" -> null
-       output_name         = "plugins/calitp_data_infra/auth.py" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fauth.py" -> null
-       source              = "../../../../airflow/plugins/calitp_data_infra/auth.py" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/calitp_data_infra/storage.py"] will be destroyed
  # (because key ["plugins/calitp_data_infra/storage.py"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "b87VYA==" -> null
-       detect_md5hash      = "sq1Q+wmsL8o0RKJLFUFC7g==" -> null
-       event_based_hold    = false -> null
-       generation          = 1763690417395966 -> null
-       id                  = "calitp-staging-composer-plugins/calitp_data_infra/storage.py" -> null
-       md5hash             = "sq1Q+wmsL8o0RKJLFUFC7g==" -> null
-       md5hexhash          = "b2ad50fb09ac2fca3444a24b154142ee" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fstorage.py?generation=1763690417395966&alt=media" -> null
-       metadata            = {} -> null
-       name                = "plugins/calitp_data_infra/storage.py" -> null
-       output_name         = "plugins/calitp_data_infra/storage.py" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fstorage.py" -> null
-       source              = "../../../../airflow/plugins/calitp_data_infra/storage.py" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/hooks/kuba_hook.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "pIf6jA==" -> (known after apply)
!~      detect_md5hash      = "M9n0Cr7dL9+4asfxnMHgjQ==" -> "different hash"
!~      generation          = 1763690418428084 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/hooks/kuba_hook.py"
!~      md5hash             = "M9n0Cr7dL9+4asfxnMHgjQ==" -> (known after apply)
        name                = "plugins/hooks/kuba_hook.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/hooks/soda_hook.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "Wxj+aQ==" -> (known after apply)
!~      detect_md5hash      = "CDAoj9pONq2nsD5BT43vEg==" -> "different hash"
!~      generation          = 1763690418284709 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/hooks/soda_hook.py"
!~      md5hash             = "CDAoj9pONq2nsD5BT43vEg==" -> (known after apply)
        name                = "plugins/hooks/soda_hook.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/hooks/transitland_hook.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "xolZ/w==" -> (known after apply)
!~      detect_md5hash      = "lMSV7OyTWTBE5ar8Ush1oA==" -> "different hash"
!~      generation          = 1763690417210931 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/hooks/transitland_hook.py"
!~      md5hash             = "lMSV7OyTWTBE5ar8Ush1oA==" -> (known after apply)
        name                = "plugins/hooks/transitland_hook.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/aggregator_to_gcs_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "h26Utg==" -> (known after apply)
!~      detect_md5hash      = "+2mzXgNPW36mWf5Z3ZZBag==" -> "different hash"
!~      generation          = 1763690417144134 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/aggregator_to_gcs_operator.py"
!~      md5hash             = "+2mzXgNPW36mWf5Z3ZZBag==" -> (known after apply)
        name                = "plugins/operators/aggregator_to_gcs_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/blackcat_to_gcs_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "s7BJ7Q==" -> (known after apply)
!~      detect_md5hash      = "husvfrVLOwWESpUczaAT3w==" -> "different hash"
!~      generation          = 1763690417861255 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/blackcat_to_gcs_operator.py"
!~      md5hash             = "husvfrVLOwWESpUczaAT3w==" -> (known after apply)
        name                = "plugins/operators/blackcat_to_gcs_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/dbt_manifest_to_dictionary_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "4MyK1Q==" -> (known after apply)
!~      detect_md5hash      = "FIHwYcjOm5NPB+nPb7enyg==" -> "different hash"
!~      generation          = 1763690416913428 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/dbt_manifest_to_dictionary_operator.py"
!~      md5hash             = "FIHwYcjOm5NPB+nPb7enyg==" -> (known after apply)
        name                = "plugins/operators/dbt_manifest_to_dictionary_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/dbt_manifest_to_metadata_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "UDX9og==" -> (known after apply)
!~      detect_md5hash      = "F4n9FAx9ExF1Zl3J7NaMWQ==" -> "different hash"
!~      generation          = 1763690417995295 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/dbt_manifest_to_metadata_operator.py"
!~      md5hash             = "F4n9FAx9ExF1Zl3J7NaMWQ==" -> (known after apply)
        name                = "plugins/operators/dbt_manifest_to_metadata_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/gtfs_csv_to_jsonl_hourly.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "pCzW9g==" -> (known after apply)
!~      detect_md5hash      = "n34Iv1u6y/wZeOnqO2WDEA==" -> "different hash"
!~      generation          = 1763690417647780 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/gtfs_csv_to_jsonl_hourly.py"
!~      md5hash             = "n34Iv1u6y/wZeOnqO2WDEA==" -> (known after apply)
        name                = "plugins/operators/gtfs_csv_to_jsonl_hourly.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/littlepay_raw_sync_feed_v3.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "TDDjLg==" -> (known after apply)
!~      detect_md5hash      = "zXyVjenO8Z0Xcx4u85EVYQ==" -> "different hash"
!~      generation          = 1763690417211071 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/littlepay_raw_sync_feed_v3.py"
!~      md5hash             = "zXyVjenO8Z0Xcx4u85EVYQ==" -> (known after apply)
        name                = "plugins/operators/littlepay_raw_sync_feed_v3.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/pod_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "H81Llg==" -> (known after apply)
!~      detect_md5hash      = "6vO0LHE3p5d/cOQ71Ghv8g==" -> "different hash"
!~      generation          = 1763690417200759 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/pod_operator.py"
!~      md5hash             = "6vO0LHE3p5d/cOQ71Ghv8g==" -> (known after apply)
        name                = "plugins/operators/pod_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/scrape_ntd_xlsx.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "800p7w==" -> (known after apply)
!~      detect_md5hash      = "+sUY5347tlkwmkjx/59Ytg==" -> "different hash"
!~      generation          = 1763690417819438 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/scrape_ntd_xlsx.py"
!~      md5hash             = "+sUY5347tlkwmkjx/59Ytg==" -> (known after apply)
        name                = "plugins/operators/scrape_ntd_xlsx.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/scrape_state_geoportal.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "HSTs0g==" -> (known after apply)
!~      detect_md5hash      = "kroDAzyyod9g32UxYWccew==" -> "different hash"
!~      generation          = 1763690418239800 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/scrape_state_geoportal.py"
!~      md5hash             = "kroDAzyyod9g32UxYWccew==" -> (known after apply)
        name                = "plugins/operators/scrape_state_geoportal.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/scripts/gtfs_rt_parser.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "XjBzGA==" -> (known after apply)
!~      detect_md5hash      = "BfvE2ctBSfWnSYBpIv2hOg==" -> "different hash"
!~      generation          = 1763690417837883 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/scripts/gtfs_rt_parser.py"
!~      md5hash             = "BfvE2ctBSfWnSYBpIv2hOg==" -> (known after apply)
        name                = "plugins/scripts/gtfs_rt_parser.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/utils.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "2wseZw==" -> (known after apply)
!~      detect_md5hash      = "L7u5luMr8c8OXJVHR7nzEg==" -> "different hash"
!~      generation          = 1763690417826500 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/utils.py"
!~      md5hash             = "L7u5luMr8c8OXJVHR7nzEg==" -> (known after apply)
        name                = "plugins/utils.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/benefits/fct_benefits_events.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "AmoX1g==" -> (known after apply)
!~      detect_md5hash      = "32e2YxZ8YVVpaVcbMfrV+w==" -> "different hash"
!~      generation          = 1762380824121196 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/mart/benefits/fct_benefits_events.sql"
!~      md5hash             = "32e2YxZ8YVVpaVcbMfrV+w==" -> (known after apply)
        name                = "data/warehouse/models/mart/benefits/fct_benefits_events.sql"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 18 to change, 3 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1074

@github-actions
Copy link

github-actions bot commented Nov 18, 2025

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer-dags["models/mart/benefits/fct_benefits_events.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "AmoX1g==" -> (known after apply)
!~      detect_md5hash      = "32e2YxZ8YVVpaVcbMfrV+w==" -> "different hash"
!~      generation          = 1762459059031940 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/mart/benefits/fct_benefits_events.sql"
!~      md5hash             = "32e2YxZ8YVVpaVcbMfrV+w==" -> (known after apply)
        name                = "data/warehouse/models/mart/benefits/fct_benefits_events.sql"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1074

@github-actions
Copy link

github-actions bot commented Nov 18, 2025

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

caused by a bug in the Docker build process
@thekaveman thekaveman force-pushed the fix/benefits-versions branch from c796186 to 4ac75b5 Compare November 21, 2025 23:59
@thekaveman
Copy link
Member Author

I'm pretty sure the DBT error is unrelated to these changes, it is referencing a different table:

Database Error in model dim_schedule_feeds (models/mart/gtfs/dim_schedule_feeds.sql)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update incorrect version numbers in analytics

2 participants