PIP additional requirements throws Docker error

[jrlegrand]

Problem Statement

Right now, we have CMS Part D DAG in a "hidden_dags" folder because it was causing an error for me and at least one other person when trying to docker-compose up airflow-init.

Also - I have another branch with a MEPS DAG (jrlegrand/meps) that I haven't merged because it also has a dependency on a PIP package.

So figuring out the way to handle this error would unlock two very useful DAGs.

Assuming the answer is to build a custom image like the error message says... https://airflow.apache.org/docs/docker-stack/build.html

airflow-init  | !!!!!  Installing additional requirements: 'zipfile-deflate64' !!!!!!!!!!!!
airflow-init  |
airflow-init  | WARNING: This is a development/test feature only. NEVER use it in production!
airflow-init  |          Instead, build a custom image as described in
airflow-init  |
airflow-init  |          https://airflow.apache.org/docs/docker-stack/build.html

Criteria for Success

Figure out a way to safely and correctly handle PIP dependencies.

Additional Information

This is what I had to do to fix the error (basically crippling the DAG that depended on this PIP package).

This is the Slack convo with Adam G.
https://coderx.slack.com/archives/C05S27E52N8/p1703102061867119

Full error log

</details>
C:\Dev\sagerx>docker-compose up airflow-init
time="2023-12-20T15:21:08-05:00" level=warning msg="The \"UMLS_API\" variable is not set. Defaulting to a blank string."time="2023-12-20T15:21:08-05:00" level=warning msg="The \"UMLS_API\" variable is not set. Defaulting to a blank string."time="2023-12-20T15:21:08-05:00" level=warning msg="The \"UMLS_API\" variable is not set. Defaulting to a blank string."time="2023-12-20T15:21:08-05:00" level=warning msg="The \"UMLS_API\" variable is not set. Defaulting to a blank string."[+] Running 2/0
 ✔ Container postgres      Running                                                                                 0.0s
 ✔ Container airflow-init  Created                                                                                 0.0s
Attaching to airflow-init, postgres
airflow-init  | The container is run as root user. For security, consider using a regular user account.
airflow-init  |
airflow-init  |
airflow-init  | /home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
airflow-init  | DB: postgresql+psycopg2://airflow:***@postgres:5432/airflow
airflow-init  | Performing upgrade with database postgresql+psycopg2://airflow:***@postgres:5432/airflow
airflow-init  | [2023-12-20 20:21:10,566] {migration.py:205} INFO - Context impl PostgresqlImpl.
airflow-init  | [2023-12-20 20:21:10,567] {migration.py:212} INFO - Will assume transactional DDL.
airflow-init  | [2023-12-20 20:21:10,572] {db.py:1571} INFO - Creating tables
airflow-init  | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
airflow-init  | INFO  [alembic.runtime.migration] Will assume transactional DDL.
airflow-init  | Upgrades done
airflow-init  | [2023-12-20 20:21:13,342] {providers_manager.py:238} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package
airflow-init  | [2023-12-20 20:21:13,666] {providers_manager.py:238} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package
airflow-init  | airflow already exist in the db
airflow-init  |
airflow-init  | !!!!!  Installing additional requirements: 'zipfile-deflate64' !!!!!!!!!!!!
airflow-init  |
airflow-init  | WARNING: This is a development/test feature only. NEVER use it in production!
airflow-init  |          Instead, build a custom image as described in
airflow-init  |
airflow-init  |          https://airflow.apache.org/docs/docker-stack/build.html
airflow-init  |
airflow-init  |          Adding requirements at container startup is fragile and is done every time
airflow-init  |          the container starts, so it is onlny useful for testing and trying out
airflow-init  |          of adding dependencies.
airflow-init  |
airflow-init  |
airflow-init  |
airflow-init  | You are running pip as root. Please use 'airflow' user to run pip!
airflow-init  |
airflow-init  | See: https://airflow.apache.org/docs/docker-stack/build.html#adding-a-new-pypi-package
airflow-init  |
airflow-init  |
airflow-init exited with code 1

[RiversPharmD]

@jrlegrand you already have a custom docker image, adding the dbt-postgres adapter. I'll open a branch and see if I can get it running for you