Skip to content

Commit 6e9dba3

Browse files
adwk67NickLarsenNZ
andauthored
docs: Add explanation of token usage for logins, API, DAGs etc. (#889)
* add explanation of token usage for logins, API, DAGs etc. * remove clause * Update docs/modules/airflow/pages/usage-guide/dag-run-information.adoc Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> * Update docs/modules/airflow/pages/usage-guide/security.adoc Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> * add admonition --------- Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com>
1 parent 157780c commit 6e9dba3

3 files changed

Lines changed: 105 additions & 0 deletions

File tree

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
= Reading DAG run information in Airflow 3
2+
:description: Replace direct metadata database access in DAGs with the Airflow REST API.
3+
:airflow-upgrade-docs: https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html
4+
5+
In Airflow 3, task code can no longer access the Airflow metadata database directly, so queries such as `session.query(DagRun)` no longer work.
6+
See the {airflow-upgrade-docs}[Airflow 3 upgrade guide{external-link-icon}^] for background.
7+
8+
DAGs that monitor other DAGs can use the REST API instead, authenticating as a technical user as described in xref:usage-guide/security.adoc#api-access[REST API access]:
9+
10+
[source,python]
11+
----
12+
import os
13+
14+
import pendulum
15+
import requests
16+
from airflow.sdk import dag, task
17+
18+
AIRFLOW_URL = "http://<cluster>-webserver:8080"
19+
20+
21+
@dag(schedule=None, start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), catchup=False)
22+
def monitor_dag_runs():
23+
@task
24+
def report(dag_id: str):
25+
response = requests.post(
26+
f"{AIRFLOW_URL}/auth/token",
27+
json={
28+
"username": os.environ["AIRFLOW_API_USERNAME"],
29+
"password": os.environ["AIRFLOW_API_PASSWORD"],
30+
},
31+
timeout=30,
32+
)
33+
response.raise_for_status()
34+
headers = {"Authorization": f"Bearer {response.json()['access_token']}"}
35+
36+
response = requests.get(
37+
f"{AIRFLOW_URL}/api/v2/dags/{dag_id}/dagRuns", headers=headers, timeout=30
38+
)
39+
response.raise_for_status()
40+
for run in response.json()["dag_runs"]:
41+
print(run["dag_run_id"], run["state"], run["start_date"], run["end_date"])
42+
43+
report("<monitored_dag_id>")
44+
45+
46+
monitor_dag_runs()
47+
----
48+
49+
The environment variables can be populated from a Secret holding the user's credentials with xref:usage-guide/overrides.adoc#_pod_overrides[Pod overrides].

‎docs/modules/airflow/pages/usage-guide/security.adoc‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
:description: Secure Apache Airflow by configuring user authentication and authorization.
33
:airflow-access-control-docs: https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/access-control.html
44
:keycloak: https://www.keycloak.org/
5+
:airflow-fab-token-docs: https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/token.html
56

67
Secure Apache Airflow by configuring user authentication and authorization.
78
Airflow provides built-in user and role management, but can also connect to an LDAP server or an OIDC provider to manage users centrally instead.
@@ -140,6 +141,60 @@ A minimum client configuration in Keycloak for this example looks like this:
140141

141142
Further information for specifying an AuthenticationClass for an OIDC provider can be found at the xref:concepts:authentication.adoc#_oidc[concepts page].
142143

144+
=== [[api-access]]REST API access
145+
146+
This section applies to Airflow 3.x.
147+
148+
The authentication methods described above (built-in users, LDAP and OpenID Connect) govern how users log in to the web interface.
149+
The REST API (`/api/v2`) authenticates requests differently: it accepts only JSON Web Tokens (JWTs) issued by Airflow itself.
150+
A client obtains such a token by sending a username and password to the `/auth/token` endpoint of the webserver:
151+
152+
[source,bash]
153+
----
154+
ACCESS_TOKEN=$(
155+
curl -s -X POST http://<cluster>-webserver:8080/auth/token \
156+
-H 'Content-Type: application/json' \
157+
-d '{"username": "<user>", "password": "<password>"}' | jq -r '.access_token'
158+
)
159+
curl -s -H "Authorization: Bearer $ACCESS_TOKEN" \
160+
http://<cluster>-webserver:8080/api/v2/dags/<dag_id>/dagRuns
161+
----
162+
163+
The `/auth/token` endpoint checks the credentials against Airflow's internal user database (and against the LDAP server if LDAP authentication is configured).
164+
See the {airflow-fab-token-docs}[Airflow documentation on generating tokens{external-link-icon}^] for details.
165+
166+
IMPORTANT: With OpenID Connect, Keycloak is used for the interactive login to the web interface only.
167+
Tokens issued by Keycloak (for example via the client credentials grant of a service account) are rejected by the REST API with `Invalid JWT token`.
168+
Users created through a Keycloak login have no usable password in Airflow, so their credentials are rejected by `/auth/token` with `Invalid credentials`.
169+
170+
NOTE: Airflow allows tokens to be obtained for OAuth users by overriding the `create_token` method of the FAB auth manager, as described in the {airflow-fab-token-docs}[Airflow documentation on generating tokens{external-link-icon}^].
171+
The Stackable Data Platform does not provide or test such an override.
172+
It requires a custom auth manager in a custom image, and that code is then responsible for validating the token from the identity provider.
173+
174+
IMPORTANT: The example in the Airflow documentation disables signature verification and (as also stated there) should not be used in production.
175+
176+
==== Service-to-service access
177+
178+
For API clients such as scripts, external services or DAGs, create a dedicated user in Airflow's internal user database.
179+
This works regardless of the authentication method configured for the web interface.
180+
Assign the least privileged role that covers the required API calls; for example, the `Viewer` role allows reading DAGs and DAG runs, but not modifying them:
181+
182+
[source,bash]
183+
----
184+
kubectl exec <cluster>-webserver-<rolegroup>-0 -c airflow -- \
185+
airflow users create \
186+
--username <technical-user> \
187+
--password <password> \
188+
--role Viewer \
189+
--firstname <first-name> \
190+
--lastname <last-name> \
191+
--email <email>
192+
----
193+
194+
Store the credentials in a Kubernetes Secret and make them available to the client.
195+
Request a new token for each run rather than storing one, as tokens expire.
196+
See xref:usage-guide/dag-run-information.adoc[] for an example of a DAG that reads DAG run information via the REST API.
197+
143198
== Authorization
144199

145200
The Airflow Webserver delegates the {airflow-access-control-docs}[handling of user access control{external-link-icon}^] to the https://flask-appbuilder.readthedocs.io/en/latest/security.html[Flask AppBuilder{external-link-icon}^].

‎docs/modules/airflow/partials/nav.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
** xref:airflow:usage-guide/reverse-proxy.adoc[]
1212
** xref:airflow:usage-guide/storage-resources.adoc[]
1313
** xref:airflow:usage-guide/security.adoc[]
14+
** xref:airflow:usage-guide/dag-run-information.adoc[]
1415
** xref:airflow:usage-guide/logging.adoc[]
1516
** xref:airflow:usage-guide/monitoring.adoc[]
1617
** xref:airflow:usage-guide/openlineage.adoc[]

0 commit comments

Comments
 (0)