Skip to content

DEMO.md

Latest commit

 

History

History
205 lines (161 loc) · 10.7 KB

DEMO.md

File metadata and controls

205 lines (161 loc) · 10.7 KB
Raw

data2day demo: data governance driven by the catalog

What this demo shows

The Stackable Data Platform (SDP) authorizes every Trino query through Open Policy Agent (OPA). The demo shows two ways of writing those rules, side by side, on the same dbt marts in data2day.demo:

  1. Hardcoded per group (customer_lifetime_value). The Rego policy names the columns to mask. Members of the Keycloak group /pii see clear text, everyone else sees masked values. Changing who sees what means editing a group membership, but changing what is sensitive means editing the policy.

  2. Driven by OpenMetadata (order_summary, and every other table in data2day.demo). The policy contains no table or column names. The OPA resource-info-fetcher asks OpenMetadata for each table's owner and tags:

    • No owner → no access. SELECT is denied for everybody, /admin included, until the table has an owner in OpenMetadata.
    • Tagged PII.Sensitive → masked, unless the user is in /pii.

    Data stewards govern access in the catalog, with no policy change and no deploy.

Both rules fail closed: if OpenMetadata can't be asked, access is denied and columns are masked. Only the static Trino service account admin (dbt, OpenMetadata ingestion, Apache Airflow) is exempt. The rules live in platform/manifests/opa/rego-trino-policies.yaml.

Setup

1. Deploy

just demo <cluster-name> feat/data2day    # new AKS cluster
just deploy feat/data2day                 # existing cluster

Always pass the branch, the justfile defaults to main. Allow about 15 minutes. The demo is ready when the Airflow DAG dbt_tpch_demo has finished: its task set_om_owners gives all marts except order_summary the owner team Data Engineering.

2. Users

User Keycloak groups Sees
demo-user none masked PII, no access to tables without an owner
demo-admin /admin, /pii clear text, but also no access to tables without an owner

Passwords: secrets/manifests/keycloak-manifests/keycloak-demo-passwords.yaml. Use a private window for the second user.

3. Browser

The node IP changes with every deploy:

kubectl -n platform get cm oidc-endpoints -o jsonpath='{.data.node-ip}'
  • Cockpit runs on HTTPS (https://<node-ip>:30300) with a certificate of the demo CA: accept the browser warning once, like for Trino and NiFi.
  • Bookmarks to all UIs and the demo queries in Cockpit (dashboard bookmarks in the categories Tools, Monitoring, Audit and Operations, shown as the bookmark's environment, plus the editor tabs "Demo A: hardcoded masks" and "Demo B: owner + tag"), and an S3 connection to Garage for the storage browser (key cockpit-demo: read on iceberg-warehouse, read/write on demo-files; saved per Cockpit user, so run it once per user): 
    just cockpit-bookmarks | wl-copy    # or pbcopy / xclip -sel clip
    Open Cockpit (https://<node-ip>:30300), press F12 → Console, paste, press Enter. Firefox wants allow pasting typed once first. Rerun after every redeploy, it replaces its own bookmarks and tabs and keeps yours (Cockpit allows 8 tabs; the leftmost of yours are dropped if needed).
UI Port
Cockpit (HTTPS) 30300
OpenMetadata 30585
Entropy Data (organization myorga) 30808
Grafana (metrics, OPA decisions) 30301
Keycloak admin (realm stackable-demo) 30900

Choreography

All queries run in Cockpit, logged in as demo-user unless noted.

Part A: hardcoded masking per group

  1. Query the table with hardcoded masks (editor tab "Demo A"): 
    SELECT customer_id, customer_name, account_balance, lifetime_net_revenue
FROM data2day.demo.customer_lifetime_value
ORDER BY lifetime_net_revenue DESC LIMIT 10;
    customer_name is ***MASKED***, account_balance is NULL. In OpenMetadata the columns carry no tags: these masks come from the policy itself.
  2. Keycloak → stackable-demo → Users → demo-user → Groups → Join group → pii.
  3. Wait about a minute (the group lookup is cached), rerun the query: clear text.

Part B: governance from the catalog

  1. Query a table without an owner (editor tab "Demo B"): 
    SELECT customer_name, customer_nation, net_revenue
FROM data2day.demo.order_summary
ORDER BY net_revenue DESC LIMIT 10;
    Access Denied. The table is still listed in the catalog browser; it just can't be read. Same for demo-admin.
  2. OpenMetadata → Explore → order_summary → set owner Data Engineering.
  3. Wait about 10 seconds, rerun: the query works, customer_name in clear text.
  4. OpenMetadata → order_summary → column customer_name → add tag PII.Sensitive.
  5. Wait about 10 seconds, rerun: customer_name is ***MASKED***. No policy was changed.
  6. Optional: tag net_revenue too, it comes back as NULL. As demo-admin (in /pii) both columns stay readable.

Reviewing the decisions in OpenSearch Dashboards

Every OPA decision for Trino lands in the index logs-opa-decisions-*. In OpenSearch Dashboards (http://<node-ip>:30601, log in via single sign-on as demo-admin) open Discover → saved search OPA decisions. Each document is one decision; message summarizes it, e.g. demo-user GetColumnMask data2day.demo.customer_lifetime_value 5 column mask(s).

  • query_id is Trino's query ID (Trino UI / system.runtime.queries). Filter on it to see all decisions of one query. The ExecuteQuery check itself carries no query ID.
  • groups are the Keycloak groups the User Info Fetcher returned for the user.
  • resource_info is what the Resource Info Fetcher returned from OpenMetadata: owners, has_owner, tags, column_tags. Part B's denial on order_summary shows has_owner: false.
  • masks lists column → mask expression; allowed: false marks denials.

Reviewing queries in Grafana

Grafana (http://<node-ip>:30301, Keycloak login) → folder Governance → Trino queries: every completed Trino query (Trino's HTTP event listener → index logs-trino-queries-*) with user, state, tables, the column masks Trino applied, and errors. Click a query ID for the drill-down: the query's details, its OPA decisions (with Keycloak groups and OpenMetadata owners/tags), the OPA log during the query and Trino log lines mentioning it. "open ↗" leads to the query in the Trino web UI (https://<node-ip>:30443). The dashboard OPA decisions links to the same drill-down.

Airflow authorization with OPA

Airflow's own roles no longer decide: Stackable's OPA auth manager asks OPA for every action (Rego package airflow, platform/manifests/opa/rego-airflow-policies.yaml), based on the same Keycloak groups as Trino.

  1. Log in to Airflow as demo-user (no groups): DAG dbt_tpch_demo, its runs and task logs are visible; triggering, clearing or pausing it fails, Admin (connections, variables, config) is not available, XCom is hidden.
  2. Log in as demo-admin (/admin) in a private window: everything works.
  3. Grafana → Governance → Airflow authorization: the denied requests with user, rule (dag, connection, ...), method, resource and the user's Keycloak groups; permissions per user; the decision log. Airflow caches decisions for 10 s.

To show a policy change live: add demo-user to the Keycloak group /admin (Keycloak admin console), wait ~10 s (Airflow + User Info Fetcher caches), reload Airflow.

Multi-tenancy: one Airflow, two teams

DAGs belong to a team by tag (team:marketing, team:finance, set in the DAG code), users by Keycloak group (/marketing, /finance); demo-marketing and demo-finance share demo-user's password. Airflow uses a small extension of Stackable's OPA auth manager (platform/manifests/airflow/ opa-tags-auth-manager.yaml): it sends the DAG's tags to OPA and lets OPA filter the DAG list. Example DAGs: marketing_campaign_report, finance_monthly_close (three short tasks each, manual trigger only).

  1. demo-marketing: the DAG list shows marketing_campaign_report and dbt_tpch_demo — the finance DAG is not there at all. Trigger the marketing DAG: works. dbt_tpch_demo: read-only.
  2. demo-finance (private window): sees finance_monthly_close instead, can run it.
  3. demo-user (no team): sees all DAGs, read-only. demo-admin: everything.
  4. Grafana → Airflow authorization → DAG list: hidden DAGs per user: OPA kept finance_monthly_close out of demo-marketing's list; Team DAGs: access per user and team shows the allowed and denied checks per team.

Moving a user between teams is a Keycloak group change, effective after ~10 s.

Same marts, built by SQLMesh

The Airflow DAG sqlmesh_tpch_demo builds the same 8 staging views and 6 marts with SQLMesh (dags/sqlmesh/tpch_demo, models ported 1:1 from the dbt project) into data2day.demo_sqlmesh. Differences to the dbt run:

  • One task instead of one pod per model. sqlmesh plan applies model changes, sqlmesh run refreshes marts whose (daily) interval is due.
  • Virtual layer. data2day.demo_sqlmesh holds only views. The tables behind them are versioned in data2day.sqlmesh__demo_sqlmesh; a changed model gets a new table and the view is switched over. Only admin can read that schema, so the masks on the views can't be bypassed.
  • Audits on each mart's grain (not null, unique) fail the plan if the data is wrong.
  • State in PostgreSQL (postgresql-sqlmesh in shared), Trino can't hold it.
  • OpenMetadata has no SQLMesh integration. The views show up through the Trino metadata ingestion, with the model and column descriptions (SQLMesh writes them as Trino comments), but without the model lineage the dbt ingestion adds.

The OPA rules are the same as for data2day.demo: owned marts readable, order_summary denied until it has an owner, customer_lifetime_value masked per group, other columns masked by tag.

Reset after a rehearsal

  • Keycloak: remove demo-user from /pii.
  • OpenMetadata, order_summary: remove the owner and the PII.Sensitive tags.

If something is off

  • Everything in data2day.demo is denied, even owned tables: the resource-info-fetcher can't reach OpenMetadata. Check that Secret platform/resource-info-fetcher-credentials holds a JWT (starts with eyJ, written by Job configure-openmetadata-v4), not placeholder.
  • The DAG hangs in wait_for_services: check trino-init. It needs an active Trino worker.
  • OpenMetadata answers 431: too many cookies for the node IP; clear them or use a private window. (The header limit is raised to 64 KiB, this should be rare.)
  • Avoid in Q&A: tpch.tiny.customer is readable unmasked. Only data2day.demo and data2day.demo_sqlmesh are governed.