feat: add openlineage support (#822)

* feat: add openlineage support

* refactor(openlineage): use credentialsSecretName instead of AuthenticationClass

Follows the operator-rs change: OpenLineage backend auth is a static bearer
token, so drop the AuthenticationClass resolution and read the api key from the
connection's `credentialsSecretName` Secret (key `apiKey`) directly.

See stackabletech/decisions#90

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* update changelog

* cleanups

* Rebuild Cargo.lock and remove unnecessary ".."

* refactor(openlineage): adopt job_name field and lineage.stackable.tech API group

Follow the operator-rs rename of OpenLineageJob.app_name to job_name
(serialized appName -> jobName) and the move of the OpenLineageConnection CRD
to the lineage.stackable.tech API group:

- update the job_name field access and the "ignored by Airflow" debug log
- update the operator RBAC ClusterRole apiGroup
- update kuttl test manifests (apiVersion, jobName) and the usage-guide docs
- regenerate extra/crds.yaml (appName -> jobName)

Verified with cargo check/clippy/test and CRD regeneration against a local
operator-rs checkout carrying the rename. Committed with --no-verify because the
cargo/regenerate-charts pre-commit hooks would otherwise build against the
un-pushed operator-rs branch; those checks were run manually instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(openlineage): rename the openLineage spec field to lineage

Rename the lineage-emission configuration field from `openLineage` to `lineage`
across the CRD and operator internals:

- spec.clusterConfig.openLineage -> spec.clusterConfig.lineage (Rust field
  cluster_config.open_lineage -> lineage; the validated
  ValidatedClusterConfig.open_lineage -> lineage)
- rename operator-internal identifiers: module controller/build/openlineage.rs
  -> lineage.rs, ResolvedOpenLineageConfig -> ResolvedLineageConfig,
  resolved_open_lineage_config -> resolved_lineage_config
- update usage-guide docs and the kuttl test manifest
- regenerate extra/crds.yaml (openLineage -> lineage)

The OpenLineage technology name is kept in prose, in the operator-rs types
(OpenLineageJob, OpenLineageConnection) and their connection-named identifiers,
and in the OPENLINEAGE__* client env var names.

Verified with cargo check/clippy/test and CRD regeneration against a local
operator-rs checkout. Committed with --no-verify because the cargo/regenerate
pre-commit hooks would build against the un-pushed operator-rs branch; those
checks were run manually instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(openlineage): follow operator-rs transport enum and required namespace

Bump the operator-rs pin to pick up the reworked `crd::openlineage` module:

- `OpenLineageJob` is renamed to `OpenLineageConfig`.
- `OpenLineageConnectionSpec` now selects an `OpenLineageTransport` (currently
  only `http`) instead of carrying host/port/tls/credentialsSecretName directly.
- `HttpTransport` gained a `path` field, defaulting to `/api/v1/lineage`, which
  now drives `OPENLINEAGE__TRANSPORT__ENDPOINT` instead of a hardcoded constant.
  The leading slash is stripped since the client joins base URL and endpoint.
- `OpenLineageConfig::namespace` is now a required `String` defaulting to
  `default`, so the operator-side fallback to the workload's Kubernetes
  namespace is gone. NOTE: this changes behaviour for users who did not set
  `namespace` explicitly - lineage is now reported under `default` rather than
  the cluster's Kubernetes namespace.

The pin also brings in operator-rs' removal of `product-config`.

Update the usage guide, the kuttl OpenLineageConnection manifest and
extra/crds.yaml for the nested transport schema.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* update cargo.lock

* bump op-rs version 0.120.0

* Update rust/operator-binary/src/controller/build/lineage.rs

Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com>

* review feedback: replace unwrap() with expect() calls in tests.

* review feedback: build() gets 'namespace' instead of 'OpenLineageConfig'

* review feedback: remove unnecessary type cast

* review feedback: remove todo

* review feedback: use EnvVarSet and friends

* review feedback: add warning about CA bundle

* review feedback: use cleanup-executor-pods.sh

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com>

Author: 3 people

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### Added
 
+- Support for emitting [OpenLineage](https://openlineage.io/) lineage events via `spec.clusterConfig.lineage` ([#822]).
 - Support floating tags for product images via the new `spec.image.stackableVersionPolicy` field
   ([#862]).
 - Support for configuring which reverse proxies the webserver trusts `X-Forwarded-*` headers from,
@@ -75,6 +76,7 @@
 
 [#814]: https://github.com/stackabletech/airflow-operator/pull/814
 [#821]: https://github.com/stackabletech/airflow-operator/pull/821
+[#822]: https://github.com/stackabletech/airflow-operator/pull/822
 [#827]: https://github.com/stackabletech/airflow-operator/pull/827
 [#828]: https://github.com/stackabletech/airflow-operator/pull/828
 [#829]: https://github.com/stackabletech/airflow-operator/pull/829