SS-441 Add OAUTH2 SERVER URL option to Iceberg catalog connections to support Unity Catalog

[patrickwwbutler]

Bottom of a two-PR stack. #38181 (vended credentials for Iceberg sinks) is based on this branch and should merge after it.

REST catalog connections that authenticate with CREDENTIAL exchange it for a bearer token at the endpoint the Iceberg REST specification derives from the catalog URL, <url>/v1/oauth/tokens. Not every catalog serves that endpoint.

Databricks Unity Catalog is the motivating case. It routes its whole /api/ surface through an auth gateway that requires a bearer token on every request, so posting a client-credentials grant to the derived endpoint returns:

HTTP/2 401
www-authenticate: Bearer realm="DatabricksRealm"
x-databricks-reason-phrase: Credential was not sent or was of an unsupported type for this API.

That is self-defeating for an endpoint whose job is to issue the first token. Unity Catalog's token endpoint lives at /oidc/v1/token on the workspace root instead, outside /api/.

OAUTH2 SERVER URL points the exchange at an explicit endpoint:

CREATE SECRET databricks_oauth AS '<client-id>:<client-secret>';

CREATE CONNECTION unity TO ICEBERG CATALOG (
    CATALOG TYPE = 'REST',
    URL = 'https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest',
    CREDENTIAL = SECRET databricks_oauth,
    OAUTH2 SERVER URL = 'https://<workspace-host>/oidc/v1/token',
    SCOPE = 'all-apis',
    WAREHOUSE = '<uc-catalog-name>'
);

Omitted, the derived endpoint is used exactly as before, so existing connections are unaffected.

The option is rejected on s3tablesrest catalogs, which authenticate with SigV4 through an AWS connection and perform no OAuth exchange, and alongside GCP CONNECTION, which supplies its own bearer token.

This is a connection-option change only. iceberg-rust already reads the corresponding oauth2-server-uri catalog property, and its OAuth2 provider posts the client-credentials grant as form parameters rather than HTTP Basic, which is the shape Databricks expects.

Tests

Adds a parser round-trip case to the sql-parser datadriven tests, and test/iceberg/oauth2-server-url.td with a workflow_oauth2_server_url covering the accepted form, the s3tablesrest rejection, and the unchanged default. Both are plan-time only and need no Iceberg backend.

Known gap

Connecting to Unity Catalog end to end also needs catalog-vended storage credentials, since Unity Catalog vends temporary credentials rather than exposing bucket keys. That is the follow-up PR stacked on top of this one.

Separately, iceberg-rust's OAuth2TokenProvider caches the token for the lifetime of the catalog object and never consults expires_in, and the REST client has no 401-invalidate-and-retry path. With Databricks' 3600s tokens, a long-running sink will start getting 401s after an hour and recover only by stalling and restarting the dataflow. Worth fixing in the fork before this is used in anger.

Release notes

This release will add an OAUTH2 SERVER URL option to CREATE CONNECTION ... TO ICEBERG CATALOG, allowing REST catalogs whose OAuth2 token endpoint is not at the specification-derived location to be used.

🤖 Generated with Claude Code

[linear-code]

SS-441