Feature Request: Allow iceberg secret to use credential_chain

[kevinjqliu]

Context

The credential_chain provider is currently supported for the S3 secret type and other storage secrets:
https://duckdb.org/docs/stable/core_extensions/iceberg/amazon_s3_tables#connecting-to-amazon-s3-tables

CREATE SECRET (
    TYPE s3,
    PROVIDER credential_chain
);

However, for iceberg secrets, I currently have to explicitly specify OAuth2 credentials such as a bearer token or client_id/client_secret:
https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs

CREATE SECRET iceberg_secret (
    TYPE ICEBERG,
    TOKEN 'bearer_token'
);

ATTACH 'warehouse' AS iceberg_catalog (
   TYPE iceberg,
   SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity
   ENDPOINT https://rest_endpoint.com
);

Proposal

Add support for PROVIDER credential_chain to the iceberg secret type to simplify credential management. This would allow credentials to be resolved dynamically (e.g., via environment, instance metadata, or shared profiles), just like with S3 secrets:

CREATE SECRET iceberg_secret (
    TYPE ICEBERG,
    PROVIDER credential_chain
);

This would make it easier to integrate with cloud environments without hardcoding or manually refreshing tokens.

[kevinjqliu]

Alternatively, can i reuse an existing s3 secret for the IRC integration? It looks like this is what the s3tables/glue integration is doing with SigV4

duckdb-iceberg/src/storage/authorization/sigv4.cpp

Lines 67 to 102 in ad2ceeb

	AWSInput SIGV4Authorization::CreateAWSInput(ClientContext &context, const IRCEndpointBuilder &endpoint_builder) {
	AWSInput aws_input;
	aws_input.cert_path = APIUtils::GetCURLCertPath();
	// Set the user Agent
	auto &config = DBConfig::GetConfig(context);
	aws_input.user_agent = config.UserAgent();
	// AWS service and region
	aws_input.service = GetAwsService(endpoint_builder.GetHost());
	aws_input.region = GetAwsRegion(endpoint_builder.GetHost());
	// Host decomposition
	auto decomposed_host = DecomposeHost(endpoint_builder.GetHost());
	aws_input.authority = decomposed_host.authority;
	for (auto &component : decomposed_host.path_components) {
	aws_input.path_segments.push_back(component);
	}
	for (auto &component : endpoint_builder.path_components) {
	aws_input.path_segments.push_back(component);
	}
	for (auto &param : endpoint_builder.GetParams()) {
	aws_input.query_string_parameters.emplace_back(param);
	}
	// AWS credentials
	auto secret_entry = IRCatalog::GetStorageSecret(context, secret);
	auto kv_secret = dynamic_cast<const KeyValueSecret &>(*secret_entry->secret);
	aws_input.key_id = kv_secret.secret_map["key_id"].GetValue<string>();
	aws_input.secret = kv_secret.secret_map["secret"].GetValue<string>();
	aws_input.session_token =
	kv_secret.secret_map["session_token"].IsNull() ? "" : kv_secret.secret_map["session_token"].GetValue<string>();
	return aws_input;
	}

[Tmonster]

Thanks for the suggestion!

s3tables/glue secrets can use the credential_chain yes. Interally for duckdb-iceberg, we then look for an AWS or S3 secret to communicate with the Rest catalog. I've been thinking about adding Iceberg secrets to the credential_chain or some equivalent, but in its simplest form, it means allowing the AWS extension to create Iceberg secrets. This kind of dependency is not something we implement quickly.

I think there will be a way to design around this however, but it may take some time before we get to this

[kevinjqliu]

Looking at this again, it seems that maybe a better solution is to allow the IRC integration to directly read from storage secrets.

The current integration looks like this, from https://duckdb.org/docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse:

CREATE SECRET (
    TYPE s3,
    PROVIDER credential_chain,
    CHAIN sts,
    ASSUME_ROLE_ARN 'arn:aws:iam::account_id:role/role',
    REGION 'us-east-2'
);
ATTACH 'account_id:s3tablescatalog/namespace_name' AS glue_catalog (
    TYPE iceberg,
    ENDPOINT_TYPE glue
);

where ENDPOINT_TYPE glue internally reads from the S3 storage secret.

The proposed solution can look something like:

CREATE SECRET s3_secret (
    TYPE s3,
    PROVIDER credential_chain,
    CHAIN sts,
    ASSUME_ROLE_ARN 'arn:aws:iam::account_id:role/role',
    REGION 'us-east-2'
);
ATTACH 'account_id:s3tablescatalog/namespace_name' AS glue_catalog (
   TYPE iceberg,
   SECRET s3_secret,
   ENDPOINT https://rest_endpoint.com
);

where the IRC integration derives its secret from the given S3 secret...

WDYT?