Skip to content

Add feature id tracking for credentials #3216

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 61 commits into from
Apr 14, 2025
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
29e6a37
Add comments for feature id locations
richardwang1124 Mar 18, 2025
ffea46f
Merge branch 'version-3' into feature/feature-ids
richardwang1124 Mar 18, 2025
16c86cc
Add new feature ids to METRICS
richardwang1124 Mar 18, 2025
2938a7c
Add comments and change priority of user agent
richardwang1124 Mar 20, 2025
696e1a6
Add skeleton check
richardwang1124 Mar 20, 2025
ed8a0fc
Add skeleton for all except assume role
richardwang1124 Mar 21, 2025
14db8f9
Delete metrics from opts once read
richardwang1124 Mar 21, 2025
e9a5fe0
Add assume role skeleton
richardwang1124 Mar 21, 2025
04fe61d
Fix some test failures
richardwang1124 Mar 21, 2025
374e46b
Add attribute readers
richardwang1124 Mar 21, 2025
260ef00
Revert built changes
richardwang1124 Mar 21, 2025
179901c
Revert built changes
richardwang1124 Mar 21, 2025
fc0528e
Revert built changes
richardwang1124 Mar 21, 2025
31411f8
Add with_metric for service calls
richardwang1124 Mar 21, 2025
2eb610e
Remove some puts
richardwang1124 Mar 21, 2025
8fc119c
Wrap intermediate calls with metric
richardwang1124 Mar 24, 2025
8cde58b
Change priority and revert chaining
richardwang1124 Mar 24, 2025
0806c60
Remove comments
richardwang1124 Mar 24, 2025
5434f86
Fix process credentials
richardwang1124 Mar 24, 2025
6df4fd6
Add small fixes
richardwang1124 Mar 24, 2025
af62faf
Add CHANGELOG entry
richardwang1124 Mar 24, 2025
5b7cf4b
Fix assume role logic
richardwang1124 Mar 24, 2025
015ece2
Fix name
richardwang1124 Mar 24, 2025
804826e
Start to add test
richardwang1124 Mar 25, 2025
e17ba4d
Try source approach
richardwang1124 Mar 25, 2025
01eca88
Partially fix extra metrics issue
richardwang1124 Mar 25, 2025
0299ccd
Fix extra 'i' metric error
richardwang1124 Mar 26, 2025
1dda7e2
Fix some tests
richardwang1124 Mar 26, 2025
3c42148
Fix code to align with correct credentials_code
richardwang1124 Mar 26, 2025
7cd276f
Cleanup
richardwang1124 Mar 26, 2025
1612b21
Fix SSO credential source
richardwang1124 Mar 26, 2025
b1eb340
Start to add unit tests to sign spec
richardwang1124 Mar 27, 2025
b5828b0
Add source testing
richardwang1124 Mar 27, 2025
bdac7df
Add metrics tests
richardwang1124 Mar 27, 2025
e2be224
Add missing test cases
richardwang1124 Mar 27, 2025
c68e112
Add with metrics to refresh
richardwang1124 Mar 28, 2025
cb816ab
Add partial refresh tests
richardwang1124 Mar 28, 2025
9693b9e
Remove metrics from refresh
richardwang1124 Mar 28, 2025
0398f24
Clean up code
richardwang1124 Mar 28, 2025
a509fc8
Fix merge conflicts
richardwang1124 Mar 28, 2025
ad3e569
Make changes from PR feedback
richardwang1124 Apr 1, 2025
bf62f37
Fix tests
richardwang1124 Apr 1, 2025
57e0889
Add some documentation
richardwang1124 Apr 1, 2025
1408d57
Add testing for service call metrics
richardwang1124 Apr 2, 2025
d975eab
Initial fix to simplify return
richardwang1124 Apr 2, 2025
acf8375
Fix duplicate metrics issue with AssumeRoleCredentials
richardwang1124 Apr 2, 2025
1630188
Add keyword argument for metrics_source
richardwang1124 Apr 3, 2025
4cf1af3
Clean up code
richardwang1124 Apr 4, 2025
484288b
'Disable' client side user-agent metric?
richardwang1124 Apr 4, 2025
8d84436
Fix syntax error
richardwang1124 Apr 4, 2025
c58aaec
Revert to setter
richardwang1124 Apr 7, 2025
abf7e56
Add sso stub
richardwang1124 Apr 8, 2025
734b436
Update based on PR comments
richardwang1124 Apr 9, 2025
d2d05e4
Minor edits
richardwang1124 Apr 9, 2025
18ae624
More minor edits
richardwang1124 Apr 9, 2025
ac4305f
Add double arguments
richardwang1124 Apr 9, 2025
9f1f25c
Remove dup
richardwang1124 Apr 9, 2025
281fc0d
Add changes from PR comments
richardwang1124 Apr 10, 2025
147f5e7
Add some rubocop fixes
richardwang1124 Apr 10, 2025
5a5fa02
Fix resolve source
richardwang1124 Apr 11, 2025
ab2034f
Omit unused metrics
richardwang1124 Apr 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions gems/aws-sdk-core/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
Unreleased Changes
------------------

* Feature - Add feature id tracking for credentials.

3.220.1 (2025-03-06)
------------------

Expand Down
3 changes: 3 additions & 0 deletions gems/aws-sdk-core/lib/aws-sdk-core/assume_role_credentials.rb
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,12 @@ def initialize(options = {})
# @return [Hash]
attr_reader :assume_role_params

attr_accessor :metrics

private

def refresh
puts "Refreshing credentials"
resp = @client.assume_role(@assume_role_params)
creds = resp.credentials
@credentials = Credentials.new(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ def initialize(options = {})
# @return [STS::Client]
attr_reader :client

attr_accessor :metrics

private

def refresh
Expand Down
119 changes: 109 additions & 10 deletions gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,79 @@ def initialize(config = nil)

# @return [CredentialProvider, nil]
def resolve
puts "RESOLVING CREDENTIALS"
providers.each do |method_name, options|
provider = send(method_name, options.merge(config: @config))
return provider if provider && provider.set?
if provider && provider.set?
puts method_name
puts provider
case method_name.to_s
when "static_credentials"
puts "CREDENTIALS_PROFILE: n"
when "static_profile_assume_role_web_identity_credentials"
puts "CREDENTIALS_PROFILE_STS_WEB_ID_TOKEN and CREDENTIALS_STS_ASSUME_ROLE_WEB_ID: qk"
when "static_profile_sso_credentials"
puts "CREDENTIALS_PROFILE_SSO and CREDENTIALS_SSO: rs (NEW)"
puts "OR"
puts "CREDENTIALS_PROFILE_SSO_LEGACY and CREDENTIALS_SSO_LEGACY: tu (LEGACY)"
when "static_profile_assume_role_credentials"
puts "CREDENTIALS_PROFILE_SOURCE_PROFILE: o and ( \
CREDENTIALS_PROFILE: n or \
CREDENTIALS_PROFILE_STS_WEB_ID_TOKEN and CREDENTIALS_STS_ASSUME_ROLE_WEB_ID: qk or \
CREDENTIALS_PROFILE_PROCESS and CREDENTIALS_PROCESS: vw or \
CREDENTIALS_PROFILE_SSO and CREDENTIALS_SSO: rs (NEW) or \
CREDENTIALS_PROFILE_SSO_LEGACY and CREDENTIALS_SSO_LEGACY: tu (LEGACY) \
) \
and CREDENTIALS_STS_ASSUME_ROLE: i"
puts "OR"
puts "CREDENTIALS_PROFILE_NAMED_PROVIDER: p and ( \
CREDENTIALS_IMDS: 0 or \
CREDENTIALS_HTTP: z \
) \
and CREDENTIALS_STS_ASSUME_ROLE: i"
when "static_profile_credentials"
puts "CREDENTIALS_PROFILE: n"
when "static_profile_process_credentials"
puts "CREDENTIALS_PROFILE_PROCESS and CREDENTIALS_PROCESS: vw"
when "env_credentials"
puts "CREDENTIALS_ENV_VARS: g"
when "assume_role_web_identity_credentials"
puts "CREDENTIALS_ENV_VARS_STS_WEB_ID_TOKEN and CREDENTIALS_STS_ASSUME_ROLE_WEB_ID: hk"
puts "OR"
puts "CREDENTIALS_PROFILE_STS_WEB_ID_TOKEN and CREDENTIALS_STS_ASSUME_ROLE_WEB_ID: qk"
when "sso_credentials"
puts "CREDENTIALS_PROFILE_SSO and CREDENTIALS_SSO: rs (NEW)"
puts "OR"
puts "CREDENTIALS_PROFILE_SSO_LEGACY and CREDENTIALS_SSO_LEGACY: tu (LEGACY)"
when "assume_role_credentials"
puts "CREDENTIALS_PROFILE_SOURCE_PROFILE: o and ( \
CREDENTIALS_PROFILE: n or \
CREDENTIALS_PROFILE_STS_WEB_ID_TOKEN and CREDENTIALS_STS_ASSUME_ROLE_WEB_ID: qk or \
CREDENTIALS_PROFILE_PROCESS and CREDENTIALS_PROCESS: vw or \
CREDENTIALS_PROFILE_SSO and CREDENTIALS_SSO: rs (NEW) or \
CREDENTIALS_PROFILE_SSO_LEGACY and CREDENTIALS_SSO_LEGACY: tu (LEGACY) \
) \
and CREDENTIALS_STS_ASSUME_ROLE: i"
puts "OR"
puts "CREDENTIALS_PROFILE_NAMED_PROVIDER: p and ( \
CREDENTIALS_IMDS: 0 or \
CREDENTIALS_HTTP: z \
) \
and CREDENTIALS_STS_ASSUME_ROLE: i"
when "shared_credentials"
puts "CREDENTIALS_PROFILE: n"
when "process_credentials"
puts "CREDENTIALS_PROFILE_PROCESS and CREDENTIALS_PROCESS: vw"
when "instance_profile_credentials"
puts "CREDENTIALS_HTTP (z)"
puts "OR"
puts "CREDENTIALS_IMDS (0)"
else
puts method_name
puts "!! UNKNOWN !!"
end
return provider
end
end
nil
end
Expand Down Expand Up @@ -42,12 +112,14 @@ def providers

def static_credentials(options)
if options[:config]
Credentials.new(
credentials = Credentials.new(
options[:config].access_key_id,
options[:config].secret_access_key,
options[:config].session_token,
account_id: options[:config].account_id
)
credentials.metrics = ['CREDENTIALS_PROFILE']
credentials
end
end

Expand Down Expand Up @@ -76,7 +148,9 @@ def static_profile_assume_role_credentials(options)

def static_profile_credentials(options)
if options[:config] && options[:config].profile
SharedCredentials.new(profile_name: options[:config].profile)
credentials = SharedCredentials.new(profile_name: options[:config].profile)
credentials.metrics = ['CREDENTIALS_PROFILE']
credentials
end
rescue Errors::NoSuchProfileError
nil
Expand All @@ -85,23 +159,30 @@ def static_profile_credentials(options)
def static_profile_process_credentials(options)
if Aws.shared_config.config_enabled? && options[:config] && options[:config].profile
process_provider = Aws.shared_config.credential_process(profile: options[:config].profile)
ProcessCredentials.new([process_provider]) if process_provider
if process_provider
credentials = ProcessCredentials.new([process_provider])
credentials.metrics = %w[CREDENTIALS_PROFILE_PROCESS CREDENTIALS_PROCESS]
credentials
end
end
rescue Errors::NoSuchProfileError
nil
end


def env_credentials(_options)
key = %w[AWS_ACCESS_KEY_ID AMAZON_ACCESS_KEY_ID AWS_ACCESS_KEY]
secret = %w[AWS_SECRET_ACCESS_KEY AMAZON_SECRET_ACCESS_KEY AWS_SECRET_KEY]
token = %w[AWS_SESSION_TOKEN AMAZON_SESSION_TOKEN]
account_id = %w[AWS_ACCOUNT_ID]
Credentials.new(
credentials = Credentials.new(
envar(key),
envar(secret),
envar(token),
account_id: envar(account_id)
)
credentials.metrics = ['CREDENTIALS_ENV_VARS']
credentials
end

def envar(keys)
Expand All @@ -117,7 +198,9 @@ def determine_profile_name(options)

def shared_credentials(options)
profile_name = determine_profile_name(options)
SharedCredentials.new(profile_name: profile_name)
credentials = SharedCredentials.new(profile_name: profile_name)
credentials.metrics = ['CREDENTIALS_PROFILE']
credentials
rescue Errors::NoSuchProfileError
nil
end
Expand All @@ -126,7 +209,11 @@ def process_credentials(options)
profile_name = determine_profile_name(options)
if Aws.shared_config.config_enabled?
process_provider = Aws.shared_config.credential_process(profile: profile_name)
ProcessCredentials.new([process_provider]) if process_provider
if process_provider
credentials = ProcessCredentials.new([process_provider])
credentials.metrics = %w[CREDENTIALS_PROFILE_PROCESS CREDENTIALS_PROCESS]
credentials
end
end
rescue Errors::NoSuchProfileError
nil
Expand Down Expand Up @@ -156,7 +243,11 @@ def assume_role_web_identity_credentials(options)
role_session_name: ENV['AWS_ROLE_SESSION_NAME']
}
cfg[:region] = region if region
AssumeRoleWebIdentityCredentials.new(cfg)
with_metrics('CREDENTIALS_ENV_VARS_STS_WEB_ID_TOKEN') do
credentials = AssumeRoleWebIdentityCredentials.new(cfg)
credentials.metrics = %w[CREDENTIALS_ENV_VARS_STS_WEB_ID_TOKEN CREDENTIALS_STS_ASSUME_ROLE_WEB_ID]
credentials
end
elsif Aws.shared_config.config_enabled?
profile = options[:config].profile if options[:config]
Aws.shared_config.assume_role_web_identity_credentials_from_config(
Expand All @@ -170,9 +261,13 @@ def instance_profile_credentials(options)
profile_name = determine_profile_name(options)
if ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'] ||
ENV['AWS_CONTAINER_CREDENTIALS_FULL_URI']
ECSCredentials.new(options)
credentials = ECSCredentials.new(options)
credentials.metrics = ['CREDENTIALS_HTTP']
credentials
else
InstanceProfileCredentials.new(options.merge(profile: profile_name))
credentials = InstanceProfileCredentials.new(options.merge(profile: profile_name))
credentials.metrics = ['CREDENTIALS_IMDS']
credentials
end
end

Expand All @@ -186,5 +281,9 @@ def assume_role_with_profile(options, profile_name)
end
Aws.shared_config.assume_role_credentials_from_config(assume_opts)
end

def with_metrics(metric, &block)
Aws::Plugins::UserAgent.metric(metric, &block)
end
end
end
2 changes: 2 additions & 0 deletions gems/aws-sdk-core/lib/aws-sdk-core/credentials.rb
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ def initialize(access_key_id, secret_access_key, session_token = nil,
# @return [String, nil]
attr_reader :account_id

attr_accessor :metrics

# @return [Credentials]
def credentials
self
Expand Down
2 changes: 2 additions & 0 deletions gems/aws-sdk-core/lib/aws-sdk-core/ecs_credentials.rb
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,8 @@ def initialize(options = {})
# fetch credentials from the instance metadata service. Defaults to 0.
attr_reader :retries

attr_accessor :metrics

private

def initialize_uri(options, credential_path, endpoint)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ def initialize(options = {})
# the default credential chain ({Aws::CredentialProviderChain}).
attr_reader :retries

attr_accessor :metrics

private

def resolve_endpoint_mode(options)
Expand Down
27 changes: 26 additions & 1 deletion gems/aws-sdk-core/lib/aws-sdk-core/plugins/sign.rb
Original file line number Diff line number Diff line change
Expand Up @@ -41,20 +41,43 @@ def self.signer_for(auth_scheme, config, sigv4_region_override = nil, sigv4_cred
class Handler < Seahorse::Client::Handler
def call(context)
# Skip signing if using sigv2 signing from s3_signer in S3
credentials = nil
unless v2_signing?(context.config)
signer = Sign.signer_for(
context[:auth_scheme],
context.config,
context[:sigv4_region],
context[:sigv4_credentials]
)
# TODO: temp added this, double check
if signer.is_a?(SignatureV4)
credentials = signer.signer.credentials_provider
end
signer.sign(context)
end
@handler.call(context)
with_metrics(credentials) { @handler.call(context) }
end

private

def with_metrics(credentials, &block)
puts "in with metric"
unless credentials && credentials.respond_to?(:metrics)
puts "No metrics"
return block.call
end

metrics = []
if !credentials.metrics
metrics << 'CREDENTIALS_CODE'
else
puts credentials.class.name
puts credentials.metrics
(metrics << credentials.metrics).flatten!
end
Aws::Plugins::UserAgent.metric(*metrics, &block)
end

def v2_signing?(config)
# 's3' is legacy signing, 'v4' is default
config.respond_to?(:signature_version) &&
Expand Down Expand Up @@ -92,6 +115,8 @@ def sign_event(*args)

# @api private
class SignatureV4
attr_reader :signer

def initialize(auth_scheme, config, sigv4_overrides = {})
scheme_name = auth_scheme['name']

Expand Down
34 changes: 32 additions & 2 deletions gems/aws-sdk-core/lib/aws-sdk-core/plugins/user_agent.rb
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,33 @@ class UserAgent < Seahorse::Client::Plugin
"FLEXIBLE_CHECKSUMS_REQ_WHEN_SUPPORTED" : "Z",
"FLEXIBLE_CHECKSUMS_REQ_WHEN_REQUIRED" : "a",
"FLEXIBLE_CHECKSUMS_RES_WHEN_SUPPORTED" : "b",
"FLEXIBLE_CHECKSUMS_RES_WHEN_REQUIRED" : "c"
"FLEXIBLE_CHECKSUMS_RES_WHEN_REQUIRED" : "c",
"DDB_MAPPER": "d",
"CREDENTIALS_CODE" : "e",
"CREDENTIALS_JVM_SYSTEM_PROPERTIES" : "f",
"CREDENTIALS_ENV_VARS" : "g",
"CREDENTIALS_ENV_VARS_STS_WEB_ID_TOKEN" : "h",
"CREDENTIALS_STS_ASSUME_ROLE" : "i",
"CREDENTIALS_STS_ASSUME_ROLE_SAML" : "j",
"CREDENTIALS_STS_ASSUME_ROLE_WEB_ID" : "k",
"CREDENTIALS_STS_FEDERATION_TOKEN" : "l",
"CREDENTIALS_STS_SESSION_TOKEN" : "m",
"CREDENTIALS_PROFILE" : "n",
"CREDENTIALS_PROFILE_SOURCE_PROFILE" : "o",
"CREDENTIALS_PROFILE_NAMED_PROVIDER" : "p",
"CREDENTIALS_PROFILE_STS_WEB_ID_TOKEN" : "q",
"CREDENTIALS_PROFILE_SSO" : "r",
"CREDENTIALS_SSO" : "s",
"CREDENTIALS_PROFILE_SSO_LEGACY" : "t",
"CREDENTIALS_SSO_LEGACY" : "u",
"CREDENTIALS_PROFILE_PROCESS" : "v",
"CREDENTIALS_PROCESS" : "w",
"CREDENTIALS_BOTO2_CONFIG_FILE" : "x",
"CREDENTIALS_AWS_SDK_STORE" : "y",
"CREDENTIALS_HTTP" : "z",
"CREDENTIALS_IMDS" : "0",
"SSO_LOGIN_DEVICE" : "1",
"SSO_LOGIN_AUTH" : "2"
}
METRICS

Expand Down Expand Up @@ -186,7 +212,11 @@ def metric_metadata
return
end

# Needed for AssumeRoleCredentials feature id tracking
Thread.current[:aws_sdk_core_user_agent_metric].uniq!

metrics = Thread.current[:aws_sdk_core_user_agent_metric].join(',')

# Metric metadata is limited to 1024 bytes
return "m/#{metrics}" if metrics.bytesize <= 1024

Expand All @@ -196,7 +226,7 @@ def metric_metadata
end
end

handler(Handler, step: :sign, priority: 97)
handler(Handler, step: :sign, priority: 5)
end
end
end
2 changes: 2 additions & 0 deletions gems/aws-sdk-core/lib/aws-sdk-core/process_credentials.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ def initialize(process)
super
end

attr_accessor :metrics

private

def credentials_from_process
Expand Down
Loading