Mountpoint doesn't work with credential_process in combination with source_profile #927
Labels
Comments
|
Mountpoint uses the Common Runtime (CRT) as its S3 client and I suspect that does not support IAM Roles Anywhere. I'm asking internally and will post back an update when I get it. If CRT does not support it, this will become a feature request to them and later us. |
|
Can you try using |
|
The CRT supports |
|
Hi @arsh. Thank you for your response. Here is the contests of my Checking the identity: Listing the buckets: Trying to mount S3 bucket: I've noticed that when I remove |
|
I'd like to see the logs for this last run. Can you specify a log directory with the |
|
@sdauhuchytsrf, can you try setting environment variable |
|
@dannycjones @arsh here is the output: |
|
In that log, I see the following Can you make sure that Also from the logs, it looks like it is falling back to IMDS, presumably due to the above failure. |
|
@arsh Edited Also tried to use absolute path |
|
OK. I'm going to try to reproduce this and report back any findings. |
|
I was able to reproduce this and I will be talking to the CRT team to understand what is happening. In the meantime, can you try using the profile which declares the |
|
Actually, can you try the following. Basically, not using |
|
Hi @arsh. Thanks for your fast response! I applied the configuration above and get the following error while trying to do |
|
Yes, I see that error when using the CLI. However, I do not see that when using Mountpoint. It is likely that Mountpoint (and CRT) isn't doing the right thing is this case (will need to check if it is actually assuming the role). Can you try the same setting using Mountpoint ? |
|
Unfortunately I still see the same error: |
|
Can you also provide the logs of that last run? |
|
Here they are: |
|
Thanks. In the latest log, I see you are no longer getting the Also, it seems that now this is a legitimate access denied error. Perhaps because it is not assuming the role you intended. As I said before, I will be bringing this up with the CRT team. |
dannycjones
changed the title
dannycjones
added a commit
to dannycjones/mountpoint-s3
that referenced
this issue
Sep 2, 2024
dannycjones
added a commit
to dannycjones/mountpoint-s3
that referenced
this issue
Sep 2, 2024
dannycjones
added a commit
to dannycjones/mountpoint-s3
that referenced
this issue
Sep 2, 2024
unexge
pushed a commit
to unexge/mountpoint-s3
that referenced
this issue
Sep 3, 2024
unexge
pushed a commit
to unexge/mountpoint-s3
that referenced
this issue
Sep 3, 2024
unexge
pushed a commit
to unexge/mountpoint-s3
that referenced
this issue
Sep 3, 2024
Signed-off-by: Burak Varlı <[email protected]>
unexge
pushed a commit
to unexge/mountpoint-s3
that referenced
this issue
Sep 4, 2024
Signed-off-by: Burak Varlı <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this issue
Sep 4, 2024
* Update mountpoint-s3-crt-sys crate excludes to reduce package size Signed-off-by: Daniel Carl Jones <[email protected]> Signed-off-by: Burak Varlı <[email protected]> * Update CRT submodules to latest releases Signed-off-by: Daniel Carl Jones <[email protected]> Signed-off-by: Burak Varlı <[email protected]> * WIP: Add testing for #927 Signed-off-by: Burak Varlı <[email protected]> * Gate scoped credential test Signed-off-by: Burak Varlı <[email protected]> * Update `test_credential_process_behind_source_profile` to use role Signed-off-by: Burak Varlı <[email protected]> * Remove unused import Signed-off-by: Burak Varlı <[email protected]> * Remove TODOs from CHANGELOG Signed-off-by: Burak Varlı <[email protected]> * Update CRT submodules to latest Signed-off-by: Burak Varlı <[email protected]> * Remove feature gate from `test_credential_process_behind_source_profile` Signed-off-by: Burak Varlı <[email protected]> * Fix Clippy failures Signed-off-by: Burak Varlı <[email protected]> * More Clippy fixes Signed-off-by: Burak Varlı <[email protected]> * Fix formatting Signed-off-by: Burak Varlı <[email protected]> * Update test failure message Co-authored-by: Daniel Carl Jones <[email protected]> Signed-off-by: Burak <[email protected]> --------- Signed-off-by: Daniel Carl Jones <[email protected]> Signed-off-by: Burak Varlı <[email protected]> Signed-off-by: Burak <[email protected]> Co-authored-by: Daniel Carl Jones <[email protected]> Co-authored-by: Daniel Carl Jones <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Mountpoint for Amazon S3 version
mount-s3 1.7.2
AWS Region
eu-central-1
Describe the running environment
Running on non-EC2 instance which uses IAM Roles Anywhere configuration. AWS profile configuration is the following:
aws_signing_helper tool is running in
servemode:Exported
AWS_EC2_METADATA_SERVICE_ENDPOINTenv var:Check configuration with AWS CLI to be sure that it works:
Try to mount S3:
Mountpoint options
What happened?
Mountpoint failed to use config file to sign request.
No signing credentialsfound error.Relevant log output
mount-s3 --debug --debug-crt -f my-bucket-name /path/to/mnt/dir --profile my-profile-name --region eu-central-1 2024-06-28T08:08:17.058928Z INFO mountpoint_s3::cli: mount-s3 1.7.2 2024-06-28T08:08:17.059081Z DEBUG mountpoint_s3::cli: CliArgs { bucket_name: "my-bucket-name", mount_point: "/path/to/mnt/dir", prefix: None, region: Some("eu-central-1"), endpoint_url: None, force_path_style: false, transfer_acceleration: false, dual_stack: false, requester_pays: false, bucket_type: None, no_sign_request: false, profile: Some("my-profile-name"), read_only: false, storage_class: None, allow_delete: false, allow_overwrite: false, auto_unmount: false, allow_root: false, allow_other: false, maximum_throughput_gbps: None, max_threads: 16, part_size: 8388608, uid: None, gid: None, dir_mode: None, file_mode: None, foreground: true, expected_bucket_owner: None, log_directory: None, log_metrics: false, debug: true, debug_crt: true, no_log: false, cache: None, metadata_ttl: None, max_cache_size: None, user_agent_prefix: None, sse: None, sse_kms_key_id: None, upload_checksums: None } 2024-06-28T08:08:17.059362Z DEBUG awscrt::libcrypto_resolve: searching process and loaded modules 2024-06-28T08:08:17.059387Z DEBUG awscrt::libcrypto_resolve: found static aws-lc HMAC symbols 2024-06-28T08:08:17.059413Z DEBUG awscrt::libcrypto_resolve: found static aws-lc libcrypto 1.1.1 EVP_MD symbols 2024-06-28T08:08:17.059441Z INFO awscrt::tls-handler: static: Initializing TLS using s2n. 2024-06-28T08:08:17.059701Z DEBUG awscrt::tls-handler: ctx: Based on OS, we detected the default PKI path as /etc/ssl/certs, and ca file as /etc/ssl/certs/ca-certificates.crt 2024-06-28T08:08:17.059769Z INFO awscrt::event-loop: id=0x55bc5e9e6e20: Initializing edge-triggered epoll 2024-06-28T08:08:17.059899Z INFO awscrt::event-loop: id=0x55bc5e9e6e20: Using eventfd for cross-thread notifications. 2024-06-28T08:08:17.060112Z INFO awscrt::event-loop: id=0x55bc5e9e6e20: Starting event-loop thread. 2024-06-28T08:08:17.060307Z INFO awscrt::dns: id=0x55bc5e9e1050: Initializing default host resolver with 8 max host entries. 2024-06-28T08:08:17.060383Z INFO awscrt::channel-bootstrap: id=0x55bc5e9fd120: Initializing client bootstrap with event-loop group 0x55bc5e9e3970 2024-06-28T08:08:17.060333Z INFO awscrt::event-loop: id=0x55bc5e9e6e20: main loop started 2024-06-28T08:08:17.060602Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: acquiring bootstrap reference 2024-06-28T08:08:17.060730Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: Successfully created 2024-06-28T08:08:17.060783Z INFO awscrt::event-loop: id=0x55bc5e9e6e20: default timeout 100000, and max events to process per tick 100 2024-06-28T08:08:17.060921Z INFO awscrt::exp-backoff-strategy: id=0x55bc5e9fe860: Initializing exponential backoff retry strategy with scale factor: 0 jitter mode: 0 and max retries 1 2024-06-28T08:08:17.060945Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: releasing bootstrap reference 2024-06-28T08:08:17.060987Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5e9fe860: Initializing retry token 0x55bc5e9fd3e0 2024-06-28T08:08:17.061017Z DEBUG awscrt::Unknown: (id=0x55bc5e9f4f10) IMDS client's token is invalid and is now updating. 2024-06-28T08:08:17.061318Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5e9fe860: Vending retry_token 0x55bc5e9fd3e0 2024-06-28T08:08:17.061370Z DEBUG awscrt::Unknown: id=0x55bc5e9f4f10: IMDS Client successfully acquired retry token. 2024-06-28T08:08:17.061413Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: Acquire connection 2024-06-28T08:08:17.061479Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=0, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1 2024-06-28T08:08:17.061557Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: Requesting 1 new connections from http 2024-06-28T08:08:17.061710Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: acquiring bootstrap reference 2024-06-28T08:08:17.061733Z DEBUG awscrt::dns: id=0x55bc5e9e1050: Host resolution requested for 169.254.169.254 2024-06-28T08:08:17.061747Z DEBUG awscrt::dns: id=0x55bc5e9e1050: No cached entries found for 169.254.169.254 starting new resolver thread. 2024-06-28T08:08:17.061848Z DEBUG awscrt::dns: static: resolving host 169.254.169.254 2024-06-28T08:08:17.061904Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254 2024-06-28T08:08:17.061941Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses 2024-06-28T08:08:17.061977Z DEBUG awscrt::dns: static: new address resolved 169.254.169.254 for host 169.254.169.254 caching 2024-06-28T08:08:17.062012Z DEBUG awscrt::dns: static, invoking resolution callback for host 169.254.169.254 with 1 addresses 2024-06-28T08:08:17.062265Z DEBUG awscrt::socket: id=0x7fb7d40043a0 fd=8: initializing with domain 0 and type 0 2024-06-28T08:08:17.062284Z DEBUG awscrt::socket: id=0x7fb7d40043a0 fd=8: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0. 2024-06-28T08:08:17.062295Z DEBUG awscrt::socket: id=0x7fb7d40043a0 fd=8: beginning connect. 2024-06-28T08:08:17.062304Z DEBUG awscrt::socket: id=0x7fb7d40043a0 fd=8: connecting to endpoint 169.254.169.254:80. 2024-06-28T08:08:17.063912Z INFO awscrt::socket: id=0x7fb7d40043a0 fd=8: connection success 2024-06-28T08:08:17.063962Z DEBUG awscrt::socket: id=0x7fb7d40043a0 fd=8: assigning to event loop 0x55bc5e9e6e20 2024-06-28T08:08:17.063979Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: client connection on socket 0x7fb7d40043a0 completed with error 0. 2024-06-28T08:08:17.063988Z DEBUG awscrt::channel: id=0x7fb7d4004f20: Beginning creation and setup of new channel. 2024-06-28T08:08:17.063996Z DEBUG awscrt::channel: id=0x7fb7d4004f20: setup complete, notifying caller. 2024-06-28T08:08:17.064006Z DEBUG awscrt::channel: id=0x7fb7d4004f20: no message pool is currently stored in the event-loop local storage, adding 0x7fb7d4004680 with max message size 16384, message count 4, with 4 small blocks of 128 bytes. 2024-06-28T08:08:17.064024Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: channel 0x7fb7d4004f20 setup succeeded: bootstrapping. 2024-06-28T08:08:17.064034Z DEBUG awscrt::socket-handler: id=0x7fb7d4015c80: Socket handler created with max_read_size of 16384 2024-06-28T08:08:17.064046Z INFO awscrt::http-connection: id=0x7fb7d40160a0: HTTP/1.1 client connection established. 2024-06-28T08:08:17.064054Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: Received new connection (id=0x7fb7d40160a0) from http layer 2024-06-28T08:08:17.064062Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: Grabbing pooled connection (0x7fb7d40160a0) 2024-06-28T08:08:17.064071Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.064079Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: Successfully completed connection acquisition with connection id=0x7fb7d40160a0 2024-06-28T08:08:17.064099Z DEBUG awscrt::http-stream: id=0x7fb7d4017120: Created client request on connection=0x7fb7d40160a0: PUT /latest/api/token HTTP/1.1 2024-06-28T08:08:17.064317Z DEBUG awscrt::Unknown: (id=0x55bc5e9f4f10) IMDS client query received http status code 404 for requester 0x55bc5e9f5aa0. 2024-06-28T08:08:17.064331Z DEBUG awscrt::http-stream: id=0x7fb7d4017120: Client request complete, response status: 404 (Not Found). 2024-06-28T08:08:17.064341Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: User releasing connection (id=0x7fb7d40160a0) 2024-06-28T08:08:17.064351Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.064363Z DEBUG awscrt::Unknown: (id=0x55bc5e9f4f10) IMDS client failed to fetch token for requester 0x55bc5e9f5aa0, fall back to v1 for the same requester. Received response status code: 404 2024-06-28T08:08:17.064373Z DEBUG awscrt::Unknown: (id=0x55bc5e9f4f10) IMDS client failed to fetch token, fallback to v1. requester 0x55bc5e9e38d0. 2024-06-28T08:08:17.064380Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5e9fe860: Initializing retry token 0x7fb7d4018090 2024-06-28T08:08:17.064391Z ERROR awscrt::Unknown: (id=0x55bc5e9f4f10) IMDS client failed to update the token from IMDS. 2024-06-28T08:08:17.064402Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5e9fe860: Vending retry_token 0x7fb7d4018090 2024-06-28T08:08:17.064410Z DEBUG awscrt::Unknown: id=0x55bc5e9f4f10: IMDS Client successfully acquired retry token. 2024-06-28T08:08:17.064417Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: Acquire connection 2024-06-28T08:08:17.064425Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: Grabbing pooled connection (0x7fb7d40160a0) 2024-06-28T08:08:17.064434Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.064442Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: Successfully completed connection acquisition with connection id=0x7fb7d40160a0 2024-06-28T08:08:17.064456Z DEBUG awscrt::http-stream: id=0x7fb7d4018ab0: Created client request on connection=0x7fb7d40160a0: GET /latest/dynamic/instance-identity/document HTTP/1.1 2024-06-28T08:08:17.064635Z DEBUG awscrt::Unknown: (id=0x55bc5e9f4f10) IMDS client query received http status code 404 for requester 0x55bc5e9e38d0. 2024-06-28T08:08:17.064657Z DEBUG awscrt::http-stream: id=0x7fb7d4018ab0: Client request complete, response status: 404 (Not Found). 2024-06-28T08:08:17.064682Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: User releasing connection (id=0x7fb7d40160a0) 2024-06-28T08:08:17.064701Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.065270Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: release 2024-06-28T08:08:17.065378Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: ref count now zero, starting shut down process 2024-06-28T08:08:17.065488Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: manager release, failing 0 pending acquisitions 2024-06-28T08:08:17.065597Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: snapshot - state=2, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=0 2024-06-28T08:08:17.065730Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: Releasing connection (id=0x7fb7d40160a0) 2024-06-28T08:08:17.065852Z DEBUG mountpoint_s3_client::instance_info: EC2 instance info not retrieved: ImdsQueryFailed(InvalidResponse("Not Found")) 2024-06-28T08:08:17.065910Z DEBUG awscrt::channel: id=0x7fb7d4004f20: beginning shutdown process 2024-06-28T08:08:17.065959Z DEBUG awscrt::channel: id=0x7fb7d4004f20: handler 0x7fb7d4015c80 shutdown in read dir completed. 2024-06-28T08:08:17.065997Z DEBUG awscrt::channel: id=0x7fb7d4004f20: handler 0x7fb7d40160a8 shutdown in read dir completed. 2024-06-28T08:08:17.066077Z DEBUG awscrt::channel: id=0x7fb7d4004f20: handler 0x7fb7d40160a8 shutdown in write dir completed. 2024-06-28T08:08:17.066119Z DEBUG awscrt::socket: id=0x7fb7d40043a0 fd=8: closing 2024-06-28T08:08:17.066185Z DEBUG awscrt::channel: id=0x7fb7d4004f20: handler 0x7fb7d4015c80 shutdown in write dir completed. 2024-06-28T08:08:17.066231Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: channel 0x7fb7d4004f20 shutdown with error 0. 2024-06-28T08:08:17.066315Z INFO awscrt::http-connection: 0x7fb7d40160a0: Client shutdown completed with error 0 (AWS_ERROR_SUCCESS). 2024-06-28T08:08:17.066359Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: shutdown received for connection (id=0x7fb7d40160a0) 2024-06-28T08:08:17.066396Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: manager release, failing 0 pending acquisitions 2024-06-28T08:08:17.066455Z DEBUG awscrt::connection-manager: id=0x55bc5ea004e0: snapshot - state=2, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=0 2024-06-28T08:08:17.066501Z INFO awscrt::connection-manager: id=0x55bc5ea004e0: Destroying self 2024-06-28T08:08:17.066537Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: releasing bootstrap reference 2024-06-28T08:08:17.066572Z DEBUG awscrt::channel: id=0x7fb7d4004f20: destroying channel. 2024-06-28T08:08:17.066609Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: releasing bootstrap reference 2024-06-28T08:08:17.066645Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9fd120: bootstrap destroying 2024-06-28T08:08:17.067108Z WARN mountpoint_s3::cli: failed to detect network throughput. Using 10 gbps as throughput. Use --maximum-throughput-gbps CLI flag to configure a target throughput appropriate for the instance. Detection failed due to: failed to get instance type: IMDS query failed: Invalid response from Imds: Not Found 2024-06-28T08:08:17.067134Z INFO mountpoint_s3::cli: target network throughput 10 Gbps 2024-06-28T08:08:17.067191Z INFO awscrt::event-loop: id=0x55bc5ea040d0: Initializing edge-triggered epoll 2024-06-28T08:08:17.067216Z INFO awscrt::event-loop: id=0x55bc5ea040d0: Using eventfd for cross-thread notifications. 2024-06-28T08:08:17.067237Z INFO awscrt::event-loop: id=0x55bc5ea040d0: Starting event-loop thread. 2024-06-28T08:08:17.067296Z INFO awscrt::dns: id=0x55bc5e9f6470: Initializing default host resolver with 8 max host entries. 2024-06-28T08:08:17.067311Z INFO awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: Initializing client bootstrap with event-loop group 0x55bc5e9f5ae0 2024-06-28T08:08:17.067319Z INFO awscrt::standard-retry-strategy: static: creating new standard retry strategy 2024-06-28T08:08:17.067329Z INFO awscrt::standard-retry-strategy: id=0x55bc5e9f6250: creating backing exponential backoff strategy with max_retries of 9 2024-06-28T08:08:17.067340Z INFO awscrt::exp-backoff-strategy: id=0x55bc5e9ef8a0: Initializing exponential backoff retry strategy with scale factor: 500 jitter mode: 2 and max retries 9 2024-06-28T08:08:17.067349Z DEBUG awscrt::standard-retry-strategy: id=0x55bc5e9f6250: maximum bucket capacity set to 500 2024-06-28T08:08:17.067369Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/home/admin/.aws/config" 2024-06-28T08:08:17.067400Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/home/admin/.aws/credentials" 2024-06-28T08:08:17.067442Z ERROR awscrt::common-io: static: Failed to open file. path:'/home/admin/.aws/credentials' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH) 2024-06-28T08:08:17.067456Z DEBUG awscrt::AWSProfile: Failed to read file at "/home/admin/.aws/credentials" 2024-06-28T08:08:17.067466Z DEBUG awscrt::AWSProfile: property "credential_source" has value "" replaced during merge 2024-06-28T08:08:17.067476Z DEBUG awscrt::AWSProfile: property "role_arn" has value "" replaced during merge 2024-06-28T08:08:17.067486Z DEBUG awscrt::AWSProfile: property "region" has value "" replaced during merge 2024-06-28T08:08:17.067511Z INFO awscrt::AuthCredentialsProvider: static: profile my-profile-name has role_arn property is set to arn:aws:iam::123456789012:role/my-role-name, attempting to create an STS credentials provider. 2024-06-28T08:08:17.067532Z DEBUG awscrt::AuthCredentialsProvider: static: computed session_name as aws-common-runtime-profile-config-21339 2024-06-28T08:08:17.067550Z INFO awscrt::AuthCredentialsProvider: TLS context not provided, initializing a new one for querying STS 2024-06-28T08:08:17.070487Z DEBUG awscrt::dns: static: Either no requests have been made for an address for 169.254.169.254 for the duration of the ttl, or this thread is being forcibly shutdown. Killing thread. 2024-06-28T08:08:17.070694Z DEBUG awscrt::dns: static: purging address 169.254.169.254 for host 169.254.169.254 from the cache due to cache eviction or shutdown 2024-06-28T08:08:17.072474Z INFO awscrt::AuthCredentialsProvider: static: credential_source property set to Ec2InstanceMetadata 2024-06-28T08:08:17.072490Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: acquiring bootstrap reference 2024-06-28T08:08:17.072520Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: Successfully created 2024-06-28T08:08:17.072541Z INFO awscrt::exp-backoff-strategy: id=0x55bc5eb3a4c0: Initializing exponential backoff retry strategy with scale factor: 0 jitter mode: 0 and max retries 1 2024-06-28T08:08:17.072553Z DEBUG awscrt::AuthCredentialsProvider: static: creating STS credentials provider 2024-06-28T08:08:17.072571Z DEBUG awscrt::AuthCredentialsProvider: (id=0x55bc5eb3a540): using session_name aws-common-runtime-profile-config-21339 2024-06-28T08:08:17.072581Z DEBUG awscrt::AuthCredentialsProvider: (id=0x55bc5eb3a540): using assume_role_arn arn:aws:iam::123456789012:role/my-role-name 2024-06-28T08:08:17.072616Z DEBUG awscrt::AuthCredentialsProvider: (id=0x55bc5eb3a540): using credentials duration 900 2024-06-28T08:08:17.072639Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/home/admin/.aws/config" 2024-06-28T08:08:17.072690Z DEBUG awscrt::AuthCredentialsProvider: Successfully built config profile collection from file at (/home/admin/.aws/config) 2024-06-28T08:08:17.072708Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: acquiring bootstrap reference 2024-06-28T08:08:17.072719Z INFO awscrt::connection-manager: id=0x55bc5eb3b710: Successfully created 2024-06-28T08:08:17.072729Z INFO awscrt::standard-retry-strategy: static: creating new standard retry strategy 2024-06-28T08:08:17.072739Z INFO awscrt::standard-retry-strategy: id=0x55bc5eb3bd50: creating backing exponential backoff strategy with max_retries of 3 2024-06-28T08:08:17.072750Z INFO awscrt::exp-backoff-strategy: id=0x55bc5eb3bec0: Initializing exponential backoff retry strategy with scale factor: 0 jitter mode: 0 and max retries 3 2024-06-28T08:08:17.072767Z DEBUG awscrt::standard-retry-strategy: id=0x55bc5eb3bd50: maximum bucket capacity set to 500 2024-06-28T08:08:17.072848Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: acquiring bootstrap reference 2024-06-28T08:08:17.074492Z INFO awscrt::event-loop: id=0x55bc5ea040d0: main loop started 2024-06-28T08:08:17.074661Z INFO awscrt::event-loop: id=0x55bc5e9e6e20: Destroying event_loop 2024-06-28T08:08:17.074843Z INFO awscrt::event-loop: id=0x55bc5e9e6e20: Stopping event-loop thread. 2024-06-28T08:08:17.075033Z DEBUG awscrt::event-loop: id=0x55bc5e9e6e20: exiting main loop 2024-06-28T08:08:17.078153Z INFO awscrt::event-loop: id=0x55bc5ec10450: Initializing edge-triggered epoll 2024-06-28T08:08:17.078168Z INFO awscrt::event-loop: id=0x55bc5ec10450: Using eventfd for cross-thread notifications. 2024-06-28T08:08:17.078179Z INFO awscrt::event-loop: id=0x55bc5ec10450: Starting event-loop thread. 2024-06-28T08:08:17.078466Z INFO awscrt::event-loop: id=0x55bc5ea040d0: default timeout 100000, and max events to process per tick 100 2024-06-28T08:08:17.078664Z INFO awscrt::event-loop: id=0x55bc5ec10450: main loop started 2024-06-28T08:08:17.078841Z INFO awscrt::event-loop: id=0x55bc5ec10450: default timeout 100000, and max events to process per tick 100 2024-06-28T08:08:17.079625Z DEBUG awscrt::AWSEndpointsResolution: Resolved endpoint with status 0 2024-06-28T08:08:17.079673Z DEBUG list_objects{id=0 bucket="my-bucket-name" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client::list_objects: new request 2024-06-28T08:08:17.079695Z INFO awscrt::S3Client: id=0x55bc5eb3c910 Initiating making of meta request 2024-06-28T08:08:17.079711Z DEBUG awscrt::S3MetaRequest: id=0x55bc5ecf5620 Created new Default Meta Request. operation=? 2024-06-28T08:08:17.079725Z DEBUG awscrt::dns: id=0x55bc5e9f6470: Host resolution requested for my-bucket-name.s3.eu-central-1.amazonaws.com 2024-06-28T08:08:17.079735Z DEBUG awscrt::dns: id=0x55bc5e9f6470: No cached entries found for my-bucket-name.s3.eu-central-1.amazonaws.com starting new resolver thread. 2024-06-28T08:08:17.079787Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: acquiring bootstrap reference 2024-06-28T08:08:17.079806Z INFO awscrt::connection-manager: id=0x55bc5ed02a70: Successfully created 2024-06-28T08:08:17.079825Z DEBUG awscrt::S3Endpoint: id=0x55bc5ecf5cd0: Created connection manager 0x55bc5ed02a70 for endpoint 2024-06-28T08:08:17.079848Z INFO awscrt::S3Client: id=0x55bc5eb3c910: Created meta request 0x55bc5ecf5620 2024-06-28T08:08:17.079893Z DEBUG awscrt::dns: static: resolving host my-bucket-name.s3.eu-central-1.amazonaws.com 2024-06-28T08:08:17.081884Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data. 2024-06-28T08:08:17.081936Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Processing any new meta requests. 2024-06-28T08:08:17.082017Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating meta requests. 2024-06-28T08:08:17.082077Z DEBUG awscrt::S3MetaRequest: id=0x55bc5ecf5620: Meta Request Default created request 0x7fb7d80030d0 2024-06-28T08:08:17.082125Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating connections, assigning requests where possible. 2024-06-28T08:08:17.082159Z DEBUG awscrt::S3MetaRequest: id=0x55bc5ecf5620: Meta Request prepared request 0x7fb7d80030d0 2024-06-28T08:08:17.082167Z INFO awscrt::S3ClientStats: id=0x55bc5eb3c910 Requests-in-flight(approx/exact):1/1 Requests-preparing:1 Requests-queued:0 Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0 Requests-streaming-response:0 Endpoints(in-table/allocated):1/1 2024-06-28T08:08:17.082219Z DEBUG awscrt::AuthCredentialsProvider: (id=0x55bc5eb3a540): fetching credentials 2024-06-28T08:08:17.082253Z DEBUG awscrt::standard-retry-strategy: id=0x55bc5eb3bd50: attempting to acquire retry token for partition_id 2024-06-28T08:08:17.082266Z DEBUG awscrt::standard-retry-strategy: id=0x55bc5eb3bd50: bucket for partition_id does not exist, attempting to create one 2024-06-28T08:08:17.082273Z DEBUG awscrt::standard-retry-strategy: id=0x55bc5eb3bd50: bucket 0x7fb7cc004790 for partition_id created 2024-06-28T08:08:17.082283Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3bec0: Initializing retry token 0x7fb7cc004b90 2024-06-28T08:08:17.082335Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3bec0: Vending retry_token 0x7fb7cc004b90 2024-06-28T08:08:17.082384Z DEBUG awscrt::standard-retry-strategy: id=0x55bc5eb3bd50: token acquired callback invoked with error Success. with token 0x7fb7cc004380 and nested token 0x7fb7cc004b90 2024-06-28T08:08:17.082569Z DEBUG awscrt::AuthCredentialsProvider: id=0x55bc5eb2d030: IMDS provider trying to load credentials 2024-06-28T08:08:17.084345Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3a4c0: Initializing retry token 0x7fb7d8006480 2024-06-28T08:08:17.084393Z DEBUG awscrt::Unknown: (id=0x55bc5eb2d370) IMDS client's token is invalid and is now updating. 2024-06-28T08:08:17.084433Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3a4c0: Vending retry_token 0x7fb7d8006480 2024-06-28T08:08:17.084470Z DEBUG awscrt::Unknown: id=0x55bc5eb2d370: IMDS Client successfully acquired retry token. 2024-06-28T08:08:17.084505Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Acquire connection 2024-06-28T08:08:17.084546Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=0, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1 2024-06-28T08:08:17.084585Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: Requesting 1 new connections from http 2024-06-28T08:08:17.084622Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: acquiring bootstrap reference 2024-06-28T08:08:17.084658Z DEBUG awscrt::dns: id=0x55bc5e9f6470: Host resolution requested for 169.254.169.254 2024-06-28T08:08:17.084693Z DEBUG awscrt::dns: id=0x55bc5e9f6470: No cached entries found for 169.254.169.254 starting new resolver thread. 2024-06-28T08:08:17.084797Z DEBUG awscrt::dns: static: resolving host 169.254.169.254 2024-06-28T08:08:17.084816Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254 2024-06-28T08:08:17.084823Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses 2024-06-28T08:08:17.084835Z DEBUG awscrt::dns: static: new address resolved 169.254.169.254 for host 169.254.169.254 caching 2024-06-28T08:08:17.084846Z DEBUG awscrt::dns: static, invoking resolution callback for host 169.254.169.254 with 1 addresses 2024-06-28T08:08:17.084903Z DEBUG awscrt::socket: id=0x7fb7d8008320 fd=11: initializing with domain 0 and type 0 2024-06-28T08:08:17.084950Z DEBUG awscrt::socket: id=0x7fb7d8008320 fd=11: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0. 2024-06-28T08:08:17.085018Z DEBUG awscrt::socket: id=0x7fb7d8008320 fd=11: beginning connect. 2024-06-28T08:08:17.085057Z DEBUG awscrt::socket: id=0x7fb7d8008320 fd=11: connecting to endpoint 169.254.169.254:80. 2024-06-28T08:08:17.086524Z INFO awscrt::socket: id=0x7fb7d8008320 fd=11: connection success 2024-06-28T08:08:17.086539Z DEBUG awscrt::socket: id=0x7fb7d8008320 fd=11: assigning to event loop 0x55bc5ea040d0 2024-06-28T08:08:17.086551Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: client connection on socket 0x7fb7d8008320 completed with error 0. 2024-06-28T08:08:17.086561Z DEBUG awscrt::channel: id=0x7fb7d8008df0: Beginning creation and setup of new channel. 2024-06-28T08:08:17.086570Z DEBUG awscrt::channel: id=0x7fb7d8008df0: setup complete, notifying caller. 2024-06-28T08:08:17.086582Z DEBUG awscrt::channel: id=0x7fb7d8008df0: no message pool is currently stored in the event-loop local storage, adding 0x7fb7d80067c0 with max message size 16384, message count 4, with 4 small blocks of 128 bytes. 2024-06-28T08:08:17.086602Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: channel 0x7fb7d8008df0 setup succeeded: bootstrapping. 2024-06-28T08:08:17.086612Z DEBUG awscrt::socket-handler: id=0x7fb7d8009290: Socket handler created with max_read_size of 16384 2024-06-28T08:08:17.086625Z INFO awscrt::http-connection: id=0x7fb7d80198e0: HTTP/1.1 client connection established. 2024-06-28T08:08:17.086635Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Received new connection (id=0x7fb7d80198e0) from http layer 2024-06-28T08:08:17.086644Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Grabbing pooled connection (0x7fb7d80198e0) 2024-06-28T08:08:17.086657Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.086666Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Successfully completed connection acquisition with connection id=0x7fb7d80198e0 2024-06-28T08:08:17.086682Z DEBUG awscrt::http-stream: id=0x7fb7d801a7f0: Created client request on connection=0x7fb7d80198e0: PUT /latest/api/token HTTP/1.1 2024-06-28T08:08:17.086940Z DEBUG awscrt::dns: static: resolved record: 52.219.171.214 2024-06-28T08:08:17.086955Z DEBUG awscrt::dns: static: resolved record: 3.5.134.190 2024-06-28T08:08:17.086963Z DEBUG awscrt::dns: static: resolved record: 52.219.169.170 2024-06-28T08:08:17.086972Z DEBUG awscrt::dns: static: resolved record: 3.5.138.137 2024-06-28T08:08:17.086978Z DEBUG awscrt::dns: static: resolved record: 52.219.47.41 2024-06-28T08:08:17.086988Z DEBUG awscrt::dns: static: resolved record: 52.219.169.158 2024-06-28T08:08:17.086997Z DEBUG awscrt::dns: static: resolved record: 3.5.139.118 2024-06-28T08:08:17.087006Z DEBUG awscrt::dns: static: resolved record: 3.5.137.167 2024-06-28T08:08:17.087017Z DEBUG awscrt::dns: static, resolving host my-bucket-name.s3.eu-central-1.amazonaws.com successful, returned 8 addresses 2024-06-28T08:08:17.087027Z DEBUG awscrt::dns: static: new address resolved 52.219.171.214 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087037Z DEBUG awscrt::dns: static: new address resolved 3.5.134.190 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087047Z DEBUG awscrt::dns: static: new address resolved 52.219.169.170 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087056Z DEBUG awscrt::dns: static: new address resolved 3.5.138.137 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087066Z DEBUG awscrt::dns: static: new address resolved 52.219.47.41 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087076Z DEBUG awscrt::dns: static: new address resolved 52.219.169.158 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087086Z DEBUG awscrt::dns: static: new address resolved 3.5.139.118 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087096Z DEBUG awscrt::dns: static: new address resolved 3.5.137.167 for host my-bucket-name.s3.eu-central-1.amazonaws.com caching 2024-06-28T08:08:17.087108Z DEBUG awscrt::dns: static, invoking resolution callback for host my-bucket-name.s3.eu-central-1.amazonaws.com with 1 addresses 2024-06-28T08:08:17.087164Z DEBUG awscrt::Unknown: (id=0x55bc5eb2d370) IMDS client query received http status code 404 for requester 0x7fb7d8005320. 2024-06-28T08:08:17.087214Z DEBUG awscrt::http-stream: id=0x7fb7d801a7f0: Client request complete, response status: 404 (Not Found). 2024-06-28T08:08:17.087253Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: User releasing connection (id=0x7fb7d80198e0) 2024-06-28T08:08:17.087296Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.087336Z DEBUG awscrt::Unknown: (id=0x55bc5eb2d370) IMDS client failed to fetch token for requester 0x7fb7d8005320, fall back to v1 for the same requester. Received response status code: 404 2024-06-28T08:08:17.087373Z DEBUG awscrt::Unknown: (id=0x55bc5eb2d370) IMDS client failed to fetch token, fallback to v1. requester 0x7fb7d8005280. 2024-06-28T08:08:17.087409Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3a4c0: Initializing retry token 0x7fb7d801b2d0 2024-06-28T08:08:17.087447Z ERROR awscrt::Unknown: (id=0x55bc5eb2d370) IMDS client failed to update the token from IMDS. 2024-06-28T08:08:17.087488Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3a4c0: Vending retry_token 0x7fb7d801b2d0 2024-06-28T08:08:17.087523Z DEBUG awscrt::Unknown: id=0x55bc5eb2d370: IMDS Client successfully acquired retry token. 2024-06-28T08:08:17.087559Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Acquire connection 2024-06-28T08:08:17.087595Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Grabbing pooled connection (0x7fb7d80198e0) 2024-06-28T08:08:17.087632Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.090493Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Successfully completed connection acquisition with connection id=0x7fb7d80198e0 2024-06-28T08:08:17.090520Z DEBUG awscrt::http-stream: id=0x7fb7d801b970: Created client request on connection=0x7fb7d80198e0: GET /latest/meta-data/iam/security-credentials/ HTTP/1.1 2024-06-28T08:08:17.090700Z DEBUG awscrt::Unknown: (id=0x55bc5eb2d370) IMDS client query received http status code 404 for requester 0x7fb7d8005280. 2024-06-28T08:08:17.090721Z DEBUG awscrt::http-stream: id=0x7fb7d801b970: Client request complete, response status: 404 (Not Found). 2024-06-28T08:08:17.090734Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: User releasing connection (id=0x7fb7d80198e0) 2024-06-28T08:08:17.090745Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.090758Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3a4c0: Initializing retry token 0x7fb7d8005c70 2024-06-28T08:08:17.090773Z DEBUG awscrt::exp-backoff-strategy: id=0x55bc5eb3a4c0: Vending retry_token 0x7fb7d8005c70 2024-06-28T08:08:17.090783Z DEBUG awscrt::Unknown: id=0x55bc5eb2d370: IMDS Client successfully acquired retry token. 2024-06-28T08:08:17.090793Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Acquire connection 2024-06-28T08:08:17.090802Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Grabbing pooled connection (0x7fb7d80198e0) 2024-06-28T08:08:17.090812Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.090822Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: Successfully completed connection acquisition with connection id=0x7fb7d80198e0 2024-06-28T08:08:17.090833Z ERROR awscrt::http-stream: id=static: Request path is invalid 2024-06-28T08:08:17.090847Z ERROR awscrt::http-connection: id=0x7fb7d80198e0: Cannot create request stream, error 2054 (AWS_ERROR_HTTP_INVALID_PATH) 2024-06-28T08:08:17.090866Z INFO awscrt::AuthCredentialsProvider: id=0x55bc5eb2d030: IMDS provider failed to retrieve credentials: Path is invalid. 2024-06-28T08:08:17.090883Z ERROR awscrt::AuthSigning: (id=0x7fb7d8001490) Credentials Provider failed to source credentials with error 2054(aws-c-http: AWS_ERROR_HTTP_INVALID_PATH, Path is invalid.) 2024-06-28T08:08:17.090893Z DEBUG awscrt::AuthCredentialsProvider: (id=0x55bc5eb3a540): signing completed with error code 6146 2024-06-28T08:08:17.090909Z ERROR awscrt::AuthSigning: (id=0x7fb7cc002570) Credentials Provider failed to source credentials with error 6146(aws-c-auth: AWS_AUTH_SIGNING_NO_CREDENTIALS, Attempt to sign an http request without credentials) 2024-06-28T08:08:17.090920Z ERROR awscrt::S3MetaRequest: id=0x55bc5ecf5620 Meta request could not sign HTTP request due to error code 6146 (Attempt to sign an http request without credentials) 2024-06-28T08:08:17.090930Z ERROR awscrt::S3MetaRequest: id=0x55bc5ecf5620 Could not prepare request 0x7fb7d80030d0 due to error 6146 (Attempt to sign an http request without credentials). 2024-06-28T08:08:17.090980Z DEBUG list_objects{id=0 bucket="my-bucket-name" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client: S3 request failed request_type=Default crt_error=None range=None duration=8.795349ms ttfb=None request_id=<unknown> 2024-06-28T08:08:17.091041Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: User releasing connection (id=0x7fb7d80198e0) 2024-06-28T08:08:17.091064Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1 2024-06-28T08:08:17.091087Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data. 2024-06-28T08:08:17.091103Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Processing any new meta requests. 2024-06-28T08:08:17.091120Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating meta requests. 2024-06-28T08:08:17.091141Z DEBUG awscrt::S3MetaRequest: id=0x55bc5ecf5620 Meta request finished with error code 6146 (Attempt to sign an http request without credentials) 2024-06-28T08:08:17.091190Z WARN list_objects{id=0 bucket="my-bucket-name" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client: duration=11.465463ms request_id=<unknown> error=ClientError(NoSigningCredentials) meta request failed 2024-06-28T08:08:17.091210Z DEBUG list_objects{id=0 bucket="my-bucket-name" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client: meta request result: MetaRequestResult { response_status: 0, crt_error: Error(6146, "aws-c-auth: AWS_AUTH_SIGNING_NO_CREDENTIALS, Attempt to sign an http request without credentials"), error_response_headers: None, error_response_body: None } 2024-06-28T08:08:17.091238Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: releasing bootstrap reference 2024-06-28T08:08:17.091267Z DEBUG awscrt::S3MetaRequest: id=0x55bc5ecf5620 Cleaning up meta request 2024-06-28T08:08:17.091279Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Client starting destruction. 2024-06-28T08:08:17.091326Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating connections, assigning requests where possible. 2024-06-28T08:08:17.091341Z INFO awscrt::S3ClientStats: id=0x55bc5eb3c910 Requests-in-flight(approx/exact):0/0 Requests-preparing:0 Requests-queued:0 Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0 Requests-streaming-response:0 Endpoints(in-table/allocated):1/1 2024-06-28T08:08:17.091355Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Client shutdown progress: starting_destroy_executing=0 body_streaming_elg_allocated=1 process_work_task_scheduled=1 process_work_task_in_progress=0 num_endpoints_allocated=1 s3express_provider_active=0 finish_destroy=0 2024-06-28T08:08:17.091369Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data. 2024-06-28T08:08:17.091381Z INFO awscrt::connection-manager: id=0x55bc5ed02a70: release 2024-06-28T08:08:17.091391Z INFO awscrt::connection-manager: id=0x55bc5ed02a70: ref count now zero, starting shut down process 2024-06-28T08:08:17.091400Z INFO awscrt::connection-manager: id=0x55bc5ed02a70: manager release, failing 0 pending acquisitions 2024-06-28T08:08:17.091412Z DEBUG awscrt::connection-manager: id=0x55bc5ed02a70: snapshot - state=2, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=0 2024-06-28T08:08:17.091422Z INFO awscrt::connection-manager: id=0x55bc5ed02a70: Destroying self 2024-06-28T08:08:17.091433Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: releasing bootstrap reference 2024-06-28T08:08:17.091450Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Processing any new meta requests. 2024-06-28T08:08:17.091470Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating meta requests. 2024-06-28T08:08:17.091483Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating connections, assigning requests where possible. 2024-06-28T08:08:17.091496Z INFO awscrt::S3ClientStats: id=0x55bc5eb3c910 Requests-in-flight(approx/exact):0/0 Requests-preparing:0 Requests-queued:0 Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0 Requests-streaming-response:0 Endpoints(in-table/allocated):0/0 2024-06-28T08:08:17.091508Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Client shutdown progress: starting_destroy_executing=0 body_streaming_elg_allocated=1 process_work_task_scheduled=1 process_work_task_in_progress=0 num_endpoints_allocated=0 s3express_provider_active=0 finish_destroy=0 2024-06-28T08:08:17.091519Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data. 2024-06-28T08:08:17.091528Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Processing any new meta requests. 2024-06-28T08:08:17.091537Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating meta requests. 2024-06-28T08:08:17.091545Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating connections, assigning requests where possible. 2024-06-28T08:08:17.091556Z INFO awscrt::S3ClientStats: id=0x55bc5eb3c910 Requests-in-flight(approx/exact):0/0 Requests-preparing:0 Requests-queued:0 Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0 Requests-streaming-response:0 Endpoints(in-table/allocated):0/0 2024-06-28T08:08:17.091568Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Client shutdown progress: starting_destroy_executing=0 body_streaming_elg_allocated=1 process_work_task_scheduled=0 process_work_task_in_progress=0 num_endpoints_allocated=0 s3express_provider_active=0 finish_destroy=0 2024-06-28T08:08:17.091604Z INFO mountpoint_s3::metrics: process.memory_usage: 13590528 2024-06-28T08:08:17.091628Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.mem_limit: 2013265920 2024-06-28T08:08:17.091608Z INFO awscrt::event-loop: id=0x55bc5ec10450: Destroying event_loop 2024-06-28T08:08:17.091643Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_allocated: 0 2024-06-28T08:08:17.091670Z INFO awscrt::event-loop: id=0x55bc5ec10450: Stopping event-loop thread. 2024-06-28T08:08:17.091673Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_cutoff: 33554432 2024-06-28T08:08:17.091691Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_num_blocks: 0 2024-06-28T08:08:17.091695Z DEBUG awscrt::event-loop: id=0x55bc5ec10450: exiting main loop 2024-06-28T08:08:17.091707Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_reserved: 0 2024-06-28T08:08:17.091720Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_used: 0 2024-06-28T08:08:17.091731Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.secondary_reserved: 0 2024-06-28T08:08:17.091748Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Client body streaming ELG shutdown. 2024-06-28T08:08:17.091768Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.secondary_used: 0 2024-06-28T08:08:17.091775Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data. 2024-06-28T08:08:17.091788Z INFO mountpoint_s3::metrics: s3.client.host_count[host=my-bucket-name.s3.eu-central-1.amazonaws.com]: 8 2024-06-28T08:08:17.091792Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 s_s3_client_process_work_default - Processing any new meta requests. 2024-06-28T08:08:17.091811Z INFO mountpoint_s3::metrics: s3.client.num_auto_default_network_io: 0 2024-06-28T08:08:17.091820Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating meta requests. 2024-06-28T08:08:17.091836Z INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_copy_network_io: 0 2024-06-28T08:08:17.091845Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Updating connections, assigning requests where possible. 2024-06-28T08:08:17.091861Z INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_get_network_io: 0 2024-06-28T08:08:17.091868Z INFO awscrt::S3ClientStats: id=0x55bc5eb3c910 Requests-in-flight(approx/exact):0/0 Requests-preparing:0 Requests-queued:0 Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0 Requests-streaming-response:0 Endpoints(in-table/allocated):0/0 2024-06-28T08:08:17.091881Z INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_put_network_io: 0 2024-06-28T08:08:17.091885Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Client shutdown progress: starting_destroy_executing=0 body_streaming_elg_allocated=0 process_work_task_scheduled=0 process_work_task_in_progress=0 num_endpoints_allocated=0 s3express_provider_active=0 finish_destroy=1 2024-06-28T08:08:17.091902Z INFO mountpoint_s3::metrics: s3.client.num_requests_being_prepared: 0 2024-06-28T08:08:17.091909Z DEBUG awscrt::S3Client: id=0x55bc5eb3c910 Client finishing destruction. 2024-06-28T08:08:17.091921Z INFO mountpoint_s3::metrics: s3.client.num_requests_being_processed: 0 2024-06-28T08:08:17.091932Z INFO mountpoint_s3::metrics: s3.client.num_requests_stream_queued_waiting: 0 2024-06-28T08:08:17.091943Z INFO mountpoint_s3::metrics: s3.client.num_requests_streaming_response: 0 2024-06-28T08:08:17.091954Z INFO mountpoint_s3::metrics: s3.client.num_total_network_io: 0 2024-06-28T08:08:17.091964Z INFO mountpoint_s3::metrics: s3.client.request_queue_size: 0 2024-06-28T08:08:17.091975Z INFO mountpoint_s3::metrics: s3.meta_requests.failures[op=list_objects,status=-6146]: 1 2024-06-28T08:08:17.091986Z INFO mountpoint_s3::metrics: s3.meta_requests.first_byte_latency_us[op=list_objects]: n=1: min=11456 p10=11519 p50=11519 avg=11488.00 p90=11519 p99=11519 p99.9=11519 max=11519 2024-06-28T08:08:17.091997Z INFO mountpoint_s3::metrics: s3.meta_requests.total_latency_us[op=list_objects]: n=1: min=11456 p10=11519 p50=11519 avg=11488.00 p90=11519 p99=11519 p99.9=11519 max=11519 2024-06-28T08:08:17.092008Z INFO mountpoint_s3::metrics: s3.meta_requests[op=list_objects]: 1 2024-06-28T08:08:17.092018Z INFO mountpoint_s3::metrics: s3.requests.failures[op=list_objects,type=Default,status=-1]: 1 2024-06-28T08:08:17.092029Z INFO mountpoint_s3::metrics: s3.requests.total_latency_us[op=list_objects,type=Default]: n=1: min=8768 p10=8831 p50=8831 avg=8800.00 p90=8831 p99=8831 p99.9=8831 max=8831 2024-06-28T08:08:17.092040Z INFO mountpoint_s3::metrics: s3.requests[op=list_objects,type=Default]: 1 2024-06-28T08:08:17.092054Z INFO mountpoint_s3::metrics: system.available_memory: 1548193792 2024-06-28T08:08:17.092555Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: releasing bootstrap reference 2024-06-28T08:08:17.092575Z INFO awscrt::connection-manager: id=0x55bc5eb3b710: release 2024-06-28T08:08:17.092586Z INFO awscrt::connection-manager: id=0x55bc5eb3b710: ref count now zero, starting shut down process 2024-06-28T08:08:17.092592Z INFO awscrt::connection-manager: id=0x55bc5eb3b710: manager release, failing 0 pending acquisitions 2024-06-28T08:08:17.092603Z DEBUG awscrt::connection-manager: id=0x55bc5eb3b710: snapshot - state=2, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=0 2024-06-28T08:08:17.092614Z INFO awscrt::connection-manager: id=0x55bc5eb3b710: Destroying self 2024-06-28T08:08:17.093321Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: releasing bootstrap reference 2024-06-28T08:08:17.093337Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: release 2024-06-28T08:08:17.093355Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: ref count now zero, starting shut down process 2024-06-28T08:08:17.093365Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: manager release, failing 0 pending acquisitions 2024-06-28T08:08:17.093374Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=2, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=0 2024-06-28T08:08:17.093384Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: Releasing connection (id=0x7fb7d80198e0) 2024-06-28T08:08:17.093396Z DEBUG awscrt::channel: id=0x7fb7d8008df0: beginning shutdown process 2024-06-28T08:08:17.093405Z DEBUG awscrt::channel: id=0x7fb7d8008df0: handler 0x7fb7d8009290 shutdown in read dir completed. 2024-06-28T08:08:17.093414Z DEBUG awscrt::channel: id=0x7fb7d8008df0: handler 0x7fb7d80198e8 shutdown in read dir completed. 2024-06-28T08:08:17.093423Z DEBUG awscrt::channel: id=0x7fb7d8008df0: handler 0x7fb7d80198e8 shutdown in write dir completed. 2024-06-28T08:08:17.093432Z DEBUG awscrt::socket: id=0x7fb7d8008320 fd=11: closing 2024-06-28T08:08:17.093465Z DEBUG awscrt::channel: id=0x7fb7d8008df0: handler 0x7fb7d8009290 shutdown in write dir completed. 2024-06-28T08:08:17.093483Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: channel 0x7fb7d8008df0 shutdown with error 0. 2024-06-28T08:08:17.093496Z INFO awscrt::http-connection: 0x7fb7d80198e0: Client shutdown completed with error 0 (AWS_ERROR_SUCCESS). 2024-06-28T08:08:17.093508Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: shutdown received for connection (id=0x7fb7d80198e0) 2024-06-28T08:08:17.093517Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: manager release, failing 0 pending acquisitions 2024-06-28T08:08:17.093524Z DEBUG awscrt::connection-manager: id=0x55bc5eb2d480: snapshot - state=2, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=0 2024-06-28T08:08:17.093533Z INFO awscrt::connection-manager: id=0x55bc5eb2d480: Destroying self 2024-06-28T08:08:17.093538Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: releasing bootstrap reference 2024-06-28T08:08:17.093545Z DEBUG awscrt::channel: id=0x7fb7d8008df0: destroying channel. 2024-06-28T08:08:17.093552Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: releasing bootstrap reference 2024-06-28T08:08:17.093559Z DEBUG awscrt::channel-bootstrap: id=0x55bc5e9e4bc0: bootstrap destroying Error: Failed to create S3 client Caused by: 0: initial ListObjectsV2 failed for bucket my-bucket-name in region eu-central-1 1: Client error 2: No signing credentials foundThe text was updated successfully, but these errors were encountered: