Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert to last know good docker build #668

Open
wants to merge 27 commits into
base: bkou_revert2
Choose a base branch
from
Open

Conversation

sfc-gh-bkou
Copy link

No description provided.

sfc-gh-afedorov and others added 27 commits May 14, 2024 09:20
* Remove vestigial logging

no need to log because these errors are recorded into the "error" column in the corresponding table

* Add retry and update aio* dependencies

Update aiohttp, aioboto3, and aiobotocore dependencies and add retry logic to sts.assume_role invocation.
* Update base image to Debian bookworm

* Update to py38
* Update Python and pip dependencies & fix tests

* fix share db view creation in installer script

Snowflake added a new column so the share db name is col4 now
…ory (#671)

* Tag image with commit SHA after build, and push to repository

* updates to contibution file
fixing error
> botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve container metadata: Connect timeout on endpoint URL: "http://<ip>/v2/credentials/<uuid>"

per https://repost.aws/questions/QUCFqv7OfoQlygJrmwfkJ24Q/various-aws-apis-fail-due-to-timeout
…ed'` error (#675)

* Update azure_collect.py to fix `KeyError: 'managedByExtended'` error

* Update v1_9_6-v1_10_0.md
getting throttled at metadata service returning header "X-Rate-Limit-Limit: 40.00"
* fix awsic rate limiting and OOM errors

- rate limit metadata separately
- prune cache to fix oom errors
- remove / abstract out inline retries

* optimizations

- use boto's retry mechanism for 429 responses
- cache AssumeRole failures indefinitely
- re-use sessions between regions
- refactor API rate limits as AioRateLimit instances
- increase batch size to 1000

* add pause in pagination
Putting them last because they currently time out after a few days of ingestion
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants