feat: Add AWS backend

[clemlesne]

No description provided.

[Copilot]

Copilot reviewed 8 out of 23 changed files in this pull request and generated 1 comment.

Files not reviewed (15)

• .env.example: Language not supported
• .vscode/settings.json: Language not supported
• Makefile: Language not supported
• cicd/test-unit-ci.sh: Language not supported
• .github/workflows/pipeline.yaml: Evaluated as low risk
• src/scrape_it_now/models/message.py: Evaluated as low risk
• src/scrape_it_now/index.py: Evaluated as low risk
• src/scrape_it_now/persistence/local_disk.py: Evaluated as low risk
• src/scrape_it_now/persistence/azure_queue_storage.py: Evaluated as low risk
• src/scrape_it_now/scrape.py: Evaluated as low risk
• src/scrape_it_now/helpers/persistence.py: Evaluated as low risk
• tests/blob.py: Evaluated as low risk
• src/scrape_it_now/cli.py: Evaluated as low risk
• src/scrape_it_now/persistence/iblob.py: Evaluated as low risk
• src/scrape_it_now/persistence/iqueue.py: Evaluated as low risk

Comments suppressed due to low confidence (2)

src/scrape_it_now/persistence/aws_s3.py:286

• The __aenter__ method tries to create a bucket using get_session().create_client(**client_kwargs).create_bucket(Bucket=self._config.name), which might lead to race conditions or unnecessary bucket creation attempts. Consider handling bucket creation separately or ensuring it is only done when necessary.

get_session().create_client(**client_kwargs).create_bucket(Bucket=self._config.name)

src/scrape_it_now/persistence/aws_s3.py:80

• The lease_blob method has a recursive call to itself which might lead to potential issues if not handled carefully. Consider refactoring to avoid recursion or ensure proper handling of recursive calls.

async with self.lease_blob(blob=blob, lease_duration=lease_duration) as lease_id:

[Copilot]

Copilot reviewed 8 out of 23 changed files in this pull request and generated 3 comments.

Files not reviewed (15)

• .env.example: Language not supported
• .vscode/settings.json: Language not supported
• Makefile: Language not supported
• cicd/test-unit-ci.sh: Language not supported
• .github/workflows/pipeline.yaml: Evaluated as low risk
• src/scrape_it_now/index.py: Evaluated as low risk
• src/scrape_it_now/models/message.py: Evaluated as low risk
• src/scrape_it_now/scrape.py: Evaluated as low risk
• src/scrape_it_now/persistence/azure_queue_storage.py: Evaluated as low risk
• src/scrape_it_now/persistence/local_disk.py: Evaluated as low risk
• src/scrape_it_now/helpers/persistence.py: Evaluated as low risk
• tests/blob.py: Evaluated as low risk
• src/scrape_it_now/persistence/iqueue.py: Evaluated as low risk
• pyproject.toml: Evaluated as low risk
• src/scrape_it_now/persistence/iblob.py: Evaluated as low risk

Comments suppressed due to low confidence (4)

src/scrape_it_now/persistence/aws_s3.py:114

• [nitpick] The argument name 'lease_id' could be more descriptive, such as 'existing_lease_id', to clarify its purpose.

lease_id: str | None = None

src/scrape_it_now/persistence/aws_s3.py:78

• The retry mechanism in the 'lease_blob' method should be covered by tests to ensure it works as expected under different scenarios.

await asyncio.sleep(0.1)

src/scrape_it_now/persistence/aws_s3.py:224

• The timeout handling in the '_file_lock' method should be covered by tests to ensure it works as expected under different scenarios.

seconds=timeout

src/scrape_it_now/persistence/aws_sqs.py:65

• [nitpick] The variable name 'pulled_now' is ambiguous. It should be renamed to 'messages_pulled_now'.

pulled_now = min(max_messages - total_pulled, max_per_batch)

[Copilot]

The default value for 'aws_sqs_region' might lead to unintended behavior if the user is not aware of it. Consider removing the default value to ensure the user explicitly sets the region.

Suggested change

	default="eu-west-1",
	envvar="AWS_SQS_REGION",

Copilot reviewed 9 out of 24 changed files in this pull request and generated no comments.

Files not reviewed (15)

• .env.example: Language not supported
• .vscode/settings.json: Language not supported
• Makefile: Language not supported
• cicd/test-unit-ci.sh: Language not supported
• .github/workflows/pipeline.yaml: Evaluated as low risk
• src/scrape_it_now/models/message.py: Evaluated as low risk
• src/scrape_it_now/index.py: Evaluated as low risk
• src/scrape_it_now/persistence/local_disk.py: Evaluated as low risk
• src/scrape_it_now/scrape.py: Evaluated as low risk
• src/scrape_it_now/helpers/http.py: Evaluated as low risk
• src/scrape_it_now/persistence/azure_queue_storage.py: Evaluated as low risk
• src/scrape_it_now/helpers/persistence.py: Evaluated as low risk
• src/scrape_it_now/cli.py: Evaluated as low risk
• pyproject.toml: Evaluated as low risk
• src/scrape_it_now/persistence/iblob.py: Evaluated as low risk

Comments suppressed due to low confidence (4)

src/scrape_it_now/persistence/aws_s3.py:273

• The _file_lock method should handle stale lock files to prevent deadlock situations. Consider adding a mechanism to handle stale lock files.

while await self._exists(lock_file):

src/scrape_it_now/persistence/aws_s3.py:341

• The __aenter__ method should check if the S3 bucket creation is successful. Add a check to ensure that the bucket is created successfully.

get_session().create_client(**client_kwargs).create_bucket(

src/scrape_it_now/persistence/aws_sqs.py:130

• The error message should be more descriptive. Consider changing it to 'Receipt handle for message "{message.message_id}" is invalid'.

f'Message "{message.message_id}" not found'

src/scrape_it_now/persistence/aws_sqs.py:60

• The send_message method should have a test case to verify its behavior.

async def send_message(