S3 bucket.add_event_notification() does not deploy - No resource provider found for "Custom::S3BucketNotifications"

[robertchinezon]

ENV

CDK-2.97.0
CDK-Python
Python 3.10.12

Problem

As the title suggests, calling add_event_notification() with any target results in the following error as seen towards the end of the log file:

2023-09-28T22:41:21.978  WARN --- [uncthread128] l.s.c.deployment_utils :  No resource provider found for "Custom::S3BucketNotifications". 

localstack.log

If I set the notification through awslocal cli:

awslocal s3api put-bucket-notification-configuration --bucket bucket --notification-configuration file://notification.json

all my code works properly.

Code

"""AWS stack that sends an sqs notification when files land in a s3 bucket."""

import logging
import os

# from utils.utils import enable_my_environment, progress_bar
from aws_cdk import (
    CfnOutput,
    Duration,
    RemovalPolicy,
    Stack,
    aws_iam as iam,
    aws_lambda as _lambda,
    aws_s3 as s3,
    aws_s3_notifications as s3n,
    aws_sqs as sqs,
)
import constructs as core

logger = logging.getLogger(__name__)

# enable_my_environment()

class InputStack(Stack):
    """Instantiate an AWS input stack."""

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        """Initialize class attributes."""

        super().__init__(scope, id, **kwargs)
        curr_dir = os.path.dirname(os.path.abspath(__file__))

        # Create the S3 bucket
        bucket = s3.Bucket(
            self,
            "InputBucket",
            removal_policy=RemovalPolicy.DESTROY,
            auto_delete_objects=False,
        )

        # Create the SQS queue
        queue = sqs.Queue(
            self,
            "ProcessingQueue",
            visibility_timeout=Duration.seconds(300),
        )

        # Add an IAM role for the Lambda to talk to the S3 bucket.
        lambda_role = iam.Role(
            self, "LambdaRole",
            assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
            description="Role for Lambda function to upload files to S3 bucket"
        )

        # Add permissions to the role
        lambda_role.add_to_policy(
            iam.PolicyStatement(
                effect=iam.Effect.ALLOW,
                actions=[
                    "s3:*",
                ],
                resources=[
                    f"{bucket.bucket_arn}/*",
                    f"{bucket.bucket_arn}"
                ]
            )
        )

        # Create the Lambda function
        function = _lambda.Function(
            self,
            "MyFunction",
            runtime=_lambda.Runtime.PYTHON_3_8,
            handler="handler.lambda_handler",
            code=_lambda.Code.from_asset(os.path.join(curr_dir, "lambda/sqs_notifier")),
            environment={
                "QUEUE_URL": queue.queue_url,
            },
            role=lambda_role,
        )


        # Attach the policy to the bucket
        bucket.add_to_resource_policy(
            iam.PolicyStatement(
                effect=iam.Effect.ALLOW,
                principals=[iam.ArnPrincipal(lambda_role.role_arn)],
                actions=[
                    "s3:*",
                ],
                resources=[
                    f"{bucket.bucket_arn}/*",
                    f"{bucket.bucket_arn}"
                ]
            )
        )

        queue.grant_consume_messages(iam.ArnPrincipal(lambda_role.role_arn))

        # Add the S3 event notification to the bucket
        bucket.add_event_notification( ##################Problem Function
            s3.EventType.OBJECT_CREATED,
            s3n.SqsDestination(queue),
            s3.NotificationKeyFilter(prefix="flight", suffix=".csv"),
        )

        # Add the Lambda source_mapping
        function.add_event_source_mapping(
            id="SqsNotifier",
            event_source_arn=queue.queue_arn,
        )

        # Output the bucket name and queue URL
        CfnOutput(self, "BucketName", value=bucket.bucket_name)
        CfnOutput(self, "QueueUrl", value=queue.queue_url)
        CfnOutput(self, "QueueArn", value=queue.queue_arn)

[lakkeger]

Hi! We just wanted to follow up to see whether your issue has been resolved. Were you able to get it working with the latest version of LocalStack? We would appreciate your feedback!

[robertchinezon]

Hello, I upgraded to the latest version 2.3.2 and it still does not work. Please find the updated log file attached. Thanks, Robert

…

On Wed, Nov 15, 2023 at 8:22 AM Gábor Németh ***@***.***> wrote: Hi! We just wanted to follow up to see whether your issue has been resolved. Were you able to get it working with the latest version of LocalStack? We would appreciate your feedback! — Reply to this email directly, view it on GitHub <#88 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALIKGKINGC635NJ3MLIAHQLYES6YPAVCNFSM6AAAAAA5LVF25OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJSGUZDSNJTGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[robertchinezon]

Here is the log file
localstack.log

[kazazor]

Same issue here. Also posted about it here: localstack/localstack#9352 (comment)

This is a blocker for us from using LocalStack

[roeysha]

Super important! Blocker from our side as well

[lakkeger]

Hi!
Thank you for your patience. I tried to reproduce your issue and it seems to come down to some Cloudformation features, namely the custom resource types, that CDK is generating.
The usage of such is a Localstack Pro feature, you can find more about Cloud Formation coverage here:
https://docs.localstack.cloud/user-guide/aws/cloudformation/#pro-edition

Please do not hesitate to contact us if you experience any other issues.

[laurence-myers]

Here's a hack to bypass the custom resource created by CDK. We can access the underlying L1 CDK construct, and from there set notificationConfiguration.lambdaConfigurations.

This example is in TypeScript.

    const bucket = new s3.Bucket(...);
    const lambda = new NodejsFunction(...);
    if (isDeployingToLocalStack()) { // implement or replace `isDeployingToLocalStack()` as required
      const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;
      cfnBucket.notificationConfiguration = {
        lambdaConfigurations: [
          {
            creationStack: [],
            event: 's3:ObjectCreated:*',
            function: lambda.functionArn,
          },
        ],
      };
    }

This works for newly created S3 and Lambda resources.

I don't know what impact it'll have if you bypass the inline Python script added by CDK. (In the worst case scenario, you can add your own.)