Skip to content

Commit

Permalink
Merge pull request #2510 from philippewanner/philippewanner-feature-i…
Browse files Browse the repository at this point in the history
…ot-rule-action-lambda-python-cdk-ts

Philippewanner feature iot rule action lambda python cdk ts
  • Loading branch information
julianwood authored Jan 6, 2025
2 parents a4508b8 + 6e45f48 commit f3150f5
Show file tree
Hide file tree
Showing 12 changed files with 523 additions and 0 deletions.
74 changes: 74 additions & 0 deletions iot-lambda-pub-receiver-cdk/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# AWS Lambda to AWS IoT Core

This pattern deploys an AWS Lambda function, which publishes a message to an AWS IoT Core topic. The topic is watched by a rule which will trigger an action when the condition is met. The action calls an AWS Lambda function.

Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/iot-lambda-pub-receiver-cdk

Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.

## Requirements

* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* [AWS Cloud Development Kit (AWS CDK) installed](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html)

## Architecture diagram
![Architecture diagram](./doc/architecture-diagram.png)

This CDK stack creates an AWS IoT Core setup with a publisher-receiver pattern using Lambda functions.
Here's a breakdown of the main components:

1. An IoT MQTT topic named "my/mqtt/topic"

1. Publisher Lambda Function, written in Python and using ARM64 architecture, has permissions to publish messages to the specified MQTT topic (1). Its environment variables include the MQTT topic region and name.

1. Receiver Lambda Function, also in Python and using ARM64 architecture, has permissions to receive messages from the MQTT topic.

1. IoT Topic Rule, named "ProcessIoTMessages, uses SQL version 2016-03-23 to select all messages from the MQTT topic. It triggers the receiver Lambda function when messages arrive and it includes error logging to CloudWatch Logs.

## Deployment Instructions

1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
```
git clone https://github.com/aws-samples/serverless-patterns
```
1. Change directory to the pattern directory:
```
cd iot-lambda-pub-receiver-cdk
```
1. Install dependencies
```bash
npm install
```
1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
```
cdk deploy
```
2. Note the outputs from the CDK deployment process. These contain the IoT endpoint address which is not relevant if you have only one account. However, in multi-accounts deployment, especially when the IoT resources are not in the same as the lambdas, then the endpoint address has to be specified in the functions' code.
## Testing
The following steps will help you test the pattern from the AWS Console:
1. Trigger the publisher Lambda function
1. Navigate to AWS Lambda service and open the `LambdaIotCdkStack-iotPubHandler*` function.
1. From the `Test` tab, generate any event
1. A green notification should appears detailing that the execution has succeeded
1. Navigate to Amazon CloudWatch and open the Log group `/aws/lambda/pub-lambda`. The latest events will display the MQTT topic name and the AWS region where the message is sent.
1. Verify receiver execution
1. Navigate to AWS Lambda service and open the `LambdaIotCdkStack-iotReceiverHandler*` function.
1. From the `Monitor` tab, look for the Invocations metrics. A data point should be displayed indicated the execution of the receiver Lambda function just after that the publisher Lambda function has been triggered.
1. Navigate to Amazon CloudWatch and open the Log group `/aws/lambda/receiver-lambda`. The latest events will display the input of the publisher Lambda function.
## Cleanup
Run the given command to delete the resources that were created. It might take some time for the CloudFormation stack to get deleted.
```bash
cdk destroy
```
----
Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.

SPDX-License-Identifier: MIT-0
7 changes: 7 additions & 0 deletions iot-lambda-pub-receiver-cdk/bin/lambda-iot-cdk.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { LambdaIotCdkStack } from '../lib/lambda-iot-cdk-stack';

const app = new cdk.App();
new LambdaIotCdkStack(app, 'LambdaIotCdkStack', { });
80 changes: 80 additions & 0 deletions iot-lambda-pub-receiver-cdk/cdk.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
{
"app": "npx ts-node --prefer-ts-exts bin/lambda-iot-cdk.ts",
"watch": {
"include": [
"**"
],
"exclude": [
"README.md",
"cdk*.json",
"**/*.d.ts",
"**/*.js",
"tsconfig.json",
"package*.json",
"yarn.lock",
"node_modules",
"test"
]
},
"context": {
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
"@aws-cdk/core:checkSecretUsage": true,
"@aws-cdk/core:target-partitions": [
"aws",
"aws-cn"
],
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
"@aws-cdk/aws-iam:minimizePolicies": true,
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
"@aws-cdk/core:enablePartitionLiterals": true,
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
"@aws-cdk/aws-route53-patters:useCertificate": true,
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
"@aws-cdk/aws-redshift:columnId": true,
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
"@aws-cdk/aws-kms:aliasNameRef": true,
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false,
"@aws-cdk/aws-ecs:reduceEc2FargateCloudWatchPermissions": true,
"@aws-cdk/aws-dynamodb:resourcePolicyPerReplica": true,
"@aws-cdk/aws-ec2:ec2SumTImeoutEnabled": true,
"@aws-cdk/aws-appsync:appSyncGraphQLAPIScopeLambdaPermission": true,
"@aws-cdk/aws-rds:setCorrectValueForDatabaseInstanceReadReplicaInstanceResourceId": true,
"@aws-cdk/core:cfnIncludeRejectComplexResourceUpdateCreatePolicyIntrinsics": true,
"@aws-cdk/aws-lambda-nodejs:sdkV3ExcludeSmithyPackages": true,
"@aws-cdk/aws-stepfunctions-tasks:fixRunEcsTaskPolicy": true
}
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
66 changes: 66 additions & 0 deletions iot-lambda-pub-receiver-cdk/example-pattern.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
{
"title": "Lambda to IoT Core to Lambda",
"description": "Create a Lambda publishing into IoT topic, triggering an action calling another Lambda.",
"language": "Python",
"level": "200",
"framework": "CDK",
"introBox": {
"headline": "How it works",
"text": [
"This sample project demonstrates how to use an AWS Lambda Function to publish messages into AWS IoT Core topic and get a rule triggered by a condition to process relevant messages into another Lambda. This pattern is leveraging CloudWatch logs to ease the debugging and visibility into the processed messages.",
"The function are written in Python and the infrastructure is described with AWS CDKv2 in Typescript. The patterns also shows an effective way to manage Python dependencies through Docker built image and requiements.txt file.",
"This pattern deploys 2 Lambda functions, 1 IoT rule and action, 3 log groups."
]
},
"gitHub": {
"template": {
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/sfn-athena-cdk-python",
"templateURL": "serverless-patterns/sfn-athena-cdk-python",
"projectFolder": "iot-lambda-pub-receiver-cdk",
"templateFile": "lambda-iot-cdk-stack.ts"
}
},
"resources": {
"bullets": [
{
"text": "AWS IoT Core - Getting started with AWS IoT Core tutorials",
"link": "https://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html"
},
{
"text": "AWS IoT Core action resources",
"link": "https://docs.aws.amazon.com/iot/latest/developerguide/iot-action-resources.html"
},
{
"test": "Building Lambda with Python",
"link": "https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html"
},
{
"text": "Working with the AWS CDK in TypeScript",
"link": "https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-typescript.html"
}
]
},
"deploy": {
"text": [
"cdk deploy"
]
},
"testing": {
"text": [
"See the GitHub repo for detailed testing instructions."
]
},
"cleanup": {
"text": [
"Delete the stack: <code>cdk destroy</code>."
]
},
"authors": [
{
"name": "Philippe Wanner",
"image": "https://serverlessland.com/assets/images/resources/contributors/philippe-wanner.jpg",
"bio": "Philippe is a Senior Specialist Solutions Architect at Amazon Web Services based in Zurich, Switzerland. His role is to spread the migration and modernization best practices for large organisations.",
"linkedin": "philippe-wanner"
}
]
}
114 changes: 114 additions & 0 deletions iot-lambda-pub-receiver-cdk/lib/lambda-iot-cdk-stack.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
import { CloudWatchLogsAction, LambdaFunctionAction } from '@aws-cdk/aws-iot-actions-alpha';
import { IotSql, TopicRule } from '@aws-cdk/aws-iot-alpha';
import { CfnOutput, RemovalPolicy, Stack, StackProps } from 'aws-cdk-lib';
import { Effect, PolicyStatement, ServicePrincipal } from 'aws-cdk-lib/aws-iam';
import { Architecture, Code, Function, Runtime } from 'aws-cdk-lib/aws-lambda';
import { LogGroup, RetentionDays } from 'aws-cdk-lib/aws-logs';
import { AwsCustomResource, AwsCustomResourcePolicy, PhysicalResourceId } from 'aws-cdk-lib/custom-resources';
import { Construct } from 'constructs';

export class LambdaIotCdkStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);

let mqttTopicName = "my/mqtt/topic"
let mqttTopicRegion = Stack.of(this).region
let mqttTopicAccount = Stack.of(this).account
let iotEndpointAddress = this.getIoTEndpoint().getResponseField('endpointAddress')

// Publisher lambda function.
// Remark - if the Lambda is in the same account and region as the IoT Core endpoint, then setting the endpoint is optional.
const iotPubPermission = new PolicyStatement(({
effect: Effect.ALLOW,
resources: [ `arn:aws:iot:${mqttTopicRegion}:${mqttTopicAccount}:topic/${mqttTopicName}` ],
actions: [ "iot:Publish" ]
}));
const iotPubLambda = new Function(this, 'iotPubHandler', {
handler: 'publisher_function.handler',
code: Code.fromAsset('./src'),
description: 'This function publishes a message to AWS IoT Core - MTTQ',
runtime: Runtime.PYTHON_3_12,
architecture: Architecture.ARM_64,
logGroup: this.addLogGroup(`/aws/lambda/pub-lambda`),
environment: {
MQTT_TOPIC_REGION: mqttTopicRegion,
MQTT_TOPIC_NAME: mqttTopicName
}
})
iotPubLambda.addToRolePolicy(iotPubPermission)

// Receiver lambda function
const iotReceiverPermission = new PolicyStatement(({
effect: Effect.ALLOW,
resources: [ `arn:aws:iot:${mqttTopicRegion}:${mqttTopicAccount}:topic/${mqttTopicName}` ],
actions: [
"iot:Receive"
]
}));
const iotReceiverLambda = new Function(this, 'iotReceiverHandler', {
handler: 'receiver_function.handler',
description: 'This function get invoked by AWS IoT Core through the action-rule',
code: Code.fromAsset('./src', {
bundling: {
image: Runtime.PYTHON_3_12.bundlingImage,
command: [
'bash', '-c',
'pip install -r receiver_requirements.txt -t /asset-output && cp -au . /asset-output'
],
},
}),
runtime: Runtime.PYTHON_3_12,
architecture: Architecture.ARM_64,
logGroup: this.addLogGroup(`/aws/lambda/receiver-lambda`)
})
iotReceiverLambda.addToRolePolicy(iotReceiverPermission)

// Topic rule
const errorLogGroup = new LogGroup(this, 'RuleErrorLogGroup', {
logGroupName: '/aws/iot/rule-error-logs',
retention: RetentionDays.FIVE_DAYS,
removalPolicy: RemovalPolicy.DESTROY
})
let topicRule = new TopicRule(this, 'IoTTopicRule', {
topicRuleName: 'ProcessIoTMessages',
description: 'Invokes the lambda function',
sql: IotSql.fromStringAsVer20160323("SELECT * FROM 'my/mqtt/topic'"),
actions: [ new LambdaFunctionAction(iotReceiverLambda) ],
errorAction: new CloudWatchLogsAction(errorLogGroup)
})

// Grant permission for AWS IoT to invoke the Lambda function
const iotServicePrincipal = new ServicePrincipal('iot.amazonaws.com');
iotReceiverLambda.grantInvoke(iotServicePrincipal);

// Outputs
new CfnOutput(this, "IoT Endpoint Address", {
value: iotEndpointAddress ?? "Error: can't get the IoT Endpoint Address!",
});
}

// Utility function to return a log-group object
private addLogGroup(logGroupName: string) {
const retentionDays = RetentionDays.FIVE_DAYS
const removalPolicy = RemovalPolicy.DESTROY
const props = { logGroupName, retentionDays, removalPolicy }
return new LogGroup(this, `${logGroupName}`, props)
}

// Get the current account IoT-Endpoint
private getIoTEndpoint() {
const ioTEndpoint = new AwsCustomResource(this, 'IoTEndpoint', {
onCreate: {
service: 'Iot',
action: 'describeEndpoint',
physicalResourceId: PhysicalResourceId.fromResponse('endpointAddress'),
parameters: {
"endpointType": "iot:Data-ATS"
}
},
policy: AwsCustomResourcePolicy.fromSdkCalls({resources: AwsCustomResourcePolicy.ANY_RESOURCE})
})
const IOT_ENDPOINT = ioTEndpoint.getResponseField('endpointAddress')
return ioTEndpoint
}
}
24 changes: 24 additions & 0 deletions iot-lambda-pub-receiver-cdk/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"name": "lambda-iot-cdk",
"version": "0.1.0",
"bin": {
"lambda-iot-cdk": "bin/lambda-iot-cdk.js"
},
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"cdk": "cdk"
},
"devDependencies": {
"aws-cdk": "^2.173.0",
"ts-node": "^10.9.2",
"typescript": "~5.7.2"
},
"dependencies": {
"@aws-cdk/aws-iot-actions-alpha": "^2.173.0-alpha.0",
"@aws-cdk/aws-iot-alpha": "^2.173.0-alpha.0",
"aws-cdk-lib": "2.173.0",
"constructs": "^10.0.0",
"source-map-support": "^0.5.21"
}
}
Loading

0 comments on commit f3150f5

Please sign in to comment.