index.json

[{"content":"AWS offers a variety of products to create any application you can think of. With DynamoDB, you can trigger a Lambda function to perform additional work each time a DynamoDB table is updated.\nIn this post, I’ll show how these two technologies can work together. Without further ado, let’s get started.\nWhat I\u0026rsquo;m building NYC Open Data has a public API that details gasoline price information of all the New York Regions. A new gasoline price gets added every week.\nI had the idea of building a system that can notify me of the gasoline prices with the condition when the price goes down. I’m going to show you how I did this. You might think the idea sounds a little vague or easy to do but the most important thing about the project is how I did it which is what I want to emphasize here in this blog.\nSetting up the project Getting your repository set up I enjoy doing side projects with the latest trends in the market. In the project, I use terraform. For those of you that don’t know terraform is infrastructure as code that lets you build, change, and version cloud and on-premise resources safely and efficiently. In short words, instead of using the AWS UI, we use code to create AWS services.\nAnother cool technology that I’m using is Serverless. Serverless is a cloud-native development model that allows developers to build and run applications without having to manage servers. It’s really cool because we just need to provide the code and AWS will do all the heavy tasks in the background.\nBuilding the infrastructure Something I have learned over time using terraform is that I like to structure the different resources I want to build using modules. For example, if I’m planning to use an EC2 I normally create a compute module and allocate all the logics that falls in EC2. Here’s what I did:\nmodule \u0026#34;dynamodb\u0026#34; {  source = \u0026#34;./dynamodb\u0026#34; }  module \u0026#34;sns\u0026#34; {  source = \u0026#34;./sns\u0026#34;  email = var.email }  module \u0026#34;lambda\u0026#34; {  source = \u0026#34;./lambda\u0026#34;  gasoline_prices_table_arn = module.dynamodb.gasoline_prices_table_arn  gasoline_price_table_stream_arn = module.dynamodb.dynamodb_stream_arn  sns_arn = module.sns.sns_arn } Once we have the modules in place I start building each module separately and export any variable that a given module needs.\nIn AWS every resource that we want to create is private. If we want to execute a resource either programmatically or get access to it, we need to explicitly tell AWS to do it.\nI created an IAM role in the lambda module to detail all the permissions that the Lambda needs.\nresource \u0026#34;aws_iam_policy\u0026#34; \u0026#34;iam_role_policy_for_lambda\u0026#34; {   name = \u0026#34;aws_iam_policy_for_terraform_aws_lambda_role\u0026#34;  description = \u0026#34;AWS IAM Policy for managing aws lambda role\u0026#34;  policy = jsonencode({  Statement = [{  Action = [\u0026#34;s3:GetObject\u0026#34;, \u0026#34;s3:PutObject\u0026#34;],  Effect = \u0026#34;Allow\u0026#34;,  Resource = \u0026#34;${aws_s3_bucket.price_fetcher_deployment.arn}\u0026#34;  },  {  Action = [\u0026#34;dynamodb:Scan\u0026#34;, \u0026#34;dynamodb:Query\u0026#34;, \u0026#34;dynamodb:PutItem\u0026#34;],  Effect = \u0026#34;Allow\u0026#34;,  Resource = \u0026#34;${var.gasoline_prices_table_arn}\u0026#34;  },  {  Action = [\u0026#34;logs:CreateLogGroup\u0026#34;, \u0026#34;logs:CreateLogStream\u0026#34;, \u0026#34;logs:PutLogEvents\u0026#34;, \u0026#34;logs:DescribeLogStreams\u0026#34;, \u0026#34;logs:createExportTask\u0026#34;],  Effect = \u0026#34;Allow\u0026#34;,  Resource = \u0026#34;*\u0026#34;  },  {  Action = [\u0026#34;dynamodb:DescribeStream\u0026#34;, \u0026#34;dynamodb:GetRecords\u0026#34;, \u0026#34;dynamodb:GetShardIterator\u0026#34;, \u0026#34;dynamodb:ListStreams\u0026#34;],  Effect = \u0026#34;Allow\u0026#34;,  Resource = \u0026#34;${var.gasoline_price_table_stream_arn}\u0026#34;  },  {  Action = [\u0026#34;sns:Publish\u0026#34;],  Effect = \u0026#34;Allow\u0026#34;,  Resource = \u0026#34;${var.sns_arn}\u0026#34;  }  ]  Version = \u0026#34;2012-10-17\u0026#34;  }) } As you can see above, we have explicitly told AWS which permissions my AWS needs to execute the Lambda. This is something that you have to think about as you continue adding functionalities to your Lambda.\nIf you wondered before why I was passing some variables to the lambda from the dynamodb module. In order to grant permission to our Lambda to execute queries in Dynamo DB. We need to get the reference of the table and explicitly add it to the policy.\nLambda and More It only takes a few minutes to create a lambda using the serverless framework. Make sure you have the serverless package installed on your machine. Depending on what type of serverless template you want to create (e.g javascript, python) you specify it in the serverless CLI. For example\nsls create --template aws-python3 --path myService The serverless.yml is the heart of a serverless application. This file describes the entire application infrastructure, all the way from the programming language to resource access.\nA few examples of properties we can use:\n The functions we want to run Environment variables Depending on the cloud provider we’re using, we can specify the role and the region our code will get run on  For more information about this, check out this documentation.\nHere’s what my YAML looks like:\nservice: aws-price-fetcher  frameworkVersion: \u0026#34;3\u0026#34;  provider:  name: aws  runtime: python3.8  region: us-east-1  deploymentBucket:  name: price-fetcher-serverlessdeploymentbucket  iam:  role: arn:aws:iam::008735640664:role/Price-Notifier-Role  functions:  price_fetcher:  handler: handler.price_fetcher  description: Fetches gasoline price from public API  events:  - schedule: cron(* * 0 ? * WED *)  price_publisher:  handler: handler.price_publisher  description: Publish gasoline drops  plugins:  - serverless-python-requirements I won’t go to each one of the properties because some of them are self-explanatory.\nservice is the name of our lambda in AWS.\nprovider/region: Lambas are region-specific. What this means is that each Lambda function lives in a specific AWS region.\ndeploymentBucket: when we create a lambda our lambda has to live somewhere in AWS. This is why in terraform we are creating a bucket for our lambda to live in and by this means we need to specify it in the YAML file.\nrole The role created in terraform along with its permissions to execute resources in AWS.\nfunctions: When we create a lambda we can have multiple functions living in one lambda. To make this project easy, I chose to have 1 lambda with 2 functions one for fetching the gasoline price from the public API and another one to publish the gasoline drops.\nplugins are just the different libraries we are using in serverless. We\u0026rsquo;re using serverless-python-requirements because we need some libraries to execute our code, for example, requests. If you’re curious to see more libraries, check out this documentation that details all the plugins serverless has.\nIn the lambda, I created a function called price_fetcher whose purpose is to query the DynamoDB table, if there are any records it gets the most recent gasoline price, calculates against the last record we have in the database if the price has dropped it insert the record to the table otherwise, it doesn’t do the insertion.\n def price_fetcher(event: Dict[str, Any], _: Any) -\u0026gt; None:  \u0026#34;\u0026#34;\u0026#34; Fetches gasoline price from public API Calculate the most recent gasoline price with the last record saved in the DynamoDB table if the price has dropped, it inserts that record into the table otherwise, it skips it. \u0026#34;\u0026#34;\u0026#34;  items = query(GASOLINE_PRICE_TABLE_NAME)  LOG.info(f\u0026#34;Got items: {items}\u0026#34;)   gasoline_api_response = request()   gasoline_api_json = gasoline_api_response.json()   most_recent_gasoline_price: Dict = gasoline_api_json[0]  LOG.info(f\u0026#34;Got most recent gasoline price: {most_recent_gasoline_price}\u0026#34;)   published_at = most_recent_gasoline_price[\u0026#39;date\u0026#39;]  gasoline_price = most_recent_gasoline_price[\u0026#39;new_york_state_average_gal\u0026#39;]   if items:  previous_gasoline_price = float(items[0][\u0026#39;newYorkStateAverageGal\u0026#39;])  LOG.info(f\u0026#39;Got previous previous_gasoline_price {previous_gasoline_price}\u0026#39;)   recent_gasoline_price = float(gasoline_price)   price_has_dropped = recent_gasoline_price \u0026lt; previous_gasoline_price   if not price_has_dropped:  LOG.info(\u0026#34;Skipping insertion because price hasn\u0026#39;t dropped\u0026#34;)  return   response = insert(GASOLINE_PRICE_TABLE_NAME, published_at=published_at, gasoline_price=gasoline_price)  LOG.info(f\u0026#34;Got response: {response}\u0026#34;)   return None DynamoDB provides a Change Data Capture (CDC) mechanism for each table.\nThat means, if someone creates a new entry or modifies an item in a table, DynamoDB immediately emits an event containing the information about the change. You can build applications that consume these events and take action based on the contents.\nDynamoDB keeps track of every modification to data items in a table and streams them as real-time events.\nHere’s how I did it using terraform:\nresource \u0026#34;aws_lambda_event_source_mapping\u0026#34; \u0026#34;gasoline_prices_stream\u0026#34; {  event_source_arn = aws_dynamodb_table.gasoline_prices_table.stream_arn  function_name = var.price_publisher_arn  starting_position = \u0026#34;LATEST\u0026#34; } We need to provide a unique ARN that points to our table and indicates the function where we want to send these events.\nNext, see how I’m getting the events from the Lambda:\ndef price_publisher(event: Dict[str, Any], _: Any) -\u0026gt; None:  \u0026#34;\u0026#34;\u0026#34; Get DynamoDB events and send information to the user. \u0026#34;\u0026#34;\u0026#34;  LOG.info(f\u0026#39;Got event {event}\u0026#39;)  record = event[\u0026#39;Records\u0026#39;][0]  most_recent_gasoline_price = record[\u0026#39;dynamodb\u0026#39;][\u0026#39;NewImage\u0026#39;][\u0026#39;newYorkStateAverageGal\u0026#39;][\u0026#39;S\u0026#39;]  message = f\u0026#39;🛡🛡🛡 Gasoline price {most_recent_gasoline_price}have fell 🛡🛡🛡\u0026#39;  subject = \u0026#39;A gasoline price dropped has been detected\u0026#39;  send_sns(message, subject)  def send_sns(message, subject):  client = boto3.client(\u0026#34;sns\u0026#34;)  client.publish(  TopicArn=SNS_ARN, Message=message, Subject=subject) Lastly, we use AWS Simple Notification Service (or AWS SNS) a cloud-based web service that delivers messages. In SNS we have a publisher and a subscriber.\nThe publisher sends out a message to the subscriber immediately instead of storing the message.\nThe subscriber s an entity that receives the messages to the topics that they have subscribed to. In my case, I’m using my email to receive the notifications.\nHere’s the final result when I receive the mail from SNS\nSummary and where to next? This was a really interesting project that taught me two important things I won’t forget. It taught me how permissions really work on AWS. They can be confusing sometimes. Do I need a role for this project? What policy do I need to run this action? What users, groups, and roles really are, and their differences? These and other questions were the ones I was able to answer during this project. After learning how to spin up a Lambda using the Serverless framework and reading records with DynamoDB streams, it’s my turn to keep digging more into these great technologies to continue creating more projects like this.\nIf you want to send me feedback, or just want to connect with me, let me know via Twitter or LinkedIn. Also, if you are curious to see the entire project, check it out on my Github. Till the next one, stay safe and keep learning.\n","permalink":"https://JoseAngel1196.github.io/posts/serverless-application-gasoline-prices/","summary":"AWS offers a variety of products to create any application you can think of. With DynamoDB, you can trigger a Lambda function to perform additional work each time a DynamoDB table is updated.\nIn this post, I’ll show how these two technologies can work together. Without further ado, let’s get started.\nWhat I\u0026rsquo;m building NYC Open Data has a public API that details gasoline price information of all the New York Regions.","title":"Distributed Serverless Workflow"},{"content":"On June 7, 2021, A Cloud Guru released a challenge about improving the performance of an application. After a very bumpy ride, I was able to complete this challenge. One of the things that made me work on this challenge was the need to learn terraform, so I found existing building the infrastructure of this project using terraform.\nCheck it out! Let\u0026rsquo;s connect? Github LinkedIn\nMy learnings When I started this challenge, there were a couple of things that I didn\u0026rsquo;t know about AWS and one of the reasons that made me work on this. One of them was that I thought that it was possible to access a Redis cluster from the public web, after some investigation I quickly realized that is not safe to expose my Redis to the internet. I have to say that doing the networking on AWS was challenging and is something that I will spend more time practicting now that I successfully completed this project from the ground up, creating a public VPC that allows my resources to connect to that VPC and all subsequent resources that I wanted. It took me around 4 days without sleep to set up the whole infrastructure on AWS, I cannot remember how many times I had to run: terraform destroy to start again because there were some misconfigurations on the server.\nMy approach to the challenge Before I even started the challenge, I had an idea of what was terraform but did not know where to start. I saw a course on Udemy that gave me a good understanding on how to start this challenge using terraform. I was very eager to start this project and after having a good knowledge of terraform and watching the course, I moved up to create my first EC2 Instance. The things I learned on terraform are how I can reuse the same line of code called modules to create multiple resources on AWS and the way I can share variables across the code. I found this very interesting and just could stop myself from learning it.\nWhy I took the challenge? At my workplace terraform is what we use to deploy our infrastructure to AWS and I want to keep leveling up on my career by expanding my knowledge and taking on challenges that put me to test.\nThe Project Once I was able to set up my infrastructure on terraform and seeing I was able to connect to my EC2 and ping to my RDS ad Redis without any problem. The only thing I was missing was installing nginx on my server. I chose the amazon linux ami that use centos. I had to investigate how to install nginx on centos as it was my first-time using centos and I did not know that I had to use yum instead of apt, the good thing was that the set up was not complicated at all. After having all the set up in place, I had to make some modifications to the code to cache our query. The result was:\nHaving that in place, once we visit the application and because there is a timer in the sql function, the application takes some time to get the response from the server and including the time to connect to the database and return it back to the server.\nAs we can see on the image above, it takes around 5sec to get the results back to the server, which is a long time, if we compare it by caching the result with redis, time would be around: 0.023sec the timedelta is a huge difference:\nConclusion I want to thank #acloudguru for this amazing challenge and I am really waiting to see more like this on the following months. I always appreciate any and all feedback.\nThanks for reading! 😀\n","permalink":"https://JoseAngel1196.github.io/posts/application-performance-elastic/","summary":"On June 7, 2021, A Cloud Guru released a challenge about improving the performance of an application. After a very bumpy ride, I was able to complete this challenge. One of the things that made me work on this challenge was the need to learn terraform, so I found existing building the infrastructure of this project using terraform.\nCheck it out! Let\u0026rsquo;s connect? Github LinkedIn\nMy learnings When I started this challenge, there were a couple of things that I didn\u0026rsquo;t know about AWS and one of the reasons that made me work on this.","title":"Application performance using Amazon Elasticache"}]