AWS Lambda Exploration

We tried to use AWS Lambda, but it was hard

Our Intent

Our initial design intention was to use Amazon Web Service's Lambda to automatically handle the passing of data between the microservices of the app. Lambda provides the ability to implement functions that can be triggered automatically on the creation or deletion of files added to S3 buckets. It seemed like a natural choice to implement functions such that messages.htm files passed into a S3 bucket (in which all data was to be erased every 24 hours) would be parsed, and the output written to another bucket which would in turn trigger the elastic map reduce as a further lambda function and return the image file to the user.

Our Difficulties with Lambda

While using Lambda was an attractive choice we found the process of getting Python code which relied on compiled dependencies to be rather cumbersome. In the end a satisfactory result could not be achieved by the responsible team member. It is important to note that AWS Lambda is still in beta development, and the experience may become more user friendly in the future. When using code with dependencies, everything is required to be in a zipped package with the function source code in the root, and this is where the difficulties began. While there was lots of official documentation on how to achieve this, it was not entirely consistent. In the end our best approach was to:

• Launch an EC2 Linux shell instance
• ssh into the instance from home and upload any source files using terminal commands
• sudo yum install a plethora of dependencies (gcc, pip, aws cli and many more)
• Create a virtual environment for the function
• Install all the dependencies in the virtual environment
• Package all that up into a zip file
• Use aws cli to move the zip file to an S3 bucket
• Point the Lambda function to the zip file in the bucket
• And... not have it work even though it ran fine on the command line and have no helpful error messages produced

It was sour work with a considerable learning curve and way more stressful than it had any right to be. While our problems were probably mostly due to some misunderstanding on our part, it just wasn't worth it to spend more time figuring it out. Our conclusion which seems to be the consensus based upon the cries for help from other developers is that Lambda is a great choice for any simple function that can be line edited in the Lambda console, but quickly becomes unreasonable when external dependencies are required.