And The End Game Is … Serverless Scripts: Meet Lambda
And The End Game Is … Serverless Scripts: Meet Lambda
We have grown up with servers, and can only see them as the solution to provide services. But, the end game is to remove these complex computing machines and replace them with scripts. With our server infrastructures, we often setup complex databases and use files to store data in a structured way — but often we just need temporary storage of files, and, increasenly, just JSON objects to reflect our data. And so adisk on a server is often wasteful of our precious data resources, and where inexpensive data buckets are often a good solution for both short-term and long-term storage.
In the public cloud, too, we pay for our data and our compute by the hour, but why not by per nanosecond of compute time? Servers are thus computationally inefficient entities and run complex operating systems to just run one or two exposed services.
One solution is to use AWS Lambda, and where we run scripts which are invoked by events, and where we pay for the compute requirements for the script, rather than for the compute requirements of a server. In many circumstances it is the automation of tasks that would often be done with PowerShell and Python on servers that is the core advantage of using Lambda. Once tested, the script can help to automate key elements of the operation of the business.
In the following, we will pick up an email from a customer, and then process it, and send it to our support team.
Setting up Lambda script
In AWS, go to Lambda, and create our new script:
In this case, we will use Python 3.7, and have a script named ForwardEmail. Once created, we can then go to the editor:
In the following, I will explain the key elements of the script. Basically, we have an event, and which calls up the script. It processes the Python code and then returns something (normally we always return a JSON data object from the function). For this, we will use the SES (Simple Email Service) to trigger an event in Lambda, and then read the email from an S3 bucket, and forward it to users:
Hooking
With Lambda, we need a hook to capture a new event. In Python, this function is known as lamda_handler(). In the following, we will forward an email that is received to another mailbox. The event contains details about the message ID of the email from ‘ses’.
def lambda_handler(event, context):
message_id = event['Records'][0]['ses']['mail']['messageId']
print(f"Received message ID {message_id}")
# Retrieve the file from the S3 bucket.
file_dict = get_message_from_s3(message_id)
# Create the message.
message = create_message(file_dict)
# Send the email and print the result.
result = send_email(message)
print(result)Once we have the ID of the email, we can get the message from the S3 bucket by determining the location of the bucket, and reading the email:
def get_message_from_s3(message_id):
incoming_email_bucket = os.environ['MailS3Bucket']
object_path = message_id
object_http_path = (f"http://s3.console.aws.amazon.com/s3/object/{incoming_email_bucket}/{object_path}?region={region}")
# Create a new S3 client.
client_s3 = boto3.client("s3")
# Get the email object from the S3 bucket.
object_s3 = client_s3.get_object(Bucket=incoming_email_bucket,
Key=object_path)
# Read the content of the message.
file = object_s3['Body'].read()
file_dict = {
"file": file,
"path": object_http_path
}
return file_dictOnce we have this, we can then format the email for the recipient. We can see in this case, we have an environment variable (MailS3Bucket). This is the place where our email package will place the email. To set up the place for the bucket, we create environment variables. In the following, we have an S3 bucket of “myS3bucket” (along with a few other environment variables we need for the script):
We can now take the email (defined in file_dict), and create a message with a MailSender ([email protected]) and MailReceipient ([email protected]):
def create_message(file_dict):
sender = os.environ['MailSender']
recipient = os.environ['MailRecipient']
separator = ";"
# Parse the email body.
mailobject = email.message_from_string(file_dict['file'].decode('utf-8'))
# Create a new subject line.
subject_original = mailobject['Subject']
subject = subject_original + " - " + mailobject['From']
msg = MIMEMultipart()
msg.attach(mailobject)
# Add subject, from and to lines.
msg['Subject'] = subject
msg['From'] = sender
msg['To'] = recipient
# Create a new MIME object.
# att = MIMEApplication(file_dict["file"], filename)
# att.add_header("Content-Disposition", 'attachment', filename=filename)
# Attach the file object to the message.
# msg.attach(att)
message = {
"Source": sender,
"Destinations": recipient,
"Data": msg.as_string()
}
return messageThe email message now is in a JSON format which could be sent as an email, with a Source, Destinations, and Data. Now we can send the email (message), and create a log on whether the email was sent correctly or not:
def send_email(message):
aws_region = os.environ['Region']
# Create a new SES client.
client_ses = boto3.client('ses', region)
# Send the email.
try:
#Provide the contents of the email.
response = client_ses.send_raw_email(
Source=message['Source'],
Destinations=[
message['Destinations']
],
RawMessage={
'Data':message['Data']
}
)
# Display an error if something goes wrong.
except ClientError as e:
output = e.response['Error']['Message']
else:
output = "Email sent! Message ID: " + response['MessageId']
return outputThe client_ses.send_raw_email() is the method to send the email through SES. And the final return basically creates a log to define if the email is successfully forwarded or not. And basically, that’s it. Here is the final code:
# Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
import os
import boto3
import email
import re
from botocore.exceptions import ClientError
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.application import MIMEApplication
region = os.environ['Region']
def get_message_from_s3(message_id):
incoming_email_bucket = os.environ['MailS3Bucket']
object_path = message_id
object_http_path = (f"http://s3.console.aws.amazon.com/s3/object/{incoming_email_bucket}/{object_path}?region={region}")
# Create a new S3 client.
client_s3 = boto3.client("s3")
# Get the email object from the S3 bucket.
object_s3 = client_s3.get_object(Bucket=incoming_email_bucket,
Key=object_path)
# Read the content of the message.
file = object_s3['Body'].read()
file_dict = {
"file": file,
"path": object_http_path
}
return file_dict
def create_message(file_dict):
sender = os.environ['MailSender']
recipient = os.environ['MailRecipient']
separator = ";"
# Parse the email body.
mailobject = email.message_from_string(file_dict['file'].decode('utf-8'))
# Create a new subject line.
subject_original = mailobject['Subject']
subject = subject_original + " - " + mailobject['From']
msg = MIMEMultipart()
msg.attach(mailobject)
# Add subject, from and to lines.
msg['Subject'] = subject
msg['From'] = sender
msg['To'] = recipient
# Create a new MIME object.
# att = MIMEApplication(file_dict["file"], filename)
# att.add_header("Content-Disposition", 'attachment', filename=filename)
# Attach the file object to the message.
# msg.attach(att)
message = {
"Source": sender,
"Destinations": recipient,
"Data": msg.as_string()
}
return message
def send_email(message):
aws_region = os.environ['Region']
# Create a new SES client.
client_ses = boto3.client('ses', region)
# Send the email.
try:
#Provide the contents of the email.
response = client_ses.send_raw_email(
Source=message['Source'],
Destinations=[
message['Destinations']
],
RawMessage={
'Data':message['Data']
}
)
# Display an error if something goes wrong.
except ClientError as e:
output = e.response['Error']['Message']
else:
output = "Email sent! Message ID: " + response['MessageId']
return output
def lambda_handler(event, context):
# Get the unique ID of the message. This corresponds to the name of the file
# in S3.
message_id = event['Records'][0]['ses']['mail']['messageId']
print(f"Received message ID {message_id}")
# Retrieve the file from the S3 bucket.
file_dict = get_message_from_s3(message_id)
# Create the message.
message = create_message(file_dict)
# Send the email and print the result.
result = send_email(message)
print(result)Conclusions
The core advantage of using Lambda is that there is no spinning-up of instances, no setup of code environments, and no setting up of firewalls. The code is triggered, it does its work, and it leaves. Our first wave of porting into the Cloud, was basically to move our existing on-premise approaches into the public Cloud, but the next wave must integrate our applications and services within a serverless environment.
If you are interested, we are scaling up our AWS Academy, so get in contact with us, if your company is interested in a partnership around training in the Cloud.