Published on: Sat Jun 11 2022
In this module, we will use this starter template - aws-webhook-series-starter as a starting point.
By the end of this module, you should:
✅ Understand the folder structure and it’s setup
✅ Be able to setup the basic scaffold of the AWS Lambda function with AWS Lambda Alias
✅ Be able to setup modules using terraform (for the lambda functions)
✅ Understand the CI/CD process of the Lambda function in our setup
✅ Understand each of the steps of the CI/CD
✅ Be able to Test out the lambda functions manually (using AWS CLI)
Overall, by doing these steps manually on local, it will help us build a deeper understanding of each step.
So, once we start automating the process, you would know each of the steps involved, and what the script is doing during a deployment.
First, we will start adding the infrastructure for one of our functions.
Then, we will refactor it to use terraform modules, and see how these modules make it much easier for us to create multiple infrastructures.
After setting that up, we will manually test out the infrastructure by running the deployment script on local before we automate it with Github Actions.
Finally, we will test out the deployment of a new change on the lambda function (just to make sure things are working as expected).
Without further delay, let’s get started!
We are using pnpm workspace for this project.
├── functions
│ ├── ingestion
│ └── process-queue
├── infra
├── node_modules
└── scripts
functions - contains all the source code for our AWS Lambda functions (ingestion & process-queue) infra - all terraform code related for infrastructure goes here scripts - These are all the scripts for the deployment process
Within each workspace, the source code will live in src
.
In addition, there will be several commands available to manage the workspace.
These include:
node_modules
in the initial zip)As mentioned, everything will live under src
.
There is a special mapping configured for easier imports using @app/
which maps to everything understand src.
This applies to both code and test files.
Let’s start off by setting up the infrastructure for our ingestion function.
# main.tf
resource "aws_s3_bucket" "lambda_bucket" {
bucket = "lambda-bucket-assets"
acl = "private"
}
⚠️ Note: You may need to update the bucket name if it is already taken.
This will our zipped files to AWS bucket.
The uuid()
is mainly for differntiating between the AWS Lambda assets.
# main.tf
resource "aws_s3_bucket_object" "lambda_default" {
bucket = aws_s3_bucket.lambda_bucket.id
key = "main-${uuid()}.zip"
source = "../functions/ingestion/main.zip"
etag = filemd5("../functions/ingestion/main.zip")
}
# main.tf
# Prefer to put at the top
locals {
default_lambda_timeout = 10
default_lambda_log_retention = 1
}
resource "aws_lambda_function" "this" {
s3_bucket = aws_s3_bucket.lambda_bucket.id
s3_key = aws_s3_bucket_object.lambda_default.key
timeout = local.default_lambda_timeout
function_name = "Ingestion-function"
runtime = "nodejs12.x"
handler = "dist/index.handler"
publish = true
source_code_hash = "${filebase64sha256("../functions/ingestion/main.zip")}"
role = aws_iam_role.lambda_exec.arn
environment {
variables = {
DefaultRegion = var.aws_region
}
}
}
Prefer to put the locals at the top of the file below aws provider in case we need to use it for other things
This allows for easier management of the AWS Lambda versions.
# main.tf
resource "aws_lambda_alias" "this" {
name = "ingestion-dev"
description = "alias for the ingestion function"
function_name = aws_lambda_function.this.arn
function_version = aws_lambda_function.this.version
}
# main.tf
resource "aws_cloudwatch_log_group" "log_group" {
name = "/aws/lambda/${aws_lambda_function.this.function_name}"
retention_in_days = local.default_lambda_log_retention
}
This is the execution (or run-time) role for the AWS Lambda.
Within this role, we can give it IAM permissions to access other AWS Services (S3, Cloudwatch for logging and file etc).
# main.tf
resource "aws_iam_role" "lambda_exec" {
name = "ingestion-function-exec-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
Service = "lambda.amazonaws.com"
}
}]
})
}
Create the runtime policy resource with permissions then attach it to our lambda role we just added above.
# main.tf
data "aws_iam_policy_document" "runtime_policy_doc" {
version = "2012-10-17"
statement {
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
]
effect = "Allow"
resources = [
"*"
]
}
}
resource "aws_iam_policy" "lambda_runtime_policy" {
name = "ingestion-function-runtime-policy"
policy = data.aws_iam_policy_document.runtime_policy_doc.json
}
resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
name = "ingestion-function-lambda-role-attachment"
roles = [aws_iam_role.lambda_exec.name]
policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}
This will be specific details we want to output from our infrastructure.
This is mainly for convenience but sometimes we need details like API gateway endpoint url which we won’t know until the infrastructure is applied.
# outputs.tf
# Output value definitions
output "lambda_bucket_name" {
description = "Name of the S3 bucket used to store function code."
value = aws_s3_bucket.lambda_bucket.id
}
output "function_name" {
description = "Name of function"
value = aws_lambda_function.this.function_name
}
output "function_alias_name" {
description = "Name of the function alias"
value = aws_lambda_alias.this.name
}
let’s use terraform to apply the infrastructure with what we have so far.
export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1
terraform init
terraform plan
terraform apply -auto-approve
Let’s test out our ingestion function.
Let’s just make sure everything is working as expected and our infrastructure was setup properly.
aws lambda invoke \
--function-name Ingestion-function:ingestion-dev \
--invocation-type Event \
--payload 'eyAibmFtZSI6ICJCb2IifQ==' \
response.json
When invoking with Ingestion-function:ingestion-dev you can specify versions or in our case Lambda aliasesThese two are very similar but you can think of Lambda alias as a container that points to a specific Lambda version.
If it was successful, you should see a output of:
{
"StatusCode": 202
}
To verify it, you can also check the AWS Cloudwatch logs where within our function we are logging out the event.
This should be under CloudWatch > Log Groups > /aws/lambda/Ingestion-function
:
Ok, so everything looks to be working as expected with our lambda infrastructure.
That is where the scripts comes in.
Within scripts/
there is a deployment.sh
which would handle all the steps needed for the deployment.
Let’s take a look at the CI/CD for our Lambda functions:
This will prepare a zip file at the end of this step
First the production dependencies node_modules
are packaged into the zip (using the appropriate flags --production --prefer-offline
etc)
Then we do another install to bring back the devDepdencies
, in this step we will run the tests and the build with esbuild
finally, those changes will be merged into the same zip
The steps we are most interested in at the moment are 2-5 which is what we are going to run locally.
Before we run the script, add a small change to the console.log
(maybe append a string) so we can see that new change.
Under the root directory of the folder, run the following:
export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1
AWS_LAMBDA_FUNCTION_NAME=Ingestion-function \
AWS_S3_BUCKET=lambda-bucket-assets-1234567 \
AWS_LAMBDA_ALIAS_NAME=ingestion-dev \
MODULE=@function/ingestion \
./scripts/deployment.sh
under CloudWatch > Log Groups > /aws/lambda/Ingestion-function
you should see the new change:
Assuming everything was deployed as expected and there was no error, it should have deployed the new version by running the above script!
Ok, so that was just the infrastructure for the ingestion function.
Let’s do the same thing for the other functions but this time we will lean into the power of terraform modules.
Why use terraform modules when we can just copy and paste ?
Certainly we can just copy and paste but there a few benefits with abstracting this logic into a module.
What are some of the benefit of doing this ?
Better Consistency
Reducing cognitive load for future changes (less code)
Better maintaibility (less code)
Centralizing implementation for more efficient security audits & hardening efforts (especially important for IaC)
Simplify the infrastructure process
Improving developer experience (ie hiding away the complexity of the infrastructure)
Of course, there are cons to premature abstraction but I think in this case it makes sense as the infrastructure for both of these Lambda functions are almost identical with some differences (which we will support as a customization option of the module).
I won’t go into the details of all the options in the module but the gist of it is we are abstracting all the resources into a simplified interface to make it easier for us to create multiple similar infrastructure (ie Lambda functions).
if you want to learn more about this terraform module, I recommend reading through the post below - “Simplify your AWS Infrastructure in 9 steps”.
I go through this in detail on how to evolve an existing infrastructure to one using terraform module.
module "lambda_ingestion" {
source = "./modules/lambda"
code_src = "../functions/ingestion/main.zip"
bucket_id = aws_s3_bucket.lambda_bucket.id
timeout = local.default_lambda_timeout
function_name = "Ingestion-function"
runtime = "nodejs12.x"
handler = "dist/index.handler"
publish = true
alias_name = "ingestion-dev"
alias_description = "Alias for ingestion function"
log_retention = local.default_lambda_log_retention
environment_vars = {
DefaultRegion = var.aws_region
}
}
module "lambda_process_queue" {
source = "./modules/lambda"
code_src = "../functions/process-queue/main.zip"
bucket_id = aws_s3_bucket.lambda_bucket.id
timeout = local.default_lambda_timeout
function_name = "Process-Queue-function"
runtime = "nodejs12.x"
handler = "dist/index.handler"
publish = true
alias_name = "process-queue-dev"
alias_description = "Alias for ingestion function"
log_retention = local.default_lambda_log_retention
environment_vars = {
DefaultRegion = var.aws_region
}
}
Since we moved all our Lambda resources into a module and exported them via the module, we need to update our output on the top level.
Change the outputs.tf
file to the following:
#outputs.tf
# Output value definitions
output "lambda_bucket_name" {
description = "Name of the S3 bucket used to store function code."
value = aws_s3_bucket.lambda_bucket.id
}
output "function_name_ingestion" {
description = "Name of function"
value = module.lambda_ingestion.lambda[0].function_name
}
output "function_alias_name_ingestion" {
description = "Name of the function alias"
value = module.lambda_ingestion.alias[0].name
}
output "function_name_process_queue" {
description = "Name of function"
value = module.lambda_process_queue.lambda[0].function_name
}
output "function_alias_name_process_queue" {
description = "Name of the function alias"
value = module.lambda_process_queue.alias[0].name
}
export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1
terraform init
terraform plan
terraform apply -auto-approve
⚠️ Note: Remember to run terraform destroy -auto-approve after you are done with the module. Unless you wish to keep the infrastructure for personal use.You may need to run aws s3 rm --recursive s3://[your-s3-bucket-name] to remove any objects in there so terraform destroy does not throw error!
If you verify that the logs with the new changes then you are all good to go with the Lambda infrastructure setup!
If you are not going to use the infrastructure any more then remember to clean up and destroy it otherwise you may incur charges!
Here is the link to the completed version of this module - aws-webhook-series-part-1.
And that’s it! This step may not be that interesting but it covers a lot of the foundational infrastructure which we will be building on top of in the later modules.
Hopefully now you should have a better understanding of:
The file structure and the setup
The CI/CD process for our Lambda
How to set up a Lambda infrastructure using a terraform modules
How to manually invoke the lambda and verify it
In the next module, we will take all these manual steps that I showed here and automate them via Github actions.
That way, any new changes will be automatically deployed for us once we push the changes to github!
In addition, we will set up Open ID connect (OIDC) to securely manage our AWS credentials 🔒.
See you in the next module!
Then consider signing up to get notified when new content arrives!