Build a webhook microservice using AWS Technical series Part I

Goals

In this module, we will use this starter template - aws-webhook-series-starter as a starting point.

By the end of this module, you should:

• ✅ Understand the folder structure and it’s setup

• ✅ Be able to setup the basic scaffold of the AWS Lambda function with AWS Lambda Alias

• ✅ Be able to setup modules using terraform (for the lambda functions)

• ✅ Understand the CI/CD process of the Lambda function in our setup

• ✅ Understand each of the steps of the CI/CD

• ✅ Be able to Test out the lambda functions manually (using AWS CLI)

Overall, by doing these steps manually on local, it will help us build a deeper understanding of each step.

So, once we start automating the process, you would know each of the steps involved, and what the script is doing during a deployment.

Content

• Introduction
• Folder Structure
  • Top-level folder structure
  • Scripts within each workspace
  • Path mapping
• AWS Lambda function ingestion
  • 1. Create a S3 Bucket
  • 2. Create a S3 Bucket Object
  • 3. Create the Lambda Function
  • 4. Create the Lambda Alias
  • 5. Create the Lambda logging
  • 6. Create the Lambda role
  • 7. Add IAM permissions then attach it to the role
  • 8. Add outputs
• Test out the Ingestion Function
  • 1. Using AWS cli invoke the lambda function
• Create a new lambda version
  • What are the steps ?
  • Running the scripts
• Using the terraform module
  • 1. Creating resource for ingestion function
  • 2. Creating resource for process queue function
  • 3. Update our output.tf file on the top level
• Putting the modules to the test
  • 1. Apply the new infrastructure with modules
  • 2. Try to invoke the functions and test out the deployment with a change
  • 3. Remember to clean up
• Conclusion

Introduction

First, we will start adding the infrastructure for one of our functions.

Then, we will refactor it to use terraform modules, and see how these modules make it much easier for us to create multiple infrastructures.

After setting that up, we will manually test out the infrastructure by running the deployment script on local before we automate it with Github Actions.

Finally, we will test out the deployment of a new change on the lambda function (just to make sure things are working as expected).

Without further delay, let’s get started!

Folder structure

We are using pnpm workspace for this project.

Top-level folder structure

├── functions
│   ├── ingestion
│   └── process-queue
├── infra
├── node_modules
└── scripts

functions - contains all the source code for our AWS Lambda functions (ingestion & process-queue) infra - all terraform code related for infrastructure goes here scripts - These are all the scripts for the deployment process

Scripts within each workspace

Within each workspace, the source code will live in src .

In addition, there will be several commands available to manage the workspace.

These include:

• generating lambda assets (creating the zip)
• typechecking
• testing (with jest)
• build
• creating depedency (packaging the production node_modules in the initial zip)
• deploy (running the deployment script)

Path mapping

As mentioned, everything will live under src .

There is a special mapping configured for easier imports using @app/ which maps to everything understand src.

This applies to both code and test files.

AWS Lambda function Ingestion

Let’s start off by setting up the infrastructure for our ingestion function.

1. Create a S3 bucket

# main.tf

resource "aws_s3_bucket" "lambda_bucket" {
  bucket = "lambda-bucket-assets"
  acl    = "private"
}

⚠️ Note: You may need to update the bucket name if it is already taken.

📝 Helpful reference:

• Terraform - aws_s3_bucket

2. Create the S3 bucket object

This will our zipped files to AWS bucket.

The uuid() is mainly for differntiating between the AWS Lambda assets.

# main.tf

resource "aws_s3_bucket_object" "lambda_default" {
  bucket = aws_s3_bucket.lambda_bucket.id

  key    = "main-${uuid()}.zip"
  source = "../functions/ingestion/main.zip"
  etag   = filemd5("../functions/ingestion/main.zip")
}

📝 Helpful reference:

• Terraform - aws_s3_bucket

3. Create the Lambda Function

# main.tf


# Prefer to put at the top
locals {
  default_lambda_timeout = 10
  default_lambda_log_retention = 1
}

resource "aws_lambda_function" "this" {
  s3_bucket            = aws_s3_bucket.lambda_bucket.id
  s3_key               = aws_s3_bucket_object.lambda_default.key
  timeout              = local.default_lambda_timeout
  function_name        = "Ingestion-function"
  runtime              = "nodejs12.x"
  handler              = "dist/index.handler"
  publish              = true
  source_code_hash     = "${filebase64sha256("../functions/ingestion/main.zip")}"
  role                 = aws_iam_role.lambda_exec.arn
  environment {
    variables = {
      DefaultRegion = var.aws_region
    }
  }
}

Prefer to put the locals at the top of the file below aws provider in case we need to use it for other things

📝 Helpful reference:

• Terraform - aws_lambda_function

4. Create the Lambda Alias

This allows for easier management of the AWS Lambda versions.

# main.tf

resource "aws_lambda_alias" "this" {
  name             = "ingestion-dev"
  description      = "alias for the ingestion function"
  function_name    = aws_lambda_function.this.arn
  function_version = aws_lambda_function.this.version
}

📝 Helpful reference:

• The ultimate guide to AWS Lambda Alias
• Terraform - aws_lambda_alias

5. Create the Lambda logging

# main.tf

resource "aws_cloudwatch_log_group" "log_group" {
  name = "/aws/lambda/${aws_lambda_function.this.function_name}"
  retention_in_days = local.default_lambda_log_retention
}

📝 Helpful reference:

• Terraform - aws_cloudwatch_log_group

6. Create the Lambda role

This is the execution (or run-time) role for the AWS Lambda.

Within this role, we can give it IAM permissions to access other AWS Services (S3, Cloudwatch for logging and file etc).

# main.tf

resource "aws_iam_role" "lambda_exec" {
  name               = "ingestion-function-exec-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Sid       = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

📝 Helpful reference:

• Terraform - aws_iam_role

7. Add IAM permissions then attach it to the role

Create the runtime policy resource with permissions then attach it to our lambda role we just added above.

# main.tf

data "aws_iam_policy_document" "runtime_policy_doc" {
  version = "2012-10-17"
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
    effect = "Allow"
    resources = [
      "*"
    ]
  }
}

resource "aws_iam_policy" "lambda_runtime_policy" {
  name = "ingestion-function-runtime-policy"
  policy = data.aws_iam_policy_document.runtime_policy_doc.json
}

resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
  name       = "ingestion-function-lambda-role-attachment"
  roles      = [aws_iam_role.lambda_exec.name]
  policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}

📝 Helpful reference:

• Terraform - aws_iam_policy_document
• Terraform - aws_iam_policy
• Terraform - aws_iam_policy_attachment

8. Add outputs

This will be specific details we want to output from our infrastructure.

This is mainly for convenience but sometimes we need details like API gateway endpoint url which we won’t know until the infrastructure is applied.

# outputs.tf

# Output value definitions

output "lambda_bucket_name" {
  description = "Name of the S3 bucket used to store function code."
  value = aws_s3_bucket.lambda_bucket.id
}

output "function_name" {
  description = "Name of function"
  value = aws_lambda_function.this.function_name
}

output "function_alias_name" {
  description = "Name of the function alias"
  value = aws_lambda_alias.this.name
}

9. Apply our initial infrastructure

let’s use terraform to apply the infrastructure with what we have so far.

export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1

terraform init
terraform plan
terraform apply -auto-approve

Test out ingestion function

Let’s test out our ingestion function.

1. Using AWS cli invoke the lambda function

Let’s just make sure everything is working as expected and our infrastructure was setup properly.

aws lambda invoke \
    --function-name Ingestion-function:ingestion-dev \
    --invocation-type Event \
    --payload 'eyAibmFtZSI6ICJCb2IifQ==' \
    response.json

When invoking with Ingestion-function:ingestion-dev you can specify versions or in our case Lambda aliases

These two are very similar but you can think of Lambda alias as a container that points to a specific Lambda version.

If it was successful, you should see a output of:

{
    "StatusCode": 202
}

To verify it, you can also check the AWS Cloudwatch logs where within our function we are logging out the event.

This should be under CloudWatch > Log Groups > /aws/lambda/Ingestion-function :

📝 Helpful reference:

• AWS CLI - lambda invoke

Create a new lambda version

Ok, so everything looks to be working as expected with our lambda infrastructure.

How do we deploy new changes for this function ?

That is where the scripts comes in.

Within scripts/ there is a deployment.sh which would handle all the steps needed for the deployment.

What are those steps ?

Let’s take a look at the CI/CD for our Lambda functions:

1. Github trigger (push)

2. Install, test and build our code

• This will prepare a zip file at the end of this step

  • First the production dependencies node_modules are packaged into the zip (using the appropriate flags --production --prefer-offline etc)

  • Then we do another install to bring back the devDepdencies , in this step we will run the tests and the build with esbuild

  • finally, those changes will be merged into the same zip

3. Upload to S3

• After the zip file is created, it will be uploaded to AWS S3 as an artifact (labelled with an unique UUID)

4. Update our Lambda Function with the artifact and publish a new version

5. Update our Lambda Alias with the new function version

The steps we are most interested in at the moment are 2-5 which is what we are going to run locally.

Running the script

Before we run the script, add a small change to the console.log (maybe append a string) so we can see that new change.

Under the root directory of the folder, run the following:

export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1

AWS_LAMBDA_FUNCTION_NAME=Ingestion-function \
AWS_S3_BUCKET=lambda-bucket-assets-1234567 \
AWS_LAMBDA_ALIAS_NAME=ingestion-dev \
MODULE=@function/ingestion \
./scripts/deployment.sh

under CloudWatch > Log Groups > /aws/lambda/Ingestion-function you should see the new change:

Assuming everything was deployed as expected and there was no error, it should have deployed the new version by running the above script!

Using the terraform module

Ok, so that was just the infrastructure for the ingestion function.

Let’s do the same thing for the other functions but this time we will lean into the power of terraform modules.

Why use terraform modules when we can just copy and paste ?

Certainly we can just copy and paste but there a few benefits with abstracting this logic into a module.

What are some of the benefit of doing this ?

• Better Consistency

• Reducing cognitive load for future changes (less code)

• Better maintaibility (less code)

• Centralizing implementation for more efficient security audits & hardening efforts (especially important for IaC)

• Simplify the infrastructure process

• Improving developer experience (ie hiding away the complexity of the infrastructure)

Of course, there are cons to premature abstraction but I think in this case it makes sense as the infrastructure for both of these Lambda functions are almost identical with some differences (which we will support as a customization option of the module).

I won’t go into the details of all the options in the module but the gist of it is we are abstracting all the resources into a simplified interface to make it easier for us to create multiple similar infrastructure (ie Lambda functions).

if you want to learn more about this terraform module, I recommend reading through the post below - “Simplify your AWS Infrastructure in 9 steps”.

I go through this in detail on how to evolve an existing infrastructure to one using terraform module.

📝 Helpful reference:

• Terraform - Simplify your AWS Infrastructure in 9 steps

1. Creating resource for ingestion function

module "lambda_ingestion" {
  source               = "./modules/lambda"
  code_src             = "../functions/ingestion/main.zip"
  bucket_id            = aws_s3_bucket.lambda_bucket.id
  timeout              = local.default_lambda_timeout
  function_name        = "Ingestion-function"
  runtime              = "nodejs12.x"
  handler              = "dist/index.handler"
  publish              = true
  alias_name           = "ingestion-dev"
  alias_description    = "Alias for ingestion function"
  log_retention        = local.default_lambda_log_retention
  environment_vars = {
    DefaultRegion   = var.aws_region
  }
}

2. Creating resource for process queue function

module "lambda_process_queue" {
  source               = "./modules/lambda"
  code_src             = "../functions/process-queue/main.zip"
  bucket_id            = aws_s3_bucket.lambda_bucket.id
  timeout              = local.default_lambda_timeout
  function_name        = "Process-Queue-function"
  runtime              = "nodejs12.x"
  handler              = "dist/index.handler"
  publish              = true
  alias_name           = "process-queue-dev"
  alias_description    = "Alias for ingestion function"
  log_retention        = local.default_lambda_log_retention
  environment_vars = {
    DefaultRegion   = var.aws_region
  }
}

3. Update our output.tf file on the top level

Since we moved all our Lambda resources into a module and exported them via the module, we need to update our output on the top level.

Change the outputs.tf file to the following:

#outputs.tf

# Output value definitions

output "lambda_bucket_name" {
  description = "Name of the S3 bucket used to store function code."
  value = aws_s3_bucket.lambda_bucket.id
}

output "function_name_ingestion" {
  description = "Name of function"
  value = module.lambda_ingestion.lambda[0].function_name
}

output "function_alias_name_ingestion" {
  description = "Name of the function alias"
  value = module.lambda_ingestion.alias[0].name
}

output "function_name_process_queue" {
  description = "Name of function"
  value = module.lambda_process_queue.lambda[0].function_name
}

output "function_alias_name_process_queue" {
  description = "Name of the function alias"
  value = module.lambda_process_queue.alias[0].name
}

Putting the modules to the test

1. Apply the new infrastructure with modules

export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1

terraform init
terraform plan
terraform apply -auto-approve

⚠️ Note: Remember to run terraform destroy -auto-approve after you are done with the module. Unless you wish to keep the infrastructure for personal use.

You may need to run aws s3 rm --recursive s3://[your-s3-bucket-name] to remove any objects in there so terraform destroy does not throw error!

2. Try to invoke the functions and test out the deployment with a change

If you verify that the logs with the new changes then you are all good to go with the Lambda infrastructure setup!

3. Remember to clean up

If you are not going to use the infrastructure any more then remember to clean up and destroy it otherwise you may incur charges!

Conclusion

Here is the link to the completed version of this module - aws-webhook-series-part-1.

And that’s it! This step may not be that interesting but it covers a lot of the foundational infrastructure which we will be building on top of in the later modules.

Hopefully now you should have a better understanding of:

• The file structure and the setup

• The CI/CD process for our Lambda

• How to set up a Lambda infrastructure using a terraform modules

• How to manually invoke the lambda and verify it

In the next module, we will take all these manual steps that I showed here and automate them via Github actions.

That way, any new changes will be automatically deployed for us once we push the changes to github!

In addition, we will set up Open ID connect (OIDC) to securely manage our AWS credentials 🔒.

See you in the next module!