What content was recently changed ?

Update section about running command for generating the zip file before setting up the infrastructure

Goals

In this module, we will use this starter template - aws-lambda-terraform-module-refactor-starter to simplify our infrastructure by using terraform modules!

By the end of this tutorial, you should:

✅ Have a better understanding of the benefit of using terraform modules
✅ Have a better understanding of the AWS Lambda terraform module we are using in the context of this project

Content

Introduction
Terraform module breakdown
Creating the terraform module
- 1. Initialize the modules
- 2. Copy all the Lambda Functions over
- 3. Refactor the S3 bucket and Lambda to be flexible
- 4. Refactor the Lambda alais and the Cloudwach log to be flexible
- 5. Refactor the IAM role and permissions to be flexible
- 6. Add the variables for the module
- 7. Add the outputs for the module
- 8. Update our output.tf file on the top level
- 9. Update main.tf to use the modules
Putting the infrastructure to the test
Conclusion

Introduction

The task is quite simple really. We have an existing terraform infrastructure ready for one of our functions.

This infrastructure works and provisions just fine. Now when it comes to provisioning the other function, the question becomes:

Do we just copy and paste the same thing and change a few things ?

Well, I think we can do better. Let’s look at a visualization of what we really need and what is different in the infrastructure between the two functions.

Terraform module breakdown

Looking at the visualization and the services that is being used in our functions.

Apart from adding different naming for our resources, and minor configuration options, the one element that really changes the most is the IAM permissions.

In our case, it does makes sense to ceate a common module to be used between these two functions and just offer configuration options for cusomization (ie IAM permissions).

What are some of the benefit of doing this ?

Better Consistency
Reducing cognitive load for future changes (less code)
Better maintaibility (less code)
Centralizing implementation for more efficient security audits & hardening efforts (especially important for IaC)
Simplify the infrastructure process
Improving developer experience (ie hiding away the complexity of the infrastructure)

Of course, there is always a trade off to be made. Premature refactors of an infrastructure into a module can also be a bad thing.

Just be sure your abstractions offer trade off you are willing to live with in the long term!

💡 Here is what I found works for me:
If you find yourself copying and pasting similar infrastructure from your other projects and just changing a few minor things often, then its probably time put this infrastructure into a module.

Creating the terraform module

Using our starter repo, the infrastructure included is just for the ingestion function.

Let’s create a terraform module and leverage that to create both of the infrastructure for our functions.

1. Initialize the modules

Within infra/ create a new directory called modules/lambda/ then create the relevant files.

mkdir -p modules/lambda && touch modules/lambda/main.tf && touch modules/lambda/outputs.tf && touch modules/lambda/variables.tf

2. Copy all the Lambda Functions over

resource "aws_s3_bucket_object" "lambda_default" {
  bucket = aws_s3_bucket.lambda_bucket.id

  key    = "main-${uuid()}.zip"
  source = "../functions/ingestion/main.zip"
  etag   = filemd5("../functions/ingestion/main.zip")
}

resource "aws_lambda_function" "this" {
  s3_bucket            = aws_s3_bucket.lambda_bucket.id
  s3_key               = aws_s3_bucket_object.lambda_default.key

  timeout              = local.default_lambda_timeout
  function_name        = "Ingestion-function"
  runtime              = "nodejs12.x"
  handler              = "dist/index.handler"
  publish              = true
  source_code_hash = "${filebase64sha256("../functions/ingestion/main.zip")}"
  role             = aws_iam_role.lambda_exec.arn
  environment {
    variables = {
      DefaultRegion = var.aws_region
    }
  }
}

resource "aws_lambda_alias" "this" {
  name             = "ingestion-dev"
  description      = "alias for the ingestion function"
  function_name    = aws_lambda_function.this.arn
  function_version = aws_lambda_function.this.version
}

resource "aws_cloudwatch_log_group" "log_group" {
  name = "/aws/lambda/${aws_lambda_function.this.function_name}"
  retention_in_days = local.default_lambda_log_retention
}

resource "aws_iam_role" "lambda_exec" {
  name               = "ingestion-function-exec-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Sid       = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

data "aws_iam_policy_document" "runtime_policy_doc" {
  version = "2012-10-17"
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
    effect = "Allow"
    resources = [
      "*"
    ]
  }
}

resource "aws_iam_policy" "lambda_runtime_policy" {
  name = "ingestion-function-runtime-policy"
  policy = data.aws_iam_policy_document.runtime_policy_doc.json
}

resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
  name       = "ingestion-function-lambda-role-attachment"
  roles      = [aws_iam_role.lambda_exec.name]
  policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}

3. Refactor the S3 bucket and Lambda to be flexible

Notice the dynamic on the aws_lambda_function."environment" field, this allows us to iterate over the environment_vars we pass into this modules.

resource "aws_s3_bucket_object" "lambda_default" {
  bucket = var.bucket_id
  key    = "main-${uuid()}.zip"
  source = var.code_src
  etag = filemd5(var.code_src)
}

resource "aws_lambda_function" "this" {
  function_name    = var.function_name
  s3_bucket        = var.bucket_id
  s3_key           = aws_s3_bucket_object.lambda_default.key
  runtime          = var.runtime
  handler          = var.handler
  publish          = var.publish
  source_code_hash = "${filebase64sha256(var.code_src)}"
  role             = aws_iam_role.lambda_exec.arn
  dynamic "environment" {
    for_each = length(keys(var.environment_vars)) == 0 ? [] : [true]
    content {
      variables = var.environment_vars
    }
  }
}

resource "aws_lambda_alias" "this" {
  name             = "ingestion-dev"
  description      = "alias for the ingestion function"
  function_name    = aws_lambda_function.this.arn
  function_version = aws_lambda_function.this.version
}

resource "aws_cloudwatch_log_group" "log_group" {
  name = "/aws/lambda/${aws_lambda_function.this.function_name}"
  retention_in_days = local.default_lambda_log_retention
}

resource "aws_iam_role" "lambda_exec" {
  name               = "ingestion-function-exec-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Sid       = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

data "aws_iam_policy_document" "runtime_policy_doc" {
  version = "2012-10-17"
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
    effect = "Allow"
    resources = [
      "*"
    ]
  }
}

resource "aws_iam_policy" "lambda_runtime_policy" {
  name = "ingestion-function-runtime-policy"
  policy = data.aws_iam_policy_document.runtime_policy_doc.json
}

resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
  name       = "ingestion-function-lambda-role-attachment"
  roles      = [aws_iam_role.lambda_exec.name]
  policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}

📝 Helpful reference:

Terraform - dynamic blocks

4. Refactor the Lambda alais and the Cloudwach log to be flexible

resource "aws_s3_bucket_object" "lambda_default" {
  bucket = var.bucket_id
  key    = "main-${uuid()}.zip"
  source = var.code_src
  etag = filemd5(var.code_src)
}

resource "aws_lambda_function" "this" {
  function_name    = var.function_name
  s3_bucket        = var.bucket_id
  s3_key           = aws_s3_bucket_object.lambda_default.key
  runtime          = var.runtime
  handler          = var.handler
  publish          = var.publish
  source_code_hash = "${filebase64sha256(var.code_src)}"
  role             = aws_iam_role.lambda_exec.arn
  dynamic "environment" {
    for_each = length(keys(var.environment_vars)) == 0 ? [] : [true]
    content {
      variables = var.environment_vars
    }
  }
}

resource "aws_lambda_alias" "this" {
  name             = var.alias_name
  description      = var.alias_description != "" ? var.alias_description : "description ${var.alias_name}"
  function_name    = aws_lambda_function.this.arn
  function_version = aws_lambda_function.this.version
}

resource "aws_cloudwatch_log_group" "log_group" {
  name = "/aws/lambda/${aws_lambda_function.this.function_name}"
  retention_in_days = var.log_retention
}

resource "aws_iam_role" "lambda_exec" {
  name               = "ingestion-function-exec-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Sid       = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

data "aws_iam_policy_document" "runtime_policy_doc" {
  version = "2012-10-17"
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
    effect = "Allow"
    resources = [
      "*"
    ]
  }
}

resource "aws_iam_policy" "lambda_runtime_policy" {
  name = "ingestion-function-runtime-policy"
  policy = data.aws_iam_policy_document.runtime_policy_doc.json
}

resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
  name       = "ingestion-function-lambda-role-attachment"
  roles      = [aws_iam_role.lambda_exec.name]
  policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}

5. Refactor the IAM role and permissions to be flexible

Same thing as above, we will add an option to customize the IAM permissions statements by using the dynamic block.

resource "aws_s3_bucket_object" "lambda_default" {
  bucket = var.bucket_id
  key    = "main-${uuid()}.zip"
  source = var.code_src
  etag = filemd5(var.code_src)
}

resource "aws_lambda_function" "this" {
  function_name    = var.function_name
  s3_bucket        = var.bucket_id
  s3_key           = aws_s3_bucket_object.lambda_default.key
  runtime          = var.runtime
  handler          = var.handler
  publish          = var.publish
  source_code_hash = "${filebase64sha256(var.code_src)}"
  role             = aws_iam_role.lambda_exec.arn
  dynamic "environment" {
    for_each = length(keys(var.environment_vars)) == 0 ? [] : [true]
    content {
      variables = var.environment_vars
    }
  }
}

resource "aws_lambda_alias" "this" {
  name             = var.alias_name
  description      = var.alias_description != "" ? var.alias_description : "description ${var.alias_name}"
  function_name    = aws_lambda_function.this.arn
  function_version = aws_lambda_function.this.version
}

resource "aws_cloudwatch_log_group" "log_group" {
  name = "/aws/lambda/${aws_lambda_function.this.function_name}"
  retention_in_days = var.log_retention
}

resource "aws_iam_role" "lambda_exec" {
  name               = "${lower(var.function_name)}-exec-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Sid       = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

data "aws_iam_policy_document" "runtime_policy_doc" {
  version = "2012-10-17"
  statement {
      actions = [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
      ]
      effect = "Allow"
      resources = [
        "*"
      ]
  }
  dynamic "statement" {
    for_each = var.iam_statements != null ? var.iam_statements : {}
    content {
      actions   = statement.value["actions"]
      effect    = statement.value["effect"]
      resources = statement.value["resources"]
    }
  }
}

resource "aws_iam_policy" "lambda_runtime_policy" {
  name = "${lower(var.function_name)}-runtime-policy"
  policy = data.aws_iam_policy_document.runtime_policy_doc.json
}

resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
  name       = "${lower(var.function_name)}-lambda-role-attachment"
  roles      = [aws_iam_role.lambda_exec.name]
  policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}

6. Add the variables for the module

within the modules/lambda/variables.tf, add the variables defined in our main.tf file:

variable "bucket_id" {
  description = "The ID of the s3 bucket for storing Lambda artifacts"
  type        = string
}

variable "function_name" {
  description = "The name of the lambda"
  type        = string
}

variable "runtime" {
  description = "The lambda runtime"
  default     = "nodejs12.x"
  type        = string
}

variable "code_src" {
  description = "The lambda code source path"
  type        = string
}

variable "timeout" {
  description = "The lambda timeout"
  default     = 3
  type        = number
}

variable "handler" {
  description = "The lambda handler definition"
  default     = "dist/index.handler"
  type        = string
}

variable "publish" {
  description = "Whether or not to publish new version of lambda"
  default     = false
  type        = bool
}

variable "alias_name" {
  description = "The lambda alias name"
  type        = string
}

variable "alias_description" {
  description = "The description of the lambda alias"
  default     = ""
  type        = string
}

variable "log_retention" {
  description = "The log retention time"
  default     = 30
  type        = number
}

variable "iam_statements" {
  type = map(object({
    actions   = list(string)
    effect    = string
    resources = list(string)
  }))
  description = "The IAM Statements for the Lambda (runtime)"
  default     = null
}

variable "alias_iam_statements" {
  type = map(object({
    actions   = list(string)
    effect    = string
    resources = list(string)
  }))
  description = "The IAM Statements for the Lambda (runtime)"
  default     = null
}

variable "environment_vars" {
  description = "The environment variables for the Lambda Function"
  type        = map(string)
  default     = {}
}

7. Add the outputs for the module

Now onto the outputs of our module.

This is what gets exported from our module and can be referenced where the module is used. These include things like arn, lambda alias invoke arn etc.

In modules/lambda/outputs.tf, add the following:

output "lambda" {
  value = aws_lambda_function.this[*]
}

output "alias" {
  value = aws_lambda_alias.this[*]
}

output "log_group" {
  value = aws_cloudwatch_log_group.log_group[*]
}

8. Update our output.tf file on the top level

Since we moved all our Lambda resources into a module and exported them via the module, we need to update our output on the top level.

Change the outputs.tf file to the following:

#outputs.tf

# Output value definitions

output "lambda_bucket_name" {
  description = "Name of the S3 bucket used to store function code."
  value = aws_s3_bucket.lambda_bucket.id
}

output "function_name_ingestion" {
  description = "Name of function"
  value = module.lambda_ingestion.lambda[0].function_name
}

output "function_alias_name_ingestion" {
  description = "Name of the function alias"
  value = module.lambda_ingestion.alias[0].name
}

output "function_name_process_queue" {
  description = "Name of function"
  value = module.lambda_process_queue.lambda[0].function_name
}

output "function_alias_name_process_queue" {
  description = "Name of the function alias"
  value = module.lambda_process_queue.alias[0].name
}

9. Update main.tf to use the modules

Change your main.tf file to the following to leverage the terraform modules we just wrote for our lambda functions.

As you can see, we went from the verbose file with multiple resources to a simple one with a set of options to pass into one module.

Isn’t this so much nicer !?

# main.tf

# Infrastructure definitions

provider "aws" {
  version = "~> 2.0"
  region  = var.aws_region
}

# Local vars
locals {
  default_lambda_timeout = 10
  default_lambda_log_retention = 1
}

resource "aws_s3_bucket" "lambda_bucket" {
  bucket        = "lambda-bucket-assets-1"
  acl           = "private"
}

module "lambda_ingestion" {
  source               = "./modules/lambda"
  code_src             = "../functions/ingestion/main.zip"
  bucket_id            = aws_s3_bucket.lambda_bucket.id
  timeout              = local.default_lambda_timeout
  function_name        = "Ingestion-function"
  runtime              = "nodejs12.x"
  handler              = "dist/index.handler"
  publish              = true
  alias_name           = "ingestion-dev"
  alias_description    = "Alias for ingestion function"
  log_retention        = local.default_lambda_log_retention
  environment_vars = {
    DefaultRegion   = var.aws_region
  }
}

module "lambda_process_queue" {
  source               = "./modules/lambda"
  code_src             = "../functions/process-queue/main.zip"
  bucket_id            = aws_s3_bucket.lambda_bucket.id
  timeout              = local.default_lambda_timeout
  function_name        = "Process-Queue-function"
  runtime              = "nodejs12.x"
  handler              = "dist/index.handler"
  publish              = true
  alias_name           = "process-queue-dev"
  alias_description    = "Alias for ingestion function"
  log_retention        = local.default_lambda_log_retention
  environment_vars = {
    DefaultRegion   = var.aws_region
  }
}

Putting the new infrastructure to the test

⚠️ Note: Before you run the terraform command, you need to generate the zip assets for the lambda functions.
Please use pnpm version >= 6.20.x to ensure there are no issues with this step.

1. Generate zip files for lambda functions

Before we setup the infrastructure, we need to generate zip files for our lambda functions.

Under the root directory, run the following command:

pnpm run generate-assets -r

2. Apply the new infrastructure with modules

export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1

terraform init
terraform plan
terraform apply -auto-approve

⚠️ Note: Remember to run terraform destroy -auto-approve after you are done with the module. Unless you wish to keep the infrastructure for personal use.
You may need to run aws s3 rm --recursive s3://[your-s3-bucket-name] to remove any objects in there so terraform destroy does not throw error!

3. Using AWS cli invoke the lambda function

Let’s just make sure everything is working as expected and our infrastructure was setup properly.

aws lambda invoke \
    --function-name Ingestion-function:ingestion-dev \
    --invocation-type Event \
    --payload 'eyAibmFtZSI6ICJCb2IifQ==' \
    response.json

When invoking with Ingestion-function:ingestion-dev you can specify versions or in our case Lambda aliases
These two are very similar but you can think of Lambda alias as a container that points to a specific Lambda version.

If it was successful, you should see a output of:

{
    "StatusCode": 202
}

To verify it, you can also check the AWS Cloudwatch logs where within our function we are logging out the event.

This should be under CloudWatch > Log Groups > /aws/lambda/Ingestion-function:

📝 Helpful reference:

AWS CLI - lambda invoke

4. Remember to clean up

If you are not going to use the infrastructure any more then remember to clean up and destroy it otherwise you may incur charges!

Conclusion

Here is the link to the completed version of this module - aws-lambda-terraform-module-refactor-completed for reference.

Terraform module is a great way to help simplify your infrastructure by abstracting the common resources into a module that you can just include then provide it options.

I hope this helps you gain a better understanding of terraform module. They are a powerful tool but use them when the benefits outweigh the drawbacks!

Let me know if you decide create a terraform module for something else. I’d love to hear how you are using it and what for! ✌️

Jerry Chang

Go Back

Simplify your AWS Lamba Infrastructure in 9 steps

Enjoy the content ?