Published on: Sun Jun 12 2022
Last Updated on: Sun Jun 19 2022
In this module, we will use this starter template - aws-lambda-terraform-module-refactor-starter to simplify our infrastructure by using terraform modules!
By the end of this tutorial, you should:
✅ Have a better understanding of the benefit of using terraform modules
✅ Have a better understanding of the AWS Lambda terraform module we are using in the context of this project
The task is quite simple really. We have an existing terraform infrastructure ready for one of our functions.
This infrastructure works and provisions just fine. Now when it comes to provisioning the other function, the question becomes:
Do we just copy and paste the same thing and change a few things ?
Well, I think we can do better. Let’s look at a visualization of what we really need and what is different in the infrastructure between the two functions.
Looking at the visualization and the services that is being used in our functions.
Apart from adding different naming for our resources, and minor configuration options, the one element that really changes the most is the IAM permissions.
In our case, it does makes sense to ceate a common module to be used between these two functions and just offer configuration options for cusomization (ie IAM permissions).
What are some of the benefit of doing this ?
Better Consistency
Reducing cognitive load for future changes (less code)
Better maintaibility (less code)
Centralizing implementation for more efficient security audits & hardening efforts (especially important for IaC)
Simplify the infrastructure process
Improving developer experience (ie hiding away the complexity of the infrastructure)
Of course, there is always a trade off to be made. Premature refactors of an infrastructure into a module can also be a bad thing.
Just be sure your abstractions offer trade off you are willing to live with in the long term!
💡 Here is what I found works for me:If you find yourself copying and pasting similar infrastructure from your other projects and just changing a few minor things often, then its probably time put this infrastructure into a module.
Using our starter repo, the infrastructure included is just for the ingestion function.
Let’s create a terraform module and leverage that to create both of the infrastructure for our functions.
Within infra/
create a new directory called modules/lambda/
then create the relevant files.
mkdir -p modules/lambda && touch modules/lambda/main.tf && touch modules/lambda/outputs.tf && touch modules/lambda/variables.tf
resource "aws_s3_bucket_object" "lambda_default" {
bucket = aws_s3_bucket.lambda_bucket.id
key = "main-${uuid()}.zip"
source = "../functions/ingestion/main.zip"
etag = filemd5("../functions/ingestion/main.zip")
}
resource "aws_lambda_function" "this" {
s3_bucket = aws_s3_bucket.lambda_bucket.id
s3_key = aws_s3_bucket_object.lambda_default.key
timeout = local.default_lambda_timeout
function_name = "Ingestion-function"
runtime = "nodejs12.x"
handler = "dist/index.handler"
publish = true
source_code_hash = "${filebase64sha256("../functions/ingestion/main.zip")}"
role = aws_iam_role.lambda_exec.arn
environment {
variables = {
DefaultRegion = var.aws_region
}
}
}
resource "aws_lambda_alias" "this" {
name = "ingestion-dev"
description = "alias for the ingestion function"
function_name = aws_lambda_function.this.arn
function_version = aws_lambda_function.this.version
}
resource "aws_cloudwatch_log_group" "log_group" {
name = "/aws/lambda/${aws_lambda_function.this.function_name}"
retention_in_days = local.default_lambda_log_retention
}
resource "aws_iam_role" "lambda_exec" {
name = "ingestion-function-exec-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
Service = "lambda.amazonaws.com"
}
}]
})
}
data "aws_iam_policy_document" "runtime_policy_doc" {
version = "2012-10-17"
statement {
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
]
effect = "Allow"
resources = [
"*"
]
}
}
resource "aws_iam_policy" "lambda_runtime_policy" {
name = "ingestion-function-runtime-policy"
policy = data.aws_iam_policy_document.runtime_policy_doc.json
}
resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
name = "ingestion-function-lambda-role-attachment"
roles = [aws_iam_role.lambda_exec.name]
policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}
Notice the dynamic
on the aws_lambda_function
."environment" field, this allows us to iterate over the environment_vars
we pass into this modules.
resource "aws_s3_bucket_object" "lambda_default" {
bucket = var.bucket_id
key = "main-${uuid()}.zip"
source = var.code_src
etag = filemd5(var.code_src)
}
resource "aws_lambda_function" "this" {
function_name = var.function_name
s3_bucket = var.bucket_id
s3_key = aws_s3_bucket_object.lambda_default.key
runtime = var.runtime
handler = var.handler
publish = var.publish
source_code_hash = "${filebase64sha256(var.code_src)}"
role = aws_iam_role.lambda_exec.arn
dynamic "environment" {
for_each = length(keys(var.environment_vars)) == 0 ? [] : [true]
content {
variables = var.environment_vars
}
}
}
resource "aws_lambda_alias" "this" {
name = "ingestion-dev"
description = "alias for the ingestion function"
function_name = aws_lambda_function.this.arn
function_version = aws_lambda_function.this.version
}
resource "aws_cloudwatch_log_group" "log_group" {
name = "/aws/lambda/${aws_lambda_function.this.function_name}"
retention_in_days = local.default_lambda_log_retention
}
resource "aws_iam_role" "lambda_exec" {
name = "ingestion-function-exec-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
Service = "lambda.amazonaws.com"
}
}]
})
}
data "aws_iam_policy_document" "runtime_policy_doc" {
version = "2012-10-17"
statement {
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
]
effect = "Allow"
resources = [
"*"
]
}
}
resource "aws_iam_policy" "lambda_runtime_policy" {
name = "ingestion-function-runtime-policy"
policy = data.aws_iam_policy_document.runtime_policy_doc.json
}
resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
name = "ingestion-function-lambda-role-attachment"
roles = [aws_iam_role.lambda_exec.name]
policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}
resource "aws_s3_bucket_object" "lambda_default" {
bucket = var.bucket_id
key = "main-${uuid()}.zip"
source = var.code_src
etag = filemd5(var.code_src)
}
resource "aws_lambda_function" "this" {
function_name = var.function_name
s3_bucket = var.bucket_id
s3_key = aws_s3_bucket_object.lambda_default.key
runtime = var.runtime
handler = var.handler
publish = var.publish
source_code_hash = "${filebase64sha256(var.code_src)}"
role = aws_iam_role.lambda_exec.arn
dynamic "environment" {
for_each = length(keys(var.environment_vars)) == 0 ? [] : [true]
content {
variables = var.environment_vars
}
}
}
resource "aws_lambda_alias" "this" {
name = var.alias_name
description = var.alias_description != "" ? var.alias_description : "description ${var.alias_name}"
function_name = aws_lambda_function.this.arn
function_version = aws_lambda_function.this.version
}
resource "aws_cloudwatch_log_group" "log_group" {
name = "/aws/lambda/${aws_lambda_function.this.function_name}"
retention_in_days = var.log_retention
}
resource "aws_iam_role" "lambda_exec" {
name = "ingestion-function-exec-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
Service = "lambda.amazonaws.com"
}
}]
})
}
data "aws_iam_policy_document" "runtime_policy_doc" {
version = "2012-10-17"
statement {
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
]
effect = "Allow"
resources = [
"*"
]
}
}
resource "aws_iam_policy" "lambda_runtime_policy" {
name = "ingestion-function-runtime-policy"
policy = data.aws_iam_policy_document.runtime_policy_doc.json
}
resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
name = "ingestion-function-lambda-role-attachment"
roles = [aws_iam_role.lambda_exec.name]
policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}
Same thing as above, we will add an option to customize the IAM permissions statements by using the dynamic block.
resource "aws_s3_bucket_object" "lambda_default" {
bucket = var.bucket_id
key = "main-${uuid()}.zip"
source = var.code_src
etag = filemd5(var.code_src)
}
resource "aws_lambda_function" "this" {
function_name = var.function_name
s3_bucket = var.bucket_id
s3_key = aws_s3_bucket_object.lambda_default.key
runtime = var.runtime
handler = var.handler
publish = var.publish
source_code_hash = "${filebase64sha256(var.code_src)}"
role = aws_iam_role.lambda_exec.arn
dynamic "environment" {
for_each = length(keys(var.environment_vars)) == 0 ? [] : [true]
content {
variables = var.environment_vars
}
}
}
resource "aws_lambda_alias" "this" {
name = var.alias_name
description = var.alias_description != "" ? var.alias_description : "description ${var.alias_name}"
function_name = aws_lambda_function.this.arn
function_version = aws_lambda_function.this.version
}
resource "aws_cloudwatch_log_group" "log_group" {
name = "/aws/lambda/${aws_lambda_function.this.function_name}"
retention_in_days = var.log_retention
}
resource "aws_iam_role" "lambda_exec" {
name = "${lower(var.function_name)}-exec-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
Service = "lambda.amazonaws.com"
}
}]
})
}
data "aws_iam_policy_document" "runtime_policy_doc" {
version = "2012-10-17"
statement {
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
]
effect = "Allow"
resources = [
"*"
]
}
dynamic "statement" {
for_each = var.iam_statements != null ? var.iam_statements : {}
content {
actions = statement.value["actions"]
effect = statement.value["effect"]
resources = statement.value["resources"]
}
}
}
resource "aws_iam_policy" "lambda_runtime_policy" {
name = "${lower(var.function_name)}-runtime-policy"
policy = data.aws_iam_policy_document.runtime_policy_doc.json
}
resource "aws_iam_policy_attachment" "attach_policy_to_role_lambda" {
name = "${lower(var.function_name)}-lambda-role-attachment"
roles = [aws_iam_role.lambda_exec.name]
policy_arn = aws_iam_policy.lambda_runtime_policy.arn
}
within the modules/lambda/variables.tf
, add the variables defined in our main.tf
file:
variable "bucket_id" {
description = "The ID of the s3 bucket for storing Lambda artifacts"
type = string
}
variable "function_name" {
description = "The name of the lambda"
type = string
}
variable "runtime" {
description = "The lambda runtime"
default = "nodejs12.x"
type = string
}
variable "code_src" {
description = "The lambda code source path"
type = string
}
variable "timeout" {
description = "The lambda timeout"
default = 3
type = number
}
variable "handler" {
description = "The lambda handler definition"
default = "dist/index.handler"
type = string
}
variable "publish" {
description = "Whether or not to publish new version of lambda"
default = false
type = bool
}
variable "alias_name" {
description = "The lambda alias name"
type = string
}
variable "alias_description" {
description = "The description of the lambda alias"
default = ""
type = string
}
variable "log_retention" {
description = "The log retention time"
default = 30
type = number
}
variable "iam_statements" {
type = map(object({
actions = list(string)
effect = string
resources = list(string)
}))
description = "The IAM Statements for the Lambda (runtime)"
default = null
}
variable "alias_iam_statements" {
type = map(object({
actions = list(string)
effect = string
resources = list(string)
}))
description = "The IAM Statements for the Lambda (runtime)"
default = null
}
variable "environment_vars" {
description = "The environment variables for the Lambda Function"
type = map(string)
default = {}
}
Now onto the outputs of our module.
This is what gets exported from our module and can be referenced where the module is used. These include things like arn, lambda alias invoke arn etc.
In modules/lambda/outputs.tf
, add the following:
output "lambda" {
value = aws_lambda_function.this[*]
}
output "alias" {
value = aws_lambda_alias.this[*]
}
output "log_group" {
value = aws_cloudwatch_log_group.log_group[*]
}
Since we moved all our Lambda resources into a module and exported them via the module, we need to update our output on the top level.
Change the outputs.tf
file to the following:
#outputs.tf
# Output value definitions
output "lambda_bucket_name" {
description = "Name of the S3 bucket used to store function code."
value = aws_s3_bucket.lambda_bucket.id
}
output "function_name_ingestion" {
description = "Name of function"
value = module.lambda_ingestion.lambda[0].function_name
}
output "function_alias_name_ingestion" {
description = "Name of the function alias"
value = module.lambda_ingestion.alias[0].name
}
output "function_name_process_queue" {
description = "Name of function"
value = module.lambda_process_queue.lambda[0].function_name
}
output "function_alias_name_process_queue" {
description = "Name of the function alias"
value = module.lambda_process_queue.alias[0].name
}
Change your main.tf
file to the following to leverage the terraform modules we just wrote for our lambda functions.
As you can see, we went from the verbose file with multiple resources to a simple one with a set of options to pass into one module.
Isn’t this so much nicer !?
# main.tf
# Infrastructure definitions
provider "aws" {
version = "~> 2.0"
region = var.aws_region
}
# Local vars
locals {
default_lambda_timeout = 10
default_lambda_log_retention = 1
}
resource "aws_s3_bucket" "lambda_bucket" {
bucket = "lambda-bucket-assets-1"
acl = "private"
}
module "lambda_ingestion" {
source = "./modules/lambda"
code_src = "../functions/ingestion/main.zip"
bucket_id = aws_s3_bucket.lambda_bucket.id
timeout = local.default_lambda_timeout
function_name = "Ingestion-function"
runtime = "nodejs12.x"
handler = "dist/index.handler"
publish = true
alias_name = "ingestion-dev"
alias_description = "Alias for ingestion function"
log_retention = local.default_lambda_log_retention
environment_vars = {
DefaultRegion = var.aws_region
}
}
module "lambda_process_queue" {
source = "./modules/lambda"
code_src = "../functions/process-queue/main.zip"
bucket_id = aws_s3_bucket.lambda_bucket.id
timeout = local.default_lambda_timeout
function_name = "Process-Queue-function"
runtime = "nodejs12.x"
handler = "dist/index.handler"
publish = true
alias_name = "process-queue-dev"
alias_description = "Alias for ingestion function"
log_retention = local.default_lambda_log_retention
environment_vars = {
DefaultRegion = var.aws_region
}
}
⚠️ Note: Before you run the terraform command, you need to generate the zip assets for the lambda functions.Please use pnpm version >= 6.20.x to ensure there are no issues with this step.
Before we setup the infrastructure, we need to generate zip files for our lambda functions.
Under the root directory, run the following command:
pnpm run generate-assets -r
export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1
terraform init
terraform plan
terraform apply -auto-approve
⚠️ Note: Remember to run terraform destroy -auto-approve after you are done with the module. Unless you wish to keep the infrastructure for personal use.You may need to run aws s3 rm --recursive s3://[your-s3-bucket-name] to remove any objects in there so terraform destroy does not throw error!
Let’s just make sure everything is working as expected and our infrastructure was setup properly.
aws lambda invoke \
--function-name Ingestion-function:ingestion-dev \
--invocation-type Event \
--payload 'eyAibmFtZSI6ICJCb2IifQ==' \
response.json
When invoking with Ingestion-function:ingestion-dev you can specify versions or in our case Lambda aliasesThese two are very similar but you can think of Lambda alias as a container that points to a specific Lambda version.
If it was successful, you should see a output of:
{
"StatusCode": 202
}
To verify it, you can also check the AWS Cloudwatch logs where within our function we are logging out the event.
This should be under CloudWatch > Log Groups > /aws/lambda/Ingestion-function
:
If you are not going to use the infrastructure any more then remember to clean up and destroy it otherwise you may incur charges!
Here is the link to the completed version of this module - aws-lambda-terraform-module-refactor-completed for reference.
Terraform module is a great way to help simplify your infrastructure by abstracting the common resources into a module that you can just include then provide it options.
I hope this helps you gain a better understanding of terraform module. They are a powerful tool but use them when the benefits outweigh the drawbacks!
Let me know if you decide create a terraform module for something else. I’d love to hear how you are using it and what for! ✌️
Then consider signing up to get notified when new content arrives!