Published on: Thu Jan 27 2022
In this module, we will be setting up the foundation of our AWS infrastructure, which will include:
The main focus here to is setup the foundational infrastructure for us to build on. Then as we progress through the modules, we will start layering on top of the existing foundation.
Here is what that will look like:
As a refresher, the AWS Aurora architecture requires a minimum of 3 availability zones when initializing. So, we will need to ensure that our VPC supports that.
This is because AWS Aurora replicates 2 copies of our the data we write to the database in 3 availibility zones (AZs).
In your repository, please ensure you have the following:
locals {
# Omitted for brevity
vpc_azs = ["us-east-1a", "us-east-1b", "us-east-1d"]
}
module "networking" {
source = "github.com/Jareechang/tf-modules//networking?ref=v1.0.20"
env = var.env
project_id = var.project_id
subnet_public_cidrblock = [
"10.0.1.0/24",
"10.0.2.0/24",
"10.0.3.0/24"
]
subnet_private_cidrblock = [
"10.0.10.0/24",
"10.0.21.0/24",
"10.0.31.0/24"
]
azs = local.vpc_azs
}
Or at very least, a VPC which supports 3 availability zones, public & private subnets.
Feel free to use this starter repository - aws-aurora-starter.
On to the AWS Aurora, there will be a few compoennts that we will be setting up.
Specifically:
For the AWS Aurora, we are creating a RDS cluster endpoint so we don’t have connect directly to the replicas themselves. The endpoint will load balance the connections between the replicas when using that endpoint.
The Database subnet groups are a collection of subnets that you can designate for running your clusters in the VPC.
resource "aws_db_subnet_group" "default" {
name = "main"
subnet_ids = module.networking.private_subnets[*].id
tags = {
Name = "DB Subnet group"
}
}
Here we are using terraform’s splat syntax to say we want to designate all the private subnets within our VPC to run our AWS Aurora clusters in.
The security group, on the other hand, restricts the inbound and outbound traffic of your resource.
resource "aws_security_group" "private_db_sg" {
name = "aurora-custom-sg"
description = "Custom default SG"
vpc_id = module.networking.vpc_id
ingress {
description = "TLS from VPC"
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [
# For the AWS ECS - we will deal with this later in the series
aws_security_group.ecs_sg.id
]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "PostgreSQL-db-access-${var.project_id}-${var.env}"
}
}
The gist of this is we are allow inbound from port 5432 (postgreSQL’s default port) and outbound for any traffic.
Here we are being extra restrictive by limiting access to particular the Security groups IDs (Bastion host and AWS ECS security groups).
In reality, you can probably get away with specifying to allow access if the CIDR block is coming from within our VPC, like so:
resource "aws_security_group" "private_db_sg" {
# omitted for brevity
ingress {
cidr_blocks = module.networking.private_subnets[*].cidr_block
}
}
A side benefit of this is we don’t have to update our security group when we expand our CIDR ranges if we run out of ip addresses within our VPC.
As mentioned, we are using AWS SSM Parameter store which is based on the “name” which is path based (ie you can store multiple different secrets under database/*
and retrieve them all).
#### Secrets
resource "random_password" "db" {
length = 16
special = false
}
resource "aws_kms_key" "default" {
description = "Default encryption key (symmetric)"
deletion_window_in_days = 7
}
resource "aws_ssm_parameter" "db_password" {
name = "/web/${var.project_id}/database/secret"
description = "Datbase password"
type = "SecureString"
key_id = aws_kms_key.default.key_id
value = random_password.db.result
}
💡 Tip: If you like a fully end to end managed secrets manager, feel free to check out AWS Secrets Managers. AWS Parameter store work fine but it does not offer automatic secret rotation.
Here we are setting up our aurora clusters and instances along the cluster endpoint which will contain our read replicas.
We are doing some fancy filtering (using terraform for
) to get the read only instance from our cluster instances being created.
# output.tf
output "aurora_cluster_endpoint" {
value = aws_rds_cluster_endpoint.static.endpoint
}
# main.tf
locals {
# Omitted for brevity, add to your existing locals
# Database
db_username = "read_user"
db_name = "blog"
db_port = 5432
db_engine = "aurora-postgresql"
db_engine_version = "12.7"
db_retention_period = 5
db_preferred_backup_window = "19:00-22:00"
}
resource "aws_rds_cluster" "default" {
apply_immediately = true
cluster_identifier = "aurora-cluster-${var.project_id}-${var.env}"
engine = local.db_engine
engine_version = local.db_engine_version
availability_zones = local.vpc_azs
database_name = local.db_name
master_username = local.db_username
master_password = random_password.db.result
backup_retention_period = local.db_retention_period
preferred_backup_window = local.db_preferred_backup_window
db_subnet_group_name = aws_db_subnet_group.default.id
vpc_security_group_ids = [
aws_security_group.private_db_sg.id
]
}
resource "aws_rds_cluster_instance" "cluster_instances" {
apply_immediately = true
count = 2
identifier = "aurora-cluster-${var.project_id}-${count.index}"
cluster_identifier = aws_rds_cluster.default.id
instance_class = "db.t3.medium"
engine = aws_rds_cluster.default.engine
publicly_accessible = false
db_subnet_group_name = aws_db_subnet_group.default.id
}
resource "aws_rds_cluster_endpoint" "static" {
cluster_identifier = aws_rds_cluster.default.id
cluster_endpoint_identifier = "static"
custom_endpoint_type = "READER"
static_members = [for i, instance in aws_rds_cluster_instance.cluster_instances : instance.id if instance.writer == false]
}
Let’s setup encrpytion for our storage volumes, snapshots and backups using AWS KMS.
The data will be encrypted using the provided AWS KMS key.
resource "aws_kms_key" "db_key" {
description = "KMS for database"
deletion_window_in_days = 7
}
resource "aws_rds_cluster" "default" {
apply_immediately = true
cluster_identifier = "aurora-cluster-${var.project_id}-${var.env}"
engine = local.db_engine
engine_version = local.db_engine_version
availability_zones = local.vpc_azs
database_name = local.db_name
master_username = local.db_username
master_password = random_password.db.result
backup_retention_period = local.db_retention_period
preferred_backup_window = local.db_preferred_backup_window
db_subnet_group_name = aws_db_subnet_group.default.id
storage_encrypted = true
kms_key_id = aws_kms_key.db_key.arn
vpc_security_group_ids = [
aws_security_group.private_db_sg.id
]
}
This configuration forces SSL any connection to always use SSL otherwise it would fail.
resource "aws_rds_cluster_parameter_group" "default" {
name = "rds-cluster-pg"
family = "aurora-postgresql12"
description = "Postgresql RDS default cluster parameter group"
parameter {
name = "rds.force_ssl"
value = "1"
apply_method = "immediate"
}
}
resource "aws_rds_cluster" "default" {
apply_immediately = true
cluster_identifier = "aurora-cluster-${var.project_id}-${var.env}"
engine = local.db_engine
engine_version = local.db_engine_version
availability_zones = local.vpc_azs
database_name = local.db_name
master_username = local.db_username
master_password = random_password.db.result
backup_retention_period = local.db_retention_period
preferred_backup_window = local.db_preferred_backup_window
db_subnet_group_name = aws_db_subnet_group.default.id
vpc_security_group_ids = [
aws_security_group.private_db_sg.id
]
db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.default.name
}
Creating the AWS Aurora database will take some time (~10-15 mins). So, just keep that in mind.
export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1
export TF_VAR_ip_address=<your-ip-address>
terraform init
terraform plan
terraform apply -auto-approve
If you do apply it, feel free to verify you see the database under “AWS RDS” on the AWS console.
⚠️ Note: Remember to run terraform destroy -auto-approve after you are done with the module. Unless you wish to keep the infrastructure for personal use.
If you see this error:
Error: RDS Cluster FinalSnapshotIdentifier is required when a final snapshot is required
you will need to update this filed to true in terraform.tfstate
:
{
"skip_final_snapshot": true,
}
then run terraform destroy -auto-approve
again.
If you’d like a reference of the finsihed result, it is available at aws-aurora-part-1.
This was a short module, we did a lot of basic setup which sets the foundation for the rest of the technical series.
To re-cap what we did, we configured the following:
That’s it! Stay tuned for the next module where we will setup our bastion host to administer (ie run migrations) the database instances. As part of that setup, and we‘ll also setup our infrastructure so only ssh connections from our IP can go through.
Then consider signing up to get notified when new content arrives!