AWS Aurora Technical Series Part I - AWS Aurora & VPC

Published on: Thu Jan 27 2022

Jerry Chang

Content

Introduction
Aurora Architecture
AWS VPC
AWS Aurora Resources
Conclusion

Introduction

In this module, we will be setting up the foundation of our AWS infrastructure, which will include:

AWS VPC (should already be available in the starter repo.)
AWS Aurora database components

The main focus here to is setup the foundational infrastructure for us to build on. Then as we progress through the modules, we will start layering on top of the existing foundation.

Here is what that will look like:

Aurora Architecture

As a refresher, the AWS Aurora architecture requires a minimum of 3 availability zones when initializing. So, we will need to ensure that our VPC supports that.

This is because AWS Aurora replicates 2 copies of our the data we write to the database in 3 availibility zones (AZs).

AWS VPC

In your repository, please ensure you have the following:

locals {
  # Omitted for brevity
  vpc_azs = ["us-east-1a", "us-east-1b", "us-east-1d"]
}

module "networking" {
  source = "github.com/Jareechang/tf-modules//networking?ref=v1.0.20"
  env = var.env
  project_id = var.project_id
  subnet_public_cidrblock = [
    "10.0.1.0/24",
    "10.0.2.0/24",
    "10.0.3.0/24"
  ]
  subnet_private_cidrblock = [
    "10.0.10.0/24",
    "10.0.21.0/24",
    "10.0.31.0/24"
  ]
  azs = local.vpc_azs
}

Or at very least, a VPC which supports 3 availability zones, public & private subnets.

Feel free to use this starter repository - aws-aurora-starter.

AWS Aurora Resources

On to the AWS Aurora, there will be a few compoennts that we will be setting up.

Specifically:

AWS Aurora (write node)
AWS Aurora (2 xread node)
DB subnet groups and Security groups
RDS Cluster endpoint (for read replicas)
Password storage to AWS SSM Parameter store
Encryption with AWS KMS

For the AWS Aurora, we are creating a RDS cluster endpoint so we don’t have connect directly to the replicas themselves. The endpoint will load balance the connections between the replicas when using that endpoint.

1. Setting up DB subnet groups

The Database subnet groups are a collection of subnets that you can designate for running your clusters in the VPC.

resource "aws_db_subnet_group" "default" {
  name       = "main"
  subnet_ids = module.networking.private_subnets[*].id

  tags = {
    Name = "DB Subnet group"
  }
}

Here we are using terraform’s splat syntax to say we want to designate all the private subnets within our VPC to run our AWS Aurora clusters in.

📝 Helpful reference:

2. Setting up security groups

The security group, on the other hand, restricts the inbound and outbound traffic of your resource.

resource "aws_security_group" "private_db_sg" {
  name        = "aurora-custom-sg"
  description = "Custom default SG"
  vpc_id      = module.networking.vpc_id

  ingress {
    description = "TLS from VPC"
    from_port   = 5432
    to_port     = 5432
    protocol    = "tcp"
    security_groups = [
      # For the AWS ECS - we will deal with this later in the series
      aws_security_group.ecs_sg.id
    ]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "PostgreSQL-db-access-${var.project_id}-${var.env}"
  }
}

The gist of this is we are allow inbound from port 5432 (postgreSQL’s default port) and outbound for any traffic.

Here we are being extra restrictive by limiting access to particular the Security groups IDs (Bastion host and AWS ECS security groups).

In reality, you can probably get away with specifying to allow access if the CIDR block is coming from within our VPC, like so:

resource "aws_security_group" "private_db_sg" {
  # omitted for brevity

  ingress {
    cidr_blocks = module.networking.private_subnets[*].cidr_block
  }
}

A side benefit of this is we don’t have to update our security group when we expand our CIDR ranges if we run out of ip addresses within our VPC.

📝 Helpful reference:

terraform - aws_security_group

3. Creating the database password

As mentioned, we are using AWS SSM Parameter store which is based on the “name” which is path based (ie you can store multiple different secrets under database/* and retrieve them all).

#### Secrets
resource "random_password" "db" {
  length           = 16
  special          = false
}

resource "aws_kms_key" "default" {
  description             = "Default encryption key (symmetric)"
  deletion_window_in_days = 7
}

resource "aws_ssm_parameter" "db_password" {
  name        = "/web/${var.project_id}/database/secret"
  description = "Datbase password"
  type        = "SecureString"
  key_id      = aws_kms_key.default.key_id
  value       = random_password.db.result
}

💡 Tip: If you like a fully end to end managed secrets manager, feel free to check out AWS Secrets Managers. AWS Parameter store work fine but it does not offer automatic secret rotation.

📝 Helpful reference:

4. Creating the AWS Aurora infrastructure

Here we are setting up our aurora clusters and instances along the cluster endpoint which will contain our read replicas.

We are doing some fancy filtering (using terraform for) to get the read only instance from our cluster instances being created.

# output.tf

output "aurora_cluster_endpoint" {
  value = aws_rds_cluster_endpoint.static.endpoint
}

# main.tf
locals {
  # Omitted for brevity, add to your existing locals

  # Database
  db_username = "read_user"
  db_name = "blog"
  db_port = 5432
  db_engine = "aurora-postgresql"
  db_engine_version = "12.7"
  db_retention_period = 5
  db_preferred_backup_window = "19:00-22:00"
}

resource "aws_rds_cluster" "default" {
  apply_immediately       = true
  cluster_identifier      = "aurora-cluster-${var.project_id}-${var.env}"
  engine                  = local.db_engine 
  engine_version          = local.db_engine_version 
  availability_zones      = local.vpc_azs 
  database_name           = local.db_name
  master_username         = local.db_username
  master_password         = random_password.db.result
  backup_retention_period = local.db_retention_period
  preferred_backup_window = local.db_preferred_backup_window
  db_subnet_group_name    = aws_db_subnet_group.default.id
  vpc_security_group_ids  = [
    aws_security_group.private_db_sg.id
  ]
}

resource "aws_rds_cluster_instance" "cluster_instances" {
  apply_immediately       = true
  count                   = 2
  identifier              = "aurora-cluster-${var.project_id}-${count.index}"
  cluster_identifier      = aws_rds_cluster.default.id
  instance_class          = "db.t3.medium"
  engine                  = aws_rds_cluster.default.engine
  publicly_accessible     = false
  db_subnet_group_name    = aws_db_subnet_group.default.id
}

resource "aws_rds_cluster_endpoint" "static" {
  cluster_identifier          = aws_rds_cluster.default.id
  cluster_endpoint_identifier = "static"
  custom_endpoint_type        = "READER"
  static_members              = [for i, instance in aws_rds_cluster_instance.cluster_instances : instance.id if instance.writer == false]
}

5. Enable encryption at rest

Let’s setup encrpytion for our storage volumes, snapshots and backups using AWS KMS.

The data will be encrypted using the provided AWS KMS key.

resource "aws_kms_key" "db_key" {
  description             = "KMS for database"
  deletion_window_in_days = 7
}

resource "aws_rds_cluster" "default" {
  apply_immediately       = true
  cluster_identifier      = "aurora-cluster-${var.project_id}-${var.env}"
  engine                  = local.db_engine 
  engine_version          = local.db_engine_version 
  availability_zones      = local.vpc_azs 
  database_name           = local.db_name
  master_username         = local.db_username
  master_password         = random_password.db.result
  backup_retention_period = local.db_retention_period
  preferred_backup_window = local.db_preferred_backup_window
  db_subnet_group_name    = aws_db_subnet_group.default.id
  storage_encrypted       = true
  kms_key_id              = aws_kms_key.db_key.arn
  vpc_security_group_ids  = [
    aws_security_group.private_db_sg.id
  ]
}

6. Configure to force SSL connection

This configuration forces SSL any connection to always use SSL otherwise it would fail.

resource "aws_rds_cluster_parameter_group" "default" {
  name        = "rds-cluster-pg"
  family      = "aurora-postgresql12"
  description = "Postgresql RDS default cluster parameter group"

  parameter {
    name         = "rds.force_ssl"
    value        = "1"
    apply_method = "immediate"
  }
}

resource "aws_rds_cluster" "default" {
  apply_immediately       = true
  cluster_identifier      = "aurora-cluster-${var.project_id}-${var.env}"
  engine                  = local.db_engine
  engine_version          = local.db_engine_version
  availability_zones      = local.vpc_azs
  database_name           = local.db_name
  master_username         = local.db_username
  master_password         = random_password.db.result
  backup_retention_period = local.db_retention_period
  preferred_backup_window = local.db_preferred_backup_window
  db_subnet_group_name    = aws_db_subnet_group.default.id
  vpc_security_group_ids  = [
    aws_security_group.private_db_sg.id
  ]
  db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.default.name
}

📝 Helpful reference:

7. Apply the infrastructure

Creating the AWS Aurora database will take some time (~10-15 mins). So, just keep that in mind.

export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1
export TF_VAR_ip_address=<your-ip-address>

terraform init
terraform plan
terraform apply -auto-approve

If you do apply it, feel free to verify you see the database under “AWS RDS” on the AWS console.

⚠️ Note: Remember to run terraform destroy -auto-approve after you are done with the module. Unless you wish to keep the infrastructure for personal use.

If you see this error:

Error: RDS Cluster FinalSnapshotIdentifier is required when a final snapshot is required

you will need to update this filed to true in terraform.tfstate:

{
  "skip_final_snapshot": true,
}

then run terraform destroy -auto-approve again.

If you’d like a reference of the finsihed result, it is available at aws-aurora-part-1.

Conclusion

This was a short module, we did a lot of basic setup which sets the foundation for the rest of the technical series.

To re-cap what we did, we configured the following:

AWS VPC (Remember AWS Aurora always requires 3 AZs)
AWS Aurora
- Configured cluster (writer), 2 x instances (read replicas)
- Configured to force TLS/SSL (Database parameter group)
- Configured encryption (AWS KMS)
- Configured RDS endpoint
- Secure storage of password in AWS SSM Parameter store

That’s it! Stay tuned for the next module where we will setup our bastion host to administer (ie run migrations) the database instances. As part of that setup, and we‘ll also setup our infrastructure so only ssh connections from our IP can go through.

Enjoy the content ?

Then consider signing up to get notified when new content arrives!

Jerry Chang

Go Back

AWS Aurora Technical Series Part I - AWS Aurora & VPC

Content

Introduction

Aurora Architecture

AWS VPC

AWS Aurora Resources

1. Setting up DB subnet groups

📝 Helpful reference:

2. Setting up security groups

📝 Helpful reference:

3. Creating the database password

📝 Helpful reference:

4. Creating the AWS Aurora infrastructure

5. Enable encryption at rest

6. Configure to force SSL connection

📝 Helpful reference:

7. Apply the infrastructure

Conclusion

Enjoy the content ?