Security harden Github Action deployments to AWS with OIDC

Published on: Tue Dec 28 2021

Jerry Chang

Last Updated on: Sat May 14 2022

What content was recently changed ?

Update section about adding more granular permissions for Github workflow(s)
Update section to dynamically generate the sha1 thumbprint_list for aws openid connect provider

Introduction

Traditionally, when you use Github Actions, you would need to provide an AWS ID and Secrets in the “secrets” stored in Github’s vault.

While this is a valid approach to gain access to AWS resources, it does come with drawbacks:

Long expiry on these credentials (risk of leaking)
No Authentication or Authorization (granular control)
Credential Rotation

What if I told you there is a better approach ?

As of October 27th, 2021, Github has shipped OpenID Connect support for Github Actions.

Using OpenID connect (OIDC), built on top of Auth 2.0, it provides a great solution to our problem. This is Github’s recommended approach for security hardening your github actions workflows.

Let’s explore how it works and implement it using a simple example.

How does it work ?

As I briefly touched on the topic, OpenID Connect is a standard built on top of Auth 2.0 to provide a way to authenticate clients. Whereas, Auth 2.0 is predominately used for authorization (even though it is still used by many for authentication).

The benefit of using OIDC is we wouldn’t have to manage the lifecycle of our AWS credentials. That will be obtained through the exchange process highlighted below.

These credentials are obtained each time Github Actions run, and expire shortly after it finishes its task. So, all the drawbacks with using hard coded AWS ID and Secrets goes away.

How does this process work ? Let’s take a closer look at the exchange process.

Step 1: Create OIDC trust between AWS and github

In our example, we will create an OIDC trust between our cloud provider (AWS) and our Github Actions workflow in terraform.

However, this feature is not limited to just terraform and AWS. This approach can be configured with other tools (etc Cloudformation, cli) using other cloud providers (ie Google Cloud and Azure).

The syntax will be different but the general flow is the same.

📝 Helpful references:

Github - Security Hardening your deployments

Step 2: Generate OIDC token (JWT token)

After creating the trust, each time our Github Actions workflow runs, it will generate an unique OIDC token, which is just a JWT token, that will then be verified with AWS.

Upon successful verification, AWS will provide a temporary AWS ID and Credential. These credentials expire after the github actions finishes.

Here is an example of the cracked JWT (OIDC token):

{
  "jti": "586d909b-640c-4398-ad0d-c054ed57df70",
  "sub": "repo:Jareechang/test-github-oidc:ref:refs/heads/master",
  "aud": "https://github.com/Jareechang",
  "ref": "refs/heads/master",
  "sha": "674dbe138de32077da63168b5ff7468591aefa2d",
  "repository": "Jareechang/test-github-oidc",
  "repository_owner": "Jareechang",
  "run_id": "1621067485",
  "run_number": "7",
  "run_attempt": "1",
  "actor": "Jareechang",
  "workflow": "AWS example workflow",
  "head_ref": "",
  "base_ref": "",
  "event_name": "push",
  "ref_type": "branch",
  "job_workflow_ref": "Jareechang/test-github-oidc/.github/workflows/main.yml@refs/heads/master",
  "iss": "https://token.actions.githubusercontent.com",
  "nbf": 1640410081,
  "exp": 1640410981,
  "iat": 1640410681
}

You can obtain this by running this (in your Github Actions):

curl -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL"

📝 Helpful references:

Github - Updating your actions for OIDC

Step 3: Verify OIDC token with AWS

Once we have generated the OIDC token in our github actions, we can verify it along with our assumed role with the cloud provider to exchange for temporary AWS credentials.

These will be short lived credentials which will expire after the github actions finishes.

Step 4: Access AWS Resources using Temporary tokens

Then, using those temporary AWS Credentials, we will now be able to access AWS resources just like if we had added AWS ID and Secret to our environment through the Github Secrets vault.

Now we have a good understanding of the OIDC process, let’s setup this up using Terraform, Github Actions and AWS.

Setting up Github OIDC

We are going to setup a simple example that allows our github actions to upload an index.html file to a AWS S3 bucket.

1. Scaffold out files

touch index.html main.tf variables.tf output.tf

2. Creating the S3 bucket

Feel free to change the bucket name if the terraform fails, just remember to change it in the later steps.

# variables.tf

variable "aws_region" {
  default = "us-east-1"
}

# main.tf

provider "aws" {
  version = "~> 2.0"
  region  = var.aws_region
}

resource "aws_s3_bucket" "this" {
  bucket = "myTestBucket-1"
  acl    = "private"
}

3. Add the OIDC provider

In this step, we configure the OIDC provider for our github, using the generated thumbprint.

The short explanation of the thumbprint is that this part of the OpenID connect, and is the public certificate of the host. For more information, see resources below.

# main.tf

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github_actions" {
  client_id_list  = var.client_id_list
  thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
  url             = "https://token.actions.githubusercontent.com"
}

📝 Helpful references:

4. Configure the IAM assume role policy

Here we are scaffolding out the policy for our trusted OIDC role.

These are all the required fields to establish the trust between AWS and our github actions.

⚠️ Important: Remember to change the "repo_name" variable to your repository name running the Github Actions.

# variables.tf
variable "client_id_list" {
  default = [
    "sts.amazonaws.com"
  ]
}

variable "repo_name" {
  default = "Jareechang/test-github-oidc"
}


# main.tf
data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "github_actions_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = [
        format(
          "arn:aws:iam::%s:root",
          data.aws_caller_identity.current.account_id
        )
      ]
    }
  }

  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type = "Federated"
      identifiers = [
        format(
          "arn:aws:iam::%s:oidc-provider/token.actions.githubusercontent.com",
          data.aws_caller_identity.current.account_id
        )
      ]
    }
    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:${var.repo_name}:*"]
    }
  }
}

5. Create role, add IAM permission and attach permission to the role

Here we are simply creating our new role with the OIDC assume role policy then creating IAM policy document for our github actions to be able to access S3.

Finally, we finish up by attaching that policy to the role.

# main.tf
resource "aws_iam_role" "github_actions" {
  name               = "github-actions"
  assume_role_policy = data.aws_iam_policy_document.github_actions_assume_role_policy.json
}

data "aws_iam_policy_document" "github_actions" {
  statement {
    actions = [
      "s3:GetObject",
      "s3:ListBucket",
      "s3:PutObject",
    ]
    effect = "Allow"
    resources = [
      aws_s3_bucket.this.arn,
      "${aws_s3_bucket.this.arn}/*"
    ]
  }
}

resource "aws_iam_role_policy" "github_actions" {
  name   = "github-actions"
  role   = aws_iam_role.github_actions.id
  policy = data.aws_iam_policy_document.github_actions.json
}

# output.tf
output "role_arn" {
  value = aws_iam_role.github_actions.arn
}

6. Apply the infrastructure

export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1

terraform init
terraform plan
terraform apply -auto-approve

After applying the infrastructure, you should see the assume role in terminal output:

Keep the role_arn handy as we will need that in our next step! let’s move onto setting up our github actions.

⚠️ Note: Remember to run terraform destroy -auto-approve after you are done with the module otherwise you will incur charges on your AWS Account. Unless you wish to keep the infrastructure for personal use.

Setting up Github Actions

1. Create a basic static html

Feel free to change the file to whatever you want it to be.

touch index.html
echo '<!DOCTYPE html><html><body><h1>testing github OIDC</h1></body></html>' > index.html

2. Add the Github Actions

Here the scaffold of our github actions which uploads a static html file to our S3 bucket using the role we are providing it.

We are going to use aws’s github actions to help us with the OIDC exchange process to get the aws credentials.

name: AWS upload to S3 using Github OIDC 
on:
  push
env:
  BUCKET_NAME : "<my-bucket>"
  AWS_REGION : "us-east-1"
  AWS_ASSUME_ROLE: "<your-assume-role>"
# permission can be added at job level or workflow level
permissions:
      id-token: write
      contents: read    # This is required for actions/checkout@v1
jobs:
  S3PackageUpload:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v1
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@master
        with:
          role-to-assume: ${{ env.AWS_ASSUME_ROLE }} 
          aws-region: ${{ env.AWS_REGION }}
      # Upload a file to AWS s3
      - name:  Copy index.html to s3
        run: |
          aws s3 cp ./index.html s3://${{ env.BUCKET_NAME }}/

Note:

This gives us the ability for our github actions to fetch OIDC token.

permissions:
      id-token: write

Now that we have our infrastructure and github actions all ready to go. Let’s test it out.

push your changes to your repository, and verify that the github actions succeeded and verify that the new file is added to your S3 bucket.

📝 Helpful references:

Granular permissions

So, we have our setup ready, and it works. However, there is one problem. The permission we set so far allows this to work for all cases (ie all branches, forks, pull requests).

If someone forks our code or create a new branch with a pull request, they would still be able to run the workflow with the these permissions that is intended to be used when we push changes to the master or main branch.

There are a few elements at play. Let’s look the different parts to see why this is the case and see how we can fix this.

OIDC Subject

The OIDC token contains a sub (Subject) which contains details about the Github Actions trigger. These can be filtering by branch, event, enviroments and tags.

{
  "sub": "repo:Jareechang/test-github-oidc:ref:refs/heads/master",
}

📝 Helpful references:

Condition operator

We are currently using StringsLike which is a case-sensitive match with a wildcard * instead of a StringsEquals (exact matching, case sensitive) to match our token sub - token.actions.githubusercontent.com:sub.

We should be using StringEquals because we want this to be an exact match.

So, there are a few things we need to change:

Create Separate the IAM roles (one for pull request, and deployment)
Update our Subject filter and the condition operators
Separate github actions workflows (pull request and deployment)

What will the roles look like now ? Let’s take a look.

Separating the roles

Read only role (pull request)

This will be our role which have limited access. We will use this role for our pull requests.

We can perhaps allow the build to check something in our AWS Resources.

Write only role (deployments)

This will be our role where we have enough access to make a deploy, for example, sync up the assets to the s3 buckets which usually requires read, update and deletion permissions .

📝 Helpful references:

AWS - policies elements condition operators

Adding the granular permissions

1. Updating the infrastructure

Let’s update our infrastructure with our separate roles and the filters for our OIDC subject.

here are the new subject filter:

Deployment - "repo:${var.repo_name}:ref:refs/heads/master"
Pull request - "repo:${var.repo_name}:pull_request"

# main.tf
provider "aws" {
  version = "~> 2.0"
  region  = var.aws_region
}

resource "aws_s3_bucket" "this" {
  bucket = "examp1e-buck-1222azeii"
  acl    = "private"
}

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github_actions" {
  client_id_list  = var.client_id_list
  thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
  url             = "https://token.actions.githubusercontent.com"
}

data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "github_actions_assume_role_policy_write_only" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = [
        format(
          "arn:aws:iam::%s:root",
          data.aws_caller_identity.current.account_id
        )
      ]
    }
  }

  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type = "Federated"
      identifiers = [
        format(
          "arn:aws:iam::%s:oidc-provider/token.actions.githubusercontent.com",
          data.aws_caller_identity.current.account_id
        )
      ]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:${var.repo_name}:ref:refs/heads/master"]
    }
  }
}

data "aws_iam_policy_document" "github_actions_assume_role_policy_read_only" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = [
        format(
          "arn:aws:iam::%s:root",
          data.aws_caller_identity.current.account_id
        )
      ]
    }
  }

  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type = "Federated"
      identifiers = [
        format(
          "arn:aws:iam::%s:oidc-provider/token.actions.githubusercontent.com",
          data.aws_caller_identity.current.account_id
        )
      ]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:${var.repo_name}:pull_request"]
    }
  }
}

resource "aws_iam_role" "github_actions_write_only" {
  name               = "github-actions-write-only"
  assume_role_policy = data.aws_iam_policy_document.github_actions_assume_role_policy_write_only.json
}

resource "aws_iam_role" "github_actions_read_only" {
  name               = "github-actions-read-only"
  assume_role_policy = data.aws_iam_policy_document.github_actions_assume_role_policy_read_only.json
}

data "aws_iam_policy_document" "github_actions_read_only" {
  statement {
    actions = [
      "s3:ListBucket",
    ]
    effect = "Allow"
    resources = [
      aws_s3_bucket.this.arn,
      "${aws_s3_bucket.this.arn}/*"
    ]
  }
}

data "aws_iam_policy_document" "github_actions_write_only" {
  statement {
    actions = [
      "s3:PutObject",
    ]
    effect = "Allow"
    resources = [
      aws_s3_bucket.this.arn,
      "${aws_s3_bucket.this.arn}/*"
    ]
  }
}

resource "aws_iam_role_policy" "github_actions_read_only" {
  name   = "github-actions-read-only"
  role   = aws_iam_role.github_actions_read_only.id
  policy = data.aws_iam_policy_document.github_actions_read_only.json
}

resource "aws_iam_role_policy" "github_actions_write_only" {
  name   = "github-actions-write-only"
  role   = aws_iam_role.github_actions_write_only.id
  policy = data.aws_iam_policy_document.github_actions_write_only.json
}

# output.tf
output "role_arn_read_only" {
  value = aws_iam_role.github_actions_read_only.arn
}

output "role_arn_write_only" {
  value = aws_iam_role.github_actions_write_only.arn
}

2. Apply the infrastructure

export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1

terraform init
terraform plan
terraform apply -auto-approve

3. Add deployment workflow

Remember to update using your Deployment (write only) role.

name: AWS example workflow (deployment)

on:
  push:
    branches:
      - master
      - main

env:
  BUCKET_NAME : "<your-bucket>"
  AWS_REGION : "us-east-1"
  AWS_ASSUME_ROLE: "<your-write-only-assume-role>"
permissions:
      id-token: write
      contents: read    # This is required for actions/checkout@v1
jobs:
  S3PackageUpload:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v1
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@master
        with:
          role-to-assume: ${{ env.AWS_ASSUME_ROLE }} 
          aws-region: ${{ env.AWS_REGION }}
      # Upload a file to AWS s3
      - name:  Copy index.html to s3
        run: |
          aws s3 cp ./index.html s3://${{ env.BUCKET_NAME }}/

4. Add pull request workflow

Remember to update using your Pull request (read only) role.

In addition, we are only listing the the files in our bucket.

name: AWS example workflow (pull request)

on: [pull_request]

env:
  BUCKET_NAME : "<your-bucket>"
  AWS_REGION : "us-east-1"
  AWS_ASSUME_ROLE: "<your-read-only-assume-role>"
permissions:
      id-token: write
      contents: read    # This is required for actions/checkout@v1
jobs:
  S3ListFiles:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v1
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@master
        with:
          role-to-assume: ${{ env.AWS_ASSUME_ROLE }} 
          aws-region: ${{ env.AWS_REGION }}
      # Upload a file to AWS s3
      - name:  Copy index.html to s3
        run: |
          aws s3 ls s3://${{ env.BUCKET_NAME }}

Now that is ready. Test out the workflows:

Make change, push to master / main
Make change, make a pull request
Make change, update to use aws cp in pull request (optional)

Wrapping up

So, that was a lot of to take in. Let’s do a quick review on what we just did.

We essentially took our initial setup where we allowed all the workflows to access our resources to one where we have more granular control on which workflow can access what.

✅ I think its important to see this transition in order to see how important this granular control is, and what that means for the security of our Github Action workflows.

Just be mindful of these controls when setting this up for your own project.

💡 Questions to ask

Think about the different workflows you need (pull_request, certains branches)

Think about which resources you need to access in these different workflows (ie pull_request -> read only, deploy -> updates)

Always go with least permission as possible first (double check and evaluate your roles when adding mutations - deletions, updates)

This granularity is the beauty of OpenID Connect with Github and AWS. However, when mis-configured it can also become a security flaw.

As a reference, here is are the repositories:

Iteration 1: Allow all - github-oidc-example
Iteration 2: Granular permissions- github-oidc-example-granular-permissions ⭐️

Conclusion

That’s it. You won’t have to worry about AWS credentials when working with Github Actions again!

So, just to wrap up, OpenID Connect allows us to eliminate the management of AWS Credentials from our process. We essentially delegated that authentication process to AWS and Github!

We still receive crendentials but they are just short lived or temporary credentials from the AWS STS which expire after Github Actions finishes the workflow.

In terms of setup, the only part we really needed was to create the OIDC Trust bewteen AWS and our Github Actions and once that is done, it is all ready to go!

Additionally, we are able to have more granularity with our permissions within the Github workflows. This makes our pipelines even more secure as we no longer are storing the AWS credentials, just be mindful of the granularity when defining the OIDC trust. Always start with least permissions!

As a bonus, these sessions are also tracked in AWS Cloudtrail, so if you want to setup tracking and automation, that can be done too.

Even though we went through a simple example with S3, this setup can be extended to be used with other services like AWS ECS, EKS or Lambda.

I hope you found this useful, and learned a thing or two!

Enjoy the content ?

Then consider signing up to get notified when new content arrives!

Jerry Chang

Go Back

Security harden Github Action deployments to AWS with OIDC

What content was recently changed ?

Content

Introduction

How does it work ?

Step 1: Create OIDC trust between AWS and github

📝 Helpful references:

Step 2: Generate OIDC token (JWT token)

📝 Helpful references:

Step 3: Verify OIDC token with AWS

Step 4: Access AWS Resources using Temporary tokens

Setting up Github OIDC

1. Scaffold out files

2. Creating the S3 bucket

3. Add the OIDC provider

📝 Helpful references:

4. Configure the IAM assume role policy

5. Create role, add IAM permission and attach permission to the role

6. Apply the infrastructure

Setting up Github Actions

1. Create a basic static html

2. Add the Github Actions

📝 Helpful references:

Granular permissions

OIDC Subject

📝 Helpful references:

Condition operator

Separating the roles

Read only role (pull request)

Write only role (deployments)

📝 Helpful references:

Adding the granular permissions

1. Updating the infrastructure

2. Apply the infrastructure

3. Add deployment workflow

4. Add pull request workflow

Wrapping up

Conclusion

Enjoy the content ?