Published on: Mon Jul 25 2022
In this module, we will use this starter template - aws-webhook-series-part-3 as a starting point.
By the end of this module, you should:
✅ Be able to integrate AWS API gateway and SQS and AWS Lambda
✅ Know how to structure and how to write tests for lambda function uses AWS services
✅ Understand the code required to make our Ingest function work (verifySignature + SQS.sendMessage)
✅ Understand the constraints of SQS message size (ie size limit) and know how to guard against it
✅ Understand how to setup webhooks with Github
This module will be focused on building on top of our existing infrastructure and start layering in the logic for our web hook.
This will be a step-by-step process to not only write out the code but also the tests to go along with individual modules.
Then finally, to finish it off, we will integrate it with Github web hooks.
The goal at the end of this is you should have a production ready webhook microservice that works!
In the Ingestion function, the main focus is on ingestion and ensuring that the event data received is coming from a verified source.
Then, once all that is done, we will forward the relevant information to the Process-Queue
function.
The Ingestion flow consist of the following Logic:
[Optional] Validation - This is optional but recommended if you have a specific event type you want to ingest
Verify Signature - We will need to verify this payload actually came from the right source by checking the signed signature
Add to Queue - The data we want to forward gets added to the queue (AWS SQS Queue)
Response - Respond with success or error
Let’s take a look at two possible results from the webhook which is we either have success or failure.
The approach we will take is to be explicit about the errors we are getting within our microservice.
That way, when issues do occur in production, it is easier to not only debug but to isolate the module causing the problem and make it easier to identify the root cause.
The only difference here is we are forwarding our error in our function to our common error handler to:
Now we have a good idea of high level design, let’s code it out!
Let’s start off by creating a layout for what we need in our entry lambda function before we actually start the coding.
Conveniently, breaking down the components allow us to build each step in isolation, and verify it with tests.
Then, we can integrate the modules together just like lego blocks.
Within the src/index.ts
, let’s scaffold out our code:
import {
APIGatewayProxyEvent,
APIGatewayProxyResult,
} from 'aws-lambda';
// Default starter
export const handler = async(
event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> => {
console.log('Event : ', JSON.stringify({
event,
}, null, 4));
// 1. Verify Signature
// 2. Add to Queue
// 3. Error handling (final touch)
// 4. Response
return {
statusCode: 200,
body: JSON.stringify({
message: 'success'
}),
}
}
I’ve added error handling as a final touch as in the series of steps above as there could potentially be an error.
We’ll start with the verify signature function.
This will essentially be a checksum for our payload to ensure it is coming from a verified source.
Using SHA-256 as the cryptograpic hashing algorithm, we can re-produce the signature using the shared secret and the payload sent.
This allows us to match that against the signature given in the request header to verify it.
This will be our custom error thrown the signature verification fails so we can identify it later.
💡 Understanding the `CommonError`The
CommonError
contains extra methods that allows us propogate contextual information for easier debugging (ie adding it to the logging).Example:
throw new MyError('An Error Occurred') .setOperation('utils/my-util') .setContext({ userId: id, query, keySignature, });
Create the file:
touch ./errors/verify-signature-error.ts
Add the error:
// errors/verify-signature-error.ts
import CommonError from './common-error';
class VerifySignatureError extends CommonError {
constructor(message: string) {
super(message);
this.name = this.constructor.name;
}
}
export default VerifySignatureError;
Export it in index file:
// errors/index.ts
export { default as CommonError } from './common-error';
export { default as VerifySignatureError } from './verify-signature-error';
verifySignature
utilAs mentioned above, we are essentially re-generating the signature based on the inputs and checking to see if they match.
If it does not match, we will throw an error which will be handle by our global error handler (we will address later).
Create the file:
touch ./utils/verify-signature.ts
Add the code for verifying the signature:
// utils/verify-signature.ts
import { APIGatewayProxyEvent } from 'aws-lambda';
import crypto from 'crypto';
import config from '@app/config';
import { VerifySignatureError } from '@app/errors';
export default function verifySignature(
event: APIGatewayProxyEvent
): void {
const headerSignature: string = event?.headers[config.webhook.signature.header] ?? '';
const hash = crypto.createHmac(
config.webhook.signature.algo,
config.webhook.signature.secret
)
.update(event?.body ?? '')
.digest('hex');
const generatedSignature = `${config.webhook.signature.algo}=${hash}`;
if (generatedSignature !== headerSignature) {
const error: Error = new VerifySignatureError('Signature Mis-match')
.setOperation('utils/verifySignature')
.setContext({
generatedSignature,
headerSignature,
'config.webhook.signature.header': config.webhook.signature.header,
'config.webhook.signature.algo': config.webhook.signature.algo
});
throw error;
}
}
Export the utils via the index file:
// utils/index.ts
export { default as verifySignature } from './verify-signature';
There are three outcomes from this function:
We will add tests to test these outcomes.
Create the test file:
touch ./utils/__tests__/verify-signature.test.ts
Add the test:
// utils/__tests__/verify-signature.test.ts
import { mockEvent } from '@app/__mocks__';
import verifySignature from '../verify-signature';
jest.mock('@app/config', () => ({
__esModule: true,
default: {
webhook: {
signature: {
secret: 'test123',
algo: 'sha256',
header: 'x-hub-signature-256'
}
}
}
}));
describe('verifySignature', () => {
it('should proceed without errors if signature matches', () => {
// @ts-ignore
expect(() => verifySignature(mockEvent)).not.toThrow();
});
it('should throw errors if signature does not match', () => {
try {
// @ts-ignore
verifySignature({
...mockEvent,
body: '',
})
} catch (error: any) {
expect(error.name).toBe('VerifySignatureError');
expect(error.message).toBe('Signature Mis-match');
expect(error.operation).toBe('utils/verifySignature');
expect(error.context).toEqual({
generatedSignature: "sha256=bce68cb87da59c708eaff17571c501dee1c4aeed50d9552f0b8644286317bcc9",
headerSignature: "sha256=c2af8d2bc59689c28094b194bc3a38ed865c2ea4de38ca90dd6d71c61d2e12ef",
'config.webhook.signature.header': 'x-hub-signature-256',
'config.webhook.signature.algo': 'sha256' ,
});
}
});
});
Now that the verifySignature
is good to go, let’s integrate that into the entry lambda function.
import {
APIGatewayProxyEvent,
APIGatewayProxyResult,
} from 'aws-lambda';
import { verifySignature } from '@app/utils';
// Default starter
export const handler = async(
event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> => {
console.log('Event : ', JSON.stringify({
event,
}, null, 4));
// 1. Verify Signature
verifySignature(event);
// 2. Add to Queue
// 3. Error handling (final touch)
// 4. Response
return {
statusCode: 200,
body: JSON.stringify({
message: 'success'
}),
}
}
getSqsMessage
utilAWS SQS messages need to be in a specific format, and contain certain required parameters.
Let’s create an util to handle the other properties and so that we can just pass in the message from the request.
Create the file:
touch ./utils/get-sqs-message.ts
Add the logic:
import AWS from 'aws-sdk';
import {
APIGatewayProxyEvent,
} from 'aws-lambda';
import config from '@app/config';
export default function getSqsMessage(
event: APIGatewayProxyEvent,
): AWS.SQS.Types.SendMessageRequest {
return {
MessageAttributes: {
requestId: {
DataType: 'String',
StringValue: event?.requestContext?.requestId ?? '',
}
},
MessageBody: event?.body ?? '',
QueueUrl: config.queue.sqs.url ?? '',
};
}
Export in the index file:
// utils/index.ts
export { default as verifySignature } from './verify-signature';
export { default as getSqsMessage } from './get-sqs-message';
getSqsMessage
utilGiven that getSqsMessage
will return the message payload for SQS, we want to ensure that does generates it correcty, and it contains the right details.
Create the test file:
touch utils/__tests__/get-sqs-message.test.ts
Add the test:
import getSqsMessage from '../get-sqs-message';
import { mockEvent } from '@app/__mocks__';
jest.mock('@app/config', () => ({
__esModule: true,
default: {
webhook: {
signature: {
secret: 'test123',
algo: 'sha256',
header: 'x-hub-signature-256'
}
},
queue: {
sqs: {
url: 'queueUrl'
}
},
}
}));
describe('getSqsMessage', () => {
it('should return with the correct "MessageAttributes.requestId"', () => {
const mockRequestId = 'mockRequestId';
const message = getSqsMessage({
...mockEvent,
// @ts-ignore
requestContext: {
requestId: mockRequestId
}
});
expect(message?.MessageAttributes?.requestId).toEqual({
DataType: 'String',
StringValue: mockRequestId
});
});
it('should return with the correct "MessageBody"', () => {
const mockMessage = 'test-message';
// @ts-ignore
const message = getSqsMessage({
...mockEvent,
body: mockMessage,
});
expect(message?.MessageBody).toEqual(mockMessage);
});
it('should return with the correct "QueueUrl"', () => {
// @ts-ignore
const message = getSqsMessage(mockEvent);
expect(message?.QueueUrl).toEqual('queueUrl');
});
});
This custom error is to identify when the SQS service has failed.
Create the file:
touch ./errors/aws-sqs-service-error.ts
Add the error:
// errors/aws-sqs-service-error.ts
import CommonError from './common-error';
class AwsSqsServiceError extends CommonError {
constructor(message: string) {
super(message);
this.name = this.constructor.name;
}
}
export default AwsSqsServiceError;
Export it in index file:
// errors/index.ts
export { default as CommonError } from './common-error';
export { default as VerifySignatureError } from './verify-signature-error';
export { default as AwsSqsServiceError } from './aws-sqs-service-error';
Create the service file:
touch services/sqs-service.ts
Add the sendMessage
method:
// services/sqs-service.ts
import AWS from 'aws-sdk';
import { APIGatewayProxyEvent } from 'aws-lambda';
import { getSqsMessage } from '@app/utils';
import config from '@app/config';
import { AwsSqsServiceError } from '@app/errors';
export const sqs = new AWS.SQS({
apiVersion: '2012-11-05',
region: config.queue.sqs.defaultRegion,
});
/*
* adds an event message to the queue
*
* **/
export async function sendMessage(
event: APIGatewayProxyEvent
): Promise<string> {
const message: AWS.SQS.Types.SendMessageRequest = getSqsMessage(event);
try {
const result: AWS.SQS.Types.SendMessageResult = (
await sqs.sendMessage(message).promise());
return result?.MessageId ?? '';
} catch (error) {
throw new AwsSqsServiceError('failed to sendMessage')
.setOperation('services/sqs-service:sendMessage')
.setContext({
error,
message,
});
}
}
For the SQS service, there is two outcomes:
MessageId
from AWS SQSAwsSqsServiceError
Let’s write tests to ensure the expected behaviour in these two outcomes.
Create the test file:
mkdir ./services/__tests__ && \
touch ./services/__tests__/sqs-service.test.ts
Add the test:
// services/__tests__/sqs-service.test.ts
import { sqs, sendMessage } from '@app/services/sqs-service';
import { mockEvent } from '@app/__mocks__';
jest.mock('aws-sdk', () => {
const SqsMethods = {
sendMessage: jest.fn().mockReturnThis(),
promise: jest.fn(),
}
const mockAwsSdk = {
SQS: jest.fn(() => SqsMethods),
};
return mockAwsSdk;
});
describe('sqs-service', () => {
beforeEach(() => {
jest.resetAllMocks();
});
describe('sendMessage', () => {
it('should return the SQS MessageId upon success', () => {
const messageBody = 'test message';
const messageId = 'test';
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn().mockResolvedValue({
MessageId: messageId
}),
})
// @ts-ignore
expect(sendMessage({
...mockEvent,
body: messageBody
})).resolves.toBe(messageId)
});
it('should make a request to SQS with the correct message body', async() => {
const messageBody = 'test message';
const messageId = 'test';
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn().mockResolvedValue({
MessageId: messageId
}),
})
// @ts-ignore
await sendMessage({
...mockEvent,
body: messageBody
});
expect(sqs.sendMessage).toBeCalledWith({
MessageAttributes: {
requestId: {
DataType: 'String',
StringValue: mockEvent?.requestContext?.requestId
}
},
MessageBody: messageBody,
QueueUrl: ''
});
});
it('should default to empty string if no message id is found', () => {
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn().mockResolvedValue({}),
})
// @ts-ignore
expect(sendMessage({
...mockEvent,
body: 'test'
})).resolves.toBe('')
});
it('should throw AwsSqsServiceError if it surpasses the size limit', async() => {
const mockError = new Error('mockError');
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: () => { throw mockError },
})
try {
// @ts-ignore
await sendMessage({
...mockEvent,
body: 'test'
})
} catch (error: any) {
expect(error.name).toBe('AwsSqsServiceError');
expect(error.message).toBe('failed to sendMessage');
expect(error.operation).toBe('services/sqs-service:sendMessage');
expect(error.context).toEqual({
error: mockError,
message: {
MessageAttributes: {
requestId: {
DataType: 'String',
StringValue: "Vb4i_hboIAMEVMA=",
}
},
MessageBody: 'test',
QueueUrl: '',
},
});
}
});
});
});
// src/index.ts
import {
APIGatewayProxyEvent,
APIGatewayProxyResult,
} from 'aws-lambda';
import { verifySignature } from '@app/utils';
import { sendMessage } from '@app/services/sqs-service';
// Default starter
export const handler = async(
event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> => {
console.log('Event : ', JSON.stringify({
event,
}, null, 4));
// 1. Verify Signature
verifySignature(event);
// 2. Add to Queue
const messageId: string = await sendMessage(event);
// 3. Error handling (final touch)
// 4. Response
return {
statusCode: 200,
body: JSON.stringify({
messageId,
message: 'success'
}),
}
}
We made a lot of changes and integrated few modules into the lambda entry, let’s update it so passes the tests.
// src/index.test.ts
import { handler } from './index';
import { mockEvent } from '@app/__mocks__';
import { sqs } from '@app/services/sqs-service';
jest.mock('aws-sdk', () => {
const SqsMethods = {
sendMessage: jest.fn().mockReturnThis(),
promise: jest.fn(),
}
const mockAwsSdk = {
SQS: jest.fn(() => SqsMethods),
};
return mockAwsSdk;
});
jest.mock('@app/config', () => ({
__esModule: true,
default: {
webhook: {
signature: {
secret: 'test123',
algo: 'sha256',
header: 'x-hub-signature-256'
}
},
queue: {
sqs: {
url: 'queueUrl'
}
},
}
}))
describe('lambda.handler', () => {
afterEach(() => {
jest.resetAllMocks();
jest.clearAllMocks();
});
it('should return with 200 with the default message', async() => {
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn(),
})
// @ts-ignore
await expect(handler(mockEvent))
.resolves
.toEqual({
statusCode: 200,
body: expect.stringMatching(
'success'
)
});
});
});
When using AWS SQS, the messages sent has limits to the size.
The AWS SQS message limits:
Even though we can pass the message over to AWS SQS then have it fail the request, we should still add logic to handle it.
That way, if we do decided to have a fallback storage (ie S3) to store the messages then it should be a simple add. Using S3 is actually the recommended fallback by AWS.
Let’s add the message size check!
// src/config.ts
export default {
// System level
log: {
level: process.env.LOG_LEVEL || 'error',
enabled: Boolean(process.env.LOG_ENABLED) || false,
redact: process.env.LOG_REDACT?.split(',') ?? [],
},
// Application level
// configuration for AWS SQS
queue: {
sqs: {
url: process?.env?.QueueUrl ?? '',
defaultRegion: process.env.DefaultRegion ?? 'us-east-1',
sizeLimit: {
// 262,144 bytes or 256kb
max: 256 * 1024,
min: 1,
}
}
},
// configuration for the custom webhook
webhook: {
signature: {
secret: process.env.WebhookSignatureSecret ?? '',
algo: process.env.WebhookSignatureAlgo ?? 'sha256',
header: process.env.WebhookSignatureHeader ?? 'x-hub-signature-256',
}
}
};
Add the new files:
touch ./utils/is-within-sqs-message-size-limit.ts && \
touch ./utils/get-sqs-message-size.ts
Add the logic for the getSqsMessageSize utils:
// utils/get-sqs-message-size.ts
/*
*
* Return message size in bytes
*
* **/
export default function getSqsMessageSize(
message: AWS.SQS.Types.SendMessageRequest,
): number {
const payload : string = typeof message === 'string'
? message
: JSON.stringify(message);
const buffer = Buffer.from(payload, 'utf8');
return buffer.byteLength ?? 0;
}
Add the logic for the isWithinSqsMessageSizeLimit utils:
// utils/is-within-sqs-message-size-limit.ts
import AWS from 'aws-sdk';
import config from '@app/config';
import { getSqsMessageSize } from '@app/utils';
/*
*
* Check if SQS message is within size limits
*
* **/
export default function isWithinSqsMessageSizeLimit(
message: AWS.SQS.Types.SendMessageRequest,
): boolean {
const max : number = config.queue.sqs.sizeLimit.max ?? 256 * 1024;
const min : number = config.queue.sqs.sizeLimit.min ?? 1;
const size = getSqsMessageSize(message);
// Check size is above min
if (size < min) return false;
// Check size is less than max
return size < max;
}
Export via index file:
export { default as verifySignature } from './verify-signature';
export { default as getSqsMessage } from './get-sqs-message';
export { default as isWithinSqsMessageSizeLimit } from './is-within-sqs-message-size-limit';
export { default as getSqsMessageSize } from './get-sqs-message-size';
There are really two outcomes from this util:
Let’s add some tests to verify those outcomes.
Add the new files:
touch ./utils/__tests__/is-within-sqs-message-size-limit.test.ts
Add the tests:
// utils/__tests__/is-within-sqs-message-size-limit.test.ts
import { mockEvent } from '@app/__mocks__';
import isWithinSqsMessageSizeLimit from '../is-within-sqs-message-size-limit';
const withinLimit = new Array(230*1024).join('a')
const overLimit = new Array(258*1024).join('a')
describe('isWithinSqsMessageSizeLimit', () => {
it('should return true if it is more than min', () => {
// @ts-ignore
expect(isWithinSqsMessageSizeLimit({
...mockEvent,
MessageBody: 'a'
})).toBe(true);
});
it('should return true if the message size is less than the max', () => {
// @ts-ignore
expect(isWithinSqsMessageSizeLimit({
...mockEvent,
MessageBody: withinLimit,
})).toBe(true);
});
it('should return false if the message size is more than the max', () => {
// @ts-ignore
expect(isWithinSqsMessageSizeLimit({
...mockEvent,
MessageBody: overLimit,
})).toBe(false);
});
});
Using the above utils, we can determine if the size of the sqs message out of bounds then throw a custom error to be handled later.
Add the new files:
touch ./errors/aws-sqs-size-limit-error.ts
Add the error:
// errors/aws-sqs-size-limit-error.ts
import CommonError from './common-error';
class AwsSqsSizeLimitError extends CommonError {
constructor(message: string) {
super(message);
this.name = this.constructor.name;
}
}
export default AwsSqsSizeLimitError;
Export via the index file:
// errors/index.ts
export { default as CommonError } from './common-error';
export { default as VerifySignatureError } from './verify-signature-error';
export { default as AwsSqsServiceError } from './aws-sqs-service-error';
export { default as AwsSqsSizeLimitError } from './aws-sqs-size-limit-error';
// services/sqs-service.ts
import AWS from 'aws-sdk';
import { APIGatewayProxyEvent } from 'aws-lambda';
import {
getSqsMessage,
getSqsMessageSize,
isWithinSqsMessageSizeLimit,
} from '@app/utils';
import config from '@app/config';
import {
AwsSqsServiceError,
AwsSqsSizeLimitError
} from '@app/errors';
export const sqs = new AWS.SQS({
apiVersion: '2012-11-05',
region: config.queue.sqs.defaultRegion,
});
/*
* Check message size to determine if it is within limit otherwise throw error
*
* **/
export function checkMessageSize(
message: AWS.SQS.Types.SendMessageRequest
): void {
if (!isWithinSqsMessageSizeLimit(message)) {
throw new AwsSqsSizeLimitError('exceeds sqs message limit')
.setOperation('services/sqs-service:sendMessage')
.setContext({
size: getSqsMessageSize(message),
});
}
}
/*
* adds an event message to the queue
*
* **/
export async function sendMessage(
event: APIGatewayProxyEvent
): Promise<string> {
const message: AWS.SQS.Types.SendMessageRequest = getSqsMessage(event);
checkMessageSize(message);
try {
const result: AWS.SQS.Types.SendMessageResult = (
await sqs.sendMessage(message).promise());
return result?.MessageId ?? '';
} catch (error) {
throw new AwsSqsServiceError('failed to sendMessage')
.setOperation('services/sqs-service:sendMessage')
.setContext({
error,
message,
});
}
}
For the SQS service, there are now three outcomes:
MessageId
from AWS SQSAwsSqsSizeLimitError
AwsSqsServiceError
// services/__tests__/sqs-service.test.ts
import { sqs, sendMessage } from '@app/services/sqs-service';
import { mockEvent } from '@app/__mocks__';
jest.mock('aws-sdk', () => {
const SqsMethods = {
sendMessage: jest.fn().mockReturnThis(),
promise: jest.fn(),
}
const mockAwsSdk = {
SQS: jest.fn(() => SqsMethods),
};
return mockAwsSdk;
});
describe('sqs-service', () => {
beforeEach(() => {
jest.resetAllMocks();
});
describe('sendMessage', () => {
it('should return the SQS MessageId upon success', () => {
const messageBody = 'test message';
const messageId = 'test';
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn().mockResolvedValue({
MessageId: messageId
}),
})
// @ts-ignore
expect(sendMessage({
...mockEvent,
body: messageBody
})).resolves.toBe(messageId)
});
it('should make a request to SQS with the correct message body', async() => {
const messageBody = 'test message';
const messageId = 'test';
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn().mockResolvedValue({
MessageId: messageId
}),
})
// @ts-ignore
await sendMessage({
...mockEvent,
body: messageBody
});
expect(sqs.sendMessage).toBeCalledWith({
MessageAttributes: {
requestId: {
DataType: 'String',
StringValue: mockEvent?.requestContext?.requestId
}
},
MessageBody: messageBody,
QueueUrl: ''
});
});
it('should default to empty string if no message id is found', () => {
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn().mockResolvedValue({}),
})
// @ts-ignore
expect(sendMessage({
...mockEvent,
body: 'test'
})).resolves.toBe('')
});
it('should throw AwsSqsServiceError if service call fails', async() => {
const mockError = new Error('mockError');
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: () => { throw mockError },
})
try {
// @ts-ignore
await sendMessage({
...mockEvent,
body: 'test'
})
} catch (error: any) {
expect(error.name).toBe('AwsSqsServiceError');
expect(error.message).toBe('failed to sendMessage');
expect(error.operation).toBe('services/sqs-service:sendMessage');
expect(error.context).toEqual({
error: mockError,
message: {
MessageAttributes: {
requestId: {
DataType: 'String',
StringValue: "Vb4i_hboIAMEVMA=",
}
},
MessageBody: 'test',
QueueUrl: '',
},
});
}
});
it('should throw AwsSqsSizeLimitError if service call fails', async() => {
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn().mockResolvedValue(null),
})
try {
// @ts-ignore
await sendMessage({
...mockEvent,
body: new Array(257 * 1024).join('a'),
});
} catch (error: any) {
expect(error.name).toBe('AwsSqsSizeLimitError');
expect(error.message).toBe('exceeds sqs message limit');
expect(error.operation).toBe('services/sqs-service:sendMessage');
expect(error.context.size).toBeDefined();
}
});
});
});
import { handler } from './index';
import { mockEvent } from '@app/__mocks__';
import { sqs } from '@app/services/sqs-service';
jest.mock('aws-sdk', () => {
const SqsMethods = {
sendMessage: jest.fn().mockReturnThis(),
promise: jest.fn(),
}
const mockAwsSdk = {
SQS: jest.fn(() => SqsMethods),
};
return mockAwsSdk;
});
jest.mock('@app/config', () => ({
__esModule: true,
default: {
webhook: {
signature: {
secret: 'test123',
algo: 'sha256',
header: 'x-hub-signature-256'
}
},
queue: {
sqs: {
url: 'queueUrl'
sizeLimit: {
max: 256 * 1024,
min: 1,
}
}
},
}
}))
describe('lambda.handler', () => {
afterEach(() => {
jest.resetAllMocks();
jest.clearAllMocks();
});
it('should return with 200 with the default message', async() => {
(sqs.sendMessage as jest.Mock).mockReturnValue({
promise: jest.fn(),
})
// @ts-ignore
await expect(handler(mockEvent))
.resolves
.toEqual({
statusCode: 200,
body: expect.stringMatching(
'success'
)
});
});
});
During the ingestion process, we will be verifying the signature by re-generating then matching it against the signature provided in the header.
# infra/variables.tf
variable "webhook_secret" {
type = string
}
# infra/main.tf
module "lambda_ingestion" {
source = "./modules/lambda"
code_src = "../functions/ingestion/main.zip"
bucket_id = aws_s3_bucket.lambda_bucket.id
timeout = local.default_lambda_timeout
function_name = "Ingestion-function"
runtime = "nodejs12.x"
handler = "dist/index.handler"
publish = true
alias_name = "ingestion-dev"
alias_description = "Alias for ingestion function"
iam_statements = {
sqs = {
actions = [
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:SendMessage",
"sqs:ReceiveMessage",
]
effect = "Allow"
resources = [
aws_sqs_queue.ingest_queue.arn
]
}
}
environment_vars = {
QueueUrl = aws_sqs_queue.ingest_queue.id
DefaultRegion = var.aws_region
WebhookSignatureSecret = var.webhook_secret
}
}
⚠️ Note: passing the secret as plain text through the environment variable is not the most secure way to handle this.A better way is using a secret store (AWS SSM or secrets manager).
For simplicity we will go with this but I will likely do a follow-up article to go through steps to handle this securely.
When you pass in your webhook secret, make sure to remember it as we will setup our Github webhook using the shared secret (they should match to ensure our signature verification works properly).
// This will re-generate the assets
pnpm run generate-assets --filter @function/*
export AWS_ACCESS_KEY_ID=<your-key>
export AWS_SECRET_ACCESS_KEY=<your-secret>
export AWS_DEFAULT_REGION=us-east-1
export TF_VAR_webhook_secret=<secret>
terraform init
terraform plan
terraform apply -auto-approve
Go to a Github respository that you own. We will setup a webhook integration using that repo.
Add the webhook details like endpoint and secret then subscribe to specific events on your repo.
You can go back and tweak it later once we verified this is working properly without issues.
Fill out the following:
Payload URL - This will be the endpoint of your API gateway (from terraform output - <api_endpoint>
)
<endpoint>/webhooks/receive
)Content type - Set this to application/json
Secret - This is the shared secret between Github and your webhook service (Make sure its the same secret you used above for the terraform)
Under Which events would you like to trigger this webhook?, select ”Let me select individual events.” and subscribe to “Stars” for testing.
Test out the webhook endpoint push by “Starring” your repository.
Make sure you see the logs in those two spots.
That’s it!
Recap on what we did:
We added the verifySignature
util to check the signature matches
We added the getSqsMessage
to format the Sqs Message based on the event
We added a check for the sqs message size before sending the message the SQS queue
We added sendMessage
method for sending message to our SQS Queue
We setup Github webhooks and verified that it works with our code
With all these things in place, this should be good enough for a production work load!
In the next module, we’ll add in our global error handler for our lambda function and also look at how we can debug our infrastructure end-to-end with our logging setup.
Here are a few ideas:
Add support AWS S3 when message size exceeds the limit
Add logic in Process-Queue to process the event after receiving it (ie send an email or notify slack)
Add webhook integration for Stripe payment events
Integrate AWS SSM or Secrets manager to manage the webhook secret
See you in the next module!
Then consider signing up to get notified when new content arrives!