Published on: Sat Aug 27 2022
After reading the article, you should:
✅ Be able to differentiate between Logs & Metrics
✅ Know what and why of Cloudwatch Embedded metrics
✅ Gain a high-level understanding of what it takes to convert logs into metrics in AWS
If you use AWS then there is a good chance you are using AWS Cloudwatch logs for your logging.
Logs are very useful for specific events within your code between requests.
However, if you want to gain high level insights, you want to be looking at an aggregation of events at a point in time for a system rather than a specific event. This is where metrics comes in.
Now the question become — how do you go from the logs to the metrics ?
If you are using AWS, then there is an easy way to achieve this, and that is using AWS Cloudwatch embedded metrics.
The gist of it is that these are logs formatted in a specific format that AWS uses to convert these logs into metrics.
Let’s go through the details.
AWS Cloudwatch embedded metrics are just logs in a specific format.
It needs to be a valid JSON, and contain the right details so AWS can convert these logs into metrics.
Format: x-amzn-logs-format: json/emf
Here is a sample of the format:
{
"_aws": {
"Timestamp": 1574109732004,
"CloudWatchMetrics": [
{
"Namespace": "lambda-function-metrics",
"Dimensions": [["functionVersion"]],
"Metrics": [
{
"Name": "time",
"Unit": "Milliseconds"
}
]
}
]
},
"functionVersion": "$LATEST",
"time": 100,
"requestId": "989ffbf8-9ace-4817-a57c-e4dd734019ee"
}
Here are a few points of why I think AWS Cloudwatch embedded metrics matters.
You go from logs (specific events) to high level insights (metrics), you can’t improve or analyze what you can’t see.
These metrics can be used to define contracts such as SLAs, SLIs and SLOs.
AWS Cloudwatch embedded metrics make this process that much easier and simpler.
This feature is an opt-in feature, meaning its optional.
When you want to “enable” it, you just format your log in a specific way.
No friction, no extra steps — It simply just works! In my opinion, that’s a big win.
In this setup, AWS assumes almost all the responsibility of this feature and infrastructure.
This reduces the operational burdens that you or your team is responsible for.
To me, that’s another big win!
Using AWS Lambda, you don’t need any further configuration, it just works.
If you are using AWS ECS or EKS, then the cloudwatch agent is required to make it work.
AWS Lambda - Cloudwatch embedded metrics are supported out of the box
AWS ECS and EKS - Requires the CloudWatch agent to send Cloudwatch embedded metrics
Here is an example of how AWS Cloudwatch embedded metrics works with AWS Lambda. Again, this works out of the box.
Sample AWS Architecture illustrating how Cloudwatch embedded metrics work
The steps:
API gateway receives request
API Gateway invokes Lambda Alias
Lambda processes the event
Lambda logs to cloudwatch logs ⭐️
Logs are automatically converted to Cloudwatch metrics
Step 1-4 are pretty typical in an AWS Lambda setup, the big difference is that when you log in a certain format, AWS will pick that up and convert those logs into metrics.Step 4 is where the magic happens!
You can format these logs manually or use a client library provided by AWS (recommended).
To make it easier, AWS has provided client libraries to generate these formatted logs.
It provides an API to help you create formatted logs that results in desired cloudwatch metrics based on the settings you provide.
The client libraries:
Example using the client library with Node.js:
const { createMetricsLogger, Unit } = require("aws-embedded-metrics");
const sendMetrics = async () => {
const metrics = createMetricsLogger();
metrics.putDimensions({ Service: "Aggregator" });
metrics.putMetric("ProcessingLatency", 100, Unit.Milliseconds);
metrics.setProperty("RequestId", "422b1569-16f6-4a03-b8f0-fe3fd9b100f8");
// ...
await metrics.flush();
};
await sendMetrics();
So, to recap:
Metrics provide high-level insights while logs provide details about specific events
AWS Cloudwatch embedded metrics converts your AWS Cloudwatch logs into AWS Cloudwatch metrics
When using AWS Lambda, embedded metrics are supported out of the box
When using AWS ECS & EKS, the cloudwatch agent will need to be configured
AWS provides client libraries to makes it easier for you manage the resulting embedded metrics
And... That’s all. I hope you found that helpful!
If you learned something new, please share this article with a friend or co-worker 🙏❤️! (Thanks!)
Then consider signing up to get notified when new content arrives!