<h2 id="goals"><a href="#goals" aria-hidden="true" tabindex="-1"></a>Goals</h2>
After reading the article, you should:
<ul>
<li>
✅ Be able to differentiate between Logs &#x26; Metrics
</li>
<li>
✅ Know what and why of Cloudwatch Embedded metrics
</li>
<li>
✅ Gain a high-level understanding of what it takes to convert logs into metrics in AWS
</li>
</ul>
<h2 id="content"><a href="#content" aria-hidden="true" tabindex="-1"></a>Content</h2>
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#what-is-it-">What is it ?</a></li>
<li><a href="#why-it-matters">Why it matters</a></li>
<li><a href="#supported-platforms">Supported Platform</a></li>
<li><a href="#example---using-aws-lambda">Example - using AWS Lambda</a></li>
<li><a href="#client-libraries">Client Libraries</a></li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
<h2 id="introduction"><a href="#introduction" aria-hidden="true" tabindex="-1"></a>Introduction</h2>
If you use AWS then there is a good chance you are using AWS Cloudwatch logs for your logging.
Logs are very useful for specific events within your code between requests.
However, if you want to gain high level insights, you want to be looking at an aggregation of events at a point in time for a system rather than a specific event. This is where metrics comes in.
Now the question become — how do you go from the logs to the metrics ?
If you are using AWS, then there is an easy way to achieve this, and that is using AWS Cloudwatch embedded metrics.
The gist of it is that these are logs formatted in a specific format that AWS uses to convert these logs into metrics.
Let’s go through the details.
<h2 id="what-is-it-"><a href="#what-is-it-" aria-hidden="true" tabindex="-1"></a>What is it ?</h2>
AWS Cloudwatch embedded metrics are just logs in a specific format.
It needs to be a valid JSON, and contain the right details so AWS can convert these logs into metrics.
Format: <code class="language-">x-amzn-logs-format: json/emf
</code>
Here is a sample of the format:
<div class="remark-highlight"><pre class="language-json line-numbers"><code>{
 "_aws": {
 "Timestamp": 1574109732004,
 "CloudWatchMetrics": [
 {
 "Namespace": "lambda-function-metrics",
 "Dimensions": [["functionVersion"]],
 "Metrics": [
 {
 "Name": "time",
 "Unit": "Milliseconds"
 }
 ]
 }
 ]
 },
 "functionVersion": "$LATEST",
 "time": 100,
 "requestId": "989ffbf8-9ace-4817-a57c-e4dd734019ee"
}
</code></pre></div>
<h4 id="-helpful-reference"><a href="#-helpful-reference" aria-hidden="true" tabindex="-1"></a>📝 Helpful reference:</h4>
<ul>
<li><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html" target="_blank" rel="nofollow noopener noreferrer">AWS - Cloudwatch Embedded metrics specification</a></li>
</ul>
<h2 id="why-it-matters"><a href="#why-it-matters" aria-hidden="true" tabindex="-1"></a>Why it matters</h2>
Here are a few points of why I think AWS Cloudwatch embedded metrics matters.
<h3 id="metrics-are-actionable"><a href="#metrics-are-actionable" aria-hidden="true" tabindex="-1"></a>Metrics are actionable</h3>
You go from logs (specific events) to high level insights (metrics), you can’t improve or analyze what you can’t see.
These metrics can be used to define contracts such as SLAs, SLIs and SLOs.
AWS Cloudwatch embedded metrics make this process that much easier and simpler.
<h3 id="great-developer-experience-dx"><a href="#great-developer-experience-dx" aria-hidden="true" tabindex="-1"></a>Great Developer Experience (DX)</h3>
This feature is an opt-in feature, meaning its optional.
When you want to “enable” it, you just format your log in a specific way.
No friction, no extra steps — It simply just works! In my opinion, that’s a big win.
<h3 id="shifted-operational-responsibility"><a href="#shifted-operational-responsibility" aria-hidden="true" tabindex="-1"></a>Shifted operational responsibility</h3>
In this setup, AWS assumes almost all the responsibility of this feature and infrastructure.
This reduces the operational burdens that you or your team is responsible for.
To me, that’s another big win!
<h2 id="supported-platforms"><a href="#supported-platforms" aria-hidden="true" tabindex="-1"></a>Supported Platforms</h2>
Using AWS Lambda, you don’t need any further configuration, it just works.
If you are using AWS ECS or EKS, then the cloudwatch agent is required to make it work.
<ul>
<li>
AWS Lambda - Cloudwatch embedded metrics are supported out of the box
</li>
<li>
AWS ECS and EKS - Requires the CloudWatch agent to send Cloudwatch embedded metrics
</li>
</ul>
<h2 id="example---using-aws-lambda"><a href="#example---using-aws-lambda" aria-hidden="true" tabindex="-1"></a>Example - using AWS Lambda</h2>
Here is an example of how AWS Cloudwatch embedded metrics works with AWS Lambda. Again, this works out of the box.
<div style="margin: 2em auto;">
 <div style="display:flex;justify-content:center;">
 <img src="/images/cloudwatch-embedded-metrics-architecture.png" alt="Architecture demonstrating AWS Cloudwatch embedded metrics" style="width:80%" />
 </div>
 <div style="display:flex;justify-content:center;">
 <blockquote>Sample AWS Architecture illustrating how Cloudwatch embedded metrics work</blockquote>
 </div>
</div>
The steps:
<ol>
<li>
API gateway receives request
</li>
<li>
API Gateway invokes Lambda Alias
</li>
<li>
Lambda processes the event
</li>
<li>
Lambda logs to cloudwatch logs ⭐️
</li>
<li>
Logs are automatically converted to Cloudwatch metrics
</li>
</ol>
<blockquote class="common">
Step 1-4 are pretty typical in an AWS Lambda setup, the big difference is that when you log in a certain format, AWS will pick that up and convert those logs into metrics.
 Step 4 is where the magic happens! 
You can format these logs manually or use a client library provided by AWS (recommended).
</blockquote>
<h2 id="client-libraries"><a href="#client-libraries" aria-hidden="true" tabindex="-1"></a>Client Libraries</h2>
To make it easier, AWS has provided client libraries to generate these formatted logs.
It provides an API to help you create formatted logs that results in desired cloudwatch metrics based on the settings you provide.
The client libraries:
<ul>
<li><a href="https://github.com/awslabs/aws-embedded-metrics-node" target="_blank" rel="nofollow noopener noreferrer">Node.js</a></li>
<li><a href="https://github.com/awslabs/aws-embedded-metrics-python" target="_blank" rel="nofollow noopener noreferrer">Python</a></li>
<li><a href="https://github.com/awslabs/aws-embedded-metrics-java" target="_blank" rel="nofollow noopener noreferrer">Java</a></li>
<li><a href="https://github.com/awslabs/aws-embedded-metrics-dotnet" target="_blank" rel="nofollow noopener noreferrer">C#</a></li>
</ul>
Example using the client library with Node.js:
<div class="remark-highlight"><pre class="language-ts line-numbers"><code>const { createMetricsLogger, Unit } = require("aws-embedded-metrics");

const sendMetrics = async () =&gt; {
 const metrics = createMetricsLogger();
 metrics.putDimensions({ Service: "Aggregator" });
 metrics.putMetric("ProcessingLatency", 100, Unit.Milliseconds);
 metrics.setProperty("RequestId", "422b1569-16f6-4a03-b8f0-fe3fd9b100f8");
 // ...
 await metrics.flush();
};

await sendMetrics();
</code></pre></div>
<h2 id="conclusion"><a href="#conclusion" aria-hidden="true" tabindex="-1"></a>Conclusion</h2>
So, to recap:
<ul>
<li>
Metrics provide high-level insights while logs provide details about specific events
</li>
<li>
AWS Cloudwatch embedded metrics converts your AWS Cloudwatch logs into AWS Cloudwatch metrics
</li>
<li>
When using AWS Lambda, embedded metrics are supported out of the box
</li>
<li>
When using AWS ECS &#x26; EKS, the cloudwatch agent will need to be configured
</li>
<li>
AWS provides client libraries to makes it easier for you manage the resulting embedded metrics
</li>
</ul>
And... That’s all. I hope you found that helpful!
If you learned something new, please share this article with a friend or co-worker 🙏❤️! (Thanks!)

Jerry Chang

Go Back

What is Cloudwatch Embedded Metrics ?

Enjoy the content ?