Search CloudWatch Logs Using CloudWatch Insights

TL:DR

I needed to track down some information, and I used the below snippet in CloudWacth Insights to easily track down the information I needed from my logs.

fields @timestamp, @message, @logStream, @log
| filter @message ~='<id#>'
| sort @timestamp desc
| limit 20

Some Context

I have a project that I’m working on which is still in the PoC stage, and due to the nature of its deployment, a lot of the debugging is coming from the logs. The project is in AWS, and we use CloudWatch for all our logging. CloudWatch Insights is a tool that allows you to search and analyze logs for your app running in AWS.

Insights give you a shortcut for helping you look through all the logs. Especially given CloudWatch’s clunky user interface. Also, since CloudWatch organizes your logs into streams called log groups, clicking back and forth to these different log groups can only prove tedious unless you know where you are searching. AWS has even created a query language for searching through logs, empowering users to make complex queries when needed.

What I was looking for was pretty simple, but I will cover some things that might help when searching through logs for specific information.

CloudWatch Insights

First, CloudWatch Insights is found under the Logs section of CloudWatch in the AWS Dashboard.

Once there, you can choose the date range and the log group you want to search. Then below that is the query editor. The query language seems reasonably straightforward. Below is what I used to search for the information I needed.

fields @timestamp, @message, @logStream, @log
| filter @message ~='<id#>'
| sort @timestamp desc
| limit 20

I was looking for a specific ID number for some data I was looking for. This data was causing issues, and I needed to see what the logs said. So to break it down, the words with the @ symbols first are the built-in CloudWatch fields.

• @message is the printout of the log unparsed
• @logStream is going to be a link back to the log group that the row you’re looking at
• @log is the group identifier for the log
• @timestamp is the event timestamp in the log’s timestamp field

The above fields are generated for every log. I was looking for an exact string, so I added the filter statement only to display logs that contained that exact string. However, you can use it to look for string patterns and many other identifiers. There are a ton more commands you can use to find stuff. Here are the docs. I also used this site, which has many query language examples. You can even save the queries and add the result to an existing dashboard by clicking the Add to Dashboard button in the dashboard.

Whatever you’re tracking should be set up to do some actual logging. Pino is an excellent library for quickly spinning up logger objects you can use in your code. I can talk about logging at a different time, but one thing that makes Insights so valuable is if you have logging set up and you want to do something like having your code log a specific event that you don’t want to throw errors for, but you want to track you can easily set up a logging action and use that in Insights.

So, say I have some data that I regularly process and need to know the value of a specific field for additional tracking. I can create a logging message like the one below.

loggingObj.debug({object}, 'this object needs to be tracked');

Now I can create a query in Insights to be able to track the value of that field.

fields @timestamp, object
| filter object~='<value>'
| sort @timestamp desc

The destructured object becomes a top-level field in the logs now. So we can use Insights to query that field. Now I can save this query and have the time range back six months and see the value of that field coming in over six months. And the data will appear in that log group’s dashboard. All without spinning up another data stack. I’m excited to see what else we can use CloudWatch Insights for!

-George