[{"type":"blog","content":"\r\n## TL:DR\r\n\r\nI needed to track down some information, and I used the below snippet in CloudWacth Insights to easily track down the information I needed from my logs. \r\n\r\n```bash\r\nfields @timestamp, @message, @logStream, @log\r\n| filter @message ~=''\r\n| sort @timestamp desc\r\n| limit 20\r\n```\r\n\r\n## Some Context\r\n\r\nI have a project that I'm working on which is still in the PoC stage, and due to the nature of its deployment, a lot of the debugging is coming from the logs. The project is in AWS, and we use CloudWatch for all our logging. CloudWatch Insights is a tool that allows you to search and analyze logs for your app running in AWS. \r\n\r\nInsights give you a shortcut for helping you look through all the logs. Especially given CloudWatch's clunky user interface. Also, since CloudWatch organizes your logs into streams called log groups, clicking back and forth to these different log groups can only prove tedious unless you know where you are searching. AWS has even created a query language for searching through logs, empowering users to make complex queries when needed. \r\n\r\nWhat I was looking for was pretty simple, but I will cover some things that might help when searching through logs for specific information. \r\n\r\n## CloudWatch Insights\r\n\r\nFirst, CloudWatch Insights is found under the Logs section of CloudWatch in the AWS Dashboard.\r\n\r\n $\"image\"$ \r\n\r\nOnce there, you can choose the date range and the log group you want to search. Then below that is the query editor. The query language seems reasonably straightforward. Below is what I used to search for the information I needed. \r\n\r\n```bash\r\nfields @timestamp, @message, @logStream, @log\r\n| filter @message ~=''\r\n| sort @timestamp desc\r\n| limit 20\r\n```\r\n\r\nI was looking for a specific ID number for some data I was looking for. This data was causing issues, and I needed to see what the logs said. So to break it down, the words with the `@` symbols first are the built-in CloudWatch fields. \r\n\r\n- @message is the printout of the log unparsed\r\n- @logStream is going to be a link back to the log group that the row you're looking at \r\n- @log is the group identifier for the log\r\n- @timestamp is the event timestamp in the log's timestamp field\r\n\r\nThe above fields are generated for every log. I was looking for an exact string, so I added the `filter` statement only to display logs that contained that exact string. However, you can use it to look for string patterns and many other identifiers. There are a ton more commands you can use to find stuff. [Here are the docs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html). I also used [this site](https://www.pujan.net/post/2021-10-01-aws-cloudwatch-insights/#contains), which has many query language examples. You can even save the queries and add the result to an existing dashboard by clicking the Add to Dashboard button in the dashboard. \r\n\r\nWhatever you're tracking should be set up to do some actual logging. [Pino](https://github.com/pinojs/pino) is an excellent library for quickly spinning up logger objects you can use in your code. I can talk about logging at a different time, but one thing that makes Insights so valuable is if you have logging set up and you want to do something like having your code log a specific event that you don't want to throw errors for, but you want to track you can easily set up a logging action and use that in Insights. \r\n\r\nSo, say I have some data that I regularly process and need to know the value of a specific field for additional tracking. I can create a logging message like the one below.\r\n\r\n```JavaScript\r\nloggingObj.debug({object}, 'this object needs to be tracked');\r\n```\r\n\r\nNow I can create a query in Insights to be able to track the value of that field.\r\n\r\n```bash\r\nfields @timestamp, object\r\n| filter object~=''\r\n| sort @timestamp desc\r\n```\r\n\r\nThe destructured `object` becomes a top-level field in the logs now. So we can use Insights to query that field. Now I can save this query and have the time range back six months and see the value of that field coming in over six months. And the data will appear in that log group's dashboard. All without spinning up another data stack. I'm excited to see what else we can use CloudWatch Insights for!\r\n\r\n-George","frontmatter":{"title":"Search CloudWatch Logs Using CloudWatch Insights","subtitle":"I'm working on a lot of debugging for a project, and being able to query specific values in the logs is helpful.","description":"I'm working on a lot of debugging for a project, and being able to query specific values in the logs is helpful.","slug":"searching-aws-cloudwatch-logs-using-cloudwatch-insights","tags":["note","aws"],"category":"dev-blog","image":null,"date":"2023-07-07T00:00:00.000Z","canonical":"https://georgeoffley.com/searching-aws-cloudwatch-logs-using-cloudwatch-insights"},"title":"Search CloudWatch Logs Using CloudWatch Insights","subtitle":"I'm working on a lot of debugging for a project, and being able to query specific values in the logs is helpful.","description":"I'm working on a lot of debugging for a project, and being able to query specific values in the logs is helpful. ","category":"dev-blog","tags":["note","aws"],"canonical":"https://georgeoffley.com/searching-aws-cloudwatch-logs-using-cloudwatch-insights","slug":"searching-aws-cloudwatch-logs-using-cloudwatch-insights","date":"2023-07-07T00:00:00.000Z","readingTime":"4 minutes","ghMetadata":{"issueUrl":"https://github.com/georgeoffley/george-offley-blog-swyxkit/issues/10","commentsUrl":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/10/comments","title":"Searching AWS CloudWatch Logs Using CloudWatch Insights","created_at":"2023-07-07T20:27:57Z","updated_at":"2023-07-07T21:19:07Z","reactions":{"url":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/10/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0}}},{"type":"blog","content":"\r\n## TL:DR\r\n\r\nI started using ChatGPT to ask questions and clarify my understanding of a new subject I'm learning, and it's been a great help. For those of us learning things on our own, it can be a boon to our success.\r\n\r\n## Some Context\r\n\r\nI've been learning on my own for a long time now. Being as curious a person as I am, it was inevitable that I would start to analyze how I learn to optimize my learning. It took me a while, but here I am.\r\n\r\nWhen I start out learning something, here is how the workflow has been in the past:\r\n\r\n- Obsessively read about the thing.\r\n- Read more about a thing\r\n- Bite off more than I can chew and try to do as much as possible\r\n- Eventually, find equilibrium (usually takes a while)\r\n\r\nI have severe ADHD, and I've been building functional habits to live with it since second grade. Eventually, I learned to adjust these habits to facilitate my learning. So these days, when I'm trying to break something down, it goes more like:\r\n\r\n- Obsessively reading about the thing (just how my brain works)\r\n- Find the smallest piece of that thing that I can do and finish quickly\r\n- Go over what I learned and try to take notes to understand better what I did and did not do right\r\n- Iterate on that process\r\n\r\nAnd this has served me well. And I work in code, so a process like this works well when I'm learning something new built out of the same foundations I already have. Then comes along this latest thing I'm trying to understand, math. Oh god.\r\n\r\n## The Maths\r\n\r\nSo, I did the usual, found some excellent resources, and got started. I'm currently watching a live-stream [math course for game devs](https://www.youtube.com/playlist?list=PLImQaTpSAdsD88wprTConznD1OY1EfK_V) made by [Freya Holmer](https://twitter.com/FreyaHolmer) and it's dense but also accessible. However, occasionally I need help understanding a concept. While resources do exist for understanding these concepts, one thing I've learned about myself is that I need to ask questions to clarify my understanding of something. Tough to do that when you're learning in a vacuum. \r\n\r\nSo what do I do? Then I remembered that I read somewhere someone tried using ChatGPT as a better version of Googling the information you need. I wish I had kept the tweet because that idea was a lightbulb for me. So I went ahead and gave it a shot. Here's what I asked.\r\n\r\n```text\r\nWhat do two-dimensional vectors have in relation to sign?\r\n```\r\n\r\nAnd there it pops out the following.\r\n\r\n```text\r\nTwo-dimensional vectors do not have a concept of sign in the same way that one-dimensional numbers do. In one...\r\n```\r\n\r\nDoing this helped me get unblocked so much quicker than I would have been Googling that question and trying to sift through everything Google decided to put on the first couple of pages (after the ads). And even better, I could ask follow-up questions. This was great.\r\n\r\n### A Note\r\n\r\nI've seen plenty of videos of people doing something similar in which they ask ChatGPT how to do things. While that is also cool, I want to differentiate from what I'm talking about. I've adopted ChatGPT as a sounding board to clarify my understanding better, much like a live instructor would be. I'm still using high-quality resources from intelligent and qualified people, taking notes, and trying to undertake active practice.\r\n\r\nHigh-quality resources created by talented and intelligent people are the best way to learn something for me. Whether in video form, written, visual, or whatever form it takes. If I'm sitting down and listening to an instructor, I'll raise my hand and ask a bunch of follow-up questions. However, ChatGPT is a great tool to help me clarify what I'm learning if I'm learning independently. \r\n\r\n-George","frontmatter":{"title":"Using ChatGPT for Learning","subtitle":"I started using CharGPT as a learning aid, and it's kinda cool.","description":"I started using CharGPT as a learning aid, and it's kinda cool.","slug":"using-chatgpt-for-learning","tags":["note"],"category":"dev-blog","image":null,"date":"2023-06-30T00:00:00.000Z","canonical":"https://georgeoffley.com/using-chatgpt-for-learning"},"title":"Using ChatGPT for Learning","subtitle":"I started using CharGPT as a learning aid, and it's kinda cool.","description":"I started using CharGPT as a learning aid, and it's kinda cool. ","category":"dev-blog","tags":["note"],"canonical":"https://georgeoffley.com/using-chatgpt-for-learning","slug":"using-chatgpt-for-learning","date":"2023-06-30T00:00:00.000Z","readingTime":"3 minutes","ghMetadata":{"issueUrl":"https://github.com/georgeoffley/george-offley-blog-swyxkit/issues/8","commentsUrl":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/8/comments","title":"Using ChatGPT for Learning","created_at":"2023-06-30T20:54:48Z","updated_at":"2023-06-30T22:06:54Z","reactions":{"url":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/8/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0}}},{"type":"blog","content":"\r\n## TL:DR\r\n\r\n```bash\r\nhttps://{restapi_id}.execute-api.{region}.amazonaws.com/{stage_name}/\r\n```\r\n\r\nThe `restapi_id` is the API ID found in API Gateway on the AWS Management Console, pictured below. Remember that each deployment you have will have an API ID set up for it. So your `test` API ID won't be the same as your `prod` API.\r\n\r\n $\"Image$ \r\n\r\nI put in the closest `region` for me (`us-east-1`), but that can be any region. \r\n\r\nThen you put in the stage the API is deployed to. So `prod`, `dev`, `test`, or whatever environments you have set up.\r\n\r\nAnd that's it. There's a more extensive conversation about permissions, deployments, etc. I might have that conversation on this blog. Stay tuned.\r\n\r\n-George","frontmatter":{"title":"How to Call A Deployed API Endpoint in Amazon API Gateway","subtitle":"Snippit used for calling API endpoints","description":"Snippit used for calling API endpoints","slug":"how-to-call-a-deployed-api-endpoint-in-amazon-api-gateway","tags":["note","aws"],"category":"dev-blog","image":null,"date":"2023-02-01T00:00:00.000Z","canonical":"https://georgeoffley.com/how-to-call-a-deployed-api-endpoint-in-amazon-api-gateway"},"title":"How to Call A Deployed API Endpoint in Amazon API Gateway","subtitle":"Snippit used for calling API endpoints","description":"Snippit used for calling API endpoints ","category":"dev-blog","tags":["note","aws"],"canonical":"https://georgeoffley.com/how-to-call-a-deployed-api-endpoint-in-amazon-api-gateway","slug":"how-to-call-a-deployed-api-endpoint-in-amazon-api-gateway","date":"2023-02-01T00:00:00.000Z","readingTime":"1 minute","ghMetadata":{"issueUrl":"https://github.com/georgeoffley/george-offley-blog-swyxkit/issues/4","commentsUrl":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/4/comments","title":"How to Call A Deployed API Endpoint in Amazon API Gateway","created_at":"2023-02-01T21:59:10Z","updated_at":"2023-04-14T13:25:17Z","reactions":{"url":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/4/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0}}},{"type":"blog","content":"\r\n## TL;DR\r\n\r\nAmazon API Gateway only allows for one path variable per resource. So these endpoints work:\r\n\r\n- `/user/{id}/update-status`\r\n- `/user/{id}/search-user`\r\n\r\nThese will throw an error since I already have the `id` as a path variable, and trying to set another is a Nah dawg.\r\n\r\n- `/user/{id}/update-status`\r\n- `/user/{id}/search-user`\r\n- `/user/{status}`\r\n\r\nThe error is this:\r\n\r\n```bash\r\nAn error occurred: ApiGatewayResourceChecksIdVar - A sibling ({id}) of this resource already has a variable path part -- only one is allowed.\r\n```\r\n\r\n

\r\n\r\n## More Context\r\n\r\nSo today, while charging full speed into a deliverable deadline, I spent an hour tracking down why my deployment to AWS kept failing. So, for some context, I'm making API endpoints for an internal tool at work. \r\n\r\nSo to work, I wrote the [Cloudformation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) files for the new feature I was adding, only to run into an error with my deployment.\r\n\r\nIn my case, we had a resource for `/user` with a bunch of endpoints under it. Like so: \r\n\r\n- `/user/{id}/update-status`\r\n- `/user/{id}/search-user`\r\n\r\n If you notice the `{id}` in those endpoints, those are [path variables](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-routes.html#http-api-routes-path-variables). These let us pass information into the API call. In the case above, we use them to pass in a user's ID to know which user to perform an action.\r\n\r\nFor this new feature I was working on, I needed to add the ability to create users with a specific `user-status`; my solution was to make an endpoint where the `user-status` was passed in via a path variable. Like so:\r\n\r\n`/user/{status}`\r\n\r\n## The Problem\r\n\r\nAPI Gateway didn't like that one bit and threw this error when I deployed:\r\n\r\n```bash\r\nAn error occurred: ApiGatewayResourceChecksIdVar - A sibling ({id}) of this resource already has a variable path part -- only one is allowed.\r\n```\r\n\r\nIt turns out [API Gateway only allows one path variable per resource](https://forum.serverless.com/t/api-gateway-similar-routes-different-methods-problem/7014), and my `user` resource already had the `id` path variable, so API Gateway threw an error when trying to set another. \r\n\r\n## Figuring It Out\r\n\r\nThis problem was easily solvable since it was a [`POST`](https://www.restapitutorial.com/lessons/httpmethods.html) endpoint, and I would send a body of `JSON` data to it. So my solution was to add the status to the request payload. I know a little more about API Gateway; hopefully, you do too!\r\n\r\n-George","frontmatter":{"title":"AWS API Gateway Resources","subtitle":null,"description":null,"slug":"aws-api-gateway-resources","tags":["note","aws"],"category":"dev-blog","image":null,"date":"2023-01-07T00:00:00.000Z","canonical":"https://georgeoffley.com/aws-api-gateway-resources"},"title":"AWS API Gateway Resources","subtitle":null,"description":"## TL;DR ","category":"dev-blog","tags":["note","aws"],"canonical":"https://georgeoffley.com/aws-api-gateway-resources","slug":"aws-api-gateway-resources","date":"2023-01-07T00:00:00.000Z","readingTime":"2 minutes","ghMetadata":{"issueUrl":"https://github.com/georgeoffley/george-offley-blog-swyxkit/issues/2","commentsUrl":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/2/comments","title":"AWS API Gateway Resources","created_at":"2023-01-08T02:19:16Z","updated_at":"2023-04-14T13:30:04Z","reactions":{"url":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/2/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0}}},{"type":"blog","content":"\r\n### Introduction \r\n\r\nRecently, I was experimenting with system testing on a new stack. While working with [Selenium Remote Driver](https://www.selenium.dev/documentation/webdriver/remote_webdriver/) to create [headless web browsers](https://en.wikipedia.org/wiki/Headless_browser) (more on that later) to run against my app, I came across a Docker option I had not known. This Docker feature allowed for easily accessible and fast file reading and writing between the host and the container. It was being used as a volume, but I learned it was much more. When I looked it up, it was referred to as a [*Shared Memory Device*](https://en.wikipedia.org/wiki/Shared_memory).\r\n\r\n\r\n\r\n### Shared Memory Device \r\n\r\nTurns out that this is an old Linux staple. The shared memory device, located in the `/dev/shm/` directory in the file system, utilizes temporary storage using RAM rather than disk storage. Using a shared volume available on the RAM and not the disk makes for a much faster read/write speeds. These devices also allows for [inter-process communication](https://en.wikipedia.org/wiki/Inter-process_communication).\r\n\r\n### Use in Docker \r\n\r\nDocker allows for the use of this device. When conducting my testing, I used the volume mapping feature to easily map local directories with directories inside the containers we’re working on. As seen below.\r\n\r\n```bash\r\ndocker run --rm -it -v /dev/shm/:/dev/shm/ --name ubuntu ubuntu\r\n```\r\n\r\nThis mapped the local shared memory device to the Docker device. Docker, by default, allows for up to 64 MB of storage space using the shared memory device. You can see the default allocation when searching for the shared storage device within the container. As seen below.\r\n\r\n```bash\r\n$ docker inspect ubuntu | grep -i shm\r\n \"ShmSize\": 67108864,\r\n```\r\n\r\nI was saving screenshots just to ensure that I was looking at the thing I was hoping to look at. Although I was only doing some foundational work, I could find myself in a pickle with only 64 MB of space available if this work ends up scaling. Should we need to, Docker lets us change the size of this device using the *--shm-size* option. As seen below.\r\n\r\n```bash\r\ndocker run --rm -it -v /dev/shm/:/dev/shm --shm-size=2gb --name ubuntu ubuntu\r\n```\r\n\r\nAbove we add in the *--shm-size* option and we can specify all the space we need! Let’s look one more time to confirm that it works.\r\n\r\n```bash\r\n$ docker inspect ubuntu | grep -i shm\r\n \"ShmSize\": 2147483648,\r\n```\r\n\r\nShazam! We have more storage in our shared memory device to play around with.\r\n\r\n### Conclusion \r\n\r\nThere is more to come about my adventures in Dockerizing system testing, but for now, I was able to learn a new thing about Linux and Docker. I hope it helps someone who came across it without knowing as I did.\r\n\r\n-George","frontmatter":{"title":"Shared Memory in Docker","subtitle":null,"description":null,"slug":"shared-memory-in-docker","tags":["note","docker"],"category":"dev-blog","image":null,"date":"2023-01-07T00:00:00.000Z","canonical":"https://georgeoffley.com/shared-memory-in-docker"},"title":"Shared Memory in Docker","subtitle":null,"description":"### Introduction ","category":"dev-blog","tags":["note","docker"],"canonical":"https://georgeoffley.com/shared-memory-in-docker","slug":"shared-memory-in-docker","date":"2023-01-07T00:00:00.000Z","readingTime":"2 minutes","ghMetadata":{"issueUrl":"https://github.com/georgeoffley/george-offley-blog-swyxkit/issues/1","commentsUrl":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/1/comments","title":"Shared Memory in Docker","created_at":"2023-01-07T20:49:14Z","updated_at":"2023-04-14T13:31:08Z","reactions":{"url":"https://api.github.com/repos/georgeoffley/george-offley-blog-swyxkit/issues/1/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0}}}]

George Offley Blog

All Posts