A few nights ago, I woke up to find that my sites had been unreachable for two hours during the night. By the time I checked, everything was back online. CloudWatch had sent alerts, but I wasn’t watching emails at that hour.
The sites recovered on their own somehow. Docker’s restart policy, traffic fluctuations, I’m not entirely sure. What I do know is that something broke, it fixed itself, and I had no clear way to understand what happened.
This is the story of that investigation, and how I ended up building a proper observability setup. Something I probably should have done from the start.
The Investigation
When I finally dug into CloudWatch the next morning, the metrics told a clear story: memory usage had spiked to 100%, the OOM killer had terminated processes, and everything fell apart from there.
But here’s the thing. I couldn’t figure out why the memory spiked. CloudWatch showed me CPU, network, disk I/O. It did not show me memory usage. Turns out, EC2 basic monitoring doesn’t include memory metrics by default. I had been flying blind without realizing it.
What made it worse: I had no easy way to search through logs. On my old VPS, I would just SSH in and check /var/log/nginx/access.log, or tail -f the Laravel log to see errors in real time. PHP errors, application logs, access logs. All right there, easy to grep.
On AWS with Docker containers, the situation was different. I had to connect via SSM to the EC2 instance, then run docker logs for each container. The output was hard to read, everything mixed together without clear separation. I honestly did not explore this much before the outage because it felt cumbersome compared to what I was used to.
Here is something worth reflecting on: my old VPS, a simpler setup managed through GoDaddy, never had memory issues. It just worked. But it also did not have the Route 53 health check that ended up causing this problem. The added infrastructure brought new monitoring capabilities, but also introduced failure modes that did not exist before.
Finding the Root Cause
After some digging through dmesg, I found a likely trigger: a Route 53 health check I had configured was hitting the homepage of my WordPress site every 30 seconds. The homepage was 154KB of PHP rendering. Multiply that by the frequency, add real traffic from users in different time zones, plus the constant bot attempts we get on WordPress sites (wp-login attacks that we’ve been fighting with Cloudflare rules), and suddenly, to nobody’s surprise, my poor little t3.small with 2GB of RAM was struggling.
The fix for the health check was embarrassingly simple: change the path from / to /health. A 7-byte static response instead of a full page render.
Was this the only cause? Probably not. Was it a significant contributor? Almost certainly. The combination of health checks, bot traffic, and real users hitting at the same time created enough pressure to trigger the OOM killer.
But this raised a bigger question: why didn’t I catch this earlier? I had metrics, I had alerts, but I didn’t have visibility into what was actually happening.
What I Decided to Build
I wanted two things:
- Memory and disk metrics: Know when resources are running low before everything crashes
- Centralized logs: Search through all my container logs from one place, like the old days but better
I considered a few approaches. I could install Prometheus and Grafana on another EC2 instance, a proper observability stack. But that felt like overkill for eight small sites, and it would add another $20-30/month to my costs.
Then I discovered that Grafana Cloud has a generous free tier. 50GB of logs, 10,000 metrics series, forever free. And I already had CloudWatch collecting some metrics. I just needed to make them visible.
The architecture I landed on:
- CloudWatch Agent on EC2 to collect memory, disk, and swap metrics
- Grafana Alloy (a lightweight log shipper) to send Docker logs to Grafana Cloud
- Grafana Cloud as the dashboard, reading both logs and CloudWatch metrics
Total additional cost: about $2/month for the custom CloudWatch metrics.
Setting It Up
I added everything through Ansible, keeping it consistent with how I manage the rest of the infrastructure.
The CloudWatch Agent was straightforward. Install the RPM, drop a JSON config file, start the service. The config tells it what metrics to collect:
{
"metrics": {
"namespace": "SharedHosting",
"metrics_collected": {
"mem": {
"measurement": ["mem_used_percent"]
},
"disk": {
"measurement": ["disk_used_percent"],
"resources": ["/", "/mnt/ebs"]
}
}
}
}
For Alloy, I created a template that discovers all running Docker containers and ships their logs to Grafana Cloud. Big shout to my intern assistant Claude for this one, hehe. It extracts labels like the site name and service type, so I can filter logs by site in the dashboard.
The Grafana credentials go in environment variables. I keep all configuration values out of the repository, not just tokens but also usernames and URLs. This way the code stays generic and shareable without exposing anything specific to my setup.
Connecting the Pieces
To let Grafana Cloud read CloudWatch metrics, I needed an IAM user with read-only access. This felt a bit odd at first, giving an external service credentials to read my AWS data. But that’s how AWS APIs work: credentials authenticate regardless of where the request comes from. As long as the permissions are scoped correctly (CloudWatchReadOnlyAccess, nothing more), the risk is minimal.
Once connected, I could build dashboards in Grafana that pull from both sources: CloudWatch for metrics, Loki for logs. Same interface, everything in one place.
A Few Bonus Fixes
While investigating the outage, I found two other issues:
WordPress cron jobs weren’t running. Cloudflare’s bot protection was blocking requests to wp-cron.php. I added a scheduler container that runs wp cron event run directly via WP-CLI, bypassing HTTP entirely.
No swap memory. When memory pressure hit, there was no buffer. The OOM killer went straight to terminating processes. I added a 2GB swap file as a safety net. Not a solution, but it buys time during spikes.
What I Learned
The obvious lesson is that monitoring isn’t optional. But the more useful lesson is about choosing the right level of complexity.
I could have avoided this entire problem by using a managed service. Elastic Beanstalk, Lightsail, something that handles monitoring for you. But I would have learned less, and I wouldn’t have the flexibility I have now.
I could have gone deeper. Prometheus with custom exporters, distributed tracing, the full observability stack. But that would have been complexity without justification for my scale.
What I built sits in the middle: enough visibility to debug problems and catch issues early, without requiring a dedicated infrastructure team to maintain.
The infrastructure repository is public if you want to see the details: github.com/raulprdev/aws-shared-hosting
Next time something breaks at 3 AM, I’ll at least know where to look.