Skip to content

Observability

The three pillars aren't three tools to enable and forget — they're three different questions, and the skill this page is actually teaching is knowing which one to reach for first. Metrics get you to "something broke," logs get you to "here's why," and traces get you to "here's which of the five services in the chain is the one that's slow." CloudWatch answers the first two; X-Ray answers the third.

Scope: the AWS module page named the three observability pillars — metrics, logs, traces — and split them across CloudWatch/CloudTrail/Config at a glance. This page goes one layer deeper: what each pillar is actually for and what question it answers, how CloudWatch implements metrics and logs concretely (alarms, dashboards, Logs Insights), and how X-Ray stitches traces across services into one request picture.

TL;DR — in 30 seconds:

  • Metrics tell you that something's wrong (aggregate, cheap to alert on), logs tell you why (a discrete event at a point in time), traces tell you where the time went across services — reach for them in that order.
  • CloudWatch covers metrics and logs (alarms, dashboards, Logs Insights, metric filters); X-Ray covers traces (segments/subsegments stitched into a service map) — CloudTrail/Config answer "who changed what," not "is the system healthy."
  • A dashboard is not a substitute for an alarm — nobody watches a dashboard 24/7; an alarm is what actually turns a crossed threshold into a page or an action.

1. Three pillars, three different questions

Each pillar exists because it answers a question the other two can't:

  • Metrics — is something wrong, in aggregate, right now? A metric is a number over time (CPU utilization, request count, p99 latency). Cheap to store, cheap to alert on, but tells you that something is wrong, not why.
  • Logs — what exactly happened, in detail, at a point in time? A log is a discrete event — a line written when a request failed, a deploy ran, or an error was thrown. Logs answer "why" once metrics have told you "when" and "where" to look.
  • Traces — where did the time go, across services, for one request? A trace follows a single request as it hops through multiple services, timing each hop. Traces answer "which service in the chain is actually slow" — the question neither metrics nor logs, taken alone, can answer once a request crosses more than one service boundary.
flowchart TB
    Request["One request<br/>enters the system"] --> Metrics["Metrics<br/>(is anything wrong?)"]
    Request --> Logs["Logs<br/>(what exactly happened?)"]
    Request --> Traces["Traces<br/>(where did the time go?)"]
    Metrics -->|alarm fires| Logs
    Logs -->|error spans multiple services| Traces

The usual investigation order follows the diagram left to right: an alarm on a metric tells you something broke; you go to logs for the specific error; if the error only shows up when a request crosses several services, you go to a trace to see which hop actually ate the latency or threw.

2. CloudWatch — metrics, alarms, dashboards

CloudWatch is AWS's native metrics-and-logs service, and it covers the first two pillars:

  • Metrics are namespaced time series (AWS/EC2, AWS/RDS, or a custom namespace you publish to). Most AWS services publish their own metrics automatically; anything you want beyond that (a business metric, an application counter) you publish yourself.
  • Alarms watch a metric against a threshold over an evaluation window and change state (OK → ALARM → INSUFFICIENT_DATA). An alarm's job is to turn "a number crossed a line" into an action — paging someone, or triggering Auto Scaling.
  • Dashboards are just saved arrangements of metric graphs — useful for a human glancing at aggregate health, not a substitute for alarms (nobody watches a dashboard 24/7; alarms are what actually notify).

CloudWatch Logs is the logs half: applications and services write log streams into log groups (one group per service is typical). Two features make raw log volume actually usable:

Feature What it's for
Logs Insights An ad-hoc query language over log groups — filter, parse, and aggregate log data without exporting it elsewhere.
Metric filters Turn a pattern match in a log stream (e.g. the string "ERROR") into a CloudWatch metric you can alarm on — this is how "an error appeared in the logs" becomes a page.
Retention Per-log-group retention (days to indefinite) — logs are not free to keep forever, so retention is a cost decision, not just a compliance one.

3. X-Ray — following one request across services

Once a request crosses more than one service — an API Gateway front door, a Lambda function, a downstream database call — metrics and logs on any single service only show that service's slice. X-Ray stitches those slices into one trace:

flowchart LR
    Client --> APIGW["API Gateway<br/>(segment)"]
    APIGW --> Lambda["Lambda<br/>(segment)"]
    Lambda --> DB["DynamoDB call<br/>(subsegment)"]
    Lambda --> Downstream["Downstream API<br/>(subsegment)"]
  • A segment is one service's contribution to a trace (its start time, duration, and any error); a subsegment breaks that further into individual downstream calls the service made.
  • The service map is X-Ray's visual product: every service that participated in recent traces, drawn as nodes, with average latency and error rate per edge — the fastest way to see which hop in a chain is the slow or failing one, without reading logs service-by-service.
  • Sampling keeps tracing cheap at scale: X-Ray doesn't have to trace every single request (typically a fixed rate plus a reservoir for rare requests) to give a statistically useful latency and error picture. (Go deeper: the X-Ray sampling rules guide — optional, not required to defend this page.)

4. How this connects

  • This extends the AWS module page's Observability theme (three pillars; the CloudWatch/CloudTrail/Config split) with what CloudWatch's metrics-vs-logs split looks like mechanically (alarms, dashboards, Logs Insights, metric filters) and what a trace actually is (segments, subsegments, the service map, sampling).
  • CloudTrail and Config stay outside this page's scope on purpose — they answer "who changed what" and "did config drift from policy," not "is the running system healthy," which is what pillars this page covers actually measure.
  • The Prove gate's observability question — which pillar answers which kind of question, and how you'd localize a slow multi-service request — draws directly on §1's question-per-pillar framing and §3's service-map explanation.

Common gotchas

  • Treating a CloudWatch dashboard as monitoring. Fix: a dashboard is a saved arrangement of graphs for a human to glance at — it notifies nobody; an alarm (threshold + evaluation window) is what actually pages someone or triggers Auto Scaling.
  • Jumping to X-Ray before checking logs for a single-service error. Fix: traces answer "which hop in a multi-service chain is slow" — for an error confined to one service, logs (via Logs Insights) already have the detail; save tracing for once the problem crosses a service boundary.
  • Assuming every AWS metric you need is published automatically. Fix: most AWS services publish their own metrics, but anything beyond that — a business metric, an application counter — you have to publish yourself.
  • Keeping CloudWatch Logs retention at "indefinite" without deciding to. Fix: retention is per-log-group and logs aren't free to keep forever — treat it as a deliberate cost decision, not the default left unset.
  • Expecting X-Ray to trace every single request at scale. Fix: sampling (a fixed rate plus a reservoir for rare requests) keeps tracing affordable — it's a statistically useful picture, not a complete log of every request's trace.
Check yourself — an alarm fires on elevated error rate for a request that touches three services. What's your investigation order, and why?

Alarm → logs → trace. The alarm tells you that something's wrong; logs on the likely service tell you why (the specific error); if the error only shows up once the request has hopped across the three services, a trace shows which hop actually ate the latency or threw — going straight to a trace before checking logs skips the cheaper, faster answer for a single-service problem.

Check yourself — what's the difference between a CloudWatch metric filter and a metric alarm?

A metric filter turns a pattern match inside a log stream (e.g. the string "ERROR") into a CloudWatch metric in the first place; an alarm then watches that (or any) metric against a threshold over an evaluation window and changes state to trigger an action. The filter creates the signal; the alarm reacts to it.

You can defend this when you can say, for any observability question, which of the three pillars answers it and why the other two don't; explain the alarm → logs → trace investigation order and when you'd skip straight to a trace; and describe what a CloudWatch alarm actually does (metric vs threshold vs evaluation window) versus what a dashboard does.