Logs vs Metrics vs Traces: An Explainer for DevOps Beginners

Your phone buzzes at 2:14 a.m. A payment endpoint is failing for roughly one in five users. You open your laptop and face the question every on-call engineer faces: where do I look first?

The answer depends on what you already collect. Most teams gather three kinds of telemetry — metrics, logs, and traces — and each one answers a different question. Knowing which to reach for turns a two-hour hunt into a ten-minute fix.

The three pillars, in plain terms

Metrics: numbers over time

A metric is a measured value stored with a timestamp and a few labels. Request counts, error rates, queue depth, CPU usage, p99 latency. Metrics are cheap to store because they are aggregated, and they are fast to query because the data set stays small.

A metric tells you that something is wrong, and rarely why. When http_requests_total{status="500"} climbs, you know the shape of the problem — which service, which endpoint, how bad — but nothing about the individual requests that failed.

Logs: what happened to one request

A log is a discrete event with a timestamp. It might be a plain text line or a structured JSON object with fields like user_id, order_id, and duration_ms. Logs carry the detail metrics throw away.

The tradeoff is volume. Logs grow with traffic, and unstructured log lines are painful to query at scale. Structured logs with consistent field names are worth the extra ten minutes of setup.

Traces: the journey of a request

A trace follows one request across every service it touches. Each hop is a span with a start time, a duration, and a parent span, so you get a waterfall view of where the time went. If a checkout takes 2.4 seconds, the trace shows you that 1.9 seconds sat waiting on an inventory service call.

Traces are the only pillar that shows relationships between services. Metrics can tell you inventory is slow; a trace tells you the checkout service is slow because of inventory.

When to use each

• Alerting and SLOs: metrics. They are cheap, fast, and good at thresholds and rates.
• Capacity planning: metrics. Trends over weeks are a metrics problem.
• Debugging a specific failed request: logs. Filter by request ID or trace ID and read the story.
• "Which service is slow?" traces. Latency broken down by hop, sorted from slowest span down.
• Understanding a new code path: traces first, then logs for the details.
• Audit and compliance questions: logs, with a retention policy that actually matches the requirement.

A rough rule: use metrics to detect, traces to localise, and logs to explain.

How they work together during an incident

1. An alert fires on a metric. Error rate on the checkout service is above 5 percent for ten minutes. You confirm it on a dashboard and check when it started.
2. You narrow the scope with metrics. The dashboard splits by endpoint: only /pay is affected, and only in the EU region. That is already a much smaller haystack.
3. You pull a trace. Grab a trace ID from a failing request — or from a latency chart if you can jump straight from a slow bucket. The waterfall shows a retry loop against the payment provider, with three attempts at 800 ms each.
4. You read the logs for that trace. Filter the payment service logs by trace_id and you see connection timeout, plus a config line showing a five-second connect timeout on a client that was fine last week.
5. You fix and verify. Roll back the config change, watch the error-rate metric fall, and check a fresh trace to confirm the retries are gone.

Five steps, three pillars, each doing the job it is good at. Skipping the trace usually means grepping logs blindly. Skipping the metric usually means you never got alerted in the first place.

Correlation is the glue worth investing in

These three sources are only powerful together if you can move between them without guessing. A few habits make that possible:

• Emit logs as structured JSON and include the current trace_id and span_id in every line.
• Use the same service name everywhere — in metrics labels, log fields, and trace resource attributes. Inconsistent names quietly break every join.
• Adopt the OpenTelemetry semantic conventions rather than inventing your own field names. Your future self will thank you.
• Where your tooling supports it, use exemplars to link a metric data point to a sample trace.
• Give logs and traces a shared time source. Clock drift across hosts makes correlation unreliable.

Mistakes that make observability expensive and useless

Cardinality explosions in metrics. Putting user_id, request_id, or a raw URL path into a metric label multiplies time series until the storage bill and query latency both explode. High-cardinality detail belongs in logs and traces, not labels.

Logging everything at debug level in production. It is the fastest way to make logs unsearchable and expensive. Keep production at info, and reach for sampled debug logging or a temporary log-level change during an incident.

Aggressive head sampling on traces. If you sample the first 1 percent of requests, you will miss the rare failing path entirely. Tail sampling — deciding after the trace completes — keeps errors and slow traces while thinning out the boring ones.

Alerting on logs. Log-based alerts are noisy and slow compared to a metric. Alert on the symptom, then use logs to diagnose.

Treating retention as uniform. Hot storage for a week, cheaper storage for a month, and archives beyond that is usually plenty. Very few teams need seven years of debug logs on fast disks.

A practical starting point

You do not need all three pillars perfectly instrumented before you get value. Pick one critical user journey — login, checkout, or whatever earns the company money — and do this over the next week or two.

1. Define three metrics for each service on that path: request rate, error rate, and duration. Put them on one dashboard with the same time range.
2. Switch that path's logs to structured JSON and add a trace ID field, even if you are not tracing yet.
3. Enable tracing on two or three services and make sure the trace ID propagates in headers between them.
4. Write down, in your runbook, which pillar you check first for which symptom. "Errors on one endpoint: metric, then trace, then logs."

Then rehearse it. The next time something breaks, follow your own runbook and see where it slows you down. Observability improves fastest when you tune it against real incidents rather than imagined ones.

Photo: 3844328 / Pixabay