DevOps · Mid-level

Observability: logs, metrics, and traces (the 3 pillars)

When a system gets slow at 3 a.m., the question isn't just "is it up?" but "why is it like this?". Monitoring answers the first; observability answers the second. In distributed systems, where a single request goes through dozens of services, knowing how to investigate the unknown is what separates firefighting from understanding the system.

Monitoring vs observability

The distinction is subtle and comes up a lot in interviews. Monitoring is watching for things you already know can go wrong: CPU above 90%, a full disk, a service that is down. You know when it broke. Observability is the ability to ask new questions about the system without having to redeploy it — to find out why it broke, even in failures you never anticipated.

Monitoring answers known questions; observability lets you answer questions you didn't know you'd ask. In complex systems, most problems are of the second kind.

The three pillars

Observability rests on three types of data, each answering a different class of question:

PillarWhat it isAnswers
LogsDiscrete, detailed eventsWhat exactly happened here?
MetricsNumbers aggregated over timeHow much? How often?
TracesThe path of a request across servicesWhere in the journey is the bottleneck?

Structured logs

Logs are the detailed record of events. Modern best practice is to use structured logs (usually JSON), because machines can filter and aggregate fields instead of searching loose text.

{"ts":"2026-06-25T03:14:00Z","level":"ERROR","service":"checkout",
 "user_id":"u-991","msg":"pagamento recusado","latency_ms":820}

Metrics: aggregated numbers

Metrics are numbers measured over time — requests per second, latency, memory usage. Because they're aggregated, they're cheap to store and great for charts and alerts. Two frameworks help you choose what to measure:

  • RED (for services): Rate (requests/s), Errors (error rate), Duration (latency).
  • USE (for resources): Utilization, Saturation, Errors.

Traces: the request crossing services

In a microservices system, a user request may go through an API, authentication, a database and a cache. A trace stitches that whole journey together and shows how long each step took. It's the right tool when "the system is slow" and you need to find out where.

requisição [#abc] 820ms
 ├─ api-gateway        12ms
 ├─ auth-service       40ms
 └─ checkout-service  760ms  ← gargalo
     └─ db-query       740ms

Alerts and SLOs

There's no point collecting data if nobody is notified when something matters. An alert fires when a metric crosses a threshold. To avoid alerting on every little thing, you define an SLO (Service Level Objective) — a target such as "99.9% of requests respond in under 300ms". You alert when you're about to blow the SLO, not at every hiccup. This data also feeds scaling decisions, such as autoscaling in Kubernetes.

Conclusion

Observability is what lets you understand systems you can't fully predict. Logs tell you what happened, metrics tell you how much, traces tell you where — and together they turn a confusing incident into an objective investigation. Knowing which pillar to use in each situation is a sign of technical maturity in a mid-level DevOps engineer.

Read also

Back to track