Structured logging: what to log and what to skip

A few years ago I inherited a service where every log line was a console.log with whatever string felt useful at the time. Some lines had a user id. Some had an order id but called it oid. Some were just "here" left behind from debugging. When something broke in production, finding the right log meant grepping for a phrase and hoping it was still unique enough to matter. That service taught me most of what I know about structured logging, mainly by being a case study in what not to do.

Structured logging just means every log line is a machine-parseable record, usually JSON, with consistent field names, instead of a free-text sentence. That sounds like a small change but it’s the difference between “grep and pray” and being able to ask your logs a real question, like “show me every failed checkout for tenant X in the last hour, sorted by latency.”

What actually belongs in a log line

Not everything you can log is worth logging. The fields that consistently pay for themselves:

• Request id. Generated once at the edge (load balancer or API gateway) and threaded through every log line the request touches. Without this, correlating “what happened during this one request” across ten log lines is guesswork.
• User or tenant id. Lets you answer “what did this specific customer experience” without reconstructing it from timestamps.
• Latency. How long the operation took, in milliseconds, measured at the point where you have the most context, not just at the outermost handler.
• Error code or type. A stable, greppable identifier for what went wrong, not just the exception message, which can vary run to run.
• Outcome. Success, failure, retried, whatever the meaningful states are for that operation.

Everything else is a judgment call based on what you’ll actually need to debug, not what seems interesting in the moment.

What to leave out

The instinct when you first adopt structured logging is to log everything, since it’s now “free” in a way unstructured logging wasn’t. That instinct is wrong for two reasons.

The first is cost. Log ingestion and storage is billed by volume almost everywhere, and a service logging every field of every request at high traffic turns into a real line item. The second, more important reason is signal. If every request produces twenty fields of mostly-identical noise, the two fields that actually mattered when something went wrong get buried. I’ve sat in incident calls where someone said “it’s in the logs somewhere” and then spent fifteen minutes scrolling past request bodies nobody needed to find the one line with the actual error.

The other thing that has no business in a log line is anything secret or personal: passwords, API keys, auth tokens, full credit card numbers, unredacted emails or phone numbers where you don’t need them. It’s easy to log an entire request object for convenience and forget it contains an Authorization header. That header ends up in your log aggregator, readable by anyone with log access, retained for however long your retention policy says, and now you have a compliance problem instead of a debugging convenience. Log the presence of a field, or a hash of it, or the last four digits, not the field itself.

Log levels, used honestly

Log levels only work if they mean something. On the service I inherited, everything was logged at the same level because nobody had bothered to differentiate, which meant the “error” filter was useless, it returned thousands of lines a minute of things that weren’t actually errors.

The way I’ve settled on using levels:

• debug for detail you want during local development or active investigation, never enabled in production by default.
• info for normal operations worth a record: a request completed, a job ran, a scheduled task finished. This is your default level in production.
• warn for something unexpected that the system recovered from on its own: a retry succeeded, a fallback was used, a cache miss that’s slower but not broken.
• error for something that failed and needs a human to look at it eventually, even if it didn’t page anyone right now.

If error only fires for things a human should actually care about, an alert on “error rate spiked” is meaningful. If it fires for routine stuff, the team learns to ignore it, and that’s how real errors get missed.

Correlating logs across services

Once you have more than one service, request ids need to travel with the request, not get regenerated at every hop. The pattern that works is to generate a trace or request id at the edge, put it in a header, and have every service downstream read that header instead of minting its own id. Here’s roughly what that looks like in Go with a structured logger:

func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        reqID := r.Header.Get("X-Request-ID")
        if reqID == "" {
            reqID = uuid.NewString()
        }
        start := time.Now()

        logger.Info("request started",
            "request_id", reqID,
            "method", r.Method,
            "path", r.URL.Path,
        )

        next.ServeHTTP(w, r)

        logger.Info("request completed",
            "request_id", reqID,
            "path", r.URL.Path,
            "latency_ms", time.Since(start).Milliseconds(),
            "status", statusFromContext(r.Context()),
        )
    })
}

That produces something like this on the wire:

{"level":"info","msg":"request completed","request_id":"a1b2c3d4","path":"/api/orders","latency_ms":42,"status":200,"tenant_id":"acme-corp"}

Every downstream service call needs to forward that same X-Request-ID header, and every log line those services emit needs to include it. Once that’s in place, an incident stops being “which of these six services broke” and becomes “search for this one id across all of them and read the timeline.”

The actual payoff

None of this matters much when everything is working. It matters at two in the morning when something is on fire and you have five minutes to figure out whether it’s your database, a downstream dependency, or a bad deploy. The service that logs the right ten fields, consistently, with an id that ties a request together across every hop it takes, gets you an answer in one query. The service that logs everything, inconsistently, with no shared id, gets you a very long night of grepping and guessing. I’ve worked in both, and I know which one I’d rather be on call for.