Deciding what belongs in the request and what belongs in a job queue

An endpoint that generates a PDF report from a user’s data started its life doing exactly what its name suggests: the request comes in, the handler queries the data, builds the PDF, and returns it, all inside one HTTP request. For a while that was the right amount of engineering. Reports were small, generation took a few hundred milliseconds, and adding a queue for that would have been solving a problem we didn’t have yet.

Where the synchronous version quietly stopped being fine

As accounts grew, so did the reports. A report covering a year of data for a large account could take twenty, sometimes thirty seconds to generate. The endpoint still worked, technically, the server finished the request and returned a valid PDF, it just took a while. The problem showed up on the client side first: a thirty-second generation time was longer than the frontend’s own request timeout, so the browser gave up and showed an error, while the server, with no idea the client had stopped listening, kept right on generating the report and eventually finished it into a response nobody was there to receive.

The part that actually caused damage wasn’t the wasted work, it was what the user did next. Seeing an error, they clicked the button again. The retry hit the same endpoint, kicked off a second full report generation for the same account while the first one was still running, and for the accounts large enough to trigger this in the first place, two of those jobs running concurrently was enough to pin the database and slow down unrelated requests from other customers sharing the same instance.

The fix: get the client out of the timeout’s way

The change wasn’t to make report generation faster, that’s a real project on its own and a separate one. The fix was to stop pretending report generation belongs inside a request/response cycle at all.

func handleGenerateReport(w http.ResponseWriter, r *http.Request) {
    jobID := enqueueReportJob(r.Context(), accountIDFrom(r))
    w.WriteHeader(http.StatusAccepted)
    json.NewEncoder(w).Encode(map[string]string{"job_id": jobID})
}

func handleJobStatus(w http.ResponseWriter, r *http.Request) {
    status, url := lookupJob(r.Context(), jobIDFrom(r))
    json.NewEncoder(w).Encode(map[string]any{"status": status, "download_url": url})
}

The handler now does almost no work itself. It enqueues a job and returns immediately with a job id, well inside any client timeout, because the response no longer depends on how long the report actually takes. The frontend polls the status endpoint every couple of seconds, or in a version we added later, gets notified over a websocket the app already had open for other reasons. Either way, the report generation itself moved to a worker process pulling jobs off a queue, with no HTTP timeout of any kind hanging over it, and a lock keyed on account id so two jobs for the same account can’t run at once regardless of how many times the client retries.

Where the line actually is

The rule of thumb I settled on isn’t about a specific millisecond cutoff, it’s about what the duration depends on. If an operation reliably finishes in some tight, bounded window regardless of input size, a user profile lookup, a login check, a small write, it belongs in the request. The moment an operation’s duration scales with the size of user-controlled input, or depends on an external call whose latency you don’t control, it’s a candidate for a job queue, because nothing about a request/response cycle was designed to tolerate an open-ended wait.

Why this is more than a latency problem

The tempting way to frame this is “make slow things async so the user doesn’t wait,” which is true but understates what actually broke. HTTP’s request/response model has a built-in assumption baked into how every client behaves: if the request times out, the client treats that as “nothing happened” and it’s safe to retry. That assumption is only true if the server actually stopped, or if retrying is safe to run concurrently with whatever’s still in flight. A long synchronous handler violates that assumption silently, the server keeps working long after the client has given up and decided to try again, and nothing in the stack tells either side that the other one’s assumption just broke. Moving the work to a queue with idempotency and per-account locking isn’t a performance optimization at that point. It’s fixing a correctness bug that a slow endpoint had been quietly running the entire time.