Message queues without the buzzwords: when a plain table beats Kafka

A teammate wanted to send a welcome email after signup without blocking the request on an SMTP call, and opened a ticket titled “set up Kafka.” That’s a real overreaction, but it’s a common one. Sending an email in the background isn’t a streaming problem, it’s a “do this thing a little bit later, and don’t lose it” problem, and Postgres, which we already run and already back up, handles that fine on its own.

The table is the whole queue

A job queue doesn’t need a broker, it needs a place to put work, a way to grab one item without two workers grabbing the same one, and a record of whether it succeeded. That’s a table and one query:

CREATE TABLE jobs (
    id BIGSERIAL PRIMARY KEY,
    kind TEXT NOT NULL,
    payload JSONB NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    attempts SMALLINT NOT NULL DEFAULT 0,
    run_after TIMESTAMPTZ NOT NULL DEFAULT now(),
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX jobs_pending_idx ON jobs (run_after) WHERE status = 'pending';

status moves through pending, done, or failed. run_after is what makes retries and delayed jobs the same mechanism, a job due now has run_after <= now(), a job waiting for a retry backoff just has it set further out.

The part that actually replaces a broker is the claim query:

UPDATE jobs
SET status = 'processing', attempts = attempts + 1
WHERE id = (
    SELECT id FROM jobs
    WHERE status = 'pending' AND run_after <= now()
    ORDER BY id
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
RETURNING id, kind, payload;

FOR UPDATE locks the row a worker is about to claim. SKIP LOCKED is the piece people miss, it tells Postgres that if a row is already locked by another worker’s in-flight transaction, skip it and look at the next one instead of blocking behind it. Without that clause, ten workers polling the same table would queue up behind each other on every claim. With it, they each grab a different row and none of them wait on the others. That’s the entire concurrency story, no separate coordination service, just a row lock Postgres was already going to give you.

Retries and dead letters are just columns

A real queue’s headline features turn out to be columns you’d add anyway. A failed job gets attempts bumped and run_after pushed out by an exponential backoff, run_after = now() + (attempts * attempts) * interval '1 minute' is a fine starting point. Once attempts crosses a threshold, flip status to failed instead of pending, and that failed bucket is your dead-letter queue, a SELECT * FROM jobs WHERE status = 'failed' away from being inspected, not a separate topic you have to configure.

func processOne(ctx context.Context, db *sql.DB) error {
	var id int64
	var kind string
	var payload []byte

	err := db.QueryRowContext(ctx, claimQuery).Scan(&id, &kind, &payload)
	if err == sql.ErrNoRows {
		return nil // nothing due right now
	}
	if err != nil {
		return err
	}

	if err := handle(kind, payload); err != nil {
		_, _ = db.ExecContext(ctx,
			`UPDATE jobs SET status = 'pending',
			 run_after = now() + (attempts * attempts) * interval '1 minute'
			 WHERE id = $1`, id)
		return err
	}

	_, err = db.ExecContext(ctx, `UPDATE jobs SET status = 'done' WHERE id = $1`, id)
	return err
}

Workers are just a handful of goroutines calling this in a loop with a short sleep between empty polls. No consumer groups, no partition assignment, no separate service to keep alive, it lives inside the same application already talking to the same database.

The failure mode you have to design for either way

None of this is free of the hard part queues are actually known for, which is that “at least once” delivery is the realistic guarantee, not “exactly once.” A worker can crash after handle() succeeds but before the UPDATE ... SET status = 'done' commits, and that job gets picked up again by someone else. Kafka and RabbitMQ have this exact same failure mode, a consumer that crashes after processing but before acking gets redelivered the message. The fix isn’t a fancier broker, it’s the same one either way: handlers need to be idempotent, the same reasoning from writing idempotent API endpoints applies directly here, send the welcome email keyed on a unique constraint so a redelivery is a harmless no-op instead of a duplicate.

Where a real broker earns its place

I want to be straight about where this stops working, because it does stop working past a certain point. A polling loop against one table is fine for hundreds of jobs a minute, it starts fighting Postgres for the same disk and connections your actual application traffic needs once you’re pushing thousands per second, at which point a dedicated broker’s storage engine, built for exactly that append-and-stream workload, stops being a nice-to-have. Fan-out is the other real gap, if the same event needs to reach five independent consumers, each reading at their own pace, a table with one status column models one consumer well and gets awkward fast modeling five. And if you need strict ordering guarantees across partitions at real throughput, or replay of a full event history rather than a queue of pending work, that’s Kafka’s actual design center, not a workaround you’re avoiding.

The practical takeaway: reach for a message queue platform when you need what it structurally provides, real throughput past what your database’s connection pool tolerates, fan-out to multiple independent consumers, or ordered event replay. If what you actually need is “do this soon, retry if it fails, don’t lose it,” that’s a table and a SKIP LOCKED query on the Postgres instance you’re already running and already backing up, no new service, no new failure mode to learn at 2am.