We had a scheduler that worked beautifully in the demo and fell apart in production. Under bursty load, jobs would vanish — accepted by the API, never executed, no error. The kind of bug that erodes trust faster than an outage.
This is the story of finding the real constraint, and why the fix was to make the system slower on purpose.
The symptom vs. the problem
The instinct was "add workers." We did. It didn't help — it just moved the cliff a little further out. That's the tell that you're treating a symptom.
The actual problem: producers could enqueue faster than workers could drain, the in-memory buffer filled, and the oldest jobs were evicted to make room. Capacity wasn't the constraint. The absence of a feedback signal was.
The principle
A queue without backpressure isn't a buffer — it's a place where work goes to be quietly lost. Bounded queues force the question: what happens when it's full?
The shape of the fix
Instead of an unbounded queue, we used a credit-based scheme: workers hand
out credits as they finish, and producers may only enqueue when they hold a
credit. When the system is saturated, producers block (or get a fast 429)
rather than overflowing a buffer.
The core of the admission check is unglamorous — which is the point:
// Acquire blocks until a credit is available or the context is cancelled.
func (q *Queue) Enqueue(ctx context.Context, job Job) error {
select {
case <-ctx.Done():
return ctx.Err()
case q.credits <- struct{}{}: // take a credit (buffered channel)
q.jobs <- job
return nil
}
}
Did it work?
Before, dropped jobs climbed with offered load and tail latency lied to us by staying flat (because dropping is "fast"). After, drops went to zero and latency told the truth — it rose under load, which is exactly what you want a healthy system to do.
Outcome
Zero dropped jobs through 16k RPS, and — counterintuitively — happier users, because a clear "retry in a moment" beats a silent disappearance every time.
What I took away
- Match the symptom to the real constraint before spending. More workers was money spent on the wrong axis.
- Bounded everything. An unbounded queue is a deferred incident.
- Honest latency is a feature. A system that slows down under load is telling you the truth; one that stays fast while losing work is lying.
The whole change was a few dozen lines. Most of the work was understanding the problem well enough to make it that small.