Cache Stampedes: Test Request Coalescing at Expiry

A cache stampede occurs when many requests reach the origin at once because a useful cached result is absent or has expired. Request coalescing can let one operation fill the cache while others wait or follow a reviewed stale-response policy. Test that behavior at the key and failure boundaries before relying on it.

NGINX's proxy-cache lock documentation describes one implementation, including wait limits and conditions that can allow another request through. The design questions below also apply to application-level single-flight mechanisms, but exact semantics depend on the software and version.

Reproduce one hot-key expiry

Start with a harmless test endpoint whose origin work is observable. Choose one cacheable key and confirm that repeated requests hit the cache after the initial fill. Then let that entry expire and send a bounded concurrent burst in an authorized test environment.

Measure origin executions, not only client request count. If 100 clients arrive and the origin performs 100 identical expensive operations, the cache did not protect that expiry boundary. If it performs one operation, inspect how the other clients waited and what they received.

Keep the burst small enough not to affect unrelated services. This is a controlled reliability test, not permission to load-test a public system without authorization. Use synthetic data and an explicit request-rate ceiling.

Verify the cache key before sharing work

Coalescing is safe only when requests truly need the same result. Review tenant, authorization, language, query, and representation boundaries. A mechanism that reduces origin work by sharing a personalized response with another user has introduced a data-isolation failure.

Write two fixtures that must share and two that must not. For example, two anonymous requests for the same public document may share a fill, while two tenant-specific dashboards must remain separated. State the expected outcome before measuring performance.

Do not repair a low cache-hit rate by removing a key dimension without understanding why it exists. Key correctness comes before a more attractive origin-request graph.

Measure waiting as a user-facing cost

Coalescing moves work from the origin into a queue of waiting requests. Record the fill duration, client wait distribution, and deadline failures. Reducing origin executions is valuable, but it does not mean every user receives a fast response.

For an invented scenario, a one-second origin fill may serve a small burst comfortably, while a 20-second fill exceeds the product's interactive deadline. The appropriate wait policy depends on the application; these are teaching durations, not recommended defaults.

Check what happens when the wait limit expires. Some implementations allow waiting requests to reach the origin, and the resulting burst can still be large. Read the deployed implementation's current behavior rather than assuming a lock means exactly one origin request under all conditions.

Make stale-response policy explicit

Serving an older public article during refresh may be acceptable. Serving an outdated permission decision or account balance can be a different risk. Classify content before enabling stale responses broadly.

State the maximum acceptable staleness and which error conditions permit it. Avoid an indefinite stale fallback that hides a broken refresh path for days. Track the age of returned data where that matters to users or operators.

Our HTTP caching guide covers cache directives and validators. Coalescing addresses a separate concurrency problem: even a correct freshness policy can create a burst when many clients encounter the same miss.

Test fill failure and owner loss

Make the test origin return a controlled error or exceed a bounded deadline. Observe whether waiting requests receive an error, retry independently, use stale content, or remain stuck. A failed fill must not leave a permanent lock that prevents recovery.

If the lock owner exits or the cache service restarts, test how another worker acquires responsibility. Distributed coordination can introduce lease and timing behavior that a single-process test does not expose.

Avoid caching a transient error as a long-lived successful result. Review the status and eligibility rules separately from lock handling. The fast delivery of a wrong cached state is not a successful resilience feature.

Compare multiple keys and realistic load

After the single-key test, use several hot keys and a normal spread of traffic. A per-key mechanism may work correctly while a shared worker pool or connection limit becomes the next bottleneck. Watch both the origin and the coalescing layer.

Consider staggered refresh or expiry variation where appropriate, but do not use randomness to hide an inadequate origin budget. Document how the selected mechanism changes expected work at expiry and what happens when the budget is exceeded.

Use JSON Diff to compare redacted test configurations and verify that the lock or stale-policy setting was the intended change. It cannot simulate concurrent load or validate tenant isolation.

Conclusion

Test cache coalescing as a concurrency and correctness feature. Verify key isolation, count origin executions at expiry, measure client waiting, and exercise failed fills. A useful cache protects the origin without sharing the wrong data or hiding an unrecoverable refresh failure.