API Cursor Pagination: Avoid Missing Records in Changing Datasets

An export can finish successfully and still miss records. If new rows arrive while a client walks numbered pages, the positions can shift between requests. The client may see the same record twice or skip one without any obvious HTTP error.

Cursor pagination gives the client a continuation point rather than merely a page number. It can improve consistency for changing datasets, but the cursor alone doesn't create a snapshot. The server's ordering and change semantics still need to be explicit.

Distinguish offset from continuation

Offset pagination says, in effect, “skip this many rows and return the next group.” Cursor pagination says, “continue after this position in the defined order.” Both can be useful, but they have different behaviour when data changes.

For example, sorting by creation time alone can be ambiguous when two rows share a timestamp. Add a stable tie-breaker such as an immutable record identifier. Without a total order, the continuation position may not identify one clear boundary.

Don't describe cursors as automatically faster or perfectly consistent. Performance depends on query design and indexes; consistency depends on the server's contract. Decide what the export promises before choosing the interface.

Publish a clear response contract

A fictional response might be:

{
  "items": [{"id":"record-example-17","created_at":"2026-10-01T09:00:00Z"}],
  "next_cursor": "opaque-example-cursor",
  "has_more": true
}

The cursor is illustrative, not a real token. Treat real cursors as opaque unless the API documents their structure. The client shouldn't build a new cursor by editing decoded fields or increasing an internal identifier.

Use JSON Formatter to inspect synthetic responses. It checks readability and syntax, not whether has_more agrees with the dataset or whether the next cursor is valid. Those are server and integration-test responsibilities.

Follow the API's continuation mechanism

Some APIs return a cursor in the body; others provide a next-page link in headers. GitHub's pagination documentation illustrates following documented response links rather than guessing the next URL.

Use the method the particular endpoint supports. Not every endpoint uses cursor pagination, and a familiar platform can have more than one pagination style. Preserve relevant filters when following continuation and don't assume a page-size parameter behaves identically everywhere.

For synthetic query-component examples, URL Encoder can help inspect percent encoding. It isn't a pagination client. Don't paste private cursors, access tokens, or confidential filter values into a shared debugging example.

Define how concurrent changes are handled

Decide whether the export sees a frozen snapshot, a bounded time range, or a live evolving dataset. A cursor can keep order stable without preserving every record that existed at the start. Deleted records and changed sort keys need an explicit rule.

For a time-bounded export, record the upper boundary at the start and apply it to every page. Use a stable ordering key and a tie-breaker. If records can change the key during the export, consider a different key or a server-supported snapshot mechanism.

Document whether the client should deduplicate by immutable ID. Deduplication can remove repeated records, but it cannot recover a record that never arrived. Don't use it to hide an undefined server contract.

Define what happens when a record changes after the client has passed its position. Some exports need only creation-time consistency; others need a versioned snapshot of values. Write that distinction into the contract so consumers know whether later updates require a separate synchronisation pass. Include the snapshot or boundary identifier in the export's internal evidence record.

Bound the consumer's work

Set a maximum page count, total record count, or elapsed duration appropriate to the task. Detect a repeated cursor so a server bug doesn't create an infinite loop. Handle an empty page with a continuation according to the API's documentation rather than guessing.

Checkpoint progress when an export is long, including the original filters and snapshot boundary. A resumed client should not accidentally switch to a different dataset because only the cursor was saved. Keep authentication separate from the checkpoint where possible.

If a cursor expires, follow the API's documented restart or recovery approach. Don't silently pretend a partial export is complete. Report the verified record range and the unresolved continuation state to the caller.

Test change scenarios deliberately

In staging, insert a record between page requests, delete a record near the boundary, and create several records sharing the same timestamp. Confirm the documented result. Also test filter changes, invalid cursors, expired cursors, and a maximum-size page.

Compare synthetic page sequences with JSON Diff to locate repeated or missing IDs during review. The tool highlights structural differences; your test harness should enforce the actual completeness rules.

Test authorisation independently. A valid continuation token must not let one user access another user's records. For database-level enforcement, see PostgreSQL row-level security. A cursor is a navigation aid, not a grant of permission.

Conclusion

Reliable pagination needs more than a next token. Define stable ordering, continuation rules, concurrent-change behaviour, and bounded recovery. Follow the endpoint's documented mechanism and test the awkward cases so a completed export means what the product claims it means.