API Rate Limits: How Should Your Integration Handle a Backlog?
Start with the short answer ↓
An API limit is a workflow constraint, not just an error to retry. Scope a shared request budget, prioritize permitted work, show how long records have waited and agree what happens when a business deadline cannot be met.
Prepared with AI assistance. These are practical scoping recommendations; examples are illustrative, not client results.
Start with the delayed business task
An overnight import is still running when the team starts accepting today's customer requests. Both use the same external API, and fresh work begins waiting behind historical records. Adding another worker may increase contention rather than finish the urgent work sooner. This is an illustrative planning scenario, not a customer incident or a claim about a delivered AgenticShip system.
RFC 6585 defines HTTP 429 as a rate-limit response and permits a Retry-After header. It does not prescribe one universal counting method. Confirm the actual provider's scope: which requests share the limit, which account or application it applies to, and whether separate limits govern concurrent work. Do not copy a number from an unrelated product plan.
Budget requests, not just customer records
A record may need a lookup, a write and a confirmation. Count the actual operations for the proposed workflow, including pagination and the provider's charging rules. Ask which other applications use the same allowance. A budget that ignores them is an assumption, not a capacity guarantee.
Consider a deliberately simplified example: an agreed shared budget of 60 requests per minute, with 12 new customer handoffs per minute requiring two calls each. That uses 24 requests and leaves at most 36 for background work. A backlog of 180 single-call operations would take at least five minutes at that steady spare rate. These are invented planning inputs, not a provider quota, measured throughput or service commitment; pauses, retries and other clients can extend the wait.
If fresh handoffs rise to 30 per minute at two calls each, they consume the entire example budget. The background queue cannot catch up under those assumptions. Make that trade-off visible before promising real-time synchronization. Use measured representative request counts to replace the example during discovery.
Choose queue priorities without bypassing the provider
The following is our proposed scoping worksheet, not an API standard. Different work classes may have separate queues, but requests sharing a provider limit still need coordinated control. Priority determines which eligible request runs next; it does not override a provider-imposed wait or grant extra capacity.
| Work class | Proposed scheduling rule | What the team must see |
|---|---|---|
| Today's customer handoff | Prefer eligible current work within the shared budget; preserve required order within the same case. | Original request time, pending destination step and deadline risk. |
| Scheduled internal summary | Allow a bounded delay agreed with the report owner. | The last completed data-through time, not a fresh-looking success badge. |
| Historical import | Use agreed spare capacity and a saved checkpoint; pause when current work needs it. | Completed range, remaining work and the next permitted attempt. |
Make retry ownership explicit
Microsoft Graph advises honoring Retry-After when throttled and using exponential backoff when that header is absent. Its guidance also cautions against repeated full scans where supported change tracking or notifications are available. Those recommendations describe that platform; verify the rules for your actual endpoint.
Decide which layer schedules retries so an SDK, worker and operator do not all repeat the same action independently. Retain the original business reference and check whether the destination already acted when the outcome is uncertain. Set a bounded automatic-retry policy and escalation owner appropriate to the workflow. Do not create extra accounts or rotate keys to evade a provider's restrictions.
Show age and completeness, not only queue size
Our proposed minimum operator view includes the oldest waiting item, its business deadline, the last completed checkpoint and the reason work is paused. A small queue containing an overdue dispatch can matter more than a large queue of historical notes. Keep diagnostic references useful without copying unnecessary customer content into logs.
Name who decides when to pause an import, notify internal staff or use an approved manual fallback. A task waiting for capacity is not complete. If staff finish it manually, reconcile that result before the queue resumes, so recovery does not create another assignment. A claimed catch-up time should state its assumptions and change when those assumptions no longer hold.
Test the busy period before promising a response time
Use an isolated environment or simulated provider with real sending and consequential writes disabled. Test a burst of new work during an import, a throttle response with a wait instruction, repeated throttling, missing retry timing, a worker restart and manual completion of a queued task. Verify that the saved checkpoint, shared budget and action identity survive each case.
Acceptance should demonstrate useful current work, understandable delays and recovery without duplicate business effects. Record actual waiting time and the oldest outstanding task; a connection-success message is not enough. If the tested capacity cannot meet the agreed deadline, change scope or scheduling before discussing any separately approved account upgrade. This guide proposes a design exercise, not a ready-made connector or a guaranteed throughput figure.
Primary references
The guidance above is AgenticShip's proposed approach. These references provide technical background for the relevant recommendations.