---

name: apify-rate-limits description: 'Handle Apify API rate limits with proper backoff and request queuing. Use when hitting 429 errors, optimizing API request throughput, or implementing rate-aware client wrappers. Trigger with "apify rate limit", "apify throttling", "apify 429", "apify retry", "apify backoff", "too many requests apify".' allowed-tools: Read, Write, Edit version: 1.5.0 license: MIT author: Jeremy Longshore jeremy@intentsolutions.io tags:

• saas
• scraping
• automation
• apify compatibility: Designed for Claude Code

---

Apify Rate Limits

Overview

The Apify API enforces rate limits per resource. The apify-client library auto-retries 429s (up to 8 times with exponential backoff), so most workloads never notice a limit. You reach for this skill when bulk operations, custom API calls, or large fan-outs push past what the built-in retry can absorb — you then batch, queue, stagger, and monitor to stay under the ceiling.

Full runnable code for every step is in implementation.md; combined scenarios are in examples.md.

Apify rate limit rules

| Scope | Limit | Notes | |-------|-------|-------| | Per resource (default) | 60 req/sec | Applies to each Actor, dataset, KV store independently | | Dataset push | 60 req/sec per dataset | Batch items to reduce call count | | Actor runs | 60 req/sec per Actor | Start runs in sequence or with delays | | Platform-wide | Higher limit | Aggregate across all resources |

"Per resource" means: calls to dataset A and dataset B each get 60 req/sec independently. Every response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.

Prerequisites

• An Apify account with API access and APIFY_TOKEN set in the environment.
• The apify-client package installed (npm install apify-client).
• For custom queuing: p-queue (npm install p-queue); crawlee for sleep and crawler-level concurrency.

Instructions

The workflow is five steps. Each is summarized here with its core lever; the full runnable code for every step is in implementation.md.

1. Understand built-in retries — apify-client already retries 429/500+ with exponential backoff. Tune maxRetries / minDelayBetweenRetriesMillis only when the defaults are wrong for your endpoint:

   import { ApifyClient } from 'apify-client';
   const client = new ApifyClient({
     token: process.env.APIFY_TOKEN,
     maxRetries: 5,                      // Default: 8
     minDelayBetweenRetriesMillis: 500,  // Default: 500
   });
   

2. Batch operations (biggest lever) — collapse per-item loops into one batched call (up to 9 MB), chunking only for very large datasets:

   await client.dataset(dsId).pushItems(items);   // 1 call, not N
   

3. Queue custom calls — gate raw API calls through p-queue (concurrency + intervalCap) so fan-out reads never exceed 60 req/sec. See implementation.md § Step 3.

4. Stagger Actor starts — insert a ~200 ms delay between start() calls so the runs endpoint never 429s, then waitForFinish() in parallel. See implementation.md § Step 4.

5. Monitor headers — feed X-RateLimit-* into a small monitor that warns before the wall and pauses exactly until reset. See implementation.md § Step 5.

Target-website throttling is a separate ceiling from the platform API — cap it with Crawlee's maxConcurrency / maxRequestsPerMinute (implementation.md § Crawlee-level concurrency).

Output

Applying this skill produces a rate-aware Apify integration:

• A configured ApifyClient with an explicit retry envelope.
• Batched/chunked dataset writes that cut API-call count by orders of magnitude.
• A p-queue-gated call path that holds requests under 60 req/sec per resource.
• Staggered Actor starts and, optionally, a header-driven monitor that pauses before exhaustion — the net effect being zero (or transparently retried) 429s under load.

Error Handling

| Scenario | Detection | Response | |----------|-----------|----------| | API 429 | apify-client auto-retries | Usually transparent; increase delays if persistent | | Target site 429 | statusCode === 429 in handler | Reduce maxConcurrency, add proxy rotation | | Burst of starts | Starting 100+ runs at once | Stagger with 200ms delays | | Large data push | Single 50MB dataset push | Chunk into 9MB batches |

Examples

Worked end-to-end scenarios live in examples.md:

• Bulk dataset push without 429s — 50,000 rows in ~50 calls via chunked batching.
• Fan-out reads through a queue — 500 Actor reads held under 50 req/sec.
• Launch 100 runs safely — staggered starts, then parallel wait-for-finish.
• Pause on header-driven exhaustion — sleep exactly until the limit resets.

Resources

• Apify API Rate Limits
• p-queue Documentation
• Crawlee Auto-scaling

For security configuration, see apify-security-basics.