apify-rate-limits
'Handle Apify API rate limits with proper backoff and request queuing.
Install / Use
npx skills add jeremylongshore/tons-of-skills-marketplace --skill apify-rate-limitsInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AutomationSupported Platforms
Our assessment of apify-rate-limits
apify-rate-limits scores 80/100 on our quality scale, 2475th of 2,780 Automation skills we index.
Its SKILL.md is 5.2 KB long, well organised into 9 sections and no code examples: a solid amount of guidance for an agent.
With 2,785 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so apify-rate-limits is actively maintained.
- Our last check on 2026-11-08 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-01. Automated pattern scan on 2026-10-01. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
apify-rate-limits compared with similar skills
All 4 of these similar skills score higher than apify-rate-limits; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| apify-rate-limits (this skill)by jeremylongshore | 80 | 2.8k | 53d ago | SKILL.md |
| taste-skillby Leonxlnx | 100 | 116.8k | 8d ago | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 81.9k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
Frequently asked questions
- How do I install apify-rate-limits?
- Run
npx skills add jeremylongshore/tons-of-skills-marketplace --skill apify-rate-limits. The install tabs above show the steps for each supported agent. - Which AI agents does apify-rate-limits work with?
- It is written for Claude Code, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is apify-rate-limits safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is apify-rate-limits still maintained?
- The repository was last updated about 2 months ago, so apify-rate-limits is actively maintained.
Skill content
View source on GitHubname: apify-rate-limits description: 'Handle Apify API rate limits with proper backoff and request queuing. Use when hitting 429 errors, optimizing API request throughput, or implementing rate-aware client wrappers. Trigger with "apify rate limit", "apify throttling", "apify 429", "apify retry", "apify backoff", "too many requests apify".' allowed-tools: Read, Write, Edit version: 1.5.0 license: MIT author: Jeremy Longshore jeremy@intentsolutions.io tags:
- saas
- scraping
- automation
- apify compatibility: Designed for Claude Code
Apify Rate Limits
Overview
The Apify API enforces rate limits per resource. The apify-client library
auto-retries 429s (up to 8 times with exponential backoff), so most workloads never
notice a limit. You reach for this skill when bulk operations, custom API calls, or
large fan-outs push past what the built-in retry can absorb — you then batch, queue,
stagger, and monitor to stay under the ceiling.
Full runnable code for every step is in implementation.md; combined scenarios are in examples.md.
Apify rate limit rules
| Scope | Limit | Notes | |-------|-------|-------| | Per resource (default) | 60 req/sec | Applies to each Actor, dataset, KV store independently | | Dataset push | 60 req/sec per dataset | Batch items to reduce call count | | Actor runs | 60 req/sec per Actor | Start runs in sequence or with delays | | Platform-wide | Higher limit | Aggregate across all resources |
"Per resource" means: calls to dataset A and dataset B each get 60 req/sec
independently. Every response carries X-RateLimit-Limit,
X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.
Prerequisites
- An Apify account with API access and
APIFY_TOKENset in the environment. - The
apify-clientpackage installed (npm install apify-client). - For custom queuing:
p-queue(npm install p-queue);crawleeforsleepand crawler-level concurrency.
Instructions
The workflow is five steps. Each is summarized here with its core lever; the full runnable code for every step is in implementation.md.
-
Understand built-in retries —
apify-clientalready retries 429/500+ with exponential backoff. TunemaxRetries/minDelayBetweenRetriesMillisonly when the defaults are wrong for your endpoint:import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN, maxRetries: 5, // Default: 8 minDelayBetweenRetriesMillis: 500, // Default: 500 }); -
Batch operations (biggest lever) — collapse per-item loops into one batched call (up to 9 MB), chunking only for very large datasets:
await client.dataset(dsId).pushItems(items); // 1 call, not N -
Queue custom calls — gate raw API calls through
p-queue(concurrency+intervalCap) so fan-out reads never exceed 60 req/sec. See implementation.md § Step 3. -
Stagger Actor starts — insert a ~200 ms delay between
start()calls so the runs endpoint never 429s, thenwaitForFinish()in parallel. See implementation.md § Step 4. -
Monitor headers — feed
X-RateLimit-*into a small monitor that warns before the wall and pauses exactly until reset. See implementation.md § Step 5.
Target-website throttling is a separate ceiling from the platform API — cap it with
Crawlee's maxConcurrency / maxRequestsPerMinute
(implementation.md § Crawlee-level concurrency).
Output
Applying this skill produces a rate-aware Apify integration:
- A configured
ApifyClientwith an explicit retry envelope. - Batched/chunked dataset writes that cut API-call count by orders of magnitude.
- A
p-queue-gated call path that holds requests under 60 req/sec per resource. - Staggered Actor starts and, optionally, a header-driven monitor that pauses before exhaustion — the net effect being zero (or transparently retried) 429s under load.
Error Handling
| Scenario | Detection | Response |
|----------|-----------|----------|
| API 429 | apify-client auto-retries | Usually transparent; increase delays if persistent |
| Target site 429 | statusCode === 429 in handler | Reduce maxConcurrency, add proxy rotation |
| Burst of starts | Starting 100+ runs at once | Stagger with 200ms delays |
| Large data push | Single 50MB dataset push | Chunk into 9MB batches |
Examples
Worked end-to-end scenarios live in examples.md:
- Bulk dataset push without 429s — 50,000 rows in ~50 calls via chunked batching.
- Fan-out reads through a queue — 500 Actor reads held under 50 req/sec.
- Launch 100 runs safely — staggered starts, then parallel wait-for-finish.
- Pause on header-driven exhaustion — sleep exactly until the limit resets.
Resources
For security configuration, see apify-security-basics.
Related Skills
taste-skill
116.8kAgent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
ruflo
81.9k🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
