SkillAgentSearch skills...

apify-rate-limits

'Handle Apify API rate limits with proper backoff and request queuing.

Install / Use

npx skills add jeremylongshore/tons-of-skills-marketplace --skill apify-rate-limits

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

80/100

Category

Automation

Supported Platforms

Claude Code

Our assessment of apify-rate-limits

apify-rate-limits scores 80/100 on our quality scale, 2475th of 2,780 Automation skills we index.

Its SKILL.md is 5.2 KB long, well organised into 9 sections and no code examples: a solid amount of guidance for an agent.

With 2,785 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
26/30
Structure
13/20
Description
12/15
Adoption
15/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so apify-rate-limits is actively maintained.
  • Our last check on 2026-11-08 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-01. Automated pattern scan on 2026-10-01. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

apify-rate-limits compared with similar skills

All 4 of these similar skills score higher than apify-rate-limits; compare them before choosing.

SkillScoreStarsUpdatedFormat
apify-rate-limits (this skill)by jeremylongshore802.8k53d agoSKILL.md
taste-skillby Leonxlnx100116.8k8d agoCLAUDE.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
rufloby ruvnet10081.9ktodayCLAUDE.md
headroomby headroomlabs-ai10079.8ktodayCLAUDE.md

Frequently asked questions

How do I install apify-rate-limits?
Run npx skills add jeremylongshore/tons-of-skills-marketplace --skill apify-rate-limits. The install tabs above show the steps for each supported agent.
Which AI agents does apify-rate-limits work with?
It is written for Claude Code, as a SKILL.md file. Other agents that read the same format can often use it too.
Is apify-rate-limits safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is apify-rate-limits still maintained?
The repository was last updated about 2 months ago, so apify-rate-limits is actively maintained.

name: apify-rate-limits description: 'Handle Apify API rate limits with proper backoff and request queuing. Use when hitting 429 errors, optimizing API request throughput, or implementing rate-aware client wrappers. Trigger with "apify rate limit", "apify throttling", "apify 429", "apify retry", "apify backoff", "too many requests apify".' allowed-tools: Read, Write, Edit version: 1.5.0 license: MIT author: Jeremy Longshore jeremy@intentsolutions.io tags:

  • saas
  • scraping
  • automation
  • apify compatibility: Designed for Claude Code

Apify Rate Limits

Overview

The Apify API enforces rate limits per resource. The apify-client library auto-retries 429s (up to 8 times with exponential backoff), so most workloads never notice a limit. You reach for this skill when bulk operations, custom API calls, or large fan-outs push past what the built-in retry can absorb — you then batch, queue, stagger, and monitor to stay under the ceiling.

Full runnable code for every step is in implementation.md; combined scenarios are in examples.md.

Apify rate limit rules

| Scope | Limit | Notes | |-------|-------|-------| | Per resource (default) | 60 req/sec | Applies to each Actor, dataset, KV store independently | | Dataset push | 60 req/sec per dataset | Batch items to reduce call count | | Actor runs | 60 req/sec per Actor | Start runs in sequence or with delays | | Platform-wide | Higher limit | Aggregate across all resources |

"Per resource" means: calls to dataset A and dataset B each get 60 req/sec independently. Every response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.

Prerequisites

  • An Apify account with API access and APIFY_TOKEN set in the environment.
  • The apify-client package installed (npm install apify-client).
  • For custom queuing: p-queue (npm install p-queue); crawlee for sleep and crawler-level concurrency.

Instructions

The workflow is five steps. Each is summarized here with its core lever; the full runnable code for every step is in implementation.md.

  1. Understand built-in retries — apify-client already retries 429/500+ with exponential backoff. Tune maxRetries / minDelayBetweenRetriesMillis only when the defaults are wrong for your endpoint:

    import { ApifyClient } from 'apify-client';
    const client = new ApifyClient({
      token: process.env.APIFY_TOKEN,
      maxRetries: 5,                      // Default: 8
      minDelayBetweenRetriesMillis: 500,  // Default: 500
    });
    
  2. Batch operations (biggest lever) — collapse per-item loops into one batched call (up to 9 MB), chunking only for very large datasets:

    await client.dataset(dsId).pushItems(items);   // 1 call, not N
    
  3. Queue custom calls — gate raw API calls through p-queue (concurrency + intervalCap) so fan-out reads never exceed 60 req/sec. See implementation.md § Step 3.

  4. Stagger Actor starts — insert a ~200 ms delay between start() calls so the runs endpoint never 429s, then waitForFinish() in parallel. See implementation.md § Step 4.

  5. Monitor headers — feed X-RateLimit-* into a small monitor that warns before the wall and pauses exactly until reset. See implementation.md § Step 5.

Target-website throttling is a separate ceiling from the platform API — cap it with Crawlee's maxConcurrency / maxRequestsPerMinute (implementation.md § Crawlee-level concurrency).

Output

Applying this skill produces a rate-aware Apify integration:

  • A configured ApifyClient with an explicit retry envelope.
  • Batched/chunked dataset writes that cut API-call count by orders of magnitude.
  • A p-queue-gated call path that holds requests under 60 req/sec per resource.
  • Staggered Actor starts and, optionally, a header-driven monitor that pauses before exhaustion — the net effect being zero (or transparently retried) 429s under load.

Error Handling

| Scenario | Detection | Response | |----------|-----------|----------| | API 429 | apify-client auto-retries | Usually transparent; increase delays if persistent | | Target site 429 | statusCode === 429 in handler | Reduce maxConcurrency, add proxy rotation | | Burst of starts | Starting 100+ runs at once | Stagger with 200ms delays | | Large data push | Single 50MB dataset push | Chunk into 9MB batches |

Examples

Worked end-to-end scenarios live in examples.md:

  • Bulk dataset push without 429s — 50,000 rows in ~50 calls via chunked batching.
  • Fan-out reads through a queue — 500 Actor reads held under 50 req/sec.
  • Launch 100 runs safely — staggered starts, then parallel wait-for-finish.
  • Pause on header-driven exhaustion — sleep exactly until the limit resets.

Resources

For security configuration, see apify-security-basics.

Related Skills

View on GitHub
GitHub Stars2.8k
CategoryAutomation
Updated1mo ago
Forks404

Languages

Python

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions