Apify Rate Limits Skill

Apify Rate Limits

Overview

The Apify API enforces rate limits per resource. The apify-client library auto-retries 429s (up to 8 times with exponential backoff), so most workloads never notice a limit. You reach for this skill when bulk operations, custom API calls, or large fan-outs push past what the built-in retry can absorb — you then batch, queue, stagger, and monitor to stay under the ceiling.

Full runnable code for every step is in implementation.md; combined scenarios are in examples.md.

Apify rate limit rules

| Scope | Limit | Notes | |-------|-------|-------| | Per resource (default) | 60 req/sec | Applies to each Actor, dataset, KV store independently | | Dataset push | 60 req/sec per dataset | Batch items to reduce call count | | Actor runs | 60 req/sec per Actor | Start runs in sequence or with delays | | Platform-wide | Higher limit | Aggregate across all resources |

"Per resource" means: calls to dataset A and dataset B each get 60 req/sec independently. Every response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.

Prerequisites

An Apify account with API access and APIFY_TOKEN set in the environment.
The apify-client package installed (npm install apify-client).
For custom queuing: p-queue (npm install p-queue); crawlee for sleep and crawler-level concurrency.

Instructions

The workflow is five steps. Each is summarized here with its core lever; the full runnable code for every step is in implementation.md.

Understand built-in retries — apify-client already retries 429/500+ with exponential backoff. Tune maxRetries / minDelayBetweenRetriesMillis only when the defaults are wrong for your endpoint:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({
  token: process.env.APIFY_TOKEN,
  maxRetries: 5,                      // Default: 8
  minDelayBetweenRetriesMillis: 500,  // Default: 500
});

Batch operations (biggest lever) — collapse per-item loops into one batched call (up to 9 MB), chunking only for very large datasets:
```
await client.dataset(dsId).pushItems(items);   // 1 call, not N
```
Queue custom calls — gate raw API calls through p-queue (concurrency + intervalCap) so fan-out reads never exceed 60 req/sec. See implementation.md § Step 3.
Stagger Actor starts — insert a ~200 ms delay between start() calls so the runs endpoint never 429s, then waitForFinish() in parallel. See implementation.md § Step 4.
Monitor headers — feed X-RateLimit-* into a small monitor that warns before the wall and pauses exactly until reset. See implementation.md § Step 5.

Target-website throttling is a separate ceiling from the platform API — cap it with Crawlee's maxConcurrency / maxRequestsPerMinute (implementation.md § Crawlee-level concurrency).

Output

Applying this skill produces a rate-aware Apify integration:

A configured ApifyClient with an explicit retry envelope.
Batched/chunked dataset writes that cut API-call count by orders of magnitude.
A p-queue-gated call path that holds requests under 60 req/sec per resource.
Staggered Actor starts and, optionally, a header-driven monitor that pauses before exhaustion — the net effect being zero (or transparently retried) 429s under load.

Error Handling

| Scenario | Detection | Response | |----------|-----------|----------| | API 429 | apify-client auto-retries | Usually transparent; increase delays if persistent | | Target site 429 | statusCode === 429 in handler | Reduce maxConcurrency, add proxy rotation | | Burst of starts | Starting 100+ runs at once | Stagger with 200ms delays | | Large data push | Single 50MB dataset push | Chunk into 9MB batches |

Examples

Worked end-to-end scenarios live in examples.md:

Bulk dataset push without 429s — 50,000 rows in ~50 calls via chunked batching.
Fan-out reads through a queue — 500 Actor reads held under 50 req/sec.
Launch 100 runs safely — staggered starts, then parallel wait-for-finish.
Pause on header-driven exhaustion — sleep exactly until the limit resets.

Resources

For security configuration, see apify-security-basics.

Agent Skills: Apify Rate Limits

Install this agent skill to your local

Skill Files