Firecrawl Production Checklist Skill

Firecrawl Production Checklist

Overview

Pre-deployment validation checklist for applications using Firecrawl's scrape, crawl, map, and extract APIs. Covers credential management, crawl safety limits, error handling, monitoring, and rollback.

Prerequisites

Staging environment tested and passing
Production API key from firecrawl.dev/app
Monitoring infrastructure ready

Pre-Deployment Checklist

Credentials & Security

[ ] Production FIRECRAWL_API_KEY in secure vault (not in code or .env)
[ ] Key starts with fc- and is scoped to production
[ ] Different API keys for dev/staging/production
[ ] .env files in .gitignore
[ ] Webhook secrets stored securely
[ ] Git history scanned for leaked keys

Crawl Safety

[ ] All crawlUrl calls have limit parameter set
[ ] maxDepth configured to prevent unbounded crawling
[ ] includePaths / excludePaths filters applied where appropriate
[ ] Credit budget tracking implemented (daily limit alerts)
[ ] No hardcoded URLs in production code

Error Handling

[ ] 429 rate limit handling with exponential backoff
[ ] 402 credit exhaustion handled gracefully (no crash)
[ ] 401 auth failure logged and alerted
[ ] Async crawl jobs have timeout with deadline
[ ] Fallback from crawl to individual scrape on failure
[ ] Empty markdown detection (JS rendering issues)

Monitoring & Alerting

[ ] Scrape success/failure rate tracked
[ ] Credit consumption monitored
[ ] Crawl job completion rate tracked
[ ] Alert on credit balance below threshold
[ ] Alert on error rate > 5%
[ ] Webhook delivery failures logged

Instructions

Step 1: Verify API Connectivity

set -euo pipefail
# Test production key
curl -s https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}' | jq '.success'

# Check credit balance
curl -s https://api.firecrawl.dev/v1/team/credits \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" | jq .

Step 2: Health Check Endpoint

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

export async function healthCheck() {
  const start = Date.now();
  try {
    const result = await firecrawl.scrapeUrl("https://example.com", {
      formats: ["markdown"],
    });
    return {
      status: result.success ? "healthy" : "degraded",
      latencyMs: Date.now() - start,
      hasContent: (result.markdown?.length || 0) > 0,
    };
  } catch (error: any) {
    return {
      status: "unhealthy",
      latencyMs: Date.now() - start,
      error: error.statusCode || error.message,
    };
  }
}

Step 3: Production-Safe Crawl Wrapper

export async function productionCrawl(url: string, opts: {
  maxPages: number;
  paths?: string[];
  timeout?: number;
}) {
  // Hard credit safety — never exceed configured limit
  const limit = Math.min(opts.maxPages, 500);

  const job = await firecrawl.asyncCrawlUrl(url, {
    limit,
    maxDepth: 3,
    includePaths: opts.paths,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });

  // Poll with timeout
  const deadline = Date.now() + (opts.timeout || 600000);
  let pollInterval = 2000;
  let status = await firecrawl.checkCrawlStatus(job.id);

  while (status.status === "scraping" && Date.now() < deadline) {
    await new Promise(r => setTimeout(r, pollInterval));
    pollInterval = Math.min(pollInterval * 1.5, 30000);
    status = await firecrawl.checkCrawlStatus(job.id);
  }

  if (status.status !== "completed") {
    throw new Error(`Crawl ${job.id} did not complete: ${status.status}`);
  }
  return status;
}

Step 4: Rollback Procedure

set -euo pipefail
# Immediate rollback — disable Firecrawl integration
kubectl set env deployment/app FIRECRAWL_ENABLED=false
kubectl rollout restart deployment/app

# Verify rollback
curl -s https://app.example.com/health | jq '.services.firecrawl'

Alerting Rules

| Alert | Condition | Severity | |-------|-----------|----------| | API unreachable | Health check fails 3x | P1 | | Credits < 1000 | Balance check | P2 | | Error rate > 5% | 429/5xx rate | P2 | | Crawl timeout | Job stuck > 10min | P3 | | Auth failure | Any 401 response | P1 |

Resources

Next Steps

For version upgrades, see firecrawl-upgrade-migration.

Agent Skills: Firecrawl Production Checklist

Install this agent skill to your local

Skill Files