Agent-Skills.md

Agent Skills: Scraping Skill

Web scraping via progressive escalation and social media platform scrapers. USE WHEN scraping, crawl, scrape URL, bot detection, CAPTCHA, spider.

UncategorizedID: danielmiessler/personal_ai_infrastructure/scraping

Repository

danielmiessler/personal_ai_infrastructure
danielmiesslerLicense: MIT
10,8261,498

Install this agent skill to your local

pnpm dlx add-skill https://github.com/danielmiessler/Personal_AI_Infrastructure/tree/HEAD/Releases/Pi/skills/scraping

Skill Files

Browse the full folder contents for scraping.

Download Skill

Loading file tree…

Releases/Pi/skills/scraping/SKILL.md

Skill Metadata

Name
scraping
Description
Web scraping via progressive escalation and social media platform scrapers. USE WHEN scraping, crawl, scrape URL, bot detection, CAPTCHA, spider.

Scraping Skill

Progressive Escalation

  1. Direct fetch — curl/fetch with standard headers
  2. Browser rendering — Headless browser for JS-heavy sites
  3. Proxy rotation — For rate-limited or geo-restricted content
  4. Specialized APIs — Platform-specific scrapers

Rules

  • Respect robots.txt
  • Rate limit requests
  • Handle pagination
  • Extract structured data
  • Store raw + processed versions