data-base Skill | Agent Skills

Mental Model

Data acquisition is converting unstructured web content into structured data. Choose tool based on page complexity: JS-heavy → chrome-devtools MCP, static → Python requests.

Tool Selection

| Page Type | Tool | When to Use | |-----------|------|-------------| | Dynamic (JS-rendered, SPAs) | chrome-devtools MCP | React/Vue apps, infinite scroll, login gates | | Static HTML | Python requests | Blogs, news sites, simple pages | | Complex/reusable logic | Python script | Multi-step scraping, rate limiting, proxies |

Anti-Patterns (NEVER)

Don't scrape without checking robots.txt
Don't overload servers (default: 1 req/sec)
Don't scrape personal data without consent
Don't use Chinese characters in output filenames (ASCII only)
Don't forget to identify bot with User-Agent

Output Format

JSON: Nested/hierarchical data
CSV: Tabular data
Filename: {source}_{timestamp}.{ext} (ASCII only, e.g., news_20250115.csv)

Workflow

Ask: What data? Which sites? How much?
Select tool based on page type
Extract and save structured data
Deliver file path to user or pass to data-analysis

Python Environment

Auto-initialize virtual environment if needed, then execute:

cd skills/data-base

if [ ! -f ".venv/bin/python" ]; then
    echo "Creating Python environment..."
    ./setup.sh
fi

.venv/bin/python your_script.py

The setup script auto-installs: requests, beautifulsoup4, pandas, web scraping tools.

References (load on demand)

For detailed APIs and templates, load: references/REFERENCE.md, references/templates.md

Agent Skills: data-base

Install this agent skill to your local