Agent Skills: arXiv Search Skill

Mode: Cognitive/Prompt-Driven — No standalone utility script; use via agent context.

arXiv Search Skill

✅ No Installation Required

This skill uses existing tools to access arXiv:

• WebFetch - Direct access to arXiv API
• Exa - Semantic search with arXiv filtering

Works immediately - no MCP server, no restart needed.

Result Limits (Memory Safeguard)

arxiv-mcp returns academic papers. To prevent memory exhaustion:

• max_results: 20 (HARD LIMIT)
• Each paper metadata ~300 bytes
• 20 papers × 300 bytes = ~6 KB metadata
• Papers can be 100+ KB each if fetched - DON'T fetch full papers

Why the limit?

• Previous limit: 100 results → 30 KB+ metadata → context explosion
• New limit: 20 results → 6 KB metadata → memory safe
• 20 papers is usually enough to find your target

Method 1: WebFetch with arXiv API (Recommended for specific queries)

The arXiv API is publicly accessible at http://export.arxiv.org/api/query.

Recommended Pattern

// ✓ GOOD: Limit results to 20
WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=all:transformer+attention&max_results=20&sortBy=relevance',
  prompt: 'Extract paper titles, authors, abstracts, arXiv IDs, and PDF links from these results',
});

// ✓ GOOD: Use specific filters to reduce result set
WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=all:transformer+attention+2025&max_results=20&sortBy=submittedDate',
  prompt: 'Extract recent papers on transformer attention',
});

// ✗ BAD: Old behavior - unlimited or >20 results
WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=all:neural+networks',
  // Too broad - will get 100s of results
});

// ✗ BAD: Exceeds memory limit
WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=all:deep+learning&max_results=100',
  // Over limit - memory risk
});

Search by Keywords

WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=all:transformer+attention&max_results=20&sortBy=relevance',
  prompt: 'Extract paper titles, authors, abstracts, arXiv IDs, and PDF links from these results',
});

Search by Author

WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=au:LeCun&max_results=10&sortBy=submittedDate',
  prompt: 'Extract paper titles, authors, abstracts, and arXiv IDs',
});

Search by Category

WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=cat:cs.LG&max_results=15&sortBy=submittedDate',
  prompt: 'Extract paper titles, authors, abstracts, categories, and arXiv IDs',
});

Get Specific Paper by ID

WebFetch({
  url: 'http://export.arxiv.org/api/query?id_list=2301.07041',
  prompt:
    'Extract full details: title, all authors, abstract, categories, published date, PDF link',
});

API Query Parameters

| Parameter | Description | Example | | -------------- | ----------------------------------------------------------- | --------------------------------------------- | | search_query | Search terms with field prefixes | all:transformer, au:LeCun, ti:attention | | id_list | Comma-separated arXiv IDs | 2301.07041,2302.13971 | | max_results | Number of results (default 10, max 100) | max_results=20 | | start | Offset for pagination | start=10 | | sortBy | Sort order: relevance, lastUpdatedDate, submittedDate | sortBy=submittedDate | | sortOrder | ascending or descending | sortOrder=descending |

Field Prefixes for search_query

| Prefix | Field | Example | | ------ | ---------- | ------------------------- | | all: | All fields | all:machine+learning | | ti: | Title | ti:transformer | | au: | Author | au:Vaswani | | abs: | Abstract | abs:attention+mechanism | | cat: | Category | cat:cs.LG | | co: | Comment | co:accepted |

Boolean Operators

Combine terms with AND, OR, ANDNOT:

search_query=ti:transformer+AND+abs:attention
search_query=au:LeCun+OR+au:Bengio
search_query=cat:cs.LG+ANDNOT+ti:survey

When NOT to Use arxiv-mcp

• General web research → Use WebSearch/WebFetch instead
• Implementation examples → Use pnpm search:code or ripgrep skill on codebase (Grep/Glob as fallback)
• Product research → Use WebSearch with news filter
• Community discussions → Use WebSearch for forums/Stack Overflow

arxiv-mcp is best for:

• Finding academic papers on specific topics
• Understanding theoretical foundations
• Citing research in documentation
• Quick literature review (20 papers max)

Method 2: Exa Search (Better for semantic/natural language queries)

Use Exa for more natural language queries with arXiv filtering:

Semantic Search

mcp__Exa__web_search_exa({
  query: 'site:arxiv.org transformer architecture attention mechanism deep learning',
  numResults: 10,
});

Recent Papers in a Field

mcp__Exa__web_search_exa({
  query: 'site:arxiv.org large language model scaling laws 2024',
  numResults: 15,
});

Author-Focused Search

mcp__Exa__web_search_exa({
  query: 'site:arxiv.org author:"Yann LeCun" deep learning',
  numResults: 10,
});

Common arXiv Categories

| Category | Field | | ---------- | ------------------------------- | | cs.AI | Artificial Intelligence | | cs.LG | Machine Learning | | cs.CL | Computation and Language (NLP) | | cs.CV | Computer Vision | | cs.SE | Software Engineering | | cs.CR | Cryptography and Security | | stat.ML | Machine Learning (Statistics) | | math.* | Mathematics (all subcategories) | | physics.* | Physics (all subcategories) | | q-bio.* | Quantitative Biology | | econ.* | Economics |

Workflow: Complete Research Process

Step 1: Initial Search

// Start with broad Exa search for semantic matching
mcp__Exa__web_search_exa({
  query: 'site:arxiv.org transformer attention mechanism neural networks',
  numResults: 10,
});

Step 2: Get Specific Papers

// Get details for interesting papers by ID
WebFetch({
  url: 'http://export.arxiv.org/api/query?id_list=2301.07041,2302.13971',
  prompt: 'Extract full metadata for each paper: title, authors, abstract, categories, PDF URL',
});

Step 3: Find Related Work

// Search by category of interesting paper
WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=cat:cs.LG+AND+ti:attention&max_results=10&sortBy=submittedDate',
  prompt: 'Find related papers, extract titles and abstracts',
});

Step 4: Get Recent Papers

// Latest papers in the field
WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=cat:cs.LG&max_results=20&sortBy=submittedDate&sortOrder=descending',
  prompt: 'Extract the 20 most recent machine learning papers',
});

</execution_process>

<best_practices>

1. Use Exa for discovery: Natural language queries find semantically related papers
2. Use WebFetch for precision: Specific IDs, categories, or API queries
3. Combine approaches: Exa to discover, WebFetch to deep-dive
4. Use specific queries: "transformer attention mechanism" > "machine learning"
5. Check multiple categories: Papers often span cs.AI + cs.LG + cs.CL
6. Sort by date for recent work: sortBy=submittedDate&sortOrder=descending

</best_practices> </instructions>

WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=ti:transformer+AND+abs:attention&max_results=10&sortBy=relevance',
  prompt: 'Extract paper titles, authors, abstracts, and arXiv IDs',
});

Example 2: Find papers by researcher:

WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=au:Vaswani&max_results=15',
  prompt: 'List all papers by this author with titles and dates',
});

Example 3: Get recent ML papers:

WebFetch({
  url: 'http://export.arxiv.org/api/query?search_query=cat:cs.LG&max_results=20&sortBy=submittedDate&sortOrder=descending',
  prompt: 'Extract the 20 most recent machine learning papers with titles and abstracts',
});

Example 4: Semantic search with Exa:

mcp__Exa__web_search_exa({
  query: 'site:arxiv.org multimodal large language models vision 2024',
  numResults: 10,
});

Example 5: Get specific paper details:

WebFetch({
  url: 'http://export.arxiv.org/api/query?id_list=1706.03762',
  prompt: "Extract complete details for the 'Attention Is All You Need' paper",
});

</usage_example> </examples>

Agent Integration

This skill is automatically assigned to:

• researcher - Academic research, literature review
• scientific-research-expert - Deep scientific analysis
• developer - Finding technical papers for implementation

Iron Laws

1. ALWAYS enforce max_results=20 — never allow unlimited or >20 result queries; context explosion from 100+ papers is a known failure mode that stalls agent pipelines.
2. NEVER fetch full paper PDFs during literature review — extract metadata and abstracts only; full papers are 100KB+ each and will exhaust context budget in minutes.
3. ALWAYS use Exa for semantic discovery, WebFetch for precision retrieval — Exa finds semantically related papers; WebFetch gets specific IDs or category feeds; use both in sequence, not interchangeably.
4. NEVER use broad queries without field prefixes — search_query=neural+networks returns thousands of results; always scope with ti:, au:, cat:, or abs: prefixes to target the query.
5. ALWAYS cite arXiv IDs (e.g., 2301.07041) when referencing papers — titles alone are ambiguous and change; IDs are stable, machine-readable, and enable instant retrieval.

Anti-Patterns

| Anti-Pattern | Why It Fails | Correct Approach | | ---------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------- | | Using max_results=100 or no limit | Context explosion; 100 papers × 300 bytes = 30KB+ metadata | Always set max_results=20 (hard limit) | | Fetching full paper PDFs | Single paper can be 100KB+; kills context budget | Extract abstract + metadata only via API | | Broad query without field prefix | Returns irrelevant results across all fields | Use ti:, au:, cat:, or abs: prefix | | Using only WebFetch for discovery | Misses semantically related papers not matching exact terms | Use Exa for semantic discovery first | | Citing paper titles instead of arXiv IDs | Titles can be ambiguous or duplicated | Always include the arXiv ID (e.g., 1706.03762) |

Memory Protocol (MANDATORY)

Before starting:

cat .claude/context/memory/learnings.md

After completing:

• New pattern -> .claude/context/memory/learnings.md
• Issue found -> .claude/context/memory/issues.md
• Decision made -> .claude/context/memory/decisions.md

ASSUME INTERRUPTION: Your context may reset. If it's not in memory, it didn't happen.