Agent-Skills.md

Agent Skills: RAG & Retrieval Engineering

RAG and search engineering — chunking, hybrid retrieval, reranking, and nDCG evaluation. Use when building retrieval-augmented generation pipelines.

UncategorizedID: vasilyu1983/ai-agents-public/ai-rag

Repository

vasilyu1983/ai-agents-public
vasilyu1983License: MIT
4911

Install this agent skill to your local

pnpm dlx add-skill https://github.com/vasilyu1983/AI-Agents-public/tree/HEAD/frameworks/shared-skills/skills/ai-rag

Skill Files

Browse the full folder contents for ai-rag.

Download Skill

Loading file tree…

frameworks/shared-skills/skills/ai-rag/SKILL.md

Skill Metadata

Name
ai-rag
Description
"Designs retrieval-augmented generation and search systems. Use when choosing retrieval, chunking, hybrid search, grounding, or RAG evaluation patterns."

RAG & Retrieval Engineering

Build retrieval systems that are grounded, observable, and explicit about tradeoffs.

This skill covers:

  • Retrieval architecture choice: long-context vs hosted file search vs tool-first/MCP vs SQL/graph vs classic vector RAG
  • Corpus preparation: parsing, metadata, chunking, ACLs, freshness, invalidation
  • Retrieval quality: sparse, dense, hybrid, late interaction, reranking, multimodal retrieval
  • Answer quality: grounding, citation coverage, refusal on missing evidence, regression testing

July 2026 posture

  • Choose the retrieval mode before tuning chunk size. A vector index is not the default answer to every knowledge problem.
  • Separate retrieval quality from answer quality and evaluate both.
  • Treat retrieved text, tool responses, and MCP resources as untrusted input.
  • Prefer primary sources for vendor or framework recommendations; volatile facts must be verified live.
  • Treat OpenTelemetry GenAI semantic conventions as useful but still evolving.
  • Context-budget note (Opus 4.7 tokenizer): The Claude Opus 4.7 tokenizer encodes ~1.0–1.35× more tokens than the pre-2026 tokenizer for the same text. Chunk-size and token-budget heuristics from earlier than 2026 are invalid — re-measure on your own corpus with the current tokenizer before setting chunk sizes or context-window budgets.
  • Managed retrieval is a real option: Anthropic's web_search_20260209 (and web_search_20250305) server tools and OpenAI's file-search are API-native retrieval surfaces — evaluate them before building a self-hosted RAG stack. See references/managed-retrieval-vs-self-hosted.md.
  • Retrieval may not be the right answer at all: if the corpus is small and stable, CAG / long-context / fine-tune may be cheaper and more reliable than RAG. Run the decision rubric in ../ai-context-layer/references/retrieve-vs-preload-vs-finetune.md before building a RAG pipeline.

Scope note: For generation-prompt structure and output contracts after retrieval, use ai-prompt-engineering.

Implementation note: This skill owns retrieval theory and evaluation concepts. For vector-brain builds with paste-ready SQL, pgvector assets, manifests, ingest scripts, and agent retrieval tool contracts, use ai-vector-brain.

ASCII Flow

knowledge need
  |
  v
retrieval mode decision
  long context | hosted retrieval | tool/API | SQL/graph | hybrid search
  |
  v
corpus contract
  source truth + parsing + metadata + ACL + freshness + deletion path
  |
  v
retrieval quality loop
  baseline -> sparse/dense/hybrid -> rerank/filter/query rewrite -> eval
  |
  v
answer quality loop
  context pack + citation checks + refusal behavior + faithfulness eval

Workflow

  1. Pin the authority source and freshness budget first. Before touching a chunker or embedder, decide whether truth lives in documents, live tools, or a structured store, and how stale an answer is allowed to be. This decision, not chunk size, determines the architecture.
  2. Run the retrieval mode decision (see Retrieval Choice Framework below) and default to the simplest mode that meets freshness, traceability, and latency — long-context, hosted file search, tool-first/MCP, and SQL/graph all beat a custom vector stack when they fit.
  3. Write the corpus contract before indexing anything: parsing strategy, metadata schema (tenant, ACL, source, timestamp, section), freshness/invalidation rule, and a deletion path that actually propagates.
  4. Build the dumbest possible baseline first: exact/lexical search plus a golden eval set (scripts/exact_search_baseline.py, scripts/retrieval_eval.py). Measure recall@k, MRR, nDCG before adding embeddings, hybrid fusion, or a reranker — an expert never tunes a component the eval hasn't proven necessary.
  5. Add retrieval-quality layers only when the baseline eval shows a measured gap: hybrid (sparse+dense), reranking, query rewriting, late interaction/multivector, or multimodal retrieval, in that order of operational cost.
  6. Keep retrieval eval and answer eval as two separate gates. A retrieval pipeline that hits recall@10 = 0.9 can still produce ungrounded answers; faithfulness, citation coverage, and abstention behavior need their own test set and their own pass/fail bar.
  7. Instrument before you scale: evidence IDs, retrieval-mode traces, and confidence scores must exist end to end so a bad answer can be traced back to a specific retrieval decision (see observability-tracing-contract.md).
  8. Treat the eval set as a regression gate, not a one-time report. Re-run it before every chunker, embedder, index, or reranker change, and before any vendor swap (embedding model, reranker, vector DB) — silent regressions are the most common way "working" RAG systems degrade.

Quick Reference

| Need | Recommended path | Use when | Avoid when | |------|------------------|----------|------------| | Small corpus fits in context | Long-context prompt or lightweight search | Low update rate, low audit burden | Corpus is large, fast-changing, or citation-critical | | Provider-managed retrieval | Anthropic web search tool (web_search_20260209) or hosted file search | Fresh web data with citations, fast start, no infra to operate | You need residency controls, custom ranking internals, or corpus isolation beyond provider defaults | | Provider-managed file/doc retrieval | OpenAI file-search (Responses API) | Standard docs/Q&A, limited infra appetite | You need custom ranking, strict residency controls, or deep retrieval tuning | | Provider-managed doc retrieval on AWS | AWS Bedrock Knowledge Bases | AWS-native, want managed ingestion + embedding + retrieval with IAM/residency controls | You need cross-cloud portability or custom ranking internals → references/aws-bedrock-knowledge-bases.md | | Source of truth is tools/APIs | Tool-first or MCP retrieval | Data lives in SQL, CRM, ticketing, SaaS APIs, or internal tools | You actually need semantic retrieval over large prose corpora | | Website or research ingestion | Crawl/extract first, then index | The source starts as websites, reports, or broad web research | You only need one-off browsing instead of a maintained retrieval system | | Exact joins or aggregations | SQL/graph retrieval | Questions need filters, joins, counts, or relationship traversal | Natural-language corpus lookup is the main problem | | Standard knowledge retrieval | Hybrid search + rerank | Mixed lexical + semantic queries, high recall, controllable latency | A simpler hosted or tool-first option already solves the problem | | Generated repo/context hub corpus | Graph-bounded hybrid (graph scopes, vector recalls, rerank) | A compiled multi-repo/context hub with an existing knowledge/code graph | The corpus has no graph, or simple semantic lookup already passes eval | | High-precision retrieval | Late interaction / multivector retrieval | Near-duplicate docs, subtle wording differences, PDF pages, multilingual precision | Latency budget is tight and BM25+dense+rering is already sufficient | | PDFs, tables, diagrams | Multimodal document retrieval | OCR loses structure or layout meaning matters | Plain text extraction is already high quality | | Production readiness proof | Golden eval + exact baseline + traces | You need to prove quality, not just describe architecture | One-off exploratory research with no durable corpus |

Retrieval Choice Framework

Need external knowledge?
  ├─ No -> Use direct prompting / standard prompt engineering
  └─ Yes
      ├─ Data already lives behind tools, APIs, SQL, or SaaS?
      │   └─ Start with tool-first or MCP retrieval
      │
      ├─ Corpus fits comfortably in model context and changes slowly?
      │   └─ Try long-context or hosted file search before custom indexing
      │
      ├─ Need joins, counts, or relationship traversal?
      │   └─ Use SQL, graph, or graph+vector hybrid retrieval
      │       (paradigm choice: see ../software-database-design/SKILL.md#storage-paradigm-matrix-relational-vs-graph-vs-vector)
      │
      ├─ Need fresh web data or provider-managed file search without self-hosting?
      │   └─ Evaluate managed retrieval first: Anthropic web_search_20260209 / web_search_20250305,
      │      OpenAI file-search, or other provider-native tools before building a custom stack
      │      (see references/managed-retrieval-vs-self-hosted.md for decision criteria)
      │
      ├─ Need retrieval over prose or mixed documents?
      │   └─ Use sparse+dense hybrid as the default baseline
      │
      ├─ Retrieval misses subtle matches or page-level structure?
      │   └─ Add late interaction, multivector, or multimodal retrieval
      │
      ├─ Precision still poor?
      │   └─ Add reranking, filters, query rewriting, and stronger evals
      │
      └─ Same queries keep re-running synthesis? Need a reviewable knowledge asset?
          └─ Switch from retrieval to knowledge compilation (P7 in ai-context-layer)
              Route to ai-context-layer/references/knowledge-compilation-and-wiki-pattern.md
              Blocks A15 (RAG re-run per turn instead of compiled knowledge)

Core Concepts

  • Authority source: define whether truth comes from retrieved documents, live tools, structured databases, or a hybrid of them.
  • Trust boundary: retrieved chunks, uploaded files, MCP resources, tool results, and web pages are untrusted until validated.
  • Freshness model: set staleness budget, invalidation triggers, and deletion propagation rules.
  • Generated-context corpus: a compiled multi-repo/context hub is a valid corpus; graph-bounded build path is ../ai-vector-brain/references/dev-context-hub-vector-recipe.md.
  • Evidence contract: return stable evidence IDs, source metadata, and enough context for later citation verification.
  • Two eval planes: retrieval relevance and answer faithfulness are separate systems and must be measured separately.

When NOT to Use This Skill

| Need | Route to | |------|----------| | Small personal knowledge base (≲100 articles / ≲400K words) where an LLM-maintained INDEX.md is sufficient and vector infra would be premature | ../docs-notes-retrieval/SKILL.md (Karpathy scale heuristic) | | Paste-ready SQL, DDL, pgvector indexes, or backend loader scripts | ../ai-vector-brain/SKILL.md | | Building a repo/docs/compliance vector brain end to end | ../ai-vector-brain/SKILL.md | | App context architecture, memory lifecycle, and grounding boundaries | ai-context-layer | | Agent topology and tool orchestration | ../ai-agents/SKILL.md | | Bot UX, conversation flows, and escalation | ai-bot-builder | | On-device iOS retrieval-stitch composer (Apple Foundation Models, @Generable, retrieval over local chunks with no-cloud guarantee) | ../software-ios-ai-engine/SKILL.md | | End-to-end natural conversational iOS surface composed with this skill + ai-context-layer + ai-vector-brain, Path A (Foundation Models) and Path B (vector-DB-only) | ../software-ios-ai-engine/references/composition-with-rag-context-vector.md | | Cross-platform natural conversation (iOS / Android / web / Telegram-Discord-WhatsApp-Slack bots / voice / backend) with or without on-device models | ../ai-context-layer/references/conversational-surfaces-cross-platform.md |

Operational Defaults

Do

  • Start with the simplest retrieval mode that satisfies freshness, traceability, and latency.
  • Keep metadata first-class: tenant, ACL, source, timestamp, page/section, language, content type.
  • Probe database/search capabilities before depending on optional extensions, parser configs, generated search columns, or ANN index types in migrations.
  • Stage SQL-backed RAG rollout in reversible layers: schema/RLS, minimal lexical retrieval, content seed, ranking tuning, evals, then indexed hybrid retrieval.
  • Defer embeddings, ANN indexes, full-text/trigram search, and dedicated search services until an eval failure, corpus-size trigger, or measured latency problem justifies the added moving parts.
  • Make retrieval deterministic where possible: fixed candidate sizes, explicit filters, bounded retries.
  • Add citation coverage checks and refusal behavior when evidence is missing or conflicting.
  • Keep offline eval sets versioned and run them before changing chunkers, embedders, indexes, or rerankers.

Avoid

  • Assuming every knowledge feature needs a custom vector database.
  • Hard-coding prices, benchmark rankings, or model leaderboards into durable guidance.
  • Treating agentic RAG as the default for simple factual Q&A.
  • Mixing tenants, corpora, or sensitivity classes without retrieval-time isolation.
  • Using response caching without invalidation tied to corpus or tool freshness.

Known Traps

  • tuning chunk size, embedder, or reranker before deciding whether retrieval is even the right architecture
  • using pre-2026 chunk-size or token-budget heuristics with Opus 4.7 — the tokenizer produces ~1.0–1.35× more tokens for the same text; re-measure on your own corpus
  • assuming iterative/agentic multi-hop retrieval is universally superior — gains are query-distribution-dependent (strongest on chemistry-domain multi-hop; transfer to general QA is unconfirmed); measure on your own multi-hop eval set before committing to the agentic loop overhead
  • shipping vector/full-text migrations before verifying the target database supports the required extensions, text-search configs, generated columns, and index operators
  • mixing DDL, retrieval-function changes, and large content seeds in one migration so a search-feature failure blocks safe corpus rollout
  • upgrading a small passing lexical corpus to embeddings or search infrastructure just because it is on the long-term roadmap
  • reindexing synthesized reports or summaries as if they were authoritative primary evidence
  • picking an embedding or reranker vendor purely off a leaderboard snapshot without checking license terms (many open-weight rerankers ship non-commercial licenses) and ownership stability (reranker/embedding vendors are consolidating via acquisition; confirm the model's roadmap and support commitment survive a vendor change before building reindex-heavy dependencies on it)
  • treating retrieval recall problems and answer-faithfulness problems as one metric
  • relying on hosted retrieval defaults while assuming ACL, residency, freshness, and deletion semantics are handled
  • mixing tool outputs, crawled pages, uploaded files, and MCP resources without normalizing provenance and trust boundaries

Common Anti-Patterns

See references/quick-start-guide.md for detailed root-cause analysis of each anti-pattern below.

Vendor Recommendation Protocol

When users ask for "best" tools, models, or frameworks:

  1. Read data/sources.json and start from sources marked add_as_web_search: true.
  2. Verify claims against current primary docs, release notes, or official benchmarks.
  3. Prefer durable guidance: retrieval mode choice, operational tradeoffs, integration constraints, evaluation and rollback plan.
  4. If live browsing is unavailable, state that rankings, prices, and benchmark claims are unverified.

Research & Ingestion Pipelines

See references/research-and-ingestion-patterns.md for crawl/extract (Firecrawl-style), research loops (GPT Researcher-style), recurring data pipelines (dlt-style), and CLI evaluation workflows (simonw/llm-style). Core rule: separate crawl/extract from rank/retrieve, keep raw capture and chunks as distinct artifacts.

Scripts

check_sources.py — validate sources.json · retrieval_eval.py — recall@k/MRR/nDCG · check_citation_support.py — evidence-ID verification · generate_synthetic_rag_testset.py — testset scaffolds · late_interaction_eval.py — ColBERT/ColPali offline eval (no inference runner) · exact_search_baseline.py — exact cosine/dot baseline · hybrid_rrf_demo.py — BM25-lite + vector + RRF smoke test

When To Use This Skill

Use when the user asks about: choosing retrieval mode (file search / tools / SQL / vector RAG); designing a RAG or retrieval system; diagnosing retrieval quality; late interaction, reranking, or hybrid search; PDF, table, or diagram retrieval; grounding, citation, or faithfulness evaluation; or debugging freshness, ACL, or prompt-injection in retrieval.

Navigation

References (this skill)

Architecture & choice: retrieval-choice-framework.md · managed-retrieval-vs-self-hosted.md · pipeline-architecture.md · aws-bedrock-knowledge-bases.md

Corpus & chunking: chunking-strategies.md · chunking-patterns.md · index-selection-guide.md · embedding-model-guide.md · research-and-ingestion-patterns.md · pdf-heavy-retrieval-playbook.md

Retrieval patterns: retrieval-patterns.md · vector-search-patterns.md · hybrid-fusion-patterns.md · bm25-tuning.md · graph-rag-patterns.md · contextual-retrieval-guide.md · advanced-rag-patterns.md · agentic-rag-patterns.md

Ranking & query: ranking-pipeline-guide.md · query-rewriting-patterns.md · backend-comparison-fixtures.md

Grounding & eval: grounding-checklists.md · confidence-scoring.md · abstention-recipe.md · rag-evaluation-guide.md · search-evaluation-guide.md

Ops & debugging: observability-tracing-contract.md · retrieval-debugging-runbook.md · rag-troubleshooting.md · search-debugging.md · rag-caching-patterns.md · distributed-search-slos.md · user-feedback-learning.md · multilingual-domain-patterns.md · security-red-team-cases.md

Onboarding: quick-start-guide.md - workflow, full template index, detailed anti-patterns · wiki-grounded-retrieval.md

Cross-Skill References

External Sources

Related Skills

Gate before invoking: each foundation has a When to Apply / When to Skip section.

Fact-Checking

Verify bugs, framework footguns, and version-specific guidance against current primary web sources. Use web search for current vendor capabilities, prices, benchmarks, and release status. Mark unverified claims explicitly if live browsing is unavailable.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present). After applying it, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.