RAG Engineer Skill Skill

Entry point: /faion-net — invoke this skill for automatic routing to the appropriate domain.

RAG Engineer Skill

Communication: User's language. Code: English.

Purpose

Specializes in RAG (Retrieval Augmented Generation) systems. Covers document processing, embeddings, vector search, and retrieval optimization.

Context Discovery

Auto-Investigation

Check these project signals before asking questions:

| Signal | Where to Check | What to Look For | |--------|----------------|------------------| | Dependencies | package.json, requirements.txt | langchain, llamaindex, qdrant-client, chromadb, weaviate-client | | Vector DB | docker-compose.yml, .env | Qdrant, Weaviate, Chroma config/containers | | Document dirs | /docs, /data, /content | Documents to index (PDF, MD, TXT) | | Existing embeddings | Grep for "embed", "vector", "retriever" | Current RAG implementation |

Discovery Questions

question: "What's your RAG use case?"
header: "RAG Goal"
multiSelect: false
options:
  - label: "Documentation Q&A"
    description: "Answer questions from internal docs"
  - label: "Knowledge base search"
    description: "Semantic search over articles/guides"
  - label: "Code search/retrieval"
    description: "Find relevant code snippets"
  - label: "Customer support"
    description: "Context-aware support responses"

question: "Which vector database?"
header: "Vector DB"
multiSelect: false
options:
  - label: "Qdrant (recommended for production)"
    description: "Fast, scalable, rich filtering"
  - label: "Chroma (recommended for dev/prototyping)"
    description: "Simple, local, easy setup"
  - label: "Weaviate (for knowledge graphs)"
    description: "Hybrid search, graph features"
  - label: "pgvector (for PostgreSQL projects)"
    description: "Vector extension for existing Postgres"

question: "Document volume and type?"
header: "Data Characteristics"
multiSelect: false
options:
  - label: "Small (<1000 docs, mostly text)"
    description: "Simple chunking sufficient"
  - label: "Medium (1000-10000 docs)"
    description: "Consider hybrid search + reranking"
  - label: "Large (>10000 docs, mixed formats)"
    description: "Advanced chunking + metadata filtering"
  - label: "Code repository"
    description: "AST-aware chunking needed"

question: "Do you need hybrid search (vector + keyword)?"
header: "Search Strategy"
multiSelect: false
options:
  - label: "Yes - combine semantic + exact matching"
    description: "Hybrid search for best results"
  - label: "No - semantic search only"
    description: "Vector similarity sufficient"

Scope

| Area | Coverage | |------|----------| | Chunking | Text splitting, semantic chunking, overlap strategies | | Embeddings | Text vectorization, similarity search, models | | Vector DBs | Qdrant, Weaviate, Chroma, pgvector | | Retrieval | Hybrid search, reranking, metadata filtering | | RAG Systems | Architecture, evaluation, agentic RAG |

Quick Start

| Task | Files | |------|-------| | Basic RAG | chunking-basics.md → embedding-basics.md → rag-architecture.md | | Vector DB setup | db-comparison.md → db-qdrant.md (recommended) | | Advanced retrieval | hybrid-search-basics.md → reranking-basics.md | | RAG evaluation | rag-eval-metrics.md → rag-eval-methods.md | | Agentic RAG | agentic-rag.md |

Methodologies (22)

Chunking (2):

chunking-basics: Size, overlap, delimiters
chunking-advanced: Semantic, recursive, custom

Embeddings (4):

embedding-basics: Fundamentals, similarity
embedding-generation: API usage, batching
embedding-models: Comparison, selection
embedding-applications: Use cases, patterns

Vector Databases (4):

db-comparison: Feature comparison, selection
db-qdrant: Setup, indexing, search (recommended)
db-weaviate: Knowledge graphs, hybrid search
db-chroma: Local dev, prototyping
vector-database-setup: General setup patterns

Retrieval (4):

hybrid-search-basics: Vector + keyword search
hybrid-search-implementation: Production patterns
reranking-basics: Cross-encoder fundamentals
reranking-models: Cohere, MixedBread, custom

RAG Systems (7):

rag: RAG overview, fundamentals
rag-architecture: System design, components
rag-implementation: Production patterns
rag-eval-metrics: Relevance, faithfulness, correctness
rag-eval-methods: Evaluation frameworks
agentic-rag: Agent-driven retrieval
graph-rag-advanced-retrieval: Knowledge graphs

Architecture

Document Ingestion
    ↓
Chunking (semantic/fixed)
    ↓
Embedding Generation
    ↓
Vector Database Storage
    ↓
Query Processing
    ↓
Retrieval (vector + hybrid)
    ↓
Reranking
    ↓
Context Assembly
    ↓
LLM Generation

Code Examples

Basic RAG Pipeline

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

# Chunk documents
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = splitter.split_documents(docs)

# Generate embeddings and store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings)

# Retrieve
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 5}
)
results = retriever.invoke("query")

Hybrid Search with Qdrant

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, Filter

client = QdrantClient("localhost", port=6333)

# Create collection
client.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE)
)

# Hybrid search
results = client.search(
    collection_name="docs",
    query_vector=query_embedding,
    query_filter=Filter(...),
    limit=10
)

Reranking

from cohere import Client

co = Client(api_key="...")

# Rerank retrieved docs
reranked = co.rerank(
    query="query text",
    documents=[doc.text for doc in results],
    top_n=3,
    model="rerank-english-v3.0"
)

Evaluation Metrics

| Metric | Measures | |--------|----------| | Retrieval Precision | Relevant docs in results | | Retrieval Recall | Coverage of relevant docs | | MRR | Mean reciprocal rank | | NDCG | Ranking quality | | Faithfulness | Grounding in context | | Answer Relevance | Response matches query |

Related Skills

| Skill | Relationship | |-------|-------------| | faion-llm-integration | Uses embedding APIs | | faion-ai-agents | Agentic RAG patterns | | faion-ml-ops | RAG evaluation |

RAG Engineer v1.0 | 22 methodologies

Agent Skills: RAG Engineer Skill

Install this agent skill to your local

Skill Files