GrepAI Storage with Qdrant Skill

GrepAI Storage with Qdrant

This skill covers using Qdrant as the storage backend for GrepAI, offering high-performance vector search.

When to Use This Skill

Need fastest possible search performance
Very large codebases (50K+ files)
Already using Qdrant infrastructure
Want advanced vector search features

What is Qdrant?

Qdrant is a purpose-built vector database offering:

⚡ Extremely fast vector similarity search
📏 Excellent scalability
🔧 Advanced filtering capabilities
🐳 Easy Docker deployment

Prerequisites

Qdrant server running
Network access to Qdrant

Advantages

| Benefit | Description | |---------|-------------| | ⚡ Performance | Fastest vector search | | 📏 Scalability | Handles millions of vectors | | 🔍 Advanced | Filtering, payloads, sharding | | 🐳 Easy deploy | Docker-ready | | ☁️ Cloud option | Qdrant Cloud available |

Setting Up Qdrant

Option 1: Docker (Recommended)

# Run Qdrant with persistent storage
docker run -d \
  --name grepai-qdrant \
  -p 6333:6333 \
  -p 6334:6334 \
  -v qdrant_storage:/qdrant/storage \
  qdrant/qdrant

Ports:

6333: REST API
6334: gRPC API (used by GrepAI)

Option 2: Docker Compose

# docker-compose.yml
version: '3.8'
services:
  qdrant:
    image: qdrant/qdrant
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - qdrant_storage:/qdrant/storage
    environment:
      - QDRANT__SERVICE__GRPC_PORT=6334

volumes:
  qdrant_storage:

docker-compose up -d

Option 3: Qdrant Cloud

Sign up at cloud.qdrant.io
Create a cluster
Get your endpoint and API key

Configuration

Basic Configuration (Local)

# .grepai/config.yaml
store:
  backend: qdrant
  qdrant:
    endpoint: localhost
    port: 6334

With TLS (Production)

store:
  backend: qdrant
  qdrant:
    endpoint: qdrant.company.com
    port: 6334
    use_tls: true

With API Key (Qdrant Cloud)

store:
  backend: qdrant
  qdrant:
    endpoint: your-cluster.aws.cloud.qdrant.io
    port: 6334
    use_tls: true
    api_key: ${QDRANT_API_KEY}

Set the environment variable:

export QDRANT_API_KEY="your-api-key"

Configuration Options

| Option | Default | Description | |--------|---------|-------------| | endpoint | localhost | Qdrant server hostname | | port | 6334 | gRPC port | | use_tls | false | Enable TLS encryption | | api_key | none | Authentication key |

Verifying Setup

Check Qdrant is Running

# REST API health check
curl http://localhost:6333/health

# Expected: {"status":"ok"}

Check Collections (after indexing)

# List collections
curl http://localhost:6333/collections

# Get collection info
curl http://localhost:6333/collections/grepai

From GrepAI

grepai status

# Should show Qdrant backend info

Qdrant Dashboard

Access the web dashboard at http://localhost:6333/dashboard:

View collections
Browse vectors
Execute queries
Monitor performance

Performance Characteristics

Search Latency

| Codebase Size | Vectors | Search Time | |---------------|---------|-------------| | Small (1K files) | 5,000 | <10ms | | Medium (10K files) | 50,000 | <20ms | | Large (100K files) | 500,000 | <50ms |

Memory Usage

Qdrant loads vectors into memory for fast search:

| Vectors | Dimensions | Memory | |---------|------------|--------| | 10,000 | 768 | ~60 MB | | 100,000 | 768 | ~600 MB | | 1,000,000 | 768 | ~6 GB |

Advanced Configuration

Qdrant Server Configuration

Create config/production.yaml:

storage:
  storage_path: /qdrant/storage

service:
  grpc_port: 6334
  http_port: 6333
  max_request_size_mb: 32

optimizers:
  memmap_threshold_kb: 200000
  indexing_threshold_kb: 50000

Mount in Docker:

docker run -d \
  -v ./config:/qdrant/config \
  -v qdrant_storage:/qdrant/storage \
  qdrant/qdrant

Collection Settings

GrepAI creates a collection named grepai with:

Vector size: matches your embedding dimensions
Distance: Cosine similarity
On-disk storage for large datasets

Clustering (Advanced)

For very large deployments, Qdrant supports distributed mode:

# qdrant config
cluster:
  enabled: true
  p2p:
    port: 6335

Backup and Restore

Snapshot Creation

# Create snapshot via REST API
curl -X POST 'http://localhost:6333/collections/grepai/snapshots'

Restore Snapshot

# Restore from snapshot
curl -X PUT 'http://localhost:6333/collections/grepai/snapshots/recover' \
  -H 'Content-Type: application/json' \
  -d '{"location": "/path/to/snapshot"}'

Migrating from GOB

Start Qdrant:

docker run -d --name qdrant -p 6333:6333 -p 6334:6334 qdrant/qdrant

Update configuration:

store:
  backend: qdrant
  qdrant:
    endpoint: localhost
    port: 6334

Delete old index:

rm .grepai/index.gob

Re-index:

grepai watch

Migrating from PostgreSQL

Start Qdrant
Update configuration to use Qdrant
Re-index (embeddings must be regenerated)

Common Issues

❌ Problem: Connection refused ✅ Solution: Ensure Qdrant is running:

docker ps | grep qdrant
docker start grepai-qdrant

❌ Problem: gRPC connection failed ✅ Solution: Check port 6334 is exposed:

docker run -p 6334:6334 ...

❌ Problem: Authentication failed ✅ Solution: Check API key:

echo $QDRANT_API_KEY

❌ Problem: Out of memory ✅ Solutions:

Enable on-disk storage in Qdrant config
Increase Docker memory limit
Use Qdrant Cloud for managed scaling

❌ Problem: Slow initial indexing ✅ Solution: This is normal; Qdrant optimizes in background. Searches will be fast after indexing completes.

Qdrant vs PostgreSQL

| Feature | Qdrant | PostgreSQL | |---------|--------|------------| | Search speed | ⚡⚡⚡ | ⚡⚡ | | Setup complexity | Easy (Docker) | Medium | | SQL queries | ❌ | ✅ | | Scalability | Excellent | Good | | Memory efficiency | Excellent | Good | | Team familiarity | Lower | Higher |

Recommendation: Use Qdrant for large codebases or maximum performance. Use PostgreSQL if you need SQL integration or team is familiar with it.

Best Practices

Use persistent volume: Mount /qdrant/storage
Enable TLS in production: Set use_tls: true
Secure API key: Use environment variables
Monitor memory: Vector search is memory-intensive
Regular snapshots: Backup before major changes

Output Format

Qdrant storage status:

✅ Qdrant Storage Configured

   Backend: Qdrant
   Endpoint: localhost:6334
   TLS: disabled
   Collection: grepai

   Contents:
   - Files: 5,000
   - Vectors: 25,000
   - Dimensions: 768

   Performance:
   - Connection: OK
   - Indexed: Yes
   - Search latency: ~15ms

Agent Skills: GrepAI Storage with Qdrant

Install this agent skill to your local

Skill Files