GrepAI Storage with GOB Skill

GrepAI Storage with GOB

This skill covers using GOB (Go Binary) as the storage backend for GrepAI, the default and simplest option.

When to Use This Skill

Single developer projects
Small to medium codebases
Simple setup without external dependencies
Local development environments

What is GOB Storage?

GOB is Go's native binary serialization format. GrepAI uses it to store:

Vector embeddings
File metadata
Chunk information

Everything is stored in a single local file.

Advantages

| Benefit | Description | |---------|-------------| | 🚀 Simple | No external services needed | | ⚡ Fast setup | Works immediately | | 📁 Portable | Single file, easy to backup | | 💰 Free | No infrastructure costs | | 🔒 Private | Data stays local |

Limitations

| Limitation | Description | |------------|-------------| | 📏 Scalability | Not ideal for very large codebases | | 👤 Single user | No concurrent access | | 🔄 No sharing | Can't share index across machines | | 💾 Memory | Loads into RAM for searches |

Configuration

Default Configuration

GOB is the default backend. Minimal config:

# .grepai/config.yaml
store:
  backend: gob

Explicit Configuration

store:
  backend: gob
  # Index stored in .grepai/index.gob (automatic)

Storage Location

GOB storage creates files in your project's .grepai/ directory:

.grepai/
├── config.yaml    # Configuration
├── index.gob      # Vector embeddings
└── symbols.gob    # Symbol index for trace

File Sizes

Approximate .grepai/index.gob sizes:

| Codebase | Files | Chunks | Index Size | |----------|-------|--------|------------| | Small | 100 | 500 | ~5 MB | | Medium | 1,000 | 5,000 | ~50 MB | | Large | 10,000 | 50,000 | ~500 MB |

Operations

Creating the Index

# Initialize project
grepai init

# Start indexing (creates index.gob)
grepai watch

Checking Index Status

grepai status

# Output:
# Index: .grepai/index.gob
# Files: 245
# Chunks: 1,234
# Size: 12.5 MB
# Last updated: 2025-01-28 10:30:00

Backing Up the Index

# Simple file copy
cp .grepai/index.gob .grepai/index.gob.backup

Clearing the Index

# Delete and re-index
rm .grepai/index.gob
grepai watch

Moving to a New Machine

# Copy entire .grepai directory
cp -r .grepai /path/to/new/location/

# Note: Only works if using same embedding model

Performance Considerations

Memory Usage

GOB loads the entire index into RAM for searches:

| Index Size | RAM Usage | |------------|-----------| | 10 MB | ~20 MB | | 50 MB | ~100 MB | | 500 MB | ~1 GB |

Search Speed

GOB provides fast searches for typical codebases:

| Codebase Size | Search Time | |---------------|-------------| | Small (100 files) | <50ms | | Medium (1K files) | <200ms | | Large (10K files) | <1s |

When to Upgrade

Consider PostgreSQL or Qdrant when:

Index exceeds 1 GB
Need concurrent access
Want to share index across team
Codebase has 50K+ files

.gitignore Configuration

Add .grepai/ to your .gitignore:

# GrepAI (machine-specific index)
.grepai/

Why: The index is machine-specific because:

Contains binary embeddings
Tied to the embedding model used
Each machine should generate its own

Sharing Index (Not Recommended)

While you can copy the index file, it's not recommended because:

Must use identical embedding model
File paths are absolute
Different machines may have different code versions

Better approach: Each developer runs their own grepai watch.

Migrating to Other Backends

To PostgreSQL

Update config:

store:
  backend: postgres
  postgres:
    dsn: postgres://user:pass@localhost:5432/grepai

Re-index:

rm .grepai/index.gob
grepai watch

To Qdrant

Update config:

store:
  backend: qdrant
  qdrant:
    endpoint: localhost
    port: 6334

Re-index:

rm .grepai/index.gob
grepai watch

Common Issues

❌ Problem: Index file too large ✅ Solution: Add more ignore patterns or migrate to PostgreSQL/Qdrant

❌ Problem: Slow searches on large codebase ✅ Solution: Migrate to Qdrant for better performance

❌ Problem: Corrupted index ✅ Solution: Delete and re-index:

rm .grepai/index.gob .grepai/symbols.gob
grepai watch

❌ Problem: "Index not found" error ✅ Solution: Run grepai watch to create the index

Best Practices

Use for small/medium projects: Up to ~10K files
Add to .gitignore: Don't commit the index
Backup before major changes: Copy index.gob before experiments
Re-index after model changes: If you change embedding models
Monitor file size: Migrate if index exceeds 1GB

Output Format

GOB storage status:

✅ GOB Storage Configured

   Backend: GOB (local file)
   Index: .grepai/index.gob
   Size: 12.5 MB

   Contents:
   - Files: 245
   - Chunks: 1,234
   - Vectors: 1,234 × 768 dimensions

   Performance:
   - Search latency: <100ms
   - Memory usage: ~25 MB

Agent Skills: GrepAI Storage with GOB

Install this agent skill to your local

Skill Files