Agent Skills: ML Ops Skill

Entry point: /faion-net — invoke this skill for automatic routing to the appropriate domain.

ML Ops Skill

Communication: User's language. Code: English.

Purpose

Handles ML model operations. Covers fine-tuning, evaluation, cost management, and observability.

Context Discovery

Auto-Investigation

Check these project signals before asking questions:

| Signal | Where to Check | What to Look For | |--------|----------------|------------------| | Dependencies | requirements.txt | transformers, peft, openai, tiktoken, langsmith | | Training data | /data, /datasets | JSONL files for fine-tuning | | Logs/metrics | Grep for "langsmith", "wandb", "mlflow" | Existing observability tools | | Cost tracking | Grep for "tiktoken", "count_tokens" | Token counting implementation |

Discovery Questions

question: "What ML operation are you working on?"
header: "Operation Type"
multiSelect: false
options:
  - label: "Fine-tuning LLM"
    description: "Custom model training (OpenAI API, LoRA, QLoRA)"
  - label: "Model evaluation"
    description: "Benchmark performance, LLM-as-judge"
  - label: "Cost optimization"
    description: "Reduce API costs, prompt caching, batching"
  - label: "Observability/monitoring"
    description: "Track LLM usage, traces, performance"

question: "For fine-tuning: dataset size and approach?"
header: "Fine-tuning Strategy"
multiSelect: false
options:
  - label: "<100 examples - use few-shot prompting instead"
    description: "Too small for fine-tuning, improve prompts"
  - label: "100-1000 examples - OpenAI fine-tuning"
    description: "Use OpenAI API fine-tuning endpoint"
  - label: ">1000 examples - LoRA/QLoRA"
    description: "Efficient parameter fine-tuning"
  - label: "Not fine-tuning"
    description: "Skip this question"

question: "Which observability tools?"
header: "Monitoring Stack"
multiSelect: true
options:
  - label: "LangSmith (recommended)"
    description: "LangChain native tracing"
  - label: "Langfuse (open-source)"
    description: "Self-hosted observability"
  - label: "Custom logging"
    description: "Build custom tracking"
  - label: "None yet"
    description: "Starting from scratch"

Scope

| Area | Coverage | |------|----------| | Fine-tuning | LoRA, QLoRA, OpenAI fine-tuning, datasets | | Evaluation | Metrics, benchmarks, frameworks | | Cost Optimization | Token management, caching, batch APIs | | Observability | LLM monitoring, tracing, logging |

Quick Start

| Task | Files | |------|-------| | Fine-tune OpenAI | fine-tuning-openai-basics.md → fine-tuning-openai-production.md | | Fine-tune LoRA | lora-qlora.md → finetuning-basics.md | | Cost optimization | llm-cost-basics.md → cost-reduction-strategies.md | | Evaluation | evaluation-metrics.md → evaluation-framework.md | | Observability | llm-observability.md → llm-observability-stack-2026.md |

Methodologies (15)

Fine-tuning (5):

• finetuning-basics: Fundamentals, when to fine-tune
• finetuning-datasets: Data preparation, quality
• fine-tuning-openai-basics: OpenAI API fine-tuning
• fine-tuning-openai-production: Production deployment
• lora-qlora: Efficient fine-tuning, parameter selection

Evaluation (3):

• evaluation-metrics: Accuracy, F1, perplexity, task metrics
• evaluation-framework: LLM-as-judge, human eval
• evaluation-benchmarks: MMLU, HumanEval, industry benchmarks

Cost Optimization (2):

• llm-cost-basics: Token counting, pricing models
• cost-reduction-strategies: Caching, compression, batching

Observability (5):

• llm-observability: Fundamentals, why monitor
• llm-observability-stack: Tools selection
• llm-observability-stack-2026: Latest tools (LangSmith, Langfuse)
• llm-management-observability: End-to-end management

Code Examples

OpenAI Fine-tuning

from openai import OpenAI

client = OpenAI()

# Upload training data
file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={"n_epochs": 3}
)

# Monitor
while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    if job.status == "succeeded":
        break

LoRA Fine-tuning

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8b")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
    bias="none"
)

model = get_peft_model(model, lora_config)

Cost Tracking

import tiktoken

def count_tokens(text, model="gpt-4o"):
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

def estimate_cost(prompt, completion, model="gpt-4o"):
    prompt_tokens = count_tokens(prompt, model)
    completion_tokens = count_tokens(completion, model)

    # GPT-4o pricing
    prompt_cost = prompt_tokens * 0.000005
    completion_cost = completion_tokens * 0.000015

    return prompt_cost + completion_cost

LLM Observability with LangSmith

from langsmith import traceable

@traceable
def rag_pipeline(query: str) -> str:
    # Retrieval
    docs = retrieve(query)

    # Generation
    response = generate(query, docs)

    return response

Fine-tuning Decision Matrix

| Scenario | Approach | |----------|----------| | Small dataset (<100 examples) | Few-shot prompting | | Medium dataset (100-1000) | OpenAI fine-tuning | | Large dataset (>1000) | LoRA/QLoRA | | Custom behavior | Fine-tuning | | New knowledge | RAG (not fine-tuning) |

Cost Reduction Strategies

| Strategy | Savings | Trade-off | |----------|---------|-----------| | Prompt caching | 90% on cached | Cold start cost | | Batch API | 50% | 24h latency | | Smaller models | 80%+ | Lower quality | | Context pruning | Variable | May lose context | | Output limits | Variable | Truncated responses |

Evaluation Frameworks

| Framework | Use Case | |-----------|----------| | LangSmith | Production monitoring, traces | | Langfuse | Open-source observability | | PromptLayer | Prompt versioning | | Weights & Biases | Experiment tracking |

Related Skills

| Skill | Relationship | |-------|-------------| | faion-llm-integration | Provides APIs to optimize | | faion-rag-engineer | RAG evaluation | | faion-devops-engineer | Model deployment |

ML Ops v1.0 | 15 methodologies