Agent Skills: lmms-eval Codebase Guide

Guides AI coding agents through the lmms-eval codebase - a unified evaluation framework for Large Multimodal Models (LMMs). Use when integrating new models, adding evaluation tasks/benchmarks, running YAML config-driven evaluations, orchestrating non-blocking training-time evaluation via the HTTP eval server, or navigating the evaluation pipeline architecture.

ID: EvolvingLMMs-Lab/lmms-eval/lmms-eval-guide

Install this agent skill to your local

npx skills add EvolvingLMMs-Lab/lmms-eval --skill lmms-eval-guide

Skill Files

The files that make up lmms-eval-guide.

Download Skill

Loading file tree…

skills/lmms-eval-guide/SKILL.md

Skill Metadata

Name
lmms-eval-guide
Description
Guides AI coding agents through the lmms-eval codebase - a unified evaluation framework for Large Multimodal Models (LMMs). Use when integrating new models, adding evaluation tasks/benchmarks, running YAML config-driven evaluations, orchestrating non-blocking training-time evaluation via the HTTP eval server, or navigating the evaluation pipeline architecture.

lmms-eval Codebase Guide

lmms-eval evaluates Large Multimodal Models across image, video, and audio tasks. 95 model backends (14 chat + 81 simple/legacy), 230 task directories, 1377 YAML configs.

Scope of This Skill

This file is the routing layer for agents. It gives architecture context, decision points, and fast guardrails.

Detailed implementation lives in references/*.md.

Use this skill to cover end-to-end workflows:

  1. Add or update model backends.
  2. Add or update task/benchmark definitions.
  3. Run evaluations via YAML configs (--config).
  4. Insert lmms-eval into training jobs via async API server calls.
  5. Operate HTTP eval service (/evaluate, /jobs/{job_id}, /queue, /tasks, /models).
  6. Verify small-sample eval before large-scale runs.
  7. Run full-scale evaluations with proper cache, batch, and seed settings.
  8. Debug pipeline failures systematically.

When to Use This Skill

Use this skill when requests include any of the following intents:

  • "Add a new model" / "integrate model backend"
  • "Add a new task/benchmark" / "write task yaml"
  • "Run evaluation with YAML config" / "set up --config"
  • "Insert lmms-eval into training job"
  • "Run eval asynchronously without blocking training"
  • "Start or debug HTTP eval server"
  • "Configure reasoning tag stripping" / "<think> tags in output"
  • "Run full evaluation" / "production eval run"
  • "Debug eval failure" / "eval is broken" / "something failed"
  • "Where do I edit X?" / "which file handles Y?"
  • "Upgrade from v0.6" / "breaking changes"

Instructions (Execution Order)

  1. Classify the request into one of: model extension, task extension, config setup, training integration, service operations, debugging.
  2. Load the matching reference(s) from the routing matrix below.
  3. Follow existing patterns in neighboring files before writing new code/config.
  4. Validate with a smoke eval (--limit 5 or --limit 8) before broad runs.
  5. For YAML config workflows, use --config with CLI overrides for quick iteration.
  6. For training loops, use async HTTP job submission and collect by job_id.
  7. For queue or failure issues, inspect /queue and /jobs/{job_id} and handle terminal states.

Setup

uv sync && uv run pre-commit install
# Quick eval test
python -m lmms_eval --model qwen2_5_vl --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct --tasks mme --batch_size 1 --limit 8
# Lint
uv run pre-commit run --all-files

Architecture

lmms_eval/
├── __main__.py              # CLI entry (python -m lmms_eval)
├── cli/                     # Subcommand dispatch (eval, tasks, models, ui, serve, power, version)
├── evaluator.py             # Core evaluation loop + reasoning tag stripping
├── protocol.py              # ChatMessages - multimodal message protocol
├── api/
│   ├── model.py             # Base class `lmms` - all models subclass this
│   ├── instance.py          # `Instance` - request object passed to models
│   ├── task.py              # ConfigurableTask - task loading from YAML
│   ├── reasoning.py         # strip_reasoning_tags(), parse_reasoning_tags_config()
│   └── registry.py          # @register_model, @register_task decorators
├── models/
│   ├── __init__.py           # Private built-in declarations + read-only compatibility views
│   ├── registry_v2.py        # ModelManifest, ModelRegistryV2 - resolution prefers chat over simple
│   ├── chat/                 # Chat models (14+, RECOMMENDED for new models)
│   │   ├── async_openai.py   # OpenAI-compatible API (message_format parameter)
│   │   └── ...               # qwen2_5_vl, qwen3_vl, vllm, sglang, etc.
│   └── simple/               # Legacy models (81)
├── tasks/                    # Auto-registered from YAML (230 dirs, 1377 configs)
├── entrypoints/              # HTTP eval server (FastAPI)
│   ├── http_server.py        # REST endpoints: /evaluate, /jobs/{job_id}, /queue, /tasks, /models
│   ├── client.py             # EvalClient (sync), AsyncEvalClient
│   └── protocol.py           # EvaluateRequest, JobInfo
├── llm_judge/                # LLM-as-judge scoring
└── tui/                      # Web UI

Evaluation Pipeline

CLI --model X --tasks Y  (or --config experiment.yaml)
  1. CONFIG: merge defaults < YAML < CLI overrides -> resolved_cli_args
  2. MODEL: ModelRegistryV2.resolve() -> dynamic import -> cls.create_from_arg_string()
  3. TASKS: scan tasks/ for YAML -> ConfigurableTask(config)
  4. REQUESTS: build Instance objects per doc
     Chat models: (doc_to_messages, gen_kwargs, doc_id, task, split)  # 5 elements
     Simple models: (contexts, gen_kwargs, doc_to_visual, doc_id, task, split)  # 6 elements
  5. INFERENCE: model.generate_until(requests) or model.loglikelihood(requests)
  6. REASONING STRIP: strip_reasoning_tags(filtered_resps) if --reasoning_tags enabled
  7. SCORING: task.process_results(doc, filtered_resps) -> {metric: value}
  8. AGGREGATION: mean, stderr, confidence intervals -> results JSON

ChatMessages Protocol (protocol.py)

Structured multimodal message format for chat models:

# Content types: text, image, video, audio
# Message: {"role": "user"|"system"|"assistant", "content": [{type, ...}, ...]}
messages = ChatMessages(messages=raw_messages)
images, videos, audios = messages.extract_media()
hf_messages = messages.to_hf_messages()      # for apply_chat_template()
oai_messages = messages.to_openai_messages()  # for OpenAI API

Model Registration

In-tree models use private bootstrap declarations in models/__init__.py:

  • _BUILTIN_CHAT_TEMPLATE_MODELS - chat models in models/chat/
  • _BUILTIN_SIMPLE_MODELS - simple models in models/simple/
  • _BUILTIN_MODEL_ALIASES - backward-compatible names

External packages should expose a ModelManifest through the lmms_eval.models entry-point group. Registration runs at startup. AVAILABLE_SIMPLE_MODELS, AVAILABLE_CHAT_TEMPLATE_MODELS, MODEL_ALIASES, and AVAILABLE_MODELS are read-only compatibility views. Registry resolution prefers chat over simple when both exist.

Reference Routing Matrix

Load only the reference(s) relevant to your current task.

You need to doLoad this referenceWhy
Add a new model backendreferences/models.mdModel class template, registration, message_format, validation
Add a new benchmark/taskreferences/tasks.mdYAML schema, reasoning_tags, utils.py contracts, aggregation patterns
Start/operate HTTP eval servicereferences/api-server.mdServer args, endpoints, client contracts
Insert eval into training jobs (non-blocking)references/api-server.mdTraining-loop integration patterns using async job submission
Diagnose queue/job statesreferences/api-server.mdJob lifecycle, polling, cancellation, queue inspection
Run with YAML config / smoke tests / full evaluationsreferences/workflows.md--config usage, smoke test scripts, full run recipes, cache strategy
Debug pipeline failuresreferences/workflows.mdStep-by-step debug workflow, failure classification, diagnostic commands
Find which file to editreferences/workflows.mdEdit-routing table by intent (model, task, evaluator, output)

Multi-Track Requests

For requests spanning multiple tracks (for example, new model + new task + training integration), combine relevant references and execute in this order:

  1. Model/task implementation.
  2. Smoke validation.
  3. HTTP service orchestration.

Key CLI Flags

FlagDescription
--configYAML config file (replaces long CLI one-liners, see workflows.md)
--modelModel backend (e.g., qwen2_5_vl, openai, vllm)
--model_argsComma key=value pairs (e.g., pretrained=org/model,device_map=auto)
--tasksComma-separated task names
--limit NEvaluate first N samples only (always use when testing)
--batch_size NBatch size for inference
--log_samplesSave per-sample predictions
--reasoning_tagsControl <think> tag stripping: default <think>...</think>, none to disable, or custom JSON pairs
--verbosity DEBUGDetailed logging

Environment Variables

# Core
export OPENAI_API_KEY="..."                    # API-backed models
export HF_TOKEN="..."                          # Gated HuggingFace datasets
export HF_HOME="/path/to/cache"                # HF cache directory

# Video decode backends
export LMMS_VIDEO_DECODE_BACKEND="pyav"         # pyav (default) | torchcodec | dali
export LMMS_VIDEO_TORCHCODEC_THREADS="8"       # Thread count for TorchCodec (MUST set explicitly)
export LMMS_VIDEO_DALI_DEVICE="gpu"             # GPU decode with DALI

# MINERVA Lance mode
export MINERVA_LANCE_VIDEO_URI="hf://datasets/lmms-lab-eval/minerva/data/train.lance"
export MINERVA_VIDEO_DIR="/path/to/videos"      # Local-first mode

# Safety tasks
export PERSPECTIVE_API_KEY="..."                # Toxicity scoring (optional, falls back to keyword heuristic)

Common Errors

ErrorFix
ValueError: gen_kwargs['until']until must be str or list[str]
NotImplementedError: loglikelihoodImplement loglikelihood() or use generate_until tasks
Visual is None or []Guard: if visual is not None and len(visual) > 0
is_simple mismatchSet is_simple = False on chat model classes
Unknown keys in config fileTypo in YAML config key. Error message lists valid keys.
<think> tokens in scored outputEnable --reasoning_tags (default) or check per-task reasoning_tags override

Upgrading from v0.6

ChangeMigration
async_openai_qwen3_vl model removedUse async_openai with message_format=qwen3_vl in --model_args
is_qwen3_vl flag removedUse message_format=qwen3_vl instead
parse_reasoning_model_answer removedReasoning stripping is now pipeline-level via --reasoning_tags
read_video_pyav renamedUse read_video. Backward-compat alias exists.

Constraints

  • Package manager: uv only, never pip
  • Linting and formatting: Ruff checks, sorts imports, and formats Python (line-length=240). Run uv run pre-commit run --all-files
  • Testing: Always use --limit 5 or --limit 8 when testing changes
  • Follow patterns: Match the style of neighboring files exactly

More agent skills