Whisper Audio Intelligibility Test Skill

Whisper Audio Intelligibility Test

Transcribe WAV audio files using OpenAI Whisper and report whether the speech is intelligible. Optionally compare against expected text.

Setup

Whisper is installed as a uv tool: uv tool install openai-whisper.

Since this machine may lack ffmpeg, always use the Python API approach that loads WAV files with scipy (bypasses the ffmpeg requirement).

Running Transcription

Use uv run --no-project --with openai-whisper --with scipy --python 3.11 to execute the transcription script:

uv run --no-project --with openai-whisper --with scipy --python 3.11 \
  python3 ~/.claude/skills/whisper-test/transcribe.py \
  [--model tiny|base|small|medium|large-v3] \
  [--language en] \
  [--expected "expected text"] \
  [--json] \
  file1.wav [file2.wav ...]

Arguments

--model: Whisper model size (default: large-v3). See model selection guide below.
--language: Language hint (default: en).
--expected: Expected transcription text. When provided, calculates Word Error Rate (WER).
--json: Output results as JSON instead of human-readable text.
Positional: One or more WAV file paths.

Model Selection

Use large-v3 for TTS quality verification. Smaller models hallucinate or miss words in synthesized speech, making them unreliable for judging output quality.

| Model | VRAM | When to use | | ---------- | ------ | --------------------------------------------------------------- | | large-v3 | ~10 GB | Default. TTS evaluation, quality gating, regression testing | | medium | ~5 GB | GPU memory constrained, still decent accuracy | | small | ~2 GB | Quick smoke tests only | | base | ~1 GB | Not recommended for TTS — high hallucination rate | | tiny | ~1 GB | Not recommended for TTS — unreliable |

Observed with identical Qwen3-TTS 1.7B voice-cloned output:

large-v3: "That's one tank. Flash attention pipeline." (key phrase captured)
base: "That's one thing, flash attention pipeline." (close but hallucinated)

For poor-quality 0.6B output, base hallucinated "Charging Wheel" while large-v3 gave "Flat, splashes." — honest about the poor quality instead of confabulating plausible words.

Output Format

For each file, prints:

filename.wav:
  transcription: "Hello world, this is a test."
  duration: 2.96s
  rms: 0.0866
  peak: 0.6832
  silence: 49.2%
  [wer: 0.0%]  (if --expected provided)

Interpreting Results

| Transcription | Meaning | | --------------------- | ------------------------------------------------------------------------------- | | Matches expected text | Audio is intelligible and correct | | Partial match | Audio has some speech but quality issues | | Empty string "" | Audio is unintelligible (noise, silence, or garbage) | | Hallucinated text | Model heard something in noise (common with Whisper, especially smaller models) |

Audio Quality Indicators

RMS < 0.01: Essentially silent
silence > 80%: Mostly silence, likely no speech
peak < 0.05: Very quiet, may not contain useful audio

TTS-Specific Patterns

Voice-cloned TTS output often has these characteristics:

Garbled opening, clear ending: Common with ICL voice cloning on short references. The model needs a few frames to "lock in" to the target voice.
Key phrases preserved: Even when WER is high, domain-specific terms (e.g. "flash attention pipeline") often come through clearly.
Smaller models produce worse audio: 0.6B models produce significantly less intelligible output than 1.7B — expect Whisper to reflect this.

Batch Testing (TTS Variant Comparison)

When testing multiple TTS outputs against expected text:

uv run --no-project --with openai-whisper --with scipy --python 3.11 \
  python3 ~/.claude/skills/whisper-test/transcribe.py \
  --expected "Hello world, this is a test." \
  variant1.wav variant2.wav variant3.wav

This produces a comparison table showing which variants produce intelligible speech.

Docker / NGC Container Usage

When testing on a GPU box inside an NGC container (e.g. for CUDA flash-attn builds), ffmpeg isn't available and apt can be slow. Two workarounds:

Static ffmpeg binary (fast, no apt):

curl -sL https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-arm64-static.tar.xz \
  | tar xJ --strip-components=1 -C /usr/local/bin/ --wildcards "*/ffmpeg" "*/ffprobe"
pip install openai-whisper

Use scipy loader (this script's default — no ffmpeg needed):

pip install openai-whisper scipy
python3 ~/.claude/skills/whisper-test/transcribe.py --model large-v3 output.wav

The script loads WAV files directly via scipy, bypassing Whisper's ffmpeg dependency entirely. This works for WAV files (the standard TTS output format).

Agent Skills: Whisper Audio Intelligibility Test

Install this agent skill to your local

Skill Files