Agent Skills: video

• Response language follows language setting in .agents/oma-config.yaml if configured.
• Follow .agents/skills/_shared/core/execution-policy.md for authorization, clarification, verification, and completion. Execute required steps on the selected path in dependency order; apply documented branch and skip conditions.
• Key-optional by default (backend rule 11). The baseline path uses zero external API keys: the agent writes the script, oma-voice does TTS, oma-image does visuals, captions are key-free, Remotion composites. Every paid upgrade (Pexels stock, Pixelle AIGC) sits behind a key-free fallback and is off by default. Never disable the fallback to force a real call, and never silently drop a requested real path.
• Determinism boundary = render-spec.json + asset files (+ seed + embedded Pretendard). "Reproducible from script/assets, not from brief." Never edit assets or render-spec by hand after generation; re-run the stage that produced them.
• Demo capture is human-supplied. --source web --url gives URL context only; it neither opens nor records a browser. The human records the flow with Cap or another recorder, then supplies --capture <path>. Never automate login or capture credentials. Mask URL query tokens in logs and manifests.
• The oma video CLI owns the pipeline. This workflow owns the brief, the agent-authored script, the QA loop, and decision checkpoints. Do NOT reimplement orchestration, provider selection, or rendering in the workflow.
• Code intelligence and state are separate capabilities. Follow .agents/skills/_shared/core/code-intelligence.md for configured tools and native fallback. Follow .agents/skills/_shared/runtime/memory-protocol.md for run tracking and result paths. Do not require an MCP memory tool or automatically install, initialize, or track a repository.
• Read the oma-video skill BEFORE starting. Read .agents/skills/oma-video/SKILL.md and follow its Core Rules and execution protocol, including resources/execution-protocol.md. If the skill is not installed, stop and ask the user to run oma install first.

Vendor note: This workflow executes inline. The script-authoring step (Step 3) is performed by the running agent itself — the agent is the LLM key (agent-as-key). A subagent may be spawned only for broad multi-scene research; the asset pipeline (Steps 4-7) runs through the oma video CLI, not through subagents.

L1 Decision Events

Emit required L1 decisions by calling oma state emit directly, as documented in .agents/skills/_shared/runtime/event-spec.md.

This workflow has two required checkpoints: mode-selection (Step 2) and cost-confirmation (Step 5). Do not skip either emit/verify pair.

Mode Routing

Resolve the mode first — it determines aspect, source, visual track, and compositor. If the user did not name a mode, infer from intent and resolve material ambiguity at Step 2.

| mode | aspect | source | visual track (default → opt) | compositor | output | |------|:---:|------|------|------|------| | shorts | 9:16 | synthetic (topic → clip) | oma-image stills · Pexels (opt) · Pixelle AIGC (opt) | Remotion · MPT alt | shorts-<slug>.mp4 | | explainer | 16:9 / 9:16 | README · code · data | oma-slide frames + oma-image diagrams + code | Remotion (deterministic) | explainer-<slug>.mp4 | | demo | 16:9 | Human recording via --capture; --source web --url adds URL context | raw footage (default) · Remotion intro · zoom · callouts (--polish) | Remotion polish | demo-<slug>.mp4 |

Intent heuristics: "reel / TikTok / short / hook" → shorts; "walkthrough / how it works / from the README / explain the architecture" → explainer; "record / screen / show the app running / product demo" → demo.

For demo, resolve the recording path first. Use --source file --capture <path> for a recording; --source web --url <url> --capture <path> adds URL context to that human recording. The CLI does not open or record a browser. Raw footage is the default output; --polish overlays the Remotion Demo composition.

Cost Guardrail & Key-Optional Notes (read before Step 4)

• Guardrail: default cost.guardrail_usd: 0.20 in .agents/skills/oma-video/config/video-config.yaml (reused from oma-image). Any provider whose estimated cost meets or exceeds the guardrail requires spend authorization (-y / --yes or the Step 5 checkpoint). Reuse an existing authorization covering that provider and amount. --max-usd <n> overrides the threshold.

• Key-optional pairs (real path is gated; fallback is always wired):

  | capability | real (key/resource) | key-free fallback | deferred marker | |------|------|------|------| | stock video | Pexels (PEXELS_API_KEY) | oma-image stills + Ken Burns | TODO(oma-deferred): pexels | | AIGC video | Pixelle-MCP + RunningHub (RUNNINGHUB_API_KEY) | oma-image stills | TODO(oma-deferred): pixelle | | caption timing | voicebox-stt (MCP voicebox_transcribe → REST) | estimate | TODO(oma-deferred): whisper-cpp | | music mixing | Strudel offline render (oma video doctor --install-strudel) | render without music | — | | premium TTS | (not needed — oma-voice is local) | — | — |

• Pixelle AIGC is a community MCP: off by default, requires one-time explicit user consent plus a source review before connecting, and is always cost-gated on RunningHub credits.

• Asset-provider fallbacks are not failures. A run may use key-free timing, visual, caption, or music paths and record pathTaken: fallback with a warning. An unavailable, failed, or invalid compositor remains an error.

Step 1: Resolve Brief & Preflight

1. Capture the brief from the user's request. If absent, ask:

   What is the video about? Give me a one-line brief, and a mode if you have one (shorts / explainer / demo).
   

2. Run the readiness check and surface gaps before spending any time on assets:

   oma video doctor --output json
   

   This reports Node / Chromium / FFmpeg, the Remotion project, the embedded Pretendard font, Voicebox MCP (oma-voice), oma-image vendors, optional Pixelle-MCP, Cap, and MPT readiness. Doctor does NOT auto-bootstrap — plain oma video doctor only reports. If Remotion is not yet installed, run oma video doctor --install (one-time: deps + Chrome Headless Shell + Pretendard font fetch) — do not install during a run. MPT needs a one-time oma video doctor --install-mpt (clone + venv + deps).
3. If doctor reports a hard blocker for the chosen mode (e.g. no compositor for shorts/explainer), report the remediation and stop. If only an optional provider is missing (Pexels, Pixelle, Cap), note it and continue on the fallback.
4. Record run start in the configured file-memory path: brief summary, requested mode, and doctor result.

Step 2: Confirm Mode & Plan

1. State the resolved mode, aspect, locale, caption style, visual track, and compositor you intend to use, and the expected output name.
2. For demo mode, state up front: "Capture is performed by a human." Resolve the source:
   • --source file: require --capture <path>; if absent, ask the user to record and provide the file path before proceeding.
   • --source web --url <url>: also require --capture <path>. The URL provides context only; the human records the flow separately. No login or browser capture is automated, and URL query tokens are masked in logs and the manifest.
3. Apply .agents/skills/_shared/core/execution-policy.md: proceed when the requested work or decision is already authorized; ask only for a material missing decision or new authorization.
4. Once the mode is resolved under the execution policy, emit and verify the mode-selection decision with its actual authorization source:

   oma state emit "decision.made" '{"subject":"video.mode-selection","decision":"<resolved mode and pipeline plan>","rationale":"<existing instruction, delegated choice, or new user decision authorizing the plan>"}'
   oma state verify --workflow video --checkpoint mode-selection
   

Step 3: Author script.json (agent-as-key)

The agent writes the script — this is the start of the determinism boundary. Do NOT call an external LLM; you are the script provider.

1. Produce a script honoring the script.json schema (schemaVersion: "1.0" — required literal — plus mode, aspect, locale, title, scenes[{id, durationSec, narration, onScreenText, visual{kind: still|clip|mixed|slide|capture, prompt, ref, source}, transition}], music, brand). Author against .agents/skills/oma-video/resources/script-schema.md (full field reference + example) — a schema mismatch is exit 4.
2. Respect limits from .agents/skills/oma-video/config/video-config.yaml (max_duration_sec: 180, max_scenes: 40). Keep narration tight and per-scene so scene boundaries map cleanly to TTS timing.
3. Mode-specific sourcing:
   • shorts: a hook-first synthetic script from the topic; each scene gets a visual.prompt for oma-image.
   • explainer: ground scenes in the README / code / data the user pointed to; mark scenes that should become oma-slide frames vs oma-image diagrams.
   • demo: narration + on-screen callouts over the captured footage; visual refs point at the ingested capture segments.
4. Translate narration / on-screen text via oma-translation when locale differs from the source language (key-free). If oma-translation is absent, keep the source text and let the run warn.
5. Write the agent-authored script to a file and hand it to the CLI via --script <path> so it validates against the schema. --script is mandatory for the agent-as-key path: without it the CLI builds its own skeleton script from the brief and your authored script is never used. Use --dry-run for the first pass so the pipeline emits script.json + render-spec.json + manifest.json without rendering:

   oma video generate "<brief>" --mode <mode> --aspect <aspect> --locale <lang> \
     --captions <tiktok|lower-third|none> --visual <auto|generate|stock|aigc|slide> \
     --voice <profile|none> --music <upbeat|calm|cinematic|lofi|piano|none> --duration <sec|auto> \
     --compositor <remotion|mpt> --seed <n> \
     --script <path-to-agent-authored-script.json> --dry-run --output json
   

6. Review the emitted script.json for scene count, durations, and narration quality. Iterate here — fixing the script is cheap; fixing a render is not.

Step 4: Parallel Asset Generation (voice / visual / caption)

The CLI orchestrator fans out the asset tracks per the asset bus. Trigger the full (non-dry) run; the orchestrator runs the tracks and writes them into the run directory. Do not author assets by hand.

oma video generate "<brief>" --mode <mode> [same flags as Step 3, incl. --script <path>, without --dry-run] --output json

The three tracks (per .agents/skills/oma-video/SKILL.md and its execution protocol):

• Voice (oma-voice / Voicebox MCP) → a single audio/narration-01.wav (all scene lines joined into one track — not per-scene files) + timing.json. Timing source: voicebox-stt (MCP voicebox_transcribe, REST /transcribe fallback, on the generated wav) → estimated (the tts-native / whisper-cpp source values are reserved but deferred). The default voice is none → a silent video with estimated timing; pass --voice <profile> for narration. If oma-voice is down, the run falls back to silent + estimated timing and warns — it does not hard-fail.
• Visual (per-scene, fallback chain oma-image → pexels → pixelle) → visuals/scene-NN.*. Default is key-free oma-image stills (aspect snapped to the nearest 16-multiple; Remotion crops to exact frame). --visual stock engages Pexels only when PEXELS_API_KEY is set; --visual aigc engages Pixelle only after consent + cost gate. Each scene that falls back is recorded with pathTaken: fallback.
• Caption (key-free) → captions.srt / .vtt, aligned to timing.json, styled tiktok or lower-third, with platform safe-area presets. Non-source locales translate via oma-translation; if absent, captions keep the source locale and warn.

Report which path each track took (real vs fallback) and surface any warnings.

Demo capture track (--mode demo)

For demo, the orchestrator produces the footage in place of synthetic visuals, dispatched on --source:

• --source file --capture <path> — ingest the human recording (absolutized, $PWD-guarded, format-validated). Without --capture, return the guided protocol and stop.
• --source web --url <url> --capture <path> — ingest the same human recording and use the URL as context. The CLI does not open a browser, wait on selectors, record the screen, or accept interactive stop controls. The URL and query tokens are masked in logs and manifest.json.

Step 5: Cost Gate & render-spec.json

1. Inspect the cost estimate the orchestrator computed across providers (cost.usd + breakdown in the manifest/JSON output).
2. If the estimate meets or exceeds the guardrail (default $0.20, or --max-usd), present the breakdown and reuse existing spend authorization if it covers the provider and amount. Otherwise obtain authorization before the paid render proceeds. Then emit and verify the actual decision:

   oma state emit "decision.made" '{"subject":"video.cost-confirmation","decision":"Proceed with the estimated paid cost or fall back to the key-free path.","rationale":"Estimated cost crossed the guardrail; the user confirmed spend or chose the fallback."}'
   oma state verify --workflow video --checkpoint cost-confirmation
   

   If the user declines, re-run with the key-free providers (drop --visual stock|aigc) — the fallback chain keeps the run alive.
3. If the estimate is under the guardrail, note "cost under guardrail ($X.XX < $0.20)" and continue without a confirmation prompt.
4. Confirm render-spec.json was written. This is the deterministic compute boundary: compositor, composition, fps, dimensions, durationInFrames, audio, scenes[], captions, background, seed. The seed is embedded so re-renders are byte-identical.

Step 6: Composite (Remotion — you author the composition; MPT is an alternative)

1. Remotion (default, all modes) — oma ships no composition code; you write it per run on the always-latest Remotion:
   1. oma video generate already scaffolded <runDir>/remotion/ (warning composition pending). If not, or to refresh: oma video compose <runDir> --output json.
   2. Read, in order: <runDir>/remotion/AUTHORING.md (contract for this spec), the remotion-best-practices and remotion-markup SKILL.md paths it lists (remotion-dev/skills at HEAD; remotion-captions when captions.style !== "none", remotion-multimedia for video/audio), and .agents/skills/oma-video/resources/remotion-authoring/<mode>.md.
   3. Write <runDir>/remotion/src/Root.tsx (+ src/components/*): one <Composition id={composition}> consuming render-spec.json, calculateMetadata from props, deterministic (no network/randomness), Pretendard via staticFile("fonts/PretendardVariable.woff2"). Never edit the generated files.
   4. oma video render <runDir> --output json — typecheck → npx remotion render → ffprobe video-stream/duration validation. A non-zero exit or invalid output is a render failure; use diagnostics to classify toolchain, runtime, or composition causes, then fix and re-render. Use at most three render attempts within ten minutes; then report the diagnostics and recovery artifacts.
   • Demo raw vs --polish: for demo, the default is the raw captured footage copied through as the output. --polish means you author the Demo composition (intro / callouts / zoom over the capture as background).
2. MoneyPrinterTurbo (--compositor mpt, shorts e2e alternative): the agent-written script is injected in custom-script mode; provider keys are env-only and masked in logs. It needs oma video doctor --install-mpt once. Setup or render failures fail with diagnostics; only OMA_VIDEO_MOCK=1 tests may create a placeholder file.
3. If the toolchain cannot be fetched (offline, nothing cached): oma video doctor --install once online. Do not pin or hand-install Remotion.
4. Confirm the output MP4 exists, matches <mode>-<slug>.mp4, has a video stream, and has a positive ffprobe duration.

Step 7: QA Loop

Review the finished video against the brief and the quality bars. Iterate by re-running the smallest upstream stage that owns the defect — never patch the artifact.

1. Checklist (priority order: correctness → sync → readability → polish):
   • Output plays; duration matches the script total within tolerance.
   • Narration audio is present (or intentionally silent) and aligns to scenes.
   • Captions are synced to timing.json, within the safe area, and legible (static windowed cues, CSS-wrapped, Pretendard, design rule 2).
   • Visuals match each scene's intent; any placeholder visual asset is declared in warnings, and a real validated MP4 remains required.
   • Aspect / dimensions are correct for the mode; branding applied as requested. (A requested music mode yields music/bgm.wav mixed at −18 dB, or a fallback warning and a silent render when Strudel is not installed.)
2. Route each defect to its stage:
   • script/narration/scene-count → Step 3 (re-author script).
   • audio/timing → Step 4 voice track (check oma-voice, re-synthesize).
   • wrong/placeholder visual → Step 4 visual track (adjust prompt or --visual mode).
   • missing/incomplete demo capture → Step 4 demo capture track (obtain a new human recording and pass --capture; --url is context only).
   • caption sync/wrap/locale → Step 4 caption track (or oma-translation).
   • layout/transition/crop → Step 6 edit the composition (<runDir>/remotion/src) or the render-spec → oma video render (for demo, toggle --polish).
3. Determinism guard: when validating reproducibility, run the golden harness — render-spec and assets must be byte-identical:

   OMA_VIDEO_MOCK=1 oma video generate "<brief>" --mode <mode> --seed <n> --dry-run --output json
   

4. Stop after three render attempts or ten minutes of render recovery. Report diagnostics and recovery artifacts; if a further change needs a different visual track, mode framing, or compositor, obtain the user's direction before continuing.
5. Repeat until the checklist passes or the user accepts the result.

Step 8: Output & Manifest

1. Confirm the run directory is complete (mirrors .agents/results/videos/<runId>-<mode>/):

   script.json · timing.json · render-spec.json
   audio/narration-01.wav                # single narration track (all lines joined)
   visuals/scene-*.{jpg,png,mp4}        # synthetic modes
   capture.mp4                           # demo: human recording ingested by the CLI
   captions.srt (+ .vtt)
   <mode>-<slug>.mp4
   manifest.json
   

2. Verify manifest.json is the reproducibility record: runId, mode, providers{...}, assets[{path,sha256,bytes,seed}], outputs{video,durationSec,sha256}, cost{usd,breakdown}, warnings[], exitCode. All external assets are copied into the run dir and hashed — no URL refs. For demo --source web, the manifest records a masked URL context (query tokens stripped), never credentials.
3. Report to the user:
   • Output MP4 path (absolute) and duration.
   • Providers used per track, and which tracks took the fallback path.
   • Final cost ($0.00 on the all-key-free path).
   • Any warnings (silent audio, source-locale captions, placeholder visuals).
   • Reproduce command: oma video render <runDir>.
4. Record run completion in the configured file-memory path: run dir, output path, providers, cost, and warnings.

Exit Codes (aligned with oma search fetch)

0 ok · 1 generic · 2 safety · 3 not-found · 4 invalid-input · 5 auth-required · 6 timeout.

Common error → action map:

| error | exit | action | |------|:---:|------| | ProviderUnavailableError | 5 | a required provider is down → run oma video doctor, fix or fall back | | CompositorBootstrapError | 1 | Remotion not installed → oma video doctor install-once, then re-render | | CostGuardrailError | confirm | estimate crossed guardrail → Step 5 confirmation or drop paid providers | | CaptureRequiredError | guided | demo needs a human recording → provide --capture <path>; --source web also requires --url context | | SchemaValidationError | 4 | script/render-spec invalid, missing demo capture, or --source web without --url → fix in Step 3 and re-validate with --dry-run |