Agent-Skills.md

Agent Skills: Source Ingest

Fetch and normalize supported source-tutorial inputs into local, traceable text artifacts.

UncategorizedID: willoscar/research-units-pipeline-skills/source-ingest
51039

Install this agent skill to your local

pnpm dlx add-skill https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/HEAD/.codex/skills/source-ingest

Skill Files

Browse the full folder contents for source-ingest.

Download Skill

Loading file tree…

.codex/skills/source-ingest/SKILL.md

Skill Metadata

Name
source-ingest
Description
"Fetch and normalize supported source-tutorial inputs into local, traceable text artifacts."

Source Ingest

Triggers & routing

  • Trigger: source ingest, ingest sources, normalize tutorial sources, 网页抽取, 资料归一化.
  • Use when: source-tutorial 的 C1，需要把 sources/manifest.yml 中的网页/PDF/repo/docs 变成可追溯文本。
  • Skip if: source manifest 还没定，或来源尚未确认。
  • Network: required for remote URLs.
  • Guardrail: 只把成功抽取的内容当作有效 source；失败来源必须落盘记录，不能默默忽略。

Goal: normalize mixed source inputs into local tutorial-ready text while preserving provenance.

Inputs

  • sources/manifest.yml

Outputs

  • sources/index.jsonl
  • sources/provenance.jsonl

Supported kinds

  • webpage
  • pdf
  • markdown
  • repo
  • docs_site
  • video

Behavior

  • Accept local file paths or remote URLs from sources/manifest.yml.
  • For video, use transcript-first ingestion:
    • provided transcript_locator
    • Bilibili subtitles when available
    • otherwise fail clearly instead of pretending the watch page is usable text
  • Plain YouTube/Bilibili watch pages should not be modeled as kind: webpage.
  • Continue past optional-source failures and record warnings.
  • Fail closed when any source marked required: true cannot be ingested.

Script

Quick Start

  • uv run python .codex/skills/source-ingest/scripts/run.py --workspace <workspace>

All Options

  • --workspace <dir> (required)
  • --unit-id <U###>
  • --inputs <semicolon-separated>
  • --outputs <semicolon-separated>
  • --checkpoint <C#>

Examples

  • Ingest all listed sources:
    • uv run python .codex/skills/source-ingest/scripts/run.py --workspace <workspace>

Troubleshooting

Issue: repo or docs site ingests too much noise

Fix:

  • Narrow the manifest to the most relevant docs root or replace the source with a specific docs page.