Agent-Skills.md

Agent Skills: CI/CD

CI/CD reference for Megatron Bridge — pipeline structure, commit and

UncategorizedID: autohandai/community-skills/nvidia-megatron-bridge-cicd

Repository

autohandai/community-skills
autohandaiLicense: Apache-2.0
103

Install this agent skill to your local

pnpm dlx add-skill https://github.com/autohandai/community-skills/tree/HEAD/nvidia-megatron-bridge-cicd

Skill Files

Browse the full folder contents for nvidia-megatron-bridge-cicd.

Download Skill

Loading file tree…

nvidia-megatron-bridge-cicd/SKILL.md

Skill Metadata

Name
nvidia-megatron-bridge-cicd
Description
CI/CD reference for Megatron Bridge — pipeline structure, commit and

CI/CD

Commit and PR Workflow

  • Never commit directly to main — always create a feature branch.
  • Always sign commits: git commit -s -m "message".
  • PR title format: [{areas}] {type}: {description} (e.g., [model] feat: Add Qwen3 model bridge). See @CONTRIBUTING.md for the full PR workflow, area/type labels, and DCO requirements.

How CI Is Triggered

The workflow is defined in @.github/workflows/cicd-main.yml and is triggered on push — not on pull_request. This is intentional: a bot called copy-pr-bot controls when CI runs.

Mechanism:

  1. When a PR is opened, copy-pr-bot watches for a trust signal.
  2. Trust is established in one of two ways:
    • All commits on the PR branch are GPG-signed by a verified NVIDIA contributor → bot triggers automatically.
    • An NVIDIAN posts /ok to test <commit-sha> as a PR comment → bot triggers manually for that SHA.
  3. Once trusted, copy-pr-bot copies the PR's code into the remote branch pull-request/<number> and pushes it.
  4. That push fires the workflow's push trigger on refs/heads/pull-request/<number>, launching CI.

Consequences:

  • CI never runs on untrusted pushes — external contributors always need /ok to test.
  • The running workflow branch is pull-request/<number>, not the author's feature branch.
  • Pushing a new commit to a PR does not automatically re-trigger CI unless the commit is signed or /ok to test <new-sha> is posted.
  • Concurrent runs for the same PR are cancelled automatically (concurrency group per PR number).

Pipeline Structure

pre-flight
  └── lint-check
        └── cicd-wait-in-queue       # queues workflows to avoid runner interleaving across PRs
              └── cicd-container-build
                    ├── unit-tests-core
                    ├── unit-tests-diffusion
                    └── functional-tests (L0 always; L1 with needs-more-tests label; L2 on schedule or full-test-suite label)
  • Slack notifications are sent on completion for scheduled and nightly runs.

For functional test tier semantics and job-to-directory mapping, see the testing skill.

CI Failure Investigation

Locating the PR from a CI Branch

# Extract PR number from branch name (e.g. pull-request/1234)
PR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?<=pull-request/)\d+')

gh pr view "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge
gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge --name-only
gh pr checks "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge

Investigating a Failing Job

  1. Get the PR number from the branch name (see above).
  2. Review the changeset: 
    gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Megatron-Bridge
  3. Identify the failing job from gh pr checks output.
  4. Fetch job logs: 
    gh run list --repo NVIDIA-NeMo/Megatron-Bridge --branch "pull-request/$PR_NUMBER"
gh run view <run_id> --repo NVIDIA-NeMo/Megatron-Bridge --log-failed > run.log
  5. Scan logs in chunks — log files can exceed 10,000 lines, never load them whole: 
    wc -l run.log
tail -200 run.log          # start from the end
sed -n '1,200p' run.log    # or scan forward in 200-line chunks
  6. Cross-reference the changeset against the failing step.

Hugging Face Model Access In CI

Assume CI functional-test containers run with Hugging Face models offline (HF_HUB_OFFLINE=1) and a pre-populated HF_HOME. When reproducing or fixing CI failures involving HF models, mirror this locally by setting HF_HUB_OFFLINE=1 after warming the cache. Test fixtures must not depend on live Hub API calls such as list_repo_files() or uncached downloads during CI. For trust_remote_code=True toy checkpoints, copy custom Python modules from the already loaded local/cache source files or a local snapshot, not by listing the remote repo at test time.

Common Failure Patterns

| Symptom | Likely Cause | Action | |---|---|---| | CI never started on a PR | Commits not GPG-signed and no /ok to test comment | Post /ok to test <full-sha> on the PR | | Lint job fails | ruff or pre-commit violation | Run ruff check --fix + ruff format locally | | Container build fails | Dependency conflict or stale uv.lock | Re-run uv lock inside Docker and commit updated lock | | Unit tests fail | Code regression or missing import | Run failing test locally; check the PR diff | | Functional test (L0) fails | Integration breakage | Check GPU runner logs; reproduce with L0_Launch_*.sh | | HF model fixture passes locally but fails in CI with OfflineModeIsEnabled | Test made a live Hugging Face Hub API/download call; CI has HF_HUB_OFFLINE=1 | Warm local cache, reproduce with HF_HUB_OFFLINE=1, and change the fixture to use cached/local artifacts only | | cicd-wait-in-queue running long | Many PRs queued; automation serializes runners to avoid interleaving | Wait; or check queue depth in the Actions tab | | MCore submodule mismatch | Pinned commit out of sync | Update 3rdparty/Megatron-LM submodule and re-lock | | Stale checkpoint auto-resume | nemo_experiments/ from a previous run exists | rm -rf nemo_experiments before starting fresh | | Port collision on Slurm (EADDRINUSE) | ntasks-per-node=8 with torchrun | Drop torchrun; use ntasks-per-node=8 with uv run python script.py |