Shadow Testing Skill Skill

Shadow Testing Skill

Purpose [LEVEL 1]

Shadow testing creates isolated container environments where you can test local uncommitted changes without affecting your host system or pushing to remote repositories.

Key Principle: Test exactly what's on your machine (including uncommitted changes) in a clean, isolated environment that mirrors CI.

When to Use This Skill [LEVEL 1]

Perfect For

Pre-Push Validation: Test changes before committing/pushing
Multi-Repo Coordination: Validate changes across multiple repositories work together
Clean-State Testing: "Does it work on a fresh machine?"
Library Development: Test library changes with dependent projects
CI Parity: See what CI will see before pushing
Destructive Testing: Tests that modify system state won't affect host

Use This Skill When

Making breaking changes to a library others depend on
Coordinating changes across multiple repositories
Unsure if your changes will work in CI
Need to test with specific dependency versions
Want to verify install/setup procedures work
Testing changes that require clean environment state

Don't Use This Skill When

Running unit tests on already-committed code (use local test runner)
Need to debug with live code changes (shadow captures snapshots)
Testing production deployment (use staging environments)
Simple single-file changes with good test coverage

Core Concepts [LEVEL 1]

Shadow Environment Architecture

A shadow environment is a Docker/Podman container with:

Git Bundle Snapshots - Exact working tree state (including uncommitted changes)
Embedded Gitea Server - Local git server at localhost:3000 inside container
Selective URL Rewriting - Git insteadOf rules redirect specific repos to local Gitea
Package Manager Isolation - UV, pip, npm, cargo, go caches isolated per shadow
API Key Passthrough - Common API keys automatically forwarded to container

┌─────────────────────────────────────────────────────────┐
│  Shadow Container                                       │
│  ┌───────────────────────────────────────────────────┐  │
│  │  Gitea Server (localhost:3000)                    │  │
│  │  - myorg/my-library (your snapshot)               │  │
│  └───────────────────────────────────────────────────┘  │
│                                                         │
│  Git URL Rewriting:                                     │
│  github.com/myorg/my-library → Gitea (local)           │
│  github.com/myorg/other-repo → Real GitHub             │
│                                                         │
│  /workspace (pre-cloned local sources)                 │
└─────────────────────────────────────────────────────────┘

How Git URL Rewriting Works

When you create a shadow with ~/repos/my-lib:myorg/my-lib:

Your working directory is captured exactly as-is (uncommitted changes included)
Snapshot is bundled with full git history
Container starts with Gitea server
Snapshot pushed to Gitea as myorg/my-lib

Git config adds insteadOf rules:

[url "http://shadow:shadow@localhost:3000/myorg/my-lib.git"]
    insteadOf = https://github.com/myorg/my-lib.git

Any git clone https://github.com/myorg/my-lib → uses YOUR local snapshot
All other GitHub URLs → fetch from real GitHub

Result: Only your specified repos are local; everything else uses production sources.

Quick Start [LEVEL 1]

Installation

For Amplifier Users (native integration):

# Shadow tool is built-in - no installation needed
amplifier run --bundle amplihack

For Other Agents (standalone CLI):

# Install via uvx (recommended)
uvx amplifier-shadow --version

# Or via pip
pip install amplifier-bundle-shadow

# Verify installation
amplifier-shadow --version

Prerequisites:

Docker or Podman installed and running
Git installed

Your First Shadow (CLI)

# Create shadow with your local library changes
amplifier-shadow create --local ~/repos/my-library:myorg/my-library --name test-lib

# Inside the shadow, install via git URL
# → my-library uses YOUR LOCAL snapshot
# → all other dependencies fetch from REAL GitHub
amplifier-shadow exec test-lib "uv pip install git+https://github.com/myorg/my-library"

# Run tests
amplifier-shadow exec test-lib "cd /workspace && pytest"

# See what changed
amplifier-shadow diff test-lib

# Clean up when done
amplifier-shadow destroy test-lib

Your First Shadow (Amplifier Tool)

# Create shadow with local changes
shadow.create(local_sources=["~/repos/my-library:myorg/my-library"])

# Execute commands
shadow.exec(shadow_id, "uv pip install git+https://github.com/myorg/my-library")
shadow.exec(shadow_id, "pytest tests/")

# Extract results
shadow.extract(shadow_id, "/workspace/test-results", "./results")

# Cleanup
shadow.destroy(shadow_id)

Tool Reference by Agent Type [LEVEL 2]

Amplifier (Native Integration)

Best experience - shadow is a first-class tool with automatic setup:

# All operations via shadow tool
result = shadow.create(
    local_sources=["~/repos/lib:org/lib"],
    verify=True  # Automatic smoke test
)

# Integrated error handling and observability
if result.ready:
    shadow.exec(result.shadow_id, "pytest")

Features:

Automatic API key passthrough
Built-in smoke tests and health checks
Integrated with other Amplifier tools
Session-aware cleanup

Claude Code Standalone

Use the CLI directly from bash tool:

# All operations via amplifier-shadow CLI
uvx amplifier-shadow create --local ~/repos/my-lib:org/my-lib --name test

uvx amplifier-shadow exec test "pip install -e /workspace/org/my-lib"
uvx amplifier-shadow exec test "pytest"

uvx amplifier-shadow destroy test

GitHub Copilot

Same CLI interface as Claude Code:

# Install once
pip install amplifier-bundle-shadow

# Use in workflow
amplifier-shadow create --local ~/repos/lib:org/lib
amplifier-shadow exec shadow-xxx "npm install && npm test"

Manual/DIY (Any Agent)

Use the provided shell scripts and Docker Compose examples (see Level 3).

Common Patterns [LEVEL 2]

Pattern: Test Library Changes Before Publishing

# Test your library with its dependents
amplifier-shadow create --local ~/repos/my-library:myorg/my-library --name lib-test

# Clone dependent project and install
amplifier-shadow exec lib-test "
  cd /workspace &&
  git clone https://github.com/myorg/dependent-app &&
  cd dependent-app &&
  uv venv && . .venv/bin/activate &&
  uv pip install git+https://github.com/myorg/my-library &&
  pytest
"

Pattern: Multi-Repo Changes

# Testing changes across multiple repos
amplifier-shadow create \
    --local ~/repos/core-lib:myorg/core-lib \
    --local ~/repos/cli-tool:myorg/cli-tool \
    --name multi-test

# Both local sources will be used
amplifier-shadow exec multi-test "uv pip install git+https://github.com/myorg/cli-tool"

Pattern: Iterate on Failures

# 1. Create shadow and run tests
amplifier-shadow create --local ~/repos/lib:org/lib --name test
amplifier-shadow exec test "pytest" # Fails

# 2. Fix code locally on host

# 3. Destroy and recreate (picks up your local changes)
amplifier-shadow destroy test
amplifier-shadow create --local ~/repos/lib:org/lib --name test
amplifier-shadow exec test "pytest" # Passes

# 4. Commit with confidence!
git commit -m "Fix issue"

Pattern: Pre-Push CI Validation

# Run your CI script in shadow before pushing
amplifier-shadow create --local ~/repos/project:org/project --name ci-check

amplifier-shadow exec ci-check "
  cd /workspace/org/project &&
  ./scripts/ci.sh
"

# If CI script passes, your push will likely succeed

Verification Best Practices [LEVEL 2]

Always Verify Local Sources Are Used

After creating a shadow, confirm your local code is actually being used:

# Step 1: Check snapshot commits (from create output)
amplifier-shadow create --local ~/repos/lib:org/lib
# Output shows: snapshot_commits: {"org/lib": "abc1234..."}

# Step 2: Compare with install output
amplifier-shadow exec shadow-xxx "uv pip install git+https://github.com/org/lib"
# Look for: lib @ git+...@abc1234

# If commits match, your local code is being used!

Pre-Cloned Repository Locations

Local sources are automatically cloned to /workspace/{org}/{repo}:

# Your local source microsoft/my-library is available at:
/workspace/microsoft/my-library

# Use for editable installs (Python)
amplifier-shadow exec shadow-xxx "pip install -e /workspace/microsoft/my-library"

# Or for Node.js
amplifier-shadow exec shadow-xxx "cd /workspace/microsoft/my-package && npm install"

Always check this location first - the repo is already there.

Environment Variable Verification

# Don't assume - verify API keys are present!
amplifier-shadow exec shadow-xxx "env | grep API_KEY"

# Check all passed variables
amplifier-shadow status shadow-xxx
# Shows: env_vars_passed: ["ANTHROPIC_API_KEY", ...]

Troubleshooting [LEVEL 2]

Common Issues

"UV tool install" uses cache instead of local source:

Problem: UV may bypass git URL rewriting for cached packages.

Solution:

# Option 1: Install from pre-cloned workspace (recommended)
amplifier-shadow exec xxx "pip install -e /workspace/org/lib"

# Option 2: Clear UV cache first
amplifier-shadow exec xxx "rm -rf /tmp/uv-cache && uv tool install git+https://github.com/org/lib"

"PEP 668: Externally-Managed Environment":

Solution: Always use virtual environments inside shadow:

amplifier-shadow exec xxx "
  cd /workspace &&
  uv venv &&
  . .venv/bin/activate &&
  uv pip install ...
"

"Container image not found":

Solution: Build the image locally:

amplifier-shadow build

"/workspace permission denied":

Solution: Use $HOME or /tmp as alternatives:

amplifier-shadow exec xxx "cd $HOME && git clone ..."

Level 3: Advanced Topics [LEVEL 3]

Custom Docker Images

Build your own shadow image with additional tools:

FROM ghcr.io/microsoft/amplifier-shadow:latest

# Add your tools
RUN apt-get update && apt-get install -y \
    postgresql-client \
    redis-tools

# Add custom scripts
COPY my-test-script.sh /usr/local/bin/

Build and use:

docker build -t my-shadow:latest .
amplifier-shadow create --image my-shadow:latest --local ~/repos/lib:org/lib

Shell Scripts (DIY Shadow Setup)

For agents without Amplifier access, use these standalone scripts:

Script 1: Create Git Bundle (scripts/create-bundle.sh):

#!/bin/bash
# Create git bundle snapshot of working tree

REPO_PATH=$1
OUTPUT_PATH=$2

cd "$REPO_PATH"

# Fetch all refs to ensure complete history
git fetch --all --tags --quiet 2>/dev/null || true

# Check for uncommitted changes
if [[ -n $(git status --porcelain) ]]; then
    # Create temp clone and commit changes
    TEMP_DIR=$(mktemp -d)
    git clone --quiet "$REPO_PATH" "$TEMP_DIR"

    # Sync working tree (including deletions)
    rsync -a --delete --exclude='.git' "$REPO_PATH/" "$TEMP_DIR/"

    cd "$TEMP_DIR"
    git add -A
    git commit --allow-empty -m "Shadow snapshot" --author="Shadow <shadow@localhost>"

    # Create bundle
    git bundle create "$OUTPUT_PATH" --all

    cd /
    rm -rf "$TEMP_DIR"
else
    # Clean repo - just bundle it
    git bundle create "$OUTPUT_PATH" --all
fi

echo "Bundle created: $OUTPUT_PATH"

Script 2: Setup Shadow Container (scripts/setup-shadow.sh):

#!/bin/bash
# Start container with Gitea and configure git URL rewriting

CONTAINER_NAME=$1
BUNDLE_PATH=$2
ORG=$3
REPO=$4

# Start container
docker run -d \
    --name "$CONTAINER_NAME" \
    -v "$BUNDLE_PATH:/snapshots/bundle.git:ro" \
    ghcr.io/microsoft/amplifier-shadow:latest

# Wait for Gitea
echo "Waiting for Gitea to start..."
until docker exec "$CONTAINER_NAME" curl -sf http://localhost:3000/api/v1/version > /dev/null; do
    sleep 1
done

# Create org and repo in Gitea
docker exec "$CONTAINER_NAME" bash -c "
    curl -s -u shadow:shadow \
        -H 'Content-Type: application/json' \
        -d '{\"username\":\"$ORG\"}' \
        http://localhost:3000/api/v1/orgs

    curl -s -u shadow:shadow \
        -H 'Content-Type: application/json' \
        -d '{\"name\":\"$REPO\",\"private\":false}' \
        http://localhost:3000/api/v1/orgs/$ORG/repos
"

# Push bundle to Gitea
docker exec "$CONTAINER_NAME" bash -c "
    cd /tmp &&
    git init --bare repo.git &&
    cd repo.git &&
    git fetch /snapshots/bundle.git refs/heads/*:refs/heads/* &&
    git remote add origin http://shadow:shadow@localhost:3000/$ORG/$REPO.git &&
    git push origin --all --force
"

# Configure git URL rewriting
docker exec "$CONTAINER_NAME" bash -c "
    git config --global url.'http://shadow:shadow@localhost:3000/$ORG/$REPO.git'.insteadOf 'https://github.com/$ORG/$REPO.git'
"

echo "Shadow container ready: $CONTAINER_NAME"
echo "Local source: $ORG/$REPO"

Usage:

# Create bundle from your repo
./scripts/create-bundle.sh ~/repos/my-lib /tmp/my-lib.bundle

# Setup shadow container
./scripts/setup-shadow.sh shadow-test /tmp/my-lib.bundle myorg my-lib

# Test
docker exec shadow-test bash -c "
    git clone https://github.com/myorg/my-lib /tmp/test &&
    cd /tmp/test &&
    git log -1 --oneline
"

Docker Compose Examples

Example 1: Single Repository (docker-compose/single-repo.yml):

version: "3.8"

services:
  shadow:
    image: ghcr.io/microsoft/amplifier-shadow:latest
    container_name: shadow-single
    volumes:
      - ./snapshots:/snapshots:ro
      - ./workspace:/workspace
    environment:
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    command: >
      bash -c "
        /usr/local/bin/gitea-init.sh &&
        tail -f /dev/null
      "

Example 2: Multi-Repository Testing (docker-compose/multi-repo.yml):

version: "3.8"

services:
  shadow-multi:
    image: ghcr.io/microsoft/amplifier-shadow:latest
    container_name: shadow-multi
    volumes:
      # Mount multiple bundles
      - ./snapshots/core-lib.bundle:/snapshots/org/core-lib.bundle:ro
      - ./snapshots/cli-tool.bundle:/snapshots/org/cli-tool.bundle:ro
      - ./workspace:/workspace
    environment:
      # Pass API keys from host
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      # UV cache isolation
      - UV_CACHE_DIR=/tmp/uv-cache
    command: >
      bash -c "
        /usr/local/bin/gitea-init.sh &&
        /usr/local/bin/setup-repos.sh org/core-lib org/cli-tool &&
        tail -f /dev/null
      "

Usage:

# Create bundles for your repos
git -C ~/repos/core-lib bundle create snapshots/core-lib.bundle --all
git -C ~/repos/cli-tool bundle create snapshots/cli-tool.bundle --all

# Start shadow
docker-compose -f docker-compose/multi-repo.yml up -d

# Run tests
docker-compose exec shadow-multi bash -c "
    cd /workspace &&
    git clone https://github.com/org/cli-tool &&
    cd cli-tool &&
    uv pip install -e .
    pytest
"

# Cleanup
docker-compose down

Example 3: CI Integration (docker-compose/ci-shadow.yml):

version: "3.8"

services:
  ci-shadow:
    image: ghcr.io/microsoft/amplifier-shadow:latest
    container_name: ci-shadow
    volumes:
      - ./snapshots:/snapshots:ro
      - ./test-results:/test-results
    environment:
      - CI=true
      - GITHUB_ACTIONS=true
    command: >
      bash -c "
        /usr/local/bin/gitea-init.sh &&
        /usr/local/bin/run-ci-tests.sh > /test-results/output.log 2>&1
      "

GitHub Actions Integration:

# .github/workflows/shadow-test.yml
name: Shadow Test

on: [push, pull_request]

jobs:
  shadow-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Create git bundle
        run: git bundle create snapshot.bundle --all

      - name: Run shadow tests
        run: |
          docker run --rm \
            -v $PWD/snapshot.bundle:/snapshots/bundle.git:ro \
            ghcr.io/microsoft/amplifier-shadow:latest \
            /usr/local/bin/test-in-shadow.sh org/repo

Integration with Outside-In Testing

Combine shadow environments with agentic outside-in tests:

# Create shadow with local changes
amplifier-shadow create --local ~/repos/lib:org/lib --name test

# Run outside-in test scenarios inside shadow
amplifier-shadow exec test "gadugi-agentic-test run test-scenario.yaml"

# Extract evidence
amplifier-shadow extract test /evidence ./test-evidence

See the qa-team skill for complete integration examples (outside-in-testing remains an alias).

Best Practices [LEVEL 2]

1. Always Verify Your Sources Are Used

Don't assume - verify that the shadow is actually using your local code:

# Check snapshot commits
amplifier-shadow status shadow-xxx | grep snapshot_commit

# Verify install resolves to that commit
amplifier-shadow exec shadow-xxx "pip install git+https://github.com/org/lib" | grep "org/lib @"

2. Use Pre-Cloned Workspace

Local sources are automatically at /workspace/{org}/{repo}:

# ✅ FAST: Use pre-cloned repo
amplifier-shadow exec xxx "pip install -e /workspace/org/lib"

# ❌ SLOWER: Clone again
amplifier-shadow exec xxx "git clone https://github.com/org/lib && pip install -e lib"

3. Isolate Package Manager Caches

Shadow environments automatically isolate caches to prevent stale packages:

Python UV: /tmp/uv-cache
Python pip: /tmp/pip-cache
Node npm: /tmp/npm-cache
Rust cargo: /tmp/cargo-home
Go modules: /tmp/go-mod-cache

These are set automatically - no action needed.

4. Pass Required Environment Variables

# Amplifier (automatic for common API keys)
shadow.create(local_sources=["~/repos/lib:org/lib"])

# CLI (explicit)
amplifier-shadow create \
    --local ~/repos/lib:org/lib \
    --env ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
    --env CUSTOM_VAR=value

5. Clean Up After Testing

# Always destroy shadows when done
amplifier-shadow destroy shadow-xxx

# Or destroy all
amplifier-shadow destroy-all

6. Use Named Shadows for Clarity

# ✅ GOOD: Descriptive name
amplifier-shadow create --local ~/repos/lib:org/lib --name test-breaking-change

# ❌ BAD: Auto-generated
amplifier-shadow create --local ~/repos/lib:org/lib
# Creates shadow-a3f2b8c1 (hard to remember)

Integration Patterns [LEVEL 3]

Pattern: Shadow + Outside-In Tests

Combine shadow isolation with declarative test scenarios:

# test-scenario.yaml
scenario:
  name: "Library Integration Test"
  type: cli

  steps:
    - action: launch
      target: "/workspace/org/lib/cli.py"

    - action: verify_output
      contains: "Success"

Run in shadow:

amplifier-shadow create --local ~/repos/lib:org/lib --name test
amplifier-shadow exec test "gadugi-agentic-test run test-scenario.yaml"

Pattern: Shadow + pytest

amplifier-shadow create --local ~/repos/lib:org/lib --name pytest-run

amplifier-shadow exec pytest-run "
  cd /workspace/org/lib &&
  uv venv && . .venv/bin/activate &&
  pip install -e '.[dev]' &&
  pytest --cov=src --cov-report=html
"

# Extract coverage report
amplifier-shadow extract pytest-run /workspace/org/lib/htmlcov ./coverage-report

Pattern: Shadow + npm test

amplifier-shadow create --local ~/repos/pkg:org/pkg --name npm-test

amplifier-shadow exec npm-test "
  cd /workspace/org/pkg &&
  npm install &&
  npm test
"

Pattern: Shadow + cargo test

amplifier-shadow create --local ~/repos/crate:org/crate --name cargo-test

amplifier-shadow exec cargo-test "
  cd /workspace/org/crate &&
  cargo build &&
  cargo test
"

Philosophy Alignment [LEVEL 2]

This skill follows amplihack's core principles:

Ruthless Simplicity

Minimal abstraction: Shadow = container + gitea + URL rewriting
No frameworks: Pure Docker, git, and shell scripts
Essential only: Only captures what's needed (git bundle, not entire filesystems)

Modular Design (Bricks & Studs)

Self-contained: Each shadow is independent
Clear contract: Git URLs in → local sources out
Composable: Combine with other testing tools

Zero-BS Implementation

No stubs: Every script works completely
Working defaults: Reasonable defaults for all operations
Clear errors: Actionable error messages with troubleshooting

Outside-In Thinking

User perspective: Test what users will see
Implementation agnostic: Don't care how code works internally
Behavior-driven: Focus on outcomes

CLI Reference [LEVEL 3]

Commands

# Create shadow environment
amplifier-shadow create [OPTIONS]
  --local, -l TEXT    Local source mapping: /path/to/repo:org/name (repeatable)
  --name, -n TEXT     Name for environment (auto-generated if not provided)
  --image, -i TEXT    Container image (default: amplifier-shadow:local)
  --env, -e TEXT      Environment variable: KEY=VALUE or KEY to inherit (repeatable)
  --env-file FILE     File with environment variables (one per line)
  --pass-api-keys     Auto-pass common API key env vars (default: enabled)

# Execute command in shadow
amplifier-shadow exec SHADOW_ID COMMAND
  --timeout INTEGER   Timeout in seconds (default: 300)

# Show changed files
amplifier-shadow diff SHADOW_ID [PATH]

# Extract file from shadow
amplifier-shadow extract SHADOW_ID CONTAINER_PATH HOST_PATH

# Inject file into shadow
amplifier-shadow inject SHADOW_ID HOST_PATH CONTAINER_PATH

# List all shadows
amplifier-shadow list

# Show shadow status
amplifier-shadow status SHADOW_ID

# Destroy shadow
amplifier-shadow destroy SHADOW_ID
  --force             Force destruction even on errors

# Destroy all shadows
amplifier-shadow destroy-all
  --force             Force destruction even on errors

# Build shadow image locally
amplifier-shadow build

# Open interactive shell
amplifier-shadow shell SHADOW_ID

Quick Reference Card [LEVEL 1]

# Typical workflow
amplifier-shadow create --local ~/repos/lib:org/lib --name test
amplifier-shadow exec test "pytest"
amplifier-shadow destroy test

# Multi-repo
amplifier-shadow create \
  --local ~/repos/lib1:org/lib1 \
  --local ~/repos/lib2:org/lib2 \
  --name multi

# With environment variables
amplifier-shadow create \
  --local ~/repos/lib:org/lib \
  --env API_KEY=$API_KEY \
  --name test

# Interactive shell
amplifier-shadow shell test

# Extract results
amplifier-shadow extract test /workspace/results ./local-results

Related Skills [LEVEL 1]

qa-team: Run agentic tests in shadow environments (legacy name: outside-in-testing)
test-gap-analyzer: Find untested code paths (complement shadow testing)
philosophy-guardian: Verify shadow scripts follow ruthless simplicity

Troubleshooting Checklist [LEVEL 2]

When shadow tests fail:

[ ] Verify local sources are being used (check snapshot commits)
[ ] Check pre-cloned repos exist at /workspace/{org}/{repo}
[ ] Verify environment variables are passed (run env inside shadow)
[ ] Clear package manager caches if stale
[ ] Check git URL rewriting is configured (git config --list)
[ ] Verify Gitea is accessible (curl http://localhost:3000/api/v1/version)
[ ] Use virtual environments (avoid PEP 668 errors)
[ ] Check container is running (amplifier-shadow status)

Changelog [LEVEL 3]

Version 1.0.0 (2026-01-29)

Initial skill release
Support for Amplifier, Claude Code, GitHub Copilot, manual DIY
Shell scripts for standalone usage
Docker Compose examples for CI integration
Complete CLI reference and troubleshooting guide
Integration patterns with qa-team / outside-in-testing alias
Philosophy alignment with ruthless simplicity

Remember: Shadow environments let you test exactly what's on your machine (uncommitted changes and all) in a clean, isolated environment that mirrors CI. Use them before every significant push to catch issues early.

Agent Skills: Shadow Testing Skill

Install this agent skill to your local

Skill Files