Agent Skills: Testing

Testing

Overview

Writing tests: unit/integration/e2e plus data-pipeline, ML, and infrastructure domains. This file owns test-double taxonomy, AAA, and framework patterns. For pyramid ratios, coverage targets, risk prioritization, and flaky-test diagnosis, see loom-test-strategy; for browser/Playwright/Cypress, see loom-e2e-testing.

Workflow

1. Map the unit — public interface, dependencies/side effects, invariants, error paths, boundary values.
2. Pick the altitude — unit for logic/branches; integration for real collaborators (DB, HTTP) at a boundary; e2e for user journeys. Push detail down the pyramid (loom-test-strategy).
3. Write failing test first when practical (TDD red→green→refactor): a red test proves the test actually exercises the code; a test that never failed asserts nothing.
4. Arrange-Act-Assert, one logical assertion per test, deterministic inputs.

AAA and naming

• Arrange state/doubles → Act (one call to the unit) → Assert outcome. Blank-line separate the three; more than one Act means split the test.
• Name test_<unit>_<scenario>_<expected> — e.g. test_cart_add_duplicate_increases_quantity. The name is the spec; a failing name should tell you what broke without reading the body.
• One logical assert per test. Multiple physical asserts on one behavior are fine (status, then body); asserting two unrelated behaviors is "assertion roulette" — the first failure masks the rest. Prefer one composite assert (assert_eq!(got, expected_struct)) over many field asserts.

Test Doubles — taxonomy and misuse

Precision matters: "mock" is colloquially any double, but the kind you choose decides whether the test is brittle.

| Double | Does | Verifies | Reach for when | | ------ | ---- | -------- | -------------- | | Dummy | Fills a parameter, never used | nothing | Satisfy a signature | | Stub | Returns canned values for indirect inputs | state (result) | Drive a branch from a dependency's return | | Spy | Stub that records how it was called | state + calls, after | Assert an effect happened, loosely | | Mock | Pre-set call expectations, self-verifying | interaction, strict | The interaction is the contract (e.g. "charge called once with amount") | | Fake | Lightweight working impl (in-memory DB, fake clock) | state | Collaborator too slow/awkward for real, but behavior matters |

When each is wrong:

• Mock used where a stub belongs → the test asserts how the code works (which methods it calls), not what it produces. Refactoring the internals breaks green tests. Assert the observable outcome; mock only when the call itself is the observable behavior (payment charged, email sent, event published).
• Over-mocking → every collaborator faked; the suite is green while real integration is broken. Mock only at architectural seams you don't control: network, clock, filesystem, randomness, external services. Do not mock the type under test or pure internal collaborators.
• Fakes drift from the real thing (in-memory store enforces no FK constraints the real DB does). Pair fakes with a thin layer of integration/contract tests against the real implementation.
• Mocking a value object instead of constructing a real one — always cheaper and truer to build the real value.

Determinism (non-negotiable)

Flaky tests are worse than no tests — they train the team to ignore red. Eliminate every non-deterministic input:

• No sleep/waitForTimeout. Poll/await a condition (element visible, job status == done). A fixed sleep is either flaky (too short) or slow (too long).
• Freeze the clock. Inject a clock or use a fake-timer lib; never assert against Date.now()/SystemTime::now().
• Seed RNG and any faker/UUID source used in assertions; log the seed on failure so you can reproduce.
• Isolate shared state — reset DB (transaction rollback or truncate), globals, singletons, env vars, and filesystem between tests. Unique keys (UUID) per test so parallel runs don't collide.
• Order-independent: tests must pass in any order and in isolation (--shuffle / random seed in CI).

Diagnosis workflow and the full flakiness cause table live in loom-test-strategy.

Coverage (floor, not target)

Coverage tells you what's untested, not what's tested well. Chasing a number invites assertion-free tests that execute lines without checking anything.

• Track branch coverage, not just line — line coverage hides untested else/error paths.
• Mutation testing (mutmut, cargo-mutants, StrykerJS) is the real signal: it perturbs code and checks a test fails. Surviving mutants = weak/missing assertions that line coverage rated 100%.
• Cover behavior and error paths, not getters/framework code. Targets by component: see loom-test-strategy.

Framework Patterns

Python (pytest)

import pytest
from unittest.mock import patch

@pytest.fixture
def db():
    d = Database(); d.connect()
    yield d
    d.disconnect()                       # teardown runs even on failure

@pytest.mark.parametrize("n,expected", [(0, 1), (1, 1), (5, 120)])
def test_factorial(n, expected):         # table-driven: one row per case
    assert factorial(n) == expected

@patch("payments.stripe")                # patch where it's *used*, not defined
def test_charge_declined_raises(mock_stripe):
    mock_stripe.Charge.create.side_effect = CardError("declined")
    with pytest.raises(PaymentError, match="declined"):
        PaymentProcessor().charge(1000, "tok_x")

⚠ @patch target is the name in the module under test (payments.stripe), not stripe itself — patching the wrong path is the #1 mock no-op. Prefer freezegun/time-machine for the clock and pytest.raises(match=...) to assert the message.

JavaScript/TypeScript (Vitest/Jest)

import { describe, it, expect, vi, beforeEach } from "vitest";

describe("UserService", () => {
  let db;
  beforeEach(() => { db = { query: vi.fn() }; });

  it("finds user by email", async () => {
    db.query.mockResolvedValue([{ id: 1, email: "a@b.com" }]);
    const user = await new UserService(db).findByEmail("a@b.com");
    expect(user.id).toBe(1);
    expect(db.query).toHaveBeenCalledWith(expect.stringContaining("WHERE email"), ["a@b.com"]);
  });
});

⚠ Use vi.useFakeTimers() for time; await expect(promise).rejects.toThrow() for async errors. toEqual deep-compares, toBe is Object.is — mixing them up passes/fails silently on objects. Snapshot tests (toMatchSnapshot) rot into rubber-stamps — reserve for stable serialized output, review every snapshot diff, never --updateSnapshot blindly.

Rust (built-in + mockall)

#[cfg(test)]
mod tests {
    use super::*;
    use mockall::predicate::eq;

    #[test]
    fn fetches_user_from_db() {
        let mut db = MockDatabase::new();
        db.expect_get_user()
            .with(eq(1))
            .times(1)
            .return_once(|_| Ok(User { id: 1, name: "Alice".into() }));
        assert_eq!(UserService::new(db).get_user(1).unwrap().name, "Alice");
    }
}

⚠ Use #[should_panic(expected = "...")] sparingly — prefer returning Result and asserting the Err. Tests needing serialized access to shared state use #[serial] (serial_test crate — this repo relies on it). assert_eq! prints both sides on failure; hand-rolled assert!(a == b) does not.

Domain Examples

Concrete assertions per domain; strategy/priority layering is in loom-test-strategy.

Data pipeline — assert on invariants, not exact output:

def test_etl_preserves_rows_and_keys():
    out = etl.transform(load_fixture("sales_1000.csv"))
    assert len(out) == 1000                          # no silent drops
    assert out["customer_id"].notna().all()          # required field intact
    assert out["order_id"].is_unique                  # no dup fan-out

Also test idempotency (run twice → same result), incremental (only new rows processed), and partial-failure recovery.

ML model — behavioral tests, not just aggregate accuracy:

def test_sentiment_invariant_to_punctuation():
    m = load_model("sentiment")
    assert abs(m.predict("great product") - m.predict("great product!!!")) < 0.1

Cover invariance (irrelevant change → stable output), directional expectation (adding a positive word raises score), and a minimum-functionality set the model must never miss. Guard train/test split for leakage.

Infrastructure (IaC) — assert on the plan, not a live deploy:

def test_no_public_s3_buckets():
    for r in load_plan("main.tfplan.json").resources("aws_s3_bucket"):
        assert r.get("acl", "private") != "public-read", f"{r['name']} is public"

Anti-Patterns

• Testing implementation — private methods, internal call counts. Test through the public interface; assert outcomes.
• Interdependent tests — one test's writes are another's fixture. Each test self-contains its setup.
• Logic in tests — loops/conditionals/computed expected values re-implement the code and hide bugs. Hard-code expected values; use parametrization for variants.
• Excessive mocking — see taxonomy above; brittle and false-green.
• Slow unit tests — real I/O in a "unit" test. Push to integration or fake the boundary.
• Assertion roulette / no message — a bare assert result failure tells you nothing.

Test Organization

tests/
├── unit/          # fast, isolated, no I/O
├── integration/   # real DB/HTTP at a boundary
├── e2e/           # user journeys (see loom-e2e-testing)
├── fixtures/      # shared data / factories
└── helpers/       # setup, custom assertions

CI altitude: unit on every commit, integration on PR, e2e pre-deploy — details in loom-test-strategy.

Verify before done

• [ ] Each test has one clear reason to fail (one behavior); name states scenario + expected outcome
• [ ] AAA structure; expected values hard-coded, not computed
• [ ] Doubles chosen by role (stub for inputs, mock only where the interaction is the contract); no over-mocking of internals
• [ ] Deterministic: clock frozen, RNG seeded, state reset, no sleep — passes shuffled and in isolation
• [ ] Error/edge paths asserted (types and messages), not just the happy path
• [ ] New test fails when the code under test is broken (verify red before green; spot-check with a mutant)
• [ ] No test-only logic in production code; no leftover .only/fdescribe/#[ignore]