Agent Skills: CLI Testing Patterns

CLI Testing Patterns

Comprehensive testing strategies for CLI applications using industry-standard testing frameworks. Covers command execution testing, exit code validation, output verification, interactive prompt testing, and integration testing patterns.

Instructions

When Testing Node.js CLI Tools

1. Use Jest for testing CLI commands

   • Import child_process.execSync for command execution
   • Create helper function to run CLI and capture output
   • Test exit codes, stdout, stderr separately
   • Handle both success and error cases

2. Test Structure

   • Set up CLI path relative to test location
   • Create runCLI() helper that returns {stdout, stderr, code}
   • Use try-catch to handle non-zero exit codes
   • Test common scenarios: version, help, unknown commands

3. What to Test

   • Command execution with various argument combinations
   • Exit code validation (0 for success, non-zero for errors)
   • Output content (stdout) validation
   • Error messages (stderr) validation
   • Configuration file handling
   • Interactive prompts (with mocked input)

When Testing Python CLI Tools

1. Use pytest with Click.testing.CliRunner

   • Import CliRunner from click.testing
   • Create runner fixture for reusable test setup
   • Invoke commands with runner.invoke(cli, ['args'])
   • Check result.exit_code and result.output

2. Test Structure

   • Create pytest fixture for CliRunner instance
   • Use runner.invoke() to execute CLI commands
   • Access results through result object
   • Simulate interactive input with input='responses\n'

3. What to Test

   • Command invocation with various arguments
   • Exit code validation
   • Output content verification
   • Error handling and messages
   • Interactive prompt responses
   • Configuration handling

Exit Code Testing Patterns

Standard Exit Codes:

• 0 - Success
• 1 - General error
• 2 - Misuse of command (invalid arguments)
• 126 - Command cannot execute
• 127 - Command not found
• 128+N - Fatal error signal N

Testing Strategy:

• Always test both success (0) and failure (non-zero) cases
• Verify specific exit codes for different error conditions
• Test argument validation returns appropriate codes
• Ensure help/version return 0 (success)

Output Validation Patterns

Content Testing:

• Check for presence of key text in output
• Validate format (JSON, YAML, tables)
• Test color/formatting codes (if applicable)
• Verify error messages are user-friendly

Best Practices:

• Use .toContain() for flexible matching (Jest)
• Use in result.output for Python tests
• Test both positive and negative cases
• Validate complete workflows (multi-command)

Templates

Use these templates for CLI testing:

Node.js/Jest Templates

• templates/jest-cli-test.ts - Complete Jest test suite with execSync
• templates/jest-config-test.ts - Configuration file testing
• templates/jest-integration-test.ts - Multi-command integration tests

Python/Pytest Templates

• templates/pytest-click-test.py - Click.testing.CliRunner tests
• templates/pytest-fixtures.py - Reusable pytest fixtures
• templates/pytest-integration-test.py - Integration test patterns

Test Utilities

• templates/test-helpers.ts - Node.js test helper functions
• templates/test-helpers.py - Python test helper functions

Scripts

Use these scripts for test setup and execution:

• scripts/setup-jest-testing.sh - Install Jest and configure for CLI testing
• scripts/setup-pytest-testing.sh - Install pytest and Click testing dependencies
• scripts/run-cli-tests.sh - Execute all CLI tests with coverage
• scripts/validate-test-coverage.sh - Check test coverage thresholds

Examples

See complete examples in the examples/ directory:

• examples/jest-basic/ - Basic Jest CLI testing setup
• examples/jest-advanced/ - Advanced Jest patterns with mocking
• examples/pytest-click/ - Click.testing.CliRunner examples
• examples/integration-testing/ - Full integration test suites
• examples/exit-code-testing/ - Exit code validation patterns

Requirements

Node.js Testing:

• Jest 29.x or later
• TypeScript support (ts-jest)
• Node.js 16+

Python Testing:

• pytest 7.x or later
• Click 8.x or later
• Python 3.8+

Both:

• Test coverage reporting tools
• CI/CD integration support
• Mock/stub capabilities for external dependencies

Best Practices

1. Test in Isolation - Each test should be independent
2. Mock External Dependencies - Don't make real API calls or file system changes
3. Test Error Paths - Test failures as thoroughly as successes
4. Use Fixtures - Share setup code across tests
5. Clear Test Names - Name tests to describe what they validate
6. Fast Execution - Keep tests fast for rapid feedback
7. Coverage Goals - Aim for 80%+ code coverage
8. Integration Tests - Test complete workflows, not just units

Purpose: Standardize CLI testing across Node.js and Python projects Load when: Writing tests for CLI tools, validating command execution, testing exit codes