Performance Profiling Skill

Performance Profiling

Goal

Provide tools to analyze simulation performance, identify bottlenecks, and recommend optimization strategies for computational materials science simulations.

Requirements

Python 3.10+
No external dependencies (uses Python standard library only)
Works on Linux, macOS, and Windows

Inputs to Gather

Before running profiling scripts, collect from the user:

| Input | Description | Example | |-------|-------------|---------| | Simulation log | Log file with timing information | simulation.log | | Scaling data | JSON with multi-run performance data | scaling_data.json | | Simulation parameters | JSON with mesh, fields, solver config | params.json | | Available memory | System memory in GB (optional) | 16.0 |

Decision Guidance

When to Use Each Script

Need to identify slow phases?
├── YES → Use timing_analyzer.py
│         └── Parse simulation logs for timing data
│
Need to understand parallel performance?
├── YES → Use scaling_analyzer.py
│         └── Analyze strong or weak scaling efficiency
│
Need to estimate memory requirements?
├── YES → Use memory_profiler.py
│         └── Estimate memory from problem parameters
│
Need optimization recommendations?
└── YES → Use bottleneck_detector.py
          └── Combine analyses and get actionable advice

Choosing Analysis Thresholds

| Metric | Good | Acceptable | Poor | |--------|------|------------|------| | Phase dominance | <30% | 30-50% | >50% | | Parallel efficiency | >0.80 | 0.70-0.80 | <0.70 | | Memory usage | <60% | 60-80% | >80% |

Script Outputs (JSON Fields)

All scripts wrap their payload in a top-level object with two keys: inputs and results. The fields below live under results.

| Script | Key Outputs (under results) | |--------|-------------| | timing_analyzer.py | results.phases, results.slowest_phase, results.total_time | | scaling_analyzer.py | results.results, results.efficiency_threshold_processors, results.average_efficiency, results.baseline | | memory_profiler.py | results.total_memory_gb, results.per_process_gb, results.field_memory_gb, results.solver_workspace_gb, results.matrix_storage_gb, results.warnings | | bottleneck_detector.py | results.bottlenecks, results.recommendations |

Workflow

Complete Profiling Workflow

Analyze timing from simulation logs
Analyze scaling from multi-run data (if available)
Profile memory from simulation parameters
Detect bottlenecks and get recommendations
Implement optimizations based on recommendations
Re-profile to verify improvements

Quick Profiling (Timing Only)

Run timing analyzer on simulation log
Identify dominant phases (>50% of runtime)
Apply targeted optimizations to dominant phases

CLI Examples

Timing Analysis

# Basic timing analysis
python3 scripts/timing_analyzer.py \
    --log simulation.log \
    --json

# Custom timing pattern
python3 scripts/timing_analyzer.py \
    --log simulation.log \
    --pattern 'Step\s+(\w+)\s+took\s+([\d.]+)s' \
    --json

Scaling Analysis

# Strong scaling (fixed problem size)
python3 scripts/scaling_analyzer.py \
    --data scaling_data.json \
    --type strong \
    --json

# Weak scaling (constant work per processor)
python3 scripts/scaling_analyzer.py \
    --data scaling_data.json \
    --type weak \
    --json

Memory Profiling

# Estimate memory requirements
python3 scripts/memory_profiler.py \
    --params simulation_params.json \
    --available-gb 16.0 \
    --json

Bottleneck Detection

# Detect bottlenecks from timing only
python3 scripts/bottleneck_detector.py \
    --timing timing_results.json \
    --json

# Comprehensive analysis with all inputs
python3 scripts/bottleneck_detector.py \
    --timing timing_results.json \
    --scaling scaling_results.json \
    --memory memory_results.json \
    --json

Conversational Workflow Example

User: My simulation is taking too long. Can you help me identify what's slow?

Agent workflow:

Ask for simulation log file

Run timing analyzer:

python3 scripts/timing_analyzer.py --log simulation.log --json

Interpret results (the detector flags solver/assembly phases above 50% and I/O phases above 30%; >70% is high severity):
- If solver dominates (>50%, high above 70%): Recommend preconditioner tuning
- If assembly dominates (>50%): Recommend caching or vectorization
- If I/O dominates (>30%): Recommend reducing output frequency

If user has multi-run data, analyze scaling:

python3 scripts/scaling_analyzer.py --data scaling.json --type strong --json

Generate comprehensive recommendations:

python3 scripts/bottleneck_detector.py --timing timing.json --scaling scaling.json --json

Interpretation Guidance

Timing Analysis

The detector applies per-type dominance thresholds: solver/assembly/general phases are flagged above 50% of runtime; I/O phases above 30%. Any flagged phase above 70% is reported as high severity.

| Scenario | Meaning | Action | |----------|---------|--------| | Solver >50% (high >70%) | Solver-dominated | Tune preconditioner, check tolerance | | Assembly >50% | Assembly-dominated | Cache matrices, vectorize, parallelize | | I/O >30% | I/O-dominated | Reduce frequency, use parallel I/O | | Balanced (below thresholds) | Well-balanced | Look for algorithmic improvements |

Scaling Analysis

| Efficiency | Meaning | Action | |------------|---------|--------| | >0.80 | Excellent scaling | Continue scaling up | | 0.70-0.80 | Good scaling | Monitor at larger scales | | 0.50-0.70 | Poor scaling | Investigate communication/load balance | | <0.50 | Very poor scaling | Reduce processor count or redesign |

Memory Profile

| Usage | Meaning | Action | |-------|---------|--------| | <60% available | Safe | No action needed | | 60-80% available | Moderate | Monitor, consider optimization | | >80% available | High | Reduce resolution or increase processors | | >100% available | Exceeds capacity | Must reduce problem size |

The estimate follows the three-term formula Total = Field + Solver Workspace + Matrix Storage (see references/profiling_guide.md). Matrix storage and solver workspace depend on solver.type:

iterative (default): sparse matrix (default 7-point stencil, override via solver.stencil_nnz) plus workspace vectors.
direct: sparse matrix scaled by a conservative fill-in factor (solver.fillin_factor, default 10) to reflect factorization fill-in — a direct solver estimates far more memory than an iterative one for the same mesh.
matrix-free: no assembled matrix; workspace vectors only.

The estimate is intentionally conservative so a "will it fit in RAM?" decision does not silently under-estimate.

Error Handling

| Error | Cause | Resolution | |-------|-------|------------| | Log file not found | Invalid path | Verify log file path | | No timing data found | Pattern mismatch | Provide custom pattern with --pattern | | At least 2 runs required | Insufficient data | Provide more scaling runs | | Missing required parameters | Incomplete params | Add mesh and fields to params file |

Optimization Strategies by Bottleneck Type

Solver Bottlenecks

Use algebraic multigrid (AMG) preconditioner
Tighten solver tolerance if over-solving
Consider direct solver for small problems
Profile matrix assembly vs solve time

Assembly Bottlenecks

Cache element matrices if geometry is static
Use vectorized assembly routines
Consider matrix-free methods
Parallelize assembly with coloring

I/O Bottlenecks

Reduce output frequency
Use parallel I/O (HDF5, MPI-IO)
Write to fast scratch storage
Compress output data

Scaling Bottlenecks

Investigate communication overhead
Check for load imbalance
Reduce synchronization points
Use asynchronous communication
Consider hybrid MPI+OpenMP

Memory Bottlenecks

Reduce mesh resolution
Use iterative solver (lower memory than direct)
Enable out-of-core computation
Increase number of processors
Use single precision where appropriate

Verification checklist

Before trusting a profiling result or acting on a recommendation, record the concrete evidence below:

[ ] Confirmed timing_analyzer.py actually matched entries — results.phases is non-empty and results.total_time > 0; if a custom --pattern was used and results.message/suggested_patterns appeared, the pattern was fixed and re-run (an empty phases list silently looks like a fast simulation).
[ ] Cross-checked that the sum of phases[].percentage is ~100% and that named phases cover the wall-clock time — unaccounted-for time means missing log lines, not a balanced run.
[ ] For scaling claims, used >=2 runs spanning a real processor range and recorded results.average_efficiency and results.efficiency_threshold_processors from scaling_analyzer.py; verified the --type (strong vs weak) matches how the runs were generated (fixed total size vs fixed work-per-rank).
[ ] Recorded the memory breakdown from memory_profiler.py (field_memory_gb, solver_workspace_gb, matrix_storage_gb, total_memory_gb) and confirmed solver.type (iterative / direct / matrix-free) matches the real solver — a direct solve carries the ~10x fill-in factor and a wrong type makes the "fits in RAM?" answer unsafe.
[ ] Checked results.warnings and compared total_memory_gb (and per_process_gb) against the actual --available-gb; treated >80% as the documented "high" band, not a pass.
[ ] For each bottleneck_detector.py recommendation, confirmed the driving bottleneck (its category, value, and threshold) is consistent with the timing/scaling/memory inputs that were actually supplied — recommendations only reflect the JSON files passed via --timing/--scaling/--memory.
[ ] After implementing an optimization, re-ran the relevant analyzer and recorded the before/after value to confirm the bottleneck actually moved (re-profile step of the workflow).

Common pitfalls & rationalizations

| Tempting shortcut | Why it's wrong / what to do | |-------------------|------------------------------| | "timing_analyzer.py returned no bottlenecks, so the run is balanced." | An empty/low result is often a pattern mismatch — phases may be empty or partial. Verify total_time matches wall-clock and that phase percentages sum to ~100% before concluding "balanced". | | "Two runs scaled fine, so it scales." | Two points only give an average efficiency; they cannot reveal where efficiency falls off. Add more processor counts and check efficiency_threshold_processors, and confirm you used the correct --type (strong vs weak). | | "Iterative vs direct is just a flag; memory is about the same." | memory_profiler.py applies a conservative ~10x fill-in factor for direct and stores no matrix for matrix-free. Setting the wrong solver.type can under-estimate RAM by an order of magnitude — set it to the real solver. | | "It fits in --available-gb total, so we're fine." | The relevant number for an MPI run is per_process_gb against per-node/per-rank RAM, and >80% of total already triggers a warning. Check the per-process figure and the warnings list, not just the total. | | "I/O is under 50%, so I/O isn't the bottleneck." | I/O is flagged at the lower 30% threshold, not 50%. A 30-50% I/O phase is a real bottleneck the detector reports — reduce output frequency or use parallel I/O. | | "The recommendation says tune the preconditioner, so the solver is the problem." | Recommendations are only as complete as the JSON you passed in. If --scaling/--memory were omitted, those bottlenecks are simply invisible — feed all available analyses before trusting the priority ranking. |

Security

Input Validation

User-supplied --pattern regex values are validated for length (500 chars max) and rejected if they contain constructs prone to catastrophic backtracking (ReDoS)
Scaling data entries are validated for finite time values, integer processor counts, and bounded run count (10,000 max)
available_gb is validated as a positive finite number; mesh dimensions and field parameters are validated as positive integers
--type (scaling type) is validated against a fixed allowlist (strong, weak)
All loaded JSON files must have an object (dict) as root element

File Access

timing_analyzer.py reads a single log file specified by --log; log files are capped at 500 MB and rejected before parsing
scaling_analyzer.py, memory_profiler.py, and bottleneck_detector.py read JSON files capped at 100 MB
Phase names extracted from log files are truncated to 200 characters and stripped of control characters to prevent prompt-injection payloads from propagating into agent context
No scripts write to the filesystem; all output goes to stdout

Tool Restrictions

Read: Used to inspect script source, references, simulation logs, and result files
Write: Used to save profiling reports or optimization recommendations; writes are scoped to the user's working directory
Grep/Glob: Used to locate log files, result files, and search references
The skill's allowed-tools excludes Bash to prevent the agent from executing arbitrary commands when processing untrusted simulation logs or result files

Safety Measures

No eval(), exec(), or dynamic code generation
All subprocess calls use explicit argument lists (no shell=True)
Reduced tool surface (no Bash) limits the agent to read/write operations only
Phase names and diagnostic strings are sanitized before inclusion in output to prevent injection

Limitations

Log parsing: Depends on pattern matching; may miss unusual formats
Scaling analysis: Requires at least 2 runs for meaningful results
Memory estimation: Approximate; actual usage may vary
Recommendations: General guidance; may need domain-specific tuning

References

references/profiling_guide.md - Profiling concepts and interpretation
references/optimization_strategies.md - Detailed optimization approaches

Version History

See CHANGELOG.md for the authoritative, dated release history.

Agent Skills: Performance Profiling

Install this agent skill to your local

Skill Files