Python Performance Optimization Skill

Python Performance Optimization

Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.

When to Use This Skill

Identifying performance bottlenecks in Python applications
Reducing application latency and response times
Optimizing CPU-intensive operations
Reducing memory consumption and memory leaks
Improving database query performance
Optimizing I/O operations
Speeding up data processing pipelines
Implementing high-performance algorithms
Profiling production applications

Core Concepts

1. Profiling Types

CPU Profiling: Identify time-consuming functions
Memory Profiling: Track memory allocation and leaks
Line Profiling: Profile at line-by-line granularity
Call Graph: Visualize function call relationships

2. Performance Metrics

Execution Time: How long operations take
Memory Usage: Peak and average memory consumption
CPU Utilization: Processor usage patterns
I/O Wait: Time spent on I/O operations

3. Optimization Strategies

Algorithmic: Better algorithms and data structures
Implementation: More efficient code patterns
Parallelization: Multi-threading/processing
Caching: Avoid redundant computation
Native Extensions: C/Rust for critical paths

Quick Start

Basic Timing

import time

def measure_time():
    """Simple timing measurement."""
    start = time.time()

    # Your code here
    result = sum(range(1000000))

    elapsed = time.time() - start
    print(f"Execution time: {elapsed:.4f} seconds")
    return result

# Better: use timeit for accurate measurements
import timeit

execution_time = timeit.timeit(
    "sum(range(1000000))",
    number=100
)
print(f"Average time: {execution_time/100:.6f} seconds")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Profile before optimizing - Measure to find real bottlenecks
Focus on hot paths - Optimize code that runs most frequently
Use appropriate data structures - Dict for lookups, set for membership
Avoid premature optimization - Clarity first, then optimize
Use built-in functions - They're implemented in C
Cache expensive computations - Use lru_cache
Batch I/O operations - Reduce system calls
Use generators for large datasets
Consider NumPy for numerical operations
Profile production code - Use py-spy for live systems

Common Pitfalls

Optimizing without profiling
Using global variables unnecessarily
Not using appropriate data structures
Creating unnecessary copies of data
Not using connection pooling for databases
Ignoring algorithmic complexity
Over-optimizing rare code paths
Not considering memory usage

Agent Skills: Python Performance Optimization

Install this agent skill to your local

Skill Files