Python Performance Optimization

Seth HobsonCommunityCreated by Seth Hobson

Skill
Guidelines
Software development
MIT
Low risk

From the author

Profile and optimize Python code using cProfile, memory profilers, and performance best practices. Use when debugging slow Python code, optimizing bottlenecks, or improving application performance.

— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to identifying and resolving performance bottlenecks in Python applications. It covers essential profiling techniques, including CPU, memory, and line-by-line analysis, alongside practical optimization strategies such as algorithmic improvements, caching, parallelization, and the use of efficient data structures. The guide emphasizes a data-driven approach, advocating for profiling before optimization to ensure efforts are focused on the most impactful areas of the codebase.

What it does

Provides methods for CPU, memory, and line-by-line profiling.
Offers strategies for reducing latency and memory consumption.
Demonstrates how to use caching to avoid redundant computations.
Explains techniques for parallelizing CPU-bound tasks.
Guides the use of native extensions and vectorized operations for performance.
Outlines best practices for writing high-performance Python code.

When to use it

When debugging slow Python applications or identifying bottlenecks.
When reducing application latency and response times.
When optimizing memory usage or addressing memory leaks.
When scaling data processing pipelines.
When profiling production applications to understand real-world performance.

Advantages

Promotes evidence-based optimization through profiling.
Provides actionable patterns for common performance issues.
Covers both development-time and production-time profiling tools.
Encourages efficient coding habits and data structure selection.

Usage examples

“How can I identify which functions are slowing down my Python script?”: The skill explains how to use cProfile to generate performance statistics and sort them by cumulative time to pinpoint bottlenecks.
“My application is consuming too much memory. How can I optimize it?”: The skill suggests using memory_profiler to track usage and implementing __slots__ in classes to reduce memory overhead.
“How do I speed up a CPU-intensive task in Python?”: The skill provides patterns for using the multiprocessing module to parallelize tasks and NumPy for vectorized numerical operations.
“How can I cache the results of an expensive recursive function?”: The skill demonstrates the use of the functools.lru_cache decorator to store and reuse function results.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/wshobson/python-performance-optimization/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip python-performance-optimization.zip -d .agents/skills

Documentation by Seth Hobson

Python Performance Optimization

Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.

When to Use This Skill

Identifying performance bottlenecks in Python applications
Reducing application latency and response times
Optimizing CPU-intensive operations
Reducing memory consumption and memory leaks
Improving database query performance
Optimizing I/O operations
Speeding up data processing pipelines
Implementing high-performance algorithms
Profiling production applications

Core Concepts

1. Profiling Types

CPU Profiling: Identify time-consuming functions
Memory Profiling: Track memory allocation and leaks
Line Profiling: Profile at line-by-line granularity
Call Graph: Visualize function call relationships

2. Performance Metrics

Execution Time: How long operations take
Memory Usage: Peak and average memory consumption
CPU Utilization: Processor usage patterns
I/O Wait: Time spent on I/O operations

3. Optimization Strategies

Algorithmic: Better algorithms and data structures
Implementation: More efficient code patterns
Parallelization: Multi-threading/processing
Caching: Avoid redundant computation
Native Extensions: C/Rust for critical paths

Quick Start

Basic Timing

import time

def measure_time():
    """Simple timing measurement."""
    start = time.time()

    # Your code here
    result = sum(range(1000000))

    elapsed = time.time() - start
    print(f"Execution time: {elapsed:.4f} seconds")
    return result

# Better: use timeit for accurate measurements
import timeit

execution_time = timeit.timeit(
    "sum(range(1000000))",
    number=100
)
print(f"Average time: {execution_time/100:.6f} seconds")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Profile before optimizing - Measure to find real bottlenecks
Focus on hot paths - Optimize code that runs most frequently
Use appropriate data structures - Dict for lookups, set for membership
Avoid premature optimization - Clarity first, then optimize
Use built-in functions - They're implemented in C
Cache expensive computations - Use lru_cache
Batch I/O operations - Reduce system calls
Use generators for large datasets
Consider NumPy for numerical operations
Profile production code - Use py-spy for live systems

Common Pitfalls

Optimizing without profiling
Using global variables unnecessarily
Not using appropriate data structures
Creating unnecessary copies of data
Not using connection pooling for databases
Ignoring algorithmic complexity
Over-optimizing rare code paths
Not considering memory usage

Files

ATTRIBUTION.mdAdded by AutoGineer1.0 KiB
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md3.2 KiB
references/advanced-patterns.md10.2 KiB
references/details.md7.2 KiB

references/details.md

python-performance-optimization — detailed patterns and worked examples

Profiling Tools

Pattern 1: cProfile - CPU Profiling

import cProfile
import pstats
from pstats import SortKey

def slow_function():
    """Function to profile."""
    total = 0
    for i in range(1000000):
        total += i
    return total

def another_function():
    """Another function."""
    return [i**2 for i in range(100000)]

def main():
    """Main function to profile."""
    result1 = slow_function()
    result2 = another_function()
    return result1, result2

# Profile the code
if __name__ == "__main__":
    profiler = cProfile.Profile()
    profiler.enable()

    main()

    profiler.disable()

    # Print stats
    stats = pstats.Stats(profiler)
    stats.sort_stats(SortKey.CUMULATIVE)
    stats.print_stats(10)  # Top 10 functions

    # Save to file for later analysis
    stats.dump_stats("profile_output.prof")

Command-line profiling:

# Profile a script
python -m cProfile -o output.prof script.py

# View results
python -m pstats output.prof
# In pstats:
# sort cumtime
# stats 10

Pattern 2: line_profiler - Line-by-Line Profiling

# Install: pip install line-profiler

# Add @profile decorator (line_profiler provides this)
@profile
def process_data(data):
    """Process data with line profiling."""
    result = []
    for item in data:
        processed = item * 2
        result.append(processed)
    return result

# Run with:
# kernprof -l -v script.py

Manual line profiling:

from line_profiler import LineProfiler

def process_data(data):
    """Function to profile."""
    result = []
    for item in data:
        processed = item * 2
        result.append(processed)
    return result

if __name__ == "__main__":
    lp = LineProfiler()
    lp.add_function(process_data)

    data = list(range(100000))

    lp_wrapper = lp(process_data)
    lp_wrapper(data)

    lp.print_stats()

Pattern 3: memory_profiler - Memory Usage

# Install: pip install memory-profiler

from memory_profiler import profile

@profile
def memory_intensive():
    """Function that uses lots of memory."""
    # Create large list
    big_list = [i for i in range(1000000)]

    # Create large dict
    big_dict = {i: i**2 for i in range(100000)}

    # Process data
    result = sum(big_list)

    return result

if __name__ == "__main__":
    memory_intensive()

# Run with:
# python -m memory_profiler script.py

Pattern 4: py-spy - Production Profiling

# Install: pip install py-spy

# Profile a running Python process
py-spy top --pid 12345

# Generate flamegraph
py-spy record -o profile.svg --pid 12345

# Profile a script
py-spy record -o profile.svg -- python script.py

# Dump current call stack
py-spy dump --pid 12345

Optimization Patterns

Pattern 5: List Comprehensions vs Loops

import timeit

# Slow: Traditional loop
def slow_squares(n):
    """Create list of squares using loop."""
    result = []
    for i in range(n):
        result.append(i**2)
    return result

# Fast: List comprehension
def fast_squares(n):
    """Create list of squares using comprehension."""
    return [i**2 for i in range(n)]

# Benchmark
n = 100000

slow_time = timeit.timeit(lambda: slow_squares(n), number=100)
fast_time = timeit.timeit(lambda: fast_squares(n), number=100)

print(f"Loop: {slow_time:.4f}s")
print(f"Comprehension: {fast_time:.4f}s")
print(f"Speedup: {slow_time/fast_time:.2f}x")

# Even faster for simple operations: map
def faster_squares(n):
    """Use map for even better performance."""
    return list(map(lambda x: x**2, range(n)))

Pattern 6: Generator Expressions for Memory

import sys

def list_approach():
    """Memory-intensive list."""
    data = [i**2 for i in range(1000000)]
    return sum(data)

def generator_approach():
    """Memory-efficient generator."""
    data = (i**2 for i in range(1000000))
    return sum(data)

# Memory comparison
list_data = [i for i in range(1000000)]
gen_data = (i for i in range(1000000))

print(f"List size: {sys.getsizeof(list_data)} bytes")
print(f"Generator size: {sys.getsizeof(gen_data)} bytes")

# Generators use constant memory regardless of size

Pattern 7: String Concatenation

import timeit

def slow_concat(items):
    """Slow string concatenation."""
    result = ""
    for item in items:
        result += str(item)
    return result

def fast_concat(items):
    """Fast string concatenation with join."""
    return "".join(str(item) for item in items)

def faster_concat(items):
    """Even faster with list."""
    parts = [str(item) for item in items]
    return "".join(parts)

items = list(range(10000))

# Benchmark
slow = timeit.timeit(lambda: slow_concat(items), number=100)
fast = timeit.timeit(lambda: fast_concat(items), number=100)
faster = timeit.timeit(lambda: faster_concat(items), number=100)

print(f"Concatenation (+): {slow:.4f}s")
print(f"Join (generator): {fast:.4f}s")
print(f"Join (list): {faster:.4f}s")

Pattern 8: Dictionary Lookups vs List Searches

import timeit

# Create test data
size = 10000
items = list(range(size))
lookup_dict = {i: i for i in range(size)}

def list_search(items, target):
    """O(n) search in list."""
    return target in items

def dict_search(lookup_dict, target):
    """O(1) search in dict."""
    return target in lookup_dict

target = size - 1  # Worst case for list

# Benchmark
list_time = timeit.timeit(
    lambda: list_search(items, target),
    number=1000
)
dict_time = timeit.timeit(
    lambda: dict_search(lookup_dict, target),
    number=1000
)

print(f"List search: {list_time:.6f}s")
print(f"Dict search: {dict_time:.6f}s")
print(f"Speedup: {list_time/dict_time:.0f}x")

Pattern 9: Local Variable Access

import timeit

# Global variable (slow)
GLOBAL_VALUE = 100

def use_global():
    """Access global variable."""
    total = 0
    for i in range(10000):
        total += GLOBAL_VALUE
    return total

def use_local():
    """Use local variable."""
    local_value = 100
    total = 0
    for i in range(10000):
        total += local_value
    return total

# Local is faster
global_time = timeit.timeit(use_global, number=1000)
local_time = timeit.timeit(use_local, number=1000)

print(f"Global access: {global_time:.4f}s")
print(f"Local access: {local_time:.4f}s")
print(f"Speedup: {global_time/local_time:.2f}x")

Pattern 10: Function Call Overhead

import timeit

def calculate_inline():
    """Inline calculation."""
    total = 0
    for i in range(10000):
        total += i * 2 + 1
    return total

def helper_function(x):
    """Helper function."""
    return x * 2 + 1

def calculate_with_function():
    """Calculation with function calls."""
    total = 0
    for i in range(10000):
        total += helper_function(i)
    return total

# Inline is faster due to no call overhead
inline_time = timeit.timeit(calculate_inline, number=1000)
function_time = timeit.timeit(calculate_with_function, number=1000)

print(f"Inline: {inline_time:.4f}s")
print(f"Function calls: {function_time:.4f}s")

For advanced optimization techniques including NumPy vectorization, caching, memory management, parallelization, async I/O, database optimization, and benchmarking tools, see references/advanced-patterns.md

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Seth Hobson

Publisher

Seth Hobson (Community)

Source

wshobson/agents · plugins/python-development/skills/python-performance-optimization · commit 4bde316

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Seth Hobson.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download python-performance-optimization.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Seth Hobson

Python Performance Optimization

When to Use This Skill

Core Concepts

1. Profiling Types

2. Performance Metrics

3. Optimization Strategies

Quick Start

Basic Timing

Detailed patterns and worked examples

Best Practices

Common Pitfalls

Files

references/details.md

python-performance-optimization — detailed patterns and worked examples

Profiling Tools

Pattern 1: cProfile - CPU Profiling

Pattern 2: line_profiler - Line-by-Line Profiling

Pattern 3: memory_profiler - Memory Usage

Pattern 4: py-spy - Production Profiling

Optimization Patterns

Pattern 5: List Comprehensions vs Loops

Pattern 6: Generator Expressions for Memory

Pattern 7: String Concatenation

Pattern 8: Dictionary Lookups vs List Searches

Pattern 9: Local Variable Access

Pattern 10: Function Call Overhead

Requirements and compatibility

Security

Attribution and licence

Related skills

Diagnosing Bugs

Systematic Debugging

Performance Optimization

Debugging And Error Recovery

Animation Performance

Parallel Debugging

Download ZIP