SmartProfiler

SmartProfiler is a comprehensive Python library designed to help you profile and optimize your code by tracking execution time, disk I/O, memory usage, network I/O, and function call counts. It provides an all-in-one solution for performance analysis, complete with integrated visualization to make sense of your profiling data. Whether you're debugging bottlenecks, optimizing resource usage, or monitoring function calls, SmartProfiler offers a simple and efficient toolset for your needs.

GitHub Repository

Why SmartProfiler?

Multi-Faceted Profiling: Profile execution time, disk I/O, memory usage, network I/O, and function calls in a single library, reducing the need for multiple tools.
Integrated Visualization: Generate normalized bar charts with a single command to visualize performance metrics across different sections of your code.
Flexible Profiling: Use decorators or context managers to profile at the function, block, or line level with minimal setup.
Lightweight: Designed to introduce minimal performance overhead while providing accurate profiling data.
Detailed Logging: Integration with Python's logging framework for real-time insights into your code's performance.

Features

Function-Level Profiling: Profile functions for execution time, disk writes, memory usage, network activity, and call counts using decorators.
Block and Line Profiling: Profile specific code blocks or individual lines using context managers.
Visualization: Generate a single PNG file with subplots for each metric, showing normalized values and raw data for easy comparison.
Customizable Metrics: Configure which metrics to profile (e.g., write_bytes for disk, bytes_sent for network).
Logging Support: Detailed logs of profiling stats with customizable log levels (e.g., INFO, DEBUG) or the ability to disable logging entirely.

Installation

Install SmartProfiler via pip:

pip install smartprofiler

Requirements:

Python 3.8 or higher Dependencies: matplotlib, numpy, requests, psutil

Usage Examples

SmartProfiler provides a unified interface through profiler classes (CPUProfiler, DiskProfiler, etc.) that can be used as decorators or context managers. Below are examples demonstrating how to use the library for profiling and visualization.

1. General Usage Examples

The following examples show how to use SmartProfiler to profile different aspects of your code, such as CPU usage, disk I/O, function calls, memory usage, and network I/O. These examples focus on logging the profiling results.

1.1 Profiling with Standard Logger

import logging
from smartprofiler import CPUProfiler

# Initialize the CPU profiler with standard logger
logger = logging.getLogger(__name__)
cpu_profiler = CPUProfiler(logger=logger)

@cpu_profiler.profile_function
def compute_fibonacci(n):
    if n <= 1:
        return n
    return compute_fibonacci(n-1) + compute_fibonacci(n-2)

# Run the function
compute_fibonacci(20)

1.2 Profiling with Loguru Logger

from loguru import logger
from smartprofiler import CPUProfiler

# Initialize the CPU profiler with loguru logger
cpu_profiler = CPUProfiler(logger=logger)

@cpu_profiler.profile_function
def compute_fibonacci(n):
    if n <= 1:
        return n
    return compute_fibonacci(n-1) + compute_fibonacci(n-2)

# Run the function
compute_fibonacci(20)

1.3 Profiling with Structlog Logger

import structlog
from smartprofiler import CPUProfiler

# Configure structlog
structlog.configure(
    processors=[
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer()
    ]
)

# Initialize the CPU profiler with structlog logger
logger = structlog.get_logger()
cpu_profiler = CPUProfiler(logger=logger)

@cpu_profiler.profile_function
def compute_fibonacci(n):
    if n <= 1:
        return n
    return compute_fibonacci(n-1) + compute_fibonacci(n-2)

# Run the function
compute_fibonacci(20)

1.4 Custom Logger Implementation

You can also use any custom logger that implements the required logging interface. The logger must have at least these methods:

info(msg, *args, **kwargs)
error(msg, *args, **kwargs)
debug(msg, *args, **kwargs)
warning(msg, *args, **kwargs)
log(level, msg, *args, **kwargs)

class CustomLogger:
    def info(self, msg, *args, **kwargs):
        print(f"INFO: {msg}")
    
    def error(self, msg, *args, **kwargs):
        print(f"ERROR: {msg}")
    
    def debug(self, msg, *args, **kwargs):
        print(f"DEBUG: {msg}")
    
    def warning(self, msg, *args, **kwargs):
        print(f"WARNING: {msg}")
    
    def log(self, level, msg, *args, **kwargs):
        print(f"LOG[{level}]: {msg}")

# Initialize the CPU profiler with custom logger
logger = CustomLogger()
cpu_profiler = CPUProfiler(logger=logger)

@cpu_profiler.profile_function
def compute_fibonacci(n):
    if n <= 1:
        return n
    return compute_fibonacci(n-1) + compute_fibonacci(n-2)

# Run the function
compute_fibonacci(20)

1.2 Profiling a Disk-Intensive Block


from smartprofiler import DiskProfiler

# Initialize the Disk profiler
disk_profiler = DiskProfiler(disk_path='/tmp', disk_metrics={'write_bytes': True, 'disk_usage': False})

# Profile a disk-intensive block
with disk_profiler.profile_block("disk_write_block"):
    with open('/tmp/test_file.txt', 'w') as f:
        f.write("Sample data " * 1000)

# Summarize statistics
disk_profiler.summarize_stats()

1.3 Profiling Network I/O with Logging Disabled

import requests
from smartprofiler import NetworkProfiler

# Initialize the Network profiler with logging disabled
net_profiler = NetworkProfiler(network_metrics={'bytes_sent': True, 'bytes_recv': True}, enable_logging=False)

# Profile a network-intensive block
with net_profiler.profile_block("api_request"):
    requests.get('https://api.github.com')

# Manually print stats (since logging is disabled)
stats = net_profiler.get_stats()
print("Network stats:", stats)

See examples/examples_general_usage.py for more usage examples, including profiling function calls, memory usage, and multithreaded scenarios.

2. Visualization Examples

SmartProfiler can generate visualizations of profiling data using the plot_profiling_stats function. The following example demonstrates how to profile various tasks and visualize the results.

2.1 Profiling and Visualizing CPU, Disk, and Memory Usage

import time
import random
from smartprofiler import CPUProfiler, DiskProfiler, MemoryProfiler, plot_profiling_stats

# Initialize profilers
cpu_profiler = CPUProfiler(time_func='execution_time')
disk_profiler = DiskProfiler(disk_path='/tmp', disk_metrics={'write_bytes': True, 'disk_usage': False})
mem_profiler = MemoryProfiler()

# Profile a CPU-intensive function
@cpu_profiler.profile_function
@mem_profiler.profile_function
def matrix_multiply():
    size = 500
    matrix_a = [[random.random() for _ in range(size)] for _ in range(size)]
    matrix_b = [[random.random() for _ in range(size)] for _ in range(size)]
    result = [[0 for _ in range(size)] for _ in range(size)]
    for i in range(size):
        for j in range(size):
            for k in range(size):
                result[i][j] += matrix_a[i][k] * matrix_b[k][j]
    return result

# Run the function
matrix_multiply()

# Generate visualization
plot_profiling_stats(
    [cpu_profiler, disk_profiler, mem_profiler],
    output_dir='images',
    output_file='profiling_stats_cpu_disk_memory.png'
)

Visualization Results

CPU, Disk, and Memory Profiling: This visualization includes CPU-intensive tasks (matrix multiplication), disk I/O (writing large files), and memory usage (large list allocation).
Network and Function Call Profiling: This visualization highlights network I/O (multiple API requests) and function calls (recursive Fibonacci calculation).

See examples/examples_visualization.py for the complete visualization examples, which include additional scenarios like network I/O and function call profiling.

3. Multithreaded Profiling

SmartProfiler supports profiling in multithreaded environments. Here's an example:

import threading
from smartprofiler import CPUProfiler

cpu_profiler = CPUProfiler(time_func='execution_time')

def thread_function():
    with cpu_profiler.profile_block("thread_task"):
        time.sleep(1)

# Create and run threads
threads = [threading.Thread(target=thread_function) for _ in range(5)]
for t in threads:
    t.start()
for t in threads:
    t.join()

# Summarize stats
cpu_profiler.summarize_stats()

Contributing to SmartProfiler

Contributions to SmartProfiler are welcome! Whether you're fixing a bug, adding a feature, or improving documentation, your help is appreciated. Please see the CONTRIBUTING.md file for detailed guidelines on how to contribute, including setup instructions, coding standards, and the contribution process.

Contributor Recognition

All contributors will be credited in the CONTRIBUTORS.md file, and their work will be documented in GitHub pull requests, providing clear evidence of their contributions for professional portfolios, including EB1A visa applications.

Code of Conduct

By contributing, you agree to follow the Code of Conduct, ensuring a positive environment for all.

Contributors

SmartProfiler is led by Vignesh Sundaram (GitHub: vigsun19), who is the lead developer, owner, and maintainer of the project.

The following individuals have contributed to SmartProfiler:

Vignesh Sundaram (GitHub: vigsun19)

See the CONTRIBUTORS.md file for a list of contributors and their contributions.

How to Contribute

Fork the Repository:
Start by forking the SmartProfiler repository.
Make Your Changes:

Smartprofiler

Install / Use

README