modshim

A Python library for enhancing existing modules without modifying their source code - a clean alternative to forking, vendoring, and monkey-patching.

Overview

modshim allows you to overlay custom functionality onto existing Python modules while preserving their original behavior. This is particularly useful when you need to:

• Fix bugs in third-party libraries without forking
• Modify the behavior of existing functions
• Add new features or options to existing classes
• Test alternative implementations in an isolated way

It works by creating a new, "shimmed" module that combines the original code with your enhancements, without modifying the original module.

Installation

pip install modshim

Usage

Suppose we want to enhance the standard library's textwrap module. Our goal is to add a prefix argument to TextWrapper to prepend a string to every wrapped line.

First, create a Python module containing your modifications. It should mirror the structure of the original textwrap module, redefining only the parts we want to modify. For classes like TextWrapper, this can be done by creating a subclass with the same name as the original class:

# prefixed_textwrap.py

# Import the class you want to extend from the original module
from textwrap import TextWrapper as OriginalTextWrapper

# Sub-class to override and extend functionality
class TextWrapper(OriginalTextWrapper):
    """Enhanced TextWrapper that adds a prefix to each line."""

    def __init__(self, *args, prefix: str = "", **kwargs) -> None:
        self.prefix = prefix
        super().__init__(*args, **kwargs)

    def wrap(self, text: str) -> list[str]:
        """Wrap text and add prefix to each line."""
        original_lines = super().wrap(text)
        if not self.prefix:
            return original_lines
        return [f"{self.prefix}{line}" for line in original_lines]

Next, use modshim to mount your modifications over the original textwrap module, creating a new, combined module.

>>> from modshim import shim
>>> shim(
...     lower="textwrap",           # Original module to enhance
...     upper="prefixed_textwrap",  # Module with your modifications
...     mount="super_textwrap",     # Name for the new, merged module
... )

Now, you can import from super_textwrap. Notice how we can call the original wrap() convenience function, but pass our new prefix argument. This works because our modifications are overlaid on the original textwrap module, so the merged super_textwrap module uses our enhanced TextWrapper class inside the wrap function instead of the original.

>>> from super_textwrap import wrap
>>>
>>> text = "This is a long sentence that will be wrapped into multiple lines."
>>> for line in wrap(text, width=30, prefix="> "):
...     print(line)
...
> This is a long sentence that
> will be wrapped into
> multiple lines.

Crucially, the original textwrap module remains completely unchanged. This is the key advantage over monkey-patching.

# The original module is untouched
>>> from textwrap import wrap
>>>
# It works as it always did, without the 'prefix' argument
>>> text = "This is a long sentence that will be wrapped into multiple lines."
>>> for line in wrap(text, width=30):
...     print(line)
...
This is a long sentence that
will be wrapped into
multiple lines.

# Trying to use our new feature with the original module will fail, as expected
>>> wrap(text, width=30, prefix="> ")
Traceback (most recent call last):
  ...
TypeError: TextWrapper.__init__() got an unexpected keyword argument 'prefix'

Creating Enhancement Packages

You can create packages that automatically apply a shim to another module, making your enhancements available just by importing your package.

This is done by calling shim() from within your package's code and using your own package's name as the mount point, so your package gets "replaced" with new merged module.

To adapt our textwrap example, we could create a super_textwrap.py file like this:

# super_textwrap.py
from textwrap import TextWrapper as OriginalTextWrapper
from modshim import shim

# Define your enhancements as before
class TextWrapper(OriginalTextWrapper):
    """Enhanced TextWrapper that adds a prefix to each line."""

    def __init__(self, *args, prefix: str = "", **kwargs) -> None:
        self.prefix = prefix
        super().__init__(*args, **kwargs)

    def wrap(self, text: str) -> list[str]:
        original_lines = super().wrap(text)
        if not self.prefix:
            return original_lines
        return [f"{self.prefix}{line}" for line in original_lines]

# Apply the shim at import time. This replaces the 'super_textwrap'
# module in sys.modules with the new, combined module.
shim(lower="textwrap")
# - The `upper` parameter defaults to the calling module ('super_textwrap').
# - The `mount` parameter defaults to `upper`, so it is also 'super_textwrap'.

Now, anyone can use your enhanced version simply by importing your package:

>>> from super_textwrap import wrap
>>>
>>> text = "This is a long sentence that will be wrapped into multiple lines."
>>> for line in wrap(text, width=30, prefix="* "):
...     print(line)
...
* This is a long sentence that
* will be wrapped into
* multiple lines.

Mounting Over the Original Module

In some cases, you may want your enhanced module to completely replace the original module name. This allows existing code to benefit from your enhancements without changing any import statements.

For example, suppose you want all imports of textwrap in your application to automatically use your enhanced version with the prefix feature. You can do this by setting the mount point to the same name as the lower module:

# my_app/setup_enhancements.py
from modshim import shim

shim(
    lower="textwrap",
    upper="prefixed_textwrap",
    mount="textwrap",  # Mount over the original module name
)

After this shim() call executes, any subsequent imports of textwrap will use the combined module:

# my_app/main.py
from my_app import setup_enhancements  # Apply the shim first

# Now the standard import gets our enhanced version!
from textwrap import wrap

text = "This is a long sentence that will be wrapped into multiple lines."
for line in wrap(text, width=30, prefix=">> "):
    print(line)

Output:

>> This is a long sentence that
>> will be wrapped into
>> multiple lines.

Use Case: Transparent Bug Fixes

This pattern is particularly useful when you need to apply a bug fix or security patch to a library that is used throughout your codebase or by third-party dependencies. Instead of modifying every import statement, you can apply the fix once at application startup:

# my_app/__init__.py
from modshim import shim

# Apply our fix to the json module globally
shim(lower="json", upper="json_fixes", mount="json")

Now all code—including third-party libraries—that imports json will use your patched version.

Caution

Mounting over the original module affects all subsequent imports in your application, including those from third-party libraries. Use this pattern carefully and ensure your enhancements are fully backward-compatible with the original module's API.

Advanced Example: Adding Configurable Retries to requests

Let's tackle a more complex, real-world scenario. We want to add a robust, configurable retry mechanism to requests for handling transient network issues or server errors. The standard way to do this in requests is by creating a Session object and mounting a custom HTTPAdapter.

With modshim, we can create an enhanced Session class that automatically configures retries, and then overlay it onto the original requests library. To do this correctly, our enhancement package (requests_extra) must mirror the structure of the original requests package.

Step 1: Create the Enhancement Package

We'll create a package named requests_extra with two files.

requests_extra/__init__.py

This is the entry point to our package. It contains the magic shim call.

# requests_extra/__init__.py

from modshim import shim

# This overlays our 'requests_extra' package on the original 'requests' package.
# Because 'mount' isn't specified, it defaults to our package name ('requests_extra').
# When a submodule like 'requests_extra.sessions' is imported, modshim will
# automatically merge it with the original 'requests.sessions'.
shim(lower="requests")

requests_extra/sessions.py

This file matches the location of the Session class in the original requests library (requests/sessions.py). Here, we define our enhanced Session with the new functionality, by subclassing the original Session implementation:

# requests_extra/sessions.py

from requests.adapters import HTTPAdapter
# We will create a new version of `Session` by subclassing the original
from requests.sessions import Session as OriginalSession
from urllib3.util.retry import Retry


class Session(OriginalSession):
    """
    Enhanced Session that adds automatic, configurable retries via a mounted HTTPAdapter.

    Accepts new keyword arguments in its constructor:
    - retries (int): Total number of retries to allow.
    - backoff_factor (float): A factor to apply between retry attempts.
    - status_forcelist (iterable): A set of HTTP status codes to force a retry on.
    """

    def __init__(self, *args, **kwargs):
        # Extract our custom arguments before calling the parent constructor
        retries = kwargs.pop("retries", 3)
        backoff_factor = kwargs.pop("backoff_factor", 0.1)
        status_forcelist = kwargs.pop("status_forcelist", (500, 502, 503, 504))

        super().__init__(*args, **kwargs)

        # If retries are configured, create a retry strategy and mount it
        if retries > 0: