Snex 🐍

Easy and efficient Python interop for Elixir.

Highlights

Robust & Isolated - Run any number of Python interpreters in separate OS processes, preventing GIL issues or blocking computations from affecting your Elixir application. You can call asyncio code, use PyPy instead of CPython, or even run Python in a Docker container!

Declarative Environments - Leverages [uv][uv] to manage Python versions and dependencies, embedding them into your application's release for consistent deployments. Supports custom Python environments and easy integration with Python projects.

Bidirectional communication - Powerful and efficient interface with explicit control over data. Python code running under Snex can send messages to BEAM processes and call Erlang/Elixir functions.

High quality, organic code - Every line of Snex is thought out and serves a purpose. Code is optimized to keep performance overhead low.

Forward Compatibility - Built on stable foundations independent of C-level interfaces, so future versions of Python and Elixir will work on day one!

Quick example

defmodule SnexTest.NumpyInterpreter do
  use Snex.Interpreter,
    pyproject_toml: """
    [project]
    name = "my-numpy-project"
    version = "0.0.0"
    requires-python = "==3.10.*"
    dependencies = ["numpy>=2"]
    """
end

{:ok, interpreter} = SnexTest.NumpyInterpreter.start_link(init_script: "import numpy as np")

{:ok, 6.0} =
  Snex.pyeval(interpreter, """
    width, height = await Elixir.Enum.map([3, 3], mapper)
    matrix = np.fromfunction(lambda i, j: (-1) ** (i + j), (width, height), dtype=int)

    return np.linalg.norm(matrix)
    """, %{"mapper" => &(&1 * 2)})

Installation & Requirements

Elixir >= 1.18
uv >= 0.6.8 - A fast Python package & project manager, used by Snex to create and manage Python environments. It has to be available at compilation time but isn't needed at runtime.
Python >= 3.10 - This is the minimum supported version you can run your scripts with. You don't need to have it installed - Snex will fetch it with uv.

# mix.exs
def deps do
  [
    {:snex, "~> 0.4.1"}
  ]
end

# See the Releases section in the README on how to configure mix release

Core Concepts & Usage

Custom Interpreter
Snex.pyeval
Environments
- Initialization script
- Passing Snex.Env between Erlang nodes
Python Interface Documentation
Serialization
- Encoding/decoding table
- Customizing serialization
Releases
Cookbook

Custom Interpreter

You can define your Python project settings using use Snex.Interpreter in your module.

Set a required Python version and any dependencies - both the Python binary and the dependencies will be fetched & synced at compile time with [uv][uv], and put into the _build/$MIX_ENV/snex directory.

defmodule SnexTest.NumpyInterpreter do
  use Snex.Interpreter,
    pyproject_toml: """
    [project]
    name = "my-numpy-project"
    version = "0.0.0"
    requires-python = "==3.10.*"
    dependencies = ["numpy>=2"]
    """
end

Modules using Snex.Interpreter have to be start_linked to be used. Each Snex.Interpreter (BEAM) process manages a separate Python (OS) process.

{:ok, interpreter} = SnexTest.NumpyInterpreter.start_link()
{:ok, "hello world!"} = Snex.pyeval(interpreter, "return 'hello world!'")

`Snex.pyeval`

The main way of interacting with the interpreter process is Snex.pyeval/4 (and other arities). This is the function that runs Python code, returns data from the interpreter, and more.

{:ok, interpreter} = SnexTest.NumpyInterpreter.start_link()

{:ok, 6.0} =
  Snex.pyeval(interpreter, """
    import numpy as np
    matrix = np.fromfunction(lambda i, j: (-1) ** (i + j), (6, 6), dtype=int)
    return np.linalg.norm(matrix)
    """)

Environments

The Snex.Env struct, also called "environment", is an Elixir-side reference to a Python-side variable context in which your Python code will run. New environments can be allocated with Snex.make_env/3 (and other arities).

Environments are mutable, and will be modified by your Python code. In Python parlance, they are the global & local symbol table your Python code is executed with.

[!IMPORTANT]

Environments are garbage collected
When a %Snex.Env{} value is cleaned up by the BEAM VM, the Python process is signalled to deallocate the environment associated with that value.

Reusing a single environment, you can use variables defined in the previous Snex.pyeval/4 calls:

{:ok, inp} = Snex.Interpreter.start_link()
{:ok, env} = Snex.make_env(inp)

# `pyeval` returns the return value of the code block. If there's
# no `return` statement, the success return value will always be `nil`
{:ok, nil} = Snex.pyeval(env, "x = 10")

# additional variables can be provided for `pyeval` to put in the environment
# before running the code
{:ok, nil} = Snex.pyeval(env, "y = x * z", %{"z" => 2})

{:ok, {10, 20, 2}} = Snex.pyeval(env, "return (x, y, z)")

Using Snex.make_env/2 and Snex.make_env/3, you can also create a new environment:

copying variables from an old environment

Snex.make_env(interpreter, from: old_env)

# You can also omit the `interpreter` when using `:from`
Snex.make_env(from: old_env)

copying variables from multiple environments (later override previous)

Snex.make_env(interpreter, from: [
  oldest_env,
  {older_env, only: ["pool"]},
  {old_env, except: ["pool"]}
])

setting some initial variables (taking precedence over variables from :from)

Snex.make_env(interpreter, %{"hello" => 42.0}, from: {old_env, only: ["world"]})

[!WARNING]

The environments you copy from have to belong to the same interpreter!

Initialization script

Snex.Interpreter can be given an :init_script option. The init script runs on interpreter startup, and prepares a "base" environment state that will be cloned to every new environment made with Snex.make_env/3.

{:ok, inp} = SnexTest.NumpyInterpreter.start_link(
  init_script: """
  import numpy as np
  my_var = 42
  """)
{:ok, env} = Snex.make_env(inp)

# The brand new `env` contains `np` and `my_var`
{:ok, 42} = Snex.pyeval(env, "return int(np.array([my_var])[0])")

If your init script takes significant time, you can pass sync_start: false to start_link/1. This will return early from the interpreter startup, and run the Python interpreter - and the init script - asynchronously. The downside is that an issue with Python or the initialization code will cause the process to crash asynchronously instead of returning an error directly from start_link/1.

Passing `Snex.Env` between Erlang nodes

Snex.Env garbage collection can only track usage within the node it was created on. If you send a %Snex.Env{} value to another node b@localhost, and drop any references to the value on the original node a@localhost, the garbage collector may clean up the environment even though it's used on b.

In general, it's best to use Snex.Env instances created on the same node as their interpreter - this simplifies reasoning about cleanup, especially with node disconnections and shutdowns. Agents are a great way to share Snex.Env between nodes without giving up garbage collection:

# a@localhost
{:ok, interpreter} = Snex.Interpreter.start_link()

{:ok, env_agent} = Agent.start_link(fn ->
  {:ok, env} = Snex.make_env(interpreter, %{"hello" => "hello from a@localhost!"})
  env
end)

:erpc.call(:"b@localhost", fn ->
  remote_env = Agent.get(env_agent, & &1)
  {:ok, "hello from a@localhost!"} = Snex.pyeval(remote_env, "return hello")
end)

Alternatively, you can opt into manual management of Snex.Env lifetime by calling Snex.Env.disable_gc/1 on the original node, and later destroying the env by calling Snex.destroy_env/1 on any node.

Python interface documentation

See Python Interface Documentation on HexDocs

Serialization

Elixir data is serialized using a limited subset of Python's Pickle format (version 5), and deserialized on the Python side using pickle.loads(). Python data is serialized with a subset of Erlang's External Term

Snex

Install / Use

README