pyproc

Run Python like a local function from Go — no CGO, no microservices.

🎯 Purpose & Problem Solved

The Challenge

Go excels at building high-performance web services, but sometimes you need Python:

Machine Learning Models: Your models are trained in PyTorch/TensorFlow
Data Science Libraries: You need pandas, numpy, scikit-learn
Legacy Code: Existing Python code that's too costly to rewrite
Python-Only Libraries: Some libraries only exist in Python ecosystem

Traditional solutions all have major drawbacks:

| Solution | Problems | |----------|----------| | CGO + Python C API | Complex setup, crashes can take down entire Go service, GIL still limits performance | | REST/gRPC Microservice | Network latency, deployment complexity, service discovery, more infrastructure | | Shell exec | High startup cost (100ms+), no connection pooling, process management nightmare | | Embedded Python | GIL bottleneck, memory leaks, difficult debugging |

The Solution: pyproc

pyproc lets you call Python functions from Go as if they were local functions, with:

Zero network overhead - Uses Unix Domain Sockets for IPC
Process isolation - Python crashes don't affect your Go service
True parallelism - Multiple Python processes bypass the GIL
Simple deployment - Just your Go binary + Python scripts
Connection pooling - Reuse connections for high throughput

🎯 Target Audience & Use Cases

Perfect for teams who need to:

Integrate existing Python ML models (PyTorch, TensorFlow, scikit-learn) into Go services
Process data with Python libraries (pandas, numpy) from Go applications
Handle 1-5k RPS with JSON payloads under 100KB
Deploy on the same host/pod without network complexity
Migrate gradually from Python microservices to Go while preserving Python logic

Ideal deployment scenarios:

Kubernetes same-pod deployments with shared volume for UDS
Docker containers with shared socket volumes
Traditional server deployments on Linux/macOS

❌ Non-Goals

pyproc is NOT designed for:

Cross-host communication - Use gRPC/REST APIs for distributed systems
Windows UDS support - Windows named pipes are not supported
GPU management - Use dedicated ML serving frameworks (TensorRT, Triton)
Large-scale ML serving - Consider Ray Serve, MLflow, or KServe for enterprise ML
Real-time streaming - Use Apache Kafka or similar for high-throughput streams
Database operations - Use native Go database drivers directly

🔄 Alternatives & Comparison

pyproc is a dedicated IPC engine for integrating Python ML/DS code into Go services on the same host. It differs from general-purpose plugin systems and embedded runtimes in design philosophy.

| Solution | Pros | Cons | Best For | |----------|------|------|----------| | go-embed-python | ✅ Python runtime embedded / No Python installation required on host | ❌ Increased binary size / Python operations are DIY | Tools distributed as a single binary | | go-plugin (HashiCorp) | ✅ Multi-language plugin support / Proven in Terraform, Vault | ❌ Requires gRPC proto definitions / Not optimized for Python | Language-agnostic plugin architecture | | pyproc | ✅ Optimized for ML/DS workloads / Built-in worker pool, health checks, auto-restart / Ultra-low latency (~45µs p50) | ❌ Python-only / Same-host only | Integrating Python ML/DS into Go services |

When to Choose What

Choose go-embed-python if:

You want to distribute a single binary (no Python required on host)
Increased binary size is acceptable

Choose go-plugin if:

You need multi-language support (Rust, Ruby, etc.) beyond Python
You're integrating with HashiCorp ecosystem

Choose pyproc if:

You're calling Python ML models (PyTorch, TensorFlow) or DS libraries (pandas, NumPy) from Go
You need low latency (<100µs) on the same host
You want built-in worker pool management, health checks, and auto-restart

Non-Goals (Recap)

pyproc is NOT designed for:

❌ General-purpose plugin system → Use go-plugin
❌ Embedded Python runtime → Consider go-embed-python
❌ Cross-host communication → Use gRPC/REST microservices
❌ GPU cluster management → Use Ray Serve, Triton

🔐 Trust Model & Security Considerations

pyproc is designed for trusted code execution

pyproc is NOT a sandbox environment. It operates under the following assumptions:

✅ Target: Python code developed and managed by your organization (ML models, data processing logic)
✅ Process isolation: Python crashes do not affect the Go service
❌ No security isolation: Python workers can access the same filesystem and network as the parent Go process

Intended Use Cases

✅ Recommended:

Running your own trained PyTorch/TensorFlow models for inference
Data transformation pipelines using pandas/NumPy
Integrating scikit-learn models into Go recommendation engines

❌ Not Recommended:

Executing arbitrary user-submitted Python scripts
Dynamically loading third-party plugins
Running untrusted code

Security Details

For detailed threat model, security architecture, and best practices, see SECURITY.md.

Key Guarantees:

OS-level access control via Unix Domain Socket filesystem permissions
Fault tolerance through process isolation
Configurable resource limits (memory, CPU)

Limitations:

Inter-process communication on the same host only (cross-host is out of scope)
Does not provide sandbox environment (use gVisor, Firecracker if needed)

📋 Compatibility Matrix

| Component | Requirements | |-----------|-------------| | Operating System | Linux, macOS (Unix Domain Sockets required) | | Go Version | 1.22+ | | Python Version | 3.9+ (3.12 recommended) | | Deployment | Same host/pod only | | Container Runtime | Docker, containerd, any OCI-compatible | | Orchestration | Kubernetes (same-pod), Docker Compose, systemd | | Architecture | amd64, arm64 |

✨ Features

No CGO Required - Pure Go implementation using Unix Domain Sockets
Bypass Python GIL - Run multiple Python processes in parallel
Type-Safe API - Call Python with compile-time type checking using Go generics (zero overhead)
Minimal Overhead - 45μs p50 latency, 200,000+ req/s with 8 workers
Production Ready - Health checks, graceful shutdown, automatic restarts
Easy Deployment - Single binary + Python scripts, no service mesh needed
Full Observability - OpenTelemetry tracing, Prometheus metrics, structured logging (v0.7.1+)

🚀 Quick Start (5 minutes)

1. Install

Go side:

go get github.com/YuminosukeSato/pyproc@latest

Python side:

pip install pyproc-worker

2. Create a Python Worker

# worker.py
from pyproc_worker import expose, run_worker

@expose
def predict(req):
    """Your ML model or Python logic here"""
    return {"result": req["value"] * 2}

if __name__ == "__main__":
    run_worker()

3. Call from Go (Type-Safe API - Recommended)

package main

import (
    "context"
    "fmt"
    "log"
    "github.com/YuminosukeSato/pyproc/pkg/pyproc"
)

// Define request/response types (compile-time type safety)
type PredictRequest struct {
    Value float64 `json:"value"`
}

type PredictResponse struct {
    Result float64 `json:"result"`
}

func main() {
    // Create a pool of Python workers
    pool, err := pyproc.NewPool(pyproc.PoolOptions{
        Config: pyproc.PoolConfig{
            Workers:              4,  // Run 4 Python processes
            MaxInFlight:          10, // Global concurrent requests
            MaxInFlightPerWorker: 1,  // Per-worker in-flight cap
        },
        WorkerConfig: pyproc.WorkerConfig{
            SocketPath:   "/tmp/pyproc.sock",
            PythonExec:   "python3",
            WorkerScript: "worker.py",
        },
    }, nil)
    if err != nil {
        log.Fatal(err)
    }

    // Start all workers
    ctx := context.Background()
    if err := pool.Start(ctx); err != nil {
        log.Fatal(err)
    }
    defer pool.Shutdown(ctx)

    // Call Python function with type safety (automatically load-balanced)
    result, err := pyproc.CallTyped[PredictRequest, PredictResponse](
        ctx, pool, "predict", PredictRequest{Value: 42},
    )
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Result: %v\n", result.Result) // Result: 84 (type-safe!)
}

4. Run

go run main.go

That's it! You're now calling Python from Go without CGO or microservices.

Try the demo in this repo

If you cloned this repository, you can run a working end to end example without installing a Python package by using the bundled worker module.

make demo

This starts a Python worker from examples/basic/worker.py and calls it from Go. The example adjusts PYTHONPATH to import the local worker/python/pyproc_worker package.

📊 Observability (v0.7.1+)

pyproc includes built-in support for distribute

Pyproc

Install / Use

README