DiffuGen - Advanced Local Image Generator with MCP Integration

<p align="center"> <img src="diffugen.png" alt="DiffuGen Logo" width="400"/> </p> <p align="center"> <em>Your AI art studio embedded directly in code. Generate, iterate, and perfect visual concepts through this powerful MCP server for Cursor, Windsurf, and other compatible IDEs, utilizing cutting-edge Flux and Stable Diffusion models without disrupting your development process.</em> </p> <p align="center"> <a href="https://github.com/CLOUDWERX-DEV/diffugen/stargazers"><img src="https://img.shields.io/github/stars/CLOUDWERX-DEV/diffugen" alt="Stars Badge"/></a> <a href="https://github.com/CLOUDWERX-DEV/diffugen/network/members"><img src="https://img.shields.io/github/forks/CLOUDWERX-DEV/diffugen" alt="Forks Badge"/></a> <a href="https://github.com/CLOUDWERX-DEV/diffugen/issues"><img src="https://img.shields.io/github/issues/CLOUDWERX-DEV/diffugen" alt="Issues Badge"/></a> <a href="https://github.com/CLOUDWERX-DEV/diffugen/blob/master/LICENSE"><img src="https://img.shields.io/github/license/CLOUDWERX-DEV/diffugen" alt="License Badge"/></a> </p>

⭐ New: Now includes OpenAPI server support and OpenWebUI OpenAPI Tools (OWUI Version 0.60.0 Required) integration for seamless image generation and display in chat interfaces! The OpenAPI is seperate from the MCP server and allowss for initigrations into your own projects!

📃 Table of Contents

• Introduction
• Understanding MCP and DiffuGen
• Features
• System Requirements
• Installation
• IDE Setup Instructions
• Usage
  • OpenAPI Server Usage
  • Default Parameters by Model
  • Asking a LLM to Generate Images
  • Parameter Reference
  • Model-Specific Parameter Recommendations
  • Default Parameter Changes
  • Command Line Usage Notes
• Configuration
  • Configuration Approach
  • Environment Variable Overrides
  • Setting IDE-Specific Configurations
  • Key Configuration Elements
  • IDE-Specific Options
  • Customizing Default Parameters
  • Updating Configuration Files
• Advanced Usage
  • Using the OpenAPI Server
• License
• Acknowledgments
• Contact

🚀 Introduction

DiffuGen is a powerful MCP-based image generation system that brings cutting-edge AI models directly into your development workflow. It seamlessly integrates both Flux models (Flux Schnell, Flux Dev) and Stable Diffusion variants (SDXL, SD3, SD1.5) into a unified interface, allowing you to leverage the unique strengths of each model family without switching tools. With comprehensive parameter control and multi-GPU support, DiffuGen scales from rapid concept sketches on modest hardware to production-quality visuals on high-performance systems.

Built on top of the highly optimized stable-diffusion.cpp implementation, DiffuGen offers exceptional performance even on modest hardware while maintaining high-quality output.

🧠 Understanding MCP and DiffuGen

What is MCP?

MCP (Model Context Protocol) is a protocol that enables LLMs (Large Language Models) to access custom tools and services. In simple terms, an MCP client (like Cursor, Windsurf, Roo Code, or Cline) can make requests to MCP servers to access tools that they provide.

DiffuGen as an MCP Server

DiffuGen functions as an MCP server that provides text-to-image generation capabilities. It implements the MCP protocol to allow compatible IDEs to send generation requests and receive generated images.

The server exposes two main tools:

1. generate_stable_diffusion_image: Generate with Stable Diffusion models
2. generate_flux_image: Generate with Flux models

Technical Architecture

DiffuGen consists of several key components:

• setup-diffugen.sh: The complete install utility and model downloader and manager
• diffugen.py: The core Python script that implements the MCP server functionality and defines the generation tools
• diffugen.sh: A shell script launcher that sets up the environment and launches the Python server
• diffugen.json: Template configuration file for MCP integration with various IDEs (to be copied into IDE's MCP configuration)
• stable-diffusion.cpp: The optimized C++ implementation of Stable Diffusion used for actual image generation

The system works by:

1. Receiving prompt and parameter data from an MCP client
2. Processing the request through the Python server
3. Calling the stable-diffusion.cpp binary with appropriate parameters
4. Saving the generated image to a configured output directory
5. Returning the path and metadata of the generated image to the client

About stable-diffusion.cpp

stable-diffusion.cpp is a highly optimized C++ implementation of the Stable Diffusion algorithm. Compared to the Python reference implementation, it offers:

• Significantly faster inference speed (up to 3-4x faster)
• Lower memory usage (works on GPUs with as little as 4GB VRAM)
• Optimized CUDA kernels for NVIDIA GPUs
• Support for various sampling methods and model formats
• Support for model quantization for better performance
• No Python dependencies for the core generation process

This allows DiffuGen to provide high-quality image generation with exceptional performance, even on modest hardware setups.

✨ Features

• Multiple Model Support: Generate images using various models including Flux Schnell, Flux Dev, SDXL, SD3, and SD1.5
• MCP Integration: Seamlessly integrates with IDEs that support MCP (Cursor, Windsurf, Roo Code, Cline, etc.)
• OpenAPI Server: Additional REST API interface for direct HTTP access to image generation capabilities
• Cross-Platform: Works on Linux, macOS, and Windows (via native or WSL)
• Parameter Control: Fine-tune your generations with controls for:
  • Image dimensions (width/height)
  • Sampling steps
  • CFG scale
  • Seed values
  • Negative prompts (for SD models only, Flux does not support negative prompts.)
  • Sampling methods
• CUDA Acceleration: Utilizes GPU acceleration for faster image generation
• Natural Language Interface: Generate images using simple natural language commands
• Smart Error Recovery: Robust error handling with operation-aware recovery procedures
• User-Friendly Setup: Interactive setup script with improved interrupt handling
• Resource Tracking: Session-aware resource management for efficient cleanup
• Customizable Interface: Support for custom ANSI art logos and visual enhancements

💻 System Requirements

Minimum Requirements:

• CPU: 4-core processor (Intel i5/AMD Ryzen 5 or equivalent)
• RAM: 8GB system memory
• Storage: 5GB free disk space (SSD preferred for faster model loading)
• Python: 3.8 or newer
• GPU: Integrated graphics or entry-level dedicated GPU (optional)
• Network: Broadband connection for model downloads (5+ Mbps)

Recommended Requirements:

• CPU: 8+ core processor (Intel i7/i9 or AMD Ryzen 7/9)
• RAM: 16GB+ system memory
• GPU: NVIDIA GPU with 6GB+ VRAM (RTX 2060 or better for optimal performance)
• Storage: 20GB+ free SSD space
• Python: 3.10 or newer (3.11 offers best performance)
• Network: High-speed connection (20+ Mbps) for efficient model downloads

📥 Installation

Automatic Installation (Recommended)

The easiest way to install DiffuGen is using the provided setup script:

git clone https://github.com/CLOUDWERX-DEV/diffugen.git
cd DiffuGen
chmod +x diffugen.sh
chmod +x setup_diffugen.sh
./setup_diffugen.sh

Follow the interactive prompts to complete the installation.

The setup script will:

• Install necessary dependencies
• Clone and build stable-diffusion.cpp
• Set up a Python virtual environment
• Download selected models (Note: Some models require Clip\VAE Models as well)
• Configure file paths for your system

Manual Installation

If you prefer to install manually, follow these steps:

1. Clone the repositories:

git clone https://github.com/CLOUDWERX-DEV/diffugen.git
cd DiffuGen
git clone --recursive https://github.com/leejet/stable-diffusion.cpp

2. Build stable-diffusion.cpp:

cd stable-diffusion.cpp
mkdir -p build && cd build

With CUDA:

cmake .. -DCMAKE_BUILD_TYPE=Release -DSD_CUDA=ON
make -j$(nproc)
cd ../..

Without CUDA:

cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
cd ../..

3. Create and activate a Python virtual environment:

python3 -m venv diffugen_env
source diffugen_env/bin/activate  # On Windows: diffugen_env\Scripts\activate
pip install -r requirements.txt

4. Download required models (structure shown below):

stable-diffusion.cpp/models/
├── ae.sft                           # VAE model
├── clip_l.safetensors               # CLIP model
├── flux/
│   ├── flux1-schnell-q8_0.gguf     # Flux Schnell model (default)
│   └── flux1-dev-q8_0.gguf          # Flux Dev model
├── sd3-medium.safetensors           # SD3 model
├── sdxl-1.0-base.safetensors        # SDXL model
├── sdxl_vae-fp16-fix.safetensors    # SDXL VAE
├── t5xxl_fp16.safetensors           # T5 model
└── v1-5-pruned-emaonly.safetensors  # SD1.5 model

You can download the models from the following sources:

# Create model directo