TwitCanva Video Workflow

<div align="center"> <img src="public/TwitCanva-logo.png" alt="TwitCanva Logo" width="120" /> <h1>TwitCanva</h1> </div>

A modern, AI-powered canvas application for generating and manipulating images and videos using OpenAI GPT Image, Google Gemini, Kling AI, Hailuo AI (MiniMax), and Fal.ai. Built with React, TypeScript, and Vite.

Star History

✨ Features

• 🎨 Visual Canvas Interface - Drag-and-drop node-based workflow
• 🤖 Multi-Model AI Generation - GPT Image 1.5, Gemini Pro, Kling V1-V2.5 for images
• 🎬 Multi-Model Video Generation - Veo 3.1, Kling V1-V2.6, Hailuo 2.3/O2 for videos
• 🎥 Camera Angle Control - Transform any image by adjusting camera rotation and tilt angles (Qwen-Image-Edit)
• 📋 Storyboard - Create video storyboards with consistent characters and layouts
• 💃 Motion Control - Transfer motion from reference videos to character images (Kling V2.6 via Fal.ai)
• 📥 TikTok Import - Download TikTok videos without watermark for use as motion references
• 📤 Post to X - Share generated images/videos directly to Twitter/X with one click
• 📤 Post to TikTok - Share generated videos directly to TikTok with one click
• 🖼️ Image-to-Image - Use reference images for generation
• 📽️ Frame-to-Frame Video - Animate between start and end frames
• 🔗 Smart Node Connections - Type-aware validation (IMAGE→VIDEO, TEXT→IMAGE, etc.)
• 💬 AI Chat Assistant - Built-in chat with LangGraph agent
• 📚 Asset Library - Save and reuse generated assets
• 💾 Workflow Management - Save, load, and share workflows
• ⚡ Real-time Updates - Hot module replacement for instant feedback
• 🎯 Aspect Ratio Control - Multiple preset ratios for images
• 📹 Resolution Options - 720p and 1080p for videos
• 🔒 Secure API - Backend proxy keeps API keys safe
• 🔄 Auto-Model Selection - Filters models based on input compatibility
• 🖥️ Local Open-Source Models - Run Stable Diffusion, ControlNet, Qwen on your GPU
• ⚖️ Commercial Friendly - Dual-licensed or permissive terms for commercial growth

🎥 Showcase

App Overview

https://github.com/user-attachments/assets/7a64d4df-7ade-4bfa-b2cd-d615d267dd40

Motion Control Example (Kling V2.6)

Transfer motion from a reference video to a character image - make anyone dance!

https://github.com/user-attachments/assets/1ee6cbf3-00a5-496e-852c-3304c6ebc6c9

Output Example

Download all the generated videos and use video editting tool like CapCut to create a final video. Check result below.

https://github.com/user-attachments/assets/43cf8bb8-bf85-45f9-96da-657033126d94

https://github.com/user-attachments/assets/e6f89da5-d3a6-4889-a38b-672cf37bbd79

Camera Angle Control

Transform any image by adjusting camera rotation and tilt angles.

https://github.com/user-attachments/assets/f0d678df-31ac-4431-bd7c-eea3950bfb1d

Storyboard

Create video storyboards with consistent characters and layouts.

https://github.com/user-attachments/assets/3c36de54-d37e-4875-8403-5b6e4a6216e0

🚀 Getting Started

Prerequisites

• Node.js 18+
• npm or yarn
• Google Gemini API key (get one at Google AI Studio)
• Kling AI API keys (get them at Kling AI Developer)
  • Requires purchasing API packages at Kling AI Pricing
• Hailuo AI API key (get one at MiniMax Platform)
• OpenAI API key (get one at OpenAI Platform)
  • Requires organization verification to use GPT Image models
• Fal.ai API key (get one at Fal.ai Dashboard) - Required for Kling V2.6 Motion Control

Installation

1. Clone the repository

   git clone https://github.com/SankaiAI/TwitCanva.git
   cd TwitCanva
   

2. Install dependencies

   npm install
   

3. Set up environment variables

   Create a .env file in the root directory:

   # Get from https://aistudio.google.com/app/apikey
   GEMINI_API_KEY=your_gemini_api_key_here
   
   # Get from https://app.klingai.com/global/dev/api-key
   KLING_ACCESS_KEY=your_kling_access_key_here
   KLING_SECRET_KEY=your_kling_secret_key_here
   
   # Get from https://platform.minimax.io/user-center/basic-information/interface-key
   HAILUO_API_KEY=your_hailuo_api_key_here
   
   # Get from https://platform.openai.com/api-keys
   OPENAI_API_KEY=your_openai_api_key_here
   
   # Get from https://fal.ai/dashboard/keys (for Kling V2.6 Motion Control)
   FAL_API_KEY=your_fal_api_key_here
   
   # Optional: X (Twitter) Post Feature - Get from https://developer.twitter.com/en/portal
   # See docs/post-to-x.md for detailed setup instructions
   TWITTER_CLIENT_ID=your_twitter_client_id
   TWITTER_CLIENT_SECRET=your_twitter_client_secret
   TWITTER_API_KEY=your_twitter_api_key
   TWITTER_API_SECRET=your_twitter_api_secret
   TWITTER_ACCESS_TOKEN=your_twitter_access_token
   TWITTER_ACCESS_TOKEN_SECRET=your_twitter_access_token_secret
   TWITTER_CALLBACK_URL=http://127.0.0.1:3001/api/twitter/callback
   
   # Optional: TikTok Post Feature - Get from https://developers.tiktok.com/
   # See docs/tiktok-integration.md for detailed setup instructions
   TIKTOK_CLIENT_KEY=your_tiktok_client_key
   TIKTOK_CLIENT_SECRET=your_tiktok_client_secret
   TIKTOK_CALLBACK_URL=https://your-ngrok-url.ngrok-free.app/api/tiktok-post/callback
   

   ⚠️ Security: API keys are stored server-side only and never exposed to the client.

4. Start the development server

   npm run dev
   

   This starts both:

   • Frontend dev server: http://localhost:5173
   • Backend API server: http://localhost:3001

Alternative: Docker Installation

If you prefer using Docker to run the application in a containerized environment (recommended for deployment):

1. Clone the repository and set up .env (same as steps 1-3 above)

2. Run with Docker Compose

   docker compose up -d --build
   

   • The app will run on http://localhost:3001
   • Data persists in the local library/ folder
   • To stop: docker compose down

Optional: Local Open-Source Models Setup

TwitCanva supports running open-source AI models (like Stable Diffusion, Qwen Camera Control, ControlNet) locally on your GPU. This is optional - the cloud-based AI models work without this setup.

Requirements:

• NVIDIA GPU with 8GB+ VRAM (12GB+ recommended for larger models)
• Python 3.10+
• CUDA-compatible drivers

Setup:

# Option 1: Use npm script (recommended)
npm run setup:local-models

# Option 2: Run setup script directly
# Windows:
setup-local-models.bat

# Linux/macOS:
chmod +x setup-local-models.sh
./setup-local-models.sh

This will:

1. Create a Python virtual environment (venv/)
2. Install PyTorch with CUDA support (~2.8GB download)
3. Create the models/ directory structure
4. Test GPU detection

Adding Models:

Download models from HuggingFace, Civitai, or similar sites (.safetensors, .ckpt, or .pt files) and place them in the appropriate folder:

| Folder | Model Types | Examples | |--------|-------------|----------| | models/checkpoints/ | Main image generation models | Stable Diffusion 1.5, SDXL, DreamShaper, Juggernaut XL, Flux | | models/loras/ | LoRA adapters for styles/characters | Art styles, character LoRAs, detail enhancers | | models/controlnet/ | Guided generation models | OpenPose, Canny, Depth, Tile | | models/video/ | Video generation models | AnimateDiff, Stable Video Diffusion (SVD) |

Using Local Models:

1. Right-click on canvas → Add Nodes
2. Select "Local Image Model" or "Local Video Model"
3. Choose your downloaded model from the dropdown
4. Enter a prompt and generate!

📖 For detailed documentation, see docs/local-model-support.md

Optional: Camera Angle Control Setup

Transform your generated images with AI-powered camera angle manipulation using the Qwen Image Edit model.

Option 1: Cloud Deployment (Recommended)

For users without high-end GPUs, we provide a Modal-based cloud deployment.

1. Install Modal:

   pip install modal
   modal setup
   

2. Deploy the App:

   modal deploy modal/camera_angle.py
   

3. Configure Environment: Copy the generated generate endpoint URL and add it to your .env file:

   VITE_MODAL_CAMERA_ENDPOINT=https://your-workspace--camera-angle-control-cameraangle-generate.modal.run
   

4. Managing Costs:

   • Auto scale-down: Containers automatically shut down after 5 minutes of inactivity (no charges when idle).
   • Stop the app completely: Run modal app stop camera-angle-control to disable the endpoint entirely.
   • Restart after stopping: Run modal deploy modal/camera_angle.py again to re-enable.

   Tip: Stop the app when not actively using the feature to avoid any accidental charges.

Option 2: Local Deployment (Advanced)

This feature requires a 24GB VRAM GPU (RTX 3090/4090).

Download Models (~35GB):

# Activate venv
.\venv\Scripts\activate    # Windows
source venv/bin/activate   # Linux/macOS

# Download fast transformer (~20GB)
huggingfa