THIS REPOSITORY IS NOT COMPATIBLE WITH THE LATEST VERSION OF COMFYUI, DUE TO NOT BEING UPDATED

👉🏼👉🏼👉🏼Please take note of the following information: This repository is compatible and optimized for use with MAC CPU+MPS and Windows with CPU+CUDA. The installation process is not beginner-friendly for enabling CUDA Toolkit==11.8 and cuDNN (CUDA Deep Neural Network). If you are unsure about installing CUDA, you can still use the CPU, and no CUDA installation will be necessary. However, if you are able to install CUDA correctly, the processing speed will increase significantly.

DeepFuze

Watch the 4k Quality Video on YOUTUBE

Overview

DeepFuze is a state-of-the-art deep learning tool that seamlessly integrates with ComfyUI to revolutionize facial transformations, lipsyncing, video generation, voice cloning, face swapping, and lipsync translation. Leveraging advanced algorithms, DeepFuze enables users to combine audio and video with unparalleled realism, ensuring perfectly synchronized facial movements. This innovative solution is ideal for content creators, animators, developers, and anyone seeking to elevate their video editing projects with sophisticated AI-driven features.

Watch the 4k Quality Video on Youtube

Watch the 4k Quality Video on Youtube

---

Installation & Models Download

---

Windows Installation 🖥️

Portable ComfyUI for Windows

Step 1

You must install Visual Studio, it works with the community version OR VS C++ Build Tools and select "Desktop Development with C++" under "Workloads -> Desktop & Mobile"

Step 2 (installing from ComfyUI-Manager)

From ComfyUI-Manager search for DeepFuze, and install the node. Restart your ComfyUI, and look at your terminal window to ensure there is no error, or Install from the ComfyUI manager, select "Install Via GIT URL", and copy past:

https://github.com/SamKhoze/ComfyUI-DeepFuze.git

Step 3

Restart your ComfyUI

IMPORTANT NOTE: CUDA INSTALLATION IS NOT BEGINNER-FRIENDLY, IF YOU DON'T KNOW WHAT YOU ARE DOING DO NOT TRY, USE THE CPU VERSION.

CUDA Installation

YOUTUBE LINK step by step instructions

---

Install Nvidia CUDA Toolkit==11.8 and cuDNN (CUDA Deep Neural Network) for Deep Learning, you must download the cuDNN version 8.9.2.26 from the NVIDIA DEVELOPER cuDNN Archive, if you don't have developer account you can directly download it from GoogleDrive. Ensure install Cuda1 1.8. I found this YOUTUBE video useful for installation. If you have a different version of CUDA here is a YOUTUBE link that guides you on how to uninstall your CUDA. Make sure to create paths in your Environment variable as described on YOUTUBE VIDEO Restart your computer after creating paths. Confirm your Cuda Installation, paste this code on your terminal window nvcc --version you should get a response like this:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0`

---

MAC Installation 👨🏻‍💻

Do not install via ComfyUI-Manager it will not work, you must install it manually and follow the instructions below:

Video Tutorial How To Install on MAC YOUTUBE LINK

For MAC users please set the environment variable before running it

Activate your Virtual Environment, Conda or Venv

Install Pytorch

Here how to install and test your PyTorch

This method has been tested on a M1 and M3 Mac, You must run the below code on your terminal window for Mac Metal Performance Shaders (MPS) Apple's specialized solution for high-performance GPU programming on their devices. Integrating closely with the Metal framework, MPS provides a suite of highly optimized shaders for graphics and computing tasks, which is particularly beneficial in machine learning applications.

⚠️⚠️⚠️ Important Steps (if you miss these steps it will not work)

Copy and paste the command below to your terminal window.

export PYTORCH_ENABLE_MPS_FALLBACK=1

Mac users must INSTALL ONNX RUNTIME CPU instead of onnxruntime-gpu

pip install onnxruntime

macOS needs to install the original dlib.

pip install dlib 

Install Text to Speech for Voice Cloning Node

pip install TTS 

Navigate into custom_nodes folder

cd custom_nodes
git clone https://github.com/SamKhoze/CompfyUI-DeepFuze.git

Navigate into the CompfyUI-DeepFuze folder and install requirements.txt file

cd CompfyUI-DeepFuze
pip install -r requirements.txt

Prerequisites for Voice Cloning and Lipsyncing

Below are the two ComfyUI repositories required to load video and audio. Install them into your custom_nodes folder:

Clone the repositories:

cd custom_nodes
git clone https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite.git

Errors 😾

CV Error: If you encounter the error "ComfyUI_windows_portable\ComfyUI\output\deepfuze\faceswap_file.mp4 could not be loaded with cv," it means that onnxruntime is not installed. To fix this, make sure to install onnxruntime for CPU and onnxruntime-gpu for Windows. Mac users should upgrade OpenCV using the command pip install --upgrade opencv-python-headless in their virtual environment. For Windows users, go to ComfyUI Manager, click on "pip install," paste --upgrade opencv-python-headless, click OK, and restart your ComfyUI.

---

Missing zlibwapi.dll error: Search for NVIDIA zlibwapi.dll file, download it and copy it in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\zlibwapi.dll

---

If you get an error installing TTS, it is most likely because you have different versions of Python, make sure to install the correct version

---

If you get an error: ImportError: cannot import name 'get_full_repo_name' from 'huggingface_hub' Run the below codes on your terminal it will solve the issue

conda install chardet 

pip install --upgrade transformers==4.39.2 

if you get any error for any packages, open the requirements.txt file with any text editor remove the version from the front of the package name, and reinstall requirments.txt again

Models🌟🌟🌟

You can download models directly from GoogleDrive and place models into the PATH ./ComfyUI/models/deepfuze/ Ensure to manually download each model one by one and place them, due to the size of the models some of the models won't download if you download the folder preparing the environmental variables navigate into your custom_nodes folder and git clone or manually download the code and extract it into the custom_nodes folder.

OpenAI API setup for voice cloning (Optional)

---

You need an OpenAI API Key if you wish to use the "DeepFuze Openai LLM" node for generating dialogues for voice cloning

To use the "Openai LLM" node for voice cloning dialogues, you need an OpenAI API Key. You can get this key and set it up by following the instructions in the OpenAI Developer quickstart guide. Please note that the "Openai LLM" node does not save your API key. Every time you close the node, you will need to manually copy and paste your API key. You can also add the API key as an Environment Variable using the following commands: For Windows: setx OPENAI_API_KEY "your-api-key-here", and for Mac: export OPENAI_API_KEY='your-api-key-here'. The next time you need to copy and paste your API key into the LLM Node, you can type the following command in your terminal: echo $OPENAI_API_KEY, and it will print your API Key, allowing you to copy and paste it into your Openai LLM node.

Nodes

DeepFuze Nodes Overview YOUTUBE LINK

DeepFuze Lipsync Node 🫦

This node generates lipsyncing video from, video, image, and audio files. For higher quality export the IMAGE output as an image batch instead of a video combined, you can get up to 4k quality image size. IMPORTANT: You must load audio with the "VHS load audio" node from the [VideoHelperSuit](https://github.