Important: we released CraftsMan3D-DoraVAE trained using rectified flow.

中文版

<p align="center"> <img src="asset/logo.png" height=220> </p>

<div align="center">CraftsMan3D: High-fidelity Mesh Generation <br> with 3D Native Generation and Interactive Geometry Refiner<div>

<p align="center"> Weiyu Li^*1,2, Jiarui Liu^*1,2, Hongyu Yan^*1, Rui Chen¹, Yixun Liang^1,2, Xuelin Chen³, Ping Tan^1,2, Xiaoxiao Long^1,2</p>

<p align="center"> ¹HKUST, ²LightIllusions, ³Adobe Research</p>

<div align="center"> <a href="https://craftsman3d.github.io/"><img src="https://img.shields.io/static/v1?label=Project%20Page&message=Github&color=blue&logo=github-pages"></a> &ensp; <a href="https://huggingface.co/spaces/wyysf/CraftsMan"><img src="https://www.gradio.app/_app/immutable/assets/gradio.CHB5adID.svg" height="25"/></a> &ensp; <a href="https://triverse.lightillusions.com/"><img src="asset/icon.png" height="25"/>Local Website</a> &ensp; <a href="https://arxiv.org/pdf/2405.14979"><img src="https://img.shields.io/static/v1?label=Paper&message=Arxiv&color=red&logo=arxiv"></a> &ensp; </div>

Usage

from craftsman import CraftsManPipeline
import torch

# load from local ckpt
# mkdir ckpts && cd ckpts
# mkdir craftsman-DoraVAE && cd craftsman-DoraVAE
# wget https://pub-c7137d332b4145b6b321a6c01fcf8911.r2.dev/craftsman-DoraVAE/config.yaml
# wget https://pub-c7137d332b4145b6b321a6c01fcf8911.r2.dev/craftsman-DoraVAE/model.ckpt
# pipeline = CraftsManPipeline.from_pretrained("./ckpts/craftsman-DoraVAE", device="cuda:0", torch_dtype=torch.bfloat16) 

# load from huggingface model hub, I uploading...
pipeline = CraftsManPipeline.from_pretrained("craftsman3d/craftsman-DoraVAE", device="cuda:0", torch_dtype=torch.bfloat16)

# inference
mesh = pipeline("https://pub-f9073a756ec645d692ce3d171c2e1232.r2.dev/data/werewolf.png").meshes[0]
mesh.export("werewolf.obj")

The results should be like this:

<p align="center"> <img src="asset/demo_result.png" height=220> </p>

TL; DR: <font color="red">CraftsMan3D (aka 匠心)</font> is a two-stage text/image to 3D mesh generation model. By mimicking the modeling workflow of artist/craftsman, we propose to generate a coarse mesh (5s) with smooth geometry using 3D diffusion model and then refine it (20s) using enhanced multi-view normal maps generated by 2D normal diffusion, which is also can be in a interactive manner like Zbrush.

✨ Overview

This repo contains source code (training / inference) of 3D diffusion model, pretrained weights and gradio demo code of our 3D mesh generation project, you can find more visualizations on our project page and try our demo. If you have high-quality 3D data or some other ideas, we very much welcome any form of cooperation.

<details><summary>Full abstract here</summary> We present a novel generative 3D modeling system, coined CraftsMan, which can generate high-fidelity 3D geometries with highly varied shapes, regular mesh topologies, and detailed surfaces, and, notably, allows for refining the geometry in an interactive manner. Despite the significant advancements in 3D generation, existing methods still struggle with lengthy optimization processes, irregular mesh topologies, noisy surfaces, and difficulties in accommodating user edits, consequently impeding their widespread adoption and implentation in 3D modeling softwares. Our work is inspired by the craftsman, who usually roughs out the holistic figure of the work first and elaborate the surface details subsequently. Specifically, we employ a 3D native diffusion model, which operates on latent space learned from latent set-based 3D representations, to generate coarse geometries with regular mesh topology in seconds. In particular, this process takes as input a text prompt or a reference image, and leverages a powerful multi-view (MV) diffusion model to generates multiple views of the coarse geometry, which are fed into our MV-conditioned 3D diffusion model for generating the 3D geometry, significantly improving robustness and generalizability. Following that, a normal-based geometry refiner is used to significantly enhance the surface details. This refinement can be performed automatically, or interactively with user-supplied edits. Extensive experiments demonstrate that our method achieves high efficiency in producing superior quality 3D assets compared to existing methods. </details> <p align="center"> <img src="asset/teaser.jpg" > </p>

💪 ToDo List

• [x] Inference code
• [x] Training code
• [x] Gradio & Hugging Face demo
• [x] Model zoo
• [x] Environment setup
• [x] Data sample
• [x] CraftsMan3D-DoraVAE (not the official version)
• [x] support rectified flow training
• [x] support flashVDM, thanks for their open-source
• [ ] release multiview(4 views) conditioned model (including weights and training data sample)
• [x] add data for vae training, we release the data preprocessing script in watertight_and_sampling.py
• [ ] support training and finetuning TripoSG model (almost done)
• [ ] support training Hunyuan3D-2 model(it is not release the weights for vae encoder)

Contents

• Pretrained Models
• Gradio & Huggingface Demo
• Inference
• Training
• Data Prepration
• Video
• Acknowledgement
• Citation

Environment Setup

<details> <summary>Hardware</summary> We train our model on 32x A800 GPUs with a batch size of 32 per GPU for 7 days.

The mesh refinement part is performed on a GTX 3080 GPU.

</details> <details> <summary>Setup environment</summary>

:smiley: We also provide a Dockerfile for easy installation, see Setup using Docker.

• Python 3.10.0
• PyTorch 2.5.1 (for RSMNorm)
• Cuda Toolkit 12.4.0
• Ubuntu 22.04

Clone this repository.

git clone https://github.com/wyysf-98/CraftsMan.git

Install the required packages.

conda create -n CraftsMan python=3.10 -y
conda activate CraftsMan
# conda install -c "nvidia/label/cuda-12.1.1" cudatoolkit
# conda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install torch==2.5.1 torchvision==0.20.1
pip install -r docker/requirements.txt
pip install torch-cluster -f https://data.pyg.org/whl/torch-2.5.1+cu124.html

</details>

✨ History

This repo will port some recent techniques for 3D diffusion model and the history version like the arxiv version can be found in different branch.

<details> <p align="center"> <img src="asset/history.png" > </p> </details>

🎥 Video

3D Native DiT Model (Latent Set DiT Model)

We provide the training and the inference code here for future research. The latent set VAE model is heavily build on the same structure of Michelangelo. The latent set diffusion model is based on a DiT/Pixart-alpha and with 500M parameters.

Pretrained models

Currently, We provide the models with single view image as condition with DiT. We will consider open source the further models according to the real situation. If you run the inference.py without specifying the model path, it will automatically download the model from the huggingface model hub.

Or you can download the model manually:

## you can just manually get the model using wget:
mkdir ckpts
cd ckpts
mkdir craftsman-v1-5
cd craftsman-v1-5
wget https://huggingface.co/craftsman3d/craftsman/resolve/main/config.yaml
wget https://huggingface.co/craftsman3d/craftsman/resolve/main/model.ckpt
### for DoraVAE version(https://aruichen.github.io/Dora/)
cd ..
mkdir craftsman-doravae
cd craftsman-doravae
wget https://huggingface.co/craftsman3d/craftsman-doravae/resolve/main/config.yaml
wget https://huggingface.co/craftsman3d/craftsman-doravae/resolve/main/model.ckpt

## OR you can git clone the repo:
git lfs install
git clone https://huggingface.co/craftsman3d/craftsman
### for DoraVAE version(https://aruichen.github.io/Dora/)
git clone https://huggingface.co/craftsman3d/craftsman-doravae

If you download the models using wget, you should manually put them under the ckpts/craftsman directory.

Gradio demo

We provide gradio demos for easy usage.

python gradio_app.py --model_path ./ckpts/craftsman

Inference

To generate 3D meshes from images folders via command line, simply run:

python inference.py --input eval_data --device 0 --model ./ckpts/craftsman

For more configs, please refer to the inference.py.

Train from scratch

We provide our training code to facilitate future research. And we provide a data sample in data. 100k data sample for VAE training can be downloaded from (to be uploaded)

100k data sample for diffusion training can be downloaded from https://pub-c7137d332b4145b6b321a6c01fcf8911.r2.dev/Objaverse_100k.zip

selected 190k UUID for training can be downloaded from https://pub-c7137d332b4145b6b321a6c01fcf8911.r2.dev/objaverse_190k.json

selected 320k UUID for training can be downloaded from https://pub-c7137d332b4145b6b321a6c01fcf8911.r2.dev/objaverse_320k.json

For more training details and configs, please refer to the configs folder.

### training the shape-autoencoder
python train.py --config ./configs/shape-autoencoder/michelangelo-l768-e64-ne8-