SegviGen: Repurposing 3D Generative Model for Part Segmentation

teaser

🏠 Project Page | Paper | Online Demo

SegviGen is a framework for 3D part segmentation that leverages the rich 3D structural and textural knowledge encoded in large-scale 3D generative models. It learns to predict part-indicative colors while reconstructing geometry, and unifies three settings in one architecture: interactive part segmentation, full segmentation, and 2D segmentation map–guided full segmentation with arbitrary granularity.

🌟 Features

Repurposed 3D Generative Priors for Data Efficiency: By reusing the rich structural and textural knowledge encoded in large-scale native 3D generative models, SegviGen learns 3D part segmentation with minimal task-specific supervision, requiring only 0.32% training data.
Unified and Flexible Segmentation Settings: Supports interactive part segmentation, full segmentation, and 2D segmentation map–guided full segmentation with arbitrary part granularity under a single architecture.
State-of-the-Art Accuracy: Consistently surpasses P3-SAM, delivering a 40% gain in IoU@1 for single-click interaction on PartObjaverse-Tiny and PartNeXT, and a 15% improvement in overall IoU for unguided full segmentation averaged across datasets.

🔨 Installation

Prerequisites

System: Linux
GPU: A NVIDIA GPU with at least 24GB of memory is necessary
Python: 3.10

Installation Steps

Create the environment of TRELLIS.2

git clone -b main https://github.com/microsoft/TRELLIS.2.git --recursive
cd TRELLIS.2
./setup.sh --new-env --basic --flash-attn --nvdiffrast --nvdiffrec --cumesh --o-voxel --flexgemm

Install the rest of requirements

pip install mathutils
pip install transformers==4.57.6 # https://github.com/microsoft/TRELLIS.2/issues/101
pip install bpy==4.0.0 --extra-index-url https://download.blender.org/pypi/
sudo apt-get install -y libsm6 libxrender1 libxext6
pip install --upgrade Pillow

Pretrained Weights

The checkpoints of Interactive part-segmentation, Full segmentation and Full segmentation with 2D guidance are available on Hugging Face.

📒 Usage

Interactive part-segmentation

python inference_interactive.py \
    --ckpt_path path/to/interactive_seg.ckpt \
    --glb ./data_toolkit/assets/example.glb \
    --input_vxz ./data_toolkit/assets/input.vxz \
    --transforms ./data_toolkit/transforms.json \
    --img ./data_toolkit/assets/img.png \
    --export_glb ./data_toolkit/assets/output.glb \
    --input_vxz_points 388 448 392

Full segmentation

python inference_full.py \
    --ckpt_path path/to/full_seg.ckpt \
    --glb ./data_toolkit/assets/example.glb \
    --input_vxz ./data_toolkit/assets/input.vxz \
    --transforms ./data_toolkit/transforms.json \
    --img ./data_toolkit/assets/img.png \
    --export_glb ./data_toolkit/assets/output.glb

Full segmentation with 2D guidance

python inference_full.py \
    --ckpt_path path/to/full_seg_w_2d_map.ckpt \
    --glb ./data_toolkit/assets/example.glb \
    --input_vxz ./data_toolkit/assets/input.vxz \
    --img ./data_toolkit/assets/full_seg_w_2d_map/2d_map.png \
    --export_glb ./data_toolkit/assets/output.glb \
    --two_d_map

⚖️ License

This project is licensed under the MIT License.
However, please note that the code in trellis2 originates from the TRELLIS.2 project and remains subject to its original license terms.
Users must comply with the licensing requirements of TRELLIS.2 when using or redistributing that portion of the code.

Citation

@article{li2026segvigen,
      title = {SegviGen: Repurposing 3D Generative Model for Part Segmentation}, 
      author = {Lin Li and Haoran Feng and Zehuan Huang and Haohua Chen and Wenbo Nie and Shaohua Hou and Keqing Fan and Pan Hu and Sheng Wang and Buyu Li and Lu Sheng},
      journal = {arXiv preprint arXiv:2603.16869},
      year = {2026}
}

SegviGen

Install / Use

README