BrickGPT

Generating Physically Stable and Buildable Brick Structures from Text
ICCV 2025 (Best Paper, Marr Prize)
Ava Pun*, Kangle Deng*, Ruixuan Liu*, Deva Ramanan, Changliu Liu, Jun-Yan Zhu
Carnegie Mellon University

---

This is the official repository for BrickGPT, the first approach for generating physically stable toy brick models from text prompts.

<video src='https://github.com/user-attachments/assets/6760f55c-a413-4baf-abf1-2ad61bb1a427' width=480/></video>

Results

Examples of generated brick structures assembled by humans:

<img src="assets/human_exec.jpg" alt="Examples of generated brick models assembled by humans"/>

Examples of textured and colored brick models:

<img src="assets/texture.jpg" alt="Examples of brick models generated with textures and colors"/>

Installation

Prerequisites

• Llama-3.2-1B-Instruct: BrickGPT is fine-tuned from meta-llama/Llama-3.2-1B-Instruct, a gated model. Request access to the model, then generate a Hugging Face user access token and set it as an environment variable: export HF_TOKEN=<your_token>. The model will be automatically downloaded upon running the code.
• Gurobi (optional but recommended): Running stability analysis requires a Gurobi license to use Gurobi. Academics may request a free license from the Gurobi website; after obtaining the license, place it in your home directory or another recommended location.
  • If you do not have access to Gurobi, you can run the code with the option --use_gurobi False to use a simpler but less accurate connectivity-based method instead of physics-based stability analysis.

Installing as a standalone project

This repo uses the Python project manager uv. To install this repo as a standalone project, first install all prerequisites. Then,

1. Clone the repo: git clone https://github.com/AvaLovelace1/BrickGPT.git && cd BrickGPT.
2. (Optional, required for running the infer script and texturing) Follow these instructions to install ImportLDraw, required for rendering brick structure visualizations:
   • Install the ImportLDraw submodule with git submodule update --init.
   • Download this background exr file and place it in the ImportLDraw/loadldraw subdirectory.
   • Download the LDraw parts library and extract it in your home directory: (cd ~ && wget https://library.ldraw.org/library/updates/complete.zip && unzip complete.zip).
     • If you wish to put the LDraw parts library in a different directory, set the environment variable LDRAW_LIBRARY_PATH to the path of the ldraw directory: export LDRAW_LIBRARY_PATH=path/to/ldraw.
3. Finally, install uv, and run uv sync to create a Python virtual environment with all dependencies installed. Python dependencies are defined in pyproject.toml.

Installing as a package

To install this repo as a package in your own Python project, first install all prerequisites. Then, run

uv add "https://github.com/AvaLovelace1/BrickGPT.git"

if using uv, or

pip install "https://github.com/AvaLovelace1/BrickGPT.git"

if using pip.

Running inference interactively

Gradio demo

A Gradio demo is available here. Its source code is provided in the subdirectory src/demo.

Command line interface (CLI)

You can run BrickGPT inference through the command line with the following command:

uv run infer

This script starts an interactive session where you can input a prompt and get a response from the model. The model weights will automatically be downloaded from Hugging Face.

If you wish to run inference with a different set of model weights, specify them using the --model_name_or_path option. See uv run infer -h for a full list of options.

Example interaction

Here is an example interaction using the infer script:

> uv run infer
Enter a prompt, or <Return> to exit: Table featuring a flat rectangular surface over four evenly spaced legs.
Enter a filename to save the output image (default=output.png): output.png
Enter a generation seed (default=42): 42
Generating...
Set parameter Username
Academic license - for non-commercial use only - expires 2026-02-19
--------------------
Finished generating in 63.53s.
Total # bricks: 59
Total # brick rejections: 98
Brick rejection reasons: {'collision': 5, 'already_rejected': 93}
Total # regenerations: 4
Saved results to /home/apun/BrickGPT/output.txt, /home/apun/BrickGPT/output.ldr, and /home/apun/BrickGPT/output.png
--------------------
Enter another prompt, or <Return> to exit:

Three output files are created: output.png, output.txt, and output.ldr.

output.png contains a rendered image of the generated brick structure:

<img src="assets/output_img.png" alt="Rendered brick structure output image" width="256"/>

output.txt contains the brick structure in brick-by-brick text format, where each line of the form hxw (x,y,z) represents a brick of height h and width w at position (x,y,z):

1x2 (16,18,0)
1x2 (16,13,0)
2x2 (0,18,0)
2x2 (0,13,0)
1x2 (16,18,1)
[...]

And finally, output.ldr contains the brick structure in LDraw format, which can be opened with any LDraw-compatible software.

Running texturing

The subdirectory src/texture contains the code for generating the UV texture or per-brick color given a brick design. To run texturing, cd into src/texture and follow the instructions in the README.md file there.

Running mesh-to-brick

The subdirectory src/mesh2brick contains the code for converting a 3D object mesh into a brick structure. To run mesh-to-brick, cd into src/mesh2brick and follow the instructions in the README.md file there.

Running fine-tuning

BrickGPT was created by fine-tuning Llama-3.2-1B-Instruct on the custom brick structure dataset StableText2Brick, converted into instructional format. We used Hugging Face TRL with Accelerate for fine-tuning.

To replicate the fine-tuning process, first install additional Python dependencies with uv sync --extra finetuning. Then, follow these instructions:

1. Prepare the brick structure dataset for fine-tuning with uv run prepare_finetuning_dataset --input_path AvaLovelace/StableText2Brick --output_path [FINETUNING_DATASET_PATH]. This converts the dataset into the instructional format required for fine-tuning LLaMA.
   • If you wish to run fine-tuning with your own brick structure dataset, replace AvaLovelace/StableText2Brick with the path to your dataset. This dataset should have the fields "captions" and "bricks". The "bricks" field should contain a brick structure in the text format described in the paper, and the "captions" field should contain a list of one or more descriptions of the brick structure.
2. Download the pretrained Llama-3.2-1B-Instruct model to some directory [PRETRAINED_DIR]. IMPORTANT: Replace the config.json, special_tokens_map.json, and tokenizer_config.json files with the ones in the finetuning_config_files directory. This specifies the pad_token to be different from the eos_token, fixing a fine-tuning issue where the model will not learn to output EOS tokens properly.
3. Initialize the Accelerate config file with uv run accelerate config.
4. Run fine-tuning with uv run ./scripts/finetune.zsh [PRETRAINED_DIR] [OUTPUT_DIR] [RUN_NAME] [FINETUNING_DATASET_PATH]. The fine-tuned model will be saved to [OUTPUT_DIR]/[RUN_NAME].

License

The BrickGPT model, StableText2Brick dataset, and majority of the BrickGPT code are licensed under the MIT License. The following submodules may have different licenses:

• ImportLDraw: For visualizing brick structures, we used ImportLDraw, available under the LICENSE.
• **[FlashTex](https://github.com/Ro