<div align="center"> <img width="500px" src="misc/diffmat_logo.png"> </div> <br/>

Overview

DiffMat is a PyTorch-based differentiable procedural material modeling library that reproduces the compositing graph system in Adobe Substance 3D Designer with auto-differentiation.

Primary features:

• Automatically converts Substance procedural materials (*.sbs) into differentiable computation graphs.
• Optimizes material node parameters to match user-captured material appearances (e.g., cellphone flash photos).

Requirements

Conda users

We provide an environment configuration file for Anaconda/Miniconda users. Clone the DiffMat repository and enter the source folder:

git clone git@github.com:mit-gfx/diffmat
cd diffmat

Create the virtual environment with:

conda env create -f environment.yml
conda activate diffmat

Non-conda users

Create a virtual environment (e.g., using venv) or configure your existing environment to meet the following requirements.

• Python ≥ 3.7
• PyTorch ≥ 1.11.0
• Torchvision ≥ 0.12.0
• Taichi-lang ≥ 1.3.0
• NumPy, SciPy, pandas, imageio, pyyaml, setuptools

Aside from PyTorch and Torchvision, all other packages are available via pip install:

pip install taichi numpy scipy pandas imageio pyyaml setuptools

Optional packages

The following packages are optional unless you want to experiment on alternative node parameter optimization algorithms discused in our paper or other image-space metrics.

• Bayesian optimization (BO): scikit-optimize, Ax
• Simulated annealing: simanneal
• LPIPS metric: lpips
• L*a*b* metric: kornia

This command will install them altogether.

pip install scikit-optimize ax-platform simanneal lpips kornia

In addition, you may download the FreeImage plugin to enable HDR texture images in OpenEXR format (*.exr).

python -c "import imageio; imageio.plugins.freeimage.download()"

Installation

Substance 3D Automation Tookit

DiffMat uses sbscooker and sbsrender, two command-line automation tools included in Adobe Substance 3D Designer (AS3D) or Substance 3D Automation Toolkit (SAT), to pre-cache the output textures of unsupported nodes.

DiffMat assumes that either AS3D or SAT is installed in its default, platform-specific folder and automatically detects the executables inside. A custom install location must be manually specified (see Getting Started).

NOTE: Our latest paper claims that DiffMat v2 does not rely on proprietary automation tools to generate noise textures. However, AS3D is still required from a practical perspective since implementing all generator nodes in DiffMat entails an unrealistic amount of effort.

NOTE: If you already have AS3D or SAT in your system, we recommend upgrading it to the latest version possible (12.1.1 as of 07/01/2022) to avoid any compatibility issue. We will continue to align DiffMat with newer software versions as they emerge.

Install DiffMat

At the root of the cloned repository, invoke the setup Python script using pip:

pip install .

To install in development mode:

pip install -e .

Exit the cloned repo and verify package integrity with:

cd ..
python -c "import diffmat; print(diffmat.__version__)"

Getting Started

We introduce the two most common ways to employ DiffMat in your procedural material capturing and authoring workflow: command-line scripts and Python API.

Command-Line Scripts

For independent usage, we include several Python scripts in the test/ folder that serve as basic command-line interfaces, including:

• test_nodes.py - translates a procedural material graph in *.sbs format into a differentiable program and computes output texture maps.
• test_optimizer.py - optimizes the continuous node parameters inside a procedural material graph to match the material appearance in an input texture image.
• test_hybrid_optimizer.py - optimizes both continuous and discrete node parameters in a procedural material graph.
• test_sampler.py - randomly samples or perturbs the continuous node parameters of a procedural material graph to generate texture variations.
• test_exposed_transfer.py - transfers the definition of an exposed parameter from one graph to another (demo only)

Run each testing script using the following command template. Replace content wrapped by square brackets as needed:

cd [PATH_TO_DIFFMAT]/test
python test_[NAME].py [PATH_TO_SBS_FILE] [OPTIONS]

Below are some shared command-line options across these scripts.

Command line options:
    -r PATH         Result directory (where a separate subfolder is created for every translated graph)
    --res INT       Output texture resolution after log2, must be an integer in [0, 12]
    -s SEED         Random seed; the usage varies between scripts
    -t PATH         Custom install location of AS3D or SAT
    -nf FORMAT      Normal format used for rendering output SVBRDF maps ('dx' DirectX or 'gl' OpenGL)
    -e              Force input noise textures to be generated using SAT
    -c              Change the PyTorch device to CPU; otherwise, use the default CUDA device if any
    -l LEVEL        Logging level ('none', 'quiet', 'default', or 'verbose')

NOTE: The -c option is mandatory on systems without a PyTorch-compatible GPU. Please refer to PyTorch documentation for devices accessible via torch.device('cuda').

You may inspect the complete command line options of each script using:

python test_[NAME].py -h

Result Folder Structure

For each procedural material graph, the output from testing scripts is organized into a folder that bears the same name as the graph. Without any command line options that change output folder names, the default result folder structure looks like:

📦result
 ┣ 📦[GRAPH_NAME]
 ┃  ┣ 📂default             Computed SVBRDF maps and physics-based rendering of the source material
 ┃  ┃
 ┃  ┣ 📂external_input
 ┃  ┃ ┗ 📂default           Input noises and patterns to the source material graph
 ┃  ┃
 ┃  ┣ 📂optim_[IMAGE]       Node parameter optimization result against an input texture image
 ┃  ┃ ┣ 📂checkpoints        +- Checkpoint files
 ┃  ┃ ┣ 📂export             +- Exported SBS file after optimization
 ┃  ┃ ┣ 📂render             +- Intermediate renderings
 ┃  ┃ ┣ 📂basecolor          +- Intermediate SVBRDF maps (albedo, normal, roughness, metallic, ...)
 ┃  ┃ ┗ 📂...
 ┃  ┃
 ┃  ┗ 📜summary.yml         Summary of translated material graph structure and node parameter values
 ┃
 ┣ 📦[ANOTHER_GRAPH_NAME]
 ┗ ...

Python API

You can also integrate DiffMat into your Python project via high-level API, which replaces the testing scripts above with equivalent Python classes.

| Script | Functionalities | Python Class | |----------------------------|---------------------------------------------------|-----------------------------------| | test_nodes.py | Graph translation & evaluation | diffmat.MaterialGraphTranslator | | test_optimizer.py | Gradient-based parameter optimization | diffmat.optim.Optimizer | | test_hybrid_optimizer.py | Mixed-integer parameter optimization | diffmat.optim.HybridOptimizer | | test_sampler.py | Random parameter sampling | diffmat.optim.ParamSampler |

For example, the following code snippet translates a procedural material named wood_american_cherry.sbs and optimizes graph parameters to match an input photo wood_dark_brown.jpg.

NOTE: While we use the pathlib package (internal to Python) to create platform-agnostic file paths in this example, ordinary Python strings work as well.

from pathlib import Path

from diffmat import MaterialGraphTranslator as MGT, config_logger
from diffmat.optim import Optimizer
from diffmat.core.io import read_image

# Enable on-screen logging
config_logger(level='default')

# Input and output file paths
sbs_file_path = Path('[PATH_TO_SBS_DIR]') / 'wood_american_cherry.sbs'
img_path = Path('[PATH_TO_IMG_DIR]') / 'wood_dark_brown.jpg'
result_dir = Path('[PATH_TO_RESULT_DIR]') / 'wood_american_cherry'

# Specify a location for storing pre-cached texture images from SAT
external_input_dir = result_dir / 'external_input'

# Translate the source material graph (using 512x512 resolution)
translator = MGT(sbs_file_path, res=9, external_noise=False)
graph = translator.translate(external_input_folder=external_input_dir, device='cuda')

# Compile the graph to generate a differentiable program
graph.compile()

# Read the target image (convert into a BxCxHxW tensor) and run gradient-based optimization for 1k iterations
target_img = read_image(img_path, device='cuda')[:3].unsqueeze(0)
optimizer = Optimizer(graph, lr=5e-4)
optimizer.optimize(target_img, num_iters=1000, result_dir=result_dir)

NOTE: The documentation to DiffMat Python API currently only consists of docstrings at functions and class methods (see Code Structure). We are actively planning on a documentation website fo