[ICCV 2025] <span style="font-variant: small-caps;">Neurons</span>: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction

<div align="center">

🌟 If you find our project useful, please consider giving us a star! ⭐

</div>

📌 Overview

<div align="center">

Architecture of the <span style="font-variant: small-caps;">Neurons</span> framework

</div>

<span style="font-variant: small-caps;">Neurons</span> is a novel framework that emulates the human visual cortex to achieve high-fidelity and interpretable fMRI-to-video reconstruction. Our biologically inspired approach significantly advances the state-of-the-art in brain decoding and visual reconstruction.

📣 Latest Updates

🟡 2025/10    Released model weights, training logs, testing logs, and generated images and videos — all available at Hugging Face! ⚠️ Note 1: Cloning the entire EXP folder requires over 60 GB of storage. To download selectively, use snapshot_download with the allow_patterns parameter (e.g., allow_patterns=["EXP/exp_neurons/subj_1/*"] to download only subject 1). ⚠️ Note 2: Due to a server issue, the original weights were lost. We re-cloned the repository and re-ran the experiments. While specific numerical results may vary slightly, the overall performance remains consistent with the paper, which also verifies the reproducibility of this work.

🟡 2025/06    Neurons is accepted by ICCV-2025!

🟡 2025/04    Code released!

🟡 2025/03    Project launched with paper available on arXiv!

🛠️ Installation & Setup

🖥️ Environment Setup

We recommend using separate environments for training and testing:

# Training environment
conda create -n neurons_train python==3.10
conda activate neurons_train
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

# Testing environment (to avoid package conflicts)
conda create -n neurons_test --clone neurons_train
conda activate neurons_test
pip install diffusers==0.11.1

📊 Data Preparation

1. Download the pre-processed dataset:

python download_dataset.py
tar -xzvf ./cc2017_dataset/masks/mask_cls_train_qwen_video.tar.gz -C ./cc2017_dataset/masks/
tar -xzvf ./cc2017_dataset/masks/mask_cls_test_qwen_video.tar.gz -C ./cc2017_dataset/masks/

2. Run task construction scripts:

# Rule-based Key Object Discovery
python tasks_construction/find_key_obj.py

# Generate CLIP embeddings
python -m tasks_construction.gen_GT_clip_embeds

⚙️ Pretrained Weights Preparation

mkdir pretrained_weights
cd pretrained_weights
wget -O unclip6_epoch0_step110000.ckpt -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/unclip6_epoch0_step110000.ckpt\?download\=true
wget -O last.pth -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/train_logs/final_subj01_pretrained_40sess_24bs/last.pth\?download\=true
wget -O convnext_xlarge_alpha0.75_fullckpt.ckpt -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/convnext_xlarge_alpha0.75_fullckpt.pth\?download\=true
wget -O sd_image_var_autoenc.pth https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/sd_image_var_autoenc.pth\?download\=true
cd ..

🚀 Quick Start

This codebase allows train, test, and evaluate using one single bash file.

bash train_neurons.sh 0 neurons 123456 enhance 1

---

Parameters:

$1: use which gpu to train

$2: train file postfix, e.g, train_neurons

$3: run which stage: 123456 for the whole process, 3456 for test & eval only

• 1: train brain model
• 2: train decoupler
• 3: recon decoupled outputs, prepare for video reconstruction
• 4: (Optional) caption the keyframes with BLIP-2 instead of using the outputs of GPT-2 in Neurons
• 5: video reconstruction
• 6: evaluation with all metrics

$4: inference mode: ['enhance', 'motion']

$5: train which subject: [0,1,2]

---

Note that for convenience of debugging, use_wandb is set to False be default.

If you would like to use wandb, first run wandb login and set the use_wandb to True in train_neurons.py.

📚 Citation

If you find this project useful, please consider citing:

@article{wang2025neurons,
  title={NEURONS: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction},
  author={Wang, Haonan and Zhang, Qixiang and Wang, Lehan and Huang, Xuanqi and Li, Xiaomeng},
  journal={arXiv preprint arXiv:2503.11167},
  year={2025}
}