RandOpt

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan, Phillip Isola

Paper | Project Page | Starting with a 1D Experiment:

Requirements

Option1: Python / Conda

(optional) conda activate your_env
pip install -r requirements.txt

Option2: Docker

From the directory containing RandOpt/:

| Step | Command | |------|---------| | Build | docker build -f RandOpt/docker/Dockerfile_vllm -t randopt-vllm:latest . | | Run | docker run -it --gpus all randopt-vllm:latest bash | | Run (with data) | docker run -it --gpus all -v /path/to/RandOpt/data:/workspace/data randopt-vllm:latest bash |

Run RandOpt

Post-train on your own dataset

Please follow the instructions in CUSTOM_DATASET_GUIDE.md

Post-train on a standard dataset

First download the data here: data/README.md

Then, from the RandOpt directory:

| Mode | Command | |------|---------| | Single node | sbatch scripts/single_node.sh | | Multiple nodes | sbatch scripts/multiple_nodes.sh | | Local (no Slurm) | bash scripts/local_run.sh |

Distill top-k models into a single model

Please follow the instructions in distillation/README.md.

Run Baselines

Please follow the instructions in baselines/README.md

Citation

@misc{gan2026neuralthickets,
      title={Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights}, 
      author={Yulu Gan and Phillip Isola},
      year={2026},
      eprint={2603.12228},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.12228}, 
}