Bayesian Optimization, Reinforcement Learning, & Generative Model Research

This directory contains official implementations for Bayesian Optimization, Reinforcement Learning, & Generative Model works developed by Huawei, Noah's Ark Lab.

• Bayesian Optimisation Research
  • HEBO: Heteroscedastic Evolutionary Bayesian Optimisation
  • MCBO: Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization
  • NAP: End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes
  • RDUCB: High Dimensional Bayesian Optimisation with Random Decompositions
  • AntBO: Antibody Design with Combinatorial Bayesian Optimisation
  • BOiLS: Bayesian Optimisation for Logic Synthesis
  • T-LBO: BO in a latent space shaped by metric learning
  • Bayesian Optimisation with Compositional Optimisers
  • AIRBO: Efficient Robust Bayesian Optimisation for Arbitrary Uncertain Inputs
• Reinforcement Learning Research
  • Enhancing Reinforcement Learning Agents with Local Guides
  • Sauté RL and Simmer RL: Safe Reinforcement Learning Using Safety State Augmentation
  • Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  • SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks
• Robotics Research
  • ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning
• Generative Model Research
  • EM-LLM: Human-like Episodic Memory for Infinite Context LLMs
  • Mixture of Attentions For Speculative Decoding

Further instructions are provided in the README files associated to each project.

Bayesian Optimisation Research

HEBO

<img src="./HEBO/hebo.png" alt="drawing" width="400"/>

Bayesian optimization library developed by Huawei Noahs Ark Decision Making and Reasoning (DMnR) lab. The <strong> winning submission </strong> to the NeurIPS 2020 Black-Box Optimisation Challenge.

MCBO

<p float="center"> <img src="MCBO/paper_results/images/all_mix_match.PNG" width="400"/> <img src="MCBO/paper_results/images/results.png" width="400"/> </p>

Codebase associated to: Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization accepted at NeurIPS (2023).

Abstract

This paper introduces a modular framework for Mixed-variable and Combinatorial Bayesian Optimization (MCBO) to address the lack of systematic benchmarking and standardized evaluation in the field. Current MCBO papers often introduce non-diverse or non-standard benchmarks to evaluate their methods, impeding the proper assessment of different MCBO primitives and their combinations. Additionally, papers introducing a solution for a single MCBO primitive often omit benchmarking against baselines that utilize the same methods for the remaining primitives. This omission is primarily due to the significant implementation overhead involved, resulting in a lack of controlled assessments and an inability to showcase the merits of a contribution effectively. To overcome these challenges, our proposed framework enables an effortless combination of Bayesian Optimization components, and provides a diverse set of synthetic and real-world benchmarking tasks. Leveraging this flexibility, we implement 47 novel MCBO algorithms and benchmark them against seven existing MCBO solvers and five standard black-box optimization algorithms on ten tasks, conducting over 4000 experiments. Our findings reveal a superior combination of MCBO primitives outperforming existing approaches and illustrate the significance of model fit and the use of a trust region. We make our MCBO library available under the MIT license at https://github.com/huawei-noah/HEBO/tree/master/MCBO.

NAP: End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes

Codebase associated to: End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes, accepted at NeurIPS (2023).

Abstract

Meta-Bayesian optimisation (meta-BO) aims to improve the sample efficiency of Bayesian optimisation by leveraging data from related tasks. While previous methods successfully meta-learn either a surrogate model or an acquisition function independently, joint training of both components remains an open challenge. This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data. Early on, we notice that training transformer-based neural processes from scratch with RL is challenging due to insufficient supervision, especially when rewards are sparse. We formalise this claim with a combinatorial analysis showing that the widely used notion of regret as a reward signal exhibits a logarithmic sparsity pattern in trajectory lengths. To tackle this problem, we augment the RL objective with an auxiliary task that guides part of the architecture to learn a valid probabilistic model as an inductive bias. We demonstrate that our method achieves state-of-the-art regret results against various baselines in experiments on standard hyperparameter optimisation tasks and also outperforms others in the real-world problems of mixed-integer programming tuning, antibody design, and logic synthesis for electronic design automation.

RDUCB: High Dimensional Bayesian Optimisation with Random Decompositions

<p align="center"> <img src="./RDUCB/figures/ToyProblem.PNG" width="400" /> </p>

Codebase associated to: Are Random Decomositions all we need in High Dimensional Bayesian Optimisation accepted at ICML (2023).

Abstract

Learning decompositions of expensive-to-evaluate black-box functions promises to scale Bayesian optimisation (BO) to high-dimensional problems. However, the success of these techniques depends on finding proper decompositions that accurately represent the black-box. While previous works learn those decompositions based on data, we investigate data-independent decomposition sampling rules in this paper. We find that data-driven learners of decompositions can be easily misled towards local decompositions that do not hold globally across the search space. Then, we formally show that a random tree-based decomposition sampler exhibits favourable theoretical guarantees that effectively trade off maximal information gain and functional mismatch between the actual black-box and its surrogate as provided by the decomposition. Those results motivate the development of the random decomposition upper-confidence bound algorithm (RDUCB) that is straightforward to implement - (almost) plug-and-play - and, surprisingly, yields significant empirical gains compared to the previous state-of-the-art on a comprehensive set of benchmarks. We also confirm the plug-and-play nature of our modelling component by integrating our method with HEBO, showing improved practical gains in the highest dimensional tasks from Bayesmark.

AntBO: Antibody Design with Combinatorial Bayesian Optimisation

Codebase associated to: AntBO: Towards Real-World Automated Antibody Design with Combinatorial Bayesian Optimisation published in Cell Reports Methods (2023).

Abstract

Antibodies are canonically Y-shaped multimeric proteins capable of highly specific molecular recognition. The CDRH3 region located at the tip of variable chains of an antibody dominates antigen-binding specificity. Therefore, it is a priority to design optimal antigen-specific CDRH3 regions to develop therapeutic antibodies to combat harmful pathogens. However, the combinatorial nature of CDRH3 sequence space makes it impossible to search for an optimal binding sequence exhaustively and efficiently, especially not experimentally. Here, we present AntBO: a Combinatorial Bayesian Optimisation framework enabling efficient in silico design of the CDRH3 region. Ideally, antibodies should bind to their target antigen and be free from any harmful outcomes. Therefore, we introduce the CDRH3 trust region that restricts the search to sequences with feasible developability scores. To benchmark AntBO, we use the Absolut! software suite as a black-box oracle because it can score the target specificity and affinity of designed antibodies in silico in an unconstrained fashion. The results across 188 antigens demonstrate the benefit of AntBO in designing CDRH3 regions with diverse biophysical properties. In under 200 protein designs, AntBO can suggest antibody sequences that outperform the best binding sequence drawn from 6.9 million experimentally obtained CDRH3s and a commonly used genetic algorithm baseline. Additionally, AntBO finds very-high affinity CDRH3 sequences in only 38 protein designs whilst requiring no domain knowledge. We conclude AntBO brings automated antibody design methods closer to what is practically viable for in vitro experimentation.

BOiLS: Bayesian Optimisation for Logic Synthesis

<p align="center"> <img src="./BOiLS/results/sample-eff-1.png" alt="drawing" width="500"/> </p>

Codebase associated to: BOiLS: Bayesian Optimisation for Logic Synthesis accepte