Evolution and learning in differentiable robots

Code up your favorite evolutionary algorithm and leverage the power of CUDA-accelerated differentiable simulation for fast policy learning and fitness evaluation in complex robots.

This project was developed out of the paper Evolution and learning in differentiable robots, Strgar et al. See the 2024-RSS-Strgar branch of this repository for code that reproduces experimental results from that paper.

:star: Project Site & Video :star: Paper :star: Citation

<p align="center"> <img src="./assets/1.gif" alt="animated" style="width:700px; height:auto"/> </p> <p align="center"> <img src="./assets/2.gif" alt="animated" style="width:700px; height:auto"/> </p> <p align="center"> <img src="./assets/0.gif" alt="animated" style="width:700px; height:auto"/> </p>

Table of contents

Installation

Quick start

Visualize results

Accelerating experiments with CUDA

Scaling experiments

Adapting the evolutionary algorithm

Adapting the simulator & learning objective

Installation

<pre lang="bash"> git clone git@github.com:lstrgar/ELDiR.git; cd ELDiR conda create --name ELDiR python=3.10.13 --yes conda activate ELDiR pip install -r requirements.txt </pre>

Notes:

• If you intend to make substantial modifications we recommend forking the respository.
• We find that an isolated conda environment is easy to work with; however, this is not required so long as you have a working pip installation.

Quick start

A simple python invocation will initiate evolution and learning according to the algorithm described in our paper:

<pre lang="bash"> python main.py </pre>

Visualize results

Visualization of robots is handled automatically with each generation. Automated visualization can be deactivated by running main.py with the flag --no_viz. After running the program with the defaults without automated visualization, you can visualize your results using the visualize-defaults.ipynb notebook. If you discover any exciting robots please share your results! Here is a little skipper we found :)

<p align="center"> <img src="./assets/robo.gif" alt="animated" style="width:700px; height:auto"/> </p>

Custom uneven terrain

Terrain is represented by piecewise linear functions, and the program supports uneven terrain files. To enter a terrain file, simply run main.py with the argument flag --groundfile and paste in the path of your ground file. Running the program without a ground file will simulate with flat terrain. The floor.ipynb notebook has functions that can randomly generate viable ground files as well (parameters are suggested but can be edited to whatever the simulation might need). Ground files must be composed of five float lists:

beg_x: starting x_coordinate of each line segment beg_y: starting y_coordinate of each line sgement lens: length of each line segment slopes: slope of each line segment shifts: y_intercept of each line segment

where i in all lists represents a consecutive line segment. All lists must contain the same number of objects.

Accelerating experiments with CUDA

This framework can be used with or without a CUDA-enabled GPU:

<pre lang="python"> # main.py use_cuda = False # True </pre>

If your machine has more than one GPU you can parallelize across a subset of them by specifying the desired device indices:

<pre lang="python"> # main.py device_ids = [0, 3, ...] </pre>

Scaling experiments

Subject to your available computing resources you may want to increase or decrease the scale of your experiments. There are several straightforward ways of doing this:

:diamond_shape_with_a_dot_inside: Robot population size. Robots are simulated and trained in parallel. A smaller/larger population size means decreasing/increasing simulator space complexity and, possibly, time complexity.

:diamond_shape_with_a_dot_inside: Number of generations. Time complexity scales directly with the number of generations evolution runs for.

<pre lang="python"> # main.py pop_size = 10 n_gens = 50 </pre>

Adapting the evolutionary algorithm

The codebase distinguishes three evolutionary algorithmic components: genetic representation, mutation, and selection. You can easily swap-in your own algorithm by implementing a small set of functions and overriding the defaults.

Default implementations

By default the framework follows the evolutionary method described in our paper. Briefly, we employ a direct representation of genotypes in terms of a binary grid over which random mutation occurs by flipping bits in the mask. Our selection operator chooses the top 50% of robots based on their learned ability to locomote. The code implementing the defaults is found in the operators/defaults folder.

The interface

Each of the three components and their corresponding interfaces are briefly described below. For more information see the operators folder where will find the geno_pheno.py, mutate.py, and select.py files, which specify detailed templates and instructions for for implementing the necessary functions. It may also be helpful to review our [distinction between the fitness function and the learning objective](.

:large_orange_diamond: Genetic representation

To create a custom genetic representation users must implement the following two functions:

<pre lang="python"> def random_geno(n: Int, outdir: Str) -> Str </pre>

The function random_geno takes an integer n and string outdir as input, generates a random population of n genotypes, saves the population to a single file in outdir, and returns the file path. Note: our code is agnostic to the data structure and file format used to store the population and related metadata. See the operators/geno_pheno.py file for more information.

<pre lang="python"> def geno_2_pheno(pop_fpath: Str) -> None </pre>

The function geno_2_pheno takes a string pop_fpath (from random_geno), loads the population, appends a phenotype representation of each robot, and saves the result back to pop_fpath. The phenotype representation must include the dictionary keys points and springs, which represent the vertices and edges of the robot's body. See the operators/geno_pheno.py file for more information, specifically that related to the structure of points and springs and suggestions for default values.

:large_orange_diamond: Mutation

To create a custom mutation operator users must implement the following function:

<pre lang="python"> def mutate(pop_fpath: Str, gen_idx: Int, fit: NDArray) -> Str </pre>

The function mutate takes a string pop_fpath, the path of the population genotype file (from random_geno), an integer gen_idx, and numpy array fit. gen_idx corresponds to the generation number of the offspring that will be created by calling this function, which may be useful metadata. The array fit is of shape n rows by learning_iters columns and represents the positive performance trajectories of each robot in the population throughout learning. mutate applies a mutation to the population genotypes, writes the offspring population to a file, and returns the file path. See the operators/mutate.py file for more information.

:large_orange_diamond: Selection

To create a custom selection operator users must implement the following function:

<pre lang="python"> def select(pop_fpath: Str, pop_fit: NDArray, offspring_fpath: Str, offspring_fit: NDArray, outdir: Str) -> tuple[Str, NDArray] </pre>

The select function takes two strings (pop_fpath, offspring_fpath), two numpy arrays (pop_fit, offspring_fit), and a string outdir as inputs. The _fpath strings represent file paths to the current population and offspring population (from mutate). The arrays represent the positive performance trajectories of the two populations throughout learning. outdir is the directory to which select should write results. The select function combines the two populations into the next generation of robots, writes the next generation to a file (in outdir), and returns the file path. The function must also return a numpy array containing the performance trajectories of the new population in row order matching that of robots written to the next generation file. See the `operators/sele