Downloads

Learning 4 CasADi Framework

L4CasADi enables the seamless integration of PyTorch-learned models with CasADi for efficient and potentially hardware-accelerated numerical optimization. The only requirement on the PyTorch model is to be traceable and differentiable.

<div align="center"> <img src="./examples/nerf_trajectory_optimization/media/animation.gif" alt="Collision-free minimum snap optimized trajectory through a NeRF" width="200" height="200"> <img src="./examples/fish_turbulent_flow/media/trajectory_generation_vorticity.gif" alt="Energy Efficient Fish Navigation in Turbulent Flow" width="400" height="200"> <a target="_blank" href="https://colab.research.google.com/github/Tim-Salzmann/l4casadi/blob/main/examples/nerf_trajectory_optimization/NeRF_Trajectory_Optimization.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>                                                <a target="_blank" href="https://colab.research.google.com/github/Tim-Salzmann/l4casadi/blob/main/examples/fish_turbulent_flow/Fish_Turbulent_Flow.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>                       Two L4CasADi examples: Collision-free trajectory through a NeRF and Navigation in Turbulent Flow </div>

arXiv: Learning for CasADi: Data-driven Models in Numerical Optimization

Talk: Youtube

L4CasADi v2 Breaking Changes

After feedback from first use-cases L4CasADi v2 is designed with efficiency and simplicity in mind.

This leads to the following breaking changes:

L4CasADi v2 can leverage PyTorch's batching capabilities for increased efficiency. When passing batched=True, L4CasADi will understand the first input dimension as batch dimension. Thus, first and second-order derivatives across elements of this dimension are assumed to be sparse-zero. To make use of this, instead of having multiple calls to a L4CasADi function in your CasADi program, batch all inputs together and have a single L4CasADi call. An example of this can be seen when comparing the non-batched NeRF example with the batched NeRF example which is faster by a factor of 5-10x.
L4CasADi v2 will not change the shape of an input anymore as this was a source of confusion. The tensor forwarded to the PyTorch model will resemble the exact dimension of the input variable by CasADi. You are responsible to make sure that the PyTorch model handles a two-dimensional input matrix! Accordingly, the parameter model_expects_batch_dim is removed.
By default, L4CasADi v2 will not provide the Hessian, but the Jacobian of the Adjoint. This is sufficient for most many optimization problems. However, you can explicitly request the generation of the Hessian by passing generate_jac_jac=True.

Table of Content

Projects using L4CasADi
Installation
Quick Start
Online Learning
Naive L4CasADi - Use this for small Multi Layer Perceptron Models.
Real-time L4CasADi - Use this for fast MPC with Acados.
Examples

If you use this framework please cite the following two papers:

@article{salzmann2023neural,
  title={Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms},
  author={Salzmann, Tim and Kaufmann, Elia and Arrizabalaga, Jon and Pavone, Marco and Scaramuzza, Davide and Ryll, Markus},
  journal={IEEE Robotics and Automation Letters},
  doi={10.1109/LRA.2023.3246839},
  year={2023}
}

@inproceedings{{salzmann2024l4casadi,
  title={Learning for CasADi: Data-driven Models in Numerical Optimization},
  author={Salzmann, Tim and Arrizabalaga, Jon and Andersson, Joel and Pavone, Marco and Ryll, Markus},
  booktitle={Learning for Dynamics and Control Conference (L4DC)},
  year={2024}
}

Projects using L4CasADi

Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms Paper | Code
AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control Paper
Reinforcement Learning based MPC with Neural Dynamical Models Paper
Neural Potential Field for Obstacle-Aware Local Motion Planning Paper | Video | Code
N-MPC for Deep Neural Network-Based Collision Avoidance exploiting Depth Images Paper | Code
An Integrated Framework for Autonomous Driving Planning and Tracking based on NNMPC Considering Road Surface Variations Paper

If your project is using L4CasADi and you would like to be featured here, please reach out.

Installation

Prerequisites

Independently if you install from source or via pip you will need to meet the following requirements:

Working build system: CMake compatible C++ compiler.
- On Linux/macOS: GCC version 10 or higher is recommended.
- On Windows: You must install CMake and Visual C++ Build Tools (do not use a GCC compiler).
PyTorch (>=2.0) installation in your python environment.
python -c "import torch; print(torch.__version__)"

Windows Installation

For users installing on Windows, please ensure the following steps are taken:

Install CMake and Visual C++ Build Tools: Download and install CMake and the Visual C++ Build Tools.
Ensure Tools Are on Your PATH: Make sure both CMake and the Visual C++ Build Tools are added to your system's PATH. This can be done either via the installers (by selecting the option to update the PATH) or manually.
Mind the File Path Length: When building the L4CasADi model, ensure that the directory path does not exceed Windows’ maximum character limit. Long paths might lead to build issues, so choose a directory with a short path if possible.

Pip Install (CPU Only)

Ensure Torch CPU-version is installed
pip install torch>=2.0 --index-url https://download.pytorch.org/whl/cpu
Ensure all build dependencies are installed

setuptools>=68.1
scikit-build>=0.17
cmake>=3.27
ninja>=1.11

Run
pip install l4casadi --no-build-isolation

From Source (CPU Only)

Clone the repository
git clone https://github.com/Tim-Salzmann/l4casadi.git
All build dependencies installed via
pip install -r requirements_build.txt
Build from source
pip install . --no-build-isolation

The --no-build-isolation flag is required for L4CasADi to find and link against the installed PyTorch.

GPU (CUDA)

CUDA installation requires nvcc to be installed which is part of the CUDA toolkit and can be installed on Linux via sudo apt-get -y install cuda-toolkit-XX-X (where XX-X is your installed Cuda version - e.g. 12-3). Once the CUDA toolkit is installed nvcc is commonly found at /usr/local/cuda/bin/nvcc.

Make sure nvcc -V can be executed and run pip install l4casadi --no-build-isolation or CUDACXX=<PATH_TO_NVCC> pip install . --no-build-isolation to build from source.

If nvcc is not automatically part of your path you can specify the nvcc path for L4CasADi. E.g. CUDACXX=<PATH_TO_NVCC> pip install l4casadi --no-build-isolation.

Quick Start

Defining an L4CasADi model in Python given a pre-defined PyTorch model is as easy as

import l4casadi as l4c

l4c_model = l4c.L4CasADi(pyTorch_model, device='cpu')

where the architecture of the PyTorch model is unrestricted and large models can be accelerated with dedicated hardware.

Online Learning and Updating

L4CasADi supports updating the PyTorch model online in the CasADi graph. To use this feature, pass mutable=True when initializing a L4CasADi. To update the model, call the update function on the L4CasADi object. You can optionally pass an updated model as parameter. If no model is passed, the reference passed at initia

L4casadi

Install / Use

README