NeuroMHE

The Neural Moving Horizon Estimation (NeuroMHE) is an auto-tuning and adaptive optimal estimator. It fuses a neural network with an MHE to realize accurate estimation and fast online adaptation, leveraging the advantages from both advanced machine learning techniques and control-theoretic estimation algorithms. The neural network can be trained efficiently from the robot's trajectory tracking errors using reinforcement learning without the need for the ground truth data.

| A Diagram of the NeuroMHE-based Robust Flight Control System and Its Learning Piplines | :----------------------------------------------------------------------------------------------------------------------------------:

Please find out more details in

• our paper "Neural Moving Horizon Estimation for Robust Flight Control," IEEE Xplore, arXiv
• YouTube video: https://www.youtube.com/watch?v=L5SrVr5ha-k

For learning estimation policy using the second-order trust-region method, kindly refer to our another respository https://github.com/BinghengNUS/TR-NeuroMHE

Citation

If you find this work helpful in your publications, we would appreciate citing our paper (published in IEEE T-RO)

@ARTICLE{10313083,
  author={Wang, Bingheng and Ma, Zhengtian and Lai, Shupeng and Zhao, Lin},
  journal={IEEE Transactions on Robotics}, 
  title={Neural Moving Horizon Estimation for Robust Flight Control}, 
  year={2024},
  volume={40},
  number={},
  pages={639-659},
  doi={10.1109/TRO.2023.3331064}}

Table of contents

1. Project Overview
2. Dependency Packages
3. How to Use
   1. SecVII-A: Accurate Estimation
   2. SecVII-B: Online Learning
   3. SecVII-C: Real-world Experiments
   4. Applications to other robots
4. Acknowledgement
5. Contact Us

1. Project Overview

The project consists of three folders, which correspond to the three experiments in the paper that show the following four advantages of our method.

1. NeuroMHE enjoys computationally efficient training and significantly improves the force estimation performance as compared to a state-of-the-art estimator [1].
2. A stable NeuroMHE with a fast dynamic response can be trained directly from the trajectory tracking error using Algorithm 2 (i.e., the model-based policy gradient algorithm) .
3. NeuroMHE exhibits superior estimation and robust control performance than a fixed-weighting MHE (DMHE) [2] and a state-of-the-art adaptive controller [3] for handling dynamic noise covariances.
4. Finally, NeuroMHE is efficiently transferable to different challenging flight scenarios on a real quadrotor without extra parameter tuning, such as counteracting state-dependent cable forces and flying under the downwash flow.

• SecVII-A (source code): A comprehensive comparison with the state-of-the-art NeuroBEM estimator on its real agile test dataset to show the first advantage.
• SecVII-B (source code): A robust trajectory tracking control scenario in simulation to show the second and third advantages.
• SecVII-C (source code): Physical experiments on a real quadrotor to show the fourth advantage.

2. Dependency Packages

Please make sure that the following packages have already been installed before running the source code.

• CasADi: version 3.5.5 Info: https://web.casadi.org/
• Numpy: version 1.23.0 Info: https://numpy.org/
• Pytorch: version 1.12.0+cu116 Info: https://pytorch.org/
• Matplotlib: version 3.3.0 Info: https://matplotlib.org/
• Python: version 3.9.12 Info: https://www.python.org/
• Scipy: version 1.8.1 Info: https://scipy.org/
• Pandas: version 1.4.2 Info: https://pandas.pydata.org/
• filterpy: version 1.4.5 Info: https://filterpy.readthedocs.io/en/latest/
• scikit-learn: version 1.0.2 Info: https://scikit-learn.org/stable/whats_new/v1.0.html

3. How to Use

First and foremost, the training process for NeuroMHE is both efficient and straightforward to setup. The source code has been comprehensively annotated to facilitate ease of use. To reproduce the simulation results presented in the paper, simply follow the steps outlined below, sequentially, after downloading and decompressing all the necessary folders.

SecVII-A: Accurate Estimation

1. Download 'processed_data.zip' and 'predictions.tat.xz' from https://download.ifi.uzh.ch/rpg/NeuroBEM/. The former file is utilized for training NeuroMHE, whereas the latter serves the purpose of evaluation and comparison with NeuroBEM.
2. Relocate the folder 'bem+nn' from the decomprassed archive 'predictions.tat.xz' to the downloaded folder 'SecVII-A (source code)', and place the decompressed 'processed_data.zip' within the folder 'SecVII-A (source code)' as well.
3. Run the Python file 'main_code_supervisedlearning.py'.
4. In the prompted terminal interface, you will be asked to select whether to train or evaluate NeuroMHE.
   • Training: type 'train' without the quotation mark in the terminal.
   • Evaluation: type 'evaluate' without the quotation mark in the terminal. Subsequently, you will be prompted to select a trajectory for evaluation. There are a total of 13 agile trajectories within the complete NeuroBEM test dataset, as shown below. Note that you can skip the training process and directly evaluate the performance using the trained neural network model 'Trained_model.pt' to reproduce the RMSE results presented in our paper. The retained model is saved in the folder 'trained_data'.

| Trajectory Parameters of NeuroBEM Test Dataset | :----------------------------------------------------------------------------------------------------------------------------------:

Please note that training performance may vary depending on the computer used. This variation arises from the gradient information's dependence on the MHE estimation trajectory, which is computed in the forward pass by a numerical solver. The solver's accuracy is influenced by its parameters and the computer's specifications. To uphold the training performance, we may slightly adjust the solver's parameters in Uav_mhe_supervisedlearning.py. Here are two training examples using different sets of IPOPT parameters on different computers. Training on a workstation | Training on a laptop :---------------------------------------------------------------:|:--------------------------------------------------------------: | opts['ipopt.tol']=1e-8 | opts['ipopt.tol']=1e-6 opts['ipopt.max_iter']=3e3 | opts['ipopt.max_iter']=1e3 opts['ipopt.acceptable_tol']=1e-7 | opts['ipopt.acceptable_tol']=1e-5

Another tricky point is that training stability can depend on the Python version. We found that using Python 3.10 with the IPOPT parameters from the left column of the above table may lead to instability. For Python 3.10, workable IPOPT parameters are: opts['ipopt.tol']=1e-14, opts['ipopt.max_iter']=3e3, and opts['ipopt.acceptable_tol']=1e-14. To ensure reproducibility, it is recommended to keep the Python version aligned with the one listed above (version 3.9) and fine-tune the IPOPT parameters if needed. In addition, for newer Python versions, it is good practice to create 'optimizer_p' outside the while-loop in main_code_supervisedlearning.py, since re-creating the optimizer at every iteration resets its internal state and can cause unstable learning.

One advantage of NeuroBEM is that its accuracy only declines by 20% when the training dataset encompasses a limited portion of the velocity-range space compared to the test dataset. To assess the performance of our NeuroMHE in this scenario, we select two 10-second-long segments for training: 1) one from an agile figure-8 trajectory, covering a wide velocity range of 0.05 m/s to 16.38 m/s, referred to as the 'fast training set'; 2) the other from a relatively slow wobbly circle trajectory, with a limited velocity range of 0.19 m/s to 5.18 m/s, referred to as the 'slow training set'. The following figures present a comparison of the velocity-range space in the world frame between the training sets and the partial test sets. Velocity-Range Space: Training Sets | Velocity-Range Space: Partial Test Sets :---------------------------------------------------------------:|:--------------------------------------------------------------: |

We evaluate the performance of NeuroMHE, trained on the 'slow training set', in comparison to NeuroBEM on its complete test dataset. The comparative results in terms of RMSE are summarized in Table V of our paper.

5. Run the Python file 'plotting.py' to reproduce the figures in Section VII-A of our paper.
6. Check the RMSE results. In the folder 'Check_RMSE',
   • Run the MATLAB file 'RMSE_vector_error_slowtrainingset.m' to replicate the RMSE results presented in Table V of our paper.
   • Additionally, we provide the RMSE