Litterature

A place to store relevant research papers and resources

[author: Matthieu Komorowski]

GENERIC AI/ML IN HEALTHCARE

• All models are local: time to replace external validation with recurrent local validation [2023] https://arxiv.org/abs/2305.03219
• Operationalizing Machine Learning: An Interview Study [2022] https://arxiv.org/abs/2209.09125
• Implementing machine learning in medicine [2021] https://www.cmaj.ca/content/193/34/E1351
• Do no harm: a roadmap for responsible machine learning for health care [2019] https://www.nature.com/articles/s41591-019-0548-6#Sec9

RL

• A Primer on Reinforcement Learning in Medicine for Clinicians [2024] https://www.nature.com/articles/s41746-024-01316-0

Courses

• DeepMind x UCL RL Lecture Series [2021] https://www.youtube.com/watch?v=TCCjZe0y4Qc&list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm

Textbooks

• Sutton & Barto 2nd Ed [2018] http://incompleteideas.net/book/the-book-2nd.html

Learning optimal policies

• Learning-to-defer for sequential medical decision-making under uncertainty [2022] https://arxiv.org/abs/2109.06312
• Deconfounding Actor-Critic Network with Policy Adaptation for Dynamic Treatment Regimes [2022] https://arxiv.org/abs/2205.09852
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning [2021] https://arxiv.org/abs/2108.08812
• Overcoming Model Bias for Robust Offline Deep Reinforcement Learning [2021] https://arxiv.org/abs/2008.05533
• The Difficulty of Passive Learning in Deep Reinforcement Learning [2021] https://arxiv.org/abs/2110.14020
• Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients [2021] https://pubmed.ncbi.nlm.nih.gov/33936452/
• Deep reinforcement learning from human preferences [2017] https://arxiv.org/abs/1706.03741
• High Confidence Policy Improvement [2015] https://proceedings.mlr.press/v37/thomas15.html

State representation

• Learning Markov State Abstractions for Deep Reinforcement Learning [2021] https://arxiv.org/abs/2106.04379
• SOM-VAE: Interpretable Discrete Representation Learning on Time Series [2019] https://arxiv.org/abs/1806.02199

OPE

• Reliable Off-Policy Learning for Dosage Combinations [2023] https://arxiv.org/abs/2305.19742
• Universal Off-Policy Evaluation [2021] https://arxiv.org/abs/2104.12820
• Off-Policy Deep Reinforcement Learning without Exploration [2018] https://arxiv.org/abs/1812.02900
• HCOPE [2015] https://ojs.aaai.org/index.php/AAAI/article/view/9541

XRL (and XAI in general)

• Interpretable ML textbook [update 2023] https://christophm.github.io/interpretable-ml-book/
• Multiple stakeholders drive diverse interpretability requirements for machine learning in healthcare [2023] https://www.nature.com/articles/s42256-023-00698-2
• Explainable Reinforcement Learning via Model Transforms [2022] https://arxiv.org/abs/2209.12006
• Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review [2022] https://arxiv.org/abs/2010.10596
• Policy Optimization with Sparse Global Contrastive Explanations [2022] https://arxiv.org/abs/2207.06269
• Counterfactuals with Reinforcement Learning [2022] https://docs.seldon.io/projects/alibi/en/latest/methods/CFRL.html
• Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning [2021] https://arxiv.org/abs/2106.02597
• XRL review [2020] https://towardsdatascience.com/xrl-explainable-reinforcement-learning-4cd065cdec9a
• Explainable Reinforcement Learning via Reward Decomposition [2019] https://finale.seas.harvard.edu/publications/explainable-reinforcement-learning-reward-decomposition
• Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead [2019] https://www.nature.com/articles/s42256-019-0048-x

Federated RL

• Federated Ensemble-Directed Offline Reinforcement Learning [2023] https://arxiv.org/abs/2305.03097
• Federated Reinforcement Learning with Environment Heterogeneity [2022] https://proceedings.mlr.press/v151/jin22a.html - https://github.com/pengyang7881187/FedRL/tree/main

RL in SEPSIS

• Optimal Vasopressin Initiation in Septic Shock - The OVISS Reinforcement Learning Study [2025] https://jamanetwork.com/journals/jama/fullarticle/2831858
• ADT2R: Adaptive Decision Transformer for Dynamic Treatment Regimes in Sepsis [2024] https://ieeexplore.ieee.org/abstract/document/10659167
• Towards more efficient and robust evaluation of sepsis treatment with deep reinforcement learning [2023] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9979564/
• A value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis [2023] https://www.nature.com/articles/s41746-023-00755-5 + https://github.com/CaryLi666/ID3QNE-algorithm
• The treatment of sepsis: an episodic memory-assisted deep reinforcement learning approach [2023] https://link.springer.com/article/10.1007/s10489-022-04099-7
• Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding [2023] https://arxiv.org/abs/2306.01157
• Uniformly Conservative Exploration in Reinforcement Learning [2023] https://proceedings.mlr.press/v206/xu23j.html
• Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications [2023] https://arxiv.org/abs/2302.07549
• A dosing strategy model of deep deterministic policy gradient algorithm for sepsis patients [2023] https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-023-02175-7
• Adversarial reinforcement learning for dynamic treatment regimes [2023] https://www.sciencedirect.com/science/article/pii/S1532046422002490
• Reinforcement Learning For Sepsis Treatment: A Continuous Action Space Solution [2022] https://proceedings.mlr.press/v182/huang22a.html
• An interpretable RL framework for pre-deployment modeling in ICU hypotension management [2022] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9671896/
• Establishment and Implementation of Potential Fluid Therapy Balance Strategies for ICU Sepsis Patients Based on Reinforcement Learning [2022] https://pubmed.ncbi.nlm.nih.gov/35492326/
• A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies [2022] https://arxiv.org/abs/2203.13884
• Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare [2022] https://proceedings.neurips.cc/paper_files/paper/2022/hash/dda7f9378a210c25e470e19304cce85d-Abstract-Conference.html
• Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data [2022] https://proceedings.neurips.cc/paper_files/paper/2022/hash/5ee7ed60a7e8169012224dec5fe0d27f-Abstract-Conference.html
• Federated Offline Reinforcement Learning [2022] https://arxiv.org/abs/2206.05581
• Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment [2022] https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000012
• Superhuman performance on sepsis MIMIC-III data by distributional reinforcement learning [2022] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0275358
• Establishment and Implementation of Potential Fluid Therapy Balance Strategies for ICU Sepsis Patients Based on Reinforcement Learning [2022] https://www.frontiersin.org/articles/10.3389/fmed.2022.766447/full
• Safe RL for sepsis treatment [2022] http://d-scholarship.pitt.edu/42914/
• Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space [2022] https://link.springer.com/chapter/10.1007/978-3-031-20627-6_11
• Medical Dead-ends and Learning to Identify High-Risk States and Treatments [2021] https://proceedings.neurips.cc/paper/2021/hash/26405399c51ad7b13b504e74eb7c696c-Abstract.html
• Combining Model-Based and Model-Free Reinforcement Learning Policies for More Efficient Sepsis Treatment [2021] https://www.springerprofessional.de/en/combining-model-based-and-model-free-reinforcement-learning-poli/19877428
• Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis [2021] https://pubmed.ncbi.nlm.nih.gov/33581824/
• Safety-driven design of machine learning for sepsis treatment [2021] https://www.sciencedirect.com/science/article/pii/S1532046421000915
• Individualized fluid administration for critically ill patients with sepsis with an interpretable dynamic treatment regimen model [2020] https://www.nature.com/articles/s41598-020-74906-z
• Combining Reinforcement Learning with Supervised Learning for Sepsis Treatment [2020] https://dl.acm.org/doi/10.1145/3426020.3426077
• Deep Inverse Reinforcement Learning for Sepsis Treatment [2019] https://www.computer.org/csdl/proceedings-article/ichi/2019/08904645/1f8N9kBhTgY
• Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning [2019] https://arxiv.org/abs/1901.04670

RL in the ICU outside of sepsis

• Personalized decision making for coronary artery disease treatment using offline reinforcement learning [2025] https://www.nature.com/articles/s41746-025-01498-1
• Personalizing renal replacement therapy initiation in the intensive care unit: a reinforcement learning-based strategy with external validation on the AKIKI randomized controlled trials [2024] https://academic.oup.com/jamia/article/31/5/1074/7624146
• Reinforcement learning model for optimizing dexmedetomidine dosing to prevent delirium in critically ill patients [2024] https://www.nature.com/articles/s41746-024-01335-x
• Optimizing warfarin dosing for patients with atrial fibrillation using machine learning [2024] https://www.nature.com/articles/s41598-024-55110-9
• Development and validation of a reinforcement learning model for ventilation control during emergence from general anesthesia [2023] https://www.nature.com/articles/s41746-023-00893-w
• Optimized glycemic control of t