Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation

📢 News

2026-02-08 Compute more metrics, including BertScore, SemScore, 1/RadCliQ-V1, and RATEScore
2025-05-09 Upload poster
2025-03-16 Release checkpoints for MIMIC-ABN and Two-view CXR
2025-03-01 Upload official code and checkpoints for MIMIC-CXR
2025-03-01 Release generated-radiology-reports — labels = reference reports, report = generated reports

⚙️ Requirements

# create virtual environment
conda create -n mlrg python=3.9.0

# install dependencies
pip install -r requirements.txt

torch==2.3.1+cu118
transformers==4.43.3 (As stated in Issue #4, the transformers version should be maintained to prevent potential problems. Credit goes to @Andy for this clarification)
torchvision==0.18.1+cu118
radgraph==0.0.9

Please refer to requirements.txt for more details.

📦 Checkpoints for our MLRG

| 📊 Dataset | 🪣 Download | 📄 Generated Reports | |------------------|------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| | `MIMIC-CXR` | HuggingFace | GitHub | | `MIMIC-ABN` | HuggingFace | GitHub | | `Two-view CXR` | HuggingFace | GitHub |

📂 Datasets

Medical Images

MIMIC-CXR / MIMIC-ABN — PhysioNet, with data systematically organized under root directories labeled p10 through p19, maintaining consistency with MIMIC-CXR's default configuration.
IU X-ray — NIH, its root directory is the NLMCXR_png.
Two-View CXR — aggregated studies with two views from MIMIC-CXR + IU X-ray (arXiv )

files/
├── p10
    └── p10000032
            └── s50414267
               ├── 02aa804e-bde0afdd-112c0b34-7bc16630-4e384014.jpg
               └── 174413ec-4ec4c1f7-34ea26b7-c5f994f8-79ef1962.jpg
├── p11
├── p12
├── p13
├── p14
├── p15
├── p16
├── p17
├── p18
├── p19
└── NLMCXR_png
   ├── CXR1_1_IM-0001-3001.png
   ├── CXR1_1_IM-0001-4001.png
   └── CXR2_IM-0652-1001.png

Raw Radiology Reports

MIMIC-CXR and MIMIC-ABN: PhysioNet.
Two-view CXR: HuggingFace 🤗.

Reorganization of Raw Radiology Reports

To simplify usage, we have organized multi-view longitudinal data using the study_id. The processed datasets—MIMIC-CXR, MIMIC-ABN, and Two-view CXR—are available on HuggingFace 🤗 (PhysioNet authorization required). Note that the IU X-ray dataset (NLMCXR_png) does not include previous visit data due to the absence of study_id.
MIMIC-CXR: five_work_mimic_cxr_annotation_v1.1.json
MIMIC-ABN: mlrg_mimic_abn_annotation_v1.1.json
Two-view CXR: mlrg_multiview_cxr_annotation_v1.1.json
View Position for all datasets: five_work_mimic_cxr_view_position_v1.1.json

📊 Evaluation using generated radiology reports

def compute_performance_using_generated_reports():
    from tools.metrics.metrics import compute_all_scores, compute_chexbert_details_scores
    mimic_cxr_generated_path = 'generated-radiology-reports/MIMIC-CXR/test_reports_epoch-1_20-10-2024_16-28-28.csv'
    mimic_abn_generated_path = 'generated-radiology-reports/MIMIC-ABN/test_reports_epoch-1_23-10-2024_10-25-20.csv'
    twoview_cxr_generated_path = 'generated-radiology-reports/Two-view CXR/test_reports_epoch-0_25-10-2024_11-38-35.csv'
    args = {
        'chexbert_path': "/home/miao/data/dataset/checkpoints/chexbert.pth",
        'bert_path': "/home/miao/data/dataset/checkpoints/bert-base-uncased",
        'radgraph_path': "/home/miao/data/dataset/checkpoints/radgraph",
    }
    for generated_path in [mimic_cxr_generated_path, mimic_abn_generated_path, twoview_cxr_generated_path]:
        data = pd.read_csv(generated_path)
        gts, gens = data['labels'].tolist(), data['report'].tolist()
        scores = compute_all_scores(gts, gens, args)
        print(scores)

📊 More metrics on the MIMIC-CXR test set

{
    'BertScore': 0.5716221332550049,
    'SemScore': 0.4368664622306824,
    '1/RadCliQ-V1': 1.0102079556023098,
    'RATEScore': 0.5668122046732644,
    'green': 0.353027779,
    'chexbert_5_micro_f1': 0.5503549017590783,
    'chexbert_5_macro_f1': 0.4862237881570195,
    'chexbert_all_micro_p': 0.5489597467209407,
    'chexbert_all_micro_r': 0.467591254935953,
    'chexbert_all_micro_f1': 0.5050189837208093,
    'chexbert_all_macro_p': 0.4399492462801775,
    'chexbert_all_macro_r': 0.354060820803069,
    'chexbert_all_macro_f1': 0.3641635446370755,
    'BLEU_1': 0.41114996799739173, 'BLEU_2': 0.2769778918508422,
    'BLEU_3': 0.20362264525354418, 'BLEU_4': 0.1582088781713785,
    'METEOR': 0.17633810974007486, 'ROUGE_L': 0.3195399064699496, 'CIDer': 0.3599887171235284
}

🚀 Training

1. Download checkpoints for architecture and metrics.

For CE metrics calculation: chexbert.pth, radgraph, and bert-base-uncased.
For model initialization: microsoft/rad-dino (image encoder), microsoft/BiomedVLP-CXR-BERT-specialized (text encoder), distilbert/distilgpt2 (define text generator), and cvt2distilgpt2 (initialize text generator).
Checkpoint directory: Place all checkpoints in a local directory (e.g., "/home/data/checkpoints"), and configure the --ckpt_zoo_dir /home/data/checkpoints argument in the corresponding script/**/**.sh file.

| Checkpoint | Variable name | Download | | ------------------------------------------ | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | chexbert.pth | chexbert_path | StanfordMedicine or HuggingFace | | bert-base-uncased | bert_path | HuggingFace | | radgraph | radgraph_path | PhysioNet | | microsoft/rad-dino | rad_dino_path | HuggingFace

MLRG

Install / Use

README