MIDAS — Microstructural Imaging using Diffraction Analysis Software

MIDAS is an open-source suite for reconstructing three-dimensional microstructures from High-Energy Diffraction Microscopy (HEDM) data. Developed at the Advanced Photon Source at Argonne National Laboratory, it supports the complete data-reduction pipeline — from raw detector frames to grain maps, strain tensors, spatially resolved orientation fields, and tomographic reconstructions.

Version: 11.0 Contact: Hemant Sharma (hsharma@anl.gov)

Key Capabilities

| Technique | What It Produces | Detector Distance | |-----------|-----------------|-------------------| | Far-Field HEDM (FF-HEDM) | Grain centroids, average orientations, full elastic strain tensors | ≈ 1 m | | Near-Field HEDM (NF-HEDM) | Spatially resolved 3D orientation maps, grain morphology, grain boundary networks | ≈ 5–10 mm | | Point-Focus HEDM (PF-HEDM) | High-resolution grain orientations from focused beam | ≈ 1 m | | Radial Integration (Caking) | 1D intensity vs. 2θ profiles for Rietveld refinement (GSAS-II) | — | | Grain Matching & Stitching | Track grains across load states; combine multi-layer scans | — | | Tomography (CT) | Absorption-contrast cross-sections via gridrec algorithm | — |

Version 11 Highlights

GPU acceleration — CUDA-accelerated indexing (IndexerGPU), strain fitting (FitPosOrStrainsGPU), NF orientation fitting (FitOrientationGPU), scanning HEDM (IndexerScanningGPU, FitOrStrainsScanningGPU), radial integration (IntegratorFitPeaksGPUStream), and tomographic reconstruction (tomo_gpu). See GPU_Acceleration
Consolidated binary I/O — PF/scanning HEDM replaces ~30K+ per-voxel files with 3 consolidated binary files per scan via IndexerConsolidatedIO.h
CalibrantIntegratorOMP — new primary calibration executable (replaces archived CalibrantPanelShiftsOMP)
Switchable peak fitting — unified pseudo-Voigt (pV) and Thompson-Cox-Hastings (TCH) modes across calibration, integration, and auto-calibration
Physical corrections — parallax, solid-angle, and polarization corrections in DetectorMapper (matching pyFAI convention)
Q-spacing integration — radial bins equally spaced in Q (Å⁻¹) via QBinSize/QMin/QMax parameters
CBF file format — new ReadCBFFrame() reader for CBF detector data
Streaming median — histogram-based streaming median for NF image processing (~500 MB vs 11.5 GB)
Stripe artifact removal — Vo et al. (2018) algorithms for tomography ring/stripe correction
Shared integration library — extracted IntegrationCore, MapperCore, CalibrationCore, CalibPeakFit modules
Centralized MIDAS_ParamParser — unified parameter file parsing across all executables
Gradient-aware pixel splitting — sub-pixel radial resampling for improved integration accuracy

Version 10 Highlights

Consolidated HDF5 output — all FF-HEDM results (grains, spots, strains, peak provenance) in a single .h5 file
Pipeline restart — --resume and --restart-from flags on all workflows (FF, PF, NF, dual-dataset, multi-resolution) allow resuming from any completed stage using the HDF5 checkpoint
Version & provenance tracking — every C binary and Python workflow embeds MIDAS v11.0 (<git-hash>) into output files, HDF5 attributes, and Zarr metadata for full reproducibility
Pseudo-Voigt peak fitting — two-stage decomposed fitting (Lorentzian + Gaussian) with mu-weighted effective widths
Grain matching & stitching — Python-native optimal (Hungarian) matching across load states with affine deformation support
Reprocess mode — regenerate MergeMap.csv and consolidated HDF5 on old datasets (-reprocess 1)
Dynamic detector sizes — auto-detected from data, no hardcoded dimensions
Zarr-ZIP data format — compressed, portable, self-contained analysis archives
Forward simulation engine — compressed output with OpenMP parallelism
Multi-resolution NF-HEDM — iterative reconstruction at increasing grid resolution
Interactive GUI — browser-based visualization of NF calibration, microstructure, and FF results
Multi-stage auto-calibration — geometry-first calibration strategy with panel auto-detection from masks

Repository Layout

MIDAS/
├── FF_HEDM/          # Far-field HEDM (calibration, indexing, fitting, integration)
│   ├── src/          #   C source code (peak search, fitting, merging, indexing, grains)
│   ├── workflows/    #   Python workflow drivers (ff_MIDAS.py)
│   ├── bin/          #   Compiled binaries (auto-generated by build)
│   └── Example/      #   Example dataset for testing
├── NF_HEDM/          # Near-field HEDM reconstruction
│   ├── src/          #   C source code (forward model, grid reconstruction)
│   ├── workflows/    #   Python workflow drivers (nf_MIDAS.py)
│   ├── bin/          #   Compiled binaries (auto-generated by build)
│   ├── v7/           #   Workflow templates
│   ├── seedOrientations/  # Orientation seed files (downloaded by build)
│   └── Example/      #   Example dataset for testing
├── DT/               # Diffraction tomography (peak-fit integrator)
├── TOMO/             # Tomographic reconstruction (gridrec CT engine)
├── utils/            # Python utilities
│   ├── AutoCalibrateZarr.py   # FF-HEDM auto-calibration
│   ├── match_grains.py        # Grain matching and layer stitching
│   ├── calcMiso.py            # Crystallographic misorientation calculations
│   ├── gsas_ii_refine.py      # GSAS-II integration
│   └── ...
├── gui/              # Interactive visualization GUI
├── manuals/          # Comprehensive documentation (see below)
├── cmake/            # CMake build configuration and dependency management
├── build.sh          # Build script (Linux / macOS)
├── build_wsl_windows.sh  # Build script (Windows via WSL)
├── environment.yml   # Conda environment specification
├── CMakeLists.txt    # Top-level CMake configuration
└── LICENSE           # UChicago Argonne open-source license

Documentation

Full manuals are in the manuals/ directory. Start with the manuals README for an overview of all HEDM techniques, coordinate systems, and a getting-started checklist.

| Manual | Topic | |--------|-------| | GPU_Acceleration | GPU-accelerated computation (CUDA) | | FF_Calibration | FF-HEDM geometry calibration | | FF_Analysis | FF-HEDM grain indexing and fitting | | FF_Match_Stack_Reconstructions | Grain matching across load states and layer stitching | | FF_Radial_Integration | Radial integration / caking | | FF_Interactive_Plotting | Interactive FF-HEDM visualization | | FF_Visualization | FF-HEDM result visualization | | FF_Dual_Datasets | Dual-dataset FF-HEDM analysis | | PF_Analysis | Point-Focus HEDM analysis | | NF_Calibration | NF-HEDM detector calibration | | NF_Analysis | NF-HEDM reconstruction workflow | | NF_MultiResolution_Analysis | Multi-resolution NF-HEDM | | NF_GUI | NF-HEDM interactive GUI | | Forward_Simulation | Forward simulation for validation | | GSAS-II_Integration | Importing MIDAS output into GSAS-II | | Tomography_Reconstruction | Absorption-contrast CT reconstruction | | FF_Benchmark | FF-HEDM benchmark testing | | NF_Benchmark | NF-HEDM benchmark testing |

Installation

Prerequisites

| Platform | Requirements | |----------|-------------| | macOS | Homebrew, LLVM, libomp, GCC, CMake, jemalloc | | Linux | GCC ≥ 9, CMake ≥ 3.16 | | Windows | WSL with Ubuntu |

MIDAS automatically downloads and builds these C/C++ dependencies during compilation: NLOPT, LIBTIFF, FFTW, HDF5, BLOSC, BLOSC-2, ZLIB, LIBZIP.

Clone

git clone https://github.com/marinerhemant/MIDAS.git
cd MIDAS

macOS

Install Homebrew (if not already installed):

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Without sudo access, install to your home directory:

mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/main | tar xz --strip-components 1 -C homebrew
eval "$(homebrew/bin/brew shellenv)"
brew update --force --quiet
chmod -R go-w "$(brew --prefix)/share/zsh"

Add Homebrew to your PATH:

echo 'eval $(/opt/homebrew/bin/brew shellenv)' >> ~/.zshrc
source ~/.zshrc

Install dependencies:

brew install llvm libomp gcc cmake jemalloc

Configure environment variables:

echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
echo 'export LDFLAGS="-L/opt/homebrew/opt/llvm/lib $LDFLAGS"' >> ~/.zshrc
echo 'export CPPFLAGS="-I/opt/homebrew/opt/llvm/include $CPPFLAGS"' >> ~/.zshrc
echo 'export LDFLAGS="-L/opt/homebrew/opt/libomp/lib $LDFLAGS"' >> ~/

MIDAS

Install / Use

README