OmixBench: A Systematic Evaluation Framework for Large Language Models in Multi-omics Analysis <img src="omixbench-logo.svg" alt="omixbench-logo" align="right" height="200" width="180"/>

Overview

The rapid advancement of large language models (LLMs) has opened new possibilities for automating complex analytical workflows in computational biology. However, the absence of standardized evaluation protocols makes it difficult to assess their reliability and select appropriate models for bioinformatics applications. This challenge is particularly acute in multi-omics analysis, where workflows involve intricate multi-step procedures, domain-specific software ecosystems, and diverse data modalities.

OmixBench represents an attempt to address this gap by providing a systematic evaluation framework designed specifically for assessing LLM capabilities in multi-omics analysis. The framework encompasses both theoretical understanding and practical execution, offering a dual-evaluation approach that examines methodological validity alongside real-world implementation reliability.

This repository contains the complete evaluation datasets, computational tools, and analysis scripts developed for this study, which evaluated 89 contemporary language models across bioinformatics tasks spanning ten omics domains. We hope this framework can serve as a foundation for the community to build upon, facilitating more informed decisions about LLM deployment in computational biology research.

Framework Architecture

Dual-Evaluation Strategy

The evaluation framework adopts a two-tiered approach to comprehensively assess LLM performance:

Static Evaluation (1,002 tasks): Assesses theoretical knowledge, methodological awareness, and code generation quality without requiring actual execution. This component examines whether models can articulate biologically appropriate and statistically defensible analytical strategies.

Dynamic Evaluation (405 tasks): Tests real-world implementation by executing generated code in standardized computational environments. This component evaluates whether proposed solutions can withstand practical challenges including software dependencies, version compatibility, and data schema variations.

This dual structure acknowledges that conceptual validity and executable reliability represent correlated yet partially independent capabilities. Some models may demonstrate strong theoretical understanding but struggle with execution details, while others may produce functional code with suboptimal methodological foundations.

Coverage Across Omics Domains

The evaluation spans ten major omics domains, reflecting the diversity of contemporary bioinformatics research:

1. Genomics
2. Transcriptomics
3. Epigenomics
4. Proteomics
5. Metabolomics
6. Single-cell Omics
7. Spatial Omics
8. Microbiomics
9. Pharmacogenomics
10. Multi-omics Integration

Task Complexity Classification

Dynamic evaluation tasks are stratified into three complexity levels based on multiple dimensions including computational requirements, analytical sophistication, and interpretative depth:

• Level 1: Straightforward procedures involving basic statistical operations and standard data manipulations
• Level 2: Multi-step workflows requiring integration of multiple analytical methods
• Level 3: Advanced analyses involving sophisticated algorithms, multi-omics integration, or iterative optimization

Detailed classification criteria are provided in Task Complexity Classification.md.

Key Features

1. Evaluation Framework for Multi-Omics Analysis

Building upon existing benchmarking approaches in bioinformatics, this framework offers a dual-evaluation strategy to assess LLM performance:

• Static evaluation (1,002 tasks): Assesses theoretical knowledge, methodological awareness, and code generation quality without requiring execution. Examines whether models can articulate biologically appropriate and statistically defensible analytical strategies across diverse bioinformatics scenarios.

• Dynamic evaluation (405 tasks): Tests real-world implementation by executing generated code in standardized computational environments. Evaluates whether proposed solutions can withstand practical challenges including software dependencies, version compatibility, and data schema variations.

• Multi-domain coverage: Spans ten major omics domains (genomics, transcriptomics, epigenomics, proteomics, metabolomics, single-cell omics, spatial omics, microbiomics, pharmacogenomics, and multi-omics integration).

• Complexity stratification: Dynamic tasks are classified into three levels based on computational requirements, analytical sophistication, and interpretative depth—from straightforward statistical operations to advanced multi-omics integration workflows.

This dual structure acknowledges that conceptual validity and executable reliability represent correlated yet partially independent capabilities, enabling assessment of whether models are better suited for methodological guidance versus code execution.

2. Open Benchmarking Resources

To support reproducibility and community development, this repository provides:

• Complete task databases:

  • OmixTask1002: Static evaluation tasks with metadata and complexity annotations
  • OmixQA: Dynamic execution tasks with standardized data inputs and validation protocols (data available on Synapse: syn70773121)

• Evaluation frameworks:

  • Automated static evaluation scripts with structured assessment protocols
  • Dynamic execution framework with isolated environments, error handling, and logging mechanisms
  • Visualization tools for result analysis and comparison

• Implementation examples:

  • Documented prompt engineering strategies for bioinformatics applications
  • Step-by-step tutorials for both static and dynamic evaluations
  • Example workflows demonstrating RAG-enhanced ReAct implementation

• Development tools:

  • Three R packages (llmhelper, OmixBenchR, llmflow) with comprehensive documentation
  • Python notebook examples for static evaluation
  • Environment configuration specifications for reproducible execution

All resources are openly available to facilitate community extensions and adaptations for different research contexts.

3. Practical Implementation Tools

Three R packages facilitate both evaluation and application of LLMs in bioinformatics workflows:

• llmhelper: Unified interface for integrating multiple LLM providers
• OmixBenchR: Core benchmarking framework with automated task execution
• llmflow: Implementation of RAG-enhanced ReAct architecture for complex analytical workflows

4. Technical Approaches

Based on systematic error analysis identifying method obsolescence and data schema mismatches as primary failure modes, the framework explores several strategies:

• Dual-evaluation methodology: Treats conceptual validity and executable reliability as partially decoupled capacities, enabling assessment of whether models are better positioned as methodological advisors versus code-executing agents

• Chain-of-thought (CoT) reasoning enhancement: Examines whether reasoning strategies can enable smaller models to approach the performance of larger counterparts within identical model families, as an alternative or complement to parameter scaling

• RAG-enhanced ReAct framework: Adapted specifically for bioinformatics code generation to address knowledge gaps observed in initial evaluations:

  • RAG component: Employs reasoning models to identify task-relevant functions from requirements, then retrieves current documentation with prioritized access to official function documentation over LLM-generated examples
  • Persistent coding sessions: Maintains isolated execution environments to preserve variable states and loaded dependencies across reasoning-action-observation cycles, addressing the limitation where computational context is typically lost between analytical steps
  • Error management: Implements intelligent error escalation and task degradation strategies, providing targeted guidance when systematic error patterns emerge

These approaches represent our exploration of bioinformatics-specific optimization strategies, developed iteratively based on observed failure patterns in benchmark evaluations.

Repository Structure

OmixBench/
├── llmhelper/                          # R package for LLM integration
│   ├── R/                              # Source code
│   ├── man/                            # Function documentation
│   └── DESCRIPTION                     # Package metadata
├── llmhelper_1.0.0.pdf                # Package manual
│
├── OmixBenchR/                        # Core benchmarking framework
│   ├── R/                             # Source code
│   ├── man/                           # Function documentation
│   └── inst/                          # Evaluation prompts
├── OmixBenchR_1.0.0.pdf              # Package manual
│
├── llmflow/                           # RAG-enhanced ReAct framework
│   ├── R/                             # Source code
│   ├── man/                           # Function docu