VMDragonSlayer

Project will be public by mid-October Refactoring In Progress

Advanced Virtual Machine Detection and Analysis Framework

VMDragonSlayer is a comprehensive framework for analyzing binaries protected by Virtual Machine (VM) based protectors such as VMProtect 2.x/3.x, Themida, and custom malware VMs. The framework combines multiple analysis engines including Dynamic Taint Tracking (DTT), Symbolic Execution (SE), Pattern Classification, and Machine Learning to automate the reverse engineering process.

Goal: Transform complex protected binary analysis from weeks/months of manual work into structured, automated analysis with explainable results.

Key Features

• Multi-Engine Analysis: Combines static, dynamic, and hybrid analysis techniques
• VM Detection: Automated detection of commercial and custom VM protectors
• Plugin Ecosystem: Integrations with Ghidra, IDA Pro, and Binary Ninja
• Machine Learning: Proof-of-concept ML models for pattern classification
• Extensible Architecture: Modular design for custom analysis workflows
• Research Framework: Built for malware research and reverse engineering education

---

Core Capabilities

| Domain | Engine / Module | Highlights | |--------|-----------------|-----------| | VM Discovery | analysis.vm_discovery | Dispatcher & handler table identification, nested VM heuristics | | Pattern Analysis | analysis.pattern_analysis | Rule-based + similarity + ML (hybrid auto-selection) | | Taint Tracking | analysis.taint_tracking | Intel Pin–driven byte-level taint, handler discovery, flow confidence | | Symbolic Execution | analysis.symbolic_execution.executor | PathPrioritizer ML-weighted exploration, constraint & state tracking | | Hybrid Orchestration | (Python core) | Sequential / parallel / adaptive workflows (Ghidra report indicates implemented) | | Synthetic Data | data/training/synthetic_sample_generator.py | Obfuscation mutation, multi-architecture sample generation | | Pattern DB | data/patterns/ | JSON + enhanced DB + SQLite-backed runtime patterns | | Ghidra Plugin | plugins/ghidra/ | In-progress UI integration (several templates missing) | | Schemas / Validation | data/schemas/ | JSON schema–validated analysis output & pattern formats |

---

Architecture Overview

VMDragonSlayer uses a modular architecture where multiple analysis engines work together:

graph TD
    A[VM Discovery Engine] --> B[Pattern/ML Classifier]
    B --> C[Symbolic Execution Engine]
    B --> D[Dynamic Taint Tracker]
    D --> C
    
    subgraph DataSources ["Data Sources"]
        E[Pattern Database]
        F[ML Models - PoC]
    end
    
    subgraph Coordination
        G[Orchestrator - Workflow Management & Coordination]
    end
    
    E --> B
    F --> B
    A --> G
    B --> G
    C --> G
    D --> G
    E --> G
    F --> G
    G --> H[REST API Server]
    G --> I[Plugins - RE Tools]

Core Analysis Engines

1. VM Discovery Engine (dragonslayer.analysis.vm_discovery)

• Purpose: Detect and classify VM-based protection schemes
• Techniques: Dispatcher loop detection, handler table analysis, control flow heuristics
• Targets: VMProtect, Themida, custom malware VMs, nested protection

2. Dynamic Taint Tracking (dragonslayer.analysis.taint_tracking)

• Purpose: Track data flow through VM execution to identify critical paths
• Implementation: Intel Pin-based instrumentation with byte-level precision
• Features: Shadow memory, anti-analysis evasion, automated handler discovery

3. Pattern Analysis (dragonslayer.analysis.pattern_analysis)

• Purpose: Classify and categorize VM patterns and behaviors
• Methods: Rule-based matching, similarity analysis, ML classification
• Database: Extensible pattern database with JSON schemas

4. Symbolic Execution (dragonslayer.analysis.symbolic_execution)

• Purpose: Explore VM execution paths symbolically
• Features: Constraint solving, path prioritization, state merging
• Integration: Uses taint analysis results to seed exploration

5. Machine Learning Pipeline (dragonslayer.ml)

• Purpose: Automated classification and analysis assistance
• Models: Basic proof-of-concept models for research and education
• Components: Feature extraction, model training, ensemble prediction

Repository Structure

VMDragonSlayer/
├── dragonslayer/                    # Main Python package
│   ├── analysis/                   # Analysis engines
│   │   ├── vm_discovery/          # VM detection and classification
│   │   ├── pattern_analysis/      # Pattern matching and ML classification
│   │   ├── symbolic_execution/    # Symbolic execution engine
│   │   ├── taint_tracking/        # Dynamic taint analysis
│   │   └── anti_evasion/          # Anti-analysis countermeasures
│   ├── api/                       # REST API server and client
│   ├── core/                      # Core framework components
│   ├── ml/                        # Machine learning pipeline
│   ├── analytics/                 # Analysis reporting and metrics
│   ├── gpu/                       # GPU acceleration support
│   ├── utils/                     # Utility functions
├── data/                          # Configuration and data files
│   ├── patterns/                  # Pattern database
│   ├── models/                    # ML models and metadata
│   │   ├── pretrained/           # Pre-trained models (PoC)
│   │   └── metadata/             # Model metadata and schemas
│   ├── samples/                   # Sample files and registries
│   ├── schemas/                   # JSON schemas for validation
│   └── training/                  # Training configurations
├── plugins/                       # Reverse engineering tool plugins
│   ├── ghidra/                   # Ghidra plugin (Java/Gradle)
│   ├── idapro/                   # IDA Pro plugin (Python)
│   └── binaryninja/              # Binary Ninja plugin (Python)
├── tests/                         # Tests suite
├── documentation/                 # Documentation
└── LICENSE                        # GPL v3 License

Plugin Ecosystem

VMDragonSlayer integrates with major reverse engineering tools:

Ghidra Plugin

• Language: Java with Gradle build system
• Features: VM analysis UI, pattern visualization, automated analysis workflows
• Status: Framework implemented, UI components in development

IDA Pro Plugin

• Language: Python
• Features: Seamless integration with IDA's analysis engine
• Status: Core functionality available

Binary Ninja Plugin

• Language: Python
• Features: Native Binary Ninja API integration
• Status: Basic integration implemented

Machine Learning Components

Note: The included ML models are basic proof-of-concept implementations designed for research and educational purposes.

---

Installation

Prerequisites

• Python 3.8 or higher
• One or more reverse engineering tools:
  • Ghidra 10.0+ (for Ghidra plugin)
  • IDA Pro 7.0+ (for IDA plugin)
  • Binary Ninja (for Binary Ninja plugin)

Hardware Requirements

Minimum Requirements

• CPU: Modern x64 processor
• RAM: 8GB minimum, 16GB recommended
• Storage: 5GB free space
• OS: Windows 10/11, Linux (Ubuntu 20.04+), macOS 11+

GPU Requirements (Optional but Recommended)

• NVIDIA GPU: GTX 1060 or newer for optimal performance
• CUDA: Version 11.8 or 12.1+ (installed automatically with PyTorch)
• VRAM: 4GB minimum for ML models

Important Limitations

• Virtual Machines: GPU-accelerated features require direct hardware access and may not work in VMs
• WSL: Some GPU features may have limited functionality in WSL environments
• Remote Servers: Ensure CUDA drivers are properly installed for headless GPU access

Current Status (Fixed Issues)

Verified Working Examples

# This now works perfectly:
from dragonslayer.core.orchestrator import Orchestrator, AnalysisType

orchestrator = Orchestrator()
result = orchestrator.analyze_binary("your_binary.exe", analysis_type=AnalysisType.VM_DISCOVERY)

# Results are properly structured:
vmd = result.get("vm_discovery", {})
print(f"VM detected: {vmd.get('vm_detected', False)}")
print(f"Handlers found: {len(vmd.get('handlers_found', []))}")

Quick Start

# 1. Install with all dependencies
pip install -r requirements.txt
pip install -e .

Core Framework

# Clone repository
git clone https://github.com/poppopjmp/VMDragonSlayer.git
cd VMDragonSlayer

# UPDATED INSTALLATION
# Install all required dependencies including z3-solver
pip install -r requirements.txt

# Install framework in development mode
pip install -e .

Installation for Different Hardware

# CPU-only installation (basic functionality)
pip install -r requirements.txt
pip install -e .

# NVIDIA GPU with CUDA 12.x (RTX 30xx/40xx series) 
pip install -r requirements.txt
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install -e .

# NVIDIA GPU with CUDA 11.8 (older GPUs)
pip install -r requirements.txt  
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -e .

Create virtual environment

python -m venv venv source venv/bin/activate # Linux/macOS

or

venv\Scripts\activate # Windows

Install dependencies

pip install -r requirements.txt

Install framework

cd dragonslayer pip install -e .

Plugin Installation

Choose your preferred disassembler:

Ghidra Plugin

cd plugins/ghidra
./build.bat  # Windows
# or
./build.sh   # Linux/macOS

# Install to Ghidra
cp dist/VMDragonSlayer.zip $GHIDRA_INSTALL_DIR/Extensions/

---

Quick Start

1. Basic Framework Usage

from dragonslayer.core.orchestrator import Orchestrator, AnalysisType

# Initialize orchestrator (loads default con