<div align="center">

TickVault 🏦

A high-performance Python library for downloading, storing, and accessing financial tick data from Dukascopy Bank's historical datafeed. Built for quantitative researchers and algorithmic traders who need reliable access to high-resolution market data.

</div>

---

✨ Highlights

Two clean layers

• Downloader: concurrent, fault-tolerant fetchers that mirror Dukascopy’s on-disk layout 1:1.
• Reader: decodes compressed hourly blobs on demand and returns tidy pandas DataFrames.

High-Performance Downloading

• Concurrent Downloads: Multi-worker architecture with configurable parallelism
• Resume Capability: Intelligent resume functionality with metadata tracking
• Proxy Support: Distributed downloading across multiple proxies
• Error Resilience: Comprehensive retry logic with exponential backoff
• Progress Tracking: Real-time progress monitoring with beautiful progress bars

Smart Storage

• Mirrored Structure: Local filesystem mirrors Dukascopy's directory hierarchy
• Compression-Aware: Stores data in original compressed .bi5 format
• Metadata Tracking: SQLite database tracks download status and data availability
• Space Efficient: 10-15GB for assets with decades of history (like Gold)

Powerful Data Access

• Fast Decoding: Efficient LZMA decompression and NumPy-based decoding
• Pandas Integration: Returns clean, structured DataFrames ready for analysis
• Gap Detection: Automatic data continuity verification
• Flexible Queries: Easy time-range based data retrieval

Production Ready

• Comprehensive Logging: Structured logging with configurable levels
• Configuration Management: Environment-based configuration with validation
• Type Safety: Full type annotations with Pydantic models
• Error Handling: Graceful handling of network issues and data gaps

Modern Python

• Tested on Python 3.14. Works with 3.11–3.14.
• Async I/O (httpx), Pydantic v2 settings, tqdm, NumPy, pandas.

Supported Assets

All dukascopy assets are supported for downloading. Since price scales differ between assets(each have their own pipet scale), you will need the scaling factor to correctly scale the price values. I have added some of the popular asset's price scales to the registry for convenience, and will add more as we go:

• Forex Majors: EURUSD, GBPUSD, USDJPY, AUDUSD, USDCAD, USDCHF, NZDUSD
• Precious Metals: XAUUSD (Gold), XAGUSD (Silver)
• Cryptocurrencies: BTCUSD, ETHUSD

For other assets you should identify the price scale manually for reading. Think of price scale as the lowest resolution of price for that asset. For example, the pipet size/price resolution for "XAUUSD" is "0.001".

---

🚀 Quick Start

Installation

pip install tick-vault

Or install from source:

1. Clone the repo:

git clone https://github.com/keyhankamyar/TickVault.git
cd TickVault

2. Create a custom environment:

python -m venv .venv
source .venv/bin/activate

# Or use conda
conda create --prefix .conda python -y
conda activate ./.conda

3. Install the package:

# Install dependencies
pip install -r requirements.txt

# Or install in development mode
pip install -e .

Basic Usage

1) Download Historical Data (resumable)

from datetime import datetime
from tick_vault import download_range, reload_config

# Optional: configure base directory, worker counts, etc.
reload_config(
    base_directory="./tick_vault_data",
    worker_per_proxy=10,                 # default 10
    fetch_max_retry_attempts=3,          # default 3
    fetch_base_retry_delay=1.0,          # default 1.0s
)

# Download one month of gold (XAU/USD) tick data
await download_range(
    symbol='XAUUSD',
    start=datetime(2024, 1, 1),
    end=datetime(2024, 2, 1)
)

# Download with multiple proxies for faster speeds
await download_range(
    symbol='EURUSD',
    start=datetime(2024, 1, 1),
    end=datetime.now(),
    proxies=[
        'http://proxy1.example.com:8080',
        'http://proxy2.example.com:8080'
    ]
)

Re-running the same call later will resume and only attempt hours not yet recorded in the metadata database.

2) Read and Analyze Data (on-demand decode → DataFrame)

from tick_vault import read_tick_data

# Read all available data for a symbol
df = read_tick_data(symbol='XAUUSD')

# Read specific date range
df = read_tick_data(
    symbol='EURUSD',
    start=datetime(2024, 1, 1),
    end=datetime(2024, 2, 1)
)

# Validation and progressbar
df = read_tick_data(
    symbol="XAUUSD",
    strict=True,         # verify requested range is fully present (raises on gaps)
    show_progress=True,  # tqdm while decoding hourly chunks
)

print(df.head())
print(f"Total ticks: {len(df)}")
print(f"Time range: {df['time'].min()} to {df['time'].max()}")
# columns: time, ask, bid, ask_volume, bid_volume

3) Configuration

from tick_vault import reload_config

# Customize settings programmatically
reload_config(
    base_directory='./my_tick_data',
    worker_per_proxy=15,
    fetch_max_retry_attempts=5
)

Or use environment variables

import os

os.environ['TICK_VAULT_BASE_DIRECTORY'] = './my_tick_data'
os.environ['TICK_VAULT_WORKER_PER_PROXY'] = 15

---

🏗️ Architecture

Download Pipeline

flowchart TB
    subgraph Orchestrator["Orchestrator (download_range)"]
        A[Generate Hourly Chunks] --> B[Distribute Across Proxies]
        B --> C[Manage Progress & Queues]
    end

    subgraph Workers["Download Workers (Parallel)"]
        D1[Worker 1<br/>Proxy A] 
        D2[Worker 2<br/>Proxy A]
        D3[Worker N<br/>Proxy B]
    end

    subgraph Fetching["HTTP Layer"]
        E[Fetch with Retry<br/>Exponential Backoff]
        E --> F{Response}
        F -->|200 OK| G[Decompress .bi5]
        F -->|404| H[No Data]
        F -->|429/503| I[Rate Limited]
        F -->|401/403| J[Forbidden]
        I -->|Retry After| E
        J -->|Fatal Error| K[Abort]
    end

    subgraph Storage["Storage"]
        L[(SQLite DB<br/>metadata.db)]
        M[Local Filesystem<br/>downloads/SYMBOL/YYYY/MM/DD/HHh_ticks.bi5]
    end

    subgraph Metadata["Metadata Worker"]
        N[Batch Accumulator]
        O[Batch Insert/Update]
    end

    C -->|Chunks| D1 & D2 & D3
    D1 & D2 & D3 -->|URL| E
    G -->|Binary Data| M
    H -->|Mark No Data| N
    G -->|Success| N
    N -->|100 chunks or timeout| O
    O -->|Update Status| L

    style A fill:#4CAF50,color:#fff
    style E fill:#2196F3,color:#fff
    style L fill:#FF9800,color:#fff
    style M fill:#9C27B0,color:#fff
    style N fill:#00BCD4,color:#fff
    style K fill:#f44336,color:#fff

Design Principles:

• Multiple download workers for parallelism
• Single metadata worker to avoid database contention
• Queue-based communication for clean separation
• Batch database writes for efficiency

Storage layout (mirrors source 1:1)

TickVault mirrors Dukascopy's URL structure on your local filesystem:

tick_vault_data/
├── downloads/
│   └── XAUUSD/
│       └── 2024/
│           └── 02/          # Month, 0-indexed(00=Jan ... 11=Dec)
│               └── 15/      # Day
│                   ├── 00h_ticks.bi5
│                   ├── 01h_ticks.bi5
│                   └── ...
├── metadata.db              # SQLite database tracking download status
└── logs.log                 # Detailed operation logs

Data Flow

1. Orchestrator generates hourly chunks and manages worker distribution
2. Download Workers fetch compressed .bi5 files concurrently
3. Metadata Worker batches updates to track download status
4. Reader decompresses and decodes data into structured arrays
5. Pandas provides the final DataFrame interface

Project Structure

TickVault/
├── .gitignore
├── pyproject.toml
├── requirements.txt
├── README.md
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
└── tick_vault/
    ├── __init__.py           # Public API: download_range, read_tick_data, reload_config
    ├── py.typed              # Type hints marker for mypy
    ├── constants.py          # Base URL, pipette scales
    ├── config.py             # Pydantic settings + computed paths
    ├── logger.py             # Centralized logging (console + file)
    ├── utils.py              # Date/hour generators, path formatting (0-index month)
    ├── chunk.py              # TickChunk model (url/path/save/load)
    ├── fetcher.py            # HTTP fetch with strong error taxonomy + retry
    ├── download_worker.py    # Async worker: get → save → emit
    ├── metadata.py           # SQLite DB: tracking, gap checks, available hours
    ├── metadata_worker.py    # Single writer, batched inserts
    ├── decoder.py            # LZMA decode → structured NumPy
    ├── downloader.py         # Download orchestrator
    └── reader.py             # Orchestrates decoding → pandas DataFrame

---

📖 Detailed Usage

Supported Assets

TickVault includes pre-configured pipet scales for common assets:

Forex Majors: EURUSD, AUDUSD, GBPUSD, NZDUSD, USDCAD, USDCHF, USDJPY
Metals: XAUUSD (Gold), XAGUSD (Silver)
Crypto: BTCUSD, ETHUSD

For other assets, provide the pipet_scale parameter:

df = read_tick_data(
    symbol='CUSTOM_PAIR',
    start=datetime(2024, 1, 1),
    end=datetime(2024, 2, 1),
    pipet_scale=0.01  # Custom scaling factor
)

Resumable Downloads

TickVault automatically tracks download progress in a SQLite