English | 中文 | Deutsch

SimTradeData - Quantitative Trading Data Downloader

BaoStock + Mootdx + EastMoney + yfinance Multi-Source | China A-Shares + US Stocks | PTrade Compatible | DuckDB + Parquet Storage

SimTradeData is an efficient data download tool designed for SimTradeLab. It supports China A-shares (BaoStock, Mootdx, EastMoney) and US stocks (yfinance) from multiple data sources, automatically orchestrating each source's strengths. Data is stored in DuckDB as intermediate storage and exported to Parquet format, with efficient incremental updates and querying.

---

<div align="center">

Recommended Combo: SimTradeData + SimTradeLab

Fully PTrade Compatible | A-Shares + US Stocks | 20x+ Backtesting Speedup

No PTrade Strategy Code Changes Needed | Ultra-Fast Local Backtesting | Zero-Cost Solution

</div>

---

Key Features

Efficient Storage Architecture

• DuckDB Intermediate Storage: High-performance columnar database with SQL queries and incremental updates
• Parquet Export Format: Highly compressed, cross-platform compatible, ideal for large-scale data analysis
• Automatic Incremental Updates: Intelligently detects existing data, only downloads new records

Comprehensive Data Coverage

• Market Data: OHLCV daily bars with limit-up/down prices and previous close
• Valuation Metrics: PE/PB/PS/PCF/Turnover Rate/Total Shares/Float Shares
• Financial Data: 23 quarterly financial indicators + automatic TTM calculation
• Corporate Actions: Dividends, bonus shares, rights offerings (with forward adjustment factors)
• Metadata: Stock info, trading calendar, index constituents, ST/suspension status
• US Stock Support: 6,000+ US common stocks, S&P 500 / NASDAQ-100 index constituents

Data Quality Assurance

• Auto-Validation: Data integrity validation before writes
• Export-Time Calculation: Limit prices, TTM metrics computed at export for consistency
• Detailed Logging: Comprehensive error logs and warnings

Generated Data Structure

data/
├── cn.duckdb          # DuckDB database - A-shares (download source)
├── us.duckdb             # DuckDB database - US stocks (download source)
└── export/                      # Exported Parquet files (by market)
    ├── cn/                      # A-shares export
    │   ├── stocks/              # Daily bars (one file per stock)
    │   │   ├── 000001.SZ.parquet
    │   │   └── 600519.SS.parquet
    │   ├── exrights/            # Corporate action events
    │   ├── fundamentals/        # Quarterly financials (with TTM)
    │   ├── valuation/           # Valuation metrics (daily)
    │   ├── metadata/            # Metadata
    │   └── manifest.json
    └── us/                      # US stocks export
        ├── stocks/
        │   ├── AAPL.US.parquet
        │   └── MSFT.US.parquet
        ├── exrights/
        ├── fundamentals/
        ├── valuation/
        ├── metadata/
        └── manifest.json

Prerequisites

• Python: 3.10 or higher
• Poetry: Installation guide
• Network: Required for downloading data from BaoStock/Mootdx/EastMoney/yfinance (China mainland network recommended for A-share data)

Quick Start

Option 1: Download Pre-Built Data (Recommended)

Download the latest data from Releases:

• A-shares: data-cn-v* release → extract to data/cn/
• US stocks: data-us-v* release → extract to data/us/

# A-shares
mkdir -p /path/to/SimTradeLab/data/cn
tar -xzf simtradelab-data-cn-*.tar.gz -C /path/to/SimTradeLab/data/cn/

# US stocks
mkdir -p /path/to/SimTradeLab/data/us
tar -xzf simtradelab-data-us-*.tar.gz -C /path/to/SimTradeLab/data/us/

Option 2: Download Data Yourself

1. Install Dependencies

# Clone the project
git clone https://github.com/kay-ou/SimTradeData.git
cd SimTradeData

# Install dependencies
poetry install

# Activate virtual environment
poetry shell

2. Download Data

Recommended: Unified Download Command

A single command downloads all data, using the TDX daily package for the fast OHLCV path and then orchestrating Mootdx and BaoStock for their respective strengths:

# Daily/production CN refresh (recommended)
# TDX: OHLCV full history and daily package import
# Mootdx: corporate actions, bulk financials, trading calendar, benchmark index
# BaoStock: valuation metrics, ST/suspension status, index constituents
poetry run python scripts/download.py --tdx-download --skip-mootdx-ohlcv

# Fallback full orchestration without the TDX package
poetry run python scripts/download.py

# Use an already-downloaded TDX ZIP file
poetry run python scripts/download.py --tdx-source data/downloads/hsjday.zip --skip-mootdx-ohlcv

# Check data status
poetry run python scripts/download.py --status

# Skip financial data (faster)
poetry run python scripts/download.py --tdx-download --skip-mootdx-ohlcv --skip-fundamentals

# Run Mootdx phase only
poetry run python scripts/download.py --source mootdx

# Run BaoStock phase only
poetry run python scripts/download.py --source baostock

Data Source Division of Labor

| Data Type | Source | Reason | |-----------|--------|--------| | OHLCV Market Data (daily/full history) | TDX Daily Package | Fastest, ~500MB bulk import of full history and latest daily data | | OHLCV Market Data (fallback/backfill) | Mootdx | Per-symbol fallback when the TDX package is unavailable | | Corporate Actions (XDXR) | Mootdx | More complete data | | Bulk Financial Data | Mootdx | One ZIP = all stocks, far better than per-stock queries | | Valuation PE/PB/PS/Turnover | BaoStock | Exclusive data | | ST/Suspension Status | BaoStock | Exclusive data | | Index Constituents | BaoStock | Exclusive data | | Trading Calendar | Mootdx | Provides exchange calendar data | | Benchmark Index | Mootdx | Provides benchmark series |

Using Individual Data Sources

# BaoStock (includes valuation data, but slower)
poetry run python scripts/download_efficient.py
poetry run python scripts/download_efficient.py --skip-fundamentals
poetry run python scripts/download_efficient.py --valuation-only  # Valuation + status only

# Mootdx (faster, but no valuation data)
poetry run python scripts/download_mootdx.py
poetry run python scripts/download_mootdx.py --skip-fundamentals

EastMoney Complementary Data (Money Flow, Dragon Tiger Board, Margin Trading)

# Download last 30 days of complementary data (requires existing market data)
poetry run python scripts/download_daily_extras.py

# Specify number of days (LHB API only retains ~30 days, run regularly)
poetry run python scripts/download_daily_extras.py --days 7

US Stock Data (yfinance)

Free US stock data via yfinance, no API key required:

# Full download (6,000+ US stocks with OHLCV + financials + valuation + metadata)
poetry run python scripts/download_us.py

# Specific symbols (small-scale testing)
poetry run python scripts/download_us.py --symbols AAPL,MSFT,GOOGL

# Market data only (skip time-consuming per-stock financials and metadata)
poetry run python scripts/download_us.py --skip-fundamentals --skip-metadata

# Specify start date
poetry run python scripts/download_us.py --start-date 2020-01-01

US stock ticker format: AAPL.US (consistent with A-shares 600000.SS using {code}.{market}), stored in a separate database data/us.duckdb.

TDX Official Data Package (Fastest Way to Get Full Historical Data)

# Auto-download official TDX Shanghai/Shenzhen/Beijing daily data package (~500MB)
poetry run python scripts/download_tdx_day.py

# Force re-download
poetry run python scripts/download_tdx_day.py --force-download

# Use an already-downloaded file
poetry run python scripts/download_tdx_day.py --file hsjday.zip

3. Export to Parquet

# Export A-shares → data/export/cn/
poetry run python scripts/export_parquet.py

# Export US stocks → data/export/us/
poetry run python scripts/export_parquet.py --market us

# Custom output directory
poetry run python scripts/export_parquet.py --market cn --output /custom/path

4. Publish Data (Maintainer)

The same DuckDB snapshot is exported to Parquet for both distribution channels, but the published artifacts differ:

• GitHub Releases publishes full baseline archives only.
• Tencent COS publishes a full baseline plus an incremental archive when the previous COS version can be advanced safely. Its releases.json index lets clients discover both.

# Release A-shares data
bash scripts/release_data.sh --market cn

# Release US stock data
bash scripts/release_data.sh --market us

# The release version comes from data/export/<market>/manifest.json

# Publish a baseline and, when possible, a delta to Tencent COS
COS_SECRET_ID=... COS_SECRET_KEY=... bash scripts/release_data.sh \
  --market cn --publish-targets cos --cos-bucket BUCKET --cos-region RE