DataMate All-in-One Data Work Platform

<div align="center">

DataMate is an enterprise-level data processing platform for model fine-tuning and RAG retrieval, supporting core functions such as data collection, data management, operator marketplace, data cleaning, data synthesis, data annotation, data evaluation, and knowledge generation.

简体中文 | English

If you like this project, please give it a Star⭐️!

</div>

🌟 Core Features

• Core Modules: Data Collection, Data Management, Operator Marketplace, Data Cleaning, Data Synthesis, Data Annotation, Data Evaluation, Knowledge Generation.
• Visual Orchestration: Drag-and-drop data processing workflow design.
• Operator Ecosystem: Rich built-in operators and support for custom operators.

🚀 Quick Start

Prerequisites

• Git (for pulling source code)
• Make (for building and installing)
• Docker (for building images and deploying services)
• Docker-Compose (for service deployment - Docker method)
• Kubernetes (for service deployment - k8s method)
• Helm (for service deployment - k8s method)

Docker Quick deploy

wget -qO docker-compose.yml https://raw.githubusercontent.com/ModelEngine-Group/DataMate/refs/heads/main/deployment/docker/datamate/docker-compose.yml \
 && REGISTRY=ghcr.io/modelengine-group/ docker compose up -d

Clone the Code

git clone git@github.com:ModelEngine-Group/DataMate.git
cd DataMate

Deploy the basic services

make install

This project supports deployment via two methods: docker-compose and helm. After executing the command, please enter the corresponding number for the deployment method. The command echo is as follows:

Choose a deployment method:
1. Docker/Docker-Compose
2. Kubernetes/Helm
Enter choice:

If the machine you are using does not have make installed, please run the following command to deploy it:

REGISTRY=ghcr.io/modelengine-group/ docker compose -f deployment/docker/datamate/docker-compose.yml --profile milvus up -d

Once the container is running, access http://localhost:30000 in a browser to view the front-end interface.

To list all available Make targets, flags and help text, run:

make help

If you are in an offline environment, you can run the following command to download all dependent images:

make download

Deploy Label Studio as an annotation tool

make install-label-studio

Build and deploy Mineru Enhanced PDF Processing

make build-mineru
make install-mineru

Deploy the DeerFlow service

make install-deer-flow

Local Development and Deployment

After modifying the local code, please execute the following commands to build the image and deploy using the local image.

make build
make install dev=true

Uninstall

make uninstall

When running make uninstall, the installer will prompt once whether to delete volumes; that single choice is applied to all components. The uninstall order is: milvus -> label-studio -> datamate, which ensures the datamate network is removed cleanly after services that use it have stopped.

📚 Documentation

Core Documentation

• DEVELOPMENT.md - Local development environment setup and workflow
• AGENTS.md - AI assistant guidelines and code style

Backend Documentation

• backend/README.md - Backend architecture, services, and technology stack
• backend/api-gateway/README.md - API Gateway configuration and routing
• backend/services/main-application/README.md - Main application modules
• backend/shared/README.md - Shared libraries (domain-common, security-common)

Runtime Documentation

• runtime/README.md - Runtime architecture and components
• runtime/datamate-python/README.md - FastAPI backend service
• runtime/python-executor/README.md - Ray executor framework
• runtime/ops/README.md - Operator ecosystem
• runtime/datax/README.md - DataX data framework
• runtime/deer-flow/README.md - DeerFlow LLM service

Frontend Documentation

• frontend/README.md - React frontend application

🤝 Contribution Guidelines

Thank you for your interest in this project! We warmly welcome contributions from the community. Whether it's submitting bug reports, suggesting new features, or directly participating in code development, all forms of help make a project better.

• 📮 GitHub Issues: Submit bugs or feature suggestions.

• 🔧 GitHub Pull Requests: Contribute code improvements.

📄 License

DataMate is open source under the MIT license. You are free to use, modify, and distribute the code of this project in compliance with the license terms.