Dingo
Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
Install / Use
/learn @MigoXLab/DingoREADME
Introduction
Dingo is A Comprehensive AI Data, Model and Application Quality Evaluation Tool, designed for ML practitioners, data engineers, and AI researchers. It helps you systematically assess and improve the quality of training data, fine-tuning datasets, and production AI systems.
🚀 Enterprise Dingo SaaS Version
Need a production-grade data quality platform? Try Dingo SaaS Enterprise Edition!
✨ Compared to the open-source version, SaaS provides:
- 🌐 Web UI - Visual evaluation interface, no coding required
- 🔐 Access Control - JWT + Google OAuth 2.0
- 📊 Visual Reports - Interactive charts, trend analysis, export features
- 🔌 RESTful API - Seamless integration with existing systems
📝 How to Get Free SaaS Code
👉 Apply for Dingo SaaS Repository Access
Review time: 1-5 business days | Suitable for enterprise data governance, team collaboration
Why Dingo?
🎯 Production-Grade Quality Checks - From pre-training datasets to RAG systems, ensure your AI gets high-quality data
🗄️ Multi-Source Data Integration - Seamlessly connect to Local files, SQL databases (PostgreSQL/MySQL/SQLite), HuggingFace datasets, and S3 storage
🔍 Multi-Field Evaluation - Apply different quality rules to different fields in parallel (e.g., ISBN validation for isbn, text quality for title)
🤖 RAG System Assessment - Comprehensive evaluation of retrieval and generation quality with 5 academic-backed metrics
🧠 LLM & Rule & Agent Hybrid - Combine fast heuristic rules (30+ built-in) with LLM-based deep assessment
🚀 Flexible Execution - Run locally for rapid iteration or scale with Spark for billion-scale datasets
📊 Rich Reporting - Detailed quality reports with GUI visualization and field-level insights
Architecture Diagram
Quick Start
Installation
# Core package (includes rule evaluation, LLM evaluation, MCP server, datasource support)
pip install dingo-python
# With HHEM hallucination detection model (requires transformers + torch)
pip install "dingo-python[hhem]"
# With all features (HHEM + Agent)
pip install "dingo-python[all]"
Example Use Cases of Dingo
1. Evaluate LLM chat data
from dingo.config.input_args import EvaluatorLLMArgs
from dingo.io.input import Data
from dingo.model.llm.text_quality.llm_text_quality_v4 import LLMTextQualityV4
from dingo.model.rule.rule_common import RuleSpecialCharacter
data = Data(
data_id='123',
prompt="hello, introduce the world",
content="�I am 8 years old. ^I love apple because:"
)
def llm():
LLMTextQualityV4.dynamic_config = EvaluatorLLMArgs(
key='YOUR_API_KEY',
api_url='https://api.openai.com/v1/chat/completions',
model='gpt-4o',
)
res = LLMTextQualityV4.eval(data)
print(res)
def rule():
res = RuleSpecialCharacter().eval(data)
print(res)
rule()
2. Evaluate Dataset
from dingo.config import InputArgs
from dingo.exec import Executor
# Evaluate a dataset from Hugging Face
if __name__ == '__main__':
input_data = {
"input_path": "tatsu-lab/alpaca", # Dataset from Hugging Face
"dataset": {
"source": "hugging_face",
"format": "plaintext" # Format: plaintext
},
"executor": {
"result_save": {
"bad": True # Save evaluation results
}
},
"evaluator": [
{
"evals": [
{"name": "RuleColonEnd"},
{"name": "RuleSpecialCharacter"}
]
}
]
}
input_args = InputArgs(**input_data)
executor = Executor.exec_map["local"](input_args)
result = executor.execute()
print(result)
Command Line Interface
Evaluate with Rule Sets
dingo eval --input .github/env/local_plaintext.json
Evaluate with LLM (e.g., GPT-4o)
dingo eval --input .github/env/local_json.json
MCP Server
Dingo includes a built-in Model Context Protocol (MCP) server for AI agent integration:
# Start MCP server (SSE transport, default port 8000)
dingo serve
# Custom port
dingo serve --port 9000
# stdio transport (for Claude Desktop)
dingo serve --transport stdio
For detailed setup and Cursor/Claude Desktop integration, see the dedicated documentation:
Video Demonstration
To help you get started quickly with Dingo MCP, we've created a video walkthrough:
https://github.com/user-attachments/assets/aca26f4c-3f2e-445e-9ef9-9331c4d7a37b
This video demonstrates step-by-step how to use Dingo MCP server with Cursor.
📚 Data Quality Metrics
Dingo provides 100+ evaluation metrics across multiple dimensions, combining rule-based speed with LLM-based depth.
Metric Categories
| Category | Examples | Use Case | |----------|----------|----------| | Pretrain Text Quality | Completeness, Effectiveness, Similarity, Security | LLM pre-training data filtering | | SFT Data Quality | Honest, Helpful, Harmless (3H) | Instruction fine-tuning data | | RAG Evaluation | Faithfulness, Context Precision, Answer Relevancy | RAG system assessment | | Hallucination Detection | HHEM-2.1-Open, Factuality Check | Production AI reliability | | Classification | Topic categorization, Content labeling | Data organization | | Multimodal | Image-text relevance, VLM quality, OCR visual evaluation | Vision-language data | | Security | PII detection, Perspective API toxicity | Privacy and safety |
📊 View Complete Metrics Documentation →
📖 RAG Evaluation Guide → | 中文版
🔍 Hallucination Detection Guide → | 中文版
✅ Factuality Assessment Guide → | 中文版
👁️ VLM Render Judge Guide → | 中文版
Most metrics are backed by academic research to ensure scientific rigor.
Quick Metric Usage
llm_config = {
"model": "gpt-4o",
"key": "YOUR_API_KEY",
"api_
