Go2NetSpectra

📖 Overview

Go2NetSpectra is a high-performance, distributed network traffic monitoring and analysis framework written in Go. It provides a powerful platform for network engineers, security analysts, and SREs to gain deep, multi-dimensional insights into network traffic in real-time.

By leveraging a high-speed data pipeline, a flexible pluggable aggregation engine, and AI-powered threat analysis, Go2NetSpectra enables everything from basic network performance monitoring to sophisticated security threat detection, all at scale.

✨ Key Features

🔀 Hybrid Analysis Engine: Simultaneously run multiple aggregator types (Exact + Sketch) for 100% accurate accounting and high-performance probabilistic analysis at the same time. Use Sketch to find anomalies, then use Exact to get precise details.
🤖 AI-Powered Alert Analysis: The new ns-ai microservice provides intelligent analysis for triggered alerts. It enriches notifications with root cause analysis, threat assessment, and mitigation suggestions, turning raw alerts into actionable insights.
⚡ Real-Time Alerting: A built-in alerting pipeline allows tasks to generate event messages. These are processed by a central Alerter which can trigger notifications via webhooks, providing immediate insights into network events.
🧩 Pluggable Aggregation Algorithms: The sketch aggregator is a micro-framework supporting multiple estimation algorithms via configuration:
- Count-Min Sketch: Heavy hitter detection with configurable accuracy
- SuperSpread: Cardinality estimation and super-spreader detection
⚙️ High-Performance by Design: Built from the ground up for performance:
- Lock-free atomic operations in sketches
- Worker pool concurrency with optimal goroutine scheduling
- Efficient Thrift serialization
- Zero-copy data handling where possible
📊 Full-Stack Observability: Built-in support for:
- Real-time Grafana dashboards
- Structured alerting with webhooks
- Multiple data backends (ClickHouse, File-based storage)
- Comprehensive query APIs (Thrift RPC + HTTP/JSON)
🚀 Scalable & Distributed: All components are decoupled and horizontally scalable:
- NATS for message bus decoupling
- Stateless microservices (Engine, API, AI)
- Kubernetes-ready with Helm charts
- Automatic leader election and failover
🔍 Intelligent Search: Support for both exact and approximate queries:
- Precise flow accounting with 100% accuracy
- Fast probabilistic heavy hitter detection
- Flexible multi-dimensional aggregations

🏗️ System Architecture

Go2NetSpectra operates as a multi-stage, distributed pipeline designed for performance, scalability, and real-time analysis.

graph TD
    subgraph DataPlane["📊 Data Collection Plane"]
        direction LR
        Iface["🖧 Network Interface"]
        Pcap["📁 PCAP Files"]
        Probe["🔍 ns-probe<br/>Live Capture"]
        Analyzer["🔬 pcap-analyzer<br/>Offline Analysis"]
        
        Iface -->|live traffic| Probe
        Pcap -->|offline traffic| Analyzer

        style Iface fill:#fbe9e7,stroke:#d84315,stroke-width:2px,color:#222
        style Pcap fill:#fbe9e7,stroke:#d84315,stroke-width:2px,color:#222
        style Probe fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#222
        style Analyzer fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#222
    end

    subgraph MessageBus["🚌 Message Bus"]
        direction TB
        NATS["📬 NATS<br/>Message Broker"]
        style NATS fill:#fff3e0,stroke:#fb8c00,stroke-width:3px,color:#222
    end

    subgraph ProcessingPlane["⚙️ Processing & Analysis Plane"]
        direction TB
        
        subgraph Engine["🧠 ns-engine: Core Processing"]
            Manager["📋 Manager<br/>Worker Pool Orchestrator"]
            style Manager fill:#e0f2f1,stroke:#00897b,stroke-width:2px,color:#222
        end
        
        subgraph Aggregators["📈 Aggregation Tasks"]
            ExactTask["✓ Exact Task<br/>100% Accurate"]
            SketchTask["⚡ Sketch Task<br/>Probabilistic<br/>Count-Min/SuperSpread"]
            style ExactTask fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#222
            style SketchTask fill:#bbdefb,stroke:#1e88e5,stroke-width:2px,color:#222
        end
        
        subgraph Alerting["🚨 Real-time Alerting & AI"]
            Alerter["⚠️ Alerter<br/>Rule Evaluation"]
            AI_Service["🤖 ns-ai<br/>AI Analysis Gateway"]
            Notifier["📧 Notifier<br/>Email/Webhook"]
            style Alerter fill:#fff8e1,stroke:#fbc02d,stroke-width:2px,color:#222
            style AI_Service fill:#ede7f6,stroke:#7e57c2,stroke-width:2px,color:#222
            style Notifier fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#222
        end
        
        subgraph Storage["💾 Storage Layer"]
            ClickHouse["📊 ClickHouse<br/>Time-Series DB"]
            style ClickHouse fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#222
        end
        
        Manager --> ExactTask
        Manager --> SketchTask
        ExactTask -->|snapshot| ClickHouse
        SketchTask -->|snapshot| ClickHouse
        ExactTask -->|generates event| Alerter
        SketchTask -->|generates event| Alerter
        Alerter -->|Thrift RPC| AI_Service
        AI_Service -->|enriched analysis| Alerter
        Alerter -->|formatted alert| Notifier
        
        %% --- 修改点 1：为 ProcessingPlane 内部的 subgraph 添加 color ---
        style Engine fill:#f1f8e9,stroke:#388e3c,stroke-width:2px,color:#222
        style Aggregators fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#222
        style Alerting fill:#fff8e1,stroke:#fbc02d,stroke-width:2px,color:#222
        style Storage fill:#ede7f6,stroke:#5e35b1,stroke-width:2px,color:#222
    end

    subgraph QueryPlane["🔍 Query & Interaction Plane"]
        direction TB
        
        API["🌐 ns-api<br/>Multi-Query Router"]
        GrpcClient["📱 RPC Client<br/>High-Performance"]
        HttpClient["🌍 HTTP/JSON Client<br/>Grafana Integration"]
        AI_Client["💬 AI Client<br/>Interactive QA"]
        Grafana["📊 Grafana<br/>Real-time Dashboards"]
        
        style API fill:#e8f5e9,stroke:#43a047,stroke-width:3px,color:#222
        style GrpcClient fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#222
        style HttpClient fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#222
        style AI_Client fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#222
        style Grafana fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#222
    end

    Probe -->|Thrift| NATS
    Analyzer -->|Thrift| NATS
    NATS -->|Thrift| Manager
   
    GrpcClient -->|Thrift RPC| API
    HttpClient -->|HTTP/JSON| API
    AI_Client -->|Session + Chunks| AI_Service
    API -->|SQL queries| ClickHouse
    Grafana -->|queries| API

    %% --- 修改点 2：为顶层 subgraph 添加 color ---
    style DataPlane fill:#ffebee,stroke:#d32f2f,stroke-width:2px,color:#222
    style MessageBus fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#222
    style ProcessingPlane fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#222
    style QueryPlane fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#222

Core Components

Boundary Ownership And Refactor Guardrails

cmd/ packages are process wiring only. Shared runtime assembly belongs in internal/api, internal/ai, or internal/engine/app.
internal/probe and pkg/pcap own transport packet conversion; internal/engine/manager owns fan-out and lifecycle orchestration.
api/thrift/v1/, configs/config.yaml, Docker Compose values, and Helm values are synchronized product surfaces. Any behavioral key change must update all affected runtime and deployment assets together.
Use module-boundaries.md for the current ownership map and build.md for validation entrypoints.

🚀 Quick Start

Choose the deployment option that best fits your needs.

Prerequisites

Environment Setup

All configuration uses environment variables via .env or .docker.env files:

Go2NetSpectra

Install / Use

README

Go2NetSpectra

📖 Overview

✨ Key Features

🏗️ System Architecture

Core Components

Boundary Ownership And Refactor Guardrails

🚀 Quick Start

Prerequisites

Environment Setup