LogSentinelAI — AI Log Analyzer

Declarative LLM-Based Log Analyzer for Security Events, System Errors, and Anomalies

Benefits: Transform unstructured logs into structured security intelligence by simply defining a Pydantic model—the LLM automatically extracts and validates data without manual parsing or regex rules.

Keywords: AI log analysis • cybersecurity automation • SIEM integration • threat detection • DevSecOps • log monitoring • security intelligence • anomaly detection

LogSentinelAI is an AI-powered cybersecurity tool that leverages LLM with Declarative Extraction to analyze security events, anomalies, and errors from various logs including Apache, Linux, and converts them into structured data for SIEM integration with Elasticsearch/Kibana. This DevSecOps automation solution enables real-time threat detection and security monitoring by simply declaring your desired result structure as a Pydantic class, and the AI automatically analyzes logs to return JSON matching that schema. No complex parsing or regex rules required.

---

Architecture & Internal (DeepWiki)

---

Installation & Usage Guide

Requirements: Python 3.11 or 3.12 (Python 3.13+ not supported due to dependency compatibility)

For installation, environment setup, CLI usage, Elasticsearch/Kibana integration, and all practical guides for LogSentinelAI, please refer to the installation documentation below.

Go to Installation and Usage Guide: INSTALL-and-USAGE.md

⚡️ For additional inquiries, please use GitHub Issues/Discussions!

---

Dashboard Example

---

JSON Output Example

---

Telegram Alert Example

When critical security events are detected, LogSentinelAI can automatically send real-time alerts to Telegram:

🚨 [CRITICAL+ EVENTS] 🚨
  • Highest Severity: CRITICAL
  • Immediate Attention: Not Required

📊 Alert Events Summary (1 total):
  • CRITICAL: 1

📋 Summary
  ➤ The analysis indicates several potential security events in the system logs.

🔥 Event-1
  • Severity: CRITICAL
  • Event Type: AUTH_FAILURE
  • Description: Multiple authentication failures attempted against the SSH daemon.
  • Confidence: 0.9
  • Human Review: Required
  • Related Logs:
      1. Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEV...
      2. Jun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown
      3. Jun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEV...
      ... and 5 more log entries
  • Recommended Actions:
      ➤ Review login history and account activity for suspicious patterns.
      ➤ Implement multi-factor authentication to enhance security.
      ➤ Monitor network traffic for unauthorized connections.

📊 Statistics:
  • total_events: 8
  • auth_failures: 8
  • unique_ips: 0
  • unique_users: 0

🔍 ES/Kibana Metadata:
  • Index: logsentinelai-analysis
  • @chunk_analysis_start_utc: 2025-08-17T22:42:32Z
  • @chunk_analysis_end_utc: 2025-08-17T22:43:02Z
  • @chunk_analysis_elapsed_time: 30
  • @processing_result: success
  • @log_count: 10
  • @processing_mode: batch
  • @access_mode: local
  • @llm_provider: vllm
  • @llm_model: Qwen/Qwen2.5-1.5B-Instruct
  • @log_path: /var/log/messages
  • @token_size_input: 1834
  • @token_size_output: 618
  • @timestamp: 2025-08-17T22:43:02.261161
  • @log_type: linux_system
  • @document_id: linux_system_20250817_224302_261129_chunk_1
  • @host: {"hostname":"linux.foo.com","ip_addresses":["123.123.123.123/24"]}

Configure Telegram alerts by setting TELEGRAM_ENABLED=true, TELEGRAM_TOKEN, and TELEGRAM_CHAT_ID in your config file. Alerts are automatically sent for CRITICAL+ events (configurable via TELEGRAM_ALERT_LEVEL).

---

Key Features

⚡️ Declarative Extraction

In each analyzer script, simply declare the desired result structure as a Pydantic class, and the LLM will automatically analyze logs and return results as JSON matching that schema. No complex parsing or post-processing—just declare what you want, and the AI handles the rest. This approach enables developers to focus on "what to extract" declaratively, while the LLM takes care of "how to extract"—a modern paradigm for information extraction.

# Example: Just declare the result structure you want in your HTTP Access log analyzer
from pydantic import BaseModel

class MyAccessLogResult(BaseModel):
    ip: str
    url: str
    is_attack: bool

# By defining only the result structure (Pydantic class) like above,
# the LLM automatically analyzes each log and returns JSON like this:
# {
#   "ip": "192.168.0.1",
#   "url": "/admin.php",
#   "is_attack": true
# }

---

System Architecture

• Log Sources: Logs are collected from various sources, including local files, remote SSH connections, HTTP endpoints, Apache error logs, system logs, and TCPDump outputs.
• LogSentinelAI Core: Handles parsing and extraction using a declarative approach. Log structures are defined using Pydantic models, and the actual extraction is performed by LLMs. The system validates and structures the extracted data.
• LLM Provider: Integrates with external or local LLMs (e.g., OpenAI, vLLM, Ollama, Google Gemini, Anthropic Claude) to interpret and transform raw logs into structured JSON, based on user-defined schemas.
• Elasticsearch: Structured outputs, raw logs, and metadata are indexed into Elasticsearch for searchability and event correlation.
• Kibana: Provides visualization and dashboards for immediate insight into security events and operational data.
• Telegram Alerts: Automatically sends real-time notifications to Telegram groups/channels when CRITICAL security events are detected or processing failures occur, enabling immediate incident response.

AI-powered Analysis

• Declarative Extraction: Just declare your desired result structure (Pydantic class) and the LLM analyzes logs automatically
• LLM Providers: OpenAI API, Ollama, vLLM, Google Gemini, Anthropic Claude
• Supported Log Types: HTTP Access, Apache Error, Linux System, General Log
• Threat Detection: SQL Injection, XSS, Brute Force, Network Anomaly Detection
• Output: Structured JSON validated by Pydantic
• Just define a Pydantic class and the LLM generates results in that structure automatically
• Adaptive Sensitivity: Detection sensitivity auto-adjusted by LLM model and log type prompt

Processing Modes

• Batch: Bulk analysis of historical logs
• Real-time: Sampling-based live monitoring
• Access Methods: Local files, SSH remote

Data Enrichment

• GeoIP: MaxMind GeoLite2 City lookup (including coordinates, Kibana geo_point support)
• Statistics: IP counts, response codes, various metrics
• Multi-language Support: Configurable result language (default: Korean)

Integration & Output

• Storage: Elasticsearch (ILM policy support)
• Visualization: Kibana dashboard
• Deployment: Docker containers
• Real-time Alerts: Telegram notifications for CRITICAL security events and system failures

CLI Command Mapping

# CLI commands mapped to analyzer scripts:
logsentinelai-httpd-access   → analyzers/httpd_access.py
logsentinelai-httpd-server   → analyzers/httpd_server.py  
logsentinelai-linux-system   → analyzers/linux_system.py
logsentinelai-geoip-download → utils/geoip_downloader.py

Sample Log Preview

HTTP Access Log

54.36.149.41 - - [22/Jan/2019:03:56:14 +0330] "GET /filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53 HTTP/1.1" 200 30577 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)" "-"
31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/60844/productModel/200x200 HTTP/1.1" 200 5667 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-"
31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/61474/productModel/200x200 HTTP/1.1" 200 5379 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-"
40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] "GET /image/14925/productModel/100x100 HTTP/1.1" 200 1696 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
91.99.72.15 - - [22/Jan/2019:03:56:17 +0330] "GET /produc