<div align="center">

</div>

Lerian Reporter

A service for managing and generating customizable reports using templates. Reporter connects directly to your databases (PostgreSQL and MongoDB) and renders reports in multiple formats (HTML, PDF, CSV, XML, TXT).

Table of Contents

• Overview
• Architecture
• Quick Start
• Configuration
• Data Sources
• Templates
• API Reference
• Development
• Contributing
• Security
• Code of Conduct
• License

Overview

Reporter is a report generation service that:

• Manages templates using Pongo2 (Django-like templating for Go)
• Connects to multiple databases (PostgreSQL and MongoDB) configured via environment variables
• Generates reports in various formats: HTML, PDF, CSV, XML, TXT
• Processes asynchronously using RabbitMQ for scalable report generation
• Stores files in S3-compatible storage (AWS S3, SeaweedFS, MinIO)

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                           REPORTER                                   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────┐         ┌─────────────┐         ┌─────────────┐   │
│  │   Manager   │ ──────► │  RabbitMQ   │ ──────► │   Worker    │   │
│  │  (REST API) │         │   Queue     │         │ (Generator) │   │
│  └─────────────┘         └─────────────┘         └──────┬──────┘   │
│         │                                                │          │
│         │                                                │          │
│         ▼                                                ▼          │
│  ┌─────────────┐                                 ┌─────────────┐   │
│  │   MongoDB   │                                 │ Data Sources│   │
│  │  (metadata) │                                 │ PostgreSQL  │   │
│  └─────────────┘                                 │  MongoDB    │   │
│         │                                        └─────────────┘   │
│         │                                                │          │
│         ▼                                                ▼          │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    Object Storage (S3)                       │   │
│  │         AWS S3 / SeaweedFS / MinIO (Templates & Reports)     │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Components

| Component | Description | |-----------|-------------| | Manager | REST API for templates and reports CRUD. Receives report requests and publishes to queue. | | Worker | Consumes messages from RabbitMQ, queries data sources, renders templates, and stores results. | | MongoDB | Stores metadata for templates and reports. | | RabbitMQ | Message queue for asynchronous report generation. | | Object Storage | S3-compatible storage (AWS S3, SeaweedFS, MinIO) for templates and generated reports. | | Redis/Valkey | Caching layer for data source schemas. |

Quick Start

Prerequisites

• Go 1.25+
• Docker and Docker Compose
• Make

Installation

1. Clone the repository:

   git clone https://github.com/LerianStudio/reporter.git
   cd reporter
   

2. Set up environment files:

   make set-env
   

3. Start all services:

   make up
   

4. Access the API:

   • API: http://localhost:4005
   • Swagger UI: http://localhost:4005/swagger/index.html

Configuration

Environment Variables

Reporter uses environment variables for configuration. Copy .env.example files and adjust as needed:

# In each component directory
cp .env.example .env

Key configurations:

| Variable | Description | Default | |----------|-------------|---------| | SERVER_PORT | Manager API port | 4005 | | MONGO_HOST | MongoDB hostname | reporter-mongodb | | RABBITMQ_HOST | RabbitMQ hostname | reporter-rabbitmq | | LOG_LEVEL | Log verbosity | debug |

Object Storage (S3-compatible)

Reporter supports S3-compatible object storage for templates and generated reports:

| Variable | Description | Default | |----------|-------------|---------| | OBJECT_STORAGE_ENDPOINT | S3 endpoint URL | http://reporter-seaweedfs:8333 | | OBJECT_STORAGE_REGION | AWS region | us-east-1 | | OBJECT_STORAGE_ACCESS_KEY_ID | Access key ID | - | | OBJECT_STORAGE_SECRET_KEY | Secret access key | - | | OBJECT_STORAGE_BUCKET | Bucket name | reporter-storage | | OBJECT_STORAGE_USE_PATH_STYLE | Use path-style URLs | true | | OBJECT_STORAGE_DISABLE_SSL | Disable SSL | true |

Supported providers: AWS S3, SeaweedFS S3, MinIO, and other S3-compatible services.

Data Sources

Reporter connects directly to external databases to fetch data for reports. Configure data sources using the DATASOURCE_* environment variables pattern:

# Pattern: DATASOURCE_<NAME>_<PROPERTY>

# PostgreSQL Example
DATASOURCE_MYDB_CONFIG_NAME=my_database
DATASOURCE_MYDB_HOST=postgres-host
DATASOURCE_MYDB_PORT=5432
DATASOURCE_MYDB_USER=username
DATASOURCE_MYDB_PASSWORD=password
DATASOURCE_MYDB_DATABASE=dbname
DATASOURCE_MYDB_TYPE=postgresql
DATASOURCE_MYDB_SSLMODE=disable
DATASOURCE_MYDB_SCHEMAS=public,sales,inventory  # Multi-schema support

# MongoDB Example
DATASOURCE_MYMONGO_CONFIG_NAME=my_mongo
DATASOURCE_MYMONGO_HOST=mongo-host
DATASOURCE_MYMONGO_PORT=27017
DATASOURCE_MYMONGO_USER=username
DATASOURCE_MYMONGO_PASSWORD=password
DATASOURCE_MYMONGO_DATABASE=dbname
DATASOURCE_MYMONGO_TYPE=mongodb
DATASOURCE_MYMONGO_SSL=false

Supported Databases

| Database | Type Value | Notes | |----------|------------|-------| | PostgreSQL | postgresql | Supports SSL modes | | MongoDB | mongodb | Supports replica sets |

Features

• Automatic schema discovery - Reporter introspects database schemas
• Multi-schema support - Query tables across multiple PostgreSQL schemas (e.g., public, sales, inventory)
• Connection pooling - Configurable pool sizes for performance
• Circuit breaker - Automatic failover for unavailable data sources
• Health checking - Background monitoring of data source availability

Templates

Templates use Pongo2 syntax (similar to Django/Jinja2).

Example Template

{% for row in my_database.users %}
Name: {{ row.name }}
Email: {{ row.email }}
{% endfor %}

Accessing Data

Data is available in templates using the pattern:

{{ datasource_config_name.table_name }}

For multi-schema databases, use explicit schema syntax:

{{ datasource_config_name:schema_name.table_name }}

Example with multiple schemas:

{# Access table from public schema #}
{% for account in midaz_onboarding:public.account %}
  Account: {{ account.id }} - {{ account.name }}
{% endfor %}

{# Access table from payment schema #}
{% for transfer in midaz_onboarding:payment.transfers %}
  Transfer: {{ transfer.id }} - {{ transfer.amount }}
{% endfor %}

Output Formats

| Format | Extension | Use Case | |--------|-----------|----------| | HTML | .html | Web reports, dashboards | | PDF | .pdf | Printable documents | | CSV | .csv | Data export, spreadsheets | | XML | .xml | Regulatory reports, integrations | | TXT | .txt | Plain text reports |

Custom Filters

Reporter extends Pongo2 with additional filters for report generation. See pkg/pongo/filters.go for available filters.

API Reference

Endpoints

Templates

| Method | Endpoint | Description | |--------|----------|-------------| | POST | /manager/v1/templates | Create template | | GET | /manager/v1/templates | List templates | | GET | /manager/v1/templates/{id} | Get template by ID | | PATCH | /manager/v1/templates/{id} | Update template | | DELETE | /manager/v1/templates/{id} | Delete template |

Reports

| Method | Endpoint | Description | |--------|----------|-------------| | POST | /manager/v1/reports | Generate report | | GET | /manager/v1/reports | List reports | | GET | /manager/v1/reports/{id} | Get report by ID |

Data Sources

| Method | Endpoint | Description | |--------|----------|-------------| | GET | /manager/v1/data-sources | List configured data sources | | GET | /manager/v1/data-sources/{id} | Get data source schema |

Health

| Method | Endpoint | Description | |--------|----------|-------------| | GET | /health | Health check |

Report Generation Message

Reports are generated asynchronously via RabbitMQ:

• Exchange: reporter.generate-report.exchange
• Queue: reporter.generate-report.queue
• Routing Key: reporter.generate-report.key

{
  "templateId": "019538ee-deee-769c-8859-cbe84fce9af7",
  "reportId": "019615d3-c1f6-7b1d-add4-6912b76cc4f2",
  "outputFormat": "html",
  "mappedFields": {
    "my_database": {
      "users": ["id", "name", "email"],
      "orders": ["id", "total", "