gosquito

gosquito ("go" + "mosquito") is a pluggable tool for data gathering, data processing and data transmitting to various destinations. Main goal is to replace various in-house automated tasks with a single tool and move those tasks at the edge. See docs and examples for additional info.

graph TD;
    I1(io)-->P1(process plugins);
    I2(kafka)-->P1;
    I3(resty)-->P1;
    I4(rss)-->P1;
    I5(telegram)-->P1;
    P1-->O1(kafka);
    P1-->O2(mattermost);
    P1-->O3(resty);
    P1-->O4(slack);
    P1-->O5(smtp);
    P1-->O6(telegram);

<br>

Main features:

• Pluggable architecture. Data processing organized as chains of plugins.
• Flow approach. Flow consists of: input plugin (receive), process plugins (filter/transform), output plugin (send).
• Declarative YAML configurations with templates support.
• Consider data as new by configurable signature (data timestamp by default). Force fetching is supported.
• Dependencies between process plugins. Plugin "B" will process data only if plugin "A" derived some data.
• Parallel executions limits.
• Export metrics to Prometheus.

<br>

Input plugins:

| Plugin | Description | |:-------------------------------------------|:-----------------------------------------------------------------------------------------------| | io | Use text and files as data source. | | kafka | Kafka topic as data source. | | resty | REST endpoint as data source. | | rss | RSS/Atom feed as data source. | | telegram | Telegram chat as data source. |

Process plugins:

| Plugin | Description | |:-------------------------------------------------------|:-------------------------------------------------------------------------------------------------------| | dedup | Deduplicate datums by UUID. | | echo | Echoing processing data. | | expandurl | Expand short URLs. | | fetch | Fetch remote data. | | iconv | Convert text encoding. | | io | Read/write text and files. | | jq | Extract JSON elements. | | minio | Get/put data from/to S3 bucket. | | regexpfind | Find patters in data. | | regexpmatch | Match data by patterns. | | regexpreplace | Replace patterns in data. | | resty | Perform REST queries. | | same | Match data similarity. | | split | Split single datum into multiple. | | unique | Get unique values from data. | | webchela | Process web pages with full browser support (Chrome, Firefox). | | xpath | Extract HTML/XML nodes. |

Output plugins:

| Plugin | Description | |:------------------------------------------------|:---------------------------------------------------------------------------------------------| | kafka | Send data to Kafka topic. | | mattermost | Send data to Mattermost channel/user. | | resty | Send data to REST endpoint. | | slack | Send data to Slack channel/user. | | smtp | Send data as email. | | telegram | Send data to Telegram chat. |