Logparser
A machine learning toolkit for log parsing [ICSE'19, DSN'16]
Install / Use
/learn @logpai/LogparserREADME
Logparser
<div> <a href="https://pypi.org/project/logparser3"><img src="https://img.shields.io/badge/python-3.6+-blue" style="max-width: 100%;" alt="Python version"></a> <a href="https://pypi.org/project/logparser3"><img src="https://img.shields.io/pypi/v/logparser3.svg" style="max-width: 100%;" alt="Pypi version"></a> <a href="https://github.com/logpai/logparser/actions/workflows/ci.yml"><img src="https://github.com/logpai/logparser/workflows/CI/badge.svg" style="max-width: 100%;" alt="Pypi version"></a> <a href="https://pepy.tech/project/logparser3"><img src="https://static.pepy.tech/badge/logparser3" style="max-width: 100%;" alt="Downloads"></a> <a href="https://github.com/logpai/logparser/blob/main/LICENSE.md"><img src="https://img.shields.io/badge/license-view-brightgreen" style="max-width: 100%;" alt="License"></a> <a href="https://github.com/logpai/logparser#discussion"><img src="https://img.shields.io/badge/chat-wechat-brightgreen?style=flat" style="max-width: 100%;"></a> </div> <hr/> <div> <a href="https://github.com/logpai/logparser/stargazers"><img src="http://bytecrank.com/nastyox/reporoster/php/stargazersSVG.php?user=logpai&repo=logparser" width="600"/><a/> </div>Logparser provides a machine learning toolkit and benchmarks for automated log parsing, which is a crucial step for structured log analytics. By applying logparser, users can automatically extract event templates from unstructured logs and convert raw log messages into a sequence of structured events. The process of log parsing is also known as message template extraction, log key extraction, or log message clustering in the literature.
<p align="center"><img src="https://cdn.jsdelivr.net/gh/logpai/logparser@main/docs/img/example.jpg" width="485"><br>An example of log parsing</p>🌈 New updates
- Since the first release of logparser, many PRs and issues have been submitted due to incompatibility with Python 3. Finally, we update logparser v1.0.0 with support for Python 3. Thanks for all the contributions (#PR86, #PR85, #PR83, #PR80, #PR65, #PR57, #PR53, #PR52, #PR51, #PR49, #PR18, #PR22)!
- We build the package wheel logparser3 and release it on pypi. Please install via
pip install logparser3. - We refactor the code structure and beautify the code via the Python code formatter black.
Log parsers available:
| Publication | Parser | Paper Title | Benchmark | |:-----------:|:-------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------:| | IPOM'03 | SLCT | A Data Clustering Algorithm for Mining Patterns from Event Logs, by Risto Vaarandi. | :arrow_upper_right: | | QSIC'08 | AEL | Abstracting Execution Logs to Execution Events for Enterprise Applications, by Zhen Ming Jiang, Ahmed E. Hassan, Parminder Flora, Gilbert Hamann. | :arrow_upper_right: | | KDD'09 | IPLoM | Clustering Event Logs Using Iterative Partitioning, by Adetokunbo Makanju, A. Nur Zincir-Heywood, Evangelos E. Milios. |:arrow_upper_right: | | ICDM'09 | LKE | Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis, by Qiang Fu, Jian-Guang Lou, Yi Wang, Jiang Li. [Microsoft] |:arrow_upper_right: | | MSR'10 | LFA | Abstracting Log Lines to Log Event Types for Mining Software System Logs, by Meiyappan Nagappan, Mladen A. Vouk. |:arrow_upper_right: | | CIKM'11 | LogSig | LogSig: Generating System Events from Raw Textual Logs, by Liang Tang, Tao Li, Chang-Shing Perng. |:arrow_upper_right: | | SCC'13 | SHISO | Incremental Mining of System Log Format, by Masayoshi Mizutani. |:arrow_upper_right: | | CNSM'15 | LogCluster | LogCluster - A Data Clustering and Pattern Mining Algorithm for Event Logs, by Risto Vaarandi, Mauno Pihelgas. |:arrow_upper_right: | | CNSM'15 | LenMa | Length Matters: Clustering System Log Messages using Length of Words, by Keiichi Shima. |:arrow_upper_right: | | CIKM'16 | LogMine | LogMine: Fast Pattern Recognition for Log Analytics, by Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Geoff Jiang, Adbullah Mueen. [NEC] |:arrow_upper_right: | | ICDM'16 | Spell | Spell: Streaming Parsing of System Event Logs, by Min Du, Feifei Li. |:arrow_upper_right: | | ICWS'17 | Drain | Drain: An Online Log Parsing Approach with Fixed Depth Tree, by Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. |:arrow_upper_right: | | ICPC'18 | [MoLFI]
