⚠️ DEPRECATED ⚠️

This repository is no longer actively maintained and is provided as-is for reference purposes only.

This project is deprecated and will not receive further updates, bug fixes, or support. The code remains available for historical reference and for those who wish to fork and maintain their own version.

If you're looking for modern log analysis tools, consider exploring these AI-suggested alternatives:

Ruby Log Parsers:

• Logstash - Part of the Elastic Stack (Ruby-based, supports multiple languages)
• Fluentd - Open source data collector (Ruby-based, supports multiple languages)

General Log Analysis Platforms:

• Graylog - Log management platform (Java-based)
• Loki - Log aggregation system by Grafana (Go-based)

---

Indy: A Log Archaeology Tool

Synopsis

Log files are often searched for particular strings but are not often treated as data structures. Indy attempts to deliver log content via more powerful features by allowing the ability to collect segments of a log from particular time; find a particular event; or monitor/reflect on a log to see if a particular event occurred (or not occurred).

Installation

To install Indy use the following command:

$ gem install indy

Usage

Basic Example (default log pattern)

log = Indy.search(:file => './data.log').all.first.message
# => "Entering APPLICATION."

Custom Example

# Given this log file:
#
# 2015/09/08 INFO #index Indexing application.
# 2015/09/11 ERROR #search Searching application.
# ...

log_contents = File.open('./example.log', 'r').read
pattern = /^(\d{4}\/\d{2}\/\d{2})\s+(INFO|DEBUG|WARN|ERROR)\s+(#\S+)\s+(.+)$/
fields = [:time, :severity, :method, :message]
custom_time_format = '%Y/%m/%d'

indy = Indy.search(  :source => log_contents,
                     :entry_regexp => pattern,
                     :entry_fields => fields,
                     :time_format => custom_time_format
           )
indy.after(:time => '2015/09/10').like(:severity => 'ERROR') do |entry|
    puts "#{entry.time} (#{entry.method}) #{entry.message}"
end

Specify your Source

As a process or command

Indy.search( {:cmd => 'ssh user@system "bash --login -c \"cat /var/log/standard.log\" "'} ).for(:severity => 'INFO')

As a file

Indy.search(file_object).for(:application => 'MyApp')
Indy.search(:file => file_object).for(:application => 'MyApp')
Indy.search(:file => '/log/data.log').for(:application => 'MyApp')

As a string

log_string = %{
    2000-09-07 14:07:41 INFO  MyApp - Entering application.
    2000-09-07 14:07:41 INFO  MyApp - Exiting application. }

Indy.search(log_string).for(:message => 'Entering application')

Log Pattern

Default Log Format

The default log format follows this form:

YYYY-MM-DD HH:MM:SS SEVERITY APPLICATION_NAME - MESSAGE

which uses this regular expression:

/^(\d{4}.\d{2}.\d{2}\s+\d{2}.\d{2}.\d{2})\s+(TRACE|DEBUG|INFO|WARN|ERROR|FATAL)\s+(\w+)\s+-\s+(.+)$/

and specifies these fields:

[:time, :severity, :application, :message]

allowing searches like so:

Indy.search(log_file).for(:severity => 'INFO')
Indy.search(log_file).for(:application => 'MyApp', :severity => 'DEBUG')

Custom Log Format

Brew your own log format! To do so, specify an :entry_regexp pattern that captures each field you want to reference. Also, specify an :entry_fields array of symbols that name the fields captured by your regexp. If your date/time format differs from the default (%Y-%m-%d %H:%M:%S), you will need to specify a :time_format parameter.

# If your log format is:
# YYYY-MM-DD HH:MM:SS SEVERITY APPLICATION#METHOD - MESSAGE

# Build a regexp pattern
pattern = /^(\d{4}.\d{2}.\d{2}\s+\d{2}.\d{2}.\d{2})\s*(INFO|DEBUG|WARN|ERROR)\s*([^#]+)#([^\s]+)\s*-\s*(.+)$/
# List the log fields
fields = [:time,:severity,:application,:method,:message]

# Use Indy#with to define your format
Indy.search(source).with(:entry_regexp => pattern, :entry_fields => fields)

Predefined Log Format

Several log formats have been predefined for ease of configuration. See indy/formats.rb

# Indy::COMMON_LOG_FORMAT
# Indy::COMBINED_LOG_FORMAT
# Indy::LOG4R_DEFAULT_FORMAT
#
# Example (Log4r)
#  INFO mylog: This is a message with level INFO
Indy.new(:source => log_file).with(Indy::LOG4R_DEFAULT_FORMAT).for(:application => 'mylog')

Multiline log entries

By default, Indy assumes that log lines are separated by new lines. Any lines that don't match the active pattern are ignored. To enable multiline log entries you MUST do these things:

1. Use Indy.new() and include the :multiline => true parameter
2. Use a log entry regexp that does not use $ and/or \n to define the end of the entry.
3. Add a capture group that surrounds one full log entry.

Multiline Regexp tips

• Use non-greedy matching when needed: .*? instead of .*
• Assuming your log entries do not include a unique line ending, you can use a zero-width positive lookahead assertion to verify that each line is followed by the start of a valid log entry, or the end of the string. e.g.: if 'foo' starts each entry, use this assertion (?=^foo|\z) at the end of your :entry_regexp

Check out Regexp Extensions

Example:

# Given this log containing three entries:
#
# INFO MyApp - Multiline message begins here...
# and ends here
# DEBUG MyOtherApp - Single line message.
# WARN MyOtherApp - A third entry.

severity_string = 'DEBUG|INFO|WARN|ERROR|FATAL'

multiline_regexp = /^((#{severity_string}) (\w+) - (.*?)(?=^#{severity_string}|\z))/

# For reference, a single line regexp would be:
#                  /^(#{severity_string}) (\w+) - (.*)$/


Indy.new( :multiline => true,
          :entry_regexp => multiline_regexp,
          :entry_fields => [:severity, :application, :message],
          :source => MY_LOG
        )

Explicit Time Format

If not specified, Indy tries to guess your time format (courtesy of DateTime#parse). If you supply an explicit time format, it will use DateTime#strptime. If that fails, it will then guess with DateTime#parse.

This is required when log data uses a non-standard date format, e.g.: U.S. format 12-31-2000, and must be used in conjunction with :entry_regexp and :entry_fields parameters.

# 12-31-2011 Application starting
Indy.new(   :time_format => '%m-%d-%Y',
            :source => LOG_FILE,
            :entry_regexp => /\d\d-\d\d-\d\d\d\d .*?/,
            :entry_fields => [:time, :message]
        ).all

Format directives are documented in DateTime#strftime.

Match Criteria

Exact Match

Indy.search(source).for(:message => 'Entering Application')
Indy.search(source).for(:severity => 'INFO')

Exact Match with multiple parameters

Indy.search(source).for(:message => 'Entering Application', :application => 'MyApp')
Indy.search(source).for(:severity => 'INFO', :application => 'MyApp')

Partial Match

Indy.search(source).like(:message => 'Memory')

Partial Match with multiple parameters

Indy.search(source).like(:severity => '(?:INFO|DEBUG)', :message => 'Memory')

Log Scopes

Multiple scope methods can be called on an instance. Use #reset_scope to remove scope constraints on the instance.

Time Scope

# After Dec 1
Indy.search(source).after(:time => '2010-12-01 23:59:59').all

# 20 minutes Around New Year's eve
Indy.search(source).around(:time => '2011-01-01 00:00:00', :span => 20).all

# After Jan 1 but Before Feb 1
@log = Indy.search(source)
@log.after(:time => '2011-01-01 00:00:00').before(:time => '2011-02-01 00:00:00')
@log.all

# Within Jan 1 and Feb 1 (same time scope as above)
Indy.search(source).within(:start_time => '2011-01-01 00:00:00', :end_time =>'2011-02-01 00:00:00').all

# After Jan 1
@log = Indy.search(source)
@log.after(:time => '2011-01-01 00:00:00')
@log.all)
# Reset the time scope to include entries before Jan 1
@log.reset_scope
# Before Feb 1
@log.before(:time => '2011-02-01 00:00:00')
@log.all

Process the Results

An Array is returned by #for and #like, containing a Struct::Entry for each log entry. The full entry is available with entry.raw_entry.

entries = Indy.search(source).for(:message => 'Entering Application')
entries.first.members
# => ["time", "severity", "application", "message", "raw_entry"]

Indy.search(source).for(:message => 'Entering Application').each do |entry|
  puts "[#{entry.time}] #{entry.message}: #{entry.application}"
end

Contributing

To get your improvements included, please fork and submit a pull request. Bugs and/or failing tests are very appreciated.

Any suggestions about log formats to support, and/or intelligent defaults would be great.

To create a report in /coverage, run gem install simplecov COVERAGE=true rake test

Compatibility

Indy was developed on MacOS, and should support MS Windows. Travis builds run the following ruby flavors:

• 2.4.3
• 2.3.6
• 2.2.9
• 1.9.3
• jruby

LICENSE

(The MIT License)

Copyright (c) 2010 Franklin Webber, Brandon Faloona

Permission is hereby granted, free of charge, to any per