mainevent

mainevent provides a suite of tools to gain insight into log files.

• Collect, parse and store updates from local and SSH-accessible log files.
• Easily write new parser modules for any file format.
• Searchable timeline with optional real-time updates.
• Dashboard analytics generated by MongoDB MapReduce, Redis and jqPlot.
• Write custom Pub/Sub listeners to real-time updates.
• more ...

<a href="http://codeactual.github.com/mainevent"><img alt="3-screenshot set" src="http://codeactual.github.com/mainevent/img/screenshot-set.png" /></a>

Use Cases

• Search development/production environment logs which span a variety of formats.
• Produce dashboard data and graphs specific to your insight needs.
• Filter and replicate real-time updates into additional systems for specialized processing or alerting.

How To

Create a new parser for an unsupported log format

Each parser lives in a separate directory under <code>app/parsers/</code> which holds its JS, CSS, templates and tests.

All parser classes extend a base and only need to implement a small number of interfaces.

• Required
  • parse(log): Accepts a log line string, returns an object of parsed fields.
• Optional
  • buildTemplateContext(template, log): Modify the context object sent to dust.js based on the type of template. Currently there are two types: preview and event. See the extension example to see them in action.
  • buildPreviewText(log): Build the preview string manually (rather than using on a preview template).
  • extractTime(log): The default implementation will detect millisecond/second timestamps and Date.parse()-able strings in log.time values. For incompatible formats, define this function to extract the millisecond timestamp manually.
• Utilities
  • namedCapture(subject, regex): Wrapper around XRegExp named capture expression handling.
  • candidateCapture(subject, candidates): Wrapper around namedCapture that lets you define multiple potential patterns and the first match wins.

See <code>app/parsers/prototype.js</code> for more interface details.

Extending the base class is a simple one-call process via a backbone.js-like extend() function. See the extension example for a working implementation and screenshots of the output content. Or browse any of the modules under <code>app/parsers/</code>.

Create a Pub/Sub listener for log updates

1. Create a module that exports an on(logs) function that receives an array of one or more log objects.

2. Perform any non-native tasks you need.

3. Find this <code>config/app.js</code> section and add the location of your listener module to the subscribers list:

{
  // ...
  mongodb: {
    // ...
    listeners: [
      {
        // Customize this event but do not remove.
        event: 'InsertLog',
        enabled: true,
        subscribers: [
          'app/modules/redis/InsertLogPubSub.js'
        ],
      }
    ],
    // ...
  },
  // ...
}

See <code>app/modules/redis/InsertLogPubSub.js</code> for a working example.

Display a real-time update with Web Notifications

• Enable Desktop Notifications via Timeline drop-down menu.
• Required event attribute(s):
  • WebNotifyApiBody
    • OR WebNotifyApiBodyAttr to specify the attribute to use instead.
• Optional event attribute(s):
  • WebNotifyApiTitle
    • OR WebNotifyApiTitleAttr to specify the attribute to use instead.
  • WebNotifyApiUrl (icon location)
    • OR WebNotifyApiUrlAttr to specify the attribute to use instead.

Configuration

$ cp config/app.js.dist config/app.js

Notes about the main properties:

• sources
  • path: Absolute path to the log file.
  • parser: Parser class/class-file name, e.g. Json or NginxAccess.
  • tags: (Optional) One or more tags to automatically attach to every event.
  • timeAttr: (Optional) By default, mainevent expects parsers to return a time property to represent the event's timestamp. Select a different property name here.
  • To specify a remote host (all are required):
    • sshKey: Ex. /path/to/my/ec2.pem
    • sshPort: Ex. 22
    • sshUser: Ex. ubuntu
    • sshHost: Ex. ec2-135-28-52-91.compute-1.amazonaws.com
  • previewAttr: (Optional) Allows parsers like Json, which do not have preview templates, to know which properties should be included in preview text.
• express
  • mainevent_server: Verify the default port/ip.
  • test_server: Verify the default port/ip.
• mongodb
  • Customize the collection name in collections.event.
  • Select a different pagination maximum in maxResultSize if needed.
  • Add additional indexes if needed. Future versions may automate that process based on metrics.
• redis
  • host/port/options: Passed to createClient() in node_redis.

Main Components

Frontend

mainevent_server.js

$ bin/mainevent_server.js

Has three responsibilities:

1. / serves the backbone.js MVC app via express.js.
2. /api serves JSON data including graph points, event objects and timeline pages via express.js.
3. /socket.io serves real-time timelime updates.

Triggers <code>public/build.js</code> on startup to build the static/ directory.

public/build.js

$ public/build.js [--prod]

• Combines and compresses (--prod) JS/CSS files located in <code>public/</code>.
• Relies on <code>public/js/app.build.js</code> for RequireJS configuration.
• Triggers <code>public/js/templates.build.js</code> to compile dust.js templates.

Outputs all files into static/.

public/js/templates.build.js

Compiles dust.js templates in <code>app/views/</code> and <code>app/parsers/*/templates/</code>.

public/js/app.build.js

RequireJS configuration for client-side dependencies.

Background

bin/tail.js

$ bin/tail.js $ bin/tail.js --help

Spawns tail instances for each source described in <code>config/app.js</code>.

app/graphs/CountAllPartitioned.js

Required by the dashboard.

$ app/graphs/CountAllPartitioned.js --verbose --jobWait 2 --chunkWait 2 --limit 1000 $ app/graphs/CountAllPartitioned.js --help

Generates the cached data used by the graph on /dashboard visualizing total events. Runs continually and sleeps for 1 minute if no new events are read.

Utilities

bin/import.js

$ bin/import.js --parser json --path /var/log/myApp/prod.json --tags myApp,import $ bin/import.js --help

Like <code>tail.js</code> except it processes the entire file. (The file does not need to be described in <code>config/app.js</code>.)

File Layout Notes

• <code>app/</code> : Holds most server-side modules and classes.
  • <code>controllers/</code>: Handlers for express.js routes defined in <code>bin/mainevent_server.js</code>.
  • <code>graphs/</code>: Background scripts which run at intervals to cache point data in Redis.
  • <code>jobs/</code>: Classes used by graphs scripts which define the MapReduce logic.
  • <code>modules/</code>: Covers MongoDB, Redis, static builds and global objects like mainevent.
  • <code>[parsers/](https://github.com/codeactu