Birdwatcher

Birdwatcher is a data analysis and OSINT framework for Twitter. Birdwatcher supports creating multiple workspaces where arbitrary Twitter users can be added and their Tweets harvested through the Twitter API for offline storage and analysis. Birdwatcher comes with several modules which can be envoked to further enrich collected data or work with it, e.g. Retrieving user's Klout score, generating social graphs between users and weighted word clouds based on their Tweets.

Installation

1. Ruby

Birdwatcher is written in Ruby and requires at least version 1.9.3 or above. To check which version of Ruby you have installed, simply run ruby --version in a terminal.

Should you have an older version installed, it is very easy to upgrade and manage different versions with the Ruby Version Manager (RVM). Please see the RVM website for installation instructions.

2. RubyGems

Birdwatcher is packaged as a Ruby gem to make it easy to install and update. To install Ruby gems you'll need the RubyGems tool installed. To check if you have it already, type gem in a Terminal. If you got it already, it is recommended to do a quick gem update --system to make sure you have the latest and greatest version. In case you don't have it installed, download it from here and follow the simple installation instructions.

3. PostgreSQL

Birdwatcher uses a PostgreSQL database to store all its data. If you are setting up Birdwatcher in the Kali linux distribution you already have it installed, you just need to make sure it's running by executing service postgresql start and perhaps install a dependency with apt-get install libpq-dev in a terminal. Here's an excellent guide on how to install PostgreSQL on a Debian based Linux system. If you are setting up Birdwatcher on a Mac, the easiest way to install PostgreSQL is with Homebrew. Here's a guide on how to install PostgreSQL with Homebrew.

3.1 PostgreSQL user and database

You need to set up a user and a database in PostgreSQL for Birdwatcher. Execute the following commands in a terminal:

sudo su postgres # Not necessary on Mac OS X
createuser -s birdwatcher --pwprompt
createdb -O birdwatcher birdwatcher

You now have a new PostgreSQL user with the name birdwatcher and with the password you typed into the prompt. You also created a database with the name birdwatcher which is owned by the birdwatcher user.

4. Graphviz

Some Birdwatcher modules use Graphviz to generate visual graphs and other things. On a Mac you can install Graphviz with homebrew by typing brew update && brew install graphviz in a terminal. On a Debian based Linux distro, Graphviz can be installed by typing sudo apt-get update && sudo apt-get install graphviz in a terminal.

5. ImageMagick

Some Birdwatcher modules use ImageMagick to generate images. On a Mac you can install Imagemagick with homebrew by typing brew update && brew install imagemagick in a terminal. On a Debian based Linux distro, ImageMagick can be installed by typing sudo apt-get update && sudo apt-get install libmagickwand-dev imagemagick in a terminal.

6. Birdwatcher

Finally with all the dependencies in place, Birdwatcher can now be installed with a simple command in a terminal:

$ gem install birdwatcher

This will download and set up Birdwatcher and install all its code depencies.

Configuration

Birdwatcher needs to know a bit about what database to connect to as well as API keys to use for API communication. For Twitter, you will need to register an application in order to get a consumer key and consumer secret. Head over to apps.twitter.com and set up your application. You don't need to have a valid callback URL as we won't be doing any OAuth authentication.

To make Birdwatcher even more useful, it is recommended to also obtain an API key for the Klout API which is required for some modules to work. If you don't intend to use Klout modules, you can of course skip this step.

On the first run, Birdwatcher will automatically start a configuration wizard where it will ask for the configuration it needs. If you have PostgreSQL connection details, Twitter consumer key & secret and optional Klout API key, you can start Birdwatcher for the first time:

$ birdwatcher

Enter the details into the configuration wizard:

As can be seen from the above screenshot, Birdwatcher supports multiple Twitter and Klout keys. If you configure Birdwatcher with several keys it will randomly shuffle between them when communicating with APIs to potentially avoid any rate limiting issues on extensive use.

Birdwatcher will save its configuration to ~/.birdwatcherrc. Be careful not to push this file up to any public code repositories!

System pager

Birdwatcher pages long command output with the operating system's default pager command (usually less) however the command output can be colored and will show up strangely if the pager is not configured to render terminal colors. It is advised to add the following to your ~/.bash_profile or similar file:

# Get color support for 'less'
export LESS="--raw-control-chars"
export PAGER="less"

This will set up less as your default pager command as well as configure less to support terminal colors. Execute the command source ~/.bash_profile to apply the configuration immediately instead of next time you open a terminal.

Usage

Birdwatcher is built as a console and if you have any experience with other frameworks such as Metasploit or Recon-ng, you should feel right at home as Birdwatcher has many of the same concepts and commands.

Workspaces

Birdwatcher, like Metasploit and Recon-ng, works with the concept of Workspaces. Workspaces enable you to segment and manage users and data stored in the database. You can use workspaces to create logical separation between different users. For example, you may want to create a workspace for a company, a department or for a specific topic.

Birdwatcher will always show the currently active workspace inside the square brackets on the command prompt:

There will always be a default workspace with the name default which might be enough if you plan to use Birdwatcher for a small group of Twitter users. Let's create a new workspace called top5 that we can use for experimentation:

birdwatcher[default]>  workspace create top5
[+] Created workspace: top5
birdwatcher[top5]>

The workspace create command created a new workspace and automatically made it the active workspace. Any user we add and any data we collect now will only be available in the top5 workspace.

If we want to navigate between workspaces, we can do so by using the workspace command again:

birdwatcher[top5]> workspace use default
[+] Now using workspace: default
birdwatcher[default]> workspace use top5
[+] Now using workspace: top5
birdwatcher[top5]>

With the above commands we switched over to the default workspace and then back again to the top5 workspace.

Commands

The core of the Birdwatcher framework is of course its commands. You already got an introduction to the workspace command in the previous section, but there are many more. One of the most important commands is help:

birdwatcher[top5]> help
[+] Available commands:

    back               		Unloads current module
    exit               		Exit Birdwatcher
    help [COMMAND]     		Show help and detailed command usage
    irb                		Start an interactive Ruby shell
    module ACTION      		Show modules
    query QUERY        		Execute SQL query
    query_csv QUERY    		Execute SQL query and return result as CSV
    resource FILE      		Execute commands from a resource file
    run                		Run current module
    schema [TABLE_NAME]		Show schema for database table
    set OPTION VALUE   		Set module option
    shell COMMAND      		Execute shell command
    show DETAILS       		Show module details and options
    status [ACTION]    		Manage statuses
    unset OPTION       		Unset module option
    use MODULE_PATH    		Load specified module
    user [ACTION]      		Manage users
    workspace [ACTION] 		Manage workspaces

birdwatcher[top5]>

The help command simply lists the available commands with a short description of what they do. If we want to get more information on a specific command, e.g. the workspace command, we can execute the following:

birdwatcher[top5]> help workspace

Workspaces enable you to segment and manage users and data stored in the database.
You can use workspaces to create logical separation between different users.
For example, you may want to create a workspace for a company, a department or
for a specific topic.

There will always be a default workspace with the name default which might be enough
if you plan to use Birdwatcher for a small group of Twitter users.

USAGE:

List available workspaces:
  workspace list

Create a new workspace:
  workspace create NAME [DESCRIPTION]

Switch to a workspace:
  workspace use NAME

D