Selfspy

What is this?

Selfspy is a daemon for Unix/X11, (thanks to @ljos!) Mac OS X and (thanks to @Foxboron) Windows, that continuously monitors and stores what you are doing on your computer. This way, you can get all sorts of nifty statistics and reminders on what you have been up to. It is inspired by the Quantified Self-movement and Stephen Wolfram's personal key logging.

See Example Statistics, below, for some of the fabulous things you can do with this data.

Windows

Due to Windows libs needing a external compiler to compile libs, some libs won't compile on all computers.
These are the sinners*:
pyHook==1.5.1
pyCrypto==2.5
They are added too the windows-requirements.txt, but IF you fail to build these libs, here are the precompiled binaries. pyWin32 is needed for some library dependency, pip can't install pyWin32, so please use the binary below.
pyHook: http://sourceforge.net/projects/pyhook/files/pyhook/
pyCrytpo: http://www.voidspace.org.uk/python/modules.shtml#pycrypto
pyWin32: http://sourceforge.net/projects/pywin32/files/pywin32/

*SQLAlchemy does compile without the external compiler, but it uses a clean Python version which might slow things down.

Installing Selfspy

If you run ArchLinux, here is an AUR package which may be up-to-date: https://aur.archlinux.org/packages/selfspy-git/

To install manually, either clone the repository from Github (git clone git://github.com/gurgeh/selfspy), or click on the Download link on http://github.com/gurgeh/selfspy/ to get the latest Python source.

Selfspy is only tested with Python 2.7 and has a few dependencies on other Python libraries that need to be satisfied. These are documented in the requirements.txt file. If you are on Linux, you will need subversion installed for pip to install python-xlib. If you are on Mac, you will not need to install python-xlib at all. Python-xlib is currently a tricky package to include in the requirements since it is not on PyPi.

pip install svn+https://python-xlib.svn.sourceforge.net/svnroot/python-xlib/tags/xlib_0_15rc1/ # Only do this step on Linux!
python setup.py install

You will also need the Tkinter python libraries. On ubuntu and debian

sudo apt-get install python-tk

On FreeBSD

cd /usr/ports/x11-toolkits/py-tkinter/
sudo make config-recursive && sudo make install clean

There is also a simple Makefile. Run make install as root/sudo, to install the files in /var/lib/selfspy and also create the symlinks /usr/bin/selfspy and /usr/bin/selfstats.

Report issues here: https://github.com/gurgeh/selfspy/issues

General discussion here: http://ost.io/gurgeh/selfspy

OS X

In OS X you also need to enable access for assistive devices. To do that in <10.9 there is a checkbox in System Preferences > Accessibility, in 10.9 you have to add the correct application in System Preferences > Privacy > Accessibility.

Running Selfspy

You run selfspy with selfspy. You should probably start with selfspy --help to get to know the command line arguments. As of this writing, it should look like this:

usage: selfspy [-h] [-c FILE] [-p PASSWORD] [-d DATA_DIR] [-n]
               [--change-password]

Monitor your computer activities and store them in an encrypted database for
later analysis or disaster recovery.

optional arguments:
  -h, --help            show this help message and exit
  -c FILE, --config FILE
                        Config file with defaults. Command line parameters
                        will override those given in the config file. The
                        config file must start with a "[Defaults]" section,
                        followed by [argument]=[value] on each line.
  -p PASSWORD, --password PASSWORD
                        Encryption password. If you want to keep your database
                        unencrypted, specify -p "" here. If you don't specify
                        a password in the command line arguments or in a
                        config file, a dialog will pop up, asking for the
                        password. The most secure is to not use either command
                        line or config file but instead type it in on startup.
  -d DATA_DIR, --data-dir DATA_DIR
                        Data directory for selfspy, where the database is
                        stored. Remember that Selfspy must have read/write
                        access. Default is ~/.selfspy
  -n, --no-text         Do not store what you type. This will make your
                        database smaller and less sensitive to security
                        breaches. Process name, window titles, window
                        geometry, mouse clicks, number of keys pressed and key
                        timings will still be stored, but not the actual
                        letters. Key timings are stored to enable activity
                        calculation in selfstats. If this switch is used,
                        you will never be asked for password.
  -r, --no-repeat       Do not store special characters as repeated
                        characters.
  --change-password     Change the password used to encrypt the keys columns
                        and exit.

Everything you do is stored in a Sqlite database in your DATA_DIR. Things that you type (passwords, for example) are generally too sensitive to leave in plain text, so they are encrypted with the supplied password. Other database columns, like process names and window titles, are not encrypted. This makes it faster and easier to search for them later.

Unless you use the --no-text flag, selfspy will store everything you type in two <a href="http://en.wikipedia.org/wiki/Blowfish_(cipher)">Blowfish</a> encrypted columns in the database.

Normally you would like Selfspy to start automatically when you launch X. How to do this depends on your system, but it will normally mean editing ~/.xinitrc or ~/.xsession. If you run KDE, ~/.kde/Autostart, is a good place to put startup scripts. When run, Selfspy will immediately spawn a daemon and exit.

Running on login in OS X

If you want selfspy to start automatically on login you need to copy the com.github.gurgeh.selfspy.plist file to ~/Library/LaunchAgents/.

Example Statistics

"OK, so now all this data will be stored, but what can I use it for?"

While you can access the Sqlite tables directly or, if you like Python, import models.py from the Selfspy directory and use those SqlAlchemy classes, the standard way to query your data is through selfstats.

Here are some standard use cases:

"Damn! The browser just threw away everything I wrote, because I was not logged in."

selfstats --back 30 m --showtext

Show me everything I have written the last 30 minutes. This will ask for my password, in order to decrypt the text.

"Hmm.. what is my password for Hoolaboola.com?"

selfstats -T "Hoolaboola" -P Google-chrome --showtext

This shows everything I have ever written in Chrome, where the window title contained something with "Hoolaboola". The regular expressions are case insensitive, so I actually did not need the caps. If I have written a lot on Hoolaboola, perhaps I can be more specific in the title query, to only get the login page.

"I need to remember what I worked on a few days ago, for my time report."

selfstats --date 10 --limit 1 d -P emacs --tkeys

What buffers did I have open in Emacs on the tenth of this month and one day forward? Sort by how many keystrokes I wrote in each. This only works if I have set Emacs to display the current buffer in the window title. In general, try to set your programs (editors, terminals, web apps, ...) to include information on what you are doing in the window title. This will make it easier to search for later.

On a related but opposite note: if you have the option, remove information like "mails unread" or "unread count" (for example in Gmail and Google Reader) from the window titles, to make it easier to group them in --tactive and --tkeys.

"Also, when and how much have I used my computer this last week?"

selfstats -b 1 w --periods 180

This will display my active time periods for the last week. A session is considered inactive when I have not clicked or used the keyboard in 180 seconds. Increase that number to get fewer and larger sessions listed.

"How effective have I been this week?"

selfstats -b 1 w --ratios

This will show ratios informing me about how much I have written per active second and how much I have clicked vs used the keyboard. For me, a lot of clicking means too much browsing or inefficient use of my tools. Track these ratios over time to get a sense of what is normal for you.

"I remember that I wrote something to her about the IP address of our printer a few months ago. I can't quite remember if it was a chat, a tweet, a mail, a facebook post, or what.. Should I search them separately? No."

selfstats --body printer -s --back 40 w

Show the texts where I have used the word printer in the last 10 weeks. If it turns out that the actual IP adress is not in the same text chunk as when you wrote "printer", you can note the row ID and use --id (or --date and --clock) and --limit to show what else you wrote around that time.

"What programs do I use the most?"

selfstats --pactive

List all programs I have ever used in order of time active in them.

"Which questions on the website Stack Overflow did I visit yesterday?"

./selfstats -T "Stack Overflow" -P Google-chrome --back 32 h --tactive

List all window titles that contained "Stack Overflow" the last 32 hours. Sort by time active. I add the sorting, not only because I want them sorted, but because otherwise the listing would