HPI

TLDR: I'm using HPI(Human Programming Interface) package as a means of unifying, accessing and interacting with all of my personal data.

It's a Python library (named my), a collection of modules for:

• social networks: posts, comments, favorites, searches
• shell/program histories (zsh, bash, python, mpv, firefox)
• programming (github/commits)
• instant messaging
• media histories (movies, TV shows, music, video game achievements/history); see https://purarue.xyz/feed/

Why?

This is built on top of karlicoss/HPI. These are all additional modules which aren't present in that repository - this is installed alongside the upstream repository (meaning you can use both modules from upstream and here simultaneously), see #install

My Modules

• my.zsh and my.bash, access to my shell history w/ timestamps
• my.mail.imap and my.mail.mbox to parse local IMAP sync's of my mail/mbox files -- see doc/MAIL_SETUP.md
• my.mpv.history_daemon, accesses movies/music w/ activity/metadata that have played on my machine, facilitated by a mpv history daemon
• my.discord.data_export, parses ~1,000,000 messages/events from the discord data export, parser here
• my.todotxt.active to parse my current todo.txt file; my.todotxt.git_history tracks my history using backups of those files in git_doc_history
• my.rss.newsboat, keeps track of when I added/removed RSS feeds (for newsboat)
• my.ipython, for timestamped python REPL history
• my.ttt, to parse shell/system history tracked by ttt
• my.activitywatch.active_window, to parse active window events (what application I'm using/what the window title is) using window_watcher and activitywatch on android
• my.chess.export, to track my chess.com/lichess.org games, using chess_export
• my.trakt.export, providing me a history/my ratings for Movies/TV Show (episodes) using traktexport
• my.listenbrainz.export, exporting my music listening history from ListenBrainz (open-source Last.fm) using listenbrainz_export
• my.offline.listens, for offline music listen history, using offline_listens
• my.mal.export, for anime/manga history using malexport
• my.grouvee.export, for my video game history/backlog using grouvee_export
• my.runelite.screenshots, parses data from the automatic runelite screenshots
• my.minecraft.advancements, parses advancement (local achievement data) from the ~/.minecraft directory
• my.project_euler, when I solved Project Euler problems
• my.linkedin.privacy_export, to parse the privacy export from linkedin
• my.scramble.history for merged (timed) rubiks cube solves from multiple sources, using scramble_history

'Historical' Modules

These are modules to parse GDPR exports/data from services I used to use, but don't anymore. They're here to provide more context into the past.

• my.apple.privacy_export, parses Game Center and location data from the apple privacy export
• my.facebook.gdpr, to parse the GDPR export from Facebook
• my.league.export, gives League of Legends game history using lolexport
• my.steam.scraper, for steam achievement data and game playtime using steamscraper
• my.piazza.scraper, parsing piazza (university forum) posts using piazza-scraper
• my.blizzard.gdpr, for general battle.net event data parsed from a GDPR export
• my.skype.gdpr to parse a couple datetimes from the Skype GDPR export (seems all my data from years ago is long gone)
• my.spotify.gdpr, to parse the GDPR export from Spotify, mostly to access songs from my playlists from years ago
• my.twitch, merging the data export and my messages parsed from the overrustle logs dump

See here for my HPI config

Promnesia Sources for these HPI modules

I also have some more personal scripts/modules in a separate repo; HPI-personal

In-use from karlicoss/HPI

• my.browser, to parse browser history using browserexport
• my.google.takeout.parser, parses lots of (~500,000) events (youtube, searches, phone usage, comments, location history) from google takeouts, using google_takeout_parser
• my.coding.commits to track git commits across the system
• my.github to track github events/commits and parse the GDPR export, using ghexport
• my.reddit, get saved posts, comments. Uses rexport to create backups of recent activity periodically, and pushshift to get old comments.
• my.smscalls, exports call/sms history using SMS Backup & Restore
• my.stackexchange.stexport, for stackexchange data using stexport

Partially in-use/with overrides:

• my.location, though since I also have some locations from apple.privacy_export, I have a my.location.apple which I then merge into my.location.all in my overridden all.py file on my personal repo
• similarly, I do use my.ip and my.location.via_ip from upstream, but I have overridden all.py and module files here

'Overriding' an all.py file means replacing the all.py from upstream repo (this means it can use my sources here to grab more locations/ips, since those don't exist in the upstream). For more info see reorder_editable, and the module design docs for HPI, but you might be able to get the gist by comparing:

• my.location.all in karlicoss/HPI
• my.location.all in purarue/HPI-personal

Since I've mangled my PYTHONPATH (see reorder_editable), it imports from my repo instead of karlicoss/HPI. all.py files tend to pretty small -- so overriding/changing a line to add a source is the whole point.

See here for an explanation of how that works.

Companion Tools/Libraries

Disregarding tools which actively collect data (like ttt/window_watcher) or repositories which have their own exporter/parsers which are used here, there are a couple other tools/libraries I've created for this project:

• ipgeocache - for any IPs gathered from data exports, provides geolocation info, so I have partial location info going back to 2013
• sqlite_backup - to safely copy/backup application sqlite databases that may currently be in use
• git_doc_history - a bash script to copy/backup files into git history, with a python library to help traverse and create a history/parse diffs between commits
• HPI_API - automatically creates a JSON API/server for HPI modules
• url_metadata - caches youtube subtitles, url metadata (title, description, image links), and a html/plaintext summary for any URL

I also use this in my_feed, which creates a feed of media/data using HPI, live at https://purarue.xyz/feed/

Ad-hoc and interactive

Some basic examples.

When was I most using reddit?

import collections, my.reddit.all, pprint

pprint.pprint(collections.Counter([c.created.year for c in my.reddit.all.comments()]))
Counter({2016: 3288, 2017: 801, 2015: 523, 2018: 209, 2019: 65, 2014: 4, 2020: 3})

Most common shell commands?

import collections, pprint, my.zsh

# lots of these are git-related aliases
pprint.pprint(
    collections.Counter([c.command for c in my.zsh.history()]).most_common(10)
)
[
    ("ls", 51059),
    ("gst", 11361),
    ("ranger",