Browserexport
backup and parse your browser history databases (chrome, firefox, safari, and other chrome/firefox derivatives)
Install / Use
/learn @purarue/BrowserexportREADME
browserexport
- Supported Browsers
- Install
- Usage
- Serializing to JSON
- Shell Completion
- Usage with HPI
- Library Usage
- Comparisons with promnesia
- Contributing
This:
- locates and backs up browser history by copying the underlying database files to some directory you specify
- can identify and parse the resulting database files into some common schema:
Visit:
url: the url
dt: datetime (when you went to this page)
metadata:
title: the <title> for this page
description: the <meta description> tag from this page
preview_image: 'main image' for this page, often opengraph/favicon
duration: how long you were on this page
metadata is dependent on the data available in the browser (e.g. firefox has preview images, chrome has duration, but not vice versa)
Supported Browsers
This currently supports:
This can probably extract visits from other Firefox/Chromium-based browsers, but it doesn't know how to locate them to save them
Install
python3 -m pip install --user browserexport
Requires python3.10+
Usage
save
Usage: browserexport save [OPTIONS]
Backs up a current browser database file
Options:
-b, --browser
[chrome | firefox | opera | safari | brave | qutebrowser |
waterfox | librewolf | floorp | chromium | vivaldi | palemoon |
arc | edge | edgedev]
Browser name to backup history for
--pattern TEXT Pattern for the resulting timestamped filename, should include an
str.format replacement placeholder for the date [default:
browser_name-{}.extension]
-p, --profile TEXT Use to pick the correct profile to back up. If unspecified, will assume a
single profile [default: *]
--path FILE Specify a direct path to a database to back up
-t, --to DIRECTORY Directory to store backup to. Pass '-' to print database to STDOUT
[required]
-h, --help Show this message and exit.
Must specify one of --browser, or --path
After your browser history reaches a certain size, browsers typically remove old history over time, so I'd recommend backing up your history periodically, like:
$ browserexport save -b firefox --to ~/data/browsing
$ browserexport save -b chrome --to ~/data/browsing
$ browserexport save -b safari --to ~/data/browsing
That copies the sqlite databases which contains your history --to some backup directory.
If a browser you want to backup is Firefox/Chrome-like (so this would be able to parse it), but this doesn't support locating it yet, you can directly back it up with the --path flag:
$ browserexport save --path ~/.somebrowser/profile/places.sqlite \
--to ~/data/browsing
The --pattern argument can be used to change the resulting filename for the browser, e.g. --pattern 'places-{}.sqlite' or --pattern "$(uname)-{}.sqlite". The {} is replaced by the browser name.
Feel free to create an issue/contribute a browser file to locate the browser if this doesn't support some browser you use.
Can pass the --debug flag to show sqlite_backup logs
$ browserexport --debug save -b firefox --to .
[D 220202 10:10:22 common:87] Glob /home/username/.mozilla/firefox with */places.sqlite (non recursive) matched [PosixPath('/home/username/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite')]
[I 220202 10:10:22 save:18] backing up /home/username/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite to /home/username/Repos/browserexport/firefox-20220202181022.sqlite
[D 220202 10:10:22 core:110] Source database files: '['/tmp/tmpcn6gpj1v/places.sqlite', '/tmp/tmpcn6gpj1v/places.sqlite-wal']'
[D 220202 10:10:22 core:111] Temporary Destination database files: '['/tmp/tmpcn6gpj1v/places.sqlite', '/tmp/tmpcn6gpj1v/places.sqlite-wal']'
[D 220202 10:10:22 core:64] Copied from '/home/username/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite' to '/tmp/tmpcn6gpj1v/places.sqlite' successfully; copied without file changing: True
[D 220202 10:10:22 core:64] Copied from '/home/username/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite-wal' to '/tmp/tmpcn6gpj1v/places.sqlite-wal' successfully; copied without file changing: True
[D 220202 10:10:22 core:230] Running backup, from '/tmp/tmpcn6gpj1v/places.sqlite' to '/home/username/Repos/browserexport/firefox-20220202181022.sqlite'
[D 220202 10:10:22 save:14] Copied 1840 of 1840 database pages...
[D 220202 10:10:22 core:246] Executing 'wal_checkpoint(TRUNCATE)' on destination '/home/username/Repos/browserexport/firefox-20220202181022.sqlite'
For Firefox Android Fenix, the database has to be manually backed up (probably from a rooted phone using termux) from data/data/org.mozilla.fenix/files/places.sqlite.
inspect/merge
These work very similarly, inspect is for a single database, merge is for multiple databases.
Usage: browserexport merge [OPTIONS] SQLITE_DB...
Extracts visits from multiple sqlite databases
Provide multiple sqlite databases as positional arguments, e.g.:
browserexport merge ~/data/firefox/*.sqlite
Drops you into a REPL to access the data
Pass '-' to read from STDIN
Options:
-s, --stream Stream JSON objects instead of printing a JSON list
-j, --json Print result to STDOUT as JSON
-h, --help Show this message and exit.
As an example:
browserexport --debug merge ~/data/firefox/* ~/data/chrome/*
[D 210417 21:12:18 merge:38] merging information from 24 sources...
[D 210417 21:12:18 parse:19] Reading visits from /home/username/data/firefox/places-20200828223058.sqlite...
[D 210417 21:12:18 common:40] Chrome: Running detector query 'SELECT * FROM keyword_search_terms'
[D 210417 21:12:18 common:40] Firefox: Running detector query 'SELECT * FROM moz_meta'
[D 210417 21:12:18 parse:22] Detected as Firefox
[D 210417 21:12:19 parse:19] Reading visits from /home/username/data/firefox/places-20201010031025.sqlite...
[D 210417 21:12:19 common:40] Chrome: Running detector query 'SELECT * FROM keyword_search_terms'
....
[D 210417 21:12:48 common:40] Firefox: Running detector query 'SELECT * FROM moz_meta'
[D 210417 21:12:48 common:40] Safari: Running detector query 'SELECT * FROM history_tombstones'
[D 210417 21:12:48 parse:22] Detected as Safari
[D 210417 21:12:48 merge:51] Summary: removed 3001879 duplicates...
[D 210417 21:12:48 merge:52] Summary: returning 334490 visit entries...
Use vis to interact with the data
[1] ...
You can also read from STDIN, so this can be used in conjunction with save, to merge databases you've backed up and combine your current browser history:
browserexport save -b firefox -t - | browserexport merge --json --stream - ~/data/browsing/* >all.jsonl
Or, use process substitution to save multiple dbs in parallel and then merge them:
$ browserexport merge <(browserexport save -b firefox -t -) <(browserexport save -b chrome -t -)
Logs are hidden by default. To show the debug logs set export BROWSEREXPORT_LOGS=10 (uses logging levels) or pass the --debug flag.
JSON
To dump all that info to JSON:
$ browserexport merge --json ~/data/browsing/*.sqlite > ./history.json
du -h history.json
67M history.json
Or, to create a quick searchable interface, using jq and fzf:
browserexport merge -j --stream ~/data/browsing/*.sqlite | jq '"\(.url)|\(.metadata.description)"' | awk '!seen[$0]++' | fzf
Merged files like history.json can also be used as inputs files themselves, this reads those by mapping the JSON onto the Visit schema directly.
In addition to .json files, this can parse .jsonl (JSON lines) files, which are files which contain newline delimited JSON objects. This allows you to parse JSON objects one at a time, instead of loading the entire file into memory. The .jsonl file can be generated with the --stream flag:
browserexport merge --stream --json ~/data/browsing/*.sqlite > ./history.jsonl
Additionally, this can parse compressed JSON/JSONL files (using [kompress](https://github.com/
