Typescraper

Typscraper is an open source scraper for fetching precise latency data from Typeracer.

You can read about this project on Medium:

Part 1: Data Scraping and Cleaning

Part 2: Exploration and Analytics

Part 3: Modeling and Estimation

Usage

If you want to reuse the code, you will need to download PostgresSQL. If you use MacOS, the easiest way to install is through Homebrew

brew install postgressql

Once installed, you can execute ./create_db in the project root directory to set up all of the tables and permissions in the database.

To populate the database with latency stats, run python typescraper %USERNAME where %USERNAME is the username of the individual you want to collect latency data from. I would recommend using conda or virtualenv to keep the development environment clean.

To add in keyboard layouts, run python keyboardloader.py %CONFIG where %CONFIG is a csv file found under the configs directory. configs/qwerty/qwerty.csv and configs/dvorak/dvorak.csv are included in the repository.

Lastly, to add WPM and Accuracy data, run python wpmscraper %USERNAME