SmashScan

This project is intended to perform video analysis on pre-recorded footage of the game Super Smash Bros Melee. Once the project has progressed enough, I plan to integrate it with the SmashVods video database.

TODO List

If you are interested in helping out, join this repository's Slack and send a message.

• Research ways to detect the percent counter (Either template matching, OCR, or neural networks)
• Research ways to detect stock icons (Either template matching, feature matching, or neural networks)
• Research ways to detect the timer (Either template matching, OCR, or neural networks)
• Add additional game capabilities (SSB64, SSB4, PM, Rivals)

Milestones

1. Train a neural network (DarkNet) to search for the six legal SSBM stages, regardless of a stream overlay.
2. Improve video download speed and create a stage-timeline object to represent SSBM matches.
3. Added non-stage training data to significantly improve stage-detection results.
4. Added percent template matching to segment the video for quicker stage-detection.

Instalation Guide

1. Create a Python 3.6 environment (required by tensorflow) locally or in a conda environment. Install the following python packages; The version numbers are not necessarily required, I just provided them as a reference for when I wrote this guide: numpy (1.11.0), matplotlib (1.5.1), cython (0.28.5), opencv-python (3.4.2.17), youtube-dl (2018.8.22 - 2018 version much faster than 2015-2017).
2. Install tensorflow if you don't have an NVIDIA GPU. Otherwise, install tensorflow-gpu. I followed Mark Jay's tutorial for Ubuntu 18.04.
   • CUDA 9.0 Download Docs
   • CUDNN 7.4.2 Download Docs
   • tensorflow-gpu (>=1.9.0) Download Docs
3. Install the DarkFlow repository globally. I followed Mark Jay's tutorials to understand the basics of the DarkFlow.
4. Clone this repo and create an annotations, output, and videos folder inside. The .gitignore doesn't like committing "empty" folders.

Testing Guide

Obtain the YouTube video-id for the video you wish to test on. The video-id is the set of characters after the "v=" in a YouTube video URL. Afterwards, use download.py to save the video to the videos directory.

# https://www.youtube.com/watch?v=dQw4w9WgXcQ
python download.py dQw4w9WgXcQ example-video-name.mp4

Use test.py to draw bounding boxes on the video. This uses the pre-trained weights (.pb) in the cfg folder.

python test.py example-video-name.mp4

Training Guide

Obtain the YouTube video-id for the video you wish to train on. The video-id is the set of characters after the "v=" in a YouTube video URL. Afterwards, use download.py to save the video to the videos directory.

# https://www.youtube.com/watch?v=dQw4w9WgXcQ
python download.py dQw4w9WgXcQ example-video-name.mp4

Use label.py to save frames to the images directory and labels to the annotations directory. The key-bindings used for label.py are listed below.

python label.py example-video-name.mp4

, - move backward 1 frame
. - move forward 1 frame
direction - move 10 frames
alt+direction - move 100 frames
ctrl+direction - move 1000 frames

left-click -> release - draw bounding box
r - begin recording once you are satisfied with the box drawn
enter - end recording and specify stage number in terminal
` - exit the labeler and save annotations & frames
q - exit the labeler without saving annotations & frames

Opening the images folder freezes my computer because of how many images it tries to display at once. If your running Linux, a useful command to run to see the most recently updated file in a directory is as follows.

ls -t images | head -1

# Eye of GNOME graphics viewer. Just an image viewer program.
eog 123456.png

Once you are satisfied with the new labels, move the images and annotations folders into the train folder of your darkflow installation directory. Then run the following two commands in separate terminals.

# Training command.
flow --model cfg/tiny-yolo-voc-6c.cfg --load bin/tiny-yolo-voc.weights --train --annotation train/annotations --dataset train/images --epoch 100 --gpu 1.0 --summary summary

# Optional command to track the progress visually.
tensorboard --logdir summarytrain

After some time, you can cancel training by hitting ctrl+c. The DarkFlow trainer saves checkpoints every couple minutes, so you aren't losing much progress. If I plan on training for a long period of time (over 12 hours), I'll typically set the checkpoints to be saved over a longer time span. AKA, I will add the flag --save 40000. which will save checkpoints every 2500 iterations (40000/16, where batch size = 16).

This is because DarkFlow has an issue with gradient descent over long periods of time and training will get stuck at NaN. DarkFlow's default.py recommends rmsprop, but I tried adam and sgd with no success. It seems to be a common issue with the trainer from the issues I've read on the repository :/

Next, convert the most recent checkpoint into something usable by SmashScan.

flow --model cfg/tiny-yolo-voc-6c.cfg --load -1 --savepb

The .pb (Protocol Buffer) format is a method for serializing structured data. The DarkFlow library saves the .meta and .pb files to the built_graph directory. Copy both of these files into the cfg directory of the SmashScan directory.

Label History

Below are a list of tournaments I trained on for the initial release. The size of these annotation-image pairs is >10GB, so I don't plan on uploading them anytime soon. This list is mainly for referencing what tournaments have been used.

| Tournament | Annotation Range | Frames | | :--------: | :----------: | :--------------: | | GOML 2018 | 1 - 5019 | 5019 | | EVO 2018 | 5020 - 8274 | 3255 | | Low Tier City 6 | 8275 - 9246 | 972 | | Smash Factor 7 | 9247 - 12485 | 3239 | | Saints Gaming Live 2018 | 12486 - 14019 | 1534 | | Smash N Splash 4 | 14020 - 15720 | 1701 | | Dreamhack Austin 2018 | 15721 - 17299 | 1579 | | Combo Breaker 2018 | 17300 - 19091 | 1792 | | Momocon 2018 | 19092 - 20562 | 1471 | | Pound Underground 2018 | 20563 - 22747 | 2185 | | Smash Summit 6 | 22747 - 24852 | 2106 | | Aegis 2018 | 24853 - 26005 | 1153 | | Flatiron 3 | 26006 - 27383 | 1378 | | Noods Oakland | 27384 - 29554 | 2171 | | The Mango | 29555 - 30976 | 1422 | | No Fun Allowed 2 | 30977 - 33133 | 2157 | | The Gang | 33134 - 34861 | 1730 | | Noods Noods Noods | 34862 - 36760 | 1899 | | Valhalla | 36761 - 38503 | 1743 | | Player Cams | 38503 - 41547 | 3044 | | Holiday Bash 2017 | 41547 - 42628 | 1082 | | Smash Valley VII | 42629 - 43944 | 1316 | | Heir 5 | 43945 - 45062 | 1117 | | Momocon 2018 (R) | 45063 - 46170 | 1108| | No Fun Allowed 2 (R) | 46171 - 48285 | 2115 | | Poi Poundaz | 48286 - 49782 | 1497 |