Pixel Art Reverse Image Search for OpenGameArt

What does the final search look like?

The final search with an example can be found here.

It looks like this: OpenGameArt Search

Why did I want a reverse image search for OpenGameArt?

I wanted to build a reverse image search for OpenGameArt as Google Image Search and TinEye don't give good results for it. I had previously generated a huge tile map to give an overview of similar images on OpenGameArt, but it wasn't very resource friendly on the web or image browser and had to be split into smaller files, plus it's not searchable in any way, just scrollable. So I wanted a way for people to explore what kind of art is available on OpenGameArt, and landed on using similarity search to browse the image space.

How did I do the crawling?

The first thing I had to do was retrieve the search results for the query I was interested in on OpenGameArt, mostly the 2D art. Then I had to retrieve each HTML page which was in the search results index and parse the HTML for links to files. OpenGameArt contains a lot of archive files like zip and rar files, so I then had to unpack them to get to the images.

For example here is a snippet showing how to parse the content page and get file links:

responseBody = await Common.ReadURIOrCache(blob, Common.BaseURI + page, client);

var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(responseBody);
var htmlBody = htmlDoc.DocumentNode.SelectSingleNode("//body");

foreach (var nNode in htmlBody.Descendants("a"))
{
    if (nNode.NodeType == HtmlNodeType.Element &&
        nNode.Attributes["href"] != null &&
        nNode.Attributes["href"].Value.Contains("/default/files/"))
    {
        msg.Add(HttpUtility.HtmlDecode(nNode.Attributes["href"].Value.Replace(Common.FileURI, "")));
    }
}

Which technology did I use for the crawling and how much did it cost?

I used Azure Functions to do the crawling steps, with some back and forth manual intervention to correct things as needed. Each step had its own queue and then put the job for the next step on the next queue. In the end the invocations cost around 50 USD on Azure, for let's say 10-20 million Function invocations if I remember correctly.

Which alternatives did I investigate?

I tried to use the open source Milvus database, but it crashed on my DigitalOcean server because I didn't have enough memory on it. Then I accidentally and luckily discovered the link to Pinecone in a Hacker News comment section and decided to use that instead, as the trial was free and I didn't have to expand my server memory to use Milvus. In the end I expanded my server anyway, but I didn't try Milvus again (at least not yet).

What data do you need on each image to create a reverse image search?

I used VGG16 feature extraction in my script for this. See the article for more information, but in essence it's 4096 32-bit floating point numbers for each image, which describe various features of the image, say for instance in a very simplified way how many stripes or squares it has or how green it is. But these features are based on neurons in the neural network for VGG16 (which is usually used for image classification), so the features could be more complicated than what is described by simple feature tags. And the reason we need these vectors is that it's easy to use Euclidean distance or cosine similarity or another measure on two vectors to see if they are similar, and then consequently the images are similar. Furthermore there is search technology on these vectors that enable quick search on a large amount of them.

Here's a simplified python script to show how to do the feature extraction:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# vim: ft=python ts=4 sw=4 sts=4 et fenc=utf-8

from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.vgg16 import decode_predictions, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.compiler import xla
import numpy as np
import time
import os
import sys
import PIL
import json
import math
import multiprocessing
from glob import glob
from PIL import Image
from io import BytesIO

model = VGG16(weights='imagenet', include_top=True)
feat_extractor = Model(inputs=model.input, outputs=model.get_layer("fc2").output)

def prepImage(img):
    x = np.array(img.resize((224, 224)).convert('RGB'))
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    return x

def main():
    'entry point'
    fname = 'demo.jpg'
    dt = Image.open(fname)
    pimg = prepImage(dt)

    print("Computing feature vector", fname)
    features = feat_extractor.predict(pimg)
    print(features)

if __name__ == '__main__':
    main()

Here's the output of the script:

emh@frostpunk ~/public_html/ogasearch 0% ./test.py                                                                                                                                                                                                                                                                                                                         (git)-[gh-pages]
2021-04-07 18:48:03.158023: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-04-07 18:48:03.158082: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-07 18:48:07.783109: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-07 18:48:07.783485: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-04-07 18:48:07.783530: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-04-07 18:48:07.783580: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (frostpunk): /proc/driver/nvidia/version does not exist
2021-04-07 18:48:07.784058: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-07 18:48:07.784513: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-07 18:48:08.599925: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
2021-04-07 18:48:09.194634: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
2021-04-07 18:48:09.385612: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
2021-04-07 18:48:13.033066: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
Computing feature vector demo.jpg
2021-04-07 18:48:13.706621: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-04-07 18:48:13.717564: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2199995000 Hz
[[0.        3.1128967 1.5611947 ... 1.2625191 0.7709812 0.       ]]
./test.py  12.20s user 4.66s system 132% cpu 12.731 total

How did I maintain a link between the image URLs and the vector datebase feature vectors?

I also wanted to put all the image URLs into a SQL database in the end, and have a flag for whether I had made the VGG16 feature extraction and whether it was added to the vector database (Milvus or Pinecone. It's essential to be able to map back and forth between an integer primary key, which is used in Pineone, and the URL and perhaps other metadata that belongs to the image, as [Pinecone](https://www.pinecone.io/ doesn't store other metadata than the primary key. In the end I dumbed the SQL database to a tab separated text file and loaded it on query server startup.

How long time did it take?

I think I spent a week in total to run all the code to finish, each step taking on the order of a day or two, crawl, computing feature vectors. I don't remember how much time it took to insert the vectors into the Pinecone database, but I think it was not the most time-consuming step.

Two ways of searching: Words and images

There are two ways of searching, either you can put a keyword, which just plainly (and a bit slowly at O(n)) iterates linearly through the URLs looking for a string match. I stuck with linear search since it's simple to implement and all the URLs are kept in memory anyway so it's not that slow. I dumped all the URLs to a text file and loading it to memory on query server load instead of querying the SQL server each time.
The other way of searching is that you put an image URL, which will run

PixelArtSearch

Install / Use

README