Neural Network in All Languages <!-- omit in toc -->

<img alt="Brain image" src="https://github.com/dlidstrom/NeuralNetworkInAllLangs/raw/main/doc/networks.png" width="400px">

🔥 Latest news

• added Zig implementation (thanks @Rojuinex!)
• added Java implementation (@dlidstrom)
• fixed C compilation in Linux (thanks @LordMhri!)

See Language Implementations for more detail.

🙌 Seeking assistance! I'm looking for help to add support for missing languages. If you can contribute, I'll gladly accept a PR and give proper credit 💫. It's simpler than you might expect. Just take a look at one of the existing implementations—it's mostly a few for loops. No need to worry about adding tests; I can help with that part.

• 1. Introduction
• 2. Usage
• 3. Training
  • 3.1. Logical Functions
    • 3.1.1. Lithmus Test
  • 3.2. Hand Written Digits
• 4. Learning
• 5. Implementation Goals
  • 5.1. Simple Random Number Generator
  • 5.2. License
  • 5.3. Language Implementations
    • 5.3.1. Sample Output
• 6. Reference Implementation
  • 6.1. Inputs and Randomized Starting Weights
  • 6.2. Forward Propagation
  • 6.3. Backpropagation
  • 6.4. Weight Updates
• 7. Using this in your own solution
• 8. References
• 9. Stargazers over time

1. Introduction

This repository aims to implement a vanilla neural network in all major programming languages. It is the "hello world" of ai programming. We will implement a fully connected network with a single hidden layer using the sigmoid activation function for both the hidden and the output layer. This kind of network can be used to do hand writing recognition, or other kinds of pattern recognitions, categorizations, or predictions. This is intended as your entry level into ai programming, i.e. for the enthusiast or hobby programmer (you and me). More advanced use cases should look elsewhere as there are infinitely more powerful methods available for the professional.

Disclaimer! Do not expect blazing fast performance. If you have such requirements or expectations then you should definitely look elsewhere. Stay here if you want to learn more about implementing a neural network!

We do not aim to justify the math involved (see [1] if you're interested). We prefer to focus on the code itself and will happily copy a solution from one programming language to another without worrying about the theoretical background.

2. Usage

These usage examples are taken directly from our test implementations. The general flow is to prepare a dataset, create a trainer which contains an empty neural network, and then train the network until a desired prediction accuracy is achived. All of these examples output the final predictions to the console. For any larger dataset you will need to compute the prediction accuracy. One way to do this is to compute the percentage of correct predictions and the average "confidence" of the predictions.

<details> <summary>Computing prediction score and confidences</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/CSharp/Program.cs#L92-L104 </details> <details> <summary>Rust</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/Rust/src/main.rs#L32-L73 </details> <details> <summary>F#</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/FSharp/Program.fs#L38-L66 </details> <details> <summary>C#</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/CSharp/Program.cs#L28-L58 </details> <details> <summary>C++</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/Cpp/main.cpp#L49-L101 </details> <details> <summary>C</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/C/main.c#L46-L87 </details> <details> <summary>Kotlin</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/Kotlin/src/Main.kt#L21-L60 </details> <details> <summary>Go</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/4c9c8176a9936320af3e777a2159f931a7dca8c9/Go/main.go#L67-L110 </details> <details> <summary>Java</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/572f8794ebf95e6a988c0e3aa95ae94438278331/Java/src/Main.java#L29-L54 </details> <details> <summary>Zig</summary> https://github.com/dlidstrom/NeuralNetworkInAllLangs/blob/9de376aad054a39b8fe90bfe780f207098418391/Zig/src/main.zig#L48-L101 </details>

3. Training

For training and verifying our implementations we will use two datasets.

3.1. Logical Functions

The first is simple and will be these logical functions: xor, xnor, or, nor, and, and nand. This truth table represents the values that the network will learn, given two inputs; $i_1$ and $i_2$:

$$\begin{array}{rcl} i_1 & i_2 & xor & xnor & or & nor & and & nand \ 0 & 0 & 0 & 1 & 0 & 1 & 0 & 1 \ 0 & 1 & 1 & 0 & 1 & 0 & 0 & 1 \ 1 & 0 & 1 & 0 & 1 & 0 & 0 & 1 \ 1 & 1 & 0 & 1 & 1 & 0 & 1 & 0 \end{array}$$

This test is interesting as it shows how flexible a simple neural network can be. There are two inputs, 6 outputs, and it is sufficient to have two hidden neurons. Such a network consists of a total of 24 weights:

• 4 hidden weights (2 inputs * 2 hidden neurons)
• 2 hidden biases (one for each hidden neuron)
• 12 output weights (2 hidden neurons * 6 output neurons)
• 6 output biases (one for each output neuron)

<img src="https://github.com/dlidstrom/NeuralNetworkInAllLangs/raw/main/doc/nn.png" width="400px">

💯 We expect each implementation to learn exactly the same network weights!

3.1.1. Lithmus Test

The logical functions example can be used as a "lithmus test" of neural network implementations. A proper implementation will be able to learn the 6 functions using the 24 weights as detailed above. An improper implementation (one that doesn't implement biases correctly, for example) likely will need more hidden nodes to learn successfully (if at all). A larger network means more mathematical operations so keep this in mind when you evaluate other implementations. You don't want to waste cpu cycles unnecessarily.

3.2. Hand Written Digits

The second dataset consists of thousands of hand written digits. This is actually also a "toy" dataset but training a network to recognize all digits correctly is still a bit of a challenge. This dataset was originally downloaded from https://archive.ics.uci.edu/dataset/178/semeion+handwritten+digit.

Each line consists of 256 inputs (16x16 pixels) corresponding to one hand written digit. At the end of the line are 10 digits which signify the handwritten digit:

0: 1 0 0 0 0 0 0 0 0 0
1: 0 1 0 0 0 0 0 0 0 0
2: 0 0 1 0 0 0 0 0 0 0
3: 0 0 0 1 0 0 0 0 0 0
4: 0 0 0 0 1 0 0 0 0 0
...
9: 0 0 0 0 0 0 0 0 0 1

Parsing this dataset needs to be implemented for each language.

4. Learning

Our code will perform backpropagation to learn the weights. We update the weights after each input. This is called stochastic learning, as opposed to batch learning where multiple inputs are presented before updating weights. Stochastic learning is generally preferred [2]. Note that inputs need to be shuffled for effective learning.

5. Implementation Goals

One of our goals is to have as few or no dependencies. These implementations should be easy to integrate and that requires dependency-free code. Another goal is to implement fast code. Nifty, one-liners which look good but have bad performance should be avoided. It is fine to use for loops for matrix multiplication, as an example (i.e. no fancy linear algebra libraries are needed unless this is available in the standard library of the programming language).

We strive for:

• code that is easy to copy/paste for reuse
• dependency-free code
• straight forward code, no excessive object orientation which makes the code look like an OOAD excercise from the 90s
• adequate performance in favour of nifty (but slow) one-liners
• making it easy to serialize weights for storing and loading, but leave it for the users own preference
• implementations in all major languages
• simple tests that verify our implementations and secure them for the future
• having fun exploring neural networks!

5.1. Simple Random Number G