PyTsetlinMachineCUDA
Massively Parallel and Asynchronous Architecture for Logic-based AI
Install / Use
/learn @cair/PyTsetlinMachineCUDAREADME
PyTsetlinMachineCUDA - Massively Parallel and Asynchronous Architecture for Logic-based AI
Contents
Overview
Using logical clauses to represent patterns, Tsetlin machines (https://arxiv.org/abs/1804.01508) have obtained competitive performance in terms of accuracy, memory footprint, energy, and learning speed on several benchmarks (image classification, regression and natural language understanding).
<p align="center"> <img width="75%" src="https://github.com/olegranmo/blob/blob/master/MassiveParallel.png"> </p>In the parallel and asynchronous architecture implemented here, each clause runs in its own thread for massive parallelism. The clauses access the training examples simultaneously, updating themselves and local voting tallies in parallel (see figure).
A team of Tsetlin Automata composes each clause. The Tsetlin Automata thus drive the entire learning process. These are rewarded/penalized according to three local rules that optimize global behaviour (see https://github.com/cair/TsetlinMachine).
There is no synchronization among the clause threads, apart from atomic adds to the local voting tallies. Hence, the speed up!
<p> The architecture currently supports multi-class classification, multiple layers (https://arxiv.org/abs/1804.01508), integer clause weighting (https://arxiv.org/abs/2005.05131, https://arxiv.org/abs/2002.01245), regression (https://royalsocietypublishing.org/doi/full/10.1098/rsta.2019.0165) and convolution (https://arxiv.org/abs/1905.09688). Support for local and global interpretability and Boolean embedding coming soon. </p>Installation
pip install PyTsetlinMachineCUDA
Examples
Multiclass Demo
Code: NoisyXORDemo.py
from PyTsetlinMachineCUDA.tm import MultiClassTsetlinMachine
import numpy as np
train_data = np.loadtxt("NoisyXORTrainingData.txt")
X_train = train_data[:,0:-1]
Y_train = train_data[:,-1]
test_data = np.loadtxt("NoisyXORTestData.txt")
X_test = test_data[:,0:-1]
Y_test = test_data[:,-1]
tm = MultiClassTsetlinMachine(10, 15, 3.9, boost_true_positive_feedback=0)
tm.fit(X_train, Y_train, epochs=200,batch_size=100)
print("Accuracy:", 100*(tm.predict(X_test) == Y_test).mean())
print("Prediction: x1 = 1, x2 = 0, ... -> y = %d" % (tm.predict(np.array([[1,0,1,0,1,0,1,1,1,1,0,0]]))))
print("Prediction: x1 = 0, x2 = 1, ... -> y = %d" % (tm.predict(np.array([[0,1,1,0,1,0,1,1,1,1,0,0]]))))
print("Prediction: x1 = 0, x2 = 0, ... -> y = %d" % (tm.predict(np.array([[0,0,1,0,1,0,1,1,1,1,0,0]]))))
print("Prediction: x1 = 1, x2 = 1, ... -> y = %d" % (tm.predict(np.array([[1,1,1,0,1,0,1,1,1,1,0,0]]))))
Output
python3 ./NoisyXORDemo.py
Accuracy: 100.00%
Prediction: x1 = 1, x2 = 0, ... -> y = 1
Prediction: x1 = 0, x2 = 1, ... -> y = 1
Prediction: x1 = 0, x2 = 0, ... -> y = 0
Prediction: x1 = 1, x2 = 1, ... -> y = 0
Interpretability Demo
Code: InterpretabilityDemo.py
from PyTsetlinMachineCUDA.tm import MultiClassTsetlinMachine
import numpy as np
number_of_features = 20
noise = 0.1
X_train = np.random.randint(0, 2, size=(5000, number_of_features), dtype=np.uint32)
Y_train = np.logical_xor(X_train[:,0], X_train[:,1]).astype(dtype=np.uint32)
Y_train = np.where(np.random.rand(5000) <= noise, 1-Y_train, Y_train) # Adds noise
X_test = np.random.randint(0, 2, size=(5000, number_of_features), dtype=np.uint32)
Y_test = np.logical_xor(X_test[:,0], X_test[:,1]).astype(dtype=np.uint32)
tm = MultiClassTsetlinMachine(10, 15, 3.0, boost_true_positive_feedback=0)
tm.fit(X_train, Y_train, epochs=200)
print("Accuracy:", 100*(tm.predict(X_test) == Y_test).mean())
print("\nClass 0 Positive Clauses:\n")
for j in range(0, 10, 2):
print("Clause #%d: " % (j), end=' ')
l = []
for k in range(number_of_features*2):
if tm.ta_action(0, j, k) == 1:
if k < number_of_features:
l.append(" x%d" % (k))
else:
l.append("¬x%d" % (k-number_of_features))
print(" ∧ ".join(l))
print("\nClass 0 Negative Clauses:\n")
for j in range(1, 10, 2):
print("Clause #%d: " % (j), end=' ')
l = []
for k in range(number_of_features*2):
if tm.ta_action(0, j, k) == 1:
if k < number_of_features:
l.append(" x%d" % (k))
else:
l.append("¬x%d" % (k-number_of_features))
print(" ∧ ".join(l))
print("\nClass 1 Positive Clauses:\n")
for j in range(0, 10, 2):
print("Clause #%d: " % (j), end=' ')
l = []
for k in range(number_of_features*2):
if tm.ta_action(1, j, k) == 1:
if k < number_of_features:
l.append(" x%d" % (k))
else:
l.append("¬x%d" % (k-number_of_features))
print(" ∧ ".join(l))
print("\nClass 1 Negative Clauses:\n")
for j in range(1, 10, 2):
print("Clause #%d: " % (j), end=' ')
l = []
for k in range(number_of_features*2):
if tm.ta_action(1, j, k) == 1:
if k < number_of_features:
l.append(" x%d" % (k))
else:
l.append("¬x%d" % (k-number_of_features))
print(" ∧ ".join(l))
Output
python ./InterpretabilityDemo.py
Accuracy: 100.0
Class 0 Positive Clauses:
Clause #0: ¬x0 ∧ ¬x1
Clause #2: x0 ∧ x1
Clause #4: x0 ∧ x1
Clause #6: ¬x0 ∧ ¬x1
Clause #8: ¬x0 ∧ ¬x1
Class 0 Negative Clauses:
Clause #1: x0 ∧ ¬x1
Clause #3: x0 ∧ ¬x1
Clause #5: x1 ∧ ¬x0
Clause #7: x1 ∧ ¬x0
Clause #9: x0 ∧ ¬x1
Class 1 Positive Clauses:
Clause #0: x1 ∧ ¬x0
Clause #2: x1 ∧ ¬x0
Clause #4: x0 ∧ ¬x1
Clause #6: x0 ∧ ¬x1
Clause #8: x0 ∧ ¬x1
Class 1 Negative Clauses:
Clause #1: x0 ∧ x1
Clause #3: ¬x0 ∧ ¬x1
Clause #5: ¬x0 ∧ ¬x1
Clause #7: ¬x0 ∧ ¬x1
Clause #9: x0 ∧ x1
MNIST Demo w/Weighted Clauses
Code: MNISTDemoWeightedClauses.py
from PyTsetlinMachineCUDA.tm import MultiClassTsetlinMachine
import numpy as np
from time import time
from keras.datasets import mnist
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
X_train = np.where(X_train.reshape((X_train.shape[0], 28*28)) > 75, 1, 0)
X_test = np.where(X_test.reshape((X_test.shape[0], 28*28)) > 75, 1, 0)
tm = MultiClassTsetlinMachine(2000, 50*16, 10.0, max_weight=16)
print("\nAccuracy over 100 epochs:\n")
for i in range(100):
start_training = time()
tm.fit(X_train, Y_train, epochs=1, incremental=True)
stop_training = time()
start_testing = time()
result = 100*(tm.predict(X_test) == Y_test).mean()
stop_testing = time()
print("#%d Accuracy: %.2f%% Training: %.2fs Testing: %.2fs" % (i+1, result, stop_training-start_training, stop_testing-start_testing))
Output
python ./MNISTDemoWeightedClauses.py
Accuracy over 100 epochs:
#1 Accuracy: 93.00% Training: 4.91s Testing: 0.63s
#2 Accuracy: 94.48% Training: 3.29s Testing: 0.42s
#3 Accuracy: 95.57% Training: 3.32s Testing: 0.42s
...
#98 Accuracy: 98.12% Training: 3.06s Testing: 0.42s
#99 Accuracy: 98.20% Training: 3.06s Testing: 0.42s
#100 Accuracy: 98.16% Training: 3.06s Testing: 0.42s
MNIST 2D Convolution Demo w/Weighted Clauses
Code: MNISTDemo2DConvolutionWeightedClauses.py
from PyTsetlinMachineCUDA.tm import MultiClassConvolutionalTsetlinMachine2D
import numpy as np
from time import time
from keras.datasets import mnist
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
X_train = np.where(X_train >= 75, 1, 0)
X_test = np.where(X_test >= 75, 1, 0)
tm = MultiClassConvolutionalTsetlinMachine2D(2000, 50*15, 5.0, (10, 10), max_weight=16)
print("\nAccuracy over 50 epochs:\n")
for i in range(50):
start_training = time()
tm.fit(X_train, Y_train, epochs=1, incremental=True)
stop_training = time()
start_testing = time()
result = 100*(tm.predict(X_test) == Y_test).mean()
stop_testing = time()
print("#%d Accuracy: %.2f%% Training: %.2fs Testing: %.2fs" % (i+1, result, stop_training-start_training, stop_testing-start_testing))
Output
python ./MNISTDemo2DConvolutionWeightedClauses.py
Accuracy over 50 epochs:
#1 Accuracy: 97.14% Training: 13.83s Testing: 1.82s
#2 Accuracy: 98.22% Training: 12.17s Testing: 1.39s
#3 Accuracy: 98.57% Training: 11.88s Testing: 1.39s
...
#48 Accuracy: 99.13% Training: 9.74s Testing: 1.38s
#49 Accuracy: 99.14% Training: 9.93s Testing: 1.38s
#50 Accuracy: 99.10% Training: 9.14s Testing: 1.38s
Fashion MNIST 2D Convolution Demo w/Weighted Clauses
Code: FashionMNISTDemo2DConvolutionWeightedClauses.py
from PyTsetlinMachineCUDA.tm import MultiClassConvolutionalTsetlinMachine2D
import numpy as np
from time import time
import cv2
from keras.datasets import fashion_mnist
(X_train, Y_train), (X_test, Y_test) = fashion_mnist.load_data()
X_train = np.copy(X_train)
X_test = np.copy(X_test)
for i in range(X_train.shape[0]):
X_train[i,:] = cv2.adaptiveThreshold(X_train[i], 1, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
for i in range(X_test.shape[0]):
X_test[i,:] = cv2.adaptiveThreshold(X_test[i], 1, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
tm = MultiClassConvolutionalTsetlinMachine2D(8000, 100*100, 10.0, (10, 10), max_weight=255)
print("\nAccuracy over 3
