CogniSync: Machine Learning & EEG-Based Directional Controls

Tulane University Computer Science: FALL 2023 - SPRING 2024

Contributors: Justin Haysbert, Gabriel Sagrera, Shayne Shelton, Ryan Stevens

Faculty Mentor: Saad Hassan

SSE EXPO: April 23, 2024

There are millions of paralysis patients in the United States. Electroencephalogram (EEG) headsets offer a way for these patients to operate mechanical devices provided motor imagery related brain waves. Machine learning approaches like SciKit-Learn (LDA, LRC, DTC, and RFC) and PyTorch can accurately and precisely categorize a user’s brain state into directional controls for this application. However, training time is long, training is user specific, and EEGs experience a low signal-to-noise ratio. This project aims to reduce user training time by (1) decreasing the number of movements users perform during training and (2) building a comprehensive graphical user interface (GUI) to streamline the recording, modeling, and prediction processes.

YouTube Demo

EEG Headset

Project Goals

• [X] Create an easy-to-use GUI for EEG data and modeling
• [X] Provide motor imagery predictions for controls
• [ ] Use predictions to successfully play the snake game
• [ ] Use predictions to accurately control the robotic wheelchair

Project Features (Tkinter Pages)

• [X] Plotting eeg - plotEEG()
  • [X] Gui for easily customizing Plotly
• [X] User recording - UserRecording()
  • [X] Streaming EEG data (8 Channels, 3 AUX)
  • [X] Prompting (1 Label)
  • [X] Recording (Combining the 11 dimensions of data into one .csv)
• [X] Modeling - Modeling()
  • [X] SciKit-Learn (LDA, DTC, RFC, LRC)
  • [X] PyTorch
  • [ ] Tensorflow (CPU possible, but GPU is too finicky)
• [X] Snake Game - SnakeGame()
  • [X] Model selection/input
  • [X] Prediction outputs
  • [ ] Sending predictions as controls
• [X] USB Output - USBOutput()
  • [X] Model selection/input
  • [X] Prediction outputs
  • [ ] Sending predictions as controls

Future Improvements

• [ ] Further improve model accuracy, f1, precision, and recalls
• [ ] Online training with PyTorch (feedback function)
• [ ] Tensorflow model using a GPU
• [ ] EEG headset alignment protocol/GUI - Possibility that the headset placement variability between uses changes the entire feed of data every time
• [ ] Switch to ESP32 for Bluetooth control to easier connect robotic wheelchair - Currently requires serial connection to read ip address
• [ ] Fix stopping of threads - current solution is to close the GUI and reopen it

gui_bci.py

Class App()

• __init__(self, parent, controller)
  Intializes the application by creating a container frame. Uses an empty dictionary to store all frames used to iterate through all the pages of the GUI.
• show_frame(self, cont)
  Takes a frame and raises it to the top making it visible to the user

Class Home()

• __init__(self, parent, controller)
  Intializes the home page of the GUI with the needed buttons for all pages plotEEG, UserRecording, Modeling, SnakeGame, and USBOutput. When buttons are clicked they call.
• show_frame(self, cont)
  Takes a frame and raises it to the top making it visible to the user

Class PlotEEG()

• __init__(self, parent, controller)
  This creates all the widgets for the PlotEEG page and creates a default range for the sliders between 0 and 1,000,000.
• updateList(self)
  This goes to the data folder and updates the list of files that are present in that folder while excluding any folders and .DS_Store. The data and label dropdown are then updated with the new list.
• slide1(self, value)
  This creates failsafes for the first slider to not be above the value set by the second slider. It also prevents the range from being significantly less than the second slider value. The text for the sliders will update when the slider is moved as well with this function running every time slider oen is moved.
• slide2(self, value)
  This creates failsafes for the second slider to not be below the value set by the second slider. It also prevents the range from being significantly less than the second slider value. The text for the sliders will update when the slider is moved as well with this function running every time slider one is moved.
• plot_eeg(self)
  This function plots the selected data in a web browser using the plotly package. The data file chosen is read using pandas.read_csv() and the unique labels are taken using .unique() as well as removing any rows with NA values. Colors are then assigned to each column and the interval is taken from the values the sliders were placed at. The minimum and maximum values are taken from the data with min() and max() and the labels are assigned colors. Each column is then individually graphed using fig.add_trace(go.Scatter()) and dashed vertical lines are added to the graph whenever a new movement occurs in the data. The columns graphed change based on what the user has selected.

Class UserRecording()

• __init__(self, parent, controller)
  Initialization function that creates the instruction canvas, initializes the prompt time settings, some default movement settings and all widgets.
• update_movements(self, event)
  This function is used to update the movement list given that there is user input. If user input is void movements default to right arm, left arm, legs, and jaw.
• start_prompting(self)
  This function is called when the start button is pressed. It disables the start button and enables the stop button. When pressed sets is prompting to True and calls prompt_next_movement(self) and start_record(self). It sets our total_prompts count to the product of the user selected iterations and movement count. If thhe user has not specified it defaults to 4 movements by 40 iterations. This function sleeps 15 seconds to ensure data stream can begin.
• stop_prompting(self)
  This function is called when the stop button is pressed while recording. is_promting is set to false and the start button is enables while the stop button is disabled. A label stating 'Training canceled' is printed and the movement index is set to 0. movements are shuffled and prompting times are reset. If the canvas till holds text it is wiped.
• end_prompting(self)
  This function is called when the prompting reaches its conclusion. Is_prompting is set to False, start button is enabled, and stop button is disabled. A label stating training completed is printed and the output file name is set to user input. If the output file is blank or filled with whitespace the output file is defaulted to 'YOU_DATA.csv'. The experiment results are then copied into the user specified file. Movement index is set to 0, movements are shuffled and prompt times are reset.
• shuffle_movements(self)
  Randomly shuffles order of movements.
• prompt_next_movement(self)
  If the prompt_count is less that the total_prompts and is_prompting is true we give a text label "Prepare for next movement", delete what was previously held in the canvas and after 5 seconds call show_movement_instruction(self). If not thenend_prompting(self) is called.
• show_movement_instruction(self) If is_prompting is true the text label 'Hold the movement for 10 seconds' is thrown. If no custom movements have been defined recall default movements. current movement is marked as the current movement index of the list of movements. Shows the user the current movement for the full 10 second interval. 2 seconds into the iterval begin writing to our csv data file using ,start_writing_to_file(self) after 6 seconds stop writing with stop_writing_to_file(self). After 10 second interval show the rest period using show_rest_period(self). If not promting close our output file, cancel training, and delete canvas content.
• start_writing_to_file(self)
  Function closes the temp_val.txt then reopens it and writes the current movement and closes it again.
• stop_writing_to_file(self)
  Function ensures temp_val.txt is closed
• show_rest_period(self)
  If is_prompting is true throw text "Rest for 10 seconds' and clear the canvas. Close the temp_val.txt file. Increment the prompt_count by 1 and set the current movement index to the next slot. if the current movement index is 0 shuffle the movements using shuffle_movements(self). After the 10 second rest time prompt the next movement using prompt_next_movement(self).
• start_record(self)
  This function checks for a record thread then creates one.
• stop_record(self)
  This function stops the record thread.
• record_data(self)
  This function calls record() from connect.py.

Class Modeling()

• __init__(self, parent, controller)
  This creates all the widgets for the model page including dropdowns for selecting the model, data file and label file. The widgets for the checkbox also have an IntVar created to help determine whether it has been clicked or not.
• model_input(self)
  This function runs the model and returns the results and fitted model and is activated by pressing the run button. The data file, label file chosen, model selected, and output file selected are all stored using the tkinter get() function. The following if statement determines which model was selec