Project Lethe: Privacy-Preserving Indoor Localization System
Named after the river Lethe from Greek mythology, project Lethe is a privacy-preserving indoor localization system designed to allow for room-level localization of individuals in a home or building. The system preserves user anonymity by using a novel detection algorithm that limits both the memory available onboard the camera and the data rate of camera communication to prevent a full image from being extracted. Briefly, the algorithm detects the direction of movement by breaking the camera's FOV into three quadrants and tracking the order in which the person enters and exits each quadrant.
System Design
Lethe consists of both a custom-built hardware system and a novel data processing pipeline.
Hardware Components
FLIR Thermal Sensor
The system utilizes a FLIR Lepton Thermal Imaging Module as the main component of its detection module. The thermal imager captures infrared radiation input in its nominal response wavelength band (from 8 to 14 microns) and has a thermal sensitivity of 50 milli-Kelvins. The imager has an effective frame rate of 8.6 Hz and a resolution of 60 (w) × 80 (h) active pixels (each 17 μm in size and covering a 0.6375-degree angle).
Breakout Board
The thermal imager is embedded in a breakout board which provides the socket for the Lepton, on-board power supplies, 25Mhz reference clock (can be by-passed), power-efficient 1.2v core voltage (can be by-passed), dual low noise LDO for 2.8V voltage (can be by-passed), 100 mil header for use in a breadboard or wiring to any host system. A few things to consider about this kit: the breakout board will accept a 3-5.5V input and regulate it to what the Lepton® wants, to read an image from the lepton module all you need is an SPI port, and to configure the camera settings you also need an I2C port, although this is not required.
Raspberry Pi 1
Frames obtained by the thermal imager are sent over an SPI port to the Pi, where each pixel is processed independently and then permanently deleted. The stream of pixel values is processed by a novel algorithm that determines the direction of the crossing.
Software Components
Customized Driver
The FLIR Thermal Module is operated using a custom written driver which uses a limited memory buffer to transfer one pixel at a time.
Presence Detection
We detect the presence of a person in the field of view based on their temperature. Indoor environments are typically air-conditioned to a 20°C to 22°C range and most objects in the environment conform to this temperature. Human skin temperature ranges between 32°C and 34°C. Additionally, hair and clothing tend to absorb the skin’s heat and retain a temperature lower than skin temperature, but higher than the surrounding environment (e.g., 25°C to 30°C). Hence, a person can be detected in the view of a thermal camera by identifying pixels that are warmer than the background temperature. . In Lethe, we determine the background temperature 
as the average temperature of the last frame absent human presence. This value is updated with each new human-less frame to adjust the background temperature to changes in the environment temperature. We detect a human in a pixel whenever that pixel has a value 
degrees higher than the background reference . If at any point in the streaming evaluation of a frame a pixel passes this threshold, the frame is declared as having a presence. If this is the beginning of an event, 
is updated. Once a person is detected, we consider all frames 0.366 seconds after the last frame with a presence ( 
) to be part of the event. This aims to cover any gap caused by lost frames in an event, without merging two events in succession. This lag time value is conservatively based on the average walking speed of a person (1.39 m/s) and the average shoulder span of an adult male (0.508 m)
Direction Detection
Once presence detection has ended an event with the creation of the ( , 
) tuple, direction detection either assigns a direction d to that event or removes that event as a non-crossing interaction. Direction detection collects the information needed to make this determination in parallel to presence detection. Each streaming pixel evaluated by presence detection is also used by direction. Direction detection uses the same and as presence to determine if someone is present in a pixel. However, direction detection goes beyond presence to use this information to determine a person’s location in the frame and how that location changes over time. When a person crosses through the field of view they first appear on one side of the frame and then progress in intervals to the other side before leaving the field of view. A canonical example of this behavior can be seen in the figure above. Lethe captures the progress of the person over time through the field of view by leftmost sector and rightmost sector location of the person in frame 
. If the pixels of a frame are processed from left to right, this means Lethe sets 
to the column of the first pixel with a human presence and loci to the column of the last pixel with a human presence. In this way, Lethe identifies the region a person is located in for each frame.
Privacy Preserving Algorithm
The Lethe thermal camera requires only 21 values in memory for it's operation. For the detection algorithms, 4 values are required for presence detection ( , , , ), 12 values for direction ( 
, 
, 
, 
, 
, 
, 
, 
, 
, 
, 
, 
), and a single value for pixel height detection ( 
). Additionally, the thermal camera requires space to store the current pixel temperature the current location of the pixel (row and column), and the current time of the device. Reference information, such as the height and width of the image, the lag time allowed in frames, and the value can be kept statically. Many of the dynamic values, depending on the size of the image, can be represented using a single byte. Only the timestamps require a larger storage space at 4 bytes a piece. Hence the memory requirement for the Lethe thermal camera is only 
bytes. In the 60x80 pixel thermal camera used in this work, that means only 33 pixels (0.69%) of the image can ever be stored on the device.
Cross Detection
Once presence detection has ended an event with the creation of the ( ,) tuple, direction detection either assigns a direction 
to that event or removes that event as a non-crossing interaction. Direction detection collects the information needed to make this determination in parallel to presence detection. Each streaming pixel evaluated by presence detection is also used by direction. Direction detection uses the same and as presence to determine if someone is present in a pixel. However, direction detection goes beyond presence to use this information to determine a person's location in the frame and how that location changes over time. When a person crosses through the field of view they first appear on one side of the frame and then progress in intervals to the other side before leaving the field of view. A canonical example of this behavior can be seen in the figure above. Lethe captures the progress of the person over time through the field of view by leftmost, 
, and rightmost, 
, the location of the person in frame . If the pixels of a frame are processed from left to right, this means Lethe sets to the column of the first pixel with a human presence and to the column of the last pixel with a human presence. In this way, Lethe identifies the region a person is located in for each frame.
Once and 
