monitor

TL;DR: Passive Bluetooth presence detection of beacons, cell phones, and other Bluetooth devices. Useful for mqtt-based home automation, especially when the script runs on multiple devices, distributed throughout a property.

Frequently Asked Questions

<details><summary><b>Installation Instructions</b></summary> <br> <details><summary><i>Set Up Raspberry Pi From Scratch</i></summary>

Installation Instructions for Raspberry Pi Zero W

Setup of SD Card

1. Download latest version of raspbian here

2. Download etcher from etcher.io

3. Image raspbian lite buster to SD card. Instructions here.

4. Mount boot partition of imaged SD card (unplug it and plug it back in)

5. To enable ssh, create blank file, without any extension, in the root directory called ssh

6. To setup Wi-Fi, create wpa_supplicant.conf file in root directory and add Wi-Fi details for home Wi-Fi:

country=US
    ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
    update_config=1

network={
    ssid="Your Network Name"
    psk="Your Network Password"
    key_mgmt=WPA-PSK
}

7. On the first startup, insert SD card and power on Raspberry Pi Zero W. On first boot, the newly-created wpa_supplicant.conf file and ssh will be moved to appropriate directories. Find the IP address of the Pi via your router.

Configuration and Setup

1. SSH into the Raspberry Pi (default password: raspberry):

ssh pi@theipaddress

2. Change the default password:

sudo passwd pi

3. Update and upgrade:

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
sudo reboot

5. Install Bluetooth Firmware, if necessary:

#install Bluetooth drivers for Pi Zero W
sudo apt-get install pi-bluetooth

6. Reboot:

sudo reboot

7. Install Mosquitto 1.5+ (important step!):

# get repo key
wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key

#add repo
sudo apt-key add mosquitto-repo.gpg.key

#download appropriate lists file 
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-buster.list

#update caches and install 
sudo apt-cache search mosquitto
sudo apt-get update
sudo apt-get install -f libmosquitto-dev mosquitto mosquitto-clients libmosquitto1

</details> <details><summary><i>Monitor Setup</i></summary>

Setup monitor

1. Clone monitor git:

#install git
cd ~
sudo apt-get install git

#clone this repo
git clone https://github.com/andrewjfreyer/monitor.git

#enter `monitor` directory
cd monitor/

#(optional) switch to beta branch for latest updates and features (may be unstable)
git checkout beta       

2. Initial run:

Configuration files will be created with default preferences. Any executables that are not installed will be reported. All can be installed via apt-get install ...

sudo bash monitor.sh

3. Edit mqtt_preferences file:

sudo nano mqtt_preferences

4. Edit known_static_addresses (phones, laptops, some smart watches):

sudo nano known_static_addresses

Alternatively, send an mqtt message to monitor/setup/ADD STATIC DEVICE with a message including a mac address and an alias separated by a space:

topic: monitor/setup/ADD STATIC DEVICE message: 00:11:22:33:44:55 alias

Use, monitor/setup/DELETE STATIC DEVICE with a message containing a mac address to remove a device from all monitor nodes.

5. Read helpfile:

sudo bash monitor.sh -h

Now the basic setup is complete. Your broker should be receiving messages and the monitor service will restart each time the Raspberry Pi boots. As currently configured, you should run sudo bash monitor.sh a few times from your command line to get a sense of how the script works.

</details>

---

</details> <details><summary><b>Background & Technical Details</b></summary>

Highlights

monitor sends a JSON-formatted MQTT message including a confidence value from 0 to 100 to a specified broker when a specified Bluetooth device responds to a name query. By default, name queries are triggered after receiving an anonymous advertisement from a previously-unseen device (e.g., a device in peripheral mode advertising an ability to connect).

Example JSON package:

topic: monitor/{{name of monitor install}}/{{mac address}}
message: {
    "id":"{{mac address}}",
    "confidence":"{{ranging from 0-100}}",
    "name":"{{if available}}",
    "manufacturer":{{if available}}",
    "type":"KNOWN_MAC",
    "retained":"{{message retained?}}",
    "timestamp":"{{formatted date at which message is sent}}",
    "version":"{{monitor version}}"
 }

In addition, optionally, a JSON-formatted MQTT message can be reported to the same broker whenever a publicly-advertising beacon device or an iBeacon device advertises.

Example JSON package:

topic: monitor/{{name of monitor install}}/{{mac address or ibeacon uuid}}
message: {
    "id":"{{mac address or ibeacon uuid}}",
    "report_delay":"{{delay from first detection to this message in seconds}}",
    "flags":"{{GAP flags}}",
    "movement":"stationary",
    "confidence":"{{ranging from 0-100}}",
    "name":"{{if available}}",
    "power":"{{if available}}",
    "rssi":"{{if available}}",
    "mac":"{{if ibeacon, the current mac address associated with the uuid}}",
    "manufacturer":{{if available}}",
    "type":"{{GENERIC_BEACON_PUBLIC or APPLE_IBEACON}},
    "retained":"{{message retained?}}",
    "timestamp":"{{formatted date at which message is sent}}",
    "version":"{{monitor version}}"
 }

---

Oversimplified Analogy of the Bluetooth Presence Problem

Imagine you're blindfolded in a large room with other people. We want to find out who of your friends is present and who of your friends isn't present:

Some of the people in the room periodically make sounds (e.g., eating a chip, sneeze, cough, etc.), others sit quietly and don’t make a sound unless you specifically ask for them by name, and still others periodically announce their own name out loud at regular intervals whether or not you want them to do that:

Here's the problem. You can’t just shout “WHO’S HERE” because then everyone would say their name at the same time and you couldn’t tell anything apart. Similarly, for obvious reasons, you can't simply ask "WHO ISN'T HERE?"

So, you take attendance like in a classroom. Everyone in the room responds only when their own name is shouted.

So, one way to take attendance is to shout for each friend on a list by name, one at a time, repeatedly. Ask for someone, get a response, wait for a moment, and ask again.

Once a friend stops responding (for some period of time), you presume that he or she has left:

This technique should work just fine, but there's a minor problem. You're constantly shouting into the room, which means that it's difficult for you to hear quiet responses and it's difficult for other people to carry on conversations. What else can we do? Can we use those random sounds for anything?

Yes! A smarter approach is to wait for an anonymous sound, then start asking whether a friend you know isn't present has just arrived:

This way, you're not constantly asking the room for all of your friends. Efficient!

This technique is a very simplified description of how monitor works for devices like cell phones (friends on a list) and beacons (announce a name out loud). This also gives an idea of how monitor uses anonymous sounds to reduce the number of times that it has to send inquiries into the Bluetooth environment.

---

Oversimplified Technical Background

The Bluetooth Low Energy spec was designed to make connecting Bluetooth devices simpler for the user. No more pin codes, no more code verifications, no more “discovery mode” - for the most part. It was also designed to be more private than previous Bluetooth implementations. That said, it’s hard to maintain privacy when you want to be able to connect to an unknown device without intervention.

Name Requests

A part of the Blueooth spec is a special function called a name request that asks another Bluetooth device to send back a human-readable name of itself. In order to send a name request, however, we need to know a private (unchanging) address of the target device.

Issuing a name request to the same private mac address every few seconds is a reliable - albeit rudimentary - way of detecting whether that device is "present" (it responds to the name request) or "absent" (no response to the name request is received). However, issuing name requests too frequently (e.g., every few seconds) uses quite a bit of 2.4GHz spectrum, which can cause interference with Wi-Fi or other wireless communications.

Not all devices respond to name requests, however. For example, beacon devices do not respond.

Connectible Devices

Blueooth devices that can exchange information with other devices (almost always) advertise a random/anonymous address that other devices can use to negotiate a secure connection and receive the first device's real, private, Bluetooth address. Using a random address in this way when publicly advertising prevents bad actors from tracking via passive Bluetooth monitoring.

Beacon/Advertising Dev