Radio Frequency Machine Learning (RFML) in PyTorch

The concept of deep learning has revitalized machine learning research in recent years.
In particular, researchers have demonstrated the use of deep learning for a multitude of tasks in wireless communications, such as signal classification and cognitive radio.
These technologies have been colloquially coined Radio Frequency Machine Learning (RFML) by the Defense Advanced Research Projects Agency (DARPA).
This repository hosts two key components to enable you to further your RFML research: a library with PyTorch implementations of common RFML networks, wrappers for downloading and utilizing an open source signal classification dataset, and adversarial evasion and training methods along with multiple tutorial notebooks for signal classification, adversarial evasion, and adversarial training.

---

.. raw:: html

<div align="center">

|LICENSE_BADGE| |PYTHON_BADGE| |PYTORCH_BADGE| |BLACK_BADGE|

.. raw:: html

</div>

.. |LICENSE_BADGE| image:: https://img.shields.io/badge/License-BSD%203--Clause-blue.svg?style=plastic :alt: License :target: /LICENSE.rst .. |PYTHON_BADGE| image:: https://img.shields.io/badge/Python-3-informational.svg?style=plastic :alt: Uses Python 3 .. |PYTORCH_BADGE| image:: https://img.shields.io/badge/Made%20With-PyTorch-informational.svg?style=plastic :alt: Deep Learning by PyTorch :target: https://pytorch.org .. |BLACK_BADGE| image:: https://img.shields.io/badge/Code%20Style-Black-000000.svg?style=plastic :target: https://github.com/python/black

---

.. contents:: Table of Contents

Highlights ##########

:code:rfml.attack Implementation of the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) that are aware of signal-to-perturbation ratios :code:rfml.data Classes for creating datasets from raw-IQ samples, splitting amongst training/validation/test datasets while keeping classes and signal-to-noise ratios (SNR) balanced, and converting into a PyTorch :code:TensorDataset :code:rfml.data.converters Wrappers to load open source datasets (including downloading them from the internet if necessary) from DeepSig, Inc :code:rfml.nn.eval Compute Top-K accuracy (overall and vs SNR) and confusion matrices from the models and datasets contained in this library :code:rfml.nn.model Implementations of state of the art signal classification deep neural networks (DNNs) in PyTorch :code:rfml.nn.train Implementation of standard training and adversarial training algorithms for classification problems in PyTorch :code:rfml.ptradio PyTorch implementations of linearly modulated modems (such as PSK, QAM, etc) and simple channel models

Quick Start ###########

Installation

The :code:rfml library can be installed directly from :code:pip (for Python >= 3.5).

.. code:: bash

pip install git+https://github.com/brysef/rfml.git@1.0.1

If you plan to directly edit the underlying library then you can install the library as editable after cloning this repository.

.. code:: bash

git clone git@github.com:brysef/rfml.git # OR https://github.com/brysef/rfml.git
pip install --user -e rfml/

Signal Classification (AMC)

.. raw:: html

<details>
<summary>Click to Expand</summary>

The following code (located at :code:examples/signal_classification.py) will:

• Download the RML2016.10a Dataset from deepsig.io/datasets
• Load the dataset into a PyTorch format with categorical labels
• Create a Convolutional Neural Network model with PyTorch
• Train the model to perform modulation classification
• Evaluate the model on the test set in terms of overall accuracy, accuracy vs SNR, and a confusion matrix amongst classes
• Save the model weights for later use

.. code:: python :number-lines:

from rfml.data import build_dataset
from rfml.nn.eval import (
    compute_accuracy,
    compute_accuracy_on_cross_sections,
    compute_confusion,
)
from rfml.nn.model import build_model
from rfml.nn.train import build_trainer, PrintingTrainingListener

train, val, test, le = build_dataset(dataset_name="RML2016.10a")
model = build_model(model_name="CNN", input_samples=128, n_classes=len(le))
trainer = build_trainer(
    strategy="standard", max_epochs=3, gpu=True
)  # Note: Disable the GPU here if you do not have one
trainer.register_listener(PrintingTrainingListener())
trainer(model=model, training=train, validation=val, le=le)
acc = compute_accuracy(model=model, data=test, le=le)
acc_vs_snr, snr = compute_accuracy_on_cross_sections(
    model=model, data=test, le=le, column="SNR"
)
cmn = compute_confusion(model=model, data=test, le=le)

# Calls to a plotting function could be inserted here
# For simplicity, this script only prints the contents as an example
print("===============================")
print("Overall Testing Accuracy: {:.4f}".format(acc))
print("SNR (dB)\tAccuracy (%)")
print("===============================")
for acc, snr in zip(acc_vs_snr, snr):
    print("{snr:d}\t{acc:0.1f}".format(snr=snr, acc=acc * 100))
print("===============================")
print("Confusion Matrix:")
print(cmn)

model.save("cnn.pt")

Running the above code will produce an output similar to the following. Additionally, the weights file will be saved off (:code:cnn.py) along with a local copy of the RML2016.10a dataset (:code:RML2016.10a.*).

.. code:: bash

> python3 signal_classification.py
.../rfml/data/converters/rml_2016.py:42: UserWarning:
About to attempt downloading the RML2016.10A dataset from deepsig.io/datasets.
Depending on your network connection, this process can be slow and error prone.  Any
errors raised during network operations are not silenced and will therefore cause your
code to crash.  If you require robustness in your experimentation, you should manually
download the file locally and pass the file path to the load_RML201610a_dataset
function.

Further, this dataset is provided by DeepSig Inc. under Creative Commons Attribution
- NonCommercial - ShareAlike 4.0 License (CC BY-NC-SA 4.0).  By calling this function,
you agree to that license -- If an alternative license is needed, please contact DeepSig
Inc. at info@deepsig.io

warn(self.WARNING_MSG)

Epoch 0 completed!
                -Mean Training Loss: 1.367
                -Mean Validation Loss: 1.226
Epoch 1 completed!
                -Mean Training Loss: 1.185
                -Mean Validation Loss: 1.180
Epoch 2 completed!
                -Mean Training Loss: 1.128
                -Mean Validation Loss: 1.158
Training has Completed:

=======================
        Best Validation Loss: 1.158
        Best Epoch: 2
        Total Epochs: 2
=======================
===============================
Overall Testing Accuracy: 0.6024
SNR (dB)        Accuracy (%)
===============================
-4      72.3
16      82.8
-12     25.2
10      84.0
-8      49.8
-10     34.8
-14     19.0
18      83.0
-6      63.5
6       83.4
-20     12.0
12      82.2
14      82.5
2       81.3
-2      77.6
-16     13.4
-18     12.3
4       81.6
0       80.9
8       83.3
===============================
Confusion Matrix:
...

.. raw:: html

</details>

Evading Signal Classification (FGSM)

.. raw:: html

<details>
<summary>Click to Expand</summary>

The following code (located at :code:examples/adversarial_evasion.py) will:

• Download the RML2016.10a Dataset from deepsig.io/datasets
• Load the dataset into a PyTorch format with categorical labels and only keep high SNR samples
• Create a Convolutional Neural Network model with PyTorch
• Load pre-trained weights (see Signal Classification (AMC)_)
• Evaluate the model on the dataset with no adversarial evasion for a baseline
• Perform an FGSM attack with a signal-to-perturbation ratio of 10 dB

Note that its likely that this script would evaluate the network on data it also used for training and that is certainly not desired. This script is merely meant to serve as an easy example and shouldn't be directly used for evaluation.

.. code:: python :number-lines:

from rfml.attack import fgsm
from rfml.data import build_dataset
from rfml.nn.eval import compute_accuracy
from rfml.nn.model import build_model

from torch.utils.data import DataLoader

_, _, test, le = build_dataset(dataset_name="RML2016.10a", test_pct=0.9)
mask = test.df["SNR"] >= 18
model = build_model(model_name="CNN", input_samples=128, n_classes=len(le))
model.load("cnn.pt")

acc = compute_accuracy(model=model, data=test, le=le, mask=mask)
print("Normal (no attack) Accuracy on Dataset: {:.3f}".format(acc))

spr = 10  # dB
right = 0
total = 0
dl = DataLoader(test.as_torch(le=le, mask=mask), shuffle=True, batch_size=512)
for x, y in dl:
    adv_x = fgsm(x, y, spr=spr, input_size=128, sps=8, net=model)

    predictions = model.predict(adv_x)
    right += (predictions == y).sum().item()
    total += len(y)

adv_acc = float(right) / total
print("Adversarial Accuracy with SPR of {} dB attack: {:.3f}".format(spr, adv_acc))
print("FGSM Degraded Model Accuracy by {:.3f}".format(acc - adv_acc))

Running the above code will produce an output similar to the following.

.. code:: bash

> python3 examples/adversarial_evasion.py
    Normal (no attack) Accuracy on Dataset: 0.831
    Adversarial Accuracy with SPR of 10 dB attack: 0.092
    FGSM Degraded Model Accuracy by 0.740

.. raw:: html