About

This repository includes my notes on enabling a true bridge mode setup with AT&T U-Verse and pfSense. This method utilizes netgraph which is a graph based kernel networking subsystem of FreeBSD. This low-level solution was required to account for the unique issues surrounding bridging 802.1X traffic and tagging a VLAN with an id of 0. I've tested and confirmed this setup works with AT&T U-Verse Internet on the ARRIS NVG589, NVG599 and BGW210-700 residential gateways (probably others too). For Pace 5268AC see special details below.

There are a few other methods to accomplish true bridge mode, so be sure to see what easiest for you. True Bridge Mode is also possible in a Linux via ebtables or using hardware with a VLAN swap trick. For me, I was not using a Linux-based router and the VLAN swap did not seem to work for me.

While many AT&T residential gateways offer something called IP Passthrough, it does not provide the same advantages of a true bridge mode. For example, the NAT table is still managed by the gateway, which is limited to a measly 8192 sessions (although it becomes unstable at even 60% capacity).

The netgraph method will allow you to fully utilize your own router and fully bypass your residential gateway. It survives reboots, re-authentications, IPv6, and new DHCP leases.

How it Works

Before continuing to the setup, it's important to understand how this method works. This will make configuration and troubleshooting much easier.

Standard Procedure

First, let's talk about what happens in the standard setup (without any bypass). At a high level, the following process happens when the gateway boots up:

All traffic on the ONT is protected with 802.1/X. So in order to talk to anything, the Router Gateway must first perform the authentication procedure. This process uses a unique certificate that is hardcoded on your residential gateway.
Once the authentication completes, you'll be able to properly "talk" to the outside. However, all of your traffic will need to be tagged with VLAN ID 0 (a.k.a. VLAN Priority Tagging<sup>[1][2]</sup>) before the IP gateway will respond.
Once traffic is tagged with VLAN0, your residential gateway needs to request a public IPv4 address via DHCP. The MAC address in the DHCP request needs to match that of the MAC address that's assigned to your AT&T account. Other than that, there's nothing special about the DCHPv4 handshake.
After the DHCP lease is issued, the WAN setup is complete. Your LAN traffic is then NAT'd and routed to the outside.

Bypass Procedure

To bypass the gateway using pfSense, we can emulate the standard procedure. If we connect our Residential Gateway and ONT to our pfSense box, we can bridge the 802.1/X authentication sequence, tag our WAN traffic as VLAN0, and request a public IPv4 via DHCP using a spoofed MAC address.

Unfortunately, there are some challenges with emulating this process. First, it's against RFC to bridge 802.1/X traffic and it is not supported. Second, tagging traffic as VLAN0 is not supported through the standard interfaces.

This is where netgraph comes in. Netgraph allows you to break some rules and build the proper plumbing to make this work. So, our cabling looks like this:

Residential Gateway
[ONT Port]
  |
  |
[nic0] pfSense [nic1]
                 |
                 |
               [ONT]
              Outside

With netgraph, our procedure looks like this (at a high level):

The Residential Gateway initiates a 802.1/X EAPOL-START.
The packet then is bridged through netgraph to the ONT interface.
If the packet matches an 802.1/X type (which is does), it is passed to the ONT interface. If it does not, the packet is discarded. This prevents our Residential Gateway from initiating DHCP. We want pfSense to handle that.
The ONT should then see and respond to the EAPOL-START, which is passed back through our netgraph back to the residential gateway. At this point, the 802.1/X authentication should be complete.
netgraph has also created an interface for us called ngeth0. This interface is connected to ng_vlan which is configured to tag all traffic as VLAN0 before sending it on to the ONT interface.
pfSense can then be configured to use ngeth0 as the WAN interface.
Next, we spoof the MAC address of the residential gateway and request a DHCP lease on ngeth0. The packets get tagged as VLAN0 and exit to the ONT.
Now the DHCP handshake should complete and we should be on our way!

Hopefully, that now gives you an idea of what we are trying to accomplish. See the comments and commands bin/pfatt.sh for details about the netgraph setup.

But enough talk. Now for the fun part!

Setup

Prerequisites

At least three physical network interfaces on your pfSense server
The MAC address of your Residential Gateway
Local or console access to pfSense
pfSense 2.4.5 running on amd64 architecture (If you are running pfSense 2.4.4 please see instruction in the Before-pfSense-2.4.5 branch)

At this time there is a bug in pFsense 2.4.5 and ng_etf module is only included in pFsense 2.4.5 amd64 build. Should be fixed in 2.4.5-p1.

PFSense Builds for Netgate hardware may not include ng_etf (Confimred on SG4860-Desktop 2.4.5-p1). Confirm ng_etf exists before continuing and look at Before-pfSense-2.4.5 branch for gudiance if it doesn't exist.

If you are running pfSense on anything other than amd64 architecture you should compile your own version of ng_etf. Look at Before-pfSense-2.4.5 branch for some guidance on compiling and running your own ng_etf.

If you only have two NICs, you can buy this cheap USB 100Mbps NIC from Amazon as your third. It has the Asix AX88772 chipset, which is supported in FreeBSD with the axe driver. I've confirmed it works in my setup. The driver was already loaded and I didn't have to install or configure anything to get it working. Also, don't worry about the poor performance of USB or 100Mbps NICs. This third NIC will only send/recieve a few packets periodicaly to authenticate your Router Gateway. The rest of your traffic will utilize your other (and much faster) NICs.

Install

Edit the following configuration variables in bin/pfatt.sh as noted below. $RG_ETHER_ADDR should match the MAC address of your Residential Gateway. AT&T will only grant a DHCP lease to the MAC they assigned your device. In my environment, it's:
```
ONT_IF='xx0' # NIC -> ONT / Outside
RG_IF='xx1'  # NIC -> Residential Gateway's ONT port
RG_ETHER_ADDR='xx:xx:xx:xx:xx:xx' # MAC address of Residential Gateway
```
Copy bin/pfatt.sh to /root/bin (or any directory):
```
ssh root@pfsense mkdir /root/bin
scp bin/pfatt.sh root@pfsense:/root/bin/
ssh root@pfsense chmod +x /root/bin/pfatt.sh
```
NOTE: If you have the 5268AC, you'll also need to install pfatt-5268AC-startup.sh and pfatt-5268.sh. The scripts monitor your connection and disable or enable the EAP bridging as needed. It's a hacky workaround, but it enables you to keep your 5268AC connected, avoid EAP-Logoffs and survive reboots. Consider changing the PING_HOST in pfatt-5268AC.sh to a reliable host. Then perform these additional steps to install:
```
scp bin/pfatt-5268AC-startup.sh root@pfsense:/usr/local/etc/rc.d/pfatt-5268AC-startup.sh
scp bin/pfatt-5268AC.sh root@pfsense:/root/bin/
ssh root@pfsense chmod +x /usr/local/etc/rc.d/pfatt-5268AC-startup.sh /root/bin/pfatt-5268AC.sh
```
To start pfatt.sh script at the beginning of the boot process pfSense team recomments you use a package called shellcmd. Use pfSense package installer to find and install it. Once you have shellcmd package installed you can find it in Services > Shellcmd. Now add a new command and fill it up accordingly (make sure to select earlyshellcmd from a dropdown):
```
Command: /root/bin/pfatt.sh
Shellcmd Type: earlyshellcmd
```
It should look like this:

This can also be acomplished by manually editing your pfSense /conf/config.xml file. Add <earlyshellcmd>/root/bin/pfatt.sh</earlyshellcmd> above </system>. This method is not recommended and is frowned upon by pfSense team.
Connect cables:
- $RG_IF to Residential Gateway on the ONT port (not the LAN ports!)
- $ONT_IF to ONT (outside)
- LAN NIC to local switch (as normal)
Prepare for console access.
Reboot.
pfSense will detect new interfaces on bootup. Follow the prompts on the console to configure ngeth0 as your pfSense WAN. Your LAN interface should not normally change. However, if you moved or re-purposed your LAN interface for this setup, you'll need to re-apply any existing configuration (like your VLANs) to your new LAN interface. pfSense does not need to manage $RG_IF or $ONT_IF. I would advise not enabling those interfaces in pfSense as it can cause problems with the netgraph.
In the webConfigurator, configure the WAN interface (ngeth0) to DHCP using the MAC address of your Residential Gateway.

If everything is setup correctly, netgraph should be bridging EAP traffic between the ONT and RG, tagging the WAN traffic with VLAN0, and your WAN interface conf

Pfatt

Install / Use

README