APU as a router

Installation

Notes:

  • Make sure you always restart/reload a service after you have changed its config files. Alternatively, follow this howto and do a clean boot at the very end.

  • All commands in this howto need to be executed by root (or via sudo).

System

Install a regular Debian amd64.

Boot Loader

The APU doesn’t come with a VGA or HDMI output, so for emergency debugging we want to enable the serial console.

Edit /etc/default/grub and add/change the following values:

GRUB_CMDLINE_LINUX_DEFAULT="verbose console=ttyS0,115200n8 reboot=bios"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
GRUB_TERMINAL=serial

Now we’ll have to write grub to the MBR again:

update-grub

We’ll also add a system service that provides a tty over the serial port. Create /etc/init/ttyS0.conf and write the following to it:

# ttyS0 - getty

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -8 115200 -L ttyS0

Once you reboot the system you should now be able to watch it boot via the serial console and login once it’s done.

Basic packages

Let’s install a bunch of common and useful packages:

apt install apt-transport-https build-essential curl git htop net-tools nmap tmux unzip

Configuration

/etc/fstab

Edit /etc/fstab and add noatime to the root filesystem’s options. This prevents unnecessary disk writes.

Unattended Upgrades

We want to enable unattended-upgrades (at least) for security updates:

apt install unattended-upgrades apt-listchanges

Edit /etc/apt/apt.conf.d/50unattended-upgrades and set an email address to be notified with Unattended-Upgrade::Mail.

Also check /etc/apt/apt.conf.d/20auto-upgrades to make sure unatteded upgrades are really enabled. Find more info in the Debian Wiki.

sudo

Add the user openlab to the sudo group. Edit /etc/sudoers and let users in this group run sudo without password verification:

%sudo   ALL=(ALL:ALL) NOPASSWD: ALL

Everyone with access to this system will have to add their pub-key to /home/openlab/.ssh/authorized_keys.

Network

The APU has three ethernet interfaces (from left to right: enp1s0, enp2s0 and enp3s0).

  • enp1s0 is physically connected to the LAN
  • enp2s0 is physically connected to the Ubiquiti radio-link to gamambel
  • enp3s0 is currently unused

Edit /etc/network/interfaces and add these interfaces:

# Local lab network
auto enp1s0
iface enp1s0 inet static
        address 172.16.0.1
        netmask 255.255.0.0

# Link to gamambel
auto enp2s0
iface enp2s0 inet static
        address 192.168.0.251
        netmask 255.255.0.0

        up route add -host 1.2.3.4 gw 192.168.0.1

The static host route is the only route available on startup. It allows us to connect to a VPN server with openvpn, which in turn sets up our default route to the Internet. Make sure the IP matches the VPN server’s address. See the next paragraph for more information.

VPN connection

We don’t want to connect to the Internet via our ISP directly, so we will setup a VPN connection, which all traffic will be routed through.

Install openvpn:

apt install openvpn ca-certificates

Copy the VPN configuration to /etc/openvpn/labvpn.conf. Now edit /etc/default/openvpn:

AUTOSTART="labvpn"

VPN connection via frivpn (alternative multi-threaded VPN client)

Install frivpn:

apt install cmake lua5.2 lua5.2-dev lua-posix lua-luaossl lua-cqueues libssl-dev liblzo2-dev
cd $HOME/src
git clone https://github.com/znuh/frivpn.git
mkdir frivpn/build ; cd frivpn/build
cmake ..
make

Now you can start frivpn like this:

./frivpn_client.lua config/ipredator

Firewall

We want to forward packages between ethernet interfaces, but only allow incoming connections on specific ports. All outgoing traffic is allowed to pass through.

Install ufw:

apt install ufw
ufw enable
ufw default deny incoming
ufw default allow outgoing
ufw default allow forward
ufw allow ssh

Add NAT/masquerading to the top of /etc/ufw/before.rules:

# NAT over tun0
*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 172.16.0.0/16 -o tun0 -j MASQUERADE

COMMIT

Enable IPv4 forwarding by editing /etc/ufw/sysctl.conf:

net/ipv4/ip_forward=1

DHCP server

LAN-guests will receive a dynamically allocated IP address, within the range of 172.16.0.100 through 172.16.0.249, via DHCP.

Install isc-dhcp-server and tools:

apt install isc-dhcp-server dhcpd-pools
systemctl enable isc-dhcp-server

Edit /etc/default/isc-dhcp-server and configure the interfaces:

INTERFACESv4="enp1s0"

Edit /etc/dhcp/dhcpd.conf for all other settings:

#option domain-name;
#option domain-name-servers;

default-lease-time 3600;
max-lease-time 14400;

authoritative;

class "guests" {
        match if substring (hardware, 1, 3) = 00:01:02;
}

subnet 172.16.0.0 netmask 255.255.0.0 {
        pool {
                deny members of "guests";
                range 172.16.0.100 172.16.0.249;
        }
        pool {
                allow members of "guests";
                range 172.16.1.200 172.16.1.249;
        }

        option routers 172.16.0.1;
        option domain-name "lab";
        option domain-name-servers 172.16.0.1, 8.8.8.8, 8.8.4.4;
}

Download http://standards.ieee.org/regauth/oui/oui.txt to /usr/local/etc/oui.txt to enable MAC => manufacturer resolving.

DNS server

We run a DNS server so local OpenLab services can be resolved by name.

Install dnsmasq:

apt install dnsmasq
ufw allow dns

Edit /etc/dnsmasq.conf and change the following values:

domain-needed
bogus-priv

interface=enp1s0

Add all machines with a static IP to /etc/hosts, as dnsmasq will parse this file for DNS requests.

Prometheus

Install prometheus:

apt install prometheus prometheus-node-exporter
systemctl enable prometheus-node-exporter
systemctl enable prometheus

Edit /etc/prometheus/prometheus.yml and change the following values:

  external_labels:
    monitor: 'lab'
    
...

  - job_name: 'lab'
    
    scrape_interval: 10s
    scrape_timeout: 10s

    static_configs:
      - targets: ['localhost:9100']

Next we’ll install grafana as a frontend for prometheus. Since it’s not in the official repos yet, we’ll add a new APT source. Edit /etc/apt/sources.list and add:

deb https://packagecloud.io/grafana/stable/debian/ stretch main

Now we can install the package:

curl https://packagecloud.io/gpg.key | sudo apt-key add -
apt update
apt install grafana
systemctl enable grafana-server
ufw allow 3000

Additional stats & scrapers

Install https://github.com/atonkyra/dhcp-stats-prometheus and its dependencies:

apt install dhcpd-pools python3-bottle

Traffic shaping

Install tc (part of iproute2):

apt install iproute2

Create a new bash script called /usr/local/bin/tcsetup.sh with the following content:

#!/bin/sh

# setup qdisc
tc qdisc del dev enp1s0 root handle 1: htb
tc qdisc add dev enp1s0 root handle 1: htb default 5

# define classes
# never use more than 50mbit/s total
tc class add dev enp1s0 parent 1: classid 1:1 htb rate 50mbit ceil 50mbit
# preferred class
tc class add dev enp1s0 parent 1:1 classid 1:4 htb rate 8mbit ceil 40mbit
# standard lab user
tc class add dev enp1s0 parent 1:1 classid 1:5 htb rate 4mbit ceil 40mbit
# external guests
tc class add dev enp1s0 parent 1:1 classid 1:6 htb rate 512kbit ceil 4mbit

# prefer DNS, SSH
tc filter add dev enp1s0 protocol ip parent 1:0 prio 0 u32 match ip sport 53 0xffff flowid 1:4
tc filter add dev enp1s0 protocol ip parent 1:0 prio 0 u32 match ip dport 53 0xffff flowid 1:4

tc filter add dev enp1s0 protocol ip parent 1:0 prio 0 u32 match ip sport 22 0xffff flowid 1:4
tc filter add dev enp1s0 protocol ip parent 1:0 prio 0 u32 match ip dport 22 0xffff flowid 1:4

# de-prioritize external guests
tc filter add dev enp1s0 protocol ip parent 1:0 prio 5 u32 match ip src 172.16.1.0/24 flowid 1:6
tc filter add dev enp1s0 protocol ip parent 1:0 prio 5 u32 match ip dst 172.16.1.0/24 flowid 1:6

Now make the script executable:

chmod +x /usr/local/bin/tcsetup.sh

Edit /etc/network/interfaces and amend the following section:

# Local lab network
auto enp1s0
iface enp1s0 inet static
        address 172.16.0.1
        netmask 255.255.0.0

        up /usr/local/bin/tcsetup.sh

Beehive

We’ll need to install Go in order to compile beehive:

apt install golang

SNMP

To be done.

Backup

To be done.

HowTo

Assign a static IP to a DHCP client

Edit /etc/dhcp/dhcpd.conf and add a new host. Restart isc-dhcp-server.

Add a host to the DNS

Edit /etc/hosts and add the host. Restart dnsmasq.