Anycast DNS PfSense

by
Tags: pfsense knot-resolver bgp anycast dns
Categories: dns linux firewall

Overview

I have 5 DNS servers, each are announcing the anycast ip with exabgp and are running knot-resolver as DNS resolver. I use pfSense as my router and have installed the FRR package on it.

Anycast IP: 172.16.0.1

DNS1: 172.30.31.31
DNS2: 172.30.31.32
DNS3: 172.30.31.33
DNS4: 172.30.31.34
DNS5: 172.30.31.35

BGP Router: 172.30.31.1
Local AS: 65000

 

Bind anycast ip on loopback interface

Start by binding the anycast IP to the loopback interface on each DNS server. /etc/netplan/00-installer-config.yaml

user@dns5:~$ sudo vim /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:
  ethernets:
    lo:
      match:
        name: lo
      addresses: [ 172.16.0.1/32 ]
    ens18:
      addresses:
      - 172.30.31.35/24
      nameservers:
        addresses:
        - 127.0.0.1
      routes:
      - to: default
        via: 172.30.31.1
  version: 2

user@dns5:~$ sudo netplan apply

 

As shown below, the anycast ip is now bound to the loopback interface.

user@dns5:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 172.16.0.1/32 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:a2:58:87 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    inet 172.30.31.35/24 brd 172.30.31.255 scope global ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fea2:5887/64 scope link 
       valid_lft forever preferred_lft forever

 

Configure DNS software

Install and configure knot-resolver, so it listens on both the anycast ip (172.16.0.1) and the local ip (172.30.31.35)

user@dns5:~$ wget https://secure.nic.cz/files/knot-resolver/knot-resolver-release.deb
user@dns5:~$ sudo dpkg -i knot-resolver-release.deb
user@dns5:~$ sudo apt update ; sudo apt install -y knot-resolver

/etc/knot-resolver/kresd.conf
-- SPDX-License-Identifier: CC0-1.0
-- vim:syntax=lua:set ts=4 sw=4:
-- Refer to manual: https://knot-resolver.readthedocs.org/en/stable/

-- Network interface configuration
net.listen('127.0.0.1', 53, { kind = 'dns' })
net.listen('172.16.0.1', 53, { kind = 'dns' })
net.listen('172.30.31.35', 53, { kind = 'dns' })

-- Load useful modules
modules = {
	'hints > iterate',  -- Allow loading /etc/hosts or custom root hints
	'stats',            -- Track internal statistics
	'predict',          -- Prefetch expiring/frequent records
    'prefill',          -- Prefill Cache
}

-- Prefill
prefill.config({
    ['.'] = {
        url = 'https://www.internic.net/domain/root.zone',
        interval = 86400, -- seconds
        ca_file = '/etc/pki/tls/certs/ca-bundle.crt', -- optional
    }
})

-- Cache size
cache.size = 100 * MB

-- set downstream bufsize to 4096 and upstream bufsize to 1232
net.bufsize(4096, 1232)

user@dns5:~$ sudo systemctl enable --now kresd@1.service

 

Configure Exabgp

First install exabgp and then configure the exabgp service

user@dns5:~$ sudo apt install -y exabgp
user@dns5:~$ sudo vim /etc/exabgp/exabgp.conf

Configure pfSense as BGP neighbor and tell exabgp to use script dns-check.sh to announce the route
process announce-routes {
  run /etc/exabgp/dns-check.sh;
  encoder text;
}

neighbor 172.30.31.1 {
    local-address 172.30.31.35;
    local-as 65000;
    peer-as 65000; 

    api {
        processes [ announce-routes ];
    }
}

 

dns-check.sh

Now create a script /etc/exabgp/dns-check.sh that checks if local dns is working and announce route if it’s working.

#!/usr/bin/bash
ANYCAST_IP="172.16.0.1"
LOCAL_IP="172.30.31.35"
while true; do
  /usr/bin/dig google.com @127.0.0.1 > /dev/null;
  if [ "$?" == 0 ]; then
    echo "announce route ${ANYCAST_IP} next-hop ${LOCAL_IP}"
  else       
    echo "withdraw route ${ANYCAST_IP} next-hop ${LOCAL_IP}"
  fi
  sleep 1
done

Make the script executeable
user@dns5:~$ sudo chmod +x /etc/exabgp/dns-check.sh

 

Create exabgp.service

Create a systemd service for exabgp so it auto starts /etc/systemd/system/exabgp.service

[Unit]
Description=ExaBGP
Documentation=man:exabgp(1)
Documentation=man:exabgp.conf(5)
Documentation=https://github.com/Exa-Networks/exabgp/wiki
After=network.target
ConditionPathExists=/etc/exabgp/exabgp.conf

[Service]
User=exabgp
Group=exabgp
RuntimeDirectory=exabgp
RuntimeDirectoryMode=0750
ExecStartPre=-/usr/bin/mkfifo /run/exabgp/exabgp.in
ExecStartPre=-/usr/bin/mkfifo /run/exabgp/exabgp.out
Environment=exabgp_daemon_daemonize=false
Environment=ETC=/etc
ExecStart=/usr/sbin/exabgp /etc/exabgp/exabgp.conf
ExecReload=/bin/kill -USR1 $MAINPID
Restart=always
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

user@dns5:~$ sudo systemctl daemon-reload
user@dns5:~$ sudo systemctl enable --now exabgp.service

 

Create neighbor in pfSense

BGP Neighbor

 
Let’s see if BGP if the route is working
BGP Table