Dual-Hub DMVPN with VyOS

DMVPN is a Dynamic Multipoint VPN technology that provides the capability for creating a dynamic-mesh VPN network without having to pre-configure (static) all possible tunnel end-point peers those simplifying deployment and management of the newly added remote sites. There are 3 main protocols primarily used to implement DMVPN:

  • NHRP - provides the dynamic tunnel endpoint discovery mechanism (endpoint registration, and endpoint discovery/lookup)
  • mGRE - provides the tunnel encapsulation itself
  • IPSec - protocols handle the key exchange, and crypto mechanism

For this example we are using the following devices:

  • 2 x Hubs
  • 3 x Spokes
  • 1 x Client device (VPC)
  • 1 x ISP router

The following software was used in the creation of this document:

  • Operating system: VyOS
  • Version: 1.3-beta-202112090443
  • Image name: vyos-1.3-beta-202112090443-amd64.iso

Topology

Network Topology Diagram

Network Addressing and Protocol Parameters

The following ip addressing schema used for the devices IPv4 connectivity:

10.X1.0.0/30 - p2p Hubs to ISP networks, where X is Hub site number
10.Y1.1.0/24 - p2p Spokes to ISP networks(DHCP), where Y is Spoke site number
172.16.253.0/29 - tunnels addressing for Hub-1 connections
172.16.254.0/29 - tunnels addressing for Hub-2 connections
192.168.0.0/24 - HQ site local network
192.168.Z.0/24 - remote sites local network, where Z is Spoke site number

eBGP parameters for the routers:

AS65000 - HQ (Hub-1 and Hub-2)
AS6500X - Spokes, where X is Spoke site number

Configuration

Step-1: Basic connectivity configuration

  • Hub-1:
set interfaces ethernet eth0 address '10.11.0.1/30'
set interfaces ethernet eth1 address '192.168.0.1/24'
set protocols static route 0.0.0.0/0 next-hop 10.11.0.2
set system host-name 'Hub-1'
  • Hub-2:
set interfaces ethernet eth0 address '10.21.0.1/30'
set interfaces ethernet eth1 address '192.168.0.2/24'
set protocols static route 0.0.0.0/0 next-hop 10.21.0.2
set system host-name 'Hub-2'
  • Spoke-1:
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.1.1/24'
set system host-name 'Spoke-1'
  • Spoke-2:
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.2.1/24'
set system host-name 'Spoke-2'
  • Spoke-3:
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.3.1/24'
set system host-name 'Spoke-3'
  • ISP-1:
set interfaces ethernet eth0 address '10.11.0.2/30'
set interfaces ethernet eth1 address '10.21.0.2/30'
set interfaces ethernet eth2 address '10.31.1.1/24'
set interfaces ethernet eth3 address '10.21.1.1/24'
set interfaces ethernet eth4 address '10.11.1.1/24'
set service dhcp-server shared-network-name SPK-1 authoritative
set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 default-router '10.11.1.1'
set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 range 1 start '10.11.1.10'
set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 range 1 stop '10.11.1.100'
set service dhcp-server shared-network-name SPK-2 authoritative
set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 default-router '10.21.1.1'
set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 range 1 start '10.21.1.10'
set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 range 1 stop '10.21.1.100'
set service dhcp-server shared-network-name SPK-3 authoritative
set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 default-router '10.31.1.1'
set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 range 1 start '10.31.1.10'
set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 range 1 stop '10.31.1.100'
set system host-name 'ISP1'

Step-2: VRRP configuration for HQ Local network redundancy

Here we are using VRRP as a local redundancy protocol between Hub-1 and Hub-2. Initially, Hub-1 operates as an Active and Hub-2 as a Standby router. Additionally, health-check and script are used to track uplinks and properly switch mastership between Hub nodes based on the upstream router reachability (ISP-1). Note, that before adding local paths to the scripts into configuration, you have to create and make them executable first.

Hub-1 and Hub-2 VRRP health-check script:

  • /config/scripts/vrrp-check.sh
#!/bin/bash

eth0status="$(cat /sys/class/net/eth0/operstate | grep 'up')"

if [[ ! -z ${eth0status} ]]; then
 eth0gw="$(ip -j r show 0.0.0.0/0 dev eth0 | awk 'match($0, /\"gateway":\"([[:digit:]\.]+)/, gw) {print gw[1]}')"
 if [[ ! -z $eth0gw ]]; then
  /bin/ping -I eth0 -c 1 -W 1 $eth0gw && exit 0 || exit 1
 else
  exit 1
 fi
else
 #Exit 0 because eth0 down is handled by vrrp transition
 exit 0
fi

Note: some parts of the script might be dependent on your network topology and connectivity. Be careful before using it on your own devices.

Hub-1 and Hub-2 VRRP configuration:

  • Hub-1
set high-availability vrrp group HQ health-check failure-count '3'
set high-availability vrrp group HQ health-check interval '1'
set high-availability vrrp group HQ health-check script '/config/scripts/vrrp-check.sh'
set high-availability vrrp group HQ interface 'eth1'
set high-availability vrrp group HQ no-preempt
set high-availability vrrp group HQ priority '200'
set high-availability vrrp group HQ rfc3768-compatibility
set high-availability vrrp group HQ virtual-address '192.168.0.254/24'
set high-availability vrrp group HQ vrid '1'
  • Hub-2:
set high-availability vrrp group HQ health-check failure-count '3'
set high-availability vrrp group HQ health-check interval '1'
set high-availability vrrp group HQ health-check script '/config/scripts/vrrp-check.sh'
set high-availability vrrp group HQ interface 'eth1'
set high-availability vrrp group HQ no-preempt
set high-availability vrrp group HQ priority '100'
set high-availability vrrp group HQ rfc3768-compatibility
set high-availability vrrp group HQ virtual-address '192.168.0.254/24'
set high-availability vrrp group HQ vrid '1'

Step-3: DMVPN configuration between Hub and Spoke devices

This section provides an example configuration of the DMVPN enabled devices. Hub devices are configured with static IPv4 addresses on the uplink interfaces while Spoke devices receive addresses dynamically from a pre-defined DHCP pool configured on ISP router. For redundancy purposes, we use 1 tunnel interface on each Hub device and 2 tunnel interfaces on Spoke devices destined to each of the Hubs. For the optimal tunnel operation timers are significantly decreased and set to the following values:

NHRP tunnel holding time - 30 seconds

IKE DPD enabled with “restart” action set, interval 3 and timeout 30 seconds

Note: these values are used only for the lab demonstration and may not suit exclusive production networks.

  • Hub-1:
set interfaces tunnel tun100 address '172.16.253.134/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '10.11.0.1'

set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 multicast 'dynamic'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut

set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
  • Hub-2:
set interfaces tunnel tun100 address '172.16.254.134/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '2'
set interfaces tunnel tun100 source-address '10.21.0.1'

set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 multicast 'dynamic'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut

set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
  • Spoke-1:
set interfaces tunnel tun100 address '172.16.253.131/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '0.0.0.0'
set interfaces tunnel tun200 address '172.16.254.131/29'
set interfaces tunnel tun200 encapsulation 'gre'
set interfaces tunnel tun200 multicast 'enable'
set interfaces tunnel tun200 parameters ip key '2'
set interfaces tunnel tun200 source-address '0.0.0.0'

set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
set protocols nhrp tunnel tun100 multicast 'nhs'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set protocols nhrp tunnel tun200 cisco-authentication 'secret'
set protocols nhrp tunnel tun200 holding-time '30'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
set protocols nhrp tunnel tun200 multicast 'nhs'
set protocols nhrp tunnel tun200 redirect
set protocols nhrp tunnel tun200 shortcut

set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
  • Spoke-2:
set interfaces tunnel tun100 address '172.16.253.132/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '0.0.0.0'
set interfaces tunnel tun200 address '172.16.254.132/29'
set interfaces tunnel tun200 encapsulation 'gre'
set interfaces tunnel tun200 multicast 'enable'
set interfaces tunnel tun200 parameters ip key '2'
set interfaces tunnel tun200 source-address '0.0.0.0'

set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
set protocols nhrp tunnel tun100 multicast 'nhs'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set protocols nhrp tunnel tun200 cisco-authentication 'secret'
set protocols nhrp tunnel tun200 holding-time '30'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
set protocols nhrp tunnel tun200 multicast 'nhs'
set protocols nhrp tunnel tun200 redirect
set protocols nhrp tunnel tun200 shortcut

set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
  • Spoke-3:
set interfaces tunnel tun100 address '172.16.253.133/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '0.0.0.0'
set interfaces tunnel tun200 address '172.16.254.133/29'
set interfaces tunnel tun200 encapsulation 'gre'
set interfaces tunnel tun200 multicast 'enable'
set interfaces tunnel tun200 parameters ip key '2'
set interfaces tunnel tun200 source-address '0.0.0.0'

set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
set protocols nhrp tunnel tun100 multicast 'nhs'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set protocols nhrp tunnel tun200 cisco-authentication 'secret'
set protocols nhrp tunnel tun200 holding-time '30'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
set protocols nhrp tunnel tun200 multicast 'nhs'
set protocols nhrp tunnel tun200 redirect
set protocols nhrp tunnel tun200 shortcut

set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'

Step-4: Enabling eBGP as a Dynamic Routing Protocol between Hubs and Spokes

For the simplified and better network management we’re using eBGP for routing information exchange between devices. As we’re using Active-Standby mode in this example, Hub-2 is configured with AS-prepand as an export route-policy and VRRP transition scripts are used for switching mastership based on the current link/device state. Also, we use multihop BFD for faster eBGP failure detection.

Hub-1 and Hub-2 VRRP transition scripts:

  • /config/scripts/vrrp-master.sh
#!/bin/vbash

if [ $(id -gn) != vyattacfg ]; then
    exec sg vyattacfg "$0 $*"
fi

source /opt/vyatta/etc/functions/script-template

configure
delete protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export AS65000-PREP
commit

exit
  • /config/scripts/vrrp-fail.sh
#!/bin/vbash

if [ $(id -gn) != vyattacfg ]; then
    exec sg vyattacfg "$0 $*"
fi

source /opt/vyatta/etc/functions/script-template

configure
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export AS65000-PREP
commit

exit

Note: some parts of the script might be dependent on your network topology and connectivity. Be careful before using it on your own devices.

Hub devices configuration:

  • Hub-1:
set high-availability vrrp group HQ transition-script backup '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script fault '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script master '/config/scripts/vrrp-master.sh'
set high-availability vrrp group HQ transition-script stop '/config/scripts/vrrp-fail.sh'

set policy route-map AS65000-PREP rule 1 action 'permit'
set policy route-map AS65000-PREP rule 1 set as-path-prepend '65000 65000 65000'

set protocols bfd peer 172.16.253.131 interval multiplier '3'
set protocols bfd peer 172.16.253.131 interval receive '300'
set protocols bfd peer 172.16.253.131 interval transmit '300'
set protocols bfd peer 172.16.253.131 multihop
set protocols bfd peer 172.16.253.131 source address '172.16.253.134'
set protocols bfd peer 172.16.253.132 interval multiplier '3'
set protocols bfd peer 172.16.253.132 interval receive '300'
set protocols bfd peer 172.16.253.132 interval transmit '300'
set protocols bfd peer 172.16.253.132 multihop
set protocols bfd peer 172.16.253.132 source address '172.16.253.134'
set protocols bfd peer 172.16.253.133 interval multiplier '3'
set protocols bfd peer 172.16.253.133 interval receive '300'
set protocols bfd peer 172.16.253.133 interval transmit '300'
set protocols bfd peer 172.16.253.133 multihop
set protocols bfd peer 172.16.253.133 source address '172.16.253.134'

set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 parameters network-import-check
set protocols bgp 65000 peer-group DMVPN bfd
  • Hub-2:
set high-availability vrrp group HQ transition-script backup '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script fault '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script master '/config/scripts/vrrp-master.sh'
set high-availability vrrp group HQ transition-script stop '/config/scripts/vrrp-fail.sh'

set policy route-map AS65000-PREP rule 1 action 'permit'
set policy route-map AS65000-PREP rule 1 set as-path-prepend '65000 65000 65000'

set protocols bfd peer 172.16.254.131 interval multiplier '3'
set protocols bfd peer 172.16.254.131 interval receive '300'
set protocols bfd peer 172.16.254.131 interval transmit '300'
set protocols bfd peer 172.16.254.131 multihop
set protocols bfd peer 172.16.254.131 source address '172.16.254.134'
set protocols bfd peer 172.16.254.132 interval multiplier '3'
set protocols bfd peer 172.16.254.132 interval receive '300'
set protocols bfd peer 172.16.254.132 interval transmit '300'
set protocols bfd peer 172.16.254.132 multihop
set protocols bfd peer 172.16.254.132 source address '172.16.254.134'
set protocols bfd peer 172.16.254.133 interval multiplier '3'
set protocols bfd peer 172.16.254.133 interval receive '300'
set protocols bfd peer 172.16.254.133 interval transmit '300'
set protocols bfd peer 172.16.254.133 multihop
set protocols bfd peer 172.16.254.133 source address '172.16.254.134'

set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 parameters network-import-check
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
set protocols bgp 65000 peer-group DMVPN bfd

Spoke devices configuration:

  • Spoke-1:
set protocols bfd peer 172.16.253.134 interval multiplier '3'
set protocols bfd peer 172.16.253.134 interval receive '300'
set protocols bfd peer 172.16.253.134 interval transmit '300'
set protocols bfd peer 172.16.253.134 multihop
set protocols bfd peer 172.16.253.134 source address '172.16.253.131'
set protocols bfd peer 172.16.254.134 interval multiplier '3'
set protocols bfd peer 172.16.254.134 interval receive '300'
set protocols bfd peer 172.16.254.134 interval transmit '300'
set protocols bfd peer 172.16.254.134 multihop
set protocols bfd peer 172.16.254.134 source address '172.16.254.131'

set protocols bgp 65001 address-family ipv4-unicast network 192.168.1.0/24
set protocols bgp 65001 neighbor 172.16.253.134 address-family ipv4-unicast
set protocols bgp 65001 neighbor 172.16.253.134 bfd
set protocols bgp 65001 neighbor 172.16.253.134 remote-as '65000'
set protocols bgp 65001 neighbor 172.16.254.134 address-family ipv4-unicast
set protocols bgp 65001 neighbor 172.16.254.134 bfd
set protocols bgp 65001 neighbor 172.16.254.134 remote-as '65000'
set protocols bgp 65001 parameters log-neighbor-changes
  • Spoke-2:
set protocols bfd peer 172.16.253.134 interval multiplier '3'
set protocols bfd peer 172.16.253.134 interval receive '300'
set protocols bfd peer 172.16.253.134 interval transmit '300'
set protocols bfd peer 172.16.253.134 multihop
set protocols bfd peer 172.16.253.134 source address '172.16.253.132'
set protocols bfd peer 172.16.254.134 interval multiplier '3'
set protocols bfd peer 172.16.254.134 interval receive '300'
set protocols bfd peer 172.16.254.134 interval transmit '300'
set protocols bfd peer 172.16.254.134 multihop
set protocols bfd peer 172.16.254.134 source address '172.16.254.132'

set protocols bgp 65002 address-family ipv4-unicast network 192.168.2.0/24
set protocols bgp 65002 neighbor 172.16.253.134 address-family ipv4-unicast
set protocols bgp 65002 neighbor 172.16.253.134 bfd
set protocols bgp 65002 neighbor 172.16.253.134 remote-as '65000'
set protocols bgp 65002 neighbor 172.16.254.134 address-family ipv4-unicast
set protocols bgp 65002 neighbor 172.16.254.134 bfd
set protocols bgp 65002 neighbor 172.16.254.134 remote-as '65000'
set protocols bgp 65002 parameters log-neighbor-changes
  • Spoke-3:
set protocols bfd peer 172.16.253.134 interval multiplier '3'
set protocols bfd peer 172.16.253.134 interval receive '300'
set protocols bfd peer 172.16.253.134 interval transmit '300'
set protocols bfd peer 172.16.253.134 multihop
set protocols bfd peer 172.16.253.134 source address '172.16.253.133'
set protocols bfd peer 172.16.254.134 interval multiplier '3'
set protocols bfd peer 172.16.254.134 interval receive '300'
set protocols bfd peer 172.16.254.134 interval transmit '300'
set protocols bfd peer 172.16.254.134 multihop
set protocols bfd peer 172.16.254.134 source address '172.16.254.133'

set protocols bgp 65003 address-family ipv4-unicast network 192.168.3.0/24
set protocols bgp 65003 neighbor 172.16.253.134 address-family ipv4-unicast
set protocols bgp 65003 neighbor 172.16.253.134 bfd
set protocols bgp 65003 neighbor 172.16.253.134 remote-as '65000'
set protocols bgp 65003 neighbor 172.16.254.134 address-family ipv4-unicast
set protocols bgp 65003 neighbor 172.16.254.134 bfd
set protocols bgp 65003 neighbor 172.16.254.134 remote-as '65000'
set protocols bgp 65003 parameters log-neighbor-changes

Note: In case if you’re using VyOS version that has a VRRP transition scripts issues after a device reboot, as a temporary solution you may add postconfig-bootup script that reloads keepalived process additionally after the device booted.

  • Hub devices /config/scripts/vyos-postconfig-bootup.script:
#!/bin/sh
# This script is executed at boot time after VyOS configuration is fully applied.
# Any modifications required to work around unfixed bugs
# or use services not available through the VyOS CLI system can be placed here.

echo "Reloading VRRP process"
sudo systemctl restart keepalived.service
echo "VRRP process reload completed"

Step-5: Verification

Now, it’s time to check that all protocols are working as expected and mastership during the failover switches correctly between Hub devices.

  • Checking VRRP state between Hub-1 and Hub-2:
vyos@Hub-1:~$ show vrrp
Name    Interface      VRID  State      Priority  Last Transition
------  -----------  ------  -------  ----------  -----------------
HQ      eth1v1            1  MASTER          200  14s

vyos@Hub-2:~$ show vrrp
Name    Interface      VRID  State      Priority  Last Transition
------  -----------  ------  -------  ----------  -----------------
HQ      eth1v1            1  BACKUP          100  29s
  • Checking NHRP and eBGP sessions between Hub and Spoke devices:
vyos@Hub-1:~$ show nhrp tunnel
Status: ok

Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.134
Flags: up

Interface: tun100
Type: local
Protocol-Address: 172.16.253.134/32
Flags: up

Interface: tun100
Type: dynamic
Protocol-Address: 172.16.253.131/32
NBMA-Address: 10.11.1.11
Flags: up
Expires-In: 0:23

Interface: tun100
Type: dynamic
Protocol-Address: 172.16.253.133/32
NBMA-Address: 10.31.1.11
Flags: up
Expires-In: 0:22

Interface: tun100
Type: dynamic
Protocol-Address: 172.16.253.132/32
NBMA-Address: 10.21.1.11
Flags: up
Expires-In: 0:21

vyos@Hub-1:~$ show bgp summary

IPv4 Unicast Summary:
BGP router identifier 192.168.0.1, local AS number 65000 vrf-id 0
BGP table version 20
RIB entries 7, using 1344 bytes of memory
Peers 3, using 64 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
172.16.253.131  4      65001     26519     26526        0    0    0 00:43:38            1        4
172.16.253.132  4      65002     26545     26540        0    0    0 00:46:36            1        4
172.16.253.133  4      65003     26528     26520        0    0    0 00:41:59            1        4

Total number of neighbors 3


vyos@Hub-2:~$ show nhrp tunnel
Status: ok

Interface: tun100
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.134
Flags: up

Interface: tun100
Type: local
Protocol-Address: 172.16.254.134/32
Flags: up

Interface: tun100
Type: dynamic
Protocol-Address: 172.16.254.132/32
NBMA-Address: 10.21.1.11
Flags: up
Expires-In: 0:28

Interface: tun100
Type: dynamic
Protocol-Address: 172.16.254.131/32
NBMA-Address: 10.11.1.11
Flags: up
Expires-In: 0:21

Interface: tun100
Type: dynamic
Protocol-Address: 172.16.254.133/32
NBMA-Address: 10.31.1.11
Flags: up
Expires-In: 0:20

vyos@Hub-2:~$ show bgp summary

IPv4 Unicast Summary:
BGP router identifier 192.168.0.2, local AS number 65000 vrf-id 0
BGP table version 14
RIB entries 7, using 1344 bytes of memory
Peers 3, using 64 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
172.16.254.131  4      65001     26516     26516        0    0    0 00:43:03            1        4
172.16.254.132  4      65002     26563     26562        0    0    0 00:48:27            1        4
172.16.254.133  4      65003     26518     26516        0    0    0 00:42:20            1        4

Total number of neighbors 3
  • Checking BFD sessions between Hub and Spoke devices:
vyos@Hub-1:~$ show protocols bfd peers
Session count: 6
SessionId  LocalAddress                             PeerAddress                             Status
=========  ============                             ===========                             ======
3600626867 172.16.253.134                           172.16.253.133                          up
1123939978 172.16.253.134                           172.16.253.131                          up
374394280  172.16.253.134                           172.16.253.132                          up
1786735466 172.16.253.134                           172.16.253.132                          up
1440522544 172.16.253.134                           172.16.253.131                          up
1106910911 172.16.253.134                           172.16.253.133                          up


vyos@Hub-2:~$ show protocols bfd peers
Session count: 6
SessionId  LocalAddress                             PeerAddress                             Status
=========  ============                             ===========                             ======
2442966178 172.16.254.134                           172.16.254.133                          up
393258775  172.16.254.134                           172.16.254.131                          up
2990308682 172.16.254.134                           172.16.254.133                          up
2267910949 172.16.254.134                           172.16.254.132                          up
3542474595 172.16.254.134                           172.16.254.131                          up
4239538185 172.16.254.134                           172.16.254.132                          up
  • Checking routing information and connectivity between Hub and Spoke devices:
vyos@Hub-1:~$ show ip bgp
BGP table version is 20, local router ID is 192.168.0.1, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.0.0/24   0.0.0.0                  0         32768 i
*> 192.168.1.0/24   172.16.253.131           0             0 65001 i
*> 192.168.2.0/24   172.16.253.132           0             0 65002 i
*> 192.168.3.0/24   172.16.253.133           0             0 65003 i

Displayed  4 routes and 4 total paths


vyos@Hub-2:~$ show ip bgp
BGP table version is 14, local router ID is 192.168.0.2, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.0.0/24   0.0.0.0                  0         32768 i
*> 192.168.1.0/24   172.16.254.131           0             0 65001 i
*> 192.168.2.0/24   172.16.254.132           0             0 65002 i
*> 192.168.3.0/24   172.16.254.133           0             0 65003 i

Displayed  4 routes and 4 total paths


vyos@Spoke-1:~$ show ip bgp
BGP table version is 19, local router ID is 192.168.1.1, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*  192.168.0.0/24   172.16.254.134           0             0 65000 65000 65000 65000 i
*>                  172.16.253.134           0             0 65000 i
*> 192.168.1.0/24   0.0.0.0                  0         32768 i
*  192.168.2.0/24   172.16.254.132                         0 65000 65000 65000 65000 65002 i
*>                  172.16.253.132                         0 65000 65002 i
*  192.168.3.0/24   172.16.254.133                         0 65000 65000 65000 65000 65003 i
*>                  172.16.253.133                         0 65000 65003 i

Displayed  4 routes and 7 total paths

As you can see, Hub-2 announces routes with longer(prepended) AS path as we’ve configured it previously, those, traffic towards HQ subnet will be forwarded over Hub-1 which is operating as an Active VRRP router. Let’s check connectivity and the path from Spoke-1 to the HQ local network:

vyos@Spoke-1:~$ ping 192.168.0.10 count 5 interface 192.168.1.1
PING 192.168.0.10 (192.168.0.10) from 192.168.1.1 : 56(84) bytes of data.
64 bytes from 192.168.0.10: icmp_seq=1 ttl=63 time=3.50 ms
64 bytes from 192.168.0.10: icmp_seq=2 ttl=63 time=2.45 ms
64 bytes from 192.168.0.10: icmp_seq=3 ttl=63 time=2.34 ms
64 bytes from 192.168.0.10: icmp_seq=4 ttl=63 time=2.20 ms
64 bytes from 192.168.0.10: icmp_seq=5 ttl=63 time=2.44 ms

--- 192.168.0.10 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 11ms
rtt min/avg/max/mdev = 2.195/2.583/3.496/0.465 ms

vyos@Spoke-1:~$ traceroute 192.168.0.10
traceroute to 192.168.0.10 (192.168.0.10), 30 hops max, 60 byte packets
 1  172.16.253.134 (172.16.253.134)  0.913 ms  0.884 ms  0.819 ms
 2  192.168.0.10 (192.168.0.10)  1.352 ms  1.446 ms  1.391 ms

From the output, we can confirm successful connectivity between Spoke-1 and HQ local networks. From the traceroute we see that the traffic pass through the Hub-1.

Now, let’s check traffic between Spoke sites. Based on our configuration, Spoke sites are using shortcut for direct reachability between each other. First, let’s check NHRP tunnels before passing the traffic between Spoke-1 and Spoke-2:

vyos@Spoke-1:~$ show nhrp tunnel
Status: ok

Interface: tun200
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.131
Flags: up

Interface: tun200
Type: local
Protocol-Address: 172.16.254.131/32
Flags: up

Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.131
Flags: up

Interface: tun100
Type: local
Protocol-Address: 172.16.253.131/32
Flags: up

Interface: tun200
Type: static
Protocol-Address: 172.16.254.134/29
NBMA-Address: 10.21.0.1
Flags: used up

Interface: tun100
Type: static
Protocol-Address: 172.16.253.134/29
NBMA-Address: 10.11.0.1
Flags: used up

vyos@Spoke-2:~$ show nhrp tunnel
Status: ok

Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.132
Flags: up

Interface: tun100
Type: local
Protocol-Address: 172.16.253.132/32
Flags: up

Interface: tun200
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.132
Flags: up

Interface: tun200
Type: local
Protocol-Address: 172.16.254.132/32
Flags: up

Interface: tun100
Type: static
Protocol-Address: 172.16.253.134/29
NBMA-Address: 10.11.0.1
Flags: used up

Interface: tun200
Type: static
Protocol-Address: 172.16.254.134/29
NBMA-Address: 10.21.0.1

After passing traffic we could see that there is additional shortcut tunnel created between Spoke-1 and Spoke-2 for the direct communication:

vyos@Spoke-1:~$ ping 192.168.2.1 count 5 interface 192.168.1.1
PING 192.168.2.1 (192.168.2.1) from 192.168.1.1 : 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.820 ms
64 bytes from 192.168.2.1: icmp_seq=3 ttl=64 time=1.13 ms
64 bytes from 192.168.2.1: icmp_seq=4 ttl=63 time=1.41 ms
64 bytes from 192.168.2.1: icmp_seq=5 ttl=64 time=0.988 ms

--- 192.168.2.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 0.820/1.075/1.412/0.197 ms

vyos@Spoke-1:~$ traceroute 192.168.2.1
traceroute to 192.168.2.1 (192.168.2.1), 30 hops max, 60 byte packets
 1  192.168.2.1 (192.168.2.1)  1.172 ms  1.109 ms  1.151 ms

vyos@Spoke-1:~$ show nhrp tunnel
Status: ok

Interface: tun200
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.131
Flags: up

Interface: tun200
Type: local
Protocol-Address: 172.16.254.131/32
Flags: up

Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.131
Flags: up

Interface: tun100
Type: local
Protocol-Address: 172.16.253.131/32
Flags: up

Interface: tun200
Type: static
Protocol-Address: 172.16.254.134/29
NBMA-Address: 10.21.0.1
Flags: used up

____________________________________
Interface: tun100
Type: cached
Protocol-Address: 172.16.253.132/32
NBMA-Address: 10.21.1.11
Flags: used up
Expires-In: 0:24
____________________________________

Interface: tun100
Type: static
Protocol-Address: 172.16.253.134/29
NBMA-Address: 10.11.0.1
Flags: used up

The same applies to the rest of the devices and works with the same logic. As we’ve already confirmed successfull connectivity between Hub and Spoke devices, let’s check failover process.

  • Failover on the health-check failure on Hub-1:
# disabling interface towards Hub-1 on ISP router
vyos@ISP1:~$ configure
[edit]
vyos@ISP1# set interfaces ethernet eth0 disable
[edit]
vyos@ISP1# commit
[edit]
vyos@ISP1#


# checking VRRP state and eBGP configuration on Hub-1:
vyos@Hub-1:~$ show vrrp
Name    Interface      VRID  State      Priority  Last Transition
------  -----------  ------  -------  ----------  -----------------
HQ      eth1v1            1  FAULT           200  1m15s

vyos@Hub-1:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
set protocols bgp 65000 peer-group DMVPN bfd


# consecutive pings check from Spoke-1 to the HQ local network during the failure
--- 192.168.0.10 ping statistics ---
223 packets transmitted, 219 received, 1.79372% packet loss, time 679ms
rtt min/avg/max/mdev = 0.918/2.191/2.957/0.364 ms
vyos@Spoke-1:~$


# consecutive pings check from Spoke-3 to the Spoke-2 local network during the failure
--- 192.168.2.1 ping statistics ---
265 packets transmitted, 265 received, 0% packet loss, time 690ms
rtt min/avg/max/mdev = 0.663/1.128/2.272/0.285 ms
vyos@Spoke-3:~$

Note: After bringing ISP interface towards Hub-1 back to UP state, VRRP state will remain unchanged due to “no-preempt” option enabled under the VRRP configuration on the Hub-1 and Hub-2 and will be changed only during link/device failure on Hub-2.

  • Failover during Hub-2 device failure:
# Checking VRRP state and eBGP configuration on Hub-2 before reboot
vyos@Hub-2:~$ show vrrp
Name    Interface      VRID  State      Priority  Last Transition
------  -----------  ------  -------  ----------  -----------------
HQ      eth1v1            1  MASTER          100  20m22s

vyos@Hub-2:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map
set protocols bgp 65000 peer-group DMVPN bfd


# Rebooting Hub-2
vyos@Hub-2:~$ reboot
Are you sure you want to reboot this system? [y/N]  y


# Checking VRRP state and eBGP configuration on Hub-1
vyos@Hub-1:~$ show vrrp
Name    Interface      VRID  State      Priority  Last Transition
------  -----------  ------  -------  ----------  -----------------
HQ      eth1v1            1  MASTER          200  1m57s

vyos@Hub-1:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map
set protocols bgp 65000 peer-group DMVPN bfd


# Checking VRRP state and eBGP configuration on Hub-2 after reboot completed
vyos@Hub-2:~$ show vrrp
Name    Interface      VRID  State      Priority  Last Transition
------  -----------  ------  -------  ----------  -----------------
HQ      eth1v1            1  BACKUP          100  1m46s

vyos@Hub-2:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
set protocols bgp 65000 peer-group DMVPN bfd


# consecutive pings check from Spoke-1 to the HQ local network during the failure
--- 192.168.0.10 ping statistics ---
1182 packets transmitted, 1182 received, 0% packet loss, time 1921ms
rtt min/avg/max/mdev = 0.890/1.692/3.305/0.503 ms
vyos@Spoke-1:~$


# consecutive pings check from Spoke-3 to the Spoke-2 local network during the failure
--- 192.168.2.1 ping statistics ---
1186 packets transmitted, 1186 received, 0% packet loss, time 2100ms
rtt min/avg/max/mdev = 0.506/1.236/8.497/0.369 ms
vyos@Spoke-3:~$

From the results, we can see that the switchover performed as expected with 0 packets loss both from Spoke-1 to HQ and Spoke-3 to Spoke-2 networks.