Dual-Hub DMVPN with VyOS
DMVPN is a Dynamic Multipoint VPN technology that provides the capability for creating a dynamic-mesh VPN network without having to pre-configure (static) all possible tunnel end-point peers those simplifying deployment and management of the newly added remote sites. There are 3 main protocols primarily used to implement DMVPN:
NHRP - provides the dynamic tunnel endpoint discovery mechanism (endpoint registration, and endpoint discovery/lookup)
mGRE - provides the tunnel encapsulation itself
IPSec - protocols handle the key exchange, and crypto mechanism
For this example we are using the following devices:
2 x Hubs
3 x Spokes
1 x Client device (VPC)
1 x ISP router
The following software was used in the creation of this document:
Operating system: VyOS
Version: 1.3-beta-202112090443
Image name: vyos-1.3-beta-202112090443-amd64.iso
Topology
Network Addressing and Protocol Parameters
The following ip addressing schema used for the devices IPv4 connectivity:
10.X1.0.0/30 - p2p Hubs to ISP networks, where X is Hub site number |
10.Y1.1.0/24 - p2p Spokes to ISP networks(DHCP), where Y is Spoke site number |
172.16.253.0/29 - tunnels addressing for Hub-1 connections |
172.16.254.0/29 - tunnels addressing for Hub-2 connections |
192.168.0.0/24 - HQ site local network |
192.168.Z.0/24 - remote sites local network, where Z is Spoke site number |
eBGP parameters for the routers:
AS65000 - HQ (Hub-1 and Hub-2) |
AS6500X - Spokes, where X is Spoke site number |
Configuration
Step-1: Basic connectivity configuration
Hub-1:
set interfaces ethernet eth0 address '10.11.0.1/30'
set interfaces ethernet eth1 address '192.168.0.1/24'
set protocols static route 0.0.0.0/0 next-hop 10.11.0.2
set system host-name 'Hub-1'
Hub-2:
set interfaces ethernet eth0 address '10.21.0.1/30'
set interfaces ethernet eth1 address '192.168.0.2/24'
set protocols static route 0.0.0.0/0 next-hop 10.21.0.2
set system host-name 'Hub-2'
Spoke-1:
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.1.1/24'
set system host-name 'Spoke-1'
Spoke-2:
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.2.1/24'
set system host-name 'Spoke-2'
Spoke-3:
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.3.1/24'
set system host-name 'Spoke-3'
ISP-1:
set interfaces ethernet eth0 address '10.11.0.2/30'
set interfaces ethernet eth1 address '10.21.0.2/30'
set interfaces ethernet eth2 address '10.31.1.1/24'
set interfaces ethernet eth3 address '10.21.1.1/24'
set interfaces ethernet eth4 address '10.11.1.1/24'
set service dhcp-server shared-network-name SPK-1 authoritative
set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 default-router '10.11.1.1'
set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 range 1 start '10.11.1.10'
set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 range 1 stop '10.11.1.100'
set service dhcp-server shared-network-name SPK-2 authoritative
set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 default-router '10.21.1.1'
set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 range 1 start '10.21.1.10'
set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 range 1 stop '10.21.1.100'
set service dhcp-server shared-network-name SPK-3 authoritative
set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 default-router '10.31.1.1'
set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 range 1 start '10.31.1.10'
set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 range 1 stop '10.31.1.100'
set system host-name 'ISP1'
Step-2: VRRP configuration for HQ Local network redundancy
Here we are using VRRP as a local redundancy protocol between Hub-1 and Hub-2. Initially, Hub-1 operates as an Active and Hub-2 as a Standby router. Additionally, health-check and script are used to track uplinks and properly switch mastership between Hub nodes based on the upstream router reachability (ISP-1). Note, that before adding local paths to the scripts into configuration, you have to create and make them executable first.
Hub-1 and Hub-2 VRRP health-check script:
/config/scripts/vrrp-check.sh
#!/bin/bash
eth0status="$(cat /sys/class/net/eth0/operstate | grep 'up')"
if [[ ! -z ${eth0status} ]]; then
eth0gw="$(ip -j r show 0.0.0.0/0 dev eth0 | awk 'match($0, /\"gateway":\"([[:digit:]\.]+)/, gw) {print gw[1]}')"
if [[ ! -z $eth0gw ]]; then
/bin/ping -I eth0 -c 1 -W 1 $eth0gw && exit 0 || exit 1
else
exit 1
fi
else
#Exit 0 because eth0 down is handled by vrrp transition
exit 0
fi
Note: some parts of the script might be dependent on your network topology and connectivity. Be careful before using it on your own devices.
Hub-1 and Hub-2 VRRP configuration:
Hub-1
set high-availability vrrp group HQ health-check failure-count '3'
set high-availability vrrp group HQ health-check interval '1'
set high-availability vrrp group HQ health-check script '/config/scripts/vrrp-check.sh'
set high-availability vrrp group HQ interface 'eth1'
set high-availability vrrp group HQ no-preempt
set high-availability vrrp group HQ priority '200'
set high-availability vrrp group HQ rfc3768-compatibility
set high-availability vrrp group HQ virtual-address '192.168.0.254/24'
set high-availability vrrp group HQ vrid '1'
Hub-2:
set high-availability vrrp group HQ health-check failure-count '3'
set high-availability vrrp group HQ health-check interval '1'
set high-availability vrrp group HQ health-check script '/config/scripts/vrrp-check.sh'
set high-availability vrrp group HQ interface 'eth1'
set high-availability vrrp group HQ no-preempt
set high-availability vrrp group HQ priority '100'
set high-availability vrrp group HQ rfc3768-compatibility
set high-availability vrrp group HQ virtual-address '192.168.0.254/24'
set high-availability vrrp group HQ vrid '1'
Step-3: DMVPN configuration between Hub and Spoke devices
This section provides an example configuration of the DMVPN enabled devices. Hub devices are configured with static IPv4 addresses on the uplink interfaces while Spoke devices receive addresses dynamically from a pre-defined DHCP pool configured on ISP router. For redundancy purposes, we use 1 tunnel interface on each Hub device and 2 tunnel interfaces on Spoke devices destined to each of the Hubs. For the optimal tunnel operation timers are significantly decreased and set to the following values:
NHRP tunnel holding time - 30 seconds
IKE DPD enabled with “restart” action set, interval 3 and timeout 30 seconds
Note: these values are used only for the lab demonstration and may not suit exclusive production networks.
Hub-1:
set interfaces tunnel tun100 address '172.16.253.134/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '10.11.0.1'
set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 multicast 'dynamic'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
Hub-2:
set interfaces tunnel tun100 address '172.16.254.134/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '2'
set interfaces tunnel tun100 source-address '10.21.0.1'
set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 multicast 'dynamic'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
Spoke-1:
set interfaces tunnel tun100 address '172.16.253.131/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '0.0.0.0'
set interfaces tunnel tun200 address '172.16.254.131/29'
set interfaces tunnel tun200 encapsulation 'gre'
set interfaces tunnel tun200 multicast 'enable'
set interfaces tunnel tun200 parameters ip key '2'
set interfaces tunnel tun200 source-address '0.0.0.0'
set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
set protocols nhrp tunnel tun100 multicast 'nhs'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set protocols nhrp tunnel tun200 cisco-authentication 'secret'
set protocols nhrp tunnel tun200 holding-time '30'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
set protocols nhrp tunnel tun200 multicast 'nhs'
set protocols nhrp tunnel tun200 redirect
set protocols nhrp tunnel tun200 shortcut
set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
Spoke-2:
set interfaces tunnel tun100 address '172.16.253.132/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '0.0.0.0'
set interfaces tunnel tun200 address '172.16.254.132/29'
set interfaces tunnel tun200 encapsulation 'gre'
set interfaces tunnel tun200 multicast 'enable'
set interfaces tunnel tun200 parameters ip key '2'
set interfaces tunnel tun200 source-address '0.0.0.0'
set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
set protocols nhrp tunnel tun100 multicast 'nhs'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set protocols nhrp tunnel tun200 cisco-authentication 'secret'
set protocols nhrp tunnel tun200 holding-time '30'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
set protocols nhrp tunnel tun200 multicast 'nhs'
set protocols nhrp tunnel tun200 redirect
set protocols nhrp tunnel tun200 shortcut
set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
Spoke-3:
set interfaces tunnel tun100 address '172.16.253.133/29'
set interfaces tunnel tun100 encapsulation 'gre'
set interfaces tunnel tun100 multicast 'enable'
set interfaces tunnel tun100 parameters ip key '1'
set interfaces tunnel tun100 source-address '0.0.0.0'
set interfaces tunnel tun200 address '172.16.254.133/29'
set interfaces tunnel tun200 encapsulation 'gre'
set interfaces tunnel tun200 multicast 'enable'
set interfaces tunnel tun200 parameters ip key '2'
set interfaces tunnel tun200 source-address '0.0.0.0'
set protocols nhrp tunnel tun100 cisco-authentication 'secret'
set protocols nhrp tunnel tun100 holding-time '30'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
set protocols nhrp tunnel tun100 multicast 'nhs'
set protocols nhrp tunnel tun100 redirect
set protocols nhrp tunnel tun100 shortcut
set protocols nhrp tunnel tun200 cisco-authentication 'secret'
set protocols nhrp tunnel tun200 holding-time '30'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
set protocols nhrp tunnel tun200 multicast 'nhs'
set protocols nhrp tunnel tun200 redirect
set protocols nhrp tunnel tun200 shortcut
set vpn ipsec esp-group ESP-HUB compression 'disable'
set vpn ipsec esp-group ESP-HUB lifetime '1800'
set vpn ipsec esp-group ESP-HUB mode 'transport'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
set vpn ipsec ike-group IKE-HUB close-action 'none'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
set vpn ipsec ike-group IKE-HUB lifetime '3600'
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
Step-4: Enabling eBGP as a Dynamic Routing Protocol between Hubs and Spokes
For the simplified and better network management we’re using eBGP for routing information exchange between devices. As we’re using Active-Standby mode in this example, Hub-2 is configured with AS-prepand as an export route-policy and VRRP transition scripts are used for switching mastership based on the current link/device state. Also, we use multihop BFD for faster eBGP failure detection.
Hub-1 and Hub-2 VRRP transition scripts:
/config/scripts/vrrp-master.sh
#!/bin/vbash
if [ $(id -gn) != vyattacfg ]; then
exec sg vyattacfg "$0 $*"
fi
source /opt/vyatta/etc/functions/script-template
configure
delete protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export AS65000-PREP
commit
exit
/config/scripts/vrrp-fail.sh
#!/bin/vbash
if [ $(id -gn) != vyattacfg ]; then
exec sg vyattacfg "$0 $*"
fi
source /opt/vyatta/etc/functions/script-template
configure
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export AS65000-PREP
commit
exit
Note: some parts of the script might be dependent on your network topology and connectivity. Be careful before using it on your own devices.
Hub devices configuration:
Hub-1:
set high-availability vrrp group HQ transition-script backup '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script fault '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script master '/config/scripts/vrrp-master.sh'
set high-availability vrrp group HQ transition-script stop '/config/scripts/vrrp-fail.sh'
set policy route-map AS65000-PREP rule 1 action 'permit'
set policy route-map AS65000-PREP rule 1 set as-path-prepend '65000 65000 65000'
set protocols bfd peer 172.16.253.131 interval multiplier '3'
set protocols bfd peer 172.16.253.131 interval receive '300'
set protocols bfd peer 172.16.253.131 interval transmit '300'
set protocols bfd peer 172.16.253.131 multihop
set protocols bfd peer 172.16.253.131 source address '172.16.253.134'
set protocols bfd peer 172.16.253.132 interval multiplier '3'
set protocols bfd peer 172.16.253.132 interval receive '300'
set protocols bfd peer 172.16.253.132 interval transmit '300'
set protocols bfd peer 172.16.253.132 multihop
set protocols bfd peer 172.16.253.132 source address '172.16.253.134'
set protocols bfd peer 172.16.253.133 interval multiplier '3'
set protocols bfd peer 172.16.253.133 interval receive '300'
set protocols bfd peer 172.16.253.133 interval transmit '300'
set protocols bfd peer 172.16.253.133 multihop
set protocols bfd peer 172.16.253.133 source address '172.16.253.134'
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 parameters network-import-check
set protocols bgp 65000 peer-group DMVPN bfd
Hub-2:
set high-availability vrrp group HQ transition-script backup '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script fault '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group HQ transition-script master '/config/scripts/vrrp-master.sh'
set high-availability vrrp group HQ transition-script stop '/config/scripts/vrrp-fail.sh'
set policy route-map AS65000-PREP rule 1 action 'permit'
set policy route-map AS65000-PREP rule 1 set as-path-prepend '65000 65000 65000'
set protocols bfd peer 172.16.254.131 interval multiplier '3'
set protocols bfd peer 172.16.254.131 interval receive '300'
set protocols bfd peer 172.16.254.131 interval transmit '300'
set protocols bfd peer 172.16.254.131 multihop
set protocols bfd peer 172.16.254.131 source address '172.16.254.134'
set protocols bfd peer 172.16.254.132 interval multiplier '3'
set protocols bfd peer 172.16.254.132 interval receive '300'
set protocols bfd peer 172.16.254.132 interval transmit '300'
set protocols bfd peer 172.16.254.132 multihop
set protocols bfd peer 172.16.254.132 source address '172.16.254.134'
set protocols bfd peer 172.16.254.133 interval multiplier '3'
set protocols bfd peer 172.16.254.133 interval receive '300'
set protocols bfd peer 172.16.254.133 interval transmit '300'
set protocols bfd peer 172.16.254.133 multihop
set protocols bfd peer 172.16.254.133 source address '172.16.254.134'
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 parameters network-import-check
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
set protocols bgp 65000 peer-group DMVPN bfd
Spoke devices configuration:
Spoke-1:
set protocols bfd peer 172.16.253.134 interval multiplier '3'
set protocols bfd peer 172.16.253.134 interval receive '300'
set protocols bfd peer 172.16.253.134 interval transmit '300'
set protocols bfd peer 172.16.253.134 multihop
set protocols bfd peer 172.16.253.134 source address '172.16.253.131'
set protocols bfd peer 172.16.254.134 interval multiplier '3'
set protocols bfd peer 172.16.254.134 interval receive '300'
set protocols bfd peer 172.16.254.134 interval transmit '300'
set protocols bfd peer 172.16.254.134 multihop
set protocols bfd peer 172.16.254.134 source address '172.16.254.131'
set protocols bgp 65001 address-family ipv4-unicast network 192.168.1.0/24
set protocols bgp 65001 neighbor 172.16.253.134 address-family ipv4-unicast
set protocols bgp 65001 neighbor 172.16.253.134 bfd
set protocols bgp 65001 neighbor 172.16.253.134 remote-as '65000'
set protocols bgp 65001 neighbor 172.16.254.134 address-family ipv4-unicast
set protocols bgp 65001 neighbor 172.16.254.134 bfd
set protocols bgp 65001 neighbor 172.16.254.134 remote-as '65000'
set protocols bgp 65001 parameters log-neighbor-changes
Spoke-2:
set protocols bfd peer 172.16.253.134 interval multiplier '3'
set protocols bfd peer 172.16.253.134 interval receive '300'
set protocols bfd peer 172.16.253.134 interval transmit '300'
set protocols bfd peer 172.16.253.134 multihop
set protocols bfd peer 172.16.253.134 source address '172.16.253.132'
set protocols bfd peer 172.16.254.134 interval multiplier '3'
set protocols bfd peer 172.16.254.134 interval receive '300'
set protocols bfd peer 172.16.254.134 interval transmit '300'
set protocols bfd peer 172.16.254.134 multihop
set protocols bfd peer 172.16.254.134 source address '172.16.254.132'
set protocols bgp 65002 address-family ipv4-unicast network 192.168.2.0/24
set protocols bgp 65002 neighbor 172.16.253.134 address-family ipv4-unicast
set protocols bgp 65002 neighbor 172.16.253.134 bfd
set protocols bgp 65002 neighbor 172.16.253.134 remote-as '65000'
set protocols bgp 65002 neighbor 172.16.254.134 address-family ipv4-unicast
set protocols bgp 65002 neighbor 172.16.254.134 bfd
set protocols bgp 65002 neighbor 172.16.254.134 remote-as '65000'
set protocols bgp 65002 parameters log-neighbor-changes
Spoke-3:
set protocols bfd peer 172.16.253.134 interval multiplier '3'
set protocols bfd peer 172.16.253.134 interval receive '300'
set protocols bfd peer 172.16.253.134 interval transmit '300'
set protocols bfd peer 172.16.253.134 multihop
set protocols bfd peer 172.16.253.134 source address '172.16.253.133'
set protocols bfd peer 172.16.254.134 interval multiplier '3'
set protocols bfd peer 172.16.254.134 interval receive '300'
set protocols bfd peer 172.16.254.134 interval transmit '300'
set protocols bfd peer 172.16.254.134 multihop
set protocols bfd peer 172.16.254.134 source address '172.16.254.133'
set protocols bgp 65003 address-family ipv4-unicast network 192.168.3.0/24
set protocols bgp 65003 neighbor 172.16.253.134 address-family ipv4-unicast
set protocols bgp 65003 neighbor 172.16.253.134 bfd
set protocols bgp 65003 neighbor 172.16.253.134 remote-as '65000'
set protocols bgp 65003 neighbor 172.16.254.134 address-family ipv4-unicast
set protocols bgp 65003 neighbor 172.16.254.134 bfd
set protocols bgp 65003 neighbor 172.16.254.134 remote-as '65000'
set protocols bgp 65003 parameters log-neighbor-changes
Note: In case if you’re using VyOS version that has a VRRP transition scripts issues after a device reboot, as a temporary solution you may add postconfig-bootup script that reloads keepalived process additionally after the device booted.
Hub devices /config/scripts/vyos-postconfig-bootup.script:
#!/bin/sh
# This script is executed at boot time after VyOS configuration is fully applied.
# Any modifications required to work around unfixed bugs
# or use services not available through the VyOS CLI system can be placed here.
echo "Reloading VRRP process"
sudo systemctl restart keepalived.service
echo "VRRP process reload completed"
Step-5: Verification
Now, it’s time to check that all protocols are working as expected and mastership during the failover switches correctly between Hub devices.
Checking VRRP state between Hub-1 and Hub-2:
vyos@Hub-1:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
HQ eth1v1 1 MASTER 200 14s
vyos@Hub-2:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
HQ eth1v1 1 BACKUP 100 29s
Checking NHRP and eBGP sessions between Hub and Spoke devices:
vyos@Hub-1:~$ show nhrp tunnel
Status: ok
Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.134
Flags: up
Interface: tun100
Type: local
Protocol-Address: 172.16.253.134/32
Flags: up
Interface: tun100
Type: dynamic
Protocol-Address: 172.16.253.131/32
NBMA-Address: 10.11.1.11
Flags: up
Expires-In: 0:23
Interface: tun100
Type: dynamic
Protocol-Address: 172.16.253.133/32
NBMA-Address: 10.31.1.11
Flags: up
Expires-In: 0:22
Interface: tun100
Type: dynamic
Protocol-Address: 172.16.253.132/32
NBMA-Address: 10.21.1.11
Flags: up
Expires-In: 0:21
vyos@Hub-1:~$ show bgp summary
IPv4 Unicast Summary:
BGP router identifier 192.168.0.1, local AS number 65000 vrf-id 0
BGP table version 20
RIB entries 7, using 1344 bytes of memory
Peers 3, using 64 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
172.16.253.131 4 65001 26519 26526 0 0 0 00:43:38 1 4
172.16.253.132 4 65002 26545 26540 0 0 0 00:46:36 1 4
172.16.253.133 4 65003 26528 26520 0 0 0 00:41:59 1 4
Total number of neighbors 3
vyos@Hub-2:~$ show nhrp tunnel
Status: ok
Interface: tun100
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.134
Flags: up
Interface: tun100
Type: local
Protocol-Address: 172.16.254.134/32
Flags: up
Interface: tun100
Type: dynamic
Protocol-Address: 172.16.254.132/32
NBMA-Address: 10.21.1.11
Flags: up
Expires-In: 0:28
Interface: tun100
Type: dynamic
Protocol-Address: 172.16.254.131/32
NBMA-Address: 10.11.1.11
Flags: up
Expires-In: 0:21
Interface: tun100
Type: dynamic
Protocol-Address: 172.16.254.133/32
NBMA-Address: 10.31.1.11
Flags: up
Expires-In: 0:20
vyos@Hub-2:~$ show bgp summary
IPv4 Unicast Summary:
BGP router identifier 192.168.0.2, local AS number 65000 vrf-id 0
BGP table version 14
RIB entries 7, using 1344 bytes of memory
Peers 3, using 64 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
172.16.254.131 4 65001 26516 26516 0 0 0 00:43:03 1 4
172.16.254.132 4 65002 26563 26562 0 0 0 00:48:27 1 4
172.16.254.133 4 65003 26518 26516 0 0 0 00:42:20 1 4
Total number of neighbors 3
Checking BFD sessions between Hub and Spoke devices:
vyos@Hub-1:~$ show protocols bfd peers
Session count: 6
SessionId LocalAddress PeerAddress Status
========= ============ =========== ======
3600626867 172.16.253.134 172.16.253.133 up
1123939978 172.16.253.134 172.16.253.131 up
374394280 172.16.253.134 172.16.253.132 up
1786735466 172.16.253.134 172.16.253.132 up
1440522544 172.16.253.134 172.16.253.131 up
1106910911 172.16.253.134 172.16.253.133 up
vyos@Hub-2:~$ show protocols bfd peers
Session count: 6
SessionId LocalAddress PeerAddress Status
========= ============ =========== ======
2442966178 172.16.254.134 172.16.254.133 up
393258775 172.16.254.134 172.16.254.131 up
2990308682 172.16.254.134 172.16.254.133 up
2267910949 172.16.254.134 172.16.254.132 up
3542474595 172.16.254.134 172.16.254.131 up
4239538185 172.16.254.134 172.16.254.132 up
Checking routing information and connectivity between Hub and Spoke devices:
vyos@Hub-1:~$ show ip bgp
BGP table version is 20, local router ID is 192.168.0.1, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 192.168.0.0/24 0.0.0.0 0 32768 i
*> 192.168.1.0/24 172.16.253.131 0 0 65001 i
*> 192.168.2.0/24 172.16.253.132 0 0 65002 i
*> 192.168.3.0/24 172.16.253.133 0 0 65003 i
Displayed 4 routes and 4 total paths
vyos@Hub-2:~$ show ip bgp
BGP table version is 14, local router ID is 192.168.0.2, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 192.168.0.0/24 0.0.0.0 0 32768 i
*> 192.168.1.0/24 172.16.254.131 0 0 65001 i
*> 192.168.2.0/24 172.16.254.132 0 0 65002 i
*> 192.168.3.0/24 172.16.254.133 0 0 65003 i
Displayed 4 routes and 4 total paths
vyos@Spoke-1:~$ show ip bgp
BGP table version is 19, local router ID is 192.168.1.1, vrf id 0
Default local pref 100, local AS 65001
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.0.0/24 172.16.254.134 0 0 65000 65000 65000 65000 i
*> 172.16.253.134 0 0 65000 i
*> 192.168.1.0/24 0.0.0.0 0 32768 i
* 192.168.2.0/24 172.16.254.132 0 65000 65000 65000 65000 65002 i
*> 172.16.253.132 0 65000 65002 i
* 192.168.3.0/24 172.16.254.133 0 65000 65000 65000 65000 65003 i
*> 172.16.253.133 0 65000 65003 i
Displayed 4 routes and 7 total paths
As you can see, Hub-2 announces routes with longer(prepended) AS path as we’ve configured it previously, those, traffic towards HQ subnet will be forwarded over Hub-1 which is operating as an Active VRRP router. Let’s check connectivity and the path from Spoke-1 to the HQ local network:
vyos@Spoke-1:~$ ping 192.168.0.10 count 5 interface 192.168.1.1
PING 192.168.0.10 (192.168.0.10) from 192.168.1.1 : 56(84) bytes of data.
64 bytes from 192.168.0.10: icmp_seq=1 ttl=63 time=3.50 ms
64 bytes from 192.168.0.10: icmp_seq=2 ttl=63 time=2.45 ms
64 bytes from 192.168.0.10: icmp_seq=3 ttl=63 time=2.34 ms
64 bytes from 192.168.0.10: icmp_seq=4 ttl=63 time=2.20 ms
64 bytes from 192.168.0.10: icmp_seq=5 ttl=63 time=2.44 ms
--- 192.168.0.10 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 11ms
rtt min/avg/max/mdev = 2.195/2.583/3.496/0.465 ms
vyos@Spoke-1:~$ traceroute 192.168.0.10
traceroute to 192.168.0.10 (192.168.0.10), 30 hops max, 60 byte packets
1 172.16.253.134 (172.16.253.134) 0.913 ms 0.884 ms 0.819 ms
2 192.168.0.10 (192.168.0.10) 1.352 ms 1.446 ms 1.391 ms
From the output, we can confirm successful connectivity between Spoke-1 and HQ local networks. From the traceroute we see that the traffic pass through the Hub-1.
Now, let’s check traffic between Spoke sites. Based on our configuration, Spoke sites are using shortcut for direct reachability between each other. First, let’s check NHRP tunnels before passing the traffic between Spoke-1 and Spoke-2:
vyos@Spoke-1:~$ show nhrp tunnel
Status: ok
Interface: tun200
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.131
Flags: up
Interface: tun200
Type: local
Protocol-Address: 172.16.254.131/32
Flags: up
Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.131
Flags: up
Interface: tun100
Type: local
Protocol-Address: 172.16.253.131/32
Flags: up
Interface: tun200
Type: static
Protocol-Address: 172.16.254.134/29
NBMA-Address: 10.21.0.1
Flags: used up
Interface: tun100
Type: static
Protocol-Address: 172.16.253.134/29
NBMA-Address: 10.11.0.1
Flags: used up
vyos@Spoke-2:~$ show nhrp tunnel
Status: ok
Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.132
Flags: up
Interface: tun100
Type: local
Protocol-Address: 172.16.253.132/32
Flags: up
Interface: tun200
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.132
Flags: up
Interface: tun200
Type: local
Protocol-Address: 172.16.254.132/32
Flags: up
Interface: tun100
Type: static
Protocol-Address: 172.16.253.134/29
NBMA-Address: 10.11.0.1
Flags: used up
Interface: tun200
Type: static
Protocol-Address: 172.16.254.134/29
NBMA-Address: 10.21.0.1
After passing traffic we could see that there is additional shortcut tunnel created between Spoke-1 and Spoke-2 for the direct communication:
vyos@Spoke-1:~$ ping 192.168.2.1 count 5 interface 192.168.1.1
PING 192.168.2.1 (192.168.2.1) from 192.168.1.1 : 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.820 ms
64 bytes from 192.168.2.1: icmp_seq=3 ttl=64 time=1.13 ms
64 bytes from 192.168.2.1: icmp_seq=4 ttl=63 time=1.41 ms
64 bytes from 192.168.2.1: icmp_seq=5 ttl=64 time=0.988 ms
--- 192.168.2.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 0.820/1.075/1.412/0.197 ms
vyos@Spoke-1:~$ traceroute 192.168.2.1
traceroute to 192.168.2.1 (192.168.2.1), 30 hops max, 60 byte packets
1 192.168.2.1 (192.168.2.1) 1.172 ms 1.109 ms 1.151 ms
vyos@Spoke-1:~$ show nhrp tunnel
Status: ok
Interface: tun200
Type: local
Protocol-Address: 172.16.254.135/32
Alias-Address: 172.16.254.131
Flags: up
Interface: tun200
Type: local
Protocol-Address: 172.16.254.131/32
Flags: up
Interface: tun100
Type: local
Protocol-Address: 172.16.253.135/32
Alias-Address: 172.16.253.131
Flags: up
Interface: tun100
Type: local
Protocol-Address: 172.16.253.131/32
Flags: up
Interface: tun200
Type: static
Protocol-Address: 172.16.254.134/29
NBMA-Address: 10.21.0.1
Flags: used up
____________________________________
Interface: tun100
Type: cached
Protocol-Address: 172.16.253.132/32
NBMA-Address: 10.21.1.11
Flags: used up
Expires-In: 0:24
____________________________________
Interface: tun100
Type: static
Protocol-Address: 172.16.253.134/29
NBMA-Address: 10.11.0.1
Flags: used up
The same applies to the rest of the devices and works with the same logic. As we’ve already confirmed successfull connectivity between Hub and Spoke devices, let’s check failover process.
Failover on the health-check failure on Hub-1:
# disabling interface towards Hub-1 on ISP router
vyos@ISP1:~$ configure
[edit]
vyos@ISP1# set interfaces ethernet eth0 disable
[edit]
vyos@ISP1# commit
[edit]
vyos@ISP1#
# checking VRRP state and eBGP configuration on Hub-1:
vyos@Hub-1:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
HQ eth1v1 1 FAULT 200 1m15s
vyos@Hub-1:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
set protocols bgp 65000 peer-group DMVPN bfd
# consecutive pings check from Spoke-1 to the HQ local network during the failure
--- 192.168.0.10 ping statistics ---
223 packets transmitted, 219 received, 1.79372% packet loss, time 679ms
rtt min/avg/max/mdev = 0.918/2.191/2.957/0.364 ms
vyos@Spoke-1:~$
# consecutive pings check from Spoke-3 to the Spoke-2 local network during the failure
--- 192.168.2.1 ping statistics ---
265 packets transmitted, 265 received, 0% packet loss, time 690ms
rtt min/avg/max/mdev = 0.663/1.128/2.272/0.285 ms
vyos@Spoke-3:~$
Note: After bringing ISP interface towards Hub-1 back to UP state, VRRP state will remain unchanged due to “no-preempt” option enabled under the VRRP configuration on the Hub-1 and Hub-2 and will be changed only during link/device failure on Hub-2.
Failover during Hub-2 device failure:
# Checking VRRP state and eBGP configuration on Hub-2 before reboot
vyos@Hub-2:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
HQ eth1v1 1 MASTER 100 20m22s
vyos@Hub-2:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map
set protocols bgp 65000 peer-group DMVPN bfd
# Rebooting Hub-2
vyos@Hub-2:~$ reboot
Are you sure you want to reboot this system? [y/N] y
# Checking VRRP state and eBGP configuration on Hub-1
vyos@Hub-1:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
HQ eth1v1 1 MASTER 200 1m57s
vyos@Hub-1:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map
set protocols bgp 65000 peer-group DMVPN bfd
# Checking VRRP state and eBGP configuration on Hub-2 after reboot completed
vyos@Hub-2:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
HQ eth1v1 1 BACKUP 100 1m46s
vyos@Hub-2:~$ show configuration commands | match bgp
set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
set protocols bgp 65000 peer-group DMVPN bfd
# consecutive pings check from Spoke-1 to the HQ local network during the failure
--- 192.168.0.10 ping statistics ---
1182 packets transmitted, 1182 received, 0% packet loss, time 1921ms
rtt min/avg/max/mdev = 0.890/1.692/3.305/0.503 ms
vyos@Spoke-1:~$
# consecutive pings check from Spoke-3 to the Spoke-2 local network during the failure
--- 192.168.2.1 ping statistics ---
1186 packets transmitted, 1186 received, 0% packet loss, time 2100ms
rtt min/avg/max/mdev = 0.506/1.236/8.497/0.369 ms
vyos@Spoke-3:~$
From the results, we can see that the switchover performed as expected with 0 packets loss both from Spoke-1 to HQ and Spoke-3 to Spoke-2 networks.