High availability

VRRP (Virtual Router Redundancy Protocol) provides active/backup redundancy for routers. Every VRRP router has a physical IP/IPv6 address, and a virtual address. On startup, routers elect the master, and the router with the highest priority becomes the master and assigns the virtual address to its interface. All routers with lower priorities become backup routers. The master then starts sending keepalive packets to notify other routers that it’s available. If the master fails and stops sending keepalive packets, the router with the next highest priority becomes the new master and takes over the virtual address.

VRRP keepalive packets use multicast, and VRRP setups are limited to a single datalink layer segment. You can setup multiple VRRP groups (also called virtual routers). Virtual routers are identified by a VRID (Virtual Router IDentifier). If you setup multiple groups on the same interface, their VRIDs must be unique if they use the same address family, but it’s possible (even if not recommended for readability reasons) to use duplicate VRIDs on different interfaces.

Basic setup

VRRP groups are created with the set high-availability vrrp group $GROUP_NAME commands. The required parameters are interface, vrid, and address.

minimal config

set high-availability vrrp group Foo vrid 10
set high-availability vrrp group Foo interface eth0
set high-availability vrrp group Foo address 192.0.2.1/24

You can verify your VRRP group status with the operational mode run show vrrp command:

vyos@vyos# run show vrrp
Name        Interface      VRID  State    Last Transition
----------  -----------  ------  -------  -----------------
Foo         eth1             10  MASTER   2s

IPv6 support

The address parameter can be either an IPv4 or IPv6 address, but you can not mix IPv4 and IPv6 in the same group, and will need to create groups with different VRIDs specially for IPv4 and IPv6. If you want to use IPv4 + IPv6 address you can use option excluded-address

Address

The address can be configured either on the VRRP interface or on not VRRP interface.

set high-availability vrrp group Foo address 192.0.2.1/24
set high-availability vrrp group Foo address 203.0.113.22/24 interface eth2
set high-availability vrrp group Foo address 198.51.100.33/24 interface eth3

Disabling a VRRP group

You can disable a VRRP group with disable option:

set high-availability vrrp group Foo disable

A disabled group will be removed from the VRRP process and your router will not participate in VRRP for that VRID. It will disappear from operational mode commands output, rather than enter the backup state.

Exclude address

Exclude IP addresses from VRRP packets. This option excluded-address is used when you want to set IPv4 + IPv6 addresses on the same virtual interface or when used more than 20 IP addresses.

set high-availability vrrp group Foo excluded-address '203.0.113.254/24'
set high-availability vrrp group Foo excluded-address '2001:db8:aa::1/64'
set high-availability vrrp group Foo excluded-address '2001:db8:22::1/64'

Setting VRRP group priority

VRRP priority can be set with priority option:

set high-availability vrrp group Foo priority 200

The priority must be an integer number from 1 to 255. Higher priority value increases router’s precedence in the master elections.

Sync groups

A sync group allows VRRP groups to transition together.

edit high-availability vrrp
set sync-group MAIN member VLAN9
set sync-group MAIN member VLAN20

In the following example, when VLAN9 transitions, VLAN20 will also transition:

vrrp {
    group VLAN9 {
        interface eth0.9
        address 10.9.1.1/24
        priority 200
        vrid 9
    }
    group VLAN20 {
        interface eth0.20
        priority 200
        address 10.20.20.1/24
        vrid 20
    }
    sync-group MAIN {
        member VLAN20
        member VLAN9
    }
}

Warning

All items in a sync group should be similarly configured. If one VRRP group is set to a different preemption delay or priority, it would result in an endless transition loop.

Preemption

VRRP can use two modes: preemptive and non-preemptive. In the preemptive mode, if a router with a higher priority fails and then comes back, routers with lower priority will give up their master status. In non-preemptive mode, the newly elected master will keep the master status and the virtual address indefinitely.

By default VRRP uses preemption. You can disable it with the “no-preempt” option:

set high-availability vrrp group Foo no-preempt

You can also configure the time interval for preemption with the “preempt-delay” option. For example, to set the higher priority router to take over in 180 seconds, use:

set high-availability vrrp group Foo preempt-delay 180

Track

Track option to track non VRRP interface states. VRRP changes status to FAULT if one of the track interfaces in state down.

set high-availability vrrp group Foo track interface eth0
set high-availability vrrp group Foo track interface eth1

Ignore VRRP main interface faults

set high-availability vrrp group Foo track exclude-vrrp-interface

Unicast VRRP

By default VRRP uses multicast packets. If your network does not support multicast for whatever reason, you can make VRRP use unicast communication instead.

set high-availability vrrp group Foo peer-address 192.0.2.10
set high-availability vrrp group Foo hello-source-address 192.0.2.15

rfc3768-compatibility

RFC 3768 defines a virtual MAC address to each VRRP virtual router. This virtual router MAC address will be used as the source in all periodic VRRP messages sent by the active node. When the rfc3768-compatibility option is set, a new VRRP interface is created, to which the MAC address and the virtual IP address is automatically assigned.

set high-availability vrrp group Foo rfc3768-compatibility

Verification

$show interfaces ethernet eth0v10
eth0v10@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether 00:00:5e:00:01:0a brd ff:ff:ff:ff:ff:ff
inet 172.25.0.247/16 scope global eth0v10
valid_lft forever preferred_lft forever

Global options

On most scenarios, there’s no need to change specific parameters, and using default configuration is enough. But there are cases were extra configuration is needed.

set high-availability vrrp global-parameters startup_delay <1-600>

This option specifies a delay in seconds before vrrp instances start up after keepalived starts.

Gratuitous ARP

These configuration is not mandatory and in most cases there’s no need to configure it. But if necessary, Gratuitous ARP can be configured in global-parameters and/or in group section.

set high-availability vrrp global-parameters garp interval <0.000-1000>
set high-availability vrrp group <name> garp interval <0.000-1000>

Set delay between gratuitous ARP messages sent on an interface.

0 if not defined.

set high-availability vrrp global-parameters garp master-delay <1-255>
set high-availability vrrp group <name> garp master-delay <1-255>

Set delay for second set of gratuitous ARPs after transition to MASTER.

5 if not defined.

set high-availability vrrp global-parameters garp master-refresh <1-600>
set high-availability vrrp group <name> garp master-refresh <1-600>

Set minimum time interval for refreshing gratuitous ARPs while MASTER.

0 if not defined, which means no refreshing.

set high-availability vrrp global-parameters garp master-refresh-repeat <1-600>
set high-availability vrrp group <name> garp master-refresh-repeat <1-600>

Set number of gratuitous ARP messages to send at a time while MASTER.

1 if not defined.

set high-availability vrrp global-parameters garp master-repeat <1-600>
set high-availability vrrp group <name> garp master-repeat <1-600>

Set number of gratuitous ARP messages to send at a time after transition to MASTER.

5 if not defined.

Version

set high-availability vrrp global-parameters version 2|3

Set the default VRRP version to use. This defaults to 2, but IPv6 instances will always use version 3.

Scripting

VRRP functionality can be extended with scripts. VyOS supports two kinds of scripts: health check scripts and transition scripts. Health check scripts execute custom checks in addition to the master router reachability. Transition scripts are executed when VRRP state changes from master to backup or fault and vice versa and can be used to enable or disable certain services, for example.

Health check scripts

This setup will make the VRRP process execute the /config/scripts/vrrp-check.sh script every 60 seconds, and transition the group to the fault state if it fails (i.e. exits with non-zero status) three times:

set high-availability vrrp group Foo health-check script /config/scripts/vrrp-check.sh
set high-availability vrrp group Foo health-check interval 60
set high-availability vrrp group Foo health-check failure-count 3

When the vrrp group is a member of the sync group will use only the sync group health check script. This example shows how to configure it for the sync group:

set high-availability vrrp sync-group Bar health-check script /config/scripts/vrrp-check.sh
set high-availability vrrp sync-group Bar health-check interval 60
set high-availability vrrp sync-group Bar health-check failure-count 3

Transition scripts

Transition scripts can help you implement various fixups, such as starting and stopping services, or even modifying the VyOS config on VRRP transition. This setup will make the VRRP process execute the /config/scripts/vrrp-fail.sh with argument Foo when VRRP fails, and the /config/scripts/vrrp-master.sh when the router becomes the master:

set high-availability vrrp group Foo transition-script backup "/config/scripts/vrrp-fail.sh Foo"
set high-availability vrrp group Foo transition-script fault "/config/scripts/vrrp-fail.sh Foo"
set high-availability vrrp group Foo transition-script master "/config/scripts/vrrp-master.sh Foo"

To know more about scripting, check the Command Scripting section.

Virtual-server

Call for Contributions

This section needs improvements, examples and explanations.

Please take a look at the Contributing Guide for our Write Documentation.

Virtual Server allows to Load-balance traffic destination virtual-address:port between several real servers.

Algorithm

Load-balancing schedule algorithm:

  • round-robin

  • weighted-round-robin

  • least-connection

  • weighted-least-connection

  • source-hashing

  • destination-hashing

  • locality-based-least-connection

set high-availability virtual-server 203.0.113.1 algorithm 'least-connection'

Forward method

  • NAT

  • direct

  • tunnel

set high-availability virtual-server 203.0.113.1 forward-method 'nat'

Health-check

Custom health-check script allows checking real-server availability

set high-availability virtual-server 203.0.113.1 real-server 192.0.2.11 health-check script <path-to-script>

Fwmark

Firewall mark. It possible to loadbalancing traffic based on fwmark value

set high-availability virtual-server 203.0.113.1 fwmark '111'

Real server

Real server IP address and port

set high-availability virtual-server 203.0.113.1 real-server 192.0.2.11 port '80'

Example

Virtual-server can be configured with VRRP virtual address or without VRRP.

In the next example all traffic destined to 203.0.113.1 and port 8280 protocol TCP is balanced between 2 real servers 192.0.2.11 and 192.0.2.12 to port 80

Real server is auto-excluded if port check with this server fail.

set interfaces ethernet eth0 address '203.0.113.11/24'
set interfaces ethernet eth1 address '192.0.2.1/24'
set high-availability vrrp group FOO interface 'eth0'
set high-availability vrrp group FOO no-preempt
set high-availability vrrp group FOO priority '150'
set high-availability vrrp group FOO address '203.0.113.1/24'
set high-availability vrrp group FOO vrid '10'

set high-availability virtual-server 203.0.113.1 algorithm 'source-hashing'
set high-availability virtual-server 203.0.113.1 delay-loop '10'
set high-availability virtual-server 203.0.113.1 forward-method 'nat'
set high-availability virtual-server 203.0.113.1 persistence-timeout '180'
set high-availability virtual-server 203.0.113.1 port '8280'
set high-availability virtual-server 203.0.113.1 protocol 'tcp'
set high-availability virtual-server 203.0.113.1 real-server 192.0.2.11 port '80'
set high-availability virtual-server 203.0.113.1 real-server 192.0.2.12 port '80'

A firewall mark fwmark allows using multiple ports for high-availability virtual-server. It uses fwmark value.

In this example all traffic destined to ports “80, 2222, 8888” protocol TCP marks to fwmark “111” and balanced between 2 real servers. Port “0” is required if multiple ports are used.

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth1 address '192.0.2.1/24'
set interfaces ethernet eth1 description 'LAN'

set policy route PR interface 'eth0'
set policy route PR rule 10 destination port '80,2222,8888'
set policy route PR rule 10 protocol 'tcp'
set policy route PR rule 10 set mark '111'

set high-availability virtual-server vyos fwmark '111'
set high-availability virtual-server vyos protocol 'tcp'
set high-availability virtual-server vyos real-server 192.0.2.11 health-check script '/config/scripts/check-real-server-first.sh'
set high-availability virtual-server vyos real-server 192.0.2.11 port '0'
set high-availability virtual-server vyos real-server 192.0.2.12 health-check script '/config/scripts/check-real-server-second.sh'
set high-availability virtual-server vyos real-server 192.0.2.12 port '0'

set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.0.2.0/24'
set nat source rule 100 translation address 'masquerade'

Op-mode check virtual-server status

vyos@r14:~$ run show virtual-server
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  111 lc persistent 300
  -> 192.0.2.11:0                 Masq    1      0          0
  -> 192.0.2.12:0                 Masq    1      1          0