The generic name of Quality of Service or Traffic Control involves things like shaping traffic, scheduling or dropping packets, which are the kind of things you may want to play with when you have, for instance, a bandwidth bottleneck in a link and you want to somehow prioritize some type of traffic over another.
tc is a powerful tool for Traffic Control found at the Linux kernel.
However, its configuration is often considered a cumbersome task.
Fortunately, VyOS eases the job through its CLI, while using
How to make it work¶
In order to have VyOS Traffic Control working you need to follow 2 steps:
- Create a traffic policy.
- Apply the traffic policy to an interface ingress or egress.
But before learning to configure your policy, we will warn you about the different units you can use and also show you what classes are and how they work, as some policies may require you to configure them.
When configuring your traffic policy, you will have to set data rate values, watch out the units you are managing, it is easy to get confused with the different prefixes and suffixes you can use. VyOS will always show you the different units you can use.
They can be decimal prefixes.
kbit (10^3) kilobit per second mbit (10^6) megabit per second gbit (10^9) gigabit per second tbit (10^12) terabit per second kbps (8*10^3) kilobyte per second mbps (8*10^6) megabyte per second gbps (8*10^9) gigabyte per second tbps (8*10^12) terabyte per second
Or binary prefixes.
kibit (2^10 = 1024) kibibit per second mibit (2^20 = 1024^2) mebibit per second gibit (2^30 = 1024^3) gibibit per second tbit (2^40 = 1024^4) tebibit per second kibps (1024*8) kibibyte (KiB) per second mibps (1024^2*8) mebibyte (MiB) per second gibps (1024^3*8) gibibyte (GiB) per second tibps (1024^4*8) tebibyte (TiB) per second
A bit is written as bit,
kbit (kilobits per second) mbit (megabits per second) gbit (gigabits per second) tbit (terabits per second)
while a byte is written as a single b.
kbps (kilobytes per second) mbps (megabytes per second) gbps (gigabytes per second)
In the Creating a traffic policy section you will see that some of the policies use classes. Those policies let you distribute traffic into different classes according to different parameters you can choose. So, a class is just a specific type of traffic you select.
The ultimate goal of classifying traffic is to give each class a different treatment.
In order to define which traffic goes into which class, you define filters (that is, the matching criteria). Packets go through these matching rules (as in the rules of a firewall) and, if a packet matches the filter, it is assigned to that class.
In VyOS, a class is identified by a number you can choose when configuring it.
The meaning of the Class ID is not the same for every type of policy. Normally policies just need a meaningless number to identify a class (Class ID), but that does not apply to every policy. The the number of a class in a Priority Queue it does not only identify it, it also defines its priority.
set traffic-policy <policy> <policy-name> class <class-ID> match <class-matching-rule-name>
In the command above, we set the type of policy we are going to work with and the name we choose for it; a class (so that we can differentiate some traffic) and an identifiable number for that class; then we configure a matching rule (or filter) and a name for it.
A class can have multiple match filters:
set traffic-policy shaper MY-SHAPER class 30 match HTTP set traffic-policy shaper MY-SHAPER class 30 match HTTPs
A match filter can contain multiple criteria and will match traffic if all those criteria are true.
set traffic-policy shaper MY-SHAPER class 30 match HTTP ip protocol tcp set traffic-policy shaper MY-SHAPER class 30 match HTTP ip source port 80
This will match TCP traffic with source port 80.
There are many parameters you will be able to use in order to match the traffic you want for a class:
- Ethernet (protocol, destination address or source address)
- Interface name
- IPv4 (DSCP value, maximum packet length, protocol, source address, destination address, source port, destination port or TCP flags)
- IPv6 (DSCP value, maximum payload length, protocol, source address, destination address, source port, destination port or TCP flags)
- Firewall mark
- VLAN ID
When configuring your filter, you can use the
Tab key to see the many
different parameters you can configure.
vyos@vyos# set traffic-policy shaper MY-SHAPER class 30 match MY-FIRST-FILTER Possible completions: description Description for this match > ether Ethernet header match interface Interface name for this match > ip Match IP protocol header > ipv6 Match IPV6 header mark Match on mark applied by firewall vif Virtual Local Area Network (VLAN) ID for this match
As shown in the example above, one of the possibilities to match packets is based on marks done by the firewall, that can give you a great deal of flexibility.
You can also write a description for a filter:
set traffic-policy shaper MY-SHAPER class 30 match MY-FIRST-FILTER description "My filter description"
An IPv4 TCP filter will only match packets with an IPv4 header length of 20 bytes (which is the majority of IPv4 packets anyway).
IPv6 TCP filters will only match IPv6 packets with no header extension, see https://en.wikipedia.org/wiki/IPv6_packet#Extension_headers
Often you will also have to configure your default traffic in the same way you do with a class. Default can be considered a class as it behaves like that. It contains any traffic that did not match any of the defined classes, so it is like an open class, a class without matching filters.
Once a class has a filter configured, you will also have to define what you want to do with the traffic of that class, what specific Traffic-Control treatment you want to give it. You will have different possibilities depending on the Traffic Policy you are configuring.
vyos@vyos# set traffic-policy shaper MY-SHAPER class 30 Possible completions: bandwidth Bandwidth used for this class burst Burst size for this class (default: 15kb) ceiling Bandwidth limit for this class codel-quantum fq-codel - Number of bytes used as 'deficit' (default 1514) description Description for this traffic class flows fq-codel - Number of flows (default 1024) interval fq-codel - Interval (milliseconds) used to measure the delay (default 100) +> match Class matching rule name priority Priority for usage of excess bandwidth queue-limit Maximum queue size (packets) queue-type Queue type for this class set-dscp Change the Differentiated Services (DiffServ) field in the IP header target fq-codel - Acceptable minimum queue delay (milliseconds)
For instance, with
set traffic-policy shaper MY-SHAPER class 30 set-dscp EF
you would be modifying the DSCP field value of packets in that class to
Binary value Configured value Drop rate Description 101110 46
Expedited forwarding (EF) 000000 0
Best effort traffic, default 001010 10 Low Assured Forwarding(AF) 11 001100 12 Medium Assured Forwarding(AF) 12 001110 14 High Assured Forwarding(AF) 13 010010 18 Low Assured Forwarding(AF) 21 010100 20 Medium Assured Forwarding(AF) 22 010110 22 High Assured Forwarding(AF) 23 011010 26 Low Assured Forwarding(AF) 31 011100 28 Medium Assured Forwarding(AF) 32 011110 30 High Assured Forwarding(AF) 33 100010 34 Low Assured Forwarding(AF) 41 100100 36 Medium Assured Forwarding(AF) 42 100110 38 High Assured Forwarding(AF) 43
Embedding one policy into another one¶
Often we need to embed one policy into another one. It is possible to do so on classful policies, by attaching a new policy into a class. For instance, you might want to apply different policies to the different classes of a Round-Robin policy you have configured.
A common example is the case of some policies which, in order to be effective, they need to be applied to an interface that is directly connected where the bottleneck is. If your router is not directly connected to the bottleneck, but some hop before it, you can emulate the bottleneck by embedding your non-shaping policy into a classful shaping one so that it takes effect.
You can configure a policy into a class through the
set traffic-policy shaper FQ-SHAPER bandwidth 4gbit set traffic-policy shaper FQ-SHAPER default bandwidth 100% set traffic-policy shaper FQ-SHAPER default queue-type fq-codel
As shown in the last command of the example above, the queue-type setting allows these combinations. You will be able to use it in many policies.
Some policies already include other embedded policies inside. That is the case of Shaper: each of its classes use fair-queue unless you change it.
Creating a traffic policy¶
VyOS lets you control traffic in many different ways, here we will cover every possibility. You can configure as many policies as you want, but you will only be able to apply one policy per interface and direction (inbound or outbound).
Some policies can be combined, you will be able to embed a different policy that will be applied to a class of the main policy.
If you are looking for a policy for your outbound traffic but you don’t know which one you need and you don’t want to go through every possible policy shown here, our bet is that highly likely you are looking for a Shaper policy and you want to set its queues as FQ-CoDel.
This the simplest queue possible you can apply to your traffic. Traffic must go through a finite queue before it is actually sent. You must define how many packets that queue can contain.
When a packet is to be sent, it will have to go through that queue, so the packet will be placed at the tail of it. When the packet completely goes through it, it will be dequeued emptying its place in the queue and being eventually handed to the NIC to be actually sent out.
Despite the Drop-Tail policy does not slow down packets, if many packets are to be sent, they could get dropped when trying to get enqueued at the tail. This can happen if the queue has still not been able to release enough packets from its head.
This is the policy that requieres the lowest resources for the same amount of traffic. But very likely you do not need it as you cannot get much from it. Sometimes it is used just to enable logging.
Fair Queue is a work-conserving scheduler which schedules the transmission of packets based on flows, that is, it balances traffic distributing it through different sub-queues in order to ensure fairness so that each flow is able to send data in turn, preventing any single one from drowning out the rest.
In order to separate traffic, Fair Queue uses a classifier based on source address, destination address and source port. The algorithm enqueues packets to hash buckets based on those tree parameters. Each of these buckets should represent a unique flow. Because multiple flows may get hashed to the same bucket, the hashing algorithm is perturbed at configurable intervals so that the unfairness lasts only for a short while. Perturbation may however cause some inadvertent packet reordering to occur. An advisable value could be 10 seconds.
One of the uses of Fair Queue might be the mitigation of Denial of Service attacks.
When dequeuing, each hash-bucket with data is queried in a round robin fashion. You can configure the length of the queue.
Fair Queue is a non-shaping (work-conserving) policy, so it will only be useful if your outgoing interface is really full. If it is not, VyOS will not own the queue and Fair Queue will have no effect. If there is bandwidth available on the physical link, you can embed Fair-Queue into a classful shaping policy to make sure it owns the queue.
The FQ-CoDel policy distributes the traffic into 1024 FIFO queues and tries to provide good service between all of them. It also tries to keep the length of all the queues short.
FQ-CoDel fights bufferbloat and reduces latency without the need of complex configurations. It has become the new default Queueing Discipline for the interfaces of some GNU/Linux distributions.
It uses a stochastic model to classify incoming packets into different flows and is used to provide a fair share of the bandwidth to all the flows using the queue. Each flow is managed by the CoDel queuing discipline. Reordering within a flow is avoided since Codel internally uses a FIFO queue.
FQ-CoDel is based on a modified Deficit Round Robin (DRR) queue scheduler with the CoDel Active Queue Management (AQM) algorithm operating on each queue.
FQ-Codel is a non-shaping (work-conserving) policy, so it will only be useful if your outgoing interface is really full. If it is not, VyOS will not own the queue and FQ-Codel will have no effect. If there is bandwidth available on the physical link, you can embed FQ-Codel into a classful shaping policy to make sure it owns the queue. If you are not sure if you need to embed your FQ-CoDel policy into a Shaper, do it.
FQ-CoDel is tuned to run ok with its default parameters at 10Gbit speeds. It might work ok too at other speeds without configuring anything, but here we will explain some cases when you might want to tune its parameters.
When running it at 1Gbit and lower, you may want to reduce the queue-limit to 1000 packets or less. In rates like 10Mbit, you may want to set it to 600 packets.
If you are using FQ-CoDel embedded into Shaper and you have large rates (100Mbit and above), you may consider increasing quantum to 8000 or higher so that the scheduler saves CPU.
On low rates (below 40Mbit) you may want to tune quantum down to something like 300 bytes.
At very low rates (below 3Mbit), besides tuning quantum (300 keeps being ok) you may also want to increase target to something like 15ms and increase interval to something around 150 ms.
A simple example of an FQ-CoDel policy working inside a Shaper one.
set traffic-policy shaper FQ-CODEL-SHAPER bandwidth 2gbit set traffic-policy shaper FQ-CODEL-SHAPER default bandwidth 100% set traffic-policy shaper FQ-CODEL-SHAPER default queue-type fq-codel
Limiter is one of those policies that uses classes (Ingress qdisc is actually a classless policy but filters do work in it).
The limiter performs basic ingress policing of traffic flows. Multiple classes of traffic can be defined and traffic limits can be applied to each class. Although the policer uses a token bucket mechanism internally, it does not have the capability to delay a packet as a shaping mechanism does. Traffic exceeding the defined bandwidth limits is directly dropped. A maximum allowed burst can be configured too.
You can configure classes (up to 4090) with different settings and a default policy which will be applied to any traffic not matching any of the configured classes.
In the case you want to apply some kind of shaping to your inbound traffic, check the ingress-shaping section.
Once the matching rules are set for a class, you can start configuring how you want matching traffic to behave.
VyOS Network Emulator policy emulates the conditions you can suffer in a real network. You will be able to configure things like rate, burst, delay, packet loss, packet corruption or packet reordering.
This could be helpful if you want to test how an application behaves under certain network conditions.
The Priority Queue is a classful scheduling policy. It does not delay packets (Priority Queue is not a shaping policy), it simply dequeues packets according to their priority.
Priority Queue, as other non-shaping policies, is only useful if your outgoing interface is really full. If it is not, VyOS will not own the queue and Priority Queue will have no effect. If there is bandwidth available on the physical link, you can embed Priority Queue into a classful shaping policy to make sure it owns the queue. In that case packets can be prioritized based on DSCP.
Up to seven queues -defined as classes with different priorities- can be configured. Packets are placed into queues based on associated match criteria. Packets are transmitted from the queues in priority order. If classes with a higher priority are being filled with packets continuously, packets from lower priority classes will only be transmitted after traffic volume from higher priority classes decreases.
In Priority Queue we do not define clases with a meaningless class ID number but with a class priority number (1-7). The lower the number, the higher the priority.
As with other policies, you can define different type of matching rules for your classes:
vyos@vyos# set traffic-policy priority-queue MY-PRIO class 3 match MY-MATCH-RULE Possible completions: description Description for this match > ether Ethernet header match interface Interface name for this match > ip Match IP protocol header > ipv6 Match IPV6 header mark Match on mark applied by firewall vif Virtual Local Area Network (VLAN) ID for this match
As with other policies, you can embed other policies into the classes
(and default) of your Priority Queue policy through the
vyos@vyos# set traffic-policy priority-queue MY-PRIO class 3 queue-type Possible completions: fq-codel Fair Queue Codel fair-queue Stochastic Fair Queue (SFQ) drop-tail First-In-First-Out (FIFO) priority Priority queueing based on DSCP random-detect Random Early Detection (RED)
A simple Random Early Detection (RED) policy would start randomly dropping packets from a queue before it reaches its queue limit thus avoiding congestion. That is good for TCP connections as the gradual dropping of packets acts as a signal for the sender to decrease its transmission rate.
In contrast to simple RED, VyOS’ Random-Detect uses a Generalized Random Early Detect policy that provides different virtual queues based on the IP Precedence value so that some virtual queues can drop more packets than others.
This is achieved by using the first three bits of the ToS (Type of Service) field to categorize data streams and, in accordance with the defined precedence parameters, a decision is made.
IP precedence as defined in RFC 791:
Precedence Priority 7 Network Control 6 Internetwork Control 5 CRITIC/ECP 4 Flash Override 3 Flash 2 Immediate 1 Priority 0 Routine
Random-Detect could be useful for heavy traffic. One use of this algorithm might be to prevent a backbone overload. But only for TCP (because dropped packets could be retransmitted), not for UDP.
When configuring a Random-Detect policy: the higher the precedence number, the higher the priority.
The default values for the minimum-threshold depend on IP precedence:
Precedence default min-threshold 7 16 6 15 5 14 4 13 3 12 2 11 1 10 0 9
If the average queue size is lower than the min-threshold, an arriving packet will be placed in the queue.
In the case the average queue size is between min-threshold and max-threshold, then an arriving packet would be either dropped or placed in the queue, it will depend on the defined mark-probability.
If the current queue size is larger than queue-limit, then packets will be dropped. The average queue size depends on its former average size and its current one.
If max-threshold is set but min-threshold is not, then **min-threshold is scaled to 50% of max-threshold.
In principle, values must be
Rate-Control is a classless policy that limits the packet flow to a set rate. It is a pure shaper, it does not schedule traffic. Traffic is filtered based on the expenditure of tokens. Tokens roughly correspond to bytes.
Short bursts can be allowed to exceed the limit. On creation, the Rate-Control traffic is stocked with tokens which correspond to the amount of traffic that can be burst in one go. Tokens arrive at a steady rate, until the bucket is full.
As a reference: for 10mbit/s on Intel, you might need at least 10kbyte buffer if you want to reach your configured rate.
A very small buffer will soon start dropping packets.
Rate-Control is a CPU-friendly policy. You might consider using it when you just simply want to slow traffic down.
Each class is assigned a deficit counter (the number of bytes that a flow is allowed to transmit when it is its turn) initialized to quantum. Quantum is a parameter you configure which acts like a credit of fix bytes the counter receives on each round. Then the Round-Robin policy starts moving its Round Robin pointer through the queues. If the deficit counter is greater than the packet’s size at the head of the queue, this packet will be sent and the value of the counter will be decremented by the packet size. Then, the size of the next packet will be compared to the counter value again, repeating the process. Once the queue is empty or the value of the counter is insufficient, the Round-Robin pointer will move to the next queue. If the queue is empty, the value of the deficit counter is reset to 0.
At every round, the deficit counter adds the quantum so that even large packets will have their opportunity to be dequeued.
As with other policies, Round-Robin can embed another policy into a
class through the
vyos@vyos# set traffic-policy round-robin DRR class 10 queue-type Possible completions: fq-codel Fair Queue Codel fair-queue Stochastic Fair Queue (SFQ) drop-tail First-In-First-Out (FIFO) priority Priority queueing based on DSCP
The Shaper policy does not guarantee a low delay, but it does guarantee bandwidth to different traffic classes and also lets you decide how to allocate more traffic once the guarantees are met.
Each class can have a guaranteed part of the total bandwidth defined for the whole policy, so all those shares together should not be higher than the policy’s whole bandwidth.
If guaranteed traffic for a class is met and there is room for more traffic, the ceiling parameter can be used to set how much more bandwidth could be used. If guaranteed traffic is met and there are several classes willing to use their ceilings, the priority parameter will establish the order in which that additional traffic will be allocated. Priority can be any number from 0 to 7. The lower the number, the higher the priority.
As with other policies, Shaper can embed other policies into its
classes through the
queue-type setting and then configure their
vyos@vyos# set traffic-policy shaper HTB class 10 queue-type Possible completions: fq-codel Fair Queue Codel fair-queue Stochastic Fair Queue (SFQ) drop-tail First-In-First-Out (FIFO) priority Priority queueing based on DSCP random-detect Random Early Detection (RED)
vyos@vyos# set traffic-policy shaper HTB class 10 Possible completions: bandwidth Bandwidth used for this class burst Burst size for this class (default: 15kb) ceiling Bandwidth limit for this class codel-quantum fq-codel - Number of bytes used as 'deficit' (default 1514) description Description for this traffic class flows fq-codel - Number of flows (default 1024) interval fq-codel - Interval (milliseconds) used to measure the delay (default 100) +> match Class matching rule name priority Priority for usage of excess bandwidth queue-limit Maximum queue size (packets) queue-type Queue type for this class set-dscp Change the Differentiated Services (DiffServ) field in the IP header target fq-codel - Acceptable minimum queue delay (milliseconds)
If you configure a class for VoIP traffic, don’t give it any ceiling, otherwise new VoIP calls could start when the link is available and get suddenly dropped when other classes start using their assigned bandwidth share.
A simple example of Shaper using priorities.
set traffic-policy shaper MY-HTB bandwidth '50mbit' set traffic-policy shaper MY-HTB class 10 bandwidth '20%' set traffic-policy shaper MY-HTB class 10 match DSCP ip dscp 'EF' set traffic-policy shaper MY-HTB class 10 queue-type 'fq-codel' set traffic-policy shaper MY-HTB class 20 bandwidth '10%' set traffic-policy shaper MY-HTB class 20 ceiling '50%' set traffic-policy shaper MY-HTB class 20 match PORT666 ip destination port '666' set traffic-policy shaper MY-HTB class 20 priority '3' set traffic-policy shaper MY-HTB class 20 queue-type 'fair-queue' set traffic-policy shaper MY-HTB class 30 bandwidth '10%' set traffic-policy shaper MY-HTB class 30 ceiling '50%' set traffic-policy shaper MY-HTB class 30 match ADDRESS30 ip source address '192.168.30.0/24' set traffic-policy shaper MY-HTB class 30 priority '5' set traffic-policy shaper MY-HTB class 30 queue-type 'fair-queue' set traffic-policy shaper MY-HTB default bandwidth '10%' set traffic-policy shaper MY-HTB default ceiling '100%' set traffic-policy shaper MY-HTB default priority '7' set traffic-policy shaper MY-HTB default queue-type 'fair-queue'
Applying a traffic policy¶
Once a traffic-policy is created, you can apply it to an interface:
set interfaces etherhet eth0 traffic-policy out WAN-OUT
You can only apply one policy per interface and direction, but you could reuse a policy on different interfaces and directions:
set interfaces ethernet eth0 traffic-policy in WAN-IN set interfaces etherhet eth0 traffic-policy out WAN-OUT set interfaces etherhet eth1 traffic-policy in LAN-IN set interfaces etherhet eth1 traffic-policy out LAN-OUT set interfaces ethernet eth2 traffic-policy in LAN-IN set interfaces ethernet eth2 traffic-policy out LAN-OUT set interfaces etherhet eth3 traffic-policy in TWO-WAY-POLICY set interfaces etherhet eth3 traffic-policy out TWO-WAY-POLICY set interfaces etherhet eth4 traffic-policy in TWO-WAY-POLICY set interfaces etherhet eth4 traffic-policy out TWO-WAY-POLICY
Getting queueing information¶
The case of ingress shaping¶
For the ingress traffic of an interface, there is only one policy you can directly apply, a Limiter policy. You cannot apply a shaping policy directly to the ingress traffic of any interface because shaping only works for outbound traffic.
This workaround lets you apply a shaping policy to the ingress traffic by first redirecting it to an in-between virtual interface (Intermediate Functional Block). There, in that virtual interface, you will be able to apply any of the policies that work for outbound traffic, for instance, a shaping one.
That is how it is possible to do the so-called “ingress shaping”.
set traffic-policy shaper MY-INGRESS-SHAPING bandwidth 1000kbit set traffic-policy shaper MY-INGRESS-SHAPING default bandwidth 1000kbit set traffic-policy shaper MY-INGRESS-SHAPING default queue-type fair-queue set interfaces input ifb0 traffic-policy out MY-INGRESS-SHAPING set interfaces ethernet eth0 redirect ifb0
Do not configure IFB as the first step. First create everything else
of your traffic-policy, and then you can configure IFB.
Otherwise you might get the
RTNETLINK answer: File exists error,
which can be solved with
sudo ip link delete ifb0.