Overview
This document defines Quality of Service (QoS) usage in RouterOS based on Marvell Prestera DX switch chips (CRS3xx, CRS5xx series switches, and CCR2116, CCR2216 routers).
QoS is a set of features in network switches that allow network administrators to prioritize traffic and allocate network resources to ensure that important data flows smoothly and with low latency.
The primary function of QoS in network switches is to manage network traffic in a way that meets the specific requirements of different types of network applications. For example, voice and video data require low latency and minimal packet loss to ensure high-quality communication, while file transfers and other data applications can tolerate higher levels of latency and packet loss.
QoS works by identifying the type of traffic flowing through the switch and assigning it a priority level based on its requirements. The switch can then use this information to alter packet headers and prioritize the flow of traffic, ensuring that higher-priority traffic is given preferential treatment over lower-priority traffic.
Planned QoS implementation phases:
- QoS Marking. QoS profile matching by ingress packet headers, then egress header alternation according to the assigned QoS profiles (introduced in RouterOS v7.10).
- QoS Enforcement. Avoid or resolve congestion based on the assigned QoS profiles and traffic shaping (introduced in RouterOS v7.13 for 98DX224S, 98DX226S, and 98DX3236 switch chips, and extended to all capable switch chips starting from RouterOS v7.15).
- QoS Policy. Assign QoS profiles via ACL rules (introduced in RouterOS v7.15).
- Extra QoS Features: WRED (Weighted Random Early Detection), ECN notification and processing (introduced in RouterOS v7.15 to capable switch chips).
- Traffic shaping.
QoS Terminology
These terms will be used throughout the article.
- QoS - Quality of Service.
- ACL - Access Control List, a set of switch rules used to filter network traffic based on specified criteria.
- AQM - Active Queue Management.
- DSCP - Differentiated Services Code Point, a 6-bit field in the IP header used to prioritize network traffic.
- ECN - Explicit Congestion Notification.
- PCP - Priority Code Point, a 3-bit field in the VLAN header used to prioritize traffic within a VLAN.
- WRED - Weighted Random Early Detection.
/in/eth/sw/
a shortcut for/interface/ethernet/switch/
. The shortcut works in CLI, too.
QoS Device Support
Model | Switch Chip | QoS Profiles | QoS Maps | Tx Managers | WRED | ECN | Port/Queue Usage Stats |
---|---|---|---|---|---|---|---|
CCR2116-12G-4S+ | 98DX3255 | 1024 | 12 | 15 | ✔ | ✔ | Unreliable 1 |
CCR2216-1G-12XS-2XQ | 98DX8525 | 1024 | 12 | 15 | ✔ | ✔ | Max fill 2 |
CRS305-1G-4S+ | 98DX3236 | 128 | 1 | 8 | Current values | ||
CRS309-1G-8S+ | 98DX8208 | 1024 | 12 | 15 | ✔ | ✔ | Unreliable |
CRS310-1G-5S-4S+ | 98DX226S | 128 | 1 | 8 | Current values | ||
CRS312-4C+8XG | 98DX8212 | 1024 | 12 | 15 | ✔ | ✔ | Unreliable |
CRS317-1G-16S+ | 98DX8216 | 1024 | 12 | 15 | ✔ | ✔ | Unreliable |
CRS318-1Fi-15Fr-2S | 98DX224S | 128 | 1 | 8 | Current values | ||
CRS318-16P-2S+ | 98DX226S | 128 | 1 | 8 | Current values | ||
CRS326-24G-2S+ | 98DX3236 | 128 | 1 | 8 | Current values | ||
CRS326-24S+2Q+ | 98DX8332 | 1024 | 12 | 15 | ✔ | ✔ | Unreliable |
CRS328-24P-4S+ | 98DX3236 | 128 | 1 | 8 | Current values | ||
CRS328-4C-20S-4S+ | 98DX3236 | 128 | 1 | 8 | Current values | ||
CRS354-48G-4S+2Q+ | 98DX3257 | 1024 | 12 | 15 | ✔ | ✔ | Unreliable |
CRS504-4XQ | 98DX4310 | 1024 | 12 | 15 | ✔ | ✔ | Max fill |
CRS510-8XS-2XQ | 98DX4310 | 1024 | 12 | 15 | ✔ | ✔ | Max fill |
CRS518-16XS-2XQ | 98DX8525 | 1024 | 12 | 15 | ✔ | ✔ | Max fill |
1 Due to hardware limitations, some switch chip models may break traffic flow while accessing QoS port/queue usage data.
2 The device gathers max queue fill statistics instead of displaying the current usage values. Use the reset-counters command to reset those stats.
Applications and Usage Examples
Basic Configuration Example
In this example, we define just one QoS level - VoIP (IP Telephony) on top of the standard "Best Effort" class. Let's imagine that we have a CRS326-24G-2S+ device where:
- all ports are bridged and using vlan-filtering;
- sfp-sfpplus1 is a VLAN trunk connected to another switch;
- ether1-ether9 are dedicated ports for IP phones;
- ether10-ether24 are standard ports for host connection;
First, we need to define QoS profiles. Defined dscp
and pcp
values will be used in forwarded packets on egress:
/interface ethernet switch qos profile add dscp=46 name=voip pcp=5 traffic-class=5
Port-based QoS profile assignment on dedicated ports for IP phones applies to ingress traffic. Other Ethernet ports will use the default qos-profile
(where dscp=0
and pcp=0
):
/interface ethernet switch qos port set ether1 profile=voip set ether2 profile=voip set ether3 profile=voip set ether4 profile=voip set ether5 profile=voip set ether6 profile=voip set ether7 profile=voip set ether8 profile=voip set ether9 profile=voip
Starting from RouterOS v7.13, QoS port settings moved from /interface/ethernet/switch/port
to /interface/ethernet/switch/qos/port
. The "qos-" prefix from the respective fields has been removed (since all fields are qos-related anyway).
The trunk port receives both types of QoS traffic. We need to create VLAN priority mapping with the QoS profile and enable qos-trust-l2
to differentiate them:
/interface ethernet switch qos map vlan add pcp=5 profile=voip /interface ethernet switch port set sfp-sfpplus1 trust-l2=trust
Finally, enable QoS hardware offloading for the above settings to start working:
/interface ethernet switch set switch1 qos-hw-offloading=yes
It is possible to verify the port QoS settings with print
command:
[admin@MikroTik] /interface/ethernet/switch/qos/port print Columns: NAME, SWITCH, PROFILE, MAP, TRUST-L2, TRUST-L3 # NAME SWITCH PROFILE MAP TRUST-L2 TRUST-L3 TX-MANAGER 0 ether1 switch1 voip default ignore ignore default 1 ether2 switch1 voip default ignore ignore default 2 ether3 switch1 voip default ignore ignore default 3 ether4 switch1 voip default ignore ignore default 4 ether5 switch1 voip default ignore ignore default 5 ether6 switch1 voip default ignore ignore default 6 ether7 switch1 voip default ignore ignore default 7 ether8 switch1 voip default ignore ignore default 8 ether9 switch1 voip default ignore ignore default 9 ether10 switch1 default default ignore ignore default 10 ether11 switch1 default default ignore ignore default 11 ether12 switch1 default default ignore ignore default 12 ether13 switch1 default default ignore ignore default 13 ether14 switch1 default default ignore ignore default 14 ether15 switch1 default default ignore ignore default 15 ether16 switch1 default default ignore ignore default 16 ether17 switch1 default default ignore ignore default 17 ether18 switch1 default default ignore ignore default 18 ether19 switch1 default default ignore ignore default 19 ether20 switch1 default default ignore ignore default 20 ether21 switch1 default default ignore ignore default 21 ether22 switch1 default default ignore ignore default 22 ether23 switch1 default default ignore ignore default 23 ether24 switch1 default default ignore ignore default 24 sfp-sfpplus1 switch1 default default trust ignore default 25 sfp-sfpplus2 switch1 default default ignore ignore default 26 switch1-cpu switch1
Now incoming packets on ports ether1-ether9 are marked with a Priority Code Point (PCP) value of 5 and a Differentiated Services Code Point (DSCP) value of 46, and incoming packets on ports ether10-ether24 are marked with PCP and DSCP values of 0. When packets are incoming to sfp-sfpplus1 port, any packets with a PCP value of 5 or higher will retain their PCP value of 5 and DSCP value of 46, while all other packets will be marked with PCP and DSCP values of 0.
Dante
Starting from RouterOS v7.15, all MikroTik QoS-Capable devices comply with Dante.
Dante hardware use the following DSCP / Diffserv priority values for traffic prioritization.
Dante Priority | Usage | DSCP Label | DSCP Value |
---|---|---|---|
High | Time critical PTP events | CS7 | 56 |
Medium | Audio, PTP | EF | 46 |
Low | (reserved) | CS1 | 8 |
None | Other traffic | BE | 0 |
The example assumes that the switch is using its default configuration, which includes a default "bridge" interface and all Ethernet interfaces added as bridge ports, and any of these interfaces could be used for Dante.
First, create QoS Profiles to match Dante traffic classes, there is already a pre-existing "default" profile that corresponds to Dante's None priority.
/interface/ethernet/switch/qos/profile add name=dante-ptp dscp=56 pcp=7 traffic-class=7 add name=dante-audio dscp=46 pcp=5 traffic-class=5 add name=dante-low dscp=8 pcp=1 traffic-class=0
Then, create a QoS mapping to match QoS profiles based on DSCP values.
/interface/ethernet/switch/qos/map/ip add dscp=56 profile=dante-ptp add dscp=46 profile=dante-audio add dscp=8 profile=dante-low
Configure hardware queues to enforce QoS on Dante traffic.
/interface/ethernet/switch/qos/tx-manager/queue set [find where traffic-class>=2] schedule=strict-priority set [find where traffic-class<2] schedule=low-priority-group weight=1
Dante's High and Medium priority traffic is scheduled in strict order. The devices transmits time critical PTP packets until queue7 gets empty, then proceeds with audio (queue5). Low and other traffic gets transmitted only when PTP and audio queues are empty. Since Dante does not define priority order between Low and Other traffic (usually, CS1 has lower priority than Best Effort), and the Low traffic class is reserved for future use anyway, we treat both traffic types equally by putting both into the same group with the same weight. Feel free to change the CS1/BE traffic scheduling according to the requirements if some Dante hardware in your network use the low-priority traffic class.
The next step is enable trust mode for incomming Layer3 packets (IP DSCP field):
/interface/ethernet/switch/qos/port set [find] trust-l3=keep
Finally, enable QoS hardware offloading for the above settings to start working:
/interface ethernet switch set switch1 qos-hw-offloading=yes
When using Dante in multicast mode, it is beneficial to enable IGMP snooping on the switch. This feature directs traffic only to ports with subscribed devices, preventing unnecessary flooding. Additionally, enabling an IGMP querier (if not already enabled on another device in the same LAN), adjusting query intervals, and activating fast-leave can further optimize multicast performance.
/interface/bridge set [find name=bridge] igmp-snooping=yes multicast-querier=yes query-interval=60s /interface/bridge/port set [find] fast-leave=yes
QoS Marking
Understanding Map ranges
In order to avoid defining all possible PCP and DSCP mappings, RouterOS allows setting the minimal PCP and DSCP values for QoS Profile mapping.
In the following example, PCP values 0-2 use the default QoS profile, 3-4 - streaming, 5 - voip, and 6-7 - control.
/interface ethernet switch qos map vlan add pcp=3 profile=streaming add pcp=5 profile=voip add pcp=6 profile=control
Since the pcp
parameter identifies the minimum value, all packets with a higher PCP value match too. If such behavior is undesired, add mapping for higher values. The next example sets voip profile for pcp=5 only. Packets with PCP values 6 or 7 are reset back to the default profile.
/interface ethernet switch qos map vlan add pcp=5 profile=voip add pcp=6 profile=default
Understanding Port, Profile, and Map relation
Each switch port has Layer2 and Layer3 trust settings that will change how ingress packets are classified into QoS profiles and what PCP and DSCP values will be used. Below are tables that describe all possible options:
qos-trust-l2 | qos-trust-l3 | Behavior |
---|---|---|
ignore | ignore | The port is considered untrusted. Both headers are ignored, and the port's profile is forced to all ingress packets. This is the default setting. |
ignore | trust | Trust the Layer 3 header. Use the DSCP field from the IP header of ingress packets for QoS profile lookup (see /in/eth/sw/qos/map/ip ). If the lookup fails (no QoS profiles are mapped to the given DSCP value), the default QoS profile is used (not the switch port's QoS profile). The switch port's profile field is used only for non-IP traffic. |
ignore | keep | Trust the Layer 3 header. Use the DSCP field from the IP header of ingress packets for QoS profile lookup (see /in/eth/sw/qos/map/ip ). If the lookup fails, the default QoS profile is used. The switch port's profile field is used only for non-IP traffic. If the forwarded/routed packet is VLAN-tagged, its PCP value is set from the selected QoS profile. However, the original DSCP value of the packet is kept intact. |
trust | ignore | Trust the Layer 2 header, but ignore L3. If an ingress packet is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan ). If the lookup fails (no QoS profiles are mapped to the given PCP value), the default QoS profile is used. The switch port's profile field is used only for untagged traffic. |
trust | trust | Trust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip ). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan ). If the VLAN-to-QoS lookup fails, use the default QoS profile. Non-IP untagged packets use the switch port's profile. |
trust | keep | The same as trust+trust, but the original DSCP value is preserved in forwarded/routed packets. |
keep | ignore | Trust the Layer 2 header but ignore L3. If an ingress packet is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan ). If the lookup fails (no QoS profiles are mapped to the given PCP value), the default QoS profile is used. The switch port's profile field is used only for untagged traffic. If the packet is VLAN-tagged on both ingress and egress, the original PCP value is kept. |
keep | trust | Trust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip ). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan ). If the VLAN-to-QoS lookup fails, use the default QoS profile. Non-IP untagged packets use the switch port's profile. If the packet is VLAN-tagged on both ingress and egress, the original PCP value is kept. The DSCP value in forwarded/routed packets is set from the selected QoS profile. |
keep | keep | Trust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip ). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan ). If the VLAN-to-QoS lookup fails, use the default QoS profile. Non-IP untagged packets use the switch port's profile. Keep both the original PCP and/or DSCP values intact in cases of VLAN-tagged and/or IP packets, respectively. |
Port settings | The selected QoS profile and the source for PCP / DSCP field values in forwarded/routed packets | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
qos-trust-l2 | qos-trust-l3 | VLAN-Tagged IP | Untagged IP | VLAN-Tagged Non-IP | Untagged Non-IP | ||||||||
QoS Profile | PCP | DSCP | QoS Profile | PCP 1 | DSCP | QoS Profile | PCP | DSCP | QoS Profile | PCP 1 | DSCP | ||
ignore | ignore | profile | profile | profile | profile | profile | profile | profile | profile | - | profile | profile | - |
ignore | trust | map/ip | map/ip | map/ip | map/ip | map/ip | map/ip | profile | profile | - | profile | profile | - |
ignore | keep | map/ip | map/ip | original | map/ip | map/ip | original | profile | profile | - | profile | profile | - |
trust | ignore | map/vlan | map/vlan | map/vlan | profile | profile | profile | map/vlan | map/vlan | - | profile | profile | - |
trust | trust | map/ip | map/ip | map/ip | map/ip | map/ip | map/ip | map/vlan | map/vlan | - | profile | profile | - |
trust | keep | map/ip | map/ip | original | map/ip | map/ip | original | map/vlan | map/vlan | - | profile | profile | - |
keep | ignore | map/vlan | original | map/vlan | profile | profile | profile | map/vlan | original | - | profile | profile | - |
keep | trust | map/ip | original | map/ip | map/ip | profile | map/ip | map/vlan | original | - | profile | profile | - |
keep | keep | map/ip | original | original | map/ip | profile | original | map/vlan | original | - | profile | profile | - |
1 applies only when ingress traffic is untagged, but the egress needs to be VLAN-tagged.
QoS Marking via Switch Rules (ACL)
Starting from RouterOS v7.15, it is possible to assign QoS profiles via Switch Rules (ACL).
Sub-menu:
/interface/ethernet/switch/rule
New/Changed Properties | Description |
---|---|
new-qos-profile (name) | The name of the QoS profile to assign to the matched packets. |
keep-qos-fields (yes | no; Default: no) | Should the original values of QoS fields (PCP, DSCP) be kept (yes), or replace them with the ones from the assigned QoS profile (no)? Relevant only if new-qos-profile is set. |
new-vlan-priority (0..7) | Deprecated and should be replaced with the respective new-qos-profile. Kept for backward compatibility. Relevant only if qos-hw-offloading=no. |
The following example assigns a QoS profile based on the source MAC address.
/interface ethernet switch rule add new-qos-profile=stream ports=ether1,ether2 src-mac-address=00:01:02:00:00:00/FF:FF:FF:00:00:00 switch=switch1 add new-qos-profile=voip ports=ether1,ether2 src-mac-address=04:05:06:00:00:00/FF:FF:FF:00:00:00 switch=switch1
QoS Enforcement
Hardware Queues
Each switch port has eight hardware transmission (tx) queues (queue0..queue7). Each queue corresponds to a traffic class (tc0..tc7) set by a QoS profile. Each ingress packet gets assigned to a QoS profile, which, in turn, determines the traffic class for tx queue selection on the egress port.
Hardware queues are of variable size - set by the Transmission Manager. Moreover, multiple ports and/or queues can share resources with each other (so-called Shared Buffers). For example, a device with 25 ports has memory (buffers) to queue 1200 packets in total. If we split the resources equally, each port gets 48 exclusive buffers with a maximum of 6 packets per queue (48/8) - which is usually insufficient to absorb even a short burst of traffic. However, choosing to share 50% of the buffers leaves each port with 24 exclusive buffers (3 per queue), but at the same time, a single queue can grow up to 603 buffers (3 exclusive + 600 shared).
RouterOS allows enabling/disabling the shared pool for each queue individually - for example, to prevent low-priority traffic from consuming the entire hardware memory. In addition, port buffer limits may prevent a single low-speed port from consuming the entire shared pool. See QoS Settings and Transmission Manager for details.
The default, best-effort (PCP=0, DSCP=0) traffic class is 1, while the lowest priority (PCP=1) has traffic class 0.
Hardware Resources
The hardware (switch chips) has limited resources (memory). There are two main hardware resources that are relevant to QoS:
- Packet descriptors - contain packet control information (target port, header alternation, etc).
- Data buffers - memory chunks containing the actual payload. Buffer size depends on the switch chip model. Usually - 256 bytes.
One packet descriptor may use multiple buffers (depending on the payload size); buffers may be shared by multiple descriptors - in cases of multicast/broadcast. If the hardware does not have enough free descriptors or buffers, the packet gets dropped (tail-drop).
Hardware resources can be limited per destination type (multicast/unicast), per port, and per each tx queue. If any limits are reached, no more packets can be enqueued for transmission, and further packets get dropped.
RouterOS obscures low-level hardware information, allowing to set resource limits either in terms of packets or a percentage of the total amount. RouterOS automatically calculates the required hardware descriptor and buffer count based on the user-specified packet limit and port's MTU. Moreover, RouterOS comes with preconfigured hardware resources, so there is no need to do a manual configuration in common QoS environments.
Changing any hardware resource allocation parameter in runtime results in a temporary device halt when no packets can be enqueued nor transmitted. Temporary packet loss is expected while the device is forwarding traffic.
Resource Saving
Since reallocating hardware resources in runtime is not an option, RouterOS cannot automatically free queue buffers reserved for inactive ports. Those buffers remain unused. However, if the user clearly knows that the specific ports will never be used (e.g., stay physically disconnected), the respective queue resources can be manually freed by introducing the "offline" tx-manager with minimum resources:
/interface/ethernet/switch/qos/tx-manager/ add name=offline comment="use this for always disconnected ports" /interface/ethernet/switch/qos/tx-manager/queue/ set [find where tx-manager=offline] queue-buffers=1 use-shared-buffers=no /interface/ethernet/switch/qos/port set [find where !running] tx-manager=offline
Traffic Prioritization
The hardware provides two types of traffic transmission prioritization:
- Strict Priority - traffic from higher queues is always transmitted first;
- Weighted Priority Groups - multiple queues participate in packet transmission scheduling at the same time.
Strict priority queues are straightforward. If the highest priority queue (Q7) has packets, those are transmitted first. When Q7 is empty, packets from Q6 get transmitted, and so on. The packets from the lowest priority queue (Q0) are transmitted only if all other queues are empty.
The downside of strict prioritization is increased latency in lower queues while "overprioritizing" higher queues. Suppose the acceptable latency of TC5 is 20ms, TC3 - 50ms. Traffic appearing in Q5 gets immediately transmitted due to the strict priority of the queue, adding extra latency to every packet in the lower queues (Q4..Q0). A packet burst in Q5 (e.g., a start of a voice call) may temporarily "paralyze" Q3, increasing TC3 latencies over the acceptable 50ms (or even causing packet drops due to full queue) while TC5 packets get transmitted at <1ms (way below the 20ms limit). Slightly sacrificing TC5 latency by transmitting TC3 packets in between would make everybody happy. That Weighted Priority Groups are for.
Weighted Priority Groups schedule traffic for transmission from multiple queues (group members) in a weighted round-robin manner. A queue's weight sets the number of packets transmitted from the queue in each round. For example, if Q2, Q1, and Q0 are the group members, and their weights are 3, 2, and 1, respectively, the scheduler transmits 3 packets from Q2, 2 - from Q1, and 1 - from Q0. The actual Tx order is "Q2, Q1, Q0, Q2, Q1, Q2" - for even fairer scheduling.
There are two hardware groups: low-priority-group
and high-priority-group
. There is a strict priority ordering between the two groups: the low-priority-group is transmitting only when all queues in the high-priority-group are empty. However, it is possible to use only one group for all queues.
The default (built-in) RouterOS queue setup is listed below. Q3-Q5 share the bandwidth within the high-priority group, where packets are transmitted while Q6 and Q7 are empty. Q0-Q2 are the members of the low-priority-group, where packets are transmitted while Q3-Q7 are empty.
[admin@MikroTik] /interface/ethernet/switch/qos/tx-manager/queue> print Columns: TX-MANAGER, TRAFFIC-CLASS, SCHEDULE, WEIGHT, QUEUE-BUFFERS, USE-SHARED-BUFFERS # TX-MANAGER TRAFFIC-CLASS SCHEDULE WEIGHT QUEUE-BUFFERS USE-SHARED-BUFFERS 0 default 0 low-priority-group 1 auto no 1 default 1 low-priority-group 2 auto yes 2 default 2 low-priority-group 3 auto yes 3 default 3 high-priority-group 3 auto yes 4 default 4 high-priority-group 4 auto yes 5 default 5 high-priority-group 5 auto yes 6 default 6 strict-priority auto yes 7 default 7 strict-priority auto yes
It is recommended that all group members are adjacent to each other.
Active Queue Management (AQM)
Weighted Random Early Detection (WRED)
WRED is a per-queue congestion control mechanism that signals congestion events to the end-points by dropping packets. WRED relies on the existence of rate throttling mechanisms in the end-points that react to packet loss, such as TCP/IP. WRED uses a randomized packet drop algorithm in an attempt to anticipate congestion events and respond to them by throttling traffic rates before the congestion actually happens. The randomness property of WRED prevents throughput collapse related to the global synchronization of TCP flows.
WRED can be enabled/disabled per each queue in each Tx Manager. Disable WRED for lossless traffic! Also, there is no reason to enable WRED on high-speed ports where congestion should not happen in the first place.
The behavior is controlled via WRED margins. WRED margin is the distance to the queue/pool buffer limit (cap) - where a random packet drop begins. A different margin can be applied to queues that use or don't use shared buffers. A queue that uses a shared pool may set a bigger WRED margin due to a higher overall cap (queue buffers + shared pool). RouterOS automatically chooses the actual WRED margin values according to queue or shared pool capacities. The user may shift the margins in one way or another via QoS Settings.
For example, if queue1-packet-cap=96, and WRED margin is 32 (assuming use-shared-buffers=no), then:
- first 64 packets are always enqueued (96 - 32 = 64).
- WRED kicks in starting from 65th packet; the chance of 65th packet to be dropped is 1/32 or roughly 3%; the formula: (65 - 64) / 32;
- the drop chance of 72th packet is 25%: (72 - 64) / 32 = 8/32 = 1/4;
- half of the newly enquining packets are dropped when the queue fill level reaches 80 packets: (80 - 64) / 32 = 16/32 = 1/2;
- 75% of the packets are dropped at the fill level 88: (88 - 64) / 32 = 24/32 = 3/4.
When WRED is enabled (wred=yes), the queue cap is rounded to the applied WRED margin. In the above example, setting wred-queue-margin=64 raises queue1-packet-cap to 128. That, in turn, may lower the resources available to other queues, such as shared buffers. A much safer option is to raise wred-shared-queue-margin which may reduce the shared buffers available for the affected queue but not the shared pool itself. For example, if: "wred-shared-queue-margin=256, use-shared-buffers=yes, wred=yes, shared-pool-index=0, queue2-packet-cap=30, and shared-pool0-packet-cap=900", queue2 can use up to 768 buffers (30+900=930, rounded down by the scale of 256), and WRED starting at 512, while the other ports/queues still may use the remaining 162 buffers (30+900-768) of the shared pool.
Choosing a WRED margin value is a tradeoff between congestion anticipation and burst absorption. Setting a higher WRED margin may lead to earlier traffic rate throttling and, therefore, resolve congestion. On the other hand, a high margin leads to packet drops in limited traffic bursts that could be absorbed by the queue buffers and transformed losslessly if WRED didn't kick in. For instance, initiating a remote database connection usually starts with heavier traffic ("packet burst") at the initialization phase; then, the traffic rate drops down to a "reasonable" level. Any packet drop during the initialization phase leads to nothing but a slower database connection due to the need for retransmission. Hence, lowering the WRED margin or entirely disabling WRED on such traffic is advised. The opposite case is video streaming. Early congestion detection helps select a comfortable streaming rate without losing too much bandwidth on retransmission or/and "overshooting" by sacrificing the quality level by too much.
Use Switch Rules (ACL) or other QoS Marking techniques to differentiate traffic and put packets into queues with desired WRED settings.
The following script only applies WRED to TCP/IP traffic by redirecting it to queue2. UDP and other packets are left in queue1 - since their end-points usually cannot respond to early drops. Queue1 and queue2 are scheduled equally - without prioritizing one queue over another.
/interface/ethernet/switch/qos/profile add name=tcp-wred traffic-class=2 pcp=0 dscp=0 # move TCP traffic to queue2 /interface/ethernet/switch/rule add new-qos-profile=tcp-wred ports=ether1,ether2,ether3,ether4 protocol=tcp switch=switch1 # set the same scheduling priority (weight) between queue1 and queue2 # apply WRED only to queue2 - TCP traffic /interface/ethernet/switch/qos/tx-manager/queue/ set [find where traffic-class=1] weight=2 schedule=low-priority-group use-shared-buffers=yes shared-pool-index=0 wred=no set [find where traffic-class=2] weight=2 schedule=low-priority-group use-shared-buffers=yes shared-pool-index=0 wred=yes
Explicit Congestion Notification (ECN)
Some switch chips can perform ECN marking of IP packets on the hardware level, according to RFC 3168. Hardware ECN marking is based on the WRED mechanism, but instead of dropping IP packets, they are marked with CE (Congestion Experienced, binary 11) in the ECN field (two least significant bits in IPv4/TOS or IPv6/TrafficClass octet). Only ECN-Capable IP packets may be marked - those with the ECN field value of ECT(1) or ECT(0) (binary 01 or 10, respectively). Not ECN-Capable Transport packets (ECN=00) never get marked. If a packet already has the CE mark (ECN=11), it never gets cleared, even if the device does not experience congestion.
Set ecn=yes in Tx Manager to enable ECN marking. The per-queue ECN setting is unavailable due to hardware limitations. ECN and WRED share the same queue fill threshold: wred-shared-queue-margin (see QoS Settings).
ECN marking mechanism requires the respective Tx queues to use shared buffers (use-shared-buffers=yes).
The packet receives the CE mark if all conditions below are met:
- The packet is either IPv4 or IPv6.
- The ECN field value in IP header is either ECT(1) or ECT(0).
- Egress port's Tx Manager has ecn=yes.
- The assigned Tx Queue uses shared buffers (use-shared-buffers=yes).
- The Tx Queue detects congestion via WRED margins.
Property Reference
Switch settings
Sub-menu:
/interface/ethernet/switch
Switch QoS settings (in addition to the existing ones).
Property | Description |
---|---|
qos-hw-offloading (yes | no; Default: no) | Allows enabling QoS for the given switch chip (if the latter supports QoS). |
Port settings
Sub-menu:
/interface/ethernet/switch/qos/port
Starting from RouterOS v7.13, QoS port settings moved from /interface/ethernet/switch/port
to /interface/ethernet/switch/qos/port
. The "qos-" prefix from the respective fields has been removed (since all fields are qos-related anyway).
Switch port QoS settings. Assigns a QoS profile to ingress packets on the given port. The assigned profile can be changed via match rules if the port is considered trusted.
By default, ports are untrusted and receive the default QoS profile (Best-Effort, PCP=0, DSCP=0), where priority fields are cleared from the egress packets.
Property | Description |
---|---|
map (name; Default: default) | Allows user-defined QoS priority-to-profile mapping in the case of a trusted port or host (see /in/eth/sw/qos/map ). |
profile (name; Default: default) | The name of the QoS profile to assign to the ingress packets by default (see /in/eth/sw/qos/profile ). |
trust-l2 (ignore | trust | keep; Default: ignore) | Whenever to trust the Layer 2 headers of the incoming packets (802.1p PCP field):
|
trust-l3 (ignore | trust | keep; Default: ignore) | Whenever to trust the Layer 3 headers of the incoming packets (IP DSCP field):
|
tx-manager (name; Default: default) | The name of the Transmission Manager that is responsible for enqueuing and transmitting packets from the given port (see |
L3 trust mode has higher precedence than L2 unless trust-l3=ignore or the packet does not have an IP header.
Forwarded/routed packets obtain priority field values (PCP, DSCP) from the selected QoS profile, overwriting the original values unless the respective trust mode is set to keep.
Commands.
Command | Description |
---|---|
Print the above properties in a human-friendly format. | |
print stats | Print port statistics: total and per-queue transmitted/dropped packets/bytes. |
reset-counters | Reset all counters in port statistics to zero. |
print usage | Print queue usage/resources. |
Port Stats
[admin@Mikrotik] /interface/ethernet/switch/qos/port> print stats where name=ether2 name: ether2 tx-packet: 2 887 tx-byte: 3 938 897 drop-packet: 1 799 drop-byte: 2 526 144 tx-queue0-packet: 50 tx-queue1-packet: 1 871 tx-queue3-packet: 774 tx-queue5-packet: 192 tx-queue0-byte: 3 924 tx-queue1-byte: 2 468 585 tx-queue3-byte: 1 174 932 tx-queue5-byte: 291 456 drop-queue1-packet: 1 799 drop-queue1-byte: 2 526 144
Property | Description |
---|---|
name | Port name. |
tx-packet | The total number of packets transmitted via this port. |
tx-byte | The total number of bytes transmitted via this port. |
drop-packet | The total number of packets should have been transmitted via this port but were dropped due to a lack of resources (e.g., queue buffers) or QoS Enforcement. |
drop-byte | The total number of bytes should have been transmitted via this port but were dropped. |
tx-queue0-packet .. tx-queue7-packet | The number of packets transmitted via this port from the respective queue. |
tx-queue0-byte .. tx-queue7-byte | The number of bytes transmitted via this port from the respective queue. |
drop-queue0-packet .. drop-queue7-packet | The number of packets dropped from the respective queue (or not enqueued at all due to lack of resources). |
drop-queue0-byte .. drop-queue7-byte | The number of bytes dropped from the respective queue. |
Port Resources/Usage
Due to hardware limitations, some switch chip models may break traffic flow while accessing QoS port usage
data. Use port usage
for diagnostics/troubleshooting only. For monitoring, use QoS monitor
or Port stats
instead.
[admin@crs326] /interface/ethernet/switch/qos/port> print usage where name=ether2 name: ether2 packet-cap: 136 packet-use: 5 byte-cap: 35 840 byte-use: 9 472 queue0-packet-cap: 130 queue0-packet-use: 1 queue1-packet-cap: 5 queue1-packet-use: 4 queue3-packet-cap: 65 queue3-packet-use: 2 queue0-byte-cap: 24 576 queue0-byte-use: 256 queue1-byte-cap: 7 680 queue1-byte-use: 6 144 queue3-byte-cap: 14 080 queue3-byte-use: 3 072
Property | Description |
---|---|
name | Port name. |
packet-cap | Port's packet capacity. The maximum number of packets that can be enqueued for transmission via the port. |
packet-use 1 | Port's packet usage. The number of packets that are currently enqueued in all port's queues. |
byte-cap | Port's byte capacity (buffer size). The maximum number of bytes that can be enqueued for transmission via the port. |
byte-use 1 | Port's byte usage. The size of hardware buffers (in bytes) that are currently allocated for packets the enqueued packets. Since the buffers are allocated by blocks (usually - 256B each), the allocated buffer size can be bigger than the actual payload. |
queue0-packet-cap .. queue7-packet-cap 2 | Queue capacity (in packets). The maximum number of packets that can be enqueued in the respective queues. |
queue0-packet-use .. queue7-packet-use 2 | Queue packet usage. The number of enqueued packets in the respective queues. |
queue0-byte-cap .. queue7-byte-cap 2 | Queue buffer capacity (in bytes). The maximum number of bytes that can be enqueued in the respective queues. Only the queues in use are printed. |
queue0-byte-use .. queue7-byte-use 2 | Queue buffer usage (in bytes). The size of hardware buffers (in bytes) that are currently allocated for packets in the respective queues. |
queue0-byte-max .. queue7-byte-max 2 | Maximum queue buffer fill level (in bytes). Available only on devices that provide the queue statistics service. Use the reset-counters command to reset values. |
1 Port's packet/byte usage can exceed the capacity if Shared Buffers are enabled.
2 Only the queues in use are printed.
QoS Menu
Sub-menu:
/interface/ethernet/switch/qos
Almost the entire QoS HW configuration is located under /in/eth/sw/qos
. Such an approach allows storing all QoS-related configuration items in one place, easy monitoring and exporting (/in/eth/sw/qos/export
).
QoS entries have two major flags:
- H - Hardware-offloaded.
- I - Inactive.
QoS Settings
Sub-menu:
/interface/ethernet/switch/qos/settings
Property | Description |
---|---|
multicast-buffers-percentage (integer: 1..90; Default: 10) | Maximum amount of packet buffers for multicast/broadcast traffic (% of the total buffer memory). |
shared-buffers-percentage (integer: 0..90; Default: 40) | Maximum amount of packet buffers that are shared between ports (% of the total buffer memory). Setting it to 0 disables buffer sharing. The remaining buffer memory is split between the ports. |
shared-buffers-color (all | green-only | yellow-and-green; Default: all) | Restricts shared buffer usage for specific traffic colors only. |
shared-pool0-percentage .. shared-pool7-percentage (integer: 0..100; Default: auto) | If the device supports multiple shared buffer pools, these settings allows adjusting the size of each pool (% of the shared buffer memory, where 100% means all shared buffers allocated by the shared-buffers-percentage setting). For example, if shared-buffers-percentage=40 and shared-pool0-percentage=50, the shared pool #0 (the first one) receives 20% of the total buffer memory (50% of 40% or "0.5 * 0.4 = 0.2"). Auto mode tries to equally allocate available resources between pools that uses auto setting, and provides at least a minimum of 10% of the total shared buffer size if the sum of other manually configured pools are exceeded. The default setting (auto). |
treat-yellow-as (green | red; Default: red) | For devices that support only two-color traffic marking (red/green). This setting allows using the same QoS profiles for the devices with two- and three-color traffic marking. |
wred-queue-margin (low | medium | high; Default: medium) | A relative amount of packets below a queue cap ("queueX-packet-cap " or "queueX-byte-cap ") to start a random tail drop. This margin is applied only to queues with enabled Weighed Random Early Detection (wred=yes) that do NOT use shared buffers (use-shared-buffers=no). The higher the queue buffer fill level, the higher the packet drop chance. The low margin means the random tail drop starts later; the high - sooner. |
wred-shared-queue-margin (low | medium | high; Default: medium) | Simillar to wred-queue-margin but appies to queues that use shared buffers (use-shared-buffers=yes). Also affects ECN marking. |
QoS Monitor
Command:
/interface/ethernet/switch/qos/monitor
[admin@CCR2216] /interface/ethernet/switch/qos monitor total-packet-cap: 32768 total-packet-use: 0 total-buffer-cap: 32768 total-buffer-use: 0 multicast-packet-cap: 3276 multicast-packet-use: 0 multicast-buffer-cap: 3276 multicast-buffer-use: 0 shared-pool0-packet-cap: 13107 shared-pool0-packet-use: 0 shared-pool0-buffer-cap: 13107 shared-pool0-buffer-use: 0
Monitors hardware QoS resources.
Property | Description |
---|---|
total-packet-cap (integer) | Total packet capacity. The maximum number of hardware packet descriptors that the device can store is all queues. |
total-packet-use (integer) | Total packet usage. The current number of packet descriptors residing in the hardware memory. |
total-buffer-cap (integer) | Total tx memory capacity (in tx queue buffers). Buffer size depends on the switch chip model (usually - 256 bytes). |
total-buffer-use (integer) | Total tx memory usage. The current number of buffers occupied by the packets in all tx queues. |
multicast-packet-cap (integer) | Multicast packet capacity. The maximum number of hardware packet descriptors that can be used by multicast/broadcast traffic. Depends on the multicast-buffers-percentage setting. |
multicast-packet-use (integer) | Multicast packet usage. The hardware makes a copy of the packet descriptor for each multicast destination. |
shared-packet-cap (integer) | Shared packet capacity. The maximum number of hardware packet descriptors that can be shared between ports and tx queues. Depends on the shared-buffers-percentage setting. |
shared-packet-use (integer) | Shared packet usage. The current number of shared packet descriptors used by all tx queues. |
shared-buffer-cap (integer) | Shared tx memory capacity (in tx queue buffers). Depends on the shared-buffers-percentage setting. |
shared-buffer-use (integer) | Shared tx memory usage. The current number of shared buffers occupied by the packets in all tx queues. |
shared-pool0-packet-cap .. shared-pool7-packet-cap (integer) | Shared packet capacity of the each shared pool. Only the shared pools in use are displayed. These fields are omitted if the device does not support multiple shared pools. |
shared-pool0-packet-use .. shared-pool7-packet-use (integer) | Per-pool shared packet usage. Only the shared pools in use are displayed. These fields are omitted if the device does not support multiple shared pools. |
QoS Profile
Sub-menu:
/interface/ethernet/switch/qos/profile
QoS profiles determine priority field values (PCP, DSCP) for the forwarded/routed packets. Congestion avoidance/resolution is based on QoS profiles. Each packet gets a QoS profile assigned based on the ingress switch port QoS settings (see /in/eth/sw/port
).
Property | Description |
---|---|
color (green | yellow | red; Default: green) | Trafic color for color-aware drop precedence management. Leave the default value (green) for color-blind drop precedence management. |
dscp (integer: 0..63; Default: 0) | IPv4/IPv6 DSCP field value for the egress packets assigned to the QoS profile. |
name (string; Default: ) | The user-defined name of the QoS profile. |
pcp (integer: 0..7; Default: 0) | VLAN priority value (IEEE 802.1q PCP - Priority Code Point). Used only if the egress packets assigned to the QoS profile are VLAN-tagged (have the 802.1q header). The value can be further altered via the QoS Egress Map. |
traffic-class (integer: 0..7; Default: 0) | The traffic class determines the packet priority and the egress queue (see tx-manager). The queue number is usually the same as the traffic class (packets with tc0 go into queue0, tc1 - queue1, ... tc7 - queue7). Unlike pcp, where 0 means the default priority but 1 - the lowest one (and further customizable), traffic classes are strictly ordered. TC0 always selects the lowest priority, etc. |
QoS Mapping
Sub-menu:
/interface/ethernet/switch/qos/map
Priority-to-profile mapping table(-s) for trusted packets. All switch chips have one built-in map - default. In addition, some models allow the user to define custom mapping tables and assign different maps to various switch ports via the qos-map property:
- devices based on Marvell Prestera 98DX224S, 98DX226S, or 98DX3236 switch chip models support only one map - default.
- devices based on Marvell Prestera 98DX8xxx, 98DX4xxx switch chips, or 98DX325x model devices support up to 12 maps (the default + 11 user-defined).
Property | Description |
---|---|
name (string; Default: ) | The user-defined name of the mapping table. |
VLAN Map
Sub-menu:
/interface/ethernet/switch/qos/map/vlan
Matches VLAN priorities (802.1p PCP/DEI fields) to QoS profiles. By default, all values are matched to the default QoS profile.
Property | Description |
---|---|
dei-only (yes | no; Default: no) | Map only packets with DEI (formerly CFI) bit set in the VLAN header. |
map (name; Default: default) | The name of the mapping table. |
profile (name; Default: ) | The name of the QoS profile to assign to the matched packets. |
pcp (integer: 0..7; Default: 0) | Minimum VLAN priority (PCP) value for the lookup. |
DSCP Map
Sub-menu:
/interface/ethernet/switch/qos/map/ip
Matches DSCP values to QoS profiles.
Property | Description |
---|---|
dscp (integer: 0..63; Default: 0) | Minimum DSCP value for the lookup. |
map (name; Default: default) | The name of the mapping table. If not set, the standard (built-in) mapping table gets altered. |
profile (name; Default: ) | The name of the QoS profile to assign to the matched packets. |
Transmission Manager
Sub-menu:
/interface/ethernet/switch/qos/tx-manager
Transmission (Tx) Manager controls packet enqueuing for transmission and packet tx order. Different switch ports can be assigned to different Tx managers. The maximum number of hardware Tx managers depends on the switch chip model (usually - 8).
Property | Description |
---|---|
ecn (yes | no; Default: no) | Enables/disables ECN marking of the transmitted packets. |
name (string; Default: ) | The use-defined name of the Tx Manager |
Transmission Queue Scheduler
Sub-menu:
/interface/ethernet/switch/qos/tx-manager/queue
Each port has eight Tx queues. The assigned Tx Manager controls packet enqueuing and schedules transmission orders. Each queue can have either strict priority (where packets with the highest traffic class are always transmitted first) or grouped together for a weighted round-robin tx schedule.
Creating a Tx Manager automatically creates all eight respective queue schedulers.
Changing any properties of Tx manager or queues completely halts traffic enqueueing and transmission during the offload process. Temporary packet loss is expected while the device is forwarding traffic.
Property | Description |
---|---|
tx-manager (name; read-only) | The linked Tx Manager |
traffic-class (integer: 0..7; read-only) | The traffic class (tc0..tc7) and the respective port queue (queue0..queue7) that the scheduler controls. |
schedule (strict-priority | high-priority-group | low-priority-group ) |
|
weight (integer: 0..255; Default: 1) | The weight value for the traffic class if it is a member of a schedule group. The field is not used in the case of strict priority schedule. |
queue-buffers (integer; Default: auto) | Per-queue buffer pool. The maximum number of packets that can be assigned to the queue (per each assigned port). |
use-shared-buffers (yes | no) | Allow the queue to use the shared buffer pool when queue-buffers are full. If the queue is full and the shared buffers are disabled, the packet gets dropped. If the shared buffers are enabled, the queue may use up to shared-packet-cap or shared-poolX-packet-cap (see QoS Settings for details) packets from the shared pool. |
shared-pool-index (integer; Default: 0) | The shared pool index for the queue to use. Relevant only if use-shared-buffers=yes and the device supports multiple shared pools. |
wred (yes | no; Default: no) | Enables/disables Weighted Random Early Detection for the given queue. |