Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: formatting

...

QoS works by identifying the type of traffic flowing through the switch and assigning it a priority level based on its requirements. The switch can then use this information to alter packet headers and prioritize the flow of traffic, ensuring that higher-priority traffic is given preferential treatment over lower-priority traffic.

The current implementation is for QoS Phase 1 - QoS Marking (introduced in RouterOS v7.10).

Planned QoS implementation phases:

  1. QoS Marking. QoS profile matching by ingress packet headers, then egress header alternation according to the assigned QoS profiles (introduced in RouterOS v7.10).
  2. QoS Enforcement. Avoid or resolve congestion based on the assigned QoS profiles and traffic shaping (introduced in RouterOS v7.13 for 98DX224S, 98DX226S, and 98DX3236 switch chips, and extended to all capable switch chips starting from RouterOS v7.15).
  3. QoS Policy. Assign QoS profiles via ACL rules (introduced in RouterOS v7.15).
  4. Extra QoS Features: WRED (Weighted Random Early Detection), ECN notification and processing (introduced in RouterOS v7.15 to capable switch chips).
  5. Traffic shaping.

QoS Terminology

These terms will be used throughout the article.

  1. .

QoS Terminology

These terms will be used throughout the article.

  • QoS - Quality of Service.
  • ACL -
  • QoS - Quality of Service.
  • ACL - Access Control List, a set of switch rules used to filter network traffic based on specified criteria.AQM - Active Queue Management.
  • DSCP - Differentiated Services Code Point, a 6-bit field in the IP header used to prioritize network traffic.ECN - Explicit Congestion Notification.
  • PCP - Priority Code Point, a 3-bit field in the VLAN header used to prioritize traffic within a VLAN.
  • WRED - Weighted Random Early Detection.
  • /in/eth/sw/ a shortcut for /interface/ethernet/switch/. The shortcut works in CLI, too.

...

Basic Configuration Example

In this example, we define just one QoS level - VoIP (IP Telephony) on top of the standard "Best Effort" class. Let's imagine that we have a CRS326-24G-2S+ device where:

  • all ports are bridged and using vlan-filtering;
  • sfp-sfpplus1 is a VLAN trunk connected to another switch;
  • ether1-ether9 are dedicated ports for IP phones;
  • ether10-ether24 are standard ports for host connection;

First, we need to define QoS profiles. Defined dscp and pcp values will be used in forwarded packets on egress:

Code Block
languageros
/interface ethernet switch qos profile
add dscp=46 name=voip pcp=5

Port-based QoS profile assignment on dedicated ports for IP phones applies to ingress traffic. Other Ethernet ports will use the default qos-profile (where dscp=0 and pcp=0):

Code Block
languageros
/interface ethernet switch port
set ether1 qos-profile=voip
set ether2 qos-profile=voip
set ether3 qos-profile=voip
set ether4 qos-profile=voip
set ether5 qos-profile=voip
set ether6 qos-profile=voip
set ether7 qos-profile=voip
set ether8 qos-profile=voip
set ether9 qos-profile=voip

The trunk port receives both types of QoS traffic. We need to create VLAN priority mapping with the QoS profile and enable qos-trust-l2 to differentiate them:

Code Block
languageros
/interface ethernet switch qos map vlan
add pcp=5 qos-profile=voip

/interface ethernet switch port
set sfp-sfpplus1 qos-trust-l2=trust

Finally, enable QoS hardware offloading for the above settings to start working:

Code Block
languageros
/interface ethernet switch
set switch1 qos-hw-offloading=yes

It is possible to verify the port QoS profile and Layer2, and Layer3 trust settings with print qos command:

Code Block
languageros
[admin@MikroTik] /interface/ethernet/switch/port print qos 
Columns: NAME, SWITCH, QOS-PROFILE, QOS-MAP, QOS-TRUST-L2, QOS-TRUST-L3
 # NAME          SWITCH   QOS-PROFILE  QOS-MAP  QOS-TRUST-L2  QOS-TRUST-L3
 0 ether1        switch1  voip         default  ignore        ignore      
 1 ether2        switch1  voip         default  ignore        ignore      
 2 ether3        switch1  voip         default  ignore        ignore      
 3 ether4        switch1  voip         default  ignore        ignore      
 4 ether5        switch1  voip         default  ignore        ignore      
 5 ether6        switch1  voip         default  ignore        ignore      
 6 ether7        switch1  voip         default  ignore        ignore      
 7 ether8        switch1  voip         default  ignore        ignore      
 8 ether9        switch1  voip         default  ignore        ignore      
 9 ether10       switch1  default      default  ignore        ignore      
10 ether11       switch1  default      default  ignore        ignore      
11 ether12       switch1  default      default  ignore        ignore      
12 ether13       switch1  default      default  ignore        ignore      
13 ether14       switch1  default      default  ignore    

1 Due to hardware limitations, some switch chip models may break traffic flow while accessing QoS port/queue usage data.

2 The device gathers max queue fill statistics instead of displaying the current usage values. Use the reset-counters command to reset those stats.

Applications and Usage Examples

Basic Configuration Example

In this example, we define just one QoS level - VoIP (IP Telephony) on top of the standard "Best Effort" class. Let's imagine that we have a CRS326-24G-2S+ device where:

  • all ports are bridged and using vlan-filtering;
  • sfp-sfpplus1 is a VLAN trunk connected to another switch;
  • ether1-ether9 are dedicated ports for IP phones;
  • ether10-ether24 are standard ports for host connection;

First, we need to define QoS profiles. Defined dscp and pcp values will be used in forwarded packets on egress:

Code Block
languageros
/interface ethernet switch qos profile
add dscp=46 name=voip pcp=5 traffic-class=5

Port-based QoS profile assignment on dedicated ports for IP phones applies to ingress traffic. Other Ethernet ports will use the default qos-profile (where dscp=0 and pcp=0):

Code Block
languageros
/interface ethernet switch qos port
set ether1 profile=voip
set ether2 profile=voip
set ether3 profile=voip
set ether4 profile=voip
set ether5 profile=voip
set ether6 profile=voip
set ether7 profile=voip
set ether8 profile=voip
set ether9 profile=voip
Info

Starting from RouterOS v7.13, QoS port settings moved from /interface/ethernet/switch/port to /interface/ethernet/switch/qos/port. The "qos-" prefix from the respective fields has been removed (since all fields are qos-related anyway).

The trunk port receives both types of QoS traffic. We need to create VLAN priority mapping with the QoS profile and enable qos-trust-l2 to differentiate them:

Code Block
languageros
/interface ethernet switch qos map vlan
add pcp=5 profile=voip

/interface ethernet switch port
set sfp-sfpplus1 trust-l2=trust

Finally, enable QoS hardware offloading for the above settings to start working:

Code Block
languageros
/interface ethernet switch
set switch1 qos-hw-offloading=yes

It is possible to verify the port QoS settings with print command:

Code Block
languageros
[admin@MikroTik] /interface/ethernet/switch/qos/port print
Columns: NAME, SWITCH, PROFILE, MAP, TRUST-L2, TRUST-L3
 # NAME          SWITCH   PROFILE  MAP      TRUST-L2  TRUST-L3  TX-MANAGER
 0 ether1        switch1  voip     default  ignore    ignore    default
 1 ether2        switch1  voip     default  ignore    ignore    default
 2 ether3        switch1  voip     default  ignore    ignore    default
 3 ether4
14 ether15       switch1  default voip     default  ignore        ignore    default
 4 ether5
15 ether16       switch1  default voip     default  ignore    ignore    default
ignore 5 ether6    
16 ether17   switch1  voip  switch1   default  ignore    ignore    default
 6 ether7        switch1  voip ignore      default  ignore    ignore    default
17 7 ether8 ether18       switch1  voip     default  ignore    ignore    default
 8 ether9        switch1  voip ignore      default  ignore    ignore    default
 9 ether10       switch1  default  default  ignore    ignore    default
10 ether11       switch1  default  default  ignore    ignore    default
11 ether12       switch1  default  default  ignore    ignore    default
12 ether13       switch1  default  default  ignore    ignore    default
13 ether14       switch1  default  default  ignore    ignore    default
14 ether15       switch1  default  default  ignore    ignore    default
15 ether16       switch1  default  default  ignore    ignore    default
16 ether17       switch1  default  default  ignore    ignore    default
17 ether18       switch1  default  default  ignore    ignore    default
18 ether19       switch1  default  default  ignore    ignore    default
19 ether20       switch1  default  default  ignore    ignore    default
20 ether21       switch1  default  default  ignore    ignore    default
21 ether22       switch1  default  default  ignore    ignore    default
22 ether23       switch1  default  default  ignore    ignore    default
23 ether24       switch1  default  default  ignore    ignore    default
24 sfp-sfpplus1  switch1  default  default  trust     ignore    default
25 sfp-sfpplus2  switch1  default  default  ignore    ignore    default
26 switch1-cpu   switch1   

Now incoming packets on ports ether1-ether9 are marked with a Priority Code Point (PCP) value of 5 and a Differentiated Services Code Point (DSCP) value of 46, and incoming packets on ports ether10-ether24 are marked with PCP and DSCP values of 0. When packets are incoming to sfp-sfpplus1 port, any packets with a PCP value of 5 or higher will retain their PCP value of 5 and DSCP value of 46, while all other packets will be marked with PCP and DSCP values of 0.

Dante

Starting from RouterOS v7.15, all MikroTik QoS-Capable devices comply with Dante

Dante hardware use the following DSCP / Diffserv priority values for traffic prioritization.

Dante PriorityUsageDSCP LabelDSCP Value
HighTime critical PTP eventsCS756
MediumAudio, PTPEF46
Low(reserved)CS18
NoneOther trafficBE0

The example assumes that the switch is using its default configuration, which includes a default "bridge" interface and all Ethernet interfaces added as bridge ports, and any of these interfaces could be used for Dante.

First, create QoS Profiles to match Dante traffic classes, there is already a pre-existing "default" profile that corresponds to Dante's None priority.

Code Block
languageros
/interface/ethernet/switch/qos/profile
add name=dante-ptp dscp=56 pcp=7 traffic-class=7
add name=dante-audio dscp=46 pcp=5 traffic-class=5
add name=dante-low dscp=8 pcp=1 traffic-class=0

Then, create a QoS mapping to match QoS profiles based on DSCP values. 

Code Block
languageros
/interface/ethernet/switch/qos/map/ip
add dscp=56 profile=dante-ptp
add dscp=46 profile=dante-audio
add dscp=8 profile=dante-low

Configure hardware queues to enforce QoS on Dante traffic.

Code Block
languageros
/interface/ethernet/switch/qos/tx-manager/queue
set [find where traffic-class>=2] schedule=strict-priority
set [find where traffic-class<2] schedule=low-priority-group weight=1

Dante's High and Medium priority traffic is scheduled in strict order. The devices transmits time critical PTP packets until queue7 gets empty, then proceeds with audio (queue5). Low and other traffic gets transmitted only when PTP and audio queues are empty. Since Dante does not define priority order between Low and Other traffic (usually, CS1 has lower priority than Best Effort), and the Low traffic class is reserved for future use anyway, we treat both traffic types equally by putting both into the same group with the same weight. Feel free to change the CS1/BE traffic scheduling according to the requirements if some Dante hardware in your network use the low-priority traffic class.

The next step is enable trust mode for incomming Layer3 packets (IP DSCP field):

Code Block
languageros
/interface/ethernet/switch/qos/port
set [find] trust-l3=keep

Finally, enable QoS hardware offloading for the above settings to start working:

Code Block
languageros
/interface ethernet switch
set switch1 qos-hw-offloading=yes

When using Dante in multicast mode, it is beneficial to enable IGMP snooping on the switch. This feature directs traffic only to ports with subscribed devices, preventing unnecessary flooding. Additionally, enabling an IGMP querier (if not already enabled on another device in the same LAN), adjusting query intervals, and activating fast-leave can further optimize multicast performance.

Code Block
languageros
/interface/bridge
set [find name=bridge] igmp-snooping=yes multicast-querier=yes query-interval=60s

/interface/bridge/port
set [find] fast-leave=yes

QoS Marking

Understanding Map ranges

In order to avoid defining all possible PCP and DSCP mappings, RouterOS allows setting the minimal PCP and DSCP values for QoS Profile mapping.

In the following example, PCP values 0-2 use the default QoS profile, 3-4 - streaming, 5 - voip, and 6-7 - control.

Code Block
languageros
/interface ethernet switch qos map vlan
add pcp=3 profile=streaming
add pcp=5 profile=voip
add pcp=6 profile=control

Since the pcp parameter identifies the minimum value, all packets with a higher PCP value match too. If such behavior is undesired, add mapping for higher values. The next example sets voip profile for pcp=5 only. Packets with PCP values 6 or 7 are reset back to the default profile.

Code Block
languageros
/interface ethernet switch qos map vlan
add pcp=5 profile=voip
add pcp=6 profile=default

Understanding Port, Profile, and Map relation

Each switch port has Layer2 and Layer3 trust settings that will change how ingress packets are classified into QoS profiles and what PCP and DSCP values will be used. Below are tables that describe all possible options:

qos-trust-l2qos-trust-l3Behavior
ignoreignoreThe port is considered untrusted. Both headers are ignored, and the port's profile is forced to all ingress packets. This is the default setting.
ignoretrustTrust the Layer 3 header. Use the DSCP field from the IP header of ingress packets for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the lookup fails (no QoS profiles are mapped to the given DSCP value), the default QoS profile is used (not the switch port's QoS profile). The switch port's profile field is used only for non-IP traffic.
ignorekeepTrust the Layer 3 header. Use the DSCP field from the IP header of ingress packets for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the lookup fails, the default QoS profile is used. The switch port's profile field is used only for non-IP traffic. If the forwarded/routed packet is VLAN-tagged, its PCP value is set from the selected QoS profile. However, the original DSCP value of the packet is kept intact.
trustignoreTrust the Layer 2 header, but ignore L3. If an ingress packet is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan). If the lookup fails (no QoS profiles are mapped to the given PCP value), the default QoS profile is used. The switch port's profile field is used only for untagged traffic.
trusttrustTrust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan).  If the VLAN-to-QoS lookup fails, use the default QoS profile. Non-IP untagged packets use the switch port's profile.
trustkeepThe same as trust+trust, but the original DSCP value is preserved in forwarded/routed packets.
keepignoreTrust the Layer 2 header but ignore L3. If an ingress packet is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan). If the lookup fails (no QoS profiles are mapped to the given PCP value), the default QoS profile is used. The switch port's profile field is used only for untagged traffic. If the packet is VLAN-tagged on both ingress and egress, the original PCP value is kept.
keeptrustTrust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan).  If the VLAN-to-QoS lookup fails, use the default QoS profile. Non-IP untagged packets use the switch port's profile. If the packet is VLAN-tagged on both ingress and egress, the original PCP value is kept. The DSCP value in forwarded/routed packets is set from the selected QoS profile.
keepkeepTrust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan).  If the VLAN-to-QoS lookup fails, use the default QoS profile. Non-IP untagged packets use the switch port's profile. Keep both the original PCP and/or DSCP values intact in cases of VLAN-tagged and/or IP packets, respectively.

...

1 applies only when ingress traffic is untagged, but the egress needs to be VLAN-tagged.

...

Starting from RouterOS v7.15, it is possible to assign QoS profiles via Switch Rules (ACL).

...

The following example assigns a QoS profile based on the source MAC address.

Code Block
languageros
/interface ethernet switch rule
add new-qos-profile=stream ports=ether1,ether2 src-mac-address=00:01:02:00:00:00/FF:FF:FF:00:00:00 switch=switch1
add new-qos-profile=voip ports=ether1,ether2 src-mac-address=04:05:06:00:00:00/FF:FF:FF:00:00:00 switch=switch1

QoS Enforcement

Hardware Queues

Each switch port has eight hardware transmission (tx) queues (queue0..queue7). Each queue corresponds to a traffic class (tc0..tc7) set by a QoS profile. Each ingress packet gets assigned to a QoS profile, which, in turn, determines the traffic class for tx queue selection on the egress port.

Hardware queues are of variable size - set by the Transmission Manager. Moreover, multiple ports and/or queues can share resources with each other (so-called Shared Buffers). For example, a device with 25 ports has memory (buffers) to queue 1200 packets in total. If we split the resources equally, each port gets 48 exclusive buffers with a maximum of 6 packets per queue (48/8) - which is usually insufficient to absorb even a short burst of traffic. However, choosing to share 50% of the buffers leaves each port with 24 exclusive buffers (3 per queue), but at the same time, a single queue can grow up to 603 buffers (3 exclusive + 600 shared).

RouterOS allows enabling/disabling the shared pool for each queue individually - for example, to prevent low-priority traffic from consuming the entire hardware memory. In addition, port buffer limits may prevent a single low-speed port from consuming the entire shared pool. See QoS Settings and  Transmission Manager for details.

Info

The default, best-effort (PCP=0, DSCP=0) traffic class is 1, while the lowest priority (PCP=1) has traffic class 0.

Hardware Resources

The hardware (switch chips) has limited resources (memory). There are two main hardware resources that are relevant to QoS:

  • Packet descriptors - contain packet control information (target port, header alternation, etc).
  • Data buffers - memory chunks containing the actual payload. Buffer size depends on the switch chip model. Usually - 256 bytes.

One packet descriptor may use multiple buffers (depending on the payload size); buffers may be shared by multiple descriptors - in cases of multicast/broadcast. If the hardware does not have enough free descriptors or buffers, the packet gets dropped (tail-drop).

Hardware resources can be limited per destination type (multicast/unicast), per port, and per each tx queue. If any limits are reached, no more packets can be enqueued for transmission, and further packets get dropped.

RouterOS obscures low-level hardware information, allowing to set resource limits either in terms of packets or a percentage of the total amount. RouterOS automatically calculates the required hardware descriptor and buffer count based on the user-specified packet limit and port's MTU. Moreover, RouterOS comes with preconfigured hardware resources, so there is no need to do a manual configuration in common QoS environments.

Warning

Changing any hardware resource allocation parameter in runtime results in a temporary device halt when no packets can be enqueued nor transmitted. Temporary packet loss is expected while the device is forwarding traffic.

Resource Saving

Since reallocating hardware resources in runtime is not an option, RouterOS cannot automatically free queue buffers reserved for inactive ports. Those buffers remain unused. However, if the user clearly knows that the specific ports will never be used (e.g., stay physically disconnected), the respective queue resources can be manually freed by introducing the "offline" tx-manager with minimum resources:

Code Block
languageros
/interface/ethernet/switch/qos/tx-manager/
add name=offline comment="use this for always disconnected ports"

/interface/ethernet/switch/qos/tx-manager/queue/
set [find where tx-manager=offline] queue-buffers=1 use-shared-buffers=no

/interface/ethernet/switch/qos/port
set [find where !running] tx-manager=offline

Traffic Prioritization

The hardware provides two types of traffic transmission prioritization:

  • Strict Priority - traffic from higher queues is always transmitted first;
  • Weighted Priority Groups - multiple queues participate in packet transmission scheduling at the same time.

Strict priority queues are straightforward. If the highest priority queue (Q7) has packets, those are transmitted first. When Q7 is empty, packets from Q6 get transmitted, and so on. The packets from the lowest priority queue (Q0) are transmitted only if all other queues are empty.

...

Weighted Priority Groups schedule traffic for transmission from multiple queues (group members) in a weighted round-robin manner. A queue's weight sets the number of packets transmitted from the queue in each round. For example, if Q2, Q1, and Q0 are the group members, and their weights are 3, 2, and 1, respectively, the scheduler transmits 3 packets from Q2, 2 - from Q1, and 1 - from Q0. The actual Tx order is "Q2, Q1, Q0, Q2, Q1, Q2" - for even fairer scheduling.

There are two hardware groups: low-priority-group and high-priority-group. There is a strict priority ordering between the two groups: the low-priority-group is transmitting only when all queues in the high-priority-group are empty. However, it is possible to use only one group for all queues.

The default (built-in) RouterOS queue setup is listed below. Q3-Q5 share the bandwidth within the high-priority group, where packets are transmitted while Q6 and Q7 are empty. Q0-Q2 are the members of the low-priority-group, where packets are transmitted while Q3-Q7 are empty.

Code Block
languageros
[admin@MikroTik] /interface/ethernet/switch/qos/tx-manager/queue> print 
Columns: TX-MANAGER, TRAFFIC-CLASS, SCHEDULE, WEIGHT, QUEUE-BUFFERS, USE-SHARED-BUFFERS
#  TX-MANAGER  TRAFFIC-CLASS  SCHEDULE             WEIGHT  QUEUE-BUFFERS  USE-SHARED-BUFFERS
0  default     0              low-priority-group   1       auto           no                
1  default     1              low-priority-group   2       auto           yes                
2  default     2              low-priority-group   3       auto           yes                
3  default     3              high-priority-group  3       auto           yes               
4  default     4              high-priority-group  4       auto           yes               
5  default     5              high-priority-group  5       auto           yes               
6  default     6              strict-priority              auto           yes               
7  default     7              strict-priority              auto           yes 
Info

It is recommended that all group members are adjacent to each other.

Active Queue Management (AQM)

...

WRED is a per-queue congestion control mechanism that signals congestion events to the end-points by dropping packets. WRED relies on the existence of rate throttling mechanisms in the end-points that react to packet loss, such as TCP/IP. WRED uses a randomized packet drop algorithm in an attempt to anticipate congestion events and respond to them by throttling traffic rates before the congestion actually happens. The randomness property of WRED prevents throughput collapse related to the global synchronization of TCP flows.

WRED can be enabled/disabled per each queue in each Tx Manager. Disable WRED for lossless traffic! Also, there is no reason to enable WRED on high-speed ports where congestion should not happen in the first place.

The behavior is controlled via WRED margins. WRED margin is the distance to the queue/pool buffer limit (cap) - where a random packet drop begins. A different margin can be applied to queues that use or don't use shared buffers. A queue that uses a shared pool may set a bigger WRED margin due to a higher overall cap (queue buffers + shared pool). RouterOS automatically chooses the actual WRED margin values according to queue or shared pool capacities. The user may shift the margins in one way or another via QoS Settings.

For example, if queue1-packet-cap=96, and WRED margin is 32 (assuming use-shared-buffers=no), then:

  • first 64 packets are always enqueued (96 - 32 = 64).
  • WRED kicks in starting from 65th packet; the chance of 65th packet to be dropped is 1/32 or roughly 3%; the formula: (65 - 64) / 32;
  • the drop chance of 72th packet is 25%: (72 - 64) / 32 = 8/32 = 1/4;
  • half of the newly enquining packets are dropped when the queue fill level reaches 80 packets: (80 - 64) / 32 = 16/32 = 1/2;
  • 75% of the packets are dropped at the fill level 88: (88 - 64) / 32 = 24/32 = 3/4.

When WRED is enabled (wred=yes), the queue cap is rounded to the applied WRED margin. In the above example, setting wred-queue-margin=64 raises queue1-packet-cap to 128. That, in turn, may lower the resources available to other queues, such as shared buffers. A much safer option is to raise wred-shared-queue-margin which may reduce the shared buffers available for the affected queue but not the shared pool itself. For example, if: "wred-shared-queue-margin=256, use-shared-buffers=yes, wred=yes, shared-pool-index=0, queue2-packet-cap=30, and shared-pool0-packet-cap=900", queue2 can use up to 768 buffers (30+900=930, rounded down by the scale of 256), and WRED starting at 512, while the other ports/queues still may use the remaining 162 buffers (30+900-768) of the shared pool.

Choosing a WRED margin value is a tradeoff between congestion anticipation and burst absorption. Setting a higher WRED margin may lead to earlier traffic rate throttling and, therefore, resolve congestion. On the other hand, a high margin leads to packet drops in limited traffic bursts that could be absorbed by the queue buffers and transformed losslessly if WRED didn't kick in. For instance, initiating a remote database connection usually starts with heavier traffic ("packet burst") at the initialization phase; then, the traffic rate drops down to a "reasonable" level. Any packet drop during the initialization phase leads to nothing but a slower database connection due to the need for retransmission. Hence, lowering the WRED margin or entirely disabling WRED on such traffic is advised. The opposite case is video streaming. Early congestion detection helps select a comfortable streaming rate without losing too much bandwidth on retransmission or/and "overshooting" by sacrificing the quality level by too much.

Info

Use Switch Rules (ACL) or other QoS Marking techniques to differentiate traffic and put packets into queues with desired WRED settings.

The following script only applies WRED to TCP/IP traffic by redirecting it to queue2. UDP and other packets are left in queue1 - since their end-points usually cannot respond to early drops. Queue1 and queue2 are scheduled equally - without prioritizing one queue over another.

Code Block
languageros
/interface/ethernet/switch/qos/profile
add name=tcp-wred traffic-class=2 pcp=0 dscp=0

# move TCP traffic to queue2
/interface/ethernet/switch/rule
add new-qos-profile=tcp-wred ports=ether1,ether2,ether3,ether4 protocol=tcp switch=switch1

# set the same scheduling priority (weight) between queue1 and queue2
# apply WRED only to queue2 - TCP traffic
/interface/ethernet/switch/qos/tx-manager/queue/
set [find where traffic-class=1] weight=2 schedule=low-priority-group use-shared-buffers=yes shared-pool-index=0 wred=no
set [find where traffic-class=2] weight=2 schedule=low-priority-group use-shared-buffers=yes shared-pool-index=0 wred=yes

...

Some switch chips can perform ECN marking of IP packets on the hardware level, according to RFC 3168. Hardware ECN marking is based on the WRED mechanism, but instead of dropping IP packets, they are marked with CE (Congestion Experienced, binary 11) in the ECN field (two least significant bits in IPv4/TOS or IPv6/TrafficClass octet). Only ECN-Capable IP packets may be marked - those with the ECN field value of ECT(1) or ECT(0)  (binary 01 or 10, respectively). Not ECN-Capable Transport packets (ECN=00) never get marked. If a packet already has the CE mark (ECN=11), it never gets cleared, even if the device does not experience congestion.

Set ecn=yes in Tx Manager to enable ECN marking. The per-queue ECN setting is unavailable due to hardware limitations. ECN and WRED share the same queue fill threshold: wred-shared-queue-margin (see  QoS Settings).

Warning

ECN marking mechanism requires the respective Tx queues to use shared buffers (use-shared-buffers=yes).

The packet receives the CE mark if all conditions below are met:

  1. The packet is either IPv4 or IPv6.
  2. The ECN field value in IP header is either ECT(1) or ECT(0).
  3. Egress port's Tx Manager has ecn=yes.
  4. The assigned Tx Queue uses shared buffers (use-shared-buffers=yes).
  5. The Tx Queue detects congestion via WRED margins.

Property Reference

Switch settings

...

Switch QoS settings (in addition to the existing ones).

...

Port settings

...

Info

Starting from RouterOS v7.13, QoS port settings moved from /interface/ethernet/switch/port to /interface/ethernet/switch/qos/port. The "qos-" prefix from the respective fields has been removed (since all fields are qos-related anyway).

Switch port QoS settings. Assigns a QoS profile to ingress packets on the given port. The assigned profile can be changed via match rules if the port is considered trusted.

By default, ports are untrusted and receive the default QoS profile (Best-Effort, PCP=0, DSCP=0), where priority fields are cleared from the egress packets.

...

Whenever to trust the Layer 2 headers of the incoming packets (802.1p PCP field):

  • ignore - ignore L2 header; use the port's profile value for all incoming packets;
  • trust - use PCP field of VLAN-tagged packets for QoS profile lookup in map. Untagged packets use the port's profile value. Forwarded VLAN or priority-tagged packets receive the PCP value from the selected QoS profile (overwriting the original value).
  • keep - trust but keep the original PCP value in forwarded packets. 

...

Whenever to trust the Layer 3 headers of the incoming packets (IP DSCP field):

  • ignore - ignore L3 header; use either L2 header or the port's profile (depends on trust-l2).
  • trust - use DSCP field of IP packets for QoS profile lookup in map. Forwarded/routed IP packets receive the DSCP value from the selected QoS profile (overwriting the original value).
  • keep - trust but keep the original DSCP value in forwarded/routed packets.

...

The name of the Transmission Manager that is responsible for enqueuing and transmitting packets from the given port (see /in/eth/sw/qos/tx-manager).

Info

L3 trust mode has higher precedence than L2 unless trust-l3=ignore or the packet does not have an IP header.

Info

Forwarded/routed packets obtain priority field values (PCP, DSCP) from the selected QoS profile, overwriting the original values unless the respective trust mode is set to keep.

Commands.

...

Port Stats

Code Block
languageros
titleExample
[admin@Mikrotik] /interface/ethernet/switch/qos/port> print stats where name=ether2
                  name:     ether2
             tx-packet:      2 887
               tx-byte:  3 938 897
           drop-packet:      1 799
             drop-byte:  2 526 144
      tx-queue0-packet:         50
      tx-queue1-packet:      1 871
      tx-queue3-packet:        774
      tx-queue5-packet:        192
        tx-queue0-byte:      3 924
        tx-queue1-byte:  2 468 585
        tx-queue3-byte:  1 174 932
        tx-queue5-byte:    291 456
    drop-queue1-packet:      1 799
      drop-queue1-byte:  2 526 144

...

tx-queue0-packet .. tx-queue7-packet

...

tx-queue0-byte .. tx-queue7-byte

...

drop-queue0-packet .. drop-queue7-packet

...

drop-queue0-byte .. drop-queue7-byte

...

Port Resources/Usage

Warning

Due to hardware limitations, some switch chip models may break traffic flow while accessing QoS port usage data. Use port usage for diagnostics/troubleshooting only. For monitoring, use QoS monitor or Port stats instead.

Code Block
languageros
titleExample
[admin@crs326] /interface/ethernet/switch/qos/port> print usage where name=ether2
     
18 ether19       switch1  default      default  ignore        ignore    name:  ether2
  
19 ether20       switch1  default      default  ignore        ignore packet-cap:     136
    
20 ether21       switch1  default      default  ignore       packet-use: ignore      5
21 ether22       switch1   default  byte-cap:  35 840
 default  ignore        ignore  byte-use:   9 472
22 ether23    queue0-packet-cap:   switch1  130
default    queue0-packet-use:  default  ignore   1
    queue1-packet-cap: ignore      5
23 ether24   queue1-packet-use:    switch1  default 4
    queue3-packet-cap: default  ignore   65
    queue3-packet-use: ignore      2
24 sfp-sfpplus1  switch1   queue0-byte-cap:  24 576
default      default  queue0-byte-use:trust     256
    ignore  queue1-byte-cap:   7 680

25 sfp-sfpplus2  switch1  default    queue1-byte-use:  default 6 144
ignore      queue3-byte-cap:  14ignore 080
     
26 queue3-byte-use:   3 072

...

switch1-cpu   switch1  

Now incoming packets on ports ether1-ether9 are marked with a Priority Code Point (PCP) value of 5 and a Differentiated Services Code Point (DSCP) value of 46, and incoming packets on ports ether10-ether24 are marked with PCP and DSCP values of 0. When packets are incoming to sfp-sfpplus1 port, any packets with a PCP value of 5 or higher will retain their PCP value of 5 and DSCP value of 46, while all other packets will be marked with PCP and DSCP values of 0.

Understanding Map ranges

To avoid the need to define all possible PCP and DSCP mappings, RouterOS allows setting the minimal PCP and DSCP values for QoS Profile mapping.

In the following example, PCP values 0-2 use the default QoS profile, 3-4 - streaming, 5 - voip, and 6-7 - control.

Code Block
languageros
/interface ethernet switch qos map vlan
add pcp=3 qos-profile=streaming
add pcp=5 qos-profile=voip
add pcp=6 qos-profile=control

Since the pcp parameter identifies the minimum value, all packets with a higher PCP value match too. If such behavior is undesired, add mapping for higher values. The next example sets voip profile for pcp=5 only. Packets with PCP values 6 or 7 are reset back to the default profile.

Code Block
languageros
/interface ethernet switch qos map vlan
add pcp=5 qos-profile=voip
add pcp=6 qos-profile=default

Understanding Port, Profile and Map relation

Each switch port has Layer2 and Layer3 trust settings, that will change how ingress packets are classified into QoS profiles and what PCP and DSCP values will be used. Below are tables that describe all possible options:

qos-trust-l2qos-trust-l3Behavior
ignoreignoreThe port is considered untrusted. Both headers are ignored, and the port's qos-profile is forced to all ingress packets. This is the default setting.
ignoretrustTrust the Layer 3 header. Use the DSCP field from the IP header of ingress packets for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the lookup fails (no QoS profiles are mapped to the given DSCP value), the default profile is used (not the switch port's qos-profile). The switch port's qos-profile field is used only for non-IP traffic.
ignorekeepTrust the Layer 3 header. Use the DSCP field from the IP header of ingress packets for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the lookup fails, the default profile is used. The switch port's qos-profile field is used only for non-IP traffic. If the forwarded/routed packet is VLAN-tagged, its PCP value is set from the selected QoS profile. However, the original DSCP value of the packet is kept intact.
trustignoreTrust the Layer 2 header but ignore L3. If an ingress packet is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan). If the lookup fails (no QoS profiles are mapped to the given PCP value), the default profile is used. The switch port's qos-profile field is used only for untagged traffic.
trusttrustTrust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan).  If the VLAN-to-QoS lookup fails, use the default profile. Non-IP untagged packets use the switch port's qos-profile.
trustkeepThe same as trust+trust, but the original DSCP value is preserved in forwarded/routed packets.
keepignoreTrust the Layer 2 header but ignore L3. If an ingress packet is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan). If the lookup fails (no QoS profiles are mapped to the given PCP value), the default profile is used. The switch port's qos-profile field is used only for untagged traffic. If the packet is VLAN-tagged on both ingress and egress, the original PCP value is kept.
keeptrustTrust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan).  If the VLAN-to-QoS lookup fails, use the default profile. Non-IP untagged packets use the switch port's qos-profile. If the packet is VLAN-tagged on both ingress and egress, the original PCP value is kept. The DSCP value in forwarded/routed packets is set from the selected QoS profile.
keepkeepTrust both headers, but Layer 3 has higher precedence. In the case of an IP packet, use the DSCP field for QoS profile lookup (see /in/eth/sw/qos/map/ip). If the DSCP-to-QoS lookup fails, use the default profile. If the packet is not an IP packet but is VLAN-tagged, use the PCP field from the VLAN header for QoS profile lookup (see /in/eth/sw/qos/map/vlan).  If the VLAN-to-QoS lookup fails, use the default profile. Non-IP untagged packets use the switch port's qos-profile. Keep both the original PCP and/or DSCP values intact in cases of VLAN-tagged and/or IP packets, respectively.
Port settings
The selected QoS profile and the source for PCP / DSCP field values in forwarded/routed packets
qos-trust-l2
 
qos-trust-l3
VLAN-Tagged IPUntagged IPVLAN-Tagged Non-IPUntagged Non-IP
QoS ProfilePCPDSCPQoS ProfilePCP 1DSCPQoS ProfilePCPDSCPQoS ProfilePCP 1DSCP
ignoreignoreqos-profileqos-profileqos-profileqos-profileqos-profileqos-profileqos-profileqos-profile-qos-profileqos-profile-
ignoretrustmap/ipmap/ipmap/ipmap/ipmap/ipmap/ipqos-profileqos-profile-qos-profileqos-profile-
ignorekeepmap/ipmap/iporiginalmap/ipmap/iporiginalqos-profileqos-profile-qos-profileqos-profile-
trustignoremap/vlanmap/vlanmap/vlanqos-profileqos-profileqos-profilemap/vlanmap/vlan-qos-profileqos-profile-
trusttrustmap/ipmap/ipmap/ipmap/ipmap/ipmap/ipmap/vlanmap/vlan-qos-profileqos-profile-
trustkeepmap/ipmap/iporiginalmap/ipmap/iporiginalmap/vlanmap/vlan-qos-profileqos-profile-
keepignoremap/vlanoriginalmap/vlanqos-profileqos-profileqos-profilemap/vlanoriginal-qos-profileqos-profile-
keeptrustmap/iporiginalmap/ipmap/ipqos-profilemap/ipmap/vlanoriginal-qos-profileqos-profile-
keepkeepmap/iporiginaloriginalmap/ipqos-profileoriginalmap/vlanoriginal-qos-profileqos-profile-

1 applies only when ingress traffic is untagged, but the egress needs to be VLAN-tagged.

Property Reference

Switch settings

Sub-menu:/interface/ethernet/switch

Switch QoS settings (in addition to the existing ones).

PropertyDescription
qos-hw-offloading (yes | no; Default: no)Allows enabling QoS for the given switch chip (if the latter supports QoS).

Port settings

Sub-menu:/interface/ethernet/switch/port

Switch port settings (in addition to the existing ones). Assigns a QoS profile to ingress packets on the given port. The assigned profile can be changed via match rules if the port is considered trusted.

By default, ports are untrusted and receive the default QoS profile (Best-Effort, PCP=0, DSCP=0), where priority fields are cleared from the egress packets.

PropertyDescription
qos-map (name; Default: default)Allows user-defined priority-to-profile mapping in the case of a trusted port or host (see /in/eth/sw/qos/map).
qos-profile (name; Default: default)The name of the QoS profile to assign to the ingress packets by default (see /in/eth/sw/qos/profile).
qos-trust-l2 (ignore | trust | keep; Default: ignore)

Whenever to trust the Layer 2 headers of the incoming packets (802.1p PCP field):

  • ignore - ignore L2 header; use the port's qos-profile value for all incoming packets;
  • trust - use PCP field of VLAN-tagged packets for QoS profile lookup in qos-map. Untagged packets use the port's qos-profile value. Forwarded VLAN or priority-tagged packets receive the PCP value from the selected QoS profile (overwriting the original value).
  • keep - trust but keep the original PCP value in forwarded packets. 
qos-trust-l3 (ignore | trust | keep; Default: ignore)

Whenever to trust the Layer 3 headers of the incoming packets (IP DSCP field):

  • ignore - ignore L3 header; use either L2 header or the port's qos-profile (depends on qos-trust-l2).
  • trust - use DSCP field of IP packets for QoS profile lookup in qos-map. Forwarded/routed IP packets receive the DSCP value from the selected QoS profile (overwriting the original value).
  • keep - trust but keep the original DSCP value in forwarded/routed packets.
Info

L3 trust mode has higher precedence than L2 unless qos-trust-l3=ignore or the packet does not have an IP header.

Info

Forwarded/routed packets obtain priority field values (PCP, DSCP) from the selected QoS profile, overwriting the original values, unless the respective trust mode is set to keep.

Commands (in addition to the existing ones).

CommandDescription
print qosPrints all QoS-related information in a human-friendly format.

QoS Settings

Sub-menu:/interface/ethernet/switch/qos

Almost the entire QoS HW configuration is located under /in/eth/sw/qos. Such an approach allows storing all QoS-related configuration items in one place, easy monitoring and exporting (/in/eth/sw/qos/export).

All QoS entries have two major flags:

  • H - Hardware-offloaded.
  • I - Inactive.

QoS Profile

Sub-menu:/interface/ethernet/switch/qos/profile

QoS profiles determine priority field values (PCP, DSCP) for the forwarded/routed packets. Congestion avoidance/resolution is based on QoS profiles. Each packet gets a QoS profile assigned based on the ingress switch port QoS settings (see /in/eth/sw/port).

PropertyDescription
dscp (integer: 0..63; Default: 0)IPv4/IPv6 DSCP field value for the egress packets assigned to the QoS profile.
name (string; Default: )The user-defined name of the QoS profile. 
pcp (integer: 0..7; Default: 0)VLAN priority value (IEEE 802.1q PCP - Priority Code Point). Used only if the egress packets assigned to the QoS profile are VLAN-tagged (have the 802.1q header). The value can be further altered via the QoS Egress Map.

QoS Mapping

Sub-menu:/interface/ethernet/switch/qos/map

Priority-to-profile mapping table(-s) for trusted packets. All switch chips have one built-in map - default. In addition, some models allow the user to define custom mapping tables and assign different maps to various switch ports via the qos-map property:

  • devices based on Marvell Prestera 98DX224S, 98DX226S, or 98DX3236 switch chip models support only one map - default.
  • devices based on Marvell Prestera 98DX8xxx98DX4xxx switch chips, or 98DX325x model devices support up to 12 maps (the default + 11 user-defined).
PropertyDescription
name (string; Default: )The user-defined name of the mapping table.

VLAN Map

Sub-menu:/interface/ethernet/switch/qos/map/vlan

Matches VLAN priorities (802.1p PCP field) to QoS profiles. By default, all values are matched to the default QoS profile.

PropertyDescription
qos-map (name; Default: default)The name of the mapping table.
qos-profile (name; Default: )The name of the QoS profile to assign to the matched packets.
pcp (integer: 0..7; Default: 0)MinimumVLAN priority (PCP) value for the lookup. 

DSCP Map

Sub-menu:/interface/ethernet/switch/qos/map/ip

Matches DSCP values to QoS profiles.

PropertyDescription
dscp (integer: 0..63; Default: 0)Minimum DSCP value for the lookup.
qos-map (name; Default: default)The name of the mapping table. If not set, the standard (built-in) mapping table gets altered.
qos-profile (name; Default: )The name of the QoS profile to assign to the matched packets

1 Port's packet/byte usage can exceed the capacity if Shared Buffers are enabled.

2 Only the queues in use are printed.

QoS Menu

...

Almost the entire QoS HW configuration is located under /in/eth/sw/qos. Such an approach allows storing all QoS-related configuration items in one place, easy monitoring and exporting (/in/eth/sw/qos/export).

QoS entries have two major flags:

  • H - Hardware-offloaded.
  • I - Inactive.

...

QoS Monitor

...

Code Block
languageros
titleExample
[admin@CCR2216] /interface/ethernet/switch/qos monitor 
         total-packet-cap: 32768
         total-packet-use: 0
         total-buffer-cap: 32768
         total-buffer-use: 0
     multicast-packet-cap: 3276
     multicast-packet-use: 0
     multicast-buffer-cap: 3276
     multicast-buffer-use: 0
  shared-pool0-packet-cap: 13107
  shared-pool0-packet-use: 0
  shared-pool0-buffer-cap: 13107
  shared-pool0-buffer-use: 0

Monitors hardware QoS resources.

...

QoS profiles determine priority field values (PCP, DSCP) for the forwarded/routed packets. Congestion avoidance/resolution is based on QoS profiles. Each packet gets a QoS profile assigned based on the ingress switch port QoS settings (see /in/eth/sw/port).

...

Priority-to-profile mapping table(-s) for trusted packets. All switch chips have one built-in map - default. In addition, some models allow the user to define custom mapping tables and assign different maps to various switch ports via the qos-map property:

  • devices based on Marvell Prestera 98DX224S, 98DX226S, or 98DX3236 switch chip models support only one map - default.
  • devices based on Marvell Prestera 98DX8xxx98DX4xxx switch chips, or 98DX325x model devices support up to 12 maps (the default + 11 user-defined).

...

VLAN Map

...

Matches VLAN priorities (802.1p PCP/DEI fields) to QoS profiles. By default, all values are matched to the default QoS profile.

...

DSCP Map

...

Matches DSCP values to QoS profiles.

...

Sub-menu: /interface/ethernet/switch/qos/tx-manager

Transmission (Tx) Manager controls packet enqueuing for transmission and packet tx order. Different switch ports can be assigned to different Tx managers. The maximum number of hardware Tx managers depends on the switch chip model (usually - 8). 

...

Transmission Queue Scheduler

Sub-menu: /interface/ethernet/switch/qos/tx-manager/queue

Each port has eight Tx queues. The assigned Tx Manager controls packet enqueuing and schedules transmission orders. Each queue can have either strict priority (where packets with the highest traffic class are always transmitted first) or grouped together for a weighted round-robin tx schedule.

Creating a Tx Manager automatically creates all eight respective queue schedulers.

Warning

Changing any properties of Tx manager or queues completely halts traffic enqueueing and transmission during the offload process. Temporary packet loss is expected while the device is forwarding traffic.

wred(yes | no; Default: no)Enables/disables Weighted Random Early Detection for the given queue
PropertyDescription
tx-manager (name; read-only)The linked Tx Manager
traffic-class (integer: 0..7 read-only)The traffic class (tc0..tc7) and the respective port queue (queue0..queue7) that the scheduler controls.
schedule (strict-priority | high-priority-group | low-priority-group )
  • strict-priority - packets in the respective queue are always scheduled before moving to lower traffic classes. Packets with lower traffic classes are not transmitted until the current queue is empty.
  • high-priority-group - all queues in the group are scheduled together by using a weighted round-robin principle. For example, if TC5 has weight 4, TC4 - 3, and TC3 - 2, then the scheduler transmits 4 packets from queue5, 3 packets from Q4, and 2 packets from Q3 in a single round. To achieve lower latency, each round is "sliced" between all queues in the group. In other words, the packet order in each round of the above example is "Q5, Q4, Q3, Q5, Q4, Q3, Q5, Q4, Q5".
  • low-priority-group - similar logic to the high-priority-group, but the low-priority-group is scheduled only when all queues in the high-priority-group are empty.
weight (integer: 0..255; Default: 1)The weight value for the traffic class if it is a member of a schedule group. The field is not used in the case of strict priority schedule.
queue-buffers (integer; Default: auto)Per-queue buffer pool. The maximum number of packets that can be assigned to the queue (per each assigned port).
use-shared-buffers (yes | no)Allow the queue to use the shared buffer pool when queue-buffers are full. If the queue is full and the shared buffers are disabled, the packet gets dropped. If the shared buffers are enabled, the queue may use up to shared-packet-cap or shared-poolX-packet-cap (see QoS Settings for details) packets from the shared pool.
shared-pool-index (integer; Default: 0)
The shared pool index for the queue to use. Relevant only if use-shared-buffers=yes and the device supports multiple shared pools.
.