...

1. QoS Marking. QoS profile matching by ingress packet headers, then egress header alternation according to the assigned QoS profiles (introduced in RouterOS v7.10).
2. QoS Enforcement. Avoid or resolve congestion based on the assigned QoS profiles and traffic shaping (introduced in RouterOS v7.13 for 98DX224S, 98DX226S, and 98DX3236 switch chips, and extended to all capable switch chips starting from RouterOS v7.15).
3. QoS Policy. Assign QoS profiles via ACL rules (introduced in RouterOS v7.15).
4. Extra QoS Features: WRED (Weighted Random Early Detection), ECN notification and processing , and processing, PFC (Priority-based Flow Control) (introduced in RouterOS v7.15 to capable switch chips).
5. Traffic shaping (introduced per-queue traffic shaping in RouterOS v7.15).

QoS Terminology

These terms will be used throughout the article.

• QoS - Quality of Service.
• ACL - Access Control List, a set of switch rules used to filter network traffic based on specified criteria.
• AQM - Active Queue Management.
• DSCP - Differentiated Services Code Point, a 6-bit field in the IP header used to prioritize network traffic.
• ECN - Explicit Congestion Notification.
• PCP - Priority Code Point, a 3-bit field in the VLAN header used to prioritize traffic within a VLAN.
• PFC - Priority-based Flow Control (IEEE 802.1Qbb).
• RoCE - RDMA over Converged Ethernet.
• WRED - Weighted Random Early Detection.
• /in/eth/sw/ a shortcut for /interface/ethernet/switch/. The shortcut works in CLI, too.

Anchor
device device

QoS Device Support

¹ Due to hardware limitations, some switch chip models may break traffic flow while accessing QoS port/queue usage data.

² The device gathers max queue fill statistics instead of displaying the current usage values. Use the reset-counters command to reset those stats.

³ The devices without PFC profiles do not support Priority-based Flow Control.

Applications and Usage Examples

...

...

Dante hardware use the following DSCP / Diffserv priority values for traffic prioritization.

The example assumes that the switch is using its default configuration, which includes a default "bridge" interface and all Ethernet interfaces added as bridge ports, and any of these interfaces could be used for Dante.

...

Dante's High and Medium priority traffic is scheduled in strict order. The devices transmits time-critical PTP packets until queue7 gets empty, then proceeds proceed with audio (queue5). Low and other traffic gets transmitted only when PTP and audio queues are empty. Since Dante does not define priority order between Low and Other traffic (usually, CS1 has lower priority than Best Effort), and the Low traffic class is reserved for future use anyway, we treat both traffic types equally by putting both into the same group with the same weight. Feel free to change the CS1/BE traffic scheduling according to the requirements if some Dante hardware in your network use uses the low-priority traffic class.

The next step is to enable trust mode for incomming incoming Layer3 packets (IP DSCP field):

...

/interface ethernet switch qos map vlan
add pcp=3 profile=streaming
add pcp=5 profile=voip
add pcp=6 profile=control

Since the pcp the parameter identifies the minimum value, all packets with a higher PCP value match too. If such behavior is undesired, add mapping for higher values. The next example sets voip profile for pcp=5 only. Packets with PCP values 6 or 7 are reset back to the default profile.

...

Each switch port has Layer2 and Layer3 trust settings that will change how ingress packets are classified into QoS profiles and what PCP and DSCP values will be used. Below are tables that describe all possible options:

¹ applies only when ingress traffic is untagged, but the egress needs to be VLAN-tagged.

...

Sub-menu:/interface/ethernet/switch/rule

The following example assigns a QoS profile based on the source MAC address.

...

Since reallocating hardware resources in runtime is not an option, RouterOS cannot automatically free queue buffers reserved for inactive ports. Those buffers remain unused. However, if the user clearly knows that the specific ports will will never be used (e.g., stay physically disconnected), the respective queue resources can be manually freed by introducing using the built-in "offline" tx-manager with minimum resources:


/interface/ethernet/switch/qos/tx-manager/
add name=offline comment="use this for always disconnected ports"

/interface/ethernet/switch/qos/tx-manager/queue/
set [find where tx-manager=offline] queue-buffers=1 use-shared-buffers=no

/interface/ethernet/switch/qos/port
set [find where !(running or name~"cpu")] tx-manager=offline

Traffic Prioritization

...

The behavior is controlled via WRED marginsthresholds. WRED margin threshold is the distance to the queue/pool buffer limit (cap) - where a random packet drop begins. A different margin threshold can be applied to queues that use or don't use shared buffers. A queue that uses a shared pool may set a bigger WRED margin threshold due to a higher overall cap (queue buffers + shared pool). RouterOS automatically chooses the actual WRED margin threshold values according to queue or shared pool capacities. The user may shift the margins thresholds in one way or another via QoS Settings.

For example, if queue1-packet-cap=96, and WRED margin threshold is 32 32 (assuming use-shared-buffers=no), then:

• first 64 packets are always enqueued (96 - 32 = 64).
• WRED kicks in starting from 65th packet; the chance of 65th packet to be dropped is 1/32 or roughly 3%; the formula: (65 - 64) / 32;
• the drop chance of 72th packet is 25%: (72 - 64) / 32 = 8/32 = 1/4;
• half of the newly enquining packets are dropped when the queue fill level reaches 80 packets: (80 - 64) / 32 = 16/32 = 1/2;
• 75% of the packets are dropped at the fill level 88: (88 - 64) / 32 = 24/32 = 3/4.

When Choosing a WRED is enabled (wred=yes), the queue cap is rounded to the applied WRED margin. In the above example, setting wred-queue-margin=64 raises queue1-packet-cap to 128. That, in turn, may lower the resources available to other queues, such as shared buffers. A much safer option is to raise wred-shared-queue-margin which may reduce the shared buffers available for the affected queue but not the shared pool itself. For example, if: "wred-shared-queue-margin=256, use-shared-buffers=yes, wred=yes, shared-pool-index=0, queue2-packet-cap=30, and shared-pool0-packet-cap=900", queue2 can use up to 768 buffers (30+900=930, rounded down by the scale of 256), and WRED starting at 512, while the other ports/queues still may use the remaining 162 buffers (30+900-768) of the shared pool.


Choosing a WRED margin value is a tradeoff between congestion anticipation and burst absorption. Setting a higher WRED margin may lead to earlier traffic rate throttling and, therefore, resolve congestion. On the other hand, a high margin leads to packet drops in limited traffic bursts that could be absorbed by the queue buffers and transformed losslessly if WRED didn't kick in. For instance, initiating a remote database connection usually starts with heavier traffic ("packet burst") at the initialization phase; then, the traffic rate drops down to a "reasonable" level. Any packet drop during the initialization phase leads to nothing but a slower database connection due to the need for retransmission. Hence, lowering the WRED margin or entirely disabling WRED on such traffic is advised. The opposite case is video streaming. Early congestion detection helps select a comfortable streaming rate without losing too much bandwidth on retransmission or/and "overshooting" by sacrificing the quality level by too much.

The following script only applies WRED to TCP/IP traffic by redirecting it to queue2. UDP and other packets are left in queue1 - since their end-points usually cannot respond to early drops. Queue1 and queue2 are scheduled equally - without prioritizing one queue over another.

threshold value is a tradeoff between congestion anticipation and burst absorption. Setting a higher WRED threshold may lead to earlier traffic rate throttling and, therefore, resolve congestion. On the other hand, a high threshold leads to packet drops in limited traffic bursts that could be absorbed by the queue buffers and transformed losslessly if WRED didn't kick in. For instance, initiating a remote database connection usually starts with heavier traffic ("packet burst") at the initialization phase; then, the traffic rate drops down to a "reasonable" level. Any packet drop during the initialization phase leads to nothing but a slower database connection due to the need for retransmission. Hence, lowering the WRED threshold or entirely disabling WRED on such traffic is advised. The opposite case is video streaming. Early congestion detection helps select a comfortable streaming rate without losing too much bandwidth on retransmission or/and "overshooting" by sacrificing the quality level by too much.

The following script only applies WRED to TCP/IP traffic by redirecting it to queue2. UDP and other packets are left in queue1 - since their end-points usually cannot respond to early drops. Queue1 and queue2 are scheduled equally - without prioritizing one queue over another.

/

/interface/ethernet/switch/qos/profile
add name=tcp-wred traffic-class=2 pcp=0 dscp=0

# move TCP traffic to queue2
/interface/ethernet/switch/rule
add new-qos-profile=tcp-wred ports=ether1,ether2,ether3,ether4 protocol=tcp switch=switch1

# set the same scheduling priority (weight) between queue1 and queue2
# apply WRED only to queue2 - TCP traffic
/interface/ethernet/switch/qos/tx-manager/queue/
set [find where traffic-class=1] weight=2 schedule=low-priority-group use-shared-buffers=yes shared-pool-index=0 wred=no
set [find where traffic-class=2] weight=2 schedule=low-priority-group use-shared-buffers=yes shared-pool-index=0 wred=yes

...

Set ecn=yes in Tx Manager to enable ECN marking. The per-queue ECN setting is unavailable due to hardware limitations. ECN and WRED share the same queue fill threshold: wred-shared-queue-marginthreshold (see  QoS Settings).


...

1. The packet is either IPv4 or IPv6.
2. The ECN field value in IP header is either ECT(1) or ECT(0).
3. Egress port's Tx Manager has ecn=yes.
4. The assigned Tx Queue uses shared buffers (use-shared-buffers=yes).
5. The Tx Queue detects congestion via WRED margins threshold.

Anchor

...

pfc

...

pfc

...

Switch QoS settings (in addition to the existing ones).

...

Port settings

...

Priority-based Flow Control (PFC)

Priority-Based Flow Control (PFC) provides lossless operation for up to eight traffic classes, so that congestion in one traffic class does not pause other traffic classes. In addition, PFC enables co-existence of loss-sensitive traffic types with loss tolerant traffic type in the same network.

PFC-capable switch chips are complaint with IEEE 802.1Qbb PFC, meaning that the respective devices are capable of generating and responding to PFC frames. On the triggering part, the PFC frame is sent by the source port and traffic class experiencing the congestion. The timer values of the generated PFC frames are 0xFFFF for pause (XOFF) and 0x0 for resume (XON), and the appropriate bit in the priority enable vector is set. On the response part, the received PFC frame pauses the specific priority queues on the port that received the PFC frame for the duration specified by the PFC frame.

In RouterOS, PFC configuration is organized in profiles, where each port can be assigned to a specific profile. A PFC profile defines the traffic classes to enable PFC on, pause/resume thresholds to send XOFF/XON PFC frames, respectively, and whenever the assigned ports should transmit or/and receive PFC frames.

While congestion occurs on egress ports, PFC is triggered on the ingress port. Shared buffers must be used to associate the amount of ingressed traffic with the respective packets waiting in Tx queues. For each PFC-enabled traffic class, set use-shared-buffers=yes to the respective Tx Queues. It is also recommended that a separate shared pool (shared-pool-index) be used for each PFC-enabled queue, especially not to mix it with PFC-disabled traffic classes.

When choosing pause and resume thresholds, consider a delay in transmitting a PFC frame and processing it by the other side. For example, device A experienced congestion at time T, transmitted a PFC pause frame to device B, and B processed the frame and halted transmission at time T+D. During the delta time D, device B still kept sending traffic. If device A has configured the pause threshold to 100%, it has no free buffers available, and, therefore, packets may drop, which is unacceptable for lossless traffic classes. Lowering the pause threshold, let's say, down to 80% issues a PFC pause frame while still having free memory to accumulate trafic during the delta time D. The same applies to resume threshold. Setting it to 0% keeps the device idle during the delta time, lowering the overall throughput.

Property Reference

Switch settings

Sub-menu:/interface/ethernet/switch

Switch QoS settings (in addition to the existing ones).

Port settings

Sub-menu:/interface/ethernet/switch/qos/port

Switch port QoS settings. Assigns a QoS profile to ingress packets on the given port. The assigned profile can be changed via match rules if the port is considered trusted.

By default, ports are untrusted and receive the default QoS profile (Best-Effort, PCP=0, DSCP=0), where priority fields are cleared from the egress packets.

Commands.

Port Stats

[admin@Mikrotik] /interface/ethernet/switch/qos/port> print stats where name=ether2
                  name:     ether2
             tx-packet:      2 887
               tx-byte:  3 938 897
           drop-packet:      1 799
             drop-byte:  2 526 144
      tx-queue0-packet:         50
      tx-queue1-packet:      1 871
      tx-queue3-packet:        774
      tx-queue5-packet:        192
        tx-queue0-byte:      3 924
        tx-queue1-byte:  2 468 585
        tx-queue3-byte:  1 174 932
        tx-queue5-byte:    291 456
    drop-queue1-packet:      1 799
      drop-queue1-byte:  2 526 144

Port Resources/Usage

[admin@crs326] /interface/ethernet/switch/qos/port> print usage

Switch port QoS settings. Assigns a QoS profile to ingress packets on the given port. The assigned profile can be changed via match rules if the port is considered trusted.

By default, ports are untrusted and receive the default QoS profile (Best-Effort, PCP=0, DSCP=0), where priority fields are cleared from the egress packets.

...

Whenever to trust the Layer 2 headers of the incoming packets (802.1p PCP field):

• ignore - ignore L2 header; use the port's profile value for all incoming packets;
• trust - use PCP field of VLAN-tagged packets for QoS profile lookup in map. Untagged packets use the port's profile value. Forwarded VLAN or priority-tagged packets receive the PCP value from the selected QoS profile (overwriting the original value).
• keep - trust but keep the original PCP value in forwarded packets. 

...

Whenever to trust the Layer 3 headers of the incoming packets (IP DSCP field):

• ignore - ignore L3 header; use either L2 header or the port's profile (depends on trust-l2).
• trust - use DSCP field of IP packets for QoS profile lookup in map. Forwarded/routed IP packets receive the DSCP value from the selected QoS profile (overwriting the original value).
• keep - trust but keep the original DSCP value in forwarded/routed packets.

...

The name of the Transmission Manager that is responsible for enqueuing and transmitting packets from the given port (see /in/eth/sw/qos/tx-manager).

Commands.

...

Port Stats

[admin@Mikrotik] /interface/ethernet/switch/qos/port> print stats where name=ether2
                  name:  ether2
           ether2
packet-cap:     136
           txpacket-packetuse:      2 8875
               txbyte-bytecap:  35 3840
 938 897
           dropbyte-packetuse:   9   1 799472
    queue0-packet-cap:     130
    dropqueue0-packet-byteuse:   2 526 144
  1
    txqueue1-queue0packet-packetcap:         505
      tx-queue1-packet-use:      1 8714
      tx-queue3-packet-cap:        77465
      txqueue3-queue5packet-packetuse:        1922
        tx-queue0-byte-cap:      3 92424 576
        txqueue0-queue1byte-byteuse:  2  468 585256
        txqueue1-queue3byte-bytecap:  1 1747 932680
        txqueue1-queue5byte-byteuse:   6 144
 291 456
    dropqueue3-queue1byte-packetcap:      1 79914 080
      dropqueue3-queue1byte-byteuse:   2 526 144

...

tx-queue0-packet .. tx-queue7-packet

...

tx-queue0-byte .. tx-queue7-byte

...

drop-queue0-packet .. drop-queue7-packet

...

drop-queue0-byte .. drop-queue7-byte

...

Port Resources/Usage

...

3 072

1 Port's packet/byte usage can exceed the capacity if Shared Buffers are enabled.

2 Only the queues in use are printed.

Port PFC Stats

[admin@crs326admin@crs317] /interface/ethernet/switch/qos/port> print usagepfc where name=ether2
    interval=1 where running 
             name:  ether2
           packet-cap:     136
           packet-use:   sfp-sfpplus1 sfp-sfpplus2   ether1
    5
          pfc:   byte-cap:  35 840
    roce     disabled disabled
   byte-use:   9 472
    queue0pfc-packet-captx:     130
    queue0-packet-use:    46   1
    queue1-packet-cap:       5
    queue1pfc-packetpaused-usetc:       4
    queue3-packet-cap:    3  65
    queue3-packet-use:       2
       queue0pfc3-byte-cappause:  24 576
  1 048 576  queue0-byte-use:     256
      queue1-byte-cap:
   7 680
  pfc3-resume:    queue1-byte-use:   6 144
10 240     queue3-byte-cap:  14 080
     
 queue3-byte-use:   3 072

...

     pfc3-use:     1 075 200 

1 Port's packet/byte usage can exceed the capacity if Shared Buffers are enabled.

...

QoS Menu

Sub-menu:/interface/ethernet/switch/qos

...

Sub-menu:/interface/ethernet/switch/qos/settings

QoS Monitor

Command:/interface/ethernet/switch/qos/monitor

[admin@CCR2216admin@crs312] /interface/ethernet/switch/qosqos> monitor once
         total-packet-cap: 11 32768480
         total-packet-use: 0454
           total-bufferbyte-cap: 327683072.0KiB
           total-buffer-byte-use: 681.0KiB
     multicast-packet-cap: 1 148
     multicast-packet-use: 0
      multicast multicast-byte-cap: 307.0KiB
       multicast-byte-use: 0
  shared-pool0-packet-cap: 2 3276296
     multicastshared-pool0-packet-use: 0
     multicast-buffershared-pool3-packet-cap: 2 3276296
     multicast-buffershared-pool3-packet-use: 0190
    shared-pool0-packetbyte-cap: 13107614.2KiB
    shared-pool0-packetbyte-use: 0
    shared-pool0pool3-bufferbyte-cap: 13107614.2KiB
    shared-pool0pool3-bufferbyte-use: 0610.5KiB

Monitors hardware QoS resources.

Anchor
qosprofile qosprofile

QoS Profile

...

QoS profiles determine priority field values (PCP, DSCP) for the forwarded/routed packets. Congestion avoidance/resolution is based on QoS profiles. Each packet gets a QoS profile assigned based on the ingress switch port QoS settings (see /in/eth/sw/port).

...

Anchor
qosmap qosmap

QoS Mapping

Sub-menu:/interface/ethernet/switch/qos/map

Priority-to-profile mapping table(-s) for trusted packets. All switch chips have one built-in map - default. In addition, some models allow the user to define custom mapping tables and assign different maps to various switch ports via the qos-map property:

• devices based on Marvell Prestera 98DX224S, 98DX226S, or 98DX3236 switch chip models support only one map - default.
• devices based on Marvell Prestera 98DX8xxx, 98DX4xxx switch chips, or 98DX325x model devices support up to 12 maps (the default + 11 user-defined).

VLAN Map

...

Sub-menu:/interface/ethernet/switch/qos/map

Priority-to-profile mapping table(-s) for trusted packets. All switch chips have one built-in map - default. In addition, some models allow the user to define custom mapping tables and assign different maps to various switch ports via the qos-map property:

• devices based on Marvell Prestera 98DX224S, 98DX226S, or 98DX3236 switch chip models support only one map - default.
• devices based on Marvell Prestera 98DX8xxx, 98DX4xxx switch chips, or 98DX325x model devices support up to 12 maps (the default + 11 user-defined).

/vlan

Matches VLAN priorities (802.1p PCP/DEI fields) to QoS profiles. By default, all values are matched to the default QoS profile.

name (string user-defined mapping table.

Property	Description
dei-only (yes | no; Default: no)	Map only packets with DEI (formerly CFI) bit set in the VLAN header.
map (name; Default: default)	The name of the mapping table.
profile (name
Property	Description
; Default: )	The
name of the

...

QoS profile to assign to the matched packets.
pcp (integer: 0..7; Default: 0)	MinimumVLAN priority (PCP) value for the lookup.

DSCP Map

Sub-menu:/interface/ethernet/switch/qos/map/vlanip

Matches VLAN priorities (802.1p PCP/DEI fields) DSCP values to QoS profiles. By default, all values are matched to the default QoS profile.

...

Anchor
txmanager txmanager

Transmission Manager

Sub-menu: /interface/ethernet/switch/qos/map/iptx-manager

Transmission (Tx) Manager controls packet enqueuing for transmission and packet tx order. Different switch ports can be assigned to different Tx managers. The maximum number of hardware Tx managers depends on the switch chip model (usually - 8). Matches DSCP values to QoS profiles.

...

Transmission Queue Scheduler

Sub-menu: /interface/ethernet/switch/qos/tx-manager

Transmission (Tx) Manager controls packet enqueuing for transmission and packet tx order. Different switch ports can be assigned to different Tx managers. The maximum number of hardware Tx managers depends on the switch chip model (usually - 8). 

...

/queue

Each port has eight Tx queues. The assigned Tx Manager controls packet enqueuing and schedules transmission orders. Each queue can have either strict priority (where packets with the highest traffic class are always transmitted first) or grouped together for a weighted round-robin tx schedule.

Creating a Tx Manager automatically creates all eight respective queue schedulers.

Anchor
pfcsettings pfcsettings

Priority-based Flow Control (PFC)

Sub-menu: /interface/ethernet/switch/qos/priority-flow-control

PFC configuration is organized in profiles. Different switch ports can be assigned to different PFC profiles. The maximum number of hardware Tx managers depends on the switch chip model. The builtin profile named "disabled" cannot be changed.

Transmission Queue Scheduler

Sub-menu: /interface/ethernet/switch/qos/tx-manager/queue

Each port has eight Tx queues. The assigned Tx Manager controls packet enqueuing and schedules transmission orders. Each queue can have either strict priority (where packets with the highest traffic class are always transmitted first) or grouped together for a weighted round-robin tx schedule.

Creating a Tx Manager automatically creates all eight respective queue schedulers.