...
Consider the following scenario, you set up a bridge and have enabled hardware offloading in order to maximize the throughput for your device, as a result, your device is working as a switch, but you want to use Sniffer or Torch tools for debugging purposes, or maybe you want to implement packet logging.
...
Code Block | ||
---|---|---|
| ||
/interface bridge add name=bridge1 /interface bridge port add bridge=bridge1 hw=yes interface=ether1 learn=yes add bridge=bridge1 hw=yes interface=ether2 learn=yes |
Problem
When running Sniffer or Torch tools to capture packets you might notice that barely any packets are visible, only some unicast packets, but mostly broadcast/multicast packets are captured, while the interfaces report that much larger traffic is flowing through certain interfaces than the traffic that was captured. Since RouterOS v6.41 if you add two or more Ethernet interfaces to a bridge and enable Hardware Offloading, then the switch chip will be used to forward packets between ports. To understand why only some packets are captured, we must first examine how the switch chip is interconnected with the CPU, in this example, we can use a block diagram from a generic 5-Port Ethernet router:
...
For this device, each Ethernet port is connected to the switch chip and the switch chip is connected to the CPU using the CPU port (sometimes called the switch-cpu port). For packets to be visible in Sniffer or TorchSniffer or Torch tools, the packet must be sent from an Ethernet port to the CPU port, this means that the packet must be destined to the CPU port (destination MAC address of the packet matches the bridge's MAC address) or the packet's MAC address has not be learnt (packet is flooded to all ports), this behavior is because of MAC learning·
The switch chip keeps a list of MAC addresses and ports called the Host table· Whenever a packet needs to be forwarded, the switch chip checks the packet's destination MAC address against the hosts table to find which port should it use to forward the packet. If the switch chip cannot find the destination MAC address, then the packet is flooded to all ports (including the CPU port). In situations where a packet is supposed to be forwarded from, for example, ether1 to ether2 and the MAC address for the device behind ether2 is in the host table, then the packet is never sent to the CPU and therefore will not be visible to Sniffer or Torch tool.
Symptoms
Below is a list of possible symptoms that might be a result of this kind of misconfiguration:
- Packets not visible by Sniffer or Torch tool
- Filter rules not working
...
Consider the following scenario, you have created a LAG interface to increase total bandwidth between 2 network nodes, usually, these are switches. For testing purposes to make sure that the LAG interface is working properly you have attached two servers that transfer data, most commonly the well-known network performance measurement tool https://en.wikipedia.org/wiki/Iperf is used to test such setups. For example, you might have made a LAG interface out of two Gigabit Ethernet ports, which gives you a 2Gbps interface while virtual interface that can load balance traffic over both interfaces and theoretically reach 2Gbps throughput, while the servers are connected using a 10Gbps interface, for example, SFP+.
...
After initial tests, you immediately notice that your network throughput never exceeds the 1Gbps limit even though the CPU load on the servers is low as well as on the network nodes (switches in this case), but the throughput is still limited to only 1Gbps. The reason behind this is because LACP (802.ad) uses transmit hash policy in order to determine if traffic can be balanced over multiple LAG members, in this case, a LAG interface does not create a 2Gbps interface, but rather an interface that can balance traffic over multiple slave interface whenever it is possible. For each packet a transmit hash is generated, this determines through which LAG member will the packet be sent, this is needed in order to avoid packets being out of order, there is an option to select the transmit hash policy, usually, there is an option to choose between Layer2 (MAC), Layer3 (IP) and Layer4 (Port), in RouterOS, this can be selected by using the transmit-hash-policy
parameter. In this case, the transmit hash is the same since you are sending packets to the same destination MAC address, as well as the same IP address and Iperf uses the same port as well, this generates the same transmit hash for all packets and load balancing between LAG members is not possible. Note that not always packets will get balanced over LAG members even though the destination is different, this is because the standardized transmit hash policy can generate the same transmit hash for different destinations, for example, 192.168.0.1/192.168.0.2 will get balanced, but 192.168.0.2/192.168.0.4 will NOT get balanced in case layer2-and-3
transmit hash policy is used and the destination MAC address is the same.
...
Choose the proper transmit hash policy and test your network's throughput properly. The simplest way to test such setups is to use multiple destinations, for example, instead of sending data to just one server, rather send data to multiple servers, this will generate a different transmit hash for each packet and will make load balancing across LAG members possible. For some setups, you might want to change the bonding interface mode to increase the total throughput, for UDP traffic balance-rr
mode might be sufficient, but can cause issues for TCP traffic, you can read more about selecting the right mode for your setup Herehere.
VLAN interface on a slave interface
...
Code Block | ||
---|---|---|
| ||
[admin@switch] /interface bridge host print where where !local Flags: X - disabled, I - invalid, D - dynamic, L - local, E - external # MAC-ADDRESS VID ON-INTERFACE BRIDGE AGE 0 D CC:2D:E0:E4:B3:A1 ether1 bridge1 42s 1 D CC:2D:E0:E4:B3:A2 ether2 bridge1 42s 2 D CC:2D:E0:E4:B3:A1 VLAN bridge2 42s 3 D CC:2D:E0:E4:B3:A2 VLAN bridge2 4 42s D 4 D CC:2D:E0:E4:B3:A3 ether3 bridge2 42s |
Devices on ether1 and ether2 need to send tagged packets with VLAN-ID 99 in order to reach the host on ether3 (other packets do not get passed towards VLAN interface and further bridged with ether3). We can see in the host table that bridge2 has learned these hosts. Packets coming from ether3 to ether1 will be correctly sent out tagged and traffic will not be flooded in bridge1. But since MAC learning is only possible between bridge ports and not on interfaces that are created on top of the bridge interface, packets sent from ether2 to ether3 will be flooded in bridge1.
...
Warning |
---|
By enabling |
VLAN in a bridge with a physical interface
...
Very similar case to VLAN on a bridge in a bridge, there are multiple possible scenarios where this could have been used, . The most popular use case is when you want to send out tagged traffic through bridge a physical interface , in such a with a VLAN (simplified trunk/access port setup). In such setup you might want traffic from one interface to receive only certain to send out tagged traffic and send out this tagged traffic as tagged through a physical interface (simplified trunk/access port setup) by just using VLAN interfaces and a bridgeon one side and untagged on the other side. To acomplish this, you create a VLAN interface on the trunk port (the tagged side), then create a bridge and add both the VLAN interface and the physical interface (the untagged side) as bridge ports.
Configuration
Code Block | ||
---|---|---|
| ||
/interface vlan add interface=ether1 name=VLAN99 vlan-id=99 /interface bridge add name=bridge1 /interface bridge port add interface=ether2 bridge=bridge1 add interface=VLAN99 bridge=bridge1 |
...
This setup and configuration will work in most cases, but it violates the IEEE 802.1W standard when (R/M)STP is used. If this is the only device in your Layer2 domain, then this should not cause problems, but problems can arise when there are other vendor switches. The reason for this is that (R)STP RSTP on a bridge interface is enabled by default and BPDUs coming from ether1 will be sent out tagged since everything sent into ether1 will be sent out through ether2 as tagged traffic, not , allowing Bridge Protocol Data Units (BPDUs) to be sent from each bridge port. While ether2 transmits BPDUs correctly without tagging, VLAN99 interface, being a bridge port, sends tagged BPDUs over ether1. Not all switches can understand tagged BPDUs. Precautions Precautions should be made with this configuration in a more complex network where there are multiple network topologies for certain (group of) VLANs, this is relevant to MSTP and PVSTP(+) with mixed vendor devices. In a ring-like topology with multiple network topologies for certain VLANs, one port from the switch will be blocked, but in MSTP and PVSTP(+) a path can be opened for a certain VLAN, in such a situation it is possible that devices that don't support PVSTP(+) will untag the BPDUs and forward the BPDU, as a result, the switch will receive its own packet, trigger a loop detection and block a port, this can happen to other protocols as well, but (R)STP is the most common case. If a switch is using a BPDU guard function, then this type of configuration can trigger it and cause a port to be blocked by STP. It has been reported that this type of configuration can prevent traffic from being forwarded over certain bridge ports over time when using 6.41 or later. This type of configuration does not only break (R/M)STP, but it can cause loop warnings, this can be caused by MNDP packets or any other packets that are directly sent out from an interface.
...
Code Block | ||
---|---|---|
| ||
/interface bridge add name=bridge1 vlan-filtering=yes /interface bridge port add bridge=bridge1 interface=ether1 pvid=99 add bridge=bridge1 interface=ether2 pvid=99 /interface bridge vlan add bridge=bridge1 tagged=ether2ether1 untagged=ether1ether2 vlan-ids=99 |
Warning |
---|
By enabling |
Bridged VLAN on physical interfaces
...
A very similar case to VLAN on a bridge in a bridge, consider the following scenario, you have a couple of switches in your network and you are using VLANs to isolate certain Layer2 domains and connect these switches to a router that assigns addresses and routes the traffic to the world. For redundancy, you connect all switches directly to the router and have enabled RSTP, but to be able to setup DHCP Server you decide that you can create a VLAN interface for each VLAN on each physical interface that is connected to a switch and add these VLAN interfaces in a bridge. A network diagram can be found below:
...
You might notice that the network is having some weird delays or even the network is unresponsive, you might notice that there is a loop detected (packet received with own MAC address) and some traffic is being generated out of nowhere. The problem occurs because a broadcast packet that is coming from either one of the VLAN interface created on the Router will be sent out the physical interface, packet will be forwarded through the physical interface, through a switch and will be received back on a different physical interface, in this case, broadcast packets sent out ether1_v10 will be received on ether2, packet will be captured by ether2_v10, which is bridged with ether1_v10 and will get forwarded again the same path (loop). (R)STP might not always detect this loop since (R)STP is not aware of any VLANs, a loop does not exist with untagged traffic, but exists with tagged traffic. In this scenario, it is quite obvious to spot the loop, but in more complex setups it is not always easy to detect the network design flaw. Sometimes this network design flaw might get unnoticed for a very long time if your network does not use broadcast traffic, usually, Neighbor Discovery Protocol is broadcasting packets from the VLAN interface and will usually trigger a loop detection in such a setup. Sometimes it is useful to capture the packet that triggered a loop detection, this can by using sniffer and analyzing the packet capture file:
...
Warning |
---|
By enabling |
Bridged VLAN
...
A more simplified scenario of Bridged VLAN on physical interfaces, but in this case, you simply want to bridge two or more VLANs together that are created on different physical interfaces. This is a very common type of setup that deserves a separate article since misconfiguring this type of setup has caused multiple network failures. This type of setup is also used for VLAN translation.
...
You may notice that certain parts of the network are not accessible and/or certain links keep flapping. This is due to (R)STP, this type of configuration forces the device to send out tagged BPDUs, that might not be supported by other devices, including RouterOS. Since a device receives a malformed packet (tagged BPDUs should not exist in your network when running (R)STP, this violates IEEE 802.1W and IEEE 802.1Q), the device will not interpret the packet correctly and can have unexpected behaviourbehavior.
Symptoms
Below is a list of possible symptoms that might be a result of this kind of misconfiguration:
...
Warning |
---|
By enabling |
Bridge VLAN filtering on non-CRS3xx
...
Before using bridge VLAN filtering check if your device supports it at the hardware level, a table with compatibility can be found at the Bridge Hardware Offloading section. Each type of device currently requires a different configuration method, below is a list of which configuration should be used on a device in order to use the benefits of hardware offloading:
VLAN filtering with multiple switch chips
...
Code Block | ||
---|---|---|
| ||
[admin@SW1] /interface bridge vlan print where tagged=ether2 FlagsColumns: XBRIDGE, VLAN-IDS, disabledCURRENT-TAGGED, D - dynamic # CURRENT-UNTAGGED # BRIDGE VLAN-IDS CURRENT-TAGGED CURRENT-UNTAGGED ;;; 0port with pvid bridge1added to untagged group which might cause problems, consider adding a separate VLAN entry 0 bridge1 10 ether2 ether3 ether3 20 ether4 ether4 |
Symptoms
- Traffic is flooded between different VLANs
- Red warning:
port with pvid added to untagged group which might cause problems, consider adding a separate VLAN entry
Solution
When access ports have been configured using the pvid property, they get dynamically added to the appropriate VLAN entry. After creating a static VLAN entry with multiple VLANs or VLAN range, the untagged access port with a matching pvid also gets included in the same VLAN group or range. It might be useful to define a large number of VLANs using a single configuration line, but extra caution should be taken when access ports are configured. For this example, separate VLAN entries should be created:
...
Problem
As soon as you start Bandwidth test or Traffic generator you notice that the throughput is much smaller than expected. For very powerful routers, which should be able to forward many Gigabits per second (Gbps) you notice that only a few Gigabits per second gets forwarded. The reason why this is happening is because of the testing method you are using, you should never test throughput on a router while using the same router for generating traffic because you are adding an additional load on the CPU that reduces the total throughput.
...
You should only use supported SFP modules. Always check the SFP compatibility table if you are intending to use SFP modules manufactured by MikroTik. There are other SFP modules that do work with MikroTik devices as well, check the Supported peripherals table to find other SFP modules that have been confirmed to work with MikroTik devices. Some unsupported modules might not be working properly at certain speeds and with auto-negotiation, you might want to try to disable it and manually set a link speed.