...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 add name=bridge2 /interface bridge port add bridge=bridge1 interface=ether1 add bridge=bridge1 interface=ether2 add bridge=bridge2 interface=ether3 add bridge=bridge2 interface=ether4 |
Problem
After a simple performance test, you might notice that one bridge is capable of forwarding traffic at wire-speed while the second, third, etc. bridge is not able to forward as much data as the first bridge. Another symptom might be that there exists a huge latency for packets that need to be routed. After a quick inspection, you might notice that the CPU is always at full load, this is because hardware offloading is not available on all bridges, but is available only on one bridge. By checking the hardware offloading status you will notice that only one bridge has it active:
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 /interface bridge port add bridge=bridge1 hw=yes interface=ether1 learn=yes add bridge=bridge1 hw=yes interface=ether2 learn=yes |
Problem
When running Sniffer or Torch tools to capture packets you might notice that barely any packets are visible, only some unicast packets, but mostly broadcast/multicast packets are captured, while the interfaces report that much larger traffic is flowing through certain interfaces than the traffic that was captured. Since RouterOS v6.41 if you add two or more Ethernet interfaces to a bridge and enable Hardware Offloading, then the switch chip will be used to forward packets between ports. To understand why only some packets are captured, we must first examine how the switch chip is interconnected with the CPU, in this example, we can use a block diagram from a generic 5-Port Ethernet router:
...
| Code Block | ||
|---|---|---|
| ||
/interface bonding add mode=802.3ad name=bond1 slaves=ether1,ether2 /interface bridge add name=bridge1 /interface bridge port add bridge=bridge1 interface=bond1 add bridge=bridge1 interface=sfp-sfpplus1 |
Problem
After initial tests, you immediately notice that your network throughput never exceeds the 1Gbps limit even though the CPU load on the servers is low as well as on the network nodes (switches in this case), but the throughput is still limited to only 1Gbps. The reason behind this is because LACP (802.ad) uses transmit hash policy in order to determine if traffic can be balanced over multiple LAG members, in this case, a LAG interface does not create a 2Gbps interface, but rather an interface that can balance traffic over multiple slave interface whenever it is possible. For each packet a transmit hash is generated, this determines through which LAG member will the packet be sent, this is needed in order to avoid packets being out of order, there is an option to select the transmit hash policy, usually, there is an option to choose between Layer2 (MAC), Layer3 (IP) and Layer4 (Port), in RouterOS, this can be selected by using the transmit-hash-policy parameter. In this case, the transmit hash is the same since you are sending packets to the same destination MAC address, as well as the same IP address and Iperf uses the same port as well, this generates the same transmit hash for all packets and load balancing between LAG members is not possible. Note that not always packets will get balanced over LAG members even though the destination is different, this is because the standardized transmit hash policy can generate the same transmit hash for different destinations, for example, 192.168.0.1/192.168.0.2 will get balanced, but 192.168.0.2/192.168.0.4 will NOT get balanced in case layer2-and-3 transmit hash policy is used and the destination MAC address is the same.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 /interface bridge port add interface=ether1 bridge=bridge1 add interface=ether2 bridge=bridge1 /interface vlan add name=VLAN99 interface=ether1 vlan-id=99 /ip pool add name=VLAN99_POOL range=192.168.99.100-192.168.99.200 /ip address add address=192.168.99.1/24 interface=VLAN99 /ip dhcp-server add interface=VLAN99 address-pool=VLAN99_POOL disabled=no /ip dhcp-server network add address=192.168.99.0/24 gateway=192.168.99.1 dns-server=192.168.99.1 |
Problem
When you add an interface to a bridge, the bridge becomes the master interface and all bridge ports become slave ports, this means that all traffic that is received on a bridge port is captured by the bridge interface and all traffic is forwarded to the CPU using the bridge interface instead of the physical interface. As a result VLAN interface that is created on a slave interface will never capture any traffic at all since it is immediately forwarded to the master interface before any packet processing is being done. The usual side effect is that some DHCP clients receive IP addresses and some don't.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 add name=bridge2 /interface vlan add interface=bridge1 name=VLAN vlan-id=99 /interface bridge port add bridge=bridge1 interface=ether1 add bridge=bridge1 interface=ether2 add bridge=bridge2 interface=VLAN add bridge=bridge2 interface=ether3 |
Problem
To better understand the underlying problems, lets first look at the bridge host table.
...
| Code Block | ||
|---|---|---|
| ||
/interface vlan add interface=ether1 name=VLAN99 vlan-id=99 /interface bridge add name=bridge1 /interface bridge port add interface=ether2 bridge=bridge1 add interface=VLAN99 bridge=bridge1 |
Problem
This setup and configuration will work in most cases, but it violates the IEEE 802.1W standard when (R)STP is used. If this is the only device in your Layer2 domain, then this should not cause problems, but problems can arise when there are other vendor switches. The reason for this is that (R)STP on a bridge interface is enabled by default and BPDUs coming from ether1 will be sent out tagged since everything sent into ether1 will be sent out through ether2 as tagged traffic, not all switches can understand tagged BPDUs. Precautions should be made with this configuration in a more complex network where there are multiple network topologies for certain (group of) VLANs, this is relevant to MSTP and PVSTP(+) with mixed vendor devices. In a ring-like topology with multiple network topologies for certain VLANs, one port from the switch will be blocked, but in MSTP and PVSTP(+) a path can be opened for a certain VLAN, in such a situation it is possible that devices that don't support PVSTP(+) will untag the BPDUs and forward the BPDU, as a result, the switch will receive its own packet, trigger a loop detection and block a port, this can happen to other protocols as well, but (R)STP is the most common case. If a switch is using a BPDU guard function, then this type of configuration can trigger it and cause a port to be blocked by STP. It has been reported that this type of configuration can prevent traffic from being forwarded over certain bridge ports over time when using 6.41 or later. This type of configuration does not only break (R/M)STP, but it can cause loop warnings, this can be caused by MNDP packets or any other packets that are directly sent out from an interface.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge10 add name=bridge20 /interface vlan add interface=ether1 name=ether1_v10 vlan-id=10 add interface=ether1 name=ether1_v20 vlan-id=20 add interface=ether2 name=ether2_v10 vlan-id=10 add interface=ether2 name=ether2_v20 vlan-id=20 /interface bridge port add bridge=bridge10 interface=ether1_v10 add bridge=bridge10 interface=ether2_v10 add bridge=bridge20 interface=ether1_v20 add bridge=bridge20 interface=ether2_v20 |
Problem
You might notice that the network is having some weird delays or even the network is unresponsive, you might notice that there is a loop detected (packet received with own MAC address) and some traffic is being generated out of nowhere. The problem occurs because a broadcast packet that is coming from either one of the VLAN interface created on the Router will be sent out the physical interface, packet will be forwarded through the physical interface, through a switch and will be received back on a different physical interface, in this case, broadcast packets sent out ether1_v10 will be received on ether2, packet will be captured by ether2_v10, which is bridged with ether1_v10 and will get forwarded again the same path (loop). (R)STP might not always detect this loop since (R)STP is not aware of any VLANs, a loop does not exist with untagged traffic, but exists with tagged traffic. In this scenario, it is quite obvious to spot the loop, but in more complex setups it is not always easy to detect the network design flaw. Sometimes this network design flaw might get unnoticed for a very long time if your network does not use broadcast traffic, usually, Neighbor Discovery Protocol is broadcasting packets from the VLAN interface and will usually trigger a loop detection in such a setup. Sometimes it is useful to capture the packet that triggered a loop detection, this can by using sniffer and analyzing the packet capture file:
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 vlan-filtering=yes /interface bridge port add bridge=bridge1 interface=ether1 add bridge=bridge1 interface=ether2 pvid=20 add bridge=bridge1 interface=ether3 pvid=30 add bridge=bridge1 interface=ether4 pvid=40 /interface bridge vlan add bridge=bridge1 tagged=ether1 untagged=ether2 vlan-ids=20 add bridge=bridge1 tagged=ether1 untagged=ether3 vlan-ids=30 add bridge=bridge1 tagged=ether1 untagged=ether4 vlan-ids=40 |
Problem
For example, you use this configuration on a CRS1xx/CRS2xx series device and you started to notice that the CPU usage is very high and when running a performance test to check the network's throughput you notice that the total throughput is only a fraction of the wire-speed performance that it should easily reach. The cause of the problem is that not all devices support bridge VLAN filtering on a hardware level. All devices are able to be configured with bridge VLAN filtering, but only a few of them will be able to offload the traffic to the switch chip. If an improper configuration method is used on a device with a built-in switch chip, then the CPU will be used to forward the traffic.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 /interface bridge port add bridge=bridge1 interface=ether1 add bridge=bridge1 interface=ether2 add bridge=bridge1 interface=ether3 add bridge=bridge1 interface=ether4 add bridge=bridge1 interface=ether5 add bridge=bridge1 interface=ether6 add bridge=bridge1 interface=ether7 add bridge=bridge1 interface=ether8 add bridge=bridge1 interface=ether9 add bridge=bridge1 interface=ether10 /interface vlan add interface=bridge1 name=VLAN10 vlan-id=10 /interface ethernet switch port set ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9 default-vlan-id=10 vlan-header=always-strip vlan-mode=secure set ether10 vlan-header=add-if-missing vlan-mode=secure set switch1-cpu,switch2-cpu vlan-mode=secure /interface ethernet switch vlan add ports=ether1,ether2,ether3,ether4,ether5,switch1-cpu switch=switch1 vlan-id=10 add ports=ether6,ether7,ether8,ether9,ether10,switch2-cpu switch=switch2 vlan-id=10 |
Problem
After running a few tests you might notice that packets from ether6-ether10 are forwarded as expected, but packets from ether1-ether5 are not always forwarded correctly (especially through the trunk port). The most noticeable issue would be that packets from ether1-ether5 through ether10 are simply dropped, this is because these ports are located on different switch chip, this means that VLAN filtering is not possible on a hardware level since the switch chip is not aware of the VLAN table's contents on a different switch chip. Packets that are being forwarded between ports that are located on different switch chips are also processed by the CPU, which means you won't be able to achieve wire-speed performance.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 vlan-filtering=yes /interface bridge port add bridge=bridge1 interface=ether2 add bridge=bridge1 interface=ether3 pvid=10 add bridge=bridge1 interface=ether4 pvid=20 /interface bridge vlan add bridge=bridge1 tagged=ether2 vlan-ids=10,20 |
Problem
Traffic is correctly forwarded and tagged from access ports to trunk port, but you might notice that some broadcast or multicast packets are actually flooded between both untagged access ports, although they should be on different VLANs. Furthermore, broadcast and multicast traffic from the tagged port is also flooded to both access ports. This might raise some security concerns as traffic from different networks can be sniffed. When you look at the bridge VLAN table, you notice that a single entry has been created for VLANs 10 and 20, and both untagged ports are part of the same VLAN group.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 /interface bridge port add bridge=bridge1 interface=ether1 add bridge=bridge1 interface=ether2 /interface vlan add interface=bridge1 name=VLAN99 vlan-id=99 |
Problem
As soon as you try to increase the MTU size on the VLAN interface, you receive an error that RouterOS Could not set MTU. This can happen when you are trying to set MTU larger than the L2MTU. In this case, you need to increase the L2MTU size on all slave interfaces, which will update the L2MTU size on the bridge interface. After this has been done, you will be able to set a larger MTU on the VLAN interface. The same principle applies to bond interfaces. You can increase the MTU on interfaces like VLAN, MPLS, VPLS, Bonding and other interfaces only when all physical slave interfaces have proper L2MTU set.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 /interface bridge port add interface=ether1 bridge=bridge1 add interface=ether2 bridge=bridge1 |
Problem
This is a very simplified problem, but in larger networks, this might not be very easy to detect. For instance, ping might be working since a generic ping packet will be 70 bytes long (14 bytes for Ethernet header, 20 bytes for IPv4 header, 8 bytes for ICMP header, 28 bytes for ICMP payload), but data transfer might not work properly. The reason why some packets might not get forwarded is that MikroTik devices running RouterOS by default has MTU set to 1500 and L2MTU set to something around 1580 bytes (depends on the device), but the Ethernet interface will silently drop anything that does not fit into the L2MTU size. Note that the L2MTU parameter is not relevant to x86 or CHR devices. For a device that is only supposed to forward packets, there is no need to increase the MTU size, it is only required to increase the L2MTU size, RouterOS will not allow you to increase the MTU size that is larger than the L2MTU size. If you require the packet to be received on the interface and the device needs to process this packet rather than just forwarding it, for example, in case of routing, then it is required to increase the L2MTU and the MTU size, but you can leave the MTU size on the interface to the default value if you are using only IP traffic (that supports packet fragmentation) and don't mind that packets are being fragmented. You can use the ping utility to make sure that all devices are able to forward jumbo frames:
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 protocol-mode=rstp /interface bridge port add interface=ether1 bridge=bridge1 add interface=eoip1 bridge=bridge1 |
Problem
Both devices are able to communicate with each other, but some protocols do not work properly. The reason is that as soon as you use any STP variant (STP, RSTP, MSTP), you make the bridge compliant with IEEE 802.1D and IEEE 802.1Q, these standards recommend that packets that are destined to 01:80:C2:00:00:0X should NOT be forwarded. In cases where there are only 2 ports added to a bridge (R/M)STP should not be used since a loop cannot occur from 2 interfaces and if a loop does occur, the cause is elsewhere and should be fixed on a different bridge. Since (R/M)STP is not needed in transparent bridge setups, it can be disabled. As soon as (R/M)STP is disabled, the RouterOS bridge is not compliant with IEEE 802.1D and IEEE 802.1Q and therefore will forward packets that are destined to 01:80:C2:00:00:0X.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 protocol-mode=none /interface bridge port add interface=ether1 bridge=bridge1 add interface=wlan1 bridge=bridge1 /ip address add address=192.168.1.X/24 interface=bridge1 |
Problem
While traffic is being forwarded properly between R1 and R2, load balancing, link failover is working properly as well, but devices between R1 and R2 are not always accessible or some of them are completely inaccessible (in most cases AP2 and ST2 are inaccessible). After examining the problem you might notice that packets do not always get forwarded over the required bonding slave and as a result, never is received by the device you are trying to access. This is a network design and bonding protocol limitation. As soon as a packet needs to be sent out through a bonding interface (in this case you might be trying to send ICMP packets to AP2 or ST2), the bonding interface will create a hash based on the selected bonding mode and transmit-hash-policy and will select an interface, through which to send the packet out, regardless if the destination is only reachable through a certain interface. Some devices will be accessible because the generated hash matches the interface, on which the device is located on, but it might not choose the needed interface as well, which will result in inaccessible device. Only broadcast bonding mode does not have this kind of protocol limitation, but this bonding mode has a very limited use case.
...
Consider the following scenario, you set up a link between two devices, this can be any link, an Ethernet cable, a wireless link, a tunnel or any other connection. You decide that you want to test the link's bandwidth, but for convenience reasons, you decide to start testing the link with the same devices that are running the link.
Problem
As soon as you start Bandwidth test or Traffic generator you notice that the throughput is much smaller than expected. For very powerful routers, which should be able to forward many Gigabits per second (Gbps) you notice that only a few Gigabits per second gets forwarded. The reason why this is happening is because of the testing method you are using, you should never test throughput on a router while using the same router for generating traffic because you are adding an additional load on the CPU that reduces the total throughput.
...
| Code Block | ||
|---|---|---|
| ||
/interface bridge add name=bridge1 /interface bridge port add bridge=bridge1 horizon=1 hw=no interface=ether1 add bridge=bridge1 horizon=2 hw=no interface=ether2 add bridge=bridge1 horizon=3 hw=no interface=ether3 add bridge=bridge1 horizon=4 hw=no interface=ether4 |
Problem
After setting the bridge split-horizon on each port, you start to notice that each port is still able to send data between each other. The reason for this is the misuse of bridge split-horizon. A bridge port is only not able to communicate with ports that are in the same horizon, for example, horizon=1 is not able to communicate with horizon=1, but is able to communicate with horizon=2, horizon=3 and so on.
...
Consider the following scenario, you have decided to use optical fiber cables to connect your devices together by using SFP or SFP+ optical modules, but for convenience reasons, you have decided to use SFP optical modules that were available.
Problem
As soon as you configure your devices to have connectivity on the ports that are using these SFP optical modules, you might notice that either the link is working properly or experiencing random connectivity issues. There are many vendors that manufacture SFP optical modules, but not all vendors strictly follow SFP MSA, SFF and IEEE 802.3 standards, which can lead to unpredictable compatibility issues, which is a very common issue when using not well known or unsupported SFP optical modules in MikroTik devices.
...