Page tree

Introduction

Layer 3 Hardware Offloading (L3HW, otherwise known as IP switching or HW routing) allows to offload some router features onto the switch chip. This allows reaching wire speeds when routing packets, which simply would not be possible with the CPU. 

Version Overview

Layer 3 Hardware Offloading has been introduced in RouterOS v7.1beta1. 

RouterOS v7.1beta2 added the firewall-compatible mode (l3hw=fw), which allowed running hardware-accelerated L4 stateful firewall on compatible CRS3xx devices.

RouterOS v7.1beta3 should not be used for l3hw.

RouterOS v7.1beta4: both modes (l3hw=yes and l3hw=fw) merged, allowing to run full hardware routing on some ports and firewall-compatible mode on another. Also, l3hw renamed to l3-hw-offloading.


Full Routing ConfigurationFirewall-Compatible Configuration
ROS v7.1beta1/interface/ethernet/switch set 0 l3hw=yes(Not Supported)
ROS v7.1beta2/interface/ethernet/switch set 0 l3hw=yes/interface/ethernet/switch set 0 l3hw=fw
ROS v7.1beta4

/interface/ethernet/switch set 0 l3-hw-offloading=yes

/interface/ethernet/switch/port set [find] l3-hw-offloading=yes

/interface/ethernet/switch set 0 l3-hw-offloading=yes

/interface/ethernet/switch/port set [find] l3-hw-offloading=no

RouterOS v7.1beta6: all CRS3xx devices received l3hw support.

RouterOS v7.1beta7: variable MTU / Jumbo frame support.

Switch Configuration

To enable Layer 3 Hardware Offloading, set l3-hw-offloading=yes for the switch:

/interface/ethernet/switch set 0 l3-hw-offloading=yes

Switch Port Configuration

Layer 3 Hardware Offloading can be configured for each physical switch port. For example:

/interface/ethernet/switch/port set sfp-sfpplus1 l3-hw-offloading=yes

Note that l3hw settings for switch and ports are different:

  • Setting l3-hw-offloading=no for the switch completely disables offloading - all packets will be routed by CPU.
  • However, setting l3-hw-offloading=no for a switch port only disables hardware routing from/to this particular port. Moreover, the port can still participate in Fastrack connection offloading. 

To enable full hardware routing, enable l3hw on all switch ports:

/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch/port set [find] l3-hw-offloading=yes

To make all packets go through the CPU first, and offload only the Fasttrack connections, disable l3hw on all ports but keep it enabled on the switch chip itself:

/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch/port set [find] l3-hw-offloading=no

Packets get routed by the hardware only if both source and destination ports have l3-hw-offloading=yes. If at least one of them has l3-hw-offloading=no, packets will go through the CPU/Firewall while offloading only the Fasttrack connections.

The next example enables hardware routing on all ports but the upstream port (sfp-sfpplus16). Packets going to/from sfp-sfpplus16 will enter the CPU and, therefore, subject to Firewall/NAT processing.

/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch/port set [find] l3-hw-offloading=yes
/interface/ethernet/switch/port set sfp-sfpplus16 l3-hw-offloading=no

The existing connections may be unaffected by the l3-hw-offloading setting change.

Interface Lists

It is impossible to use interface lists directly to control l3-hw-offloading because an interface list may contain virtual interfaces (such as VLAN) while the l3-hw-offloading setting must be applied to physical switch ports only. For example, if there are two VLAN interfaces (vlan20 and vlan30) running on the same switch port (trunk port), it is impossible to enable hardware routing on vlan20 but keep it disabled on vlan30.

However, an interface list may be used as a port selector. The following example demonstrates how to enable hardware routing on LAN ports (ports that belong to the "LAN" interface list) and disable it on WAN ports:

:foreach i in=[/interface/list/member/find where list=LAN] do={
    /interface/ethernet/switch/port set [/interface/list/member/get $i interface] l3-hw-offloading=yes
}

:foreach i in=[/interface/list/member/find where list=WAN] do={
    /interface/ethernet/switch/port set [/interface/list/member/get $i interface] l3-hw-offloading=no
}

Please take into account that since interface lists are not used directly in the hardware routing control, modifying the interface list also does not automatically reflect into l3hw changes. For instance, adding a switch port to the "LAN" interface list does not automatically enable l3-hw-offloading on that. The user has to rerun the above script to apply the changes.

MTU

Starting with RouterOS v7.1beta7, it is possible to change MTU on the hardware interfaces. Previously, only the default MTU 1500 was supported by the hardware, but others were processed by the CPU. The hardware supports up to 8 MTU profiles, meaning that the user can set up to 8 different MTU values for interfaces: the default 1500 + seven custom ones.

It is recommended to disable l3-hw-offloading while changing the MTU/L2MTU values on the interfaces.
MTU Change Example
/interface/ethernet/switch set 0 l3-hw-offloading=no
/interface set sfp-sfpplus1 mtu=9000 l2mtu=9022
/interface set sfp-sfpplus2 mtu=9000 l2mtu=9022
/interface set sfp-sfpplus3 mtu=10000 l2mtu=10022
/interface/ethernet/switch set 0 l3-hw-offloading=yes

Layer 2 Dependency

Layer 3 hardware processing lies on top of Layer 2 hardware processing. Therefore, L3HW offloading requires L2HW offloading on the underlying interfaces. The latter is enabled by default, but there are some exceptions. For example, CRS3xx devices support only one hardware bridge. If there are multiple bridges, others are processed by the CPU and are not subject to L3HW. 

Another example is ACL rules. If a rule redirects traffic to the CPU for software processing, then hardware routing (L3HW) is not triggered:

ACL rule to disable hardware processing on a specific port
/interface/ethernet/switch/rule/add switch=switch1 ports=ether1 redirect-to-cpu=yes
It is recommended to turn off L3HW offloading during L2 configuration.

To make sure that Layer 3 is in sync with Layer 2 on both the software and hardware sides, we recommend disabling L3HW while configuring Layer 2 features. The recommendation applies to the following configuration:

  • adding/removing/enabling/disabling bridge;
  • adding/removing switch ports to/from the bridge;
  • changing VLAN settings;
  • changing MTU/L2MTU on switch ports.

In short, disable l3-hw-offloading while making changes under /interface/bridge/ and /interface/vlan/:

Layer 2 Configuration Template
/interface/ethernet/switch set 0 l3-hw-offloading=no

/interface/bridge
# put bridge configuration changes here

/interface/vlan
# define/change VLAN interfaces

/interface/ethernet/switch set 0 l3-hw-offloading=yes

Inter-VLAN Routing

Since L3HW depends on L2HW, and L2HW is the one that does VLAN processing, Inter-VLAN hardware routing requires a hardware bridge underneath. Even if a particular VLAN has only one tagged port member, the latter must be a bridge member. Do not assign a VLAN interface directly on a switch port! Otherwise, L3HW offloading fails and the traffic will get processed by the CPU:

/interface/vlan add interface=ether2 name=vlan20 vlan-id=20

Assign VLAN interface to the bridge instead. This way, VLAN configuration gets offloaded to the hardware, and, with L3HW enabled, the traffic is subject to inter-VLAN hardware routing.

VLAN Configuration Example
/interface/ethernet/switch set 0 l3-hw-offloading=no
/interface/bridge/port add bridge=bridge interface=ether2 pvid=20
/interface/bridge/vlan add bridge=bridge tagged=bridge,ether2 vlan-ids=20
/interface/vlan add interface=bridge name=vlan20 vlan-id=20
/ip/address add address=192.0.2.1/24 interface=vlan20
/interface/bridge set bridge vlan-filtering=yes
/interface/ethernet/switch set 0 l3-hw-offloading=yes
For Inter-VLAN routing, the bridge interface must be a tagged member of every routable /interface/bridge/vlan/ entry.

Route Configuration

Suppressing HW Offload

By default, all the routes are participating to be hardware candidate routes. To further fine-tune which traffic to offload, there is an option for each route to disable/enable suppress-hw-offload

For example, if we know that majority of traffic flows to the network where servers are located, we can enable offloading only to that specific destination:

/ip/route set [find where static && dst-address!="192.168.3.0/24"] suppress-hw-offload=yes

Now only the route to 192.168.3.0/24 has H-flag, indicating that it will be the only one eligible to be selected for HW offloading:

[admin@MikroTik] > /ip/route print where static
Flags: A - ACTIVE; s - STATIC, y - COPY; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#     DST-ADDRESS       GATEWAY         D
0 As  0.0.0.0/0         172.16.2.1      1
1 As  10.0.0.0/8        10.155.121.254  1
2 AsH 192.168.3.0/24    172.16.2.1      1

H-flag does not indicate that route is actually HW offloaded, it indicates only that route can be selected to be HW offloaded.

Offloading Fasttrack Connections

Firewall filter rules have hw-offload option for Fasttrack, allowing fine-tuning connection offloading. Since the hardware memory for Fasttrack connections is very limited, we can choose what type of connections to offload and, therefore, benefit from near-the-wire-speed traffic. The next example offloads only TCP connections while UDP packets are routed via the CPU and do not occupy HW memory:

/ip/firewall/filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes protocol=tcp
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=no
add action=accept chain=forward connection-state=established,related

Configuration Examples

Inter-VLAN Routing with Upstream Port Behind Firewall/NAT

This example demonstrates how to benefit from near-to-wire-speed inter-VLAN routing while keeping Firewall and NAT running on the upstream port. Moreover, Fasttrack connections to the upstream port get offloaded to hardware as well, boosting the traffic speed close to wire-level. Inter-VLAN traffic is fully routed by the hardware, not entering the CPU/Firewall, and, therefore, not occupying the hardware memory of Fasttrack connections.

We use the CRS317-1G-16S+ model with the following setup:

  • sfp1-sfp4 - bridged ports, VLAN ID 20, untagged
  • sfp5-sfp8 - bridged ports, VLAN ID 30, untagged
  • sfp16 - the upstream port
  • ether1 - management port


Setup interface lists for easy access:

Interface Lists
/interface list
add name=LAN
add name=WAN
add name=MGMT 

/interface list member
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN
add interface=sfp-sfpplus16 list=WAN
add interface=ether1 list=MGMT 
Bridge Setup
/interface bridge
add name=bridge vlan-filtering=yes

/interface bridge port
add bridge=bridge interface=sfp-sfpplus1 pvid=20
add bridge=bridge interface=sfp-sfpplus2 pvid=20
add bridge=bridge interface=sfp-sfpplus3 pvid=20
add bridge=bridge interface=sfp-sfpplus4 pvid=20
add bridge=bridge interface=sfp-sfpplus5 pvid=30
add bridge=bridge interface=sfp-sfpplus6 pvid=30
add bridge=bridge interface=sfp-sfpplus7 pvid=30
add bridge=bridge interface=sfp-sfpplus8 pvid=30

/interface bridge vlan
add bridge=bridge tagged=bridge untagged=sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4 vlan-ids=20
add bridge=bridge tagged=bridge untagged=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8 vlan-ids=30

Routing requires dedicated VLAN interfaces. For standard L2 VLAN bridging (without inter-VLAN routing), the next step can be omitted.

VLAN Interface Setup for Routing
/interface vlan
add interface=bridge name=vlan20 vlan-id=20
add interface=bridge name=vlan30 vlan-id=30

/ip address
add address=192.168.20.17/24 interface=vlan20 network=192.168.20.0
add address=192.168.30.17/24 interface=vlan30 network=192.168.30.0

Configure management and upstream ports, a basic firewall, NAT, and enable hardware offloading of Fasttrack connections:

Firewall Setup
/ip address
add address=192.168.88.1/24 interface=ether1
add address=10.0.0.17/24 interface=sfp-sfpplus16

/ip route
add gateway=10.0.0.1

/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related

/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN

At this moment, all routing still is performed by the CPU. Enable hardware routing on the switch chip:

Enable Layer 3 Hardware Offloading
# Enable full hardware routing on LAN ports
:foreach i in=[/interface/list/member/find where list=LAN] do={ 
    /interface/ethernet/switch/port set [/interface/list/member/get $i interface] l3-hw-offloading=yes 
} 

# Disable full hardware routing on WAN or Management ports
:foreach i in=[/interface/list/member/find where list=WAN or list=MGMT] do={ 
    /interface/ethernet/switch/port set [/interface/list/member/get $i interface] l3-hw-offloading=no 
}

# Activate Layer 3 Hardware Offloading on the switch chip
/interface/ethernet/switch/set 0 l3-hw-offloading=yes

Results:

  • Within the same VLAN (e.g., sfp1-sfp4), traffic is forwarded by the hardware on Layer 2 (L2HW).
  • Inter-VLAN traffic (e.g. sfp1-sfp5) is routed by the hardware on Layer 3 (L3HW).
  • Traffic from/to WAN port gets processed by the CPU/Firewall first. Then Fasttrack connections get offloaded to the hardware (Hardware-Accelerated L4 Stateful Firewall). NAT applies both on CPU- and HW-processed packets.
  • Traffic to the management port is protected by the Firewall.

L3HW Feature Support

  • HW - the feature is supported and offloaded to the hardware.
  • CPU - the feature is supported but performed by software (CPU)
  • N/A - the feature is not available together with L3HW. Layer 3 hardware offloading must be completely disabled (switch l3-hw-offloading=no) to make this feature work.
  • FW - the feature requires l3-hw-offloading=no for a given switch port. On the switch level, l3-hw-offloading=yes.


FeatureSupportCommentsRelease
IPv4 Unicast RoutingHW
7.1beta1
IPv6 Unicast RoutingCPU

IPv4 Multicast RoutingCPU

IPv6 Multicast RoutingCPU

ECMPHWMultipath routing7.1beta1
BlackholesHW
/ip/route add dst-address=10.0.99.0/24 type=blackhole
7.1beta1
Prohibited routesCPU
/ip/route add dst-address=10.0.99.0/24 type=prohibit 

The hardware is incapable of sending ICMP replies. Therefore the packets must go through the CPU.
Use blackhole to silently drop traffic on the hardware level.


Unreachable routesCPU /ip/route add dst-address=10.0.99.0/24 type=unreachable 
gateway=<interface_name>CPU/HW
/ip/route add dst-address=10.0.0.0/24 gateway=ether1 

This works only for directly connected networks. Since HW does not know how to send ARP requests,
CPU sends an ARP request and waits for a reply to find out a DST MAC address on the first received packet of the connection that matches a DST IP address.
After DST MAC is determined, HW entry is added and all further packets will be processed by the switch chip.

7.1beta1
BRIDGEHWIP Routing from/to bridge interface7.1beta1
VLANHWRouting between VLAN interfaces7.1beta1
BondingHW
/interface/bonding
7.1beta1
FirewallFWUsers must choose either HW-accelerated routing or firewall.
Firewall rules get processed by the CPU. Fasttrack connections get offloaded to HW.
7.1beta4
NATFWNAT rules applied to the offloaded Fasttrack connections get processed by HW too.7.1beta4
VRFN/A

VRRPN/A

MTUHWThe hardware supports up to 8 MTU profiles.7.1beta7

Only the devices listed in the table below support L3 HW Offloading.

L3HW Device Support

Only the devices listed in the table below support L3 HW Offloading.

CRS300: Switch DX8000 and DX4000 Series

The devices below are based on Marvell 98DX8xxx, 98DX4xxx switch chips, or the 98DX3257 model.

ModelReleaseIPv4 Routes 1NexthopsFasttrack connections 2,3,4NAT entries 2 
CRS317-1G-16S+7.1beta1160K - 240K8K4.5K8K 5
CRS309-1G-8S+7.1beta216K - 30K8K4.5K8K
CRS312-4C+8XG7.1beta216K - 30K8K2.25K8K
CRS326-24S+2Q+7.1beta216K - 30K8K2.25K8K
CRS354-48G-4S+2Q+7.1beta416K - 30K8K2.25K8K

 1 Depends on the complexity of the routing table. Whole-byte IP prefixes (/8, /16, /24, etc.) occupy less HW space than others (e.g., /22). When the HW route limit is reached, new route entries are ignored by the switch chip! The user should choose the device with HW capability large enough to store all the routes or/and suppress offloading of the routes that do not fit in the HW memory.

2 When the HW limit of Fasttrack or NAT entries is reached, other connections will fall back to the CPU. MikroTik's smart connection offload algorithm ensures that the connections with the most traffic are offloaded to the hardware.

3 Fasttrack connections share the same HW memory with ACL rules. Depending on the complexity, one ACL rule may occupy the memory of 3-6 Fasttrack connections.

4 MPLS shares the HW memory with Fasttrack connections. Moreover, enabling MPLS requires the allocation of the entire memory region, which could store up to 768 (0.75K) Fasttrack connections otherwise. The same applies to Bridge Port Extender. However, MPLS and BPE may use the same memory region, so enabling them both doesn't double the limitation of Fasttrack connections.

5 All NAT entries cannot be used due to the limited amount of Fasttrack connections.

CRS300: Switch DX3000 and DX2000 Series

The devices below are based on Marvell 98DX224S, 98DX226S, or 98DX3236 switch chip model. These devices do not support Fasttrack or NAT connection offloading.

The 98DX3257 model is an exception, which has a feature set of the DX8000 rather than the DX3000 series.
ModelReleaseIPv4 Route Prefixes1NexthopsECMP paths per prefix2
CRS305-1G-4S+7.1beta6133124K8
CRS318-1Fi-15Fr-2S7.1beta6133124K8
CRS318-16P-2S+7.1beta6133124K8
CRS326-24G-2S+7.1beta6133124K8
CRS328-24P-4S+7.1beta6133124K8
CRS328-4C-20S-4S+7.1beta6133124K8

1 Since the total amount of routes that can be offloaded is limited, prefixes with higher netmask are preferred to be forwarded by hardware (e.g /32 /30 /29, etc.), any other prefixes that do not fit in the HW table will be processed by the CPU.

2 If a route has more paths than the hardware ECMP limit (X), only the first X paths get offloaded.

  • No labels