Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Hotspot-in - allows to capture traffic that otherwise would be discarded by connection tracking - this way our Hotspot feature is able to provide connectivity even if networks settings are an incomplete mess ;
  • RAW Prerouting - RAW table prerouting chain;
  • Connection tracking - packet is processed by connection tracking;
  • Mangle Prerouting - Mangle prerouting chain;
  • Mangle Input - Mangle input chain;
  • Filter Input - Firewall filter input chain;
  • HTB Global - Queue tree;
  • Simple Queues - is a feature that can be used to limit traffic for a particular target;
  • TTL - indicates an exact place where the Time To Live (TTL) of the routed packet is reduced by 1 if TTL becomes 0, a packet will be discarded;
  • Mangle Forward - Mangle forward chain;
  • Filter Forward - Mangle Filter forward chain;
  • Accounting - Authentication, Authorization, and Accounting feature processing;
  • RAW Output - RAW table output chain;
  • Mangle Output - Mangle output chain;
  • Filter Output - Firewall filter output chain;
  • Routing Adjustment - this is a workaround that allows to set up policy routing in mangle chain output (routing-mark) ;
  • Mangle Postrouting - Mangle postrouting chain;
  • Src Nat - Network Address Translation srcnat chain;
  • Dst Nat - Network Address Translation dstnat chain;
  • Hotspot-out - undo all that was done by hotspot-in for the packets that are going back to the client;

...

Section


Column
width49%
  1. A packet goes through the bridge NAT dst-nat chain, where MAC destination and priority can be changed, apart from that, a packet can be simply accepted, dropped, or marked;
  2. Checks whether the use-ip-firewall option is enabled in the bridge settings;
  3. Run packet through the bridge host table to make a forwarding decision. A packet that ends up being flooded (e.g. broadcast, multicast, unknown unicast traffic), gets multiplied per bridge port and then processed further in the bridge forward chain. When using vlan-filtering=yes, packets that are not allowed due to the "/interface bridge vlan" table, will be dropped at this stage.
  4. A packet goes through the bridge filter forward chain, where priority can be changed or the packet can be simply accepted, dropped, or marked;
  5. Checks whether the use-ip-firewall option is enabled in the bridge settings;
  6. A packet goes through the bridge NAT src-nat chain, where MAC source and priority can be changed, apart from that, a packet can be simply accepted, dropped, or marked;
  7. Checks whether the use-ip-firewall option is enabled in the bridge settings;


Column
width49%




Info

For RouterOS v6:
When bridge vlan-filtering is enabled, received untagged packets might get encapsulated into the VLAN header before the "DST-NAT" block, which means these packets can be filtered using the mac-protocol=vlan and vlan-encap settings. Encapsulation can happen if the outgoing interface has frame-types is set  set to admit-all or admit-only-untagged-and-priority-tagged.

Tagged packets might get decapsulated on the "BRIDGING DECISION" block, which means these packets will no longer match the mac-protocol=vlan and vlan-encap settings. Decapsulation can happen if the packet's VLAN ID matches the outgoing port's untagged VLAN membership.

Bridge Input

Bridge input is a process that takes place when a packet is destined for the bridge interface. Most commonly this happens when you need to reach some services that are running on the bridge interface (e.g. a DHCP server) or you need to route traffic to other networks. The very first steps are similar to the bridge forward process - after receiving a packet on the in-interface, the device determines that the in-interface is a bridge port, so it gets passed through the bridging process:


For RouterOS v7 and newer:

When bridge vlan-filtering is enabled, received untagged packets might get encapsulated into the VLAN header on the "BRIDGING-DECISION" block, which means these packets can be filtered using the mac-protocol=vlan and vlan-encap settings.  Encapsulation can happen if the outgoing interface has frame-types set to admit-all or admit-only-untagged-and-priority-tagged.

Tagged packets might get decapsulated on the "BRIDGING DECISION" block, which means these packets will no longer match the mac-protocol=vlan and vlan-encap settings. Decapsulation can happen if the packet's VLAN ID matches the outgoing port's untagged VLAN membership.

Bridge Input

Bridge input is a process that takes place when a packet is destined for the bridge interface. Most commonly this happens when you need to reach some services that are running on the bridge interface (e.g. a DHCP server) or you need to route traffic to other networks. The very first steps are similar to the bridge forward process - after receiving a packet on the in-interface, the device determines that the in-interface is a bridge port, so it gets passed through the bridging process:

Section


Column
width400px
  1. A packet goes through the bridge NAT dst-nat chain, where MAC destination and priority can be changed, apart from that, a
Section
Column
width400px
  1. A packet goes through the bridge NAT dst-nat chain, where MAC destination and priority can be changed, apart from that, a packet can be simply accepted, dropped, or marked;
  2. Checks whether the use-ip-firewall option is enabled in the bridge settings;
  3. Run packet through the bridge host table to make a forwarding decision. A packet where the destination MAC address matches the bridge MAC address will be passed to the bridge input chain. A packet that ends up being flooded (e.g. broadcast, multicast, unknown unicast traffic), also reaches the bridge input chain as the bridge interface itself is one of the many destinations;
  4. A packet goes through the bridge filter input chain, where priority can be changed or the packet can be simply accepted, dropped, or marked;
Column
width50%

Image Removed

Bridge Output

Bridge output is a process that takes place when a packet should exit the device through one or multiple bridge ports. Most commonly this happens when a bridge interface itself tries to reach a device connected to a certain bridge port (e.g. when a DHCP server running on a bridge interface is responding to a DHCP client). After a packet is processed on other higher-level RouterOS processes and the device finally determines that the output interface is a bridge, the packet gets passed through the bridging process:

  1. Checks whether the use-ip-firewall option is enabled in the bridge settings;
  2. Run packet through the bridge host table to make a forwarding decision. A packet where the destination MAC address matches the bridge MAC address will be passed to the bridge input chain. A packet that ends up being flooded (e.g. broadcast, multicast, unknown unicast traffic), also reaches the bridge input chain as the bridge interface itself is one of the many destinations;
  3. A packet goes through the bridge filter input chain, where priority can be changed or the packet can be simply accepted, dropped, or marked;


Column
width50%

Image Added


Bridge Output

Bridge output is a process that takes place when a packet should exit the device through one or multiple bridge ports. Most commonly this happens when a bridge interface itself tries to reach a device connected to a certain bridge port (e.g. when a DHCP server running on a bridge interface is responding to a DHCP client). After a packet is processed on other higher-level RouterOS processes and the device finally determines that the output interface is a bridge, the packet gets passed through the bridging process:

Section


Column
width400px
  1. Run packet through the bridge host table to make a forwarding decision. A packet that ends up being flooded (e.g. broadcast, multicast, unknown unicast traffic), gets multiplied per bridge port and then processed further in the bridge output chain.
  2. A packet
Section
Column
width400px
  1. Run packet through the bridge host table to make a forwarding decision. A packet that ends up being flooded (e.g. broadcast, multicast, unknown unicast traffic), gets multiplied per bridge port and then processed further in the bridge output chain.
  2. A packet goes through the bridge filter output chain, where priority can be changed or the packet can be simply accepted, dropped, or marked;
  3. A packet goes through the bridge NAT src-nat chain, where MAC source and priority can be changed, apart from that, a packet can be simply accepted, dropped, or marked;
  4. Checks whether the use-ip-firewall option is enabled in the bridge settings;


Column
width50%


...

Section


Column
width400px
  1. A packet goes through the bridge NAT dst-nat chain;
  2. With the use-ip-firewall option enabled, the packet will be further processed in the prerouting chain;
  3. A packet enters prerouting processing;
  4. Run packet through the bridge host table to make forwarding decision;
  5. A packet goes through the bridge filter forward chain;
  6. With the use-ip-firewall option enabled, the packet will be further processed in the routing forward chain;
  7. A packet enters routing forward processing;
  8. A packet goes through the bridge NAT src-nat chain;
  9. With the use-ip-firewall option enabled, the packet will be further processed in the postrouting chain;
  10. A packet enters prerouting postrouting processing;


Column
width50%


...

On the previous topic, we solely discussed a software bridging that requires the main CPU processing to forward packets through the correct bridge port. Most of the MikroTik devices are equipped with dedicated switching hardware, the so-called switch chip or switch ASIC. This allows us to offload some of the bridging functions, like packet forwarding between bridge ports or packet filtering, to this specialized hardware chip without consuming any CPU resources. In RouterOS, we have named this function Bridge Hardware (HW) Offloading. Different MikroTik devices might have different switch chips and each chip has a different set of features available, so make sure to visit this article to get more details - Bridge Hardware Offloading.


Note

Interface HTB will not work correctly when the out-interface is hardware offloaded and the bridge Fast Path is not active.

...

  • switching decision - widely depends on the switch model. This block controls all the switching-related tasks, like host learning, packet forwarding, filtering, rate-limiting, VLAN tagging/untagging, mirroring, etc. Certain switch configurations can alter the packet flow;
  • switch-cpu port - a special purpose switch port for communication between the main CPU and other switch ports. Note that the switch-cpu port does not show up anywhere on RouterOS except for the switch menu, none of the software-related configurations (e.g. interface-list) can be applied to this port. Packets that reach the CPU are automatically associated with the physical in-interface.

The hardware offloading, however, does not restrict a device to only hardware limited features, rather it is possible to take advantage of the hardware and software processing at the same time. This does require a profound understanding of how packets travel through the switch chip and when exactly they are passed to the main CPU.

Info

Switch features found in the "/interface/ethernet/switch" menu and its sub-menus, like ACL rules, mirroring, ingress/egress rate limiters, QoS, and L3HW (except inter-VLAN routing) may not rely on bridge hardware offloading. Therefore, they can potentially be applied to interfaces not configured within a hardware-offloaded bridge.

Switch ForwardSwitch Forward

We will further discuss a packet flow when bridge hardware offloading is enabled and a packet is forwarded between two switched ports on a single switch chip. This is the most common and also the simplest example:

...

Section



Column



Push Label

Section



Column


Logical Interfaces

So far we looked at examples when in or out interfaces are actual physical interfaces (Ethernet, wireless), but how packets will flow if the router receives tunnel encapsulated packets?

Image Removed

Let's assume that there is an IPIP packet coming into the router. Since it is a regular ipv4 packet it will be processed through all routing-related facilities ( until "J" in the diagram). Then the router will look if the packet needs to be decapsulated., in this case, it is an IPIP packet so "yes" send the packet to decapsulation. After that packet will go another loop through all the facilities but this time as a decapsulated IPv4 packet.

It is very important because the packet actually travels through the firewall twice, so if there is a strict firewall, then there should be "accept" rules for IPIP encapsulated packets as well as decapsulated IP packets.

Info

Packet encapsulation and decapsulation using a bridge with enabled vlan-filtering do not relate to logical interfaces. See more details in the bridging section.

IPSec Policies


MPLS IP VPN

In VPNv4 setups packet arriving at PE router that needs to be forwarded to the CE router is not a typical "forward". 

If incoming label and destination is bound to the VRF Then after MPLS label is popped and:

  • destination address is local to the router, then packet is moved to LOCAL_IN
  • destination address is in the CE network, then packet is moved to LOCAL_OUT
Warning

Forwarded packets from MPLS cloud to the CE network will not show up in the forward.


For example traffic from src:111.15.0.1 to dst:111.13.0.1

Code Block
languagetext
[admin@CCR2004_2XS] /mpls/forwarding-table> print 
Flags: L - LDP, P - VPN
Columns: LABEL, VRF, PREFIX, NEXTHOPS
#   LABEL  VRF      PREFIX         NEXTHOPS                                                                                                           
0 P    17  myVrf    111.13.0.0/24  
4 L    20  main     203.0.113.2    { label=impl-null; nh=111.11.0.1; interface=sfp-sfpplus1 }
[admin@CCR2004_2XS] /mpls/forwarding-table>                                                           
...

[admin@CCR2004_2XS] /ip/route> print detail 
Flags: D - dynamic; X - disabled, I - inactive, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, i - is-is, d - dhcp, v - vpn, m - modem, y - bgp-mpls-vpn; 
H - hw-offloaded; + - ecmp 

   DAc   dst-address=111.11.0.0/24 routing-table=main gateway=sfp-sfpplus1 immediate-gw=sfp-sfpplus1 distance=0 scope=10 suppress-hw-offload=no 
         local-address=111.11.0.2%sfp-sfpplus1 
   DAc   dst-address=203.0.113.1/32 routing-table=main gateway=lo immediate-gw=lo distance=0 scope=10 suppress-hw-offload=no local-address=203.0.113.1%lo 
   DAo   dst-address=203.0.113.2/32 routing-table=main gateway=111.11.0.1%sfp-sfpplus1 immediate-gw=111.11.0.1%sfp-sfpplus1 distance=110 scope=20 target-scope=10 
         suppress-hw-offload=no 
   DAc   dst-address=111.13.0.0/24 routing-table=myVrf gateway=sfp-sfpplus2@myVrf immediate-gw=sfp-sfpplus2 distance=0 scope=10 suppress-hw-offload=no 
         local-address=111.13.0.2%sfp-sfpplus2@myVrf 
   DAy   dst-address=111.15.0.0/24 routing-table=myVrf gateway=203.0.113.2 immediate-gw=111.11.0.1%sfp-sfpplus1 distance=200 scope=40 target-scope=30 suppress-hw-offload=no [admin@CCR2004_2XS] /ip/route>              

Packet will be seen in the output and postrouting chains, because now it is locally originated packet with source MAC address equal to vrfInterface:

Code Block
languagetext
08:10:55 firewall,info output: in:(unknown 0) out:sfp-sfpplus2, connection-state:established src-mac f2:b5:e9:17:18:3b, proto ICMP (type 8, code 0), 111.15.0.1->111.13.0.1, len 56
08:10:55 firewall,info postrouting: in:myVrf out:sfp-sfpplus2, connection-state:established src-mac f2:b5:e9:17:18:3b, proto ICMP (type 8, code 0), 111.15.0.1->111.13.0.1, len 56

On the other hand, for packets routed in the direction CE→PE will be seen in the "forward" as any other routed IP traffic before being sent to the MPLS:

Code Block
languagetext
 08:10:55 firewall,info prerouting: in:sfp-sfpplus2 out:(unknown 0), connection-state:established src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, len 56
 08:10:55 firewall,info forward: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, len 56
 08:10:55 firewall,info postrouting: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, len 56
 


But there can be an exception. If destination IP of reply packet is local to the router and connection tracking is performing NAT translation then connection tracking will "force" packet to move through the firewall prerouting/forward/postrouting chains.

Logical Interfaces

So far we looked at examples when in or out interfaces are actual physical interfaces (Ethernet, wireless), but how packets will flow if the router receives tunnel encapsulated packets?

Image Added

Let's assume that there is an IPIP packet coming into the router. Since it is a regular ipv4 packet it will be processed through all routing-related facilities ( until "J" in the diagram). Then the router will look if the packet needs to be decapsulated., in this case, it is an IPIP packet so "yes" send the packet to decapsulation. After that packet will go another loop through all the facilities but this time as a decapsulated IPv4 packet.

It is very important because the packet actually travels through the firewall twice, so if there is a strict firewall, then there should be "accept" rules for IPIP encapsulated packets as well as decapsulated IP packets.

Info

Packet encapsulation and decapsulation using a bridge with enabled vlan-filtering do not relate to logical interfaces. See more details in the bridging section.


IPSec Policies

Let's take a look at another tunnel type - IPSec. This type of VPN does not have logical interfaces but is processed in a similar Let's take a look at another tunnel type - IPSec. This type of VPN does not have logical interfaces but is processed in a similar manner.

Section


Column
width400px
Instead of logical interfaces packets are processed through IPSec policies. After routing decision (2) and input firewall processing (3), the router tries to match the source and destination to the IPsec policy. When policy matches the packet it is sent to decryption (5). After the decryption packet enters PREROUTING processing again (6) and starts another processing loop, but now with the decapsulated packet.


Column



...

RouterBoardInterfaces
RB6xx seriesether1,2RB7xx seriesall ports
RB800ether1,2
RB9xx seriesall ports
RB1000all ports
RB1100 seriesether1-11
RB2011 seriesall ports
RB3011 seriesall ports
CRS series routersall ports
CCR series routersall ports
All deviceswireless interfaces
bridge interfacesVLAN, VRRP interfacesbonding interfaces (RX only)PPPoE, L2TP interfacesEoIP, GRE, IPIP interfaces.

EoIP, Gre, IPIP, and L2TP interfaces have per-interface setting allow-fast-path. Allowing a fast path on these interfaces has a side effect of bypassing firewall, connection tracking, simple queues, queue tree with parent=global, IP accounting, IPsec, hotspot universal client, vrf assignment for encapsulated packets that go through a fast-path. Also, packet fragments cannot be received in FastPath.

Note

FastPath support can be verified by checking the fast-path property value in /interface print detail.

Only interface queue that guarantees FastPath is only-hardware-queue. If you need an interface queue other than hardware then the packet will not go fully FastPath, but there is not a big impact on performance, as "interface queue" is the last step in the packet flow.

The packet may go Half-FastPath by switching from FastPath to SlowPath, but not the other way around. So, for example, if the receiving interface has FastPath support, but the out interface does not, then the router will process the packet by FastPath handlers as far as it can and then proceed with SlowPath. If the receiving interface does not support FastPath but the out interface does, the packet will be processed by SlowPath all the way through the router.

Image Removed

FastTrack

Fasttrack can be decoded as Fast Path + Connection Tracking. It allows marking connections as "fast-tracked", marking packets that belong to fast-tracked connection will be sent fast-path way. The connection table entry for such a connection now will have a fast-tracked flag.

Image Removed

Note

FastTrack packets bypass firewall, connection tracking, simple queues, queue tree with parent=global, ip traffic-flow(restriction removed in 6.33), IP accounting, IPSec, hotspot universal client, VRF assignment, so it is up to the administrator to make sure FastTrack does not interfere with other configuration!

To mark a connection as fast-tracked new action was implemented "fasttrack-connection" for firewall filter and mangle. Currently, only IPv4 TCP and UDP connections can be fast-tracked and to maintain connection tracking entries some random packets will still be sent to a slow path. This must be taken into consideration when designing firewalls with enabled "fasttrack".

FastTrack handler also supports source and destination NAT, so special exceptions for NATed connections are not required.

Warning
Traffic that belongs to a fast-tracked connection travels in FastPath, which means that it will not be visible by other router L3 facilities (firewall, queues, IPsec, IP accounting, VRF assignment, etc).

The easiest way to start using this feature on home routers is to enable "fasttrack" for all established, related connections:

Code Block
languageros
linenumberstrue
/ip firewall filter 
add chain=forward action=fasttrack-connection connection-state=established,related \
  comment="fasttrack established/related"
add chain=forward action=accept connection-state=established,related \
  comment="accept established/related"

Notice that the first rule marks established/related connections as fast-tracked, the second rule is still required to accept packets belonging to those connections. The reason for this is that, as was mentioned earlier, some random packets from fast-tracked connections are still sent the slow pathway and only UDP and TCP are fast-tracked, but we still want to accept packets for other protocols.

Image RemovedImage Removed

After adding the "FastTrack" rule special dummy rule appeared at the top of the list. This is not an actual rule, it is for visual information showing that some of the traffic is traveling FastPath and will not reach other firewall rules.

These rules appear as soon as there is at least one fast-tracked connection tracking entry and will disappear after the last fast-tracked connection times out in the connection table.

...

All devicesEthernet interfaces
wireless interfaces
bridge interfaces
VLAN, VRRP interfaces
bonding interfaces (RX only)
PPPoE, L2TP interfaces
EoIP, GRE, IPIP, VXLAN interfaces.

EoIP, Gre, IPIP, VXLAN and L2TP interfaces have per-interface setting allow-fast-path. Allowing a fast path on these interfaces has a side effect of bypassing firewall, connection tracking, simple queues, queue tree with parent=global, IP accounting, IPsec, hotspot universal client, vrf assignment for encapsulated packets that go through a fast-path. Also, packet fragments cannot be received in FastPath.

Note

FastPath support can be verified by checking the fast-path property value in /interface print detail.

Only interface queue that guarantees FastPath is only-hardware-queue. If you need an interface queue other than hardware then the packet will not go fully FastPath, but there is not a big impact on performance, as "interface queue" is the last step in the packet flow.

The packet may go Half-FastPath by switching from FastPath to SlowPath, but not the other way around. So, for example, if the receiving interface has FastPath support, but the out interface does not, then the router will process the packet by FastPath handlers as far as it can and then proceed with SlowPath. If the receiving interface does not support FastPath but the out interface does, the packet will be processed by SlowPath all the way through the router.

Image Added

FastTrack

Fasttrack can be decoded as Fast Path + Connection Tracking. It allows marking connections as "fast-tracked", marking packets that belong to fast-tracked connection will be sent fast-path way. The connection table entry for such a connection now will have a fast-tracked flag.

Image Added

Note

FastTrack packets bypass firewall, connection tracking, simple queues, queue tree with parent=global, ip traffic-flow(restriction removed in 6.33), IP accounting, IPSec, hotspot universal client, VRF assignment, so it is up to the administrator to make sure FastTrack does not interfere with other configuration!

To mark a connection as fast-tracked new action was implemented "fasttrack-connection" for firewall filter and mangle. Currently, only IPv4 TCP and UDP connections can be fast-tracked and to maintain connection tracking entries some random packets will still be sent to a slow path. This must be taken into consideration when designing firewalls with enabled "fasttrack".

FastTrack handler also supports source and destination NAT, so special exceptions for NATed connections are not required.

Warning
Traffic that belongs to a fast-tracked connection travels in FastPath, which means that it will not be visible by other router L3 facilities (firewall, queues, IPsec, IP accounting, VRF assignment, etc). Fasttrack lookups route before routing marks have been set, so it works only with the main routing table.

The easiest way to start using this feature on home routers is to enable "fasttrack" for all established, related connections:

Code Block
languageros
linenumberstrue
/ip firewall filter 
add chain=forward action=fasttrack-connection connection-state=established,related \
  comment="fasttrack established/related"
add chain=forward action=accept connection-state=established,related \
  comment="accept established/related"

Notice that the first rule marks established/related connections as fast-tracked, the second rule is still required to accept packets belonging to those connections. The reason for this is that, as was mentioned earlier, some random packets from fast-tracked connections are still sent the slow pathway and only UDP and TCP are fast-tracked, but we still want to accept packets for other protocols.

Image AddedImage Added

After adding the "FastTrack" rule special dummy rule appeared at the top of the list. This is not an actual rule, it is for visual information showing that some of the traffic is traveling FastPath and will not reach other firewall rules.

Note

FastTrack can process packets only in the main routing table so it is the system administrator duty to not FastTrack connections that are going through non-main routing table (thus connections that are processed with mangle action=mark-routing rules). Otherwise packets might be misrouted though the main routing table.

These rules appear as soon as there is at least one fast-tracked connection tracking entry and will disappear after the last fast-tracked connection times out in the connection table.

Note

The connection is FastTracked until a connection is closed, timed out or the router is rebooted.

Configuration example: excluding specific host, from being Fast-Tracked

Code Block
languageros
/ip firewall filter
add action=accept chain=forward connection-state=established,related src-address=192.168.88.111
add action=accept chain=forward connection-state=established,related dst-address=192.168.88.111
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=no
add action=accept chain=forward connection-state=established,related

In this example, we exclude host 192.168.88.111, from being Fast-tracked, by first accepting it with the firewall rule, both for source and destination. The idea is - not allowing the traffic to reach the FastTrack action.

Note: the "exclusion" rules, must be placed before fasttrack filters, order is important

...

.

Requirements

IPv4 FastTrack is active if the following conditions are met:

  • no mesh, metarouter interface configuration;
  • sniffer, torch, and traffic generator are not running;
  • "/tool mac-scan" is not actively used;
  • "/tool ip-scan" is not actively used;
  • FastPath and Route cache is are enabled under IP/Settings (route cache condition does not apply to RouterOS v7 or newer);
  • bridge FastPath is enabled if a connection is going over the bridge interface;


Packet flow for the visually impaired

...