Understanding Packet Flow

More advanced firewall setups or complicated tasks, such as traffic prioritization, routing policies, where it is necessary to utilize more than one RouterOS facility, requires knowledge: How these facilities work together? What happens when and why?

RouterOS packet flow diagram and flow examples will try to answers these questions.

It would be very complicated to represent what is going on with the packet in one diagram, therefore a packet flow diagram is divided into three parts:

• overall diagram;
• detailed bridging, routing, and mpls flow diagram;
• diagram that shows what facilities and in what order is included in prerouting, input, forward, output and postrouting.

Let's look at the overall diagram. It looks complicated at first, but after we go through the diagram with examples it will become much clearer.

There are 4 boxes in the center of the diagram: Bridging, routing, mpls decisions, and local route processes. So for example, if the packet needs to be routed over the router, a packet will flow as illustrated in the image below. Without looking deeper into each facility, packet enters in-interface, the router determines that it is IP traffic and needs to be routed, packet goes through all routing processes and exits out-interface.

Let's take a look at another example that will illustrate what happens if the packet's destination is a router. For example, in-interface receives ICMP (ping) packet, its destination is the router itself, so packet will go for local-in processing. After packet is processed ICMP (ping) reply is generated inside the router (Local-out processing) and will be sent out over out-interface.

Simple explanation of each box before we go further with examples:

• physical in-interface - starting point of the packet received by the router;
• logical in-interface - starting point of the decapsulated packet (from tunnels, IPsec, etc);
• local in - last point of packet destined to router itself;
• interface HTB - interface queue;
• physical out-interface - last point of the packet before it is actually sent out;
• logical out-interface - last point of the packet before encapsulation (to tunnels);
• local out - starting point of packet generated by the router;

Now it is time to take a deeper look at what is happening inside bridging, mpls, and routing flows.

Simple explanation of each box before we go further with examples:

• routing decision - go through routes in the routing table to find a match for the destination IP address of the packet. When match is found - packet will be processed further, in case of no match - packet will be discarded.;
• mpls decision - what to do with the packet based on MPLS forwarding tables;
• bridging decision - bridge goes through the MAC address table in order to find a match for destination MAC address of the packet. When match is found - packet will be processed further, in case of no match - multiple copies of packet will be created and packet will be broadcast (sent out via all bridge ports).
• use-ip-firewall - whether use-ip-firewall option is enabled in bridge settings;
• ipsec-policy - whether packet matches any of configured ipsec policies;

Each of prerouting, input, forward, output and postrouting blocks contain even more facilities, which are illustrated in the third part of the packet flow diagram:

Simple explanation of each box before we go further with examples:

• Hotspot-in - allow to capture traffic which otherwise would be discarded by connection tracking - this way our Hotspot feature is able to provide connectivity even if networks settings are incomplete mess ;
• RAW Prerouting - RAW table prerouting chain;
• Connection tracking - packet is processed by connection tracking;
• Mangle Prerouting - Mangle prerouting chain;
• Mangle Input - Mangle input chain;
• Filter Input - Firewall filter input chain;
• HTB Global - ;
• Simple Queues - ;
• TTL - indicates exact place where Time To Live (TTL) of the routed packet is reduced by 1 if TTL becomes 0 packet will be discarded;
• Mangle Forward - Mangle forward chain;
• Filter Forward - Mangle forward chain;
• Accounting - ;
• RAW Output - RAW table output chain;
• Mangle Output - Mangle output chain;
• Filter Output - Firewall filter output chain;
• Routing Adjustment - this is a workaround that allows to set-up policy routing in mangle chain output ;
• Mangle Postrouting - Mangle postrouting chain;
• Src Nat - Network_Address_Translation srcnat chain;
• Dst Nat - Network_Address_Translation dstnat chain;
• Hotspot-out - undo all that was done by hotspot-in for the packets that are going back to client;

Flow of Routed Packet

Now, let's take our first example where packet gets routed over the router and look deeper through what facilities packet goes:

We already learned that packet goes into in-interface, the router determines that it is IP packet and need to be routed, and here starts the complicated process:

Or when packet is originated from the router (routing output):

Flow of Bridged Packet

Flow of MPLS Packet

Logical Interfaces

So far we looked at examples when in or out interfaces are actual physical interfaces (Ethernet, wireless), but how packets will flow if router receives tunnel encapsulated packet?

Lets assume that there is IPIP packet coming in the router. Since it is regular ipv4 packet it will be processed through all routing related facilities ( until "J" in the diagram). Then router will look if packet need to be decapsualted., in this case it is IPIP packet so "yes" send packet to decapsulation. After that packet will go another loop through all the facilities but this time as decapsulated IPv4 packet.

It is very important because packet actually travels through firewall twice, so if there is a strict firewall, then there should be accept rules for IPIP encapsulated packet as well as decapsulated ip packet.

IPSec Policies

Lets take a look at another tunnel type - IPSec. This type of VPN does not have logical interfaces, but is processed in similar manner.

Instead of logical interfaces packets are processed through ipsec policies. After routing decision (2) and input firewall processing (3), router tries to match source and destination to ipsec policy. When policy matches the packet it is sent to decryption (5). After decryption packet enters prerouting processing again (6) and starts another processing loop, but now with decapsulated packet.

The same process is with encapsulation but in reverse order. First ip packet gets processed through facilities, then matched against ipsec policies (5), encapsulated (6) and then sent to processing on second loop (7-10).

Fast Path

From what we learned so far, it is quite obvious that such packet processing takes a lot of CPU resources. To fast things up FastPath was introduced since the first RouterOS v6. What it does is it skips processing in the Linux kernel, basically trade some RouterOS functionality for performance. For FastPath to work, interface driver support and specific configuration conditions are required.

How Fast Path Works

FastPath is an interface driver extension, that allows driver to talk directly to specific RouterOS facility and skipping all others.

Packet can be forwarded by fast path handler only if at least source interface supports fast path. For complete fast forwarding, destination interface support is also required.

Currently RouterOS has following FastPath handlers:

• IPv4
• IPv4 FastTrack
• Traffic Generator
• MPLS
• Bridge

IPv4 FastPath handler is used if following conditions are met:

• firewall rules are not configured;
• firewall address lists are not configured;
• simple and queue trees with parent=global are not configured;
• no mesh, metarouter interface configuration;
• sniffer, torch and traffic generator is not running;
• connection tracking is not active;
• ip accounting is disabled;
• VRFs are not configured (/ip route vrf is empty);
• Hotspot is not used (/ip hotspot has no interfaces);
• IPSec policies are not configured;
• /tool mac-scan is not actively used;
• /tool ip-scan is not actively used.

Traffic Generator and MPLS automatically uses FastPath if interface support this feature. Currently MPLS fast-path applies only to MPLS switched traffic (frames that enter router as MPLS and must leave router as MPLS) - MPLS ingress and egress (including VPLS tunnel endpoints that do VPLS encap/decap) will operate as before.

Bridge handler is used if following conditions are met:

• there are no bridge firewall rules;
• use-ip-firewall is disabled;
• no mesh, MetaRouter interface configuration;
• sniffer, torch and traffic generator is not running.

Interfaces that support FastPath:

EoIP, Gre, IPIP and L2TP interfaces have per interface setting allow-fast-path. Allowing fast path on these interfaces have side effect of bypassing firewall, connection tracking, simple queues, queue tree with parent=global, ip accounting, ipsec, hotspot universal client, vrf assignment for encapsulated packets that go trough fastpath. Also packet fragments cannot be received in FastPath.

Only interface queue that guarantees FastPath is only-hardware-queue. If you need interface queue other than hardware then packet will not go fully FastPath, but there is not a big impact on performance, as "interface queue" is the last step in the packet flow.

Packet may go Half-FastPath by switching from FastPath to SlowPath, but not the other way around. So, for example, if receiving interface has FastPath suport, but out interface does not, then router will process packet by FastPath handlers as far as it can and then proceed with SlowPath. If receiving interface does not support FastPath but out interface does, packet will be processed by SlowPath all the way through the router.

Fasttrack

Fasttrack can be decoded as Fast Path + Connection Tracking. It allows to mark connections as "fasttracked", after marking packets that belong to fastracked connection will be sent fast path way. Connection table entry for such connection now will have fasttracked flag.

To mark connection as fasttracked new action was implemented fasttrack-connection for firewall filter and mangle. Currently only IPv4 TCP and UDP connections can be fasttracked and to maintain connection tracking entries some random packets will still be sent to slow path. This must be taken in consideration when designing firewalls with enabled fasttrack.

FasTrack handler also supports source and destination NAT, so special exception for NATed connections are not required.

Easiest way to start using this feature on home routers is to enable fasttrack for all established,related connections:

/ip firewall filter add chain=forward action=fasttrack-connection \
  connection-state=established,related comment = "fasttrack established/related"
/ip firewall filter add chain=forward action=accept \
  connection-state=established,related comment = "accept established/related"

Notice that first rule marks established/related connections as fasttracked, second rule is still required to accept packets belonging to those connections. The reason for this is that, as it was mentioned earlier, some random packets from fasttracked connections are still sent the slow path way and only UDP and TCP are fasttracked, but we still want to accept packets for other protocols.

After adding fasttrack rule special dummy rule appeared at the top of the list. This is not an actual rule, it is for visual information showing that some of the traffic is travelling FastPath and will not reach other firewall rules.

These rules appear as soon as there are at least one "fasttracked" connection tracking entry and will disappear after the last "fasttracked" connection times out in connection table.