Description

RouterOS allows to create multiple Virtual Routing and Forwarding instances on a single router. This is useful for BGP based MPLS VPNs. Unlike BGP VPLS, which is OSI Layer 2 technology, BGP VRF VPNs work in Layer 3 and as such exchange IP prefixes between routers. VRFs solve the problem of overlapping IP prefixes, and provide the required privacy (via separated routing for different VPNs).

It is possible to set up vrf-lite setups or use multi-protocol BGP with VPNv4 address family to distribute routes from VRF routing tables - not only to other routers, but also to different routing tables in the router itself.

Configuration

VRF table is created in /ip vrf menu. After the VRF config is created routing table mapping is added (dynamic table with the same name is created). Each active VRF will always have a mapped routing table.

[admin@arm-bgp] /ip/vrf> print 
Flags: X - disabled; * - builtin 
 0  * name="main" interfaces=all  

[admin@arm-bgp] /routing/table> print 
Flags: D - dynamic; X - disabled, I - invalid; U - used 
 0 D   name="main" fib 

Note that the order of the added VRFs is significant. To properly match which interface will belong to the VRF care must be taken to place VRFs in the correct order (matching is done starting from the top entry, just like firewall rules).

Let's look at the following example:

[admin@arm-bgp] /ip/vrf> print 
Flags: X - disabled; * - builtin 
 0  * name="main" interfaces=all 
 1    name="myVrf" interfaces=lo_vrf  

Since the first entry is matching all the interfaces, the second VRF will not have any interfaces added. To fix the problem order of the entries must be changed.

[admin@arm-bgp] /ip/vrf> move 1 0
[admin@arm-bgp] /ip/vrf> print 
Flags: X - disabled; * - builtin 
 0    name="myVrf" interfaces=lo_vrf  
 1  * name="main" interfaces=all   

Connected routes from the interfaces assigned to the VRF will be installed in the right routing table automatically.

When the interface is assigned to the VRF as well as connected routes it does not mean that RouterOS services will magically know which VRF to use just by specifying the IP address in the configuration. Each service needs VRF support to be added and explicit configuration. Whether the service has VRF support and has VRF configuration options refer to appropriate service documentation.

For example, let's make an SSH service to listen for connections on the interfaces belonging to the VRF:

[admin@arm-bgp] /ip/service> set ssh vrf=myVrf 
[admin@arm-bgp] /ip/service> print 
Flags: X, I - INVALID
Columns: NAME, PORT, CERTIFICATE, VRF
#   NAME     PORT  CERTIFICATE  VRF     
0   telnet     23               main    
1   ftp        21                       
2   www        80               main    
3   ssh        22               myVrf
4 X www-ssl   443  none         main    
5   api      8728               main    
6   winbox   8291               main    
7   api-ssl  8729  none         main  

Adding routes to the VRF is as simple as specifying the routing-table parameter when adding the route and specifying in which routing table to resolve the gateway by specifying @name after the gateway IP:

/ip route add dst-address=192.168.1.0/24 gateway=172.16.1.1@myVrf routing-table=myVrf

Traffic leaking between VRFs is possible if the gateway is explicitly set to be resolved in another VRF, for example:

# add route in the myVrf, but resolve the gateway in the main table
/ip route add dst-address=192.168.1.0/24 gateway=172.16.1.1@main routing-table=myVrf

# add route in the main table, but resolve the gateway in the myVrf
/ip route add dst-address=192.168.1.0/24 gateway=172.16.1.1@myVrf

If the gateway configuration does not have an explicitly configured table to be resolved in, then it is considered, that gateway should be resolved in the "main" table.

Supported features

Different services can be placed in specific VRF on which the service is listening for incoming or creating outgoing connections. By default, all services are using the main table, but it can be changed with a separate vrf parameter or by specifying the VRF name separated by "@" at the end of the IP address.

Below is the list of supported services.

Feature

Support

Comment

BGP+
/routing bgp template
add name=bgp-template1 vrf=vrf1
/routing bgp vpls
add name=bgp-vpls1 site-id=10 vrf=vrf1
/routing bgp vpn
add label-allocation-policy=per-vrf vrf=vrf1
E-mail+
/tool e-mail
set address=192.168.88.1 vrf=vrf1
IP Services+

VRF is supported for telnet, www, ssh, www-ssl, api, winbox, api-ssl services. The ftp service does not support changing the VRF.

/ip service
set telnet vrf=vrf1
L2TP Client+
/interface l2tp-client
add connect-to=192.168.88.1 name=l2tp-out1 user=l2tp-client vrf=vrf1
MPLS+
/mpls ldp
add vrf=vrf1
Netwatch

+

/tool netwatch
add host=192.168.88.1@vrf1
NTP

+

/system ntp client
set vrf=vrf1
/system ntp server
set vrf=vrf1
OSPF

+

/routing ospf instance
add disabled=no name=ospf-instance-1 vrf=vrf1
ping

+

/ping 192.168.88.1 vrf=vrf1
RADIUS

+

/radius add address=192.168.88.1@vrf1
/radius incoming set vrf=vrf1
RIP

+

/routing rip instance
add name=rip-instance-1 vrf=vrf1
RPKI

+

/routing rpki
add vrf=vrf1
SNMP

+

/snmp
set vrf=vrf1

Examples

Simple VRF-Lite setup

Let's consider a setup where we need two customer VRFs that require access to the internet:

/ip address
add address=172.16.1.2/24 interface=public
add address=192.168.1.1/24 interface=ether1
add address=192.168.2.1/24 interface=ether2

/ip route
add gateway=172.16.1.1

# add VRF configuration
/ip vrf
add name=cust_a interface=ether1 place-before 0
add name=cust_b interface=ether2 place-before 0

# add vrf routes
/ip route
add gateway=172.16.1.1@main routing-table=cust_a
add gateway=172.16.1.1@main routing-table=cust_b

# masquerade local source
/ip firewall nat add chain=srcnat out-interface=public action=masquerade

It might be necessary to ensure that packets coming in the "public" interface can actually reach the correct VRF. 
This can be solved by marking new connections originated by the VRF customers and steering the traffic by routing marks of incoming packets on the "public" interface.

# mark new customer connections
/ip firewall mangle 
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=\
    cust_a_conn src-address=192.168.1.0/24 passthrough=no
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=\
    cust_b_conn src-address=192.168.2.0/24 passthrough=no 

# mark routing
/ip firewall mangle  
add action=mark-routing chain=prerouting connection-mark=cust_a_conn \
    in-interface=public new-routing-mark=cust_a
add action=mark-routing chain=prerouting connection-mark=cust_b_conn \
    in-interface=public new-routing-mark=cust_b

Static inter-VRF routes

In general, it is recommended that all routes between VRF should be exchanged using BGP local import and export functionality. If that is not enough, static routes can be used to achieve this so-called route leaking.

There are two ways to install a route that has a gateway in a different routing table than the route itself.

The first way is to explicitly specify the routing table in the gateway field when adding a route. This is only possible when leaking a route and gateway from the "main" routing table to a different routing table (VRF). Example:

# add route to 5.5.5.0/24 in 'vrf1' routing table with gateway in the main routing table 
add dst-address=5.5.5.0/24 gateway=10.3.0.1@main routing-table=vrf1


The second way is to explicitly specify the interface in the gateway field. The interface specified can belong to a VRF instance. Example:

# add route to 5.5.5.0/24 in the main routing table with gateway at 'ether2' VRF interface 
add dst-address=5.5.5.0/24 gateway=10.3.0.1%ether2 routing-table=main 
# add route to 5.5.5.0/24 in the main routing table with 'ptp-link-1' VRF interface as gateway 
add dst-address=5.5.5.0/24 gateway=ptp-link-1 routing-table=main


As can be observed, there are two variations possible - to specify gateway as ip_address%interface or to simply specify interface. The first should be used for broadcast interfaces in most cases. The second should be used for point-to-point interfaces, and also for broadcast interfaces, if the route is a connected route in some VRF. For example, if you have address 1.2.3.4/24 on interface ether2 that is put in a VRF, there will be connected route to 1.2.3.0/24 in that VRF's routing table. It is acceptable to add static route 1.2.3.0/24 in a different routing table with interface-only gateway, even though ether2 is a broadcast interface:


add dst-address=1.2.3.0/24 gateway=ether2 routing-table=main

The simplest MPLS VPN setup

In this example, a rudimentary MPLS backbone (consisting of two Provider Edge (PE) routers PE1 and PE2) is created and configured to forward traffic between Customer Edge (CE) routers CE1 and CE2 routers that belong to cust-one VPN.

CE1 Router

/ip address add address=10.1.1.1/24 interface=ether1 
# use static routing 
/ip route add dst-address=10.3.3.0/24 gateway=10.1.1.2


CE2 Router

/ip address add address=10.3.3.4/24 interface=ether1 
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3


PE1 Router

/interface bridge add name=lobridge 
/ip address add address=10.1.1.2/24 interface=ether1 
/ip address add address=10.2.2.2/24 interface=ether2 
/ip address add address=10.5.5.2/32 interface=lobridge 
/ip vrf add name=cust-one interfaces=ether1 
/mpls ldp add enabled=yes transport-address=10.5.5.2 
/mpls ldp interface add interface=ether2 
/routing bgp template set default as=65000 

/routing bgp vpn 
add vrf=cust-one redistribute=connected \
  route-distinguisher=1.1.1.1:111 \
  import-route-targets=1.1.1.1:111 \
  export-route-targets=1.1.1.1:111 \
  label-allocation-policy=per-vrf
/routing bgp connection 
add template=default remote.address=10.5.5.3 address-families=vpnv4 local.address=10.5.5.2

# add route to the remote BGP peer's loopback address 
/ip route add dst-address=10.5.5.3/32 gateway=10.2.2.3


PE2 Router (Cisco)


ip vrf cust-one
rd 1.1.1.1:111
route-target export 1.1.1.1:111
route-target import 1.1.1.1:111
exit

interface Loopback0
ip address 10.5.5.3 255.255.255.255

mpls ldp router-id Loopback0 force
mpls label protocol ldp

interface FastEthernet0/0
ip address 10.2.2.3 255.255.255.0
mpls ip

interface FastEthernet1/0
ip vrf forwarding cust-one
ip address 10.3.3.3 255.255.255.0

router bgp 65000
neighbor 10.5.5.2 remote-as 65000
neighbor 10.5.5.2 update-source Loopback0
address-family vpnv4
neighbor 10.5.5.2 activate
neighbor 10.5.5.2 send-community both
exit-address-family
address-family ipv4 vrf cust-one
redistribute connected
exit-address-family

ip route 10.5.5.2 255.255.255.255 10.2.2.2

Results

Check that VPNv4 route redistribution is working:


...

Check that the 10.3.3.0 is installed in IP routes, in cust-one route table:

[admin@PE1] > /ip route print 
Flags: X - disabled, A - active, D - dynamic,
 C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE 
0 ADC 10.1.1.0/24 10.1.1.2 ether1 0 
1 ADb 10.3.3.0/24 10.5.5.3 recursi... 20 
2 ADC 10.2.2.0/24 10.2.2.2 ether2 0 
3 ADC 10.5.5.2/32 10.5.5.2 lobridge 0 
4 A S 10.5.5.3/32 10.2.2.3 reachab... 1


Let's take closer look at IP routes in cust-one VRF. The 10.1.1.0/24 IP prefix is a connected route that belongs to an interface that was configured to belong to cust-one VRF. The 10.3.3.0/24 IP prefix was advertised via BGP as VPNv4 route from PE2 and is imported in this VRF routing table, because our configured import-route-targets matched the BGP extended communities attribute it was advertised with.

[admin@PE1] /routing/route> print detail where routing-table=cust-one 
...


The same for Cisco:

PE2#show ip bgp vpnv4 all 
BGP table version is 5, local router ID is 10.5.5.3 
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
r RIB-failure, S Stale 
Origin codes: i - IGP, e - EGP, ? - incomplete 
Network Next Hop Metric LocPrf Weight Path 
Route Distinguisher: 1.1.1.1:111 (default for vrf cust-one) 
*>i10.1.1.0/24 10.5.5.2 100 0 ? 
*> 10.3.3.0/24 0.0.0.0 0 32768 ? 

PE2#show ip route vrf cust-one 
Routing Table: cust-one 
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP 
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
E1 - OSPF external type 1, E2 - OSPF external type 2 
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 
ia - IS-IS inter area, * - candidate default, U - per-user static route 
o - ODR, P - periodic downloaded static route 

Gateway of last resort is not set 
10.0.0.0/24 is subnetted, 1 subnets
B 10.1.1.0 [200/0] via 10.5.5.2, 00:05:33 
10.0.0.0/24 is subnetted, 1 subnets 
C 10.3.3.0 is directly connected, FastEthernet1/0


You should be able to ping from CE1 to CE2 and vice versa.

[admin@CE1] > /ping 10.3.3.4 
10.3.3.4 64 byte ping: ttl=62 time=18 ms 
10.3.3.4 64 byte ping: ttl=62 time=13 ms 
10.3.3.4 64 byte ping: ttl=62 time=13 ms 
10.3.3.4 64 byte ping: ttl=62 time=14 ms 
4 packets transmitted, 4 packets received, 0% packet loss 
round-trip min/avg/max = 13/14.5/18 ms


A more complicated setup (changes only)

As opposed to the simplest setup, in this example we have two customers: cust-one and cust-two.

We configure two VPNs for then, cust-one and cust-two respectively, and exchange all routes between them. (This is also called "route leaking").

Note that this could be not the most typical setup, because routes are usually not exchanged between different customers. In contrast, by default it should not be possible to gain access from one VRF site to a different VRF site in another VPN. (This is the "Private" aspect of VPNs.) Separate routing is a way to provide privacy; and it is also required to solve the problem of overlapping IP network prefixes. Route exchange is in direct conflict with these two requirement but may sometimes be needed (e.g. temp. solution when two customers are migrating to single network infrastructure).

CE1 Router, cust-one

/ip route add dst-address=10.4.4.0/24 gateway=10.1.1.2


CE2 Router, cust-one


/ip route add dst-address=10.4.4.0/24 gateway=10.3.3.3

CE1 Router,cust-two

/ip address add address=10.4.4.5 interface=ether1 
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3 
/ip route add dst-address=10.3.3.0/24 gateway=10.3.3.3


PE1 Router

# replace the old BGP VPN with this:
/routing bgp vpn 
add vrf=cust-one redistribute=connected \
  route-distinguisher=1.1.1.1:111 \
  import-route-targets=1.1.1.1:111,2.2.2.2:222  \
  export-route-targets=1.1.1.1:111

PE2 Router (Cisco)

ip vrf cust-one 
rd 1.1.1.1:111 
route-target export 1.1.1.1:111 
route-target import 1.1.1.1:111 
route-target import 2.2.2.2:222 
exit 

ip vrf cust-two 
rd 2.2.2.2:222 
route-target export 2.2.2.2:222 
route-target import 1.1.1.1:111 
route-target import 2.2.2.2:222 
exit 

interface FastEthernet2/0 
ip vrf forwarding cust-two 
ip address 10.4.4.3 255.255.255.0 

router bgp 65000 
address-family ipv4 vrf cust-two 
redistribute connected 
exit-address-family


Variation: replace the Cisco with another MT

PE2 Mikrotik config


/interface bridge add name=lobridge
/ip address
add address=10.2.2.3/24 interface=ether1
add address=10.3.3.3/24 interface=ether2
add address=10.4.4.3/24 interface=ether3
add address=10.5.5.3/32 interface=lobridge
/ip vrf
add name=cust-one interfaces=ether2
add name=cust-two interfaces=ether3
/mpls ldp add enabled=yes transport-address=10.5.5.3
/mpls ldp interface add interface=ether1

/routing bgp template set default as=65000 
/routing bgp vpn 
add vrf=cust-one redistribute=connected \
  route-distinguisher=1.1.1.1:111 \
  import-route-targets=1.1.1.1:111,2.2.2.2:222 \
  export-route-targets=1.1.1.1:111 \
add vrf=cust-two redistribute=connected \
  route-distinguisher=2.2.2.2:222 \
  import-route-targets=1.1.1.1:111,2.2.2.2:222 \
  export-route-targets=2.2.2.2:222 \

/routing bgp connection 
add template=default remote.address=10.5.5.2 address-families=vpnv4 local.address=10.5.5.3

# add route to the remote BGP peer's loopback address
/ip route add dst-address=10.5.5.2/32 gateway=10.2.2.2

Results

The output of /ip route print now is interesting enough to deserve detailed observation.


[admin@PE2] /ip route> print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 10.1.1.0/24 10.5.5.2 recurs... 20
1 ADC 10.3.3.0/24 10.3.3.3 ether2 0
2 ADb 10.4.4.0/24 20
3 ADb 10.1.1.0/24 10.5.5.2 recurs... 20
4 ADb 10.3.3.0/24 20
5 ADC 10.4.4.0/24 10.4.4.3 ether3 0
6 ADC 10.2.2.0/24 10.2.2.3 ether1 0
7 A S 10.5.5.2/32 10.2.2.2 reacha... 1
8 ADC 10.5.5.3/32 10.5.5.3 lobridge 0

The route 10.1.1.0/24 was received from remote BGP peer and is installed in both VRF routing tables.

The routes 10.3.3.0/24 and 10.4.4.0/24 are also installed in both VRF routing tables. Each is as connected route in one table and as BGP route in another table. This has nothing to do with their being advertised via BGP. They are simply being "advertised" to local VPNv4 route table and locally reimported after that. Import and export route-targets determine in which tables they will end up.

This can be deduced from its attributes - they don't have the usual BGP properties. (Route 10.4.4.0/24.)


[admin@PE2] /routing/route> print detail where routing-table=cust-one
...

References

RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)

MPLS Fundamentals, chapter 7, Luc De Ghein, Cisco Press 2006