...
| Note |
|---|
If the gateway configuration does not have an explicitly configured table to be resolved in, then it is considered, that gateway should be resolved in the "main" table. |
Supported features
Different services can be placed in specific VRF on which the service is listening for incoming or creating outgoing connections. By default, all services are using the main table, but it can be changed with a separate vrf parameter or by specifying the VRF name separated by "@" at the end of the IP address.
Below is the list of supported services.
...
Feature
...
Support
...
Comment
VRF interfaces in firewall
Started from version 7.14 when interfaces are added in VRF - virtual VRF interface is created automatically. If it is needed to match traffic which belongs to VRF interface, VRF virtual interface should be used in firewall filters, for example:
| Code Block |
|---|
|
/ip vrf add interfaces=ether5 name=vrf5
/ip firewall filter add chain=input in-interface=vrf5 action=accept |
If there are several interfaces in one VRF but it is needed to match only one of these interfaces - marks should be used. For example:
...
...
...
...
...
interface=ether15,ether16 vrf= |
...
...
...
...
mangle
add action=mark-connection chain=prerouting connection-state=new in-interface=ether15 new-connection-mark=input_allow passthrough=yes
/ip firewall filter
add action=accept chain=input connection-mark=input_allow |
Supported features
Different services can be placed in specific VRF on which the service is listening for incoming or creating outgoing connections. By default, all services are using the main table, but it can be changed with a separate vrf parameter or by specifying the VRF name separated by "@" at the end of the IP address.
Below is the list of supported services.
Feature | Support | Comment |
|---|
| BGP | name=bgp-vpls1 site-id=10 vrf=vrf1
/routing bgp vpn
add label-allocation-policy=per-vrf vrf=vrf1E-mail | + | | Code Block |
|---|
|
/tool e-mail
set address=192.168.88.1 vrf=vrf1 |
IP Services | + | VRF is supported for telnet, www, ssh, www-ssl, api, winbox, api-ssl services. The ftp service does not support changing the VRF. | Code Block |
|---|
| /ip service
set telnet vrf=vrf1 |
L2TP Client | + |
| Code Block |
|---|
| /interface l2tp-clientrouting bgp template
add connect-to=192.168.88.1@vrf1name=bgp-template1 vrf=vrf1
/routing bgp vpls
add name=l2tpbgp-out1vpls1 user=l2tp-client site-id=10 vrf=vrf1
/routing bgp vpn
add label-allocation-policy=per-vrf vrf=vrf1 |
|
| E-mail | MPLS | + | | Code Block |
|---|
| /mpls ldp
add vrf=vrf1 |
Netwatch | + |
| Code Block |
|---|
| /tool netwatche-mail
addset hostaddress=192.168.88.1@vrf11 vrf=vrf1 |
|
| IP Services | + | VRF is supported for telnet, www, ssh, www-ssl, api, winbox, api-ssl services. The ftp service does not support changing the VRF. | Code Block |
|---|
| /ip service
set telnet vrf=vrf1 |
|
| L2TP Client | NTP | + | | Code Block |
|---|
| /system ntp client
set vrf=vrf1
/system ntp server
set vrf=vrf1 |
OSPF | + |
| Code Block |
|---|
| /routing ospf instanceinterface l2tp-client
add disabled=noconnect-to=192.168.88.1@vrf1 name=ospf-instance-1 vrf=vrf1l2tp-out1 user=l2tp-client |
|
| MPLSping | + |
| Code Block |
|---|
| /ping 192.168.88.1mpls ldp
add vrf=vrf1 |
|
| RADIUSNetwatch | + |
| Code Block |
|---|
| /radiustool netwatch
add addresshost=192.168.88.1@vrf1 |
|
| NTP | + |
| Code Block |
|---|
| /system ntp client
/radius incoming set vrf=vrf1
/system ntp server
set vrf=vrf1 |
|
| RIPOSPF | + |
| Code Block |
|---|
| /routing ripospf instance
add disabled=no name=ripospf-instance-1 vrf=vrf1 |
|
| RPKIping | + |
| Code Block |
|---|
| /routing rpki
addping 192.168.88.1 vrf=vrf1 |
|
| SNMPRADIUS | + |
| Code Block |
|---|
| /snmp
radius add address=192.168.88.1@vrf1
/radius incoming set vrf=vrf1 |
|
| EoIPRIP | + |
| Code Block |
|---|
| /interfacerouting rip eoipinstance
add remote-address=192.168.1.1@vrf1name=rip-instance-1 vrf=vrf1 |
|
| RPKIIPIP | + |
| Code Block |
|---|
| /routing rpki
add vrf=vrf1 |
|
| SNMP | + |
| Code Block |
|---|
| /snmp
set vrf=vrf1 |
|
| EoIP | + |
| Code Block |
|---|
| /interface eoipinterface ipip
add remote-address=192.168.1.1@vrf1 |
|
| IPIP | + |
| Code Block |
|---|
| /interface ipip
add remote-address=192.168.1.1@vrf1 |
|
| GRE | + |
| Code Block |
|---|
| /interface gre
add remote-address=192.168.1.1@vrf1 |
|
| SSTP-client | + |
| Code Block |
|---|
| /interface sstp-client
add connect-to=192.168.1.1@vrf1 |
|
| OVPN-client | + |
| Code Block |
|---|
| /interface ovpn-client
add connect-to=192.168.1.1@vrf1 |
|
| L2TP-ether | + |
| Code Block |
|---|
| /interface l2tp-ether
add connect-to=192.168.2.2@vrf |
|
| VXLAN | + |
| Code Block |
|---|
| /interface vxlan
add vni=10 vrf=vrf1 |
|
| Fetchfetch | + |
| | Code Block |
|---|
| /tool/fetch
address=10.155.28.236@vrf1 mode=ftp src-path=my_file.pcap user=admin password="" |
| DNS | +
Starting from RouterOS v7.15
Examples
Simple VRF-Lite setup
Let's consider a setup where we need two customer VRFs that require access to the internet:
...
+
Starting from RouterOS v7.15
...
| Code Block |
|---|
|
/ip dhcp-relay set dhcp-server-vrf=vrf1 |
If dhcp-client is in vrf - special parameter in "ip dhcp-relay" configuration is not neededExamples
Simple VRF-Lite setup
Let's consider a setup where we need two customer VRFs that require access to the internet:
| Code Block |
|---|
|
/ip addressaddress
add address=172.16.1.2/24 interface=public
add address=192.168.1.1/24 interface=ether1
add address=192.168.2.1/24 interface=ether2
/ip route
add gateway=172.16.1.1
# add VRF configuration
/ip vrf
add name=cust_a interface=ether1 place-before 0
add name=cust_b interface=ether2 place-before 0
# add vrf routes
/ip route
add gateway=172.16.1.1@main routing-table=cust_a
add gateway=172.16.1.1@main routing-table=cust_b
# masquerade local source
/ip firewall nat add chain=srcnat out-interface=public action=masquerade |
...
| Code Block |
|---|
|
# mark new customer connections
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=\
cust_a_conn src-address=192.168.1.0/24 passthrough=no
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=\
cust_b_conn src-address=192.168.2.0/24 passthrough=no
# mark routing
/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=cust_a_conn \
in-interface=public new-routing-mark=cust_a
add action=mark-routing chain=prerouting connection-mark=cust_b_conn \
in-interface=public new-routing-mark=cust_b |
Static
...
inter-VRF routes
In general, it is recommended that all routes between VRF should be exchanged using BGP local import and export functionality. If that is not enough, static routes can be used to achieve this so-called route leaking.
There are two ways to install a route that has a gateway in a different routing table than the route itself.
The first way is to explicitly specify the routing table in the gateway field when adding a route. This is only possible when leaking a route and gateway from the "main" routing table to a different routing table (VRF). Example:
| Code Block |
|---|
|
# add route to 5.5.5.0/24 in 'vrf1' routing table with gateway in the main routing table
add dst-address=5.5.5.0/24 gateway=10.3.0.1@main routing-table=vrf1 |
The second way is to explicitly specify the interface in the gateway field. The interface specified can belong to a VRF instance. Example:
| Code Block |
|---|
|
# add route to 5.5.5.0/24 in the main routing table with gateway at 'ether2' VRF interface
add dst-address=5.5.5.0/24 gateway=10.3.0.1%ether2 routing-table=main
# add route to 5.5.5.0/24 in the main routing table with 'ptp-link-1' VRF interface as gateway
add dst-address=5.5.5.0/24 gateway=ptp-link-1 routing-table=main |
As can be observed, there are two variations possible - to specify gateway as ip_address%interface or to simply specify an interface. The first should be used for broadcast interfaces in most cases. The second should be used for point-to-point interfaces, and also for broadcast interfaces, if the route is a connected route in some VRF. For example, if you have an address 1.2.3.4/24 on interface ether2 that is put in a VRF, there will be a connected route to 1.2.3.0/24 in that VRF's routing table. It is acceptable to add a static route 1.2.3.0/24 in a different routing table with an interface-only gateway, even though ether2 is a broadcast interface:
| Code Block |
|---|
|
add dst-address=1.2.3.0/24 gateway=ether2 routing-table=main
|
The simplest MPLS VPN setup
Image Added
In this example, a rudimentary MPLS backbone (consisting of two Provider Edge (PE) routers PE1 and PE2) is created and configured to forward traffic between Customer Edge (CE) routers CE1 and CE2 routers that belong to cust-one VPN.
CE1 Router
| Code Block |
|---|
|
/ip address add address=10.1.1.1/24 interface=ether1
# use static routing
/ip route add dst-address=10.3.3.0/24 gateway=10.1.1.2 |
CE2 Router
| Code Block |
|---|
|
/ip address add address=10.3.3.4/24 interface=ether1
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3 |
PE1 Router
| Code Block |
|---|
|
/interface bridge add name=lobridge
/ip address add address=10.1.1.2/24 interface=ether1
/ip address add address=10.2.2.2/24 interface=ether2
/ip address add address=10.5.5.2/32 interface=lobridge
/ip vrf add name=cust-one interfaces=ether1
/mpls ldp add enabled=yes transport-address=10.5.5.2 lsr-id=10.5.5.2
/mpls ldp interface add interface=ether2
/routing bgp template set default as=65000
/routing bgp vpn
add vrf=cust-one \
route-distinguisher=1.1.1.1:111 \
import.route-targets=1.1.1.1:111 \
import.router-id=cust-one \
export.redistribute=connected \
export.route-targets=1.1.1.1:111 \
label-allocation-policy=per-vrf
/routing bgp connection
add template=default remote.address=10.5.5.3 address-families=vpnv4 local.address=10.5.5.2
# add route to the remote BGP peer's loopback address
/ip route add dst-address=10.5.5.3/32 gateway=10.2.2.3 |
PE2 Router (Cisco)
| Code Block |
|---|
|
ip vrf cust-one
rd 1.1.1.1:111
route-target export 1.1.1.1:111
route-target import 1.1.1.1:111
exit
interface Loopback0
ip address 10.5.5.3 255.255.255.255
mpls ldp router-id Loopback0 force
mpls label protocol ldp
interface FastEthernet0/0
ip address 10.2.2.3 255.255.255.0
mpls ip
interface FastEthernet1/0
ip vrf forwarding cust-one
ip address 10.3.3.3 255.255.255.0
router bgp 65000
neighbor 10.5.5.2 remote-as 65000
neighbor 10.5.5.2 update-source Loopback0
address-family vpnv4
neighbor 10.5.5.2 activate
neighbor 10.5.5.2 send-community both
exit-address-family
address-family ipv4 vrf cust-one
redistribute connected
exit-address-family
ip route 10.5.5.2 255.255.255.255 10.2.2.2
|
Results
Check that VPNv4 route redistribution is working:
| Code Block |
|---|
|
[admin@PE1] /routing/route> print detail where afi="vpn4"
Flags: X - disabled, F - filtered, U - unreachable, A - active;
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - l
dp-mapping, g - slaac, y - bgp-mpls-vpn;
H - hw-offloaded; + - ecmp, B - blackhole
Ab afi=vpn4 contribution=active dst-address=111.16.0.0/24&1.1.1.1:111 routing-table=main label=16
gateway=111.111.111.4 immediate-gw=111.13.0.2%ether9 distance=200 scope=40 target-scope=30
belongs-to="bgp-VPN4-111.111.111.4"
bgp.peer-cache-id=*2C00011 .as-path="65511" .ext-communities=rt:1.1.1.1:111 .local-pref=100
.atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x202427E0
[admin@PE1] /routing/bgp/advertisements> print
0 peer=to-pe2-1 dst=10.1.1.0/24 local-pref=100 origin=2 ext-communities=rt:1.1.1.1:111 atomic-aggregate=yes
|
Check that the 10.3.3.0 is installed in IP routes, in the cust-one route table:
| Code Block |
|---|
|
[admin@PE1] > /ip route print where routing-table="cust-one"
Flags: D - DYNAMIC; A - ACTIVE; c, b, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
# DST-ADDRESS GATEWAY DISTANCE
0 ADC 10.1.1.0/24 ether1@cust-one 0
1 ADb 10.3.3.0/24 10.5.5.3 |
...
Sometimes it is necessary to access directly connected resources from another vrf. In our example setup we have two connected networks each in its own VRF. And we want to allow client1 to be able to access client2.
| Code Block |
|---|
|
+-----------------+
|+-vrf1-+ +-vrf2-+|
client1(*.2)-------||ip *.1| |ip *.1||-------client2(*.2)
(10.11.0.0/24) |+------+ +------+| (10.12.0.0/24)
+-----------------+ |
| Code Block |
|---|
|
/ip address
add address=10.11.0.1/24 interface=sfp-sfpplus1
add address=10.12.0.1/24 interface=sfp-sfpplus2
# add VRF configuration
/ip vrf
add name=vrfTest1 interface=sfp-sfpplus1 place-before 0
add name=vrfTest2 interface=sfp-sfpplus2 place-before 0
|
We can say that connected network is reachable on specific vrf by setting gateway "interface@vrf"
| Code Block |
|---|
|
# add vrf routes
/ip route
add dst-address=10.11.0.0/24 gateway="sfp-sfpplus1@vrfTest1" routing-table=vrfTest2
add dst-address=10.12.0.0/24 gateway="sfp-sfpplus2@vrfTest2" routing-table=vrfTest1
|
Verify routes and reachability:
| Code Block |
|---|
|
[admin@CCR2004_2XS] /ip/route> print detail
Flags: D - dynamic; X - disabled, I - inactive, A - active;
c - connect, s - static, r - rip, b - bgp, o - ospf, i - is-is, d - dhcp, v - vpn, m - modem, y - bgp-mpls-vpn; H - hw-offloaded; + - ecmp
DAc dst-address=111.11.0.0/24 routing-table=vrfTest1 gateway=sfp-sfpplus1@vrfTest1 immediate-gw=sfp-sfpplus1 distance=0 scope=10 suppress-hw-offload=no
local-address=111.11.0.1%sfp-sfpplus1@vrfTest1
1 As dst-address=111.12.0.0/24 routing-table=vrfTest1 pref-src="" gateway=vrfTest2 immediate-gw=vrfTest2 distance=1 scope=30 target-scope=10
suppress-hw-offload=no
2 As dst-address=111.11.0.0/24 routing-table=vrfTest2 pref-src="" gateway=vrfTest1 immediate-gw=vrfTest1 distance=1 scope=30 target-scope=10
suppress-hw-offload=no
DAc dst-address=111.12.0.0/24 routing-table=vrfTest2 gateway=sfp-sfpplus2@vrfTest2 immediate-gw=sfp-sfpplus2 distance=0 scope=10 suppress-hw-offload=no
local-address=111.12.0.1%sfp-sfpplus2@vrfTest2
|
| Code Block |
|---|
|
[admin@cl2] > /ping 111.11.0.2 src-address=111.12.0.2
SEQ HOST SIZE TTL TIME STATUS
0 111.11.0.2 56 64 67us
1 111.11.0.2 56 64 61us
sent=2 received=2 packet-loss=0% min-rtt=61us avg-rtt=64u
|
| Note |
|---|
Keep in mind that trying to leak overlapping networks will not work. |
But now what if we want to access routers local address located in another vrf?
| Code Block |
|---|
|
[admin@cl2] > /ping 111.11.0.1 src-address=111.12.0.2
SEQ HOST SIZE TTL TIME STATUS 20 |
Let's take a closer look at IP routes in cust-one VRF. The 10.1.1.0/24 IP prefix is a connected route that belongs to an interface that was configured to belong to cust-one VRF. The 10.3.3.0/24 IP prefix was advertised via BGP as a VPNv4 route from PE2 and is imported in this VRF routing table, because our configured import-route-targets matched the BGP extended communities attribute it was advertised with.
| Code Block |
|---|
|
[admin@PE1] /routing/route> print detail where routing-table="cust-one"
Flags: X - disabled, F - filtered, U - unreachable, A - active;
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - l
dp-mapping, g - slaac, y - bgp-mpls-vpn;
H - hw-offloaded; + - ecmp, B - blackhole
Ac afi=ip4 contribution=active dst-address=10.1.1.0/24 routing-table=cust-one
gateway=ether1@cust-one immediate-gw=ether1 distance=0 scope=10 belongs-to="connected"
0 111.11.0.1local-address=10.1.1.2%ether1@cust-one
debug.fwp-ptr=0x202420C0
Ay afi=ip4 contribution=active dst-address=10.3.3.0/24 routing-table=cust-one label=16
gateway=10.5.5.3 immediate-gw=10.2.2.3%ether2 distance=20 scope=40 target-scope=30
belongs-to="bgp-mpls-vpn-1-bgp-VPN4-10.5.5.3-import"
bgp.peer-cache-id=*2C00011 .ext-communities=rt:1.1.1.1:111 .local-pref=100
timeout .atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x20242840
[admin@PE1] /routing/route> print detail where afi="vpn4"
Flags: X - disabled, F - filtered, U - unreachable, A - active;
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - l
dp-mapping, g - slaac, y
1 111.11.0.1 - bgp-mpls-vpn;
H - hw-offloaded; + - ecmp, B - blackhole
Ay afi=vpn4 contribution=active dst-address=10.1.1.0/24&1.1.1.1:111 routing-table=main label=19
gateway=ether1@cust-one immediate-gw=ether1 distance=200 scope=40 target-scope=10
belongs-to="bgp-mpls-vpn-1-connected-export"
bgp.ext-communities=rt:1.1.1.1:1111 .atomic-aggregate=no .origin=incomplete
timeout debug.fwp-ptr=0x202426C0
Ab afi=vpn4 contribution=active dst-address=10.3.3.0/24&1.1.1.1:111 routing-table=main label=16
gateway=10.5.5.3 immediate-gw=10.2.2.3%ether2 distance=200 scope=40 target-scope=30
belongs-to="bgp-VPN4-10.5.5.3"
bgp.peer-cache-id=*2C00011 .ext-communities=rt:1.1.1.1:111 .local-pref=100
.atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x202427E0
|
The same for Cisco:
| Code Block |
|---|
|
PE2#show ip bgp vpnv4 all
BGP table version is 5, local router ID is 10.5.5.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
sent=2 received=0 packet-loss=100%
|
Approach with "interface@vrf" gateways works only when router is forwarding packets. To access local vrf addresses we need to route to the vrf interface.
| Code Block |
|---|
|
# add vrf routes
/ip route
add dst-address=10.11.0.0/24 gateway=vrfTest1@vrfTest1 routing-table=vrfTest2
add dst-address=10.12.0.0/24 gateway=vrfTest2@vrfTest2 routing-table=vrfTest1
|
| Code Block |
|---|
|
[admin@cl2] > /ping 111.11.0.1 src-address=111.12.0.2
SEQ HOST SIZE TTL TIME STATUS
0 111.11.0.1 56 64 67us
1 111.11.0.1 56 64 61us
sent=2 received=2 packet-loss=0% min-rtt=61us avg-rtt=64u
|
Dynamic Vrf-Lite route leaking
With large enough setups static route leaking is not sufficient. Let's consider we have the same setup as in static route leaking example plus ipv6 addresses, just for demonstration.
| Code Block |
|---|
|
/ip address
add address=10.11.0.1/24 interface=sfp-sfpplus1
add address=10.12.0.1/24 interface=sfp-sfpplus2
# add VRF configuration
/ip vrf
add name=vrfTest1 interface=sfp-sfpplus1 place-before 0
add name=vrfTest2 interface=sfp-sfpplus2 place-before 0
/ipv6 address
add address=2001:1::1 advertise=no interface=sfp-sfpplus1
add address=2001:2::1 advertise=no interface=sfp-sfpplus2
|
We can use BGP VPN to leak local routes without actually establishing BGP session.
| Code Block |
|---|
|
/routing bgp vpn
add export.redistribute=connected .route-targets=1:1 import.route-targets=1:2 label-allocation-policy=per-vrf name=bgp-mpls-vpn-1 \
route-distinguisher=1.2.3.4:1 vrf=vrfTest1
add export.redistribute=connected .route-targets=1:2 import.route-targets=1:1 label-allocation-policy=per-vrf name=bgp-mpls-vpn-2 \
route-distinguisher=1.2.3.4:1 vrf=vrfTest2 |
| Note |
|---|
Be careful with import/export route targets, if not set up properly local vrf routes from itself will be imported. |
Now we can see that connected routes between VRFs are exchanged
| Code Block |
|---|
|
[admin@CCR2004_2XS] > /routing route print where dst-address in 111.0.0.0/8 && afi=ip4
...
Ac afi=ip4 contribution=active dst-address=111.11.0.0/24 routing-table=vrfTest1 gateway=sfp-sfpplus1@vrfTest1 immediate-gw=sfp-sfpplus1 distance=0 scope=10
belongs-to="connected" local-address=111.11.0.1%sfp-sfpplus1@vrfTest1
debug.fwp-ptr=0x202421E0
Ay afi=ip4 contribution=best-candidate dst-address=111.12.0.0/24 routing-table=vrfTest1 label=17 gateway=vrfTest2@vrfTest2 immediate-gw=sfp-sfpplus2
distance=200 scope=40 target-scope=10 belongs-to="bgp-mpls-vpn-1-bgp-mpls-vpn-2-connected-export-import"
bgp.ext-communities=rt:1:2 .atomic-aggregate=no .origin=incomplete
debug.fwp-ptr=0x202425A0
Ay afi=ip4 contribution=best-candidate dst-address=111.11.0.0/24 routing-table=vrfTest2 label=16 gateway=vrfTest1@vrfTest1 immediate-gw=sfp-sfpplus1
distance=200 scope=40 target-scope=10 belongs-to="bgp-mpls-vpn-2-bgp-mpls-vpn-1-connected-export-import"
bgp.ext-communities=rt:1:1 .atomic-aggregate=no .origin=incomplete
debug.fwp-ptr=0x202424E0
Ac afi=ip4 contribution=active dst-address=111.12.0.0/24 routing-table=vrfTest2 gateway=sfp-sfpplus2@vrfTest2 immediate-gw=sfp-sfpplus2 distance=0 scope=10
belongs-to="connected" local-address=111.12.0.1%sfp-sfpplus2@vrfTest2
debug.fwp-ptr=0x20242240
|
And IPv6 too:
| Code Block |
|---|
|
[admin@CCR2004_2XS] /routing/route> print detail where dst-address in 2001::/8 && afi=ip6
...
Ac afi=ip6 contribution=active dst-address=2001:1::/64 routing-table=vrfTest1 gateway=sfp-sfpplus1@vrfTest1 immediate-gw=sfp-sfpplus1 distance=0 scope=10
belongs-to="connected" local-address=2001:1::1%sfp-sfpplus1@vrfTest1
debug.fwp-ptr=0x20242300
Ay afi=ip6 contribution=active dst-address=2001:2::/64 routing-table=vrfTest1 label=17 gateway=vrfTest2@vrfTest2 immediate-gw=sfp-sfpplus2 distance=200
scope=40 target-scope=10 belongs-to="bgp-mpls-vpn-1-bgp-mpls-vpn-2-connected-export-import"
bgp.ext-communities=rt:1:2 .atomic-aggregate=no .origin=incomplete
debug.fwp-ptr=0x202425A0
Ay afi=ip6 contribution=active dst-address=2001:1::/64 routing-table=vrfTest2 label=16 gateway=vrfTest1@vrfTest1 immediate-gw=sfp-sfpplus1 distance=200
scope=40 target-scope=10 belongs-to="bgp-mpls-vpn-2-bgp-mpls-vpn-1-connected-export-import"
bgp.ext-communities=rt:1:1 .atomic-aggregate=no .origin=incomplete
debug.fwp-ptr=0x202424E0
Ac afi=ip6 contribution=active dst-address=2001:2::/64 routing-table=vrfTest2 gateway=sfp-sfpplus2@vrfTest2 immediate-gw=sfp-sfpplus2 distance=0 scope=10
belongs-to="connected" local-address=2001:2::1%sfp-sfpplus2@vrfTest2
debug.fwp-ptr=0x20242360
|
Dynamic Vrf-Lite route leaking (old workaround)
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:111 (default for vrf cust-one)
*>i10.1.1.0/24 10.5.5.2 100 0 ?
*> 10.3.3.0/24 0.0.0.0 0 32768 ?
PE2#show ip route vrf cust-one
Routing Table: cust-one
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
10.0.0.0/24 is subnetted, 1 subnets
B 10.1.1.0 [200/0] via 10.5.5.2, 00:05:33
10.0.0.0/24 is subnetted, 1 subnets
C 10.3.3.0 is directly connected, FastEthernet1/0 |
You should be able to ping from CE1 to CE2 and vice versa.
| Code Block |
|---|
|
[admin@CE1] > /ping 10.3.3.4
10.3.3.4 64 byte ping: ttl=62 time=18 ms
10.3.3.4 64 byte ping: ttl=62 time=13 ms
10.3.3.4 64 byte ping: ttl=62 time=13 ms
10.3.3.4 64 byte ping: ttl=62 time=14 ms
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 13/14.5/18 ms |
A more complicated setup (changes only)
Image Added
As opposed to the simplest setup, in this example, we have two customers: cust-one and cust-two.
We configure two VPNs for them, cust-one and cust-two respectively, and exchange all routes between them. (This is also called "route leaking").
Note that this could be not the most typical setup, because routes are usually not exchanged between different customers. In contrast, by default, it should not be possible to gain access from one VRF site to a different VRF site in another VPN. (This is the "Private" aspect of VPNs.) Separate routing is a way to provide privacy, and it is also required to solve the problem of overlapping IP network prefixes. Route exchange is in direct conflict with these two requirements but may sometimes be needed (e.g. temp. solution when two customers are migrating to a single network infrastructure).
CE1 Router, cust-one
| Code Block |
|---|
|
/ip route add dst-address=10.4.4.0/24 gateway=10.1.1.2 |
CE2 Router, cust-one
| Code Block |
|---|
|
/ip route add dst-address=10.4.4.0/24 gateway=10.3.3.3
|
CE1 Router,cust-two
| Code Block |
|---|
|
/ip address add address=10.4.4.5 interface=ether1
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3
/ip route add dst-address=10.3.3.0/24 gateway=10.3.3.3 |
PE1 Router
| Code Block |
|---|
|
# replace the old BGP VPN with this:
/routing bgp vpn
add vrf=cust-one \
export.redistribute=connected \
route-distinguisher=1.1.1.1:111 \
import.route-targets=1.1.1.1:111,2.2.2.2:222 \
export.route-targets=1.1.1.1:111
|
PE2 Router (Cisco)
| Code Block |
|---|
|
ip vrf cust-one
rd 1.1.1.1:111
route-target export 1.1.1.1:111
route-target import 1.1.1.1:111
route-target import 2.2.2.2:222
exit
ip vrf cust-two
rd 2.2.2.2:222
route-target export 2.2.2.2:222
route-target import 1.1.1.1:111
route-target import 2.2.2.2:222
exit
interface FastEthernet2/0
ip vrf forwarding cust-two
ip address 10.4.4.3 255.255.255.0
router bgp 65000
address-family ipv4 vrf cust-two
redistribute connected
exit-address-family |
Variation: replace the Cisco with another MT
PE2 Mikrotik config
| Code Block |
|---|
|
/interface bridge add name=lobridge
/ip address
add address=10.2.2.3/24 interface=ether1
add address=10.3.3.3/24 interface=ether2
add address=10.4.4.3/24 interface=ether3
add address=10.5.5.3/32 interface=lobridge
/ip vrf
add name=cust-one interfaces=ether2
add name=cust-two interfaces=ether3
/mpls ldp add enabled=yes transport-address=10.5.5.3
/mpls ldp interface add interface=ether1
/routing bgp template set default as=65000
/routing bgp vpn
add vrf=cust-one \
export.redistribute=connected \
route-distinguisher=1.1.1.1:111 \
import.route-targets=1.1.1.1:111,2.2.2.2:222 \
export.route-targets=1.1.1.1:111 \
add vrf=cust-two \
export.redistribute=connected \
route-distinguisher=2.2.2.2:222 \
import.route-targets=1.1.1.1:111,2.2.2.2:222 \
export.route-targets=2.2.2.2:222 \
/routing bgp connection
add template=default remote.address=10.5.5.2 address-families=vpnv4 local.address=10.5.5.3
# add route to the remote BGP peer's loopback address
/ip route add dst-address=10.5.5.2/32 gateway=10.2.2.2
|
Results
The output of /ip route print now is interesting enough to deserve detailed observation.
| Code Block |
|---|
|
[admin@PE2] /ip route> print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 10.1.1.0/24 10.5.5.2 recurs... 20
1 ADC 10.3.3.0/24 10.3.3.3 ether2 0
2 ADb 10.4.4.0/24 20
3 ADb 10.1.1.0/24 10.5.5.2 recurs... 20
4 ADb 10.3.3.0/24 20
5 ADC 10.4.4.0/24 10.4.4.3 ether3 0
6 ADC 10.2.2.0/24 10.2.2.3 ether1 0
7 A S 10.5.5.2/32 10.2.2.2 reacha... 1
8 ADC 10.5.5.3/32 10.5.5.3 lobridge 0
|
The route 10.1.1.0/24 was received from a remote BGP peer and is installed in both VRF routing tables.
The routes 10.3.3.0/24 and 10.4.4.0/24 are also installed in both VRF routing tables. Each is a connected route in one table and a BGP route in another table. This has nothing to do with their being advertised via BGP. They are simply being "advertised" to the local VPNv4 route table and locally reimported after that. Import and export route-targets determine in which tables they will end up.
This can be deduced from its attributes - they don't have the usual BGP properties. (Route 10.4.4.0/24.)
| Code Block |
|---|
|
[admin@PE2] /routing/route> print detail where routing-table=cust-one
...
|
Leaking routes between VRFs
Currently, there is Before ROS v7.14 there were no mechanism to leak routes from one VRF instance to another within the same router.
As a workaround, it was is possible to create a tunnel between two locally configure loopback addresses and assign each tunnel endpoint to its own VRF. Then it is possible to run either dynamic routing protocols or set up static routes to leak between both VRFs.
...
| Code Block |
|---|
|
[admin@rack1_b36_CCR1009] /routing/ospf/neighbor> print
Flags: V - virtual; D - dynamic
0 D instance=i2_custB area=custB_bb address=192.168.1.1 priority=128 router-id=192.168.1.2 dr=192.168.1.1 bdr=192.168.1.2
state="Full" state-changes=6 adjacency=41m28s timeout=33s
1 D instance=i2_custC area=custC_bb address=192.168.1.2 priority=128 router-id=192.168.1.1 dr=192.168.1.1 bdr=192.168.1.2
state="Full" state-changes=6 adjacency=41m28s timeout=33s
[admin@rack1_b36_CCR1009] /ip/route> print where routing-table=custB
Flags: D - DYNAMIC; A - ACTIVE; c, s, o, y - COPY
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAo 172.16.1.0/24 192.168.1.1%ipip-tunnel2@custB 110
DAc 172.16.2.0/24 dummy_custB@custB 0
DAc 192.168.1.0/24 ipip-tunnel2@custB 0
[admin@rack1_b36_CCR1009] > /ip route/print where routing-table=custC
Flags: D - DYNAMIC; A - ACTIVE; c, o, y - COPY
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAc 172.16.1.0/24 dummy_custC@custC 0
DAo 172.16.2.0/24 192.168.1.2%ipip-tunnel1@custC 110
DAc 192.168.1.0/24 ipip-tunnel1@custC 0
|
The simplest MPLS VPN setup
Image Removed
In this example, a rudimentary MPLS backbone (consisting of two Provider Edge (PE) routers PE1 and PE2) is created and configured to forward traffic between Customer Edge (CE) routers CE1 and CE2 routers that belong to cust-one VPN.
CE1 Router
| Code Block |
|---|
|
/ip address add address=10.1.1.1/24 interface=ether1
# use static routing
/ip route add dst-address=10.3.3.0/24 gateway=10.1.1.2 |
CE2 Router
| Code Block |
|---|
|
/ip address add address=10.3.3.4/24 interface=ether1
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3 |
PE1 Router
| Code Block |
|---|
|
/interface bridge add name=lobridge
/ip address add address=10.1.1.2/24 interface=ether1
/ip address add address=10.2.2.2/24 interface=ether2
/ip address add address=10.5.5.2/32 interface=lobridge
/ip vrf add name=cust-one interfaces=ether1
/mpls ldp add enabled=yes transport-address=10.5.5.2 lsr-id=10.5.5.2
/mpls ldp interface add interface=ether2
/routing bgp template set default as=65000
/routing bgp vpn
add vrf=cust-one \
route-distinguisher=1.1.1.1:111 \
import.route-targets=1.1.1.1:111 \
import.router-id=cust-one \
export.redistribute=connected \
export.route-targets=1.1.1.1:111 \
label-allocation-policy=per-vrf
/routing bgp connection
add template=default remote.address=10.5.5.3 address-families=vpnv4 local.address=10.5.5.2
# add route to the remote BGP peer's loopback address
/ip route add dst-address=10.5.5.3/32 gateway=10.2.2.3 |
PE2 Router (Cisco)
| Code Block |
|---|
|
ip vrf cust-one
rd 1.1.1.1:111
route-target export 1.1.1.1:111
route-target import 1.1.1.1:111
exit
interface Loopback0
ip address 10.5.5.3 255.255.255.255
mpls ldp router-id Loopback0 force
mpls label protocol ldp
interface FastEthernet0/0
ip address 10.2.2.3 255.255.255.0
mpls ip
interface FastEthernet1/0
ip vrf forwarding cust-one
ip address 10.3.3.3 255.255.255.0
router bgp 65000
neighbor 10.5.5.2 remote-as 65000
neighbor 10.5.5.2 update-source Loopback0
address-family vpnv4
neighbor 10.5.5.2 activate
neighbor 10.5.5.2 send-community both
exit-address-family
address-family ipv4 vrf cust-one
redistribute connected
exit-address-family
ip route 10.5.5.2 255.255.255.255 10.2.2.2
|
Results
Check that VPNv4 route redistribution is working:
| Code Block |
|---|
|
[admin@PE1] /routing/route> print detail where afi="vpn4"
Flags: X - disabled, F - filtered, U - unreachable, A - active;
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - l
dp-mapping, g - slaac, y - bgp-mpls-vpn;
H - hw-offloaded; + - ecmp, B - blackhole
Ab afi=vpn4 contribution=active dst-address=111.16.0.0/24&1.1.1.1:111 routing-table=main label=16
gateway=111.111.111.4 immediate-gw=111.13.0.2%ether9 distance=200 scope=40 target-scope=30
belongs-to="bgp-VPN4-111.111.111.4"
bgp.peer-cache-id=*2C00011 .as-path="65511" .ext-communities=rt:1.1.1.1:111 .local-pref=100
.atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x202427E0
[admin@PE1] /routing/bgp/advertisements> print
0 peer=to-pe2-1 dst=10.1.1.0/24 local-pref=100 origin=2 ext-communities=rt:1.1.1.1:111 atomic-aggregate=yes
|
Check that the 10.3.3.0 is installed in IP routes, in the cust-one route table:
| Code Block |
|---|
|
[admin@PE1] > /ip route print where routing-table="cust-one"
Flags: D - DYNAMIC; A - ACTIVE; c, b, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
# DST-ADDRESS GATEWAY DISTANCE
0 ADC 10.1.1.0/24 ether1@cust-one 0
1 ADb 10.3.3.0/24 10.5.5.3 20 |
Let's take a closer look at IP routes in cust-one VRF. The 10.1.1.0/24 IP prefix is a connected route that belongs to an interface that was configured to belong to cust-one VRF. The 10.3.3.0/24 IP prefix was advertised via BGP as a VPNv4 route from PE2 and is imported in this VRF routing table, because our configured import-route-targets matched the BGP extended communities attribute it was advertised with.
| Code Block |
|---|
|
[admin@PE1] /routing/route> print detail where routing-table="cust-one"
Flags: X - disabled, F - filtered, U - unreachable, A - active;
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - l
dp-mapping, g - slaac, y - bgp-mpls-vpn;
H - hw-offloaded; + - ecmp, B - blackhole
Ac afi=ip4 contribution=active dst-address=10.1.1.0/24 routing-table=cust-one
gateway=ether1@cust-one immediate-gw=ether1 distance=0 scope=10 belongs-to="connected"
local-address=10.1.1.2%ether1@cust-one
debug.fwp-ptr=0x202420C0
Ay afi=ip4 contribution=active dst-address=10.3.3.0/24 routing-table=cust-one label=16
gateway=10.5.5.3 immediate-gw=10.2.2.3%ether2 distance=20 scope=40 target-scope=30
belongs-to="bgp-mpls-vpn-1-bgp-VPN4-10.5.5.3-import"
bgp.peer-cache-id=*2C00011 .ext-communities=rt:1.1.1.1:111 .local-pref=100
.atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x20242840
[admin@PE1] /routing/route> print detail where afi="vpn4"
Flags: X - disabled, F - filtered, U - unreachable, A - active;
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - l
dp-mapping, g - slaac, y - bgp-mpls-vpn;
H - hw-offloaded; + - ecmp, B - blackhole
Ay afi=vpn4 contribution=active dst-address=10.1.1.0/24&1.1.1.1:111 routing-table=main label=19
gateway=ether1@cust-one immediate-gw=ether1 distance=200 scope=40 target-scope=10
belongs-to="bgp-mpls-vpn-1-connected-export"
bgp.ext-communities=rt:1.1.1.1:1111 .atomic-aggregate=no .origin=incomplete
debug.fwp-ptr=0x202426C0
Ab afi=vpn4 contribution=active dst-address=10.3.3.0/24&1.1.1.1:111 routing-table=main label=16
gateway=10.5.5.3 immediate-gw=10.2.2.3%ether2 distance=200 scope=40 target-scope=30
belongs-to="bgp-VPN4-10.5.5.3"
bgp.peer-cache-id=*2C00011 .ext-communities=rt:1.1.1.1:111 .local-pref=100
.atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x202427E0
|
The same for Cisco:
| Code Block |
|---|
|
PE2#show ip bgp vpnv4 all
BGP table version is 5, local router ID is 10.5.5.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:111 (default for vrf cust-one)
*>i10.1.1.0/24 10.5.5.2 100 0 ?
*> 10.3.3.0/24 0.0.0.0 0 32768 ?
PE2#show ip route vrf cust-one
Routing Table: cust-one
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
10.0.0.0/24 is subnetted, 1 subnets
B 10.1.1.0 [200/0] via 10.5.5.2, 00:05:33
10.0.0.0/24 is subnetted, 1 subnets
C 10.3.3.0 is directly connected, FastEthernet1/0 |
You should be able to ping from CE1 to CE2 and vice versa.
| Code Block |
|---|
|
[admin@CE1] > /ping 10.3.3.4
10.3.3.4 64 byte ping: ttl=62 time=18 ms
10.3.3.4 64 byte ping: ttl=62 time=13 ms
10.3.3.4 64 byte ping: ttl=62 time=13 ms
10.3.3.4 64 byte ping: ttl=62 time=14 ms
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 13/14.5/18 ms |
A more complicated setup (changes only)
Image Removed
As opposed to the simplest setup, in this example, we have two customers: cust-one and cust-two.
We configure two VPNs for them, cust-one and cust-two respectively, and exchange all routes between them. (This is also called "route leaking").
Note that this could be not the most typical setup, because routes are usually not exchanged between different customers. In contrast, by default, it should not be possible to gain access from one VRF site to a different VRF site in another VPN. (This is the "Private" aspect of VPNs.) Separate routing is a way to provide privacy, and it is also required to solve the problem of overlapping IP network prefixes. Route exchange is in direct conflict with these two requirements but may sometimes be needed (e.g. temp. solution when two customers are migrating to a single network infrastructure).
CE1 Router, cust-one
| Code Block |
|---|
|
/ip route add dst-address=10.4.4.0/24 gateway=10.1.1.2 |
CE2 Router, cust-one
| Code Block |
|---|
|
/ip route add dst-address=10.4.4.0/24 gateway=10.3.3.3
|
CE1 Router,cust-two
| Code Block |
|---|
|
/ip address add address=10.4.4.5 interface=ether1
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3
/ip route add dst-address=10.3.3.0/24 gateway=10.3.3.3 |
PE1 Router
| Code Block |
|---|
|
# replace the old BGP VPN with this:
/routing bgp vpn
add vrf=cust-one \
export.redistribute=connected \
route-distinguisher=1.1.1.1:111 \
import.route-targets=1.1.1.1:111,2.2.2.2:222 \
export.route-targets=1.1.1.1:111
|
PE2 Router (Cisco)
| Code Block |
|---|
|
ip vrf cust-one
rd 1.1.1.1:111
route-target export 1.1.1.1:111
route-target import 1.1.1.1:111
route-target import 2.2.2.2:222
exit
ip vrf cust-two
rd 2.2.2.2:222
route-target export 2.2.2.2:222
route-target import 1.1.1.1:111
route-target import 2.2.2.2:222
exit
interface FastEthernet2/0
ip vrf forwarding cust-two
ip address 10.4.4.3 255.255.255.0
router bgp 65000
address-family ipv4 vrf cust-two
redistribute connected
exit-address-family |
Variation: replace the Cisco with another MT
PE2 Mikrotik config
| Code Block |
|---|
|
/interface bridge add name=lobridge
/ip address
add address=10.2.2.3/24 interface=ether1
add address=10.3.3.3/24 interface=ether2
add address=10.4.4.3/24 interface=ether3
add address=10.5.5.3/32 interface=lobridge
/ip vrf
add name=cust-one interfaces=ether2
add name=cust-two interfaces=ether3
/mpls ldp add enabled=yes transport-address=10.5.5.3
/mpls ldp interface add interface=ether1
/routing bgp template set default as=65000
/routing bgp vpn
add vrf=cust-one \
export.redistribute=connected \
route-distinguisher=1.1.1.1:111 \
import.route-targets=1.1.1.1:111,2.2.2.2:222 \
export.route-targets=1.1.1.1:111 \
add vrf=cust-two \
export.redistribute=connected \
route-distinguisher=2.2.2.2:222 \
import.route-targets=1.1.1.1:111,2.2.2.2:222 \
export.route-targets=2.2.2.2:222 \
/routing bgp connection
add template=default remote.address=10.5.5.2 address-families=vpnv4 local.address=10.5.5.3
# add route to the remote BGP peer's loopback address
/ip route add dst-address=10.5.5.2/32 gateway=10.2.2.2
|
Results
The output of /ip route print now is interesting enough to deserve detailed observation.
| Code Block |
|---|
|
[admin@PE2] /ip route> print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 10.1.1.0/24 10.5.5.2 recurs... 20
1 ADC 10.3.3.0/24 10.3.3.3 ether2 0
2 ADb 10.4.4.0/24 20
3 ADb 10.1.1.0/24 10.5.5.2 recurs... 20
4 ADb 10.3.3.0/24 20
5 ADC 10.4.4.0/24 10.4.4.3 ether3 0
6 ADC 10.2.2.0/24 10.2.2.3 ether1 0
7 A S 10.5.5.2/32 10.2.2.2 reacha... 1
8 ADC 10.5.5.3/32 10.5.5.3 lobridge 0
|
The route 10.1.1.0/24 was received from a remote BGP peer and is installed in both VRF routing tables.
The routes 10.3.3.0/24 and 10.4.4.0/24 are also installed in both VRF routing tables. Each is a connected route in one table and a BGP route in another table. This has nothing to do with their being advertised via BGP. They are simply being "advertised" to the local VPNv4 route table and locally reimported after that. Import and export route-targets determine in which tables they will end up.
This can be deduced from its attributes - they don't have the usual BGP properties. (Route 10.4.4.0/24.)
| Code Block |
|---|
|
[admin@PE2] /routing/route> print detail where routing-table=cust-one
... 0
DAo 172.16.2.0/24 192.168.1.2%ipip-tunnel1@custC 110
DAc 192.168.1.0/24 ipip-tunnel1@custC 0
|
References
RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)
...