Overview
Ethernet Virtual Private Network (EVPN) is a technology to extend Layer 2 and 3 connectivity between different network segments. MPLS or VXLAN encapsulations can be used to transfer ethernet frames over MPLS or IP-based networks.
BGP EVPN is used as a control plane for multiple data-planes encapsulations (for both Layer2 and Layer3 VPN services). MP-BGP carries MAC addresses, MAC/IP bindings and IP prefixes.
RFC7432 is initial EVPN standard thad defines BGP as control plane for MPLS data plane. RFC8365 extends the use of additional data planes, VXLAN, NVGRE and MPLS over GRE and defines the use of EVPN as Network Virtualization Overlay (NVO).
Terminology
- NVO: Network Virtualization Overlay used to deliver Layer2 and Layer 3 VPN services.
- NVE: Network Virtualization Endpoint is Provider Edge (PE) node within the NVO environment. It is responsible for encapsulation/decapsulation of VPN traffic. In case of VXLAN this defines VTEP (Virtual Tunnel End Point).
- VNI: Virtual Network Identifier
- EVI: EVPN Instance
- RD: Route Distinguisher is a 64-bit prefix appended to IP prefix to make it unique, multiple tenants can use overlapping IP range.
- RT: Route Target is BGP extended community used to control import and export of routes. Typically, RT is based on the AS number and the VNI of the MAC-VRF
- MAC-VRF: VRF table for MAC addresses on a PE (VTEP). Requires RD and RT.
- BUM: Broadcast, unknown Unicast and Multicast traffic is a multi-destination layer2 traffic in vxlan networks.
- Ingress replication: unicast approach to handle BUM traffic. It uses IMET routes to auto-discover remote peers. Ingress device replicates BUM traffic to all the VTEPS associated with the Layer-2 VNI.
- ESI - Ethernet Segment Identifier
For MP-BGP to carry EVPN, new AFI/SAFI was defined 25(L2 VPN)/70(EVPN). Next-hop address within the NLRI is an IP address of the VTEP advertising the EVPN route.
There are five EVPN route types:
- Type-1: (Ethernet A-D) announces reachability of multi-homed ethernet segment
- Type-2:( MAC advertisement MACIP) advertises MAC address of MAC/IP binding learned by specific EVI
- Type-3: (Inclusive multicast IMET) advertises membership of a Layer 2 domain, allowing to auto discover VTEPs
- Type-4: (Ethernet segment) is used to discover VTEPs attached to the same shared Ethernet Segment for EVPN multi-homing model (active-active, active-standby forwarding)
- Type-5: (IP prefix) Advertising IP prefix into the EVPN domain allows to create classic Layer 3 VPN.
Data plane encapsulation is defined with encapsulation extended community value:
- 8 - VXLAN (currently only one supported by ROS)
- 9 - NVGRE
- 10 - MPLS
- 11 - MPLSoGRE
There are two methods for supporting inter-subnet routing with EVPN: symmetric and asymmetric integrated routing and bridging (IRB). The main difference between the two methods is that the symmetric method supports both routing and bridged on both the ingress and egress VTEPs, where the asymmetric method supports routing on the ingress, but only bridging on the egress.
Route list
RouterOS will show local and received EVPN routes in the /routing/route
list
Locally generated routes will hace e-evpn flag. for example:
[admin@ros_leaf_3] /routing/route> print where evpn Flags: e - EVPN Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE DST-ADDRESS GATEWAY AFI DISTANCE SCOPE TARGET-SCOPE e [10.155.101.133:1010]macip:0|0C:50:85:84:00:01 203.0.255.133 evpn 200 40 10 e [10.155.101.133:1010]imet:0|203.0.255.133 203.0.255.133 evpn 200 40 10 e [203.0.255.133:4]imet:0|203.0.255.133 203.0.255.133 evpn 200 40 10
EVPN data is encoded in dst-address parameter:
Dst [rd]type:x|y ^ ^ ^ | | \ - where x - tag or ESI; y - type specific data (can show mac addresses, ip addresses, ethernet segments etc.) | \------- name of the EVPN route type (macip, imet, es, ad, prefix) \---------- route distinguisher in square brackets
Configuration Examples
Basic Configuration Example
Basic Layer2 EVPN Vxlan configuration:
/interface bridge add name=bridge1 vlan-filtering=yes pvid=40 /interface bridge port add bridge=bridge1 interface=sfp-sfpplus3 pvid=40 /ip address add address=203.0.113.1 interface=lo /interface vxlan add bridge=bridge1 bridge-pvid=40 local-address=203.0.113.1 name=vxlan1 vni=100040 learning=no /routing bgp instance add as=65000 name=evpn-inst /routing bgp connection add afi=evpn instance=evpn-inst local.address=203.0.113.1 .role=ebgp multihop=yes name=to-leaf-lo remote.address=203.0.113.2 .as=65001 /routing bgp evpn add instance=evpn-inst name=bgp-evpn-1o vni=100040
For simple setups with only one vni, there is no need to set route distinguisher and import/export route targets.
EVPN configuration is directly mapped to VXLAN configurations with matching VNIs
Bridged EVPN VXLAN Overlay with Arista's Eos and ROS
Topology
For this example we will use the following topology:
- 203.0.255.0/24 address range is used for loopback addresses
- Subnets from 172.16.0.0/16 range is used for underlay connectivity of leaf and spine routers
- Subnets from 192.168.0.0/16 range is used for host addressing on vlans.
Underlay Configuration
Underly connectivity can depend on already existing setups, it can be eBGP, OSPF, ISIS or even static routing. In this case we will be using OSPF as underly routing protocol to advertise routes for addresses associated with loopback interfaces.
Ros_Spine
/ip address add address=203.0.255.138 interface=lo add address=172.16.1.1/30 interface=ether3 add address=172.16.2.1/30 interface=ether4 add address=172.16.3.1/30 interface=ether5 add address=172.16.4.1/30 interface=ether6 add address=172.16.5.1/30 interface=ether7 /routing ospf instance add name=evpn_underlay /routing ospf area add disabled=no instance=evpn_underlay name=evpn-underlay-bb /routing ospf interface-template add area=evpn-underlay-bb disabled=no networks=172.16.0.0/16 add area=evpn-underlay-bb disabled=no interfaces=lo passive
Ros_Leaf_3
/ip address add address=203.0.255.133 interface=lo add address=172.16.3.2/30 interface=ether10 /routing ospf instance add name=evpn_underlay /routing ospf area add disabled=no instance=evpn_underlay name=evpn-underlay-bb /routing ospf interface-template add area=evpn-underlay-bb disabled=no networks=172.16.0.0/16 add area=evpn-underlay-bb disabled=no interfaces=lo passive
Eos_Leaf
interface Ethernet1 no switchport ip address 172.16.5.2/30 ! interface Loopback0 ip address 203.0.255.128/32 ! ip routing ! router ospf 100 router-id 203.0.255.135 redistribute connected network 172.16.1.0/30 area 0.0.0.0 !
BGP EVPN Overlay
For BGP overly we will be using multihop eBGP that uses loopback addresses.
To simplify configuration we will utilize BGP template to set common parameters and set connection to listen on all loopback address range. This is great for scalability, if we will want to add more leaf routers we won't need to configure anything on the spine routers.
Also it is recommended to set nexthop-choice to propagate especially if there is more than one spine. In case if iBGP is used as overlay then spines should be route reflectors and nexthop propagation is happening by default.
Ros_Spine
/routing bgp instance add as=65000 name=bgp-instance-1 /routing bgp template set default afi=evpn multihop=yes nexthop-choice=propagate /routing bgp connection add instance=bgp-instance-1 local.address=203.0.255.138 .role=ebgp name=evpn_leafs remote.address=\ 203.0.255.0/24 templates=default
Ros_Leaf_3
/routing bgp instance add as=65003 disabled=no name=bgp-instance-1 /routing bgp connection add afi=evpn instance=bgp-instance-1 local.address=203.0.255.133 .role=ebgp multihop=yes name=to_spine remote.address=\ 203.0.255.138
Eos_Leaf
service routing protocols model multi-agent ! router bgp 65005 neighbor SPINE_EVPN peer group neighbor SPINE_EVPN remote-as 65000 neighbor SPINE_EVPN update-source Loopback0 neighbor SPINE_EVPN ebgp-multihop 10 neighbor SPINE_EVPN send-community extended neighbor 203.0.255.138 peer group SPINE_EVPN ! address-family evpn neighbor SPINE_EVPN activate !
Verify BGP Connectivity
Eos_Leaf
localhost#show bgp summary BGP summary information for VRF default Router identifier 203.0.255.128, local AS number 65005 Neighbor AS Session State AFI/SAFI AFI/SAFI State NLRI Rcd NLRI Acc ------------- ----------- ------------- ----------------------- -------------- ---------- ---------- 203.0.255.138 65000 Established IPv4 Unicast Advertised 0 0 203.0.255.138 65000 Established L2VPN EVPN Negotiated 6 6
Ros_Leaf_3
[admin@ros_leaf_3] /routing/bgp/session> print Flags: E - established 0 E name="to_spine-1" instance=bgp-instance-1 remote.address=203.0.255.138 .as=65000 .id=203.0.255.138 .capabilities=mp,rr,gr,as4 .afi=evpn .messages=7 .bytes=682 .eor="" local.address=203.0.255.133 .as=65003 .id=203.0.255.133 .cluster-id=203.0.255.133 .capabilities=mp,rr,gr,as4 .afi=evpn .messages=7 .bytes=698 .eor="" output.procid=20 input.procid=20 ebgp multihop=yes hold-time=3m keepalive-time=1m uptime=1s620ms last-started=2025-05-29 11:01:38 prefix-count=0
VXLAN and EVPN configuration
Ros_Leaf_3
Just for demonstration purposes, on RouterOS leaf we will be sending vlan tagged traffic to the host.
VXLAN learning should be disabled as we are using BGP EVPN for discovery.
/interface bridge add name=bridge1 pvid=10 vlan-filtering=yes /interface vxlan add bridge=bridge1 bridge-pvid=10 learning=no local-address=203.0.255.133 mac-address=C2:16:F6:B2:CC:D3 name=vxlan1 vni=1010 /interface bridge port add bridge=bridge1 interface=ether11 pvid=10 /ip address add address=192.168.10.133/24 interface=bridge1 /routing bgp evpn add disabled=no export.route-targets=1010:1010 import.route-targets=1010:1010 instance=bgp-instance-1 name=bgp-evpn-1 vni=1010
Eos_Leaf
On the Arista router we are setting vlan trunk, untagged traffic will be sent to the host
vlan 10 ! interface Ethernet2 switchport trunk allowed vlan 10 switchport mode trunk ! interface Vlan10 ip address 192.168.10.128/24 ! interface Vxlan1 vxlan source-interface Loopback0 vxlan vlan 10 vni 1010 ! router bgp 65501 vlan 10 rd 203.0.255.128:1010 route-target both 1010:1010 redistribute learned
Host_1
/ip address add address=192.168.10.132/24 interface=ether2
Host_3
/interface vlan add interface=ether2 name=vlan10 vlan-id=10 /ip address add address=192.168.10.129/24 interface=vlan10
Validate L2VPN Service
Lets verify that IMET routes are present on leaf routers and that vteps are discovered
[admin@ros_leaf_3] /routing/route> print where dst-address~"imet" Flags: A - ACTIVE; b - BGP, e - EVPN Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE, IMMEDIATE-GW DST-ADDRESS GATEWAY AFI DISTANCE SCOPE TARGET-SCOPE IMMEDIATE-GW Ab [203.0.255.128:1010]imet:0|203.0.255.128 203.0.255.128 evpn 20 40 30 172.16.3.1%ether10 e [203.0.255.133:256]imet:0|203.0.255.133 203.0.255.133 evpn 200 40 10 [admin@ros_leaf_3] /interface/vxlan/vteps> print Flags: D - DYNAMIC Columns: INTERFACE, REMOTE-IP # INTERFACE REMOTE-IP 0 D vxlan1 203.0.255.128
On Arista:
localhost#show bgp evpn route-type imet BGP routing table information for VRF default Router identifier 203.0.255.135, local AS number 65501 Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP c - Contributing to ECMP, % - Pending BGP convergence Origin codes: i - IGP, e - EGP, ? - incomplete AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop Network Next Hop Metric LocPref Weight Path * > RD: 203.0.255.128:1010 imet 203.0.255.128 - - - 0 i * > RD: 203.0.255.133:256 imet 203.0.255.133 203.0.255.133 - 100 0 65000 65003 i localhost#show interfaces vxlan1 Vxlan1 is up, line protocol is up (connected) Hardware is Vxlan Source interface is Loopback0 and is active with 203.0.255.128 Listening on UDP port 4789 Replication/Flood Mode is headend with Flood List Source: EVPN Remote MAC learning via EVPN VNI mapping to VLANs Static VLAN to VNI mapping is [10, 1010] Note: All Dynamic VLANs used by VCS are internal VLANs. Use 'show vxlan vni' for details. Static VRF to VNI mapping is not configured Headend replication flood vtep list is: 10 203.0.255.133 Shared Router MAC is 0000.0000.0000 localhost#show vxlan flood vtep vlan 10 VXLAN Flood VTEP Table -------------------------------------------------------------------------------- VLANS Ip Address ----------------------------- ------------------------------------------------ 10 203.0.255.133
At this point we can try to ping host_3 from host_1:
[admin@host_1] /interface> print ... 1 R ether2 ether 1500 0C:50:85:84:00:01 [admin@host_1] /ip/address> /ping 192.168.10.129 SEQ HOST SIZE TTL TIME STATUS 0 192.168.10.129 56 64 17ms26us 1 192.168.10.129 56 64 13ms119us 2 192.168.10.129 56 64 17ms192us
host-3
[admin@host_1] /interface> print ... 1 R ether2 ether 1500 0C:74:39:88:00:01
Now we should be able to see that EVPN is used to learn remote MAC addresses by looking at MACIP routes.
If we look at routes on ros_leaf, we can see that router 203.0.255.128 sent the macip route for 0C:74:39:88:00:01 mac address which is the host_1 mac address located behind eos_leaf.
Eos also sends MAC/IP binding which is used for arp/nd suppression. Unfortunately at the time of writing this article RouterOS does not have this functionality.
[admin@ros_leaf_3] /routing/route> print where dst-address~"macip" Flags: A - ACTIVE; b - BGP, e - EVPN Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE, IMMEDIATE-GW DST-ADDRESS GATEWAY AFI DISTANCE SCOPE TARGET-SCOPE IMMEDIATE-GW Ab [203.0.255.128:1010]macip:0|0C:74:39:88:00:01 203.0.255.128 evpn 20 40 30 172.16.3.1%ether10 e [203.0.255.133:256]macip:0|0C:50:85:84:00:01 203.0.255.133 evpn 200 40 10 Ab [203.0.255.128:1010]macip:0|0C:74:39:88:00:01|192.168.10.129 203.0.255.128 evpn 20 40 30 172.16.3.1%ether10
Arista allows additionally to see remotely learned mac addresses in "vxlan mac table" and "vlan mac-address table" includes local mac addresses as well:
localhost#show bgp evpn route-type mac-ip detail BGP routing table information for VRF default Router identifier 203.0.255.128, local AS number 65005 BGP routing table entry for mac-ip 0c50.8584.0001, Route Distinguisher: 203.0.255.133:256 Paths: 1 available 65000 65003 203.0.255.133 from 203.0.255.138 (203.0.255.138) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, best Extended Community: Route-Target-AS:1010:1010 TunnelEncap:tunnelTypeVxlan VNI: 0 ESI: 0000:0000:0000:0000:0000 BGP routing table entry for mac-ip 0c74.3988.0001, Route Distinguisher: 203.0.255.128:1010 Paths: 1 available Local - from - (0.0.0.0) Origin IGP, metric -, localpref -, weight 0, tag 0, valid, local, best Extended Community: Route-Target-AS:1010:1010 TunnelEncap:tunnelTypeVxlan VNI: 1010 ESI: 0000:0000:0000:0000:0000 BGP routing table entry for mac-ip 0c74.3988.0001 192.168.10.129, Route Distinguisher: 203.0.255.128:1010 Paths: 1 available Local - from - (0.0.0.0) Origin IGP, metric -, localpref -, weight 0, tag 0, valid, local, best Extended Community: Route-Target-AS:1010:1010 TunnelEncap:tunnelTypeVxlan VNI: 1010 ESI: 0000:0000:0000:0000:0000 localhost#show vxlan address-table vlan 10 Vxlan Mac Address Table ---------------------------------------------------------------------- VLAN Mac Address Type Prt VTEP Moves Last Move ---- ----------- ---- --- ---- ----- --------- 10 0c50.8584.0001 EVPN Vx1 203.0.255.133 1 1:30:49 ago Total Remote Mac Addresses for this criterion: 1 localhost#show mac address-table vlan 10 Mac Address Table ------------------------------------------------------------------ Vlan Mac Address Type Ports Moves Last Move ---- ----------- ---- ----- ----- --------- 10 0c50.8584.0001 DYNAMIC Vx1 1 1:31:17 ago 10 0c74.3988.0001 DYNAMIC Et2 1 1 day, 23:45:18 ago Total Mac Addresses for this criterion: 2 Multicast Mac Address Table ------------------------------------------------------------------ Vlan Mac Address Type Ports ---- ----------- ---- ----- Total Mac Addresses for this criterion: 0
Bridged EVPN Overlay with Arista ESI LAG
ESI-LAG (Ethernet Segment Identifier - Link Aggregation) or EVPN-LAG is a networking concept that enables multihoming using EVPN (Ethernet Virtual Private Network) technology, where client devices (like access switches) can connect to core devices (like distribution switches) through multiple links, forming a logical LAG interface.
Topology
Lets extend the topology from the previous example. Host_2 is a RouterOS device with LACP enabled on links connecting to two ARISTA Leafs, forming active-active multihoming setup.
Configuration Prerequisities
On how to set up underly and EVPN overly configuration refer to example above. In this example we will focus on ESI LAG configuration and setup validation.
Port Channel configuration
Assuming that underly and overly is configured, and is up and running, we will proceed to port channel configuration on Arista switches:
leaf_2 and leaf_4 config is exactly the same
interface Port-Channel3 switchport access vlan 10 switchport trunk allowed vlan 10 switchport mode trunk ! evpn ethernet-segment identifier 0000:0000:0000:0333:3333 route-target import 00:00:03:33:33:33 lacp system-id 0000.0333.3333 ! interface Ethernet2 channel-group 3 mode active
ros_host_2
/interface bonding add mode=802.3ad name=bond1 slaves=ether2,ether3 /interface vlan add interface=bond1 mtu=1496 name=vlan10 vlan-id=10 /ip address add address=192.168.10.130/24 interface=vlan10
Validate setup
Now if we look at evpn routes we should see some new route types. Both Arista switches are advertising Type-1 AD routes and Type-4 Ethernet Segment (ES) routes to discover multihoming VTEPs
[admin@gns3_spine1_ros] /routing/route> print where afi=evpn dst-address~"(ad|es)" Flags: A - ACTIVE; b - BGP Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE DST-ADDRESS GATEWAY AFI DISTANCE SCOPE TA Ab [203.0.255.127:1]ad:4294967295|00:00:00:00:00:00:03:33:33:33 203.0.255.127 evpn 20 40 30 Ab [203.0.255.127:1]es:00:00:00:00:00:00:03:33:33:33|203.0.255.127 203.0.255.127 evpn 20 40 30 Ab [203.0.255.127:1010]ad:0|00:00:00:00:00:00:03:33:33:33 203.0.255.127 evpn 20 40 30 Ab [203.0.255.134:1]ad:4294967295|00:00:00:00:00:00:03:33:33:33 203.0.255.134 evpn 20 40 30 Ab [203.0.255.134:1]es:00:00:00:00:00:00:03:33:33:33|203.0.255.134 203.0.255.134 evpn 20 40 30 Ab [203.0.255.134:1010]ad:0|00:00:00:00:00:00:03:33:33:33 203.0.255.134 evpn 20 40 30
If we check both Eos leafs, we will see that designated forwarder 203.0.255.127 (eos_leaf_4) is selected:
eos_leaf_2#show bgp evpn instance vlan 10 EVPN instance: VLAN 10 Route distinguisher: 203.0.255.134:1010 Route target import: Route-Target-AS:1010:1010 Route target export: Route-Target-AS:1010:1010 Service interface: VLAN-based Local VXLAN IP address: 203.0.255.134 VXLAN: enabled MPLS: disabled Local ethernet segment: ESI: 0000:0000:0000:0333:3333 Type: 0 (administratively configured) Interface: Port-Channel3 Mode: all-active State: up ES-Import RT: 00:00:03:33:33:33 DF election algorithm: modulus Designated forwarder: 203.0.255.127 Non-Designated forwarder: 203.0.255.134
Lets suspend the link from host2 to eos_leaf_4 and see what happens:
eos_leaf_2#show bgp evpn instance vlan 10 EVPN instance: VLAN 10 Route distinguisher: 203.0.255.134:1010 Route target import: Route-Target-AS:1010:1010 Route target export: Route-Target-AS:1010:1010 Service interface: VLAN-based Local VXLAN IP address: 203.0.255.134 VXLAN: enabled MPLS: disabled Local ethernet segment: ESI: 0000:0000:0000:0333:3333 Type: 0 (administratively configured) Interface: Port-Channel3 Mode: all-active State: up ES-Import RT: 00:00:03:33:33:33 DF election algorithm: modulus Designated forwarder: 203.0.255.134 [admin@spine1_ros] /routing/route> print interval=1 where dst-address~"ad|es" Flags: A - ACTIVE; b - BGP Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE DST-ADDRESS GATEWAY AFI DISTANCE SCOPE TA Ab [203.0.255.134:1]ad:4294967295|00:00:00:00:00:00:03:33:33:33 203.0.255.134 evpn 20 40 30 Ab [203.0.255.134:1]es:00:00:00:00:00:00:03:33:33:33|203.0.255.134 203.0.255.134 evpn 20 40 30 Ab [203.0.255.134:1010]ad:0|00:00:00:00:00:00:03:33:33:33 203.0.255.134 evpn 20 40 30 [admin@host_2] /interface/bonding> /ping 192.168.10.132 interval=500ms SEQ HOST SIZE TTL TIME STATUS 0 192.168.10.132 56 64 2ms90us 1 192.168.10.132 56 64 2ms172us 2 192.168.10.132 56 64 2ms503us 3 192.168.10.132 timeout 4 192.168.10.132 timeout 5 192.168.10.132 timeout 6 192.168.10.132 56 64 2ms191us 7 192.168.10.132 56 64 2ms31us
eos_leaf_2 became forwarder, eos_leaf_4 withdraw ES and AD routes and traffic switched to other LACP link.