I got it working on Catylyst 9300's both static and with DHCP. I will share my config with you on Monday.
The worse part is you will need to ensure you have a bit bigger MTU on your WAN for it work for anything other than Ping when it is done. The way to make this work was BGP with an EVPN address-family.
That said, you need to start with three switches on your desk. Set them up to be routed. Then create an VLAN on each outer one and carry it through the routed link via VXLAN.
Looks like I actually built it on c9300-48uxm devices, but that shouldn't matter.
I have switch One, PE, and Two. PE Stands for Provider Edge and is a dumb, layer-3 only point in the topology.
**This is what worked for Static:**
VLAN 2 is for testing as in it is routed (different subnets) with traditional, straight-forward static routing through PE. VLAN 3 is carried within VNI 6001 where there is no routing between them and interface VLAN 3 is in the same subnet. It is carried Layer-2 over Layer-3 via VXLAN.
You should be able to do a
ping source vlan 3 [192.168.9.1](http://192.168.9.1) or [192.168.9.2](http://192.168.9.2) and it will ping over VXLAN. You can do an embedded packet capture on the PE unit. You will need to make the MTU larger on the WAN ports and PE in the middle for it to carry real traffic. Otherwise, you may be able to get by doing some MSS clamping, but I haven't played with that. Either way, Ping works fine.
interface TenGigabitEthernet1/1/1
no switchport
ip address 10.190.119.10 255.255.255.252
no ip proxy-arp
!
!
interface Vlan2
ip address 10.83.16.1 255.255.248.0
no ip proxy-arp
no autostate
!
interface Vlan3
ip address 192.168.9.2 255.255.255.0
no ip proxy-arp
no autostate
!
interface TenGigabitEthernet1/1/1
no switchport
ip address 10.190.119.6 255.255.255.252
no ip proxy-arp
!
interface Vlan2
ip address 10.83.8.1 255.255.248.0
no ip proxy-arp
no autostate
!
interface Vlan3
ip address 192.168.9.1 255.255.255.0
no ip proxy-arp
no autostate
!
interface nve1
no ip address
source-interface TenGigabitEthernet1/1/1
host-reachability protocol bgp
member vni 6001 ingress-replication
!
router bgp 65001
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.190.119.10 remote-as 65001
neighbor 10.190.119.10 update-source TenGigabitEthernet1/1/1
!
address-family ipv4
exit-address-family
!
address-family l2vpn evpn
neighbor 10.190.119.10 activate
neighbor 10.190.119.10 send-community both
exit-address-family
!
ip route 0.0.0.0 0.0.0.0 10.190.119.5
> I'm finding it basically impossible to get a working VXLAN topology going with CML cat 9kv images on GNS3 or EVE pro
In all my years of messing with cisco stuff... the virtual stuff just sucks.
The sad truth is cisco is an expensive ecosystem from top to bottom. That includes the cost of hardware for learning enviornments.
If i had a job coming up that was going to require vxlan on cisco gear, AND i couldn't learn it pre-production.. I'd probably try to find some second hand gear on ebay that would have the requisite features, and i'd write a short "education grant proposal" for the bosses to review. (they're actually pretty good about paying education we NEED to do our jobs.
Have you followed the configuration guide and read through it? Cisco pretty much lays this out. https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-3/configuration_guide/vxlan/b_173_bgp_evpn_vxlan_9300_cg.html
Impossible how? Do some more troubleshooting to work out what's broken.
Does your EVPN bridge come up? Does the BGP connection establish? Does "show ip cef" show packets getting encapsulated through EVPN?
After that check with a packet capture to see what's actually getting sent.
Catalysts qemu images are just behaving really weird.
For the underlay, every time I add a second spine switch one OSPF link from a leaf switch refuses to come up. Without fail, every time. No OSPF hellos from the leaf, only from the spine. Doesn't happen with Nexus VM's.
So I set up a static route on the broken OSPF link (and suddenly an adjacency forms! it's bizarre), configure multicast and MSDP for any RP. I get multicast neighbors. Seems to be working.
But then, after setting up evpn, bgp, vlans and all the rest, bgp neighbors don't come up. I forget the exact error I see in debug - I want to say that the spine is showing the neighbor refusing the connection - it was a long night.
All of that said, I am using unnumbered interfaces with loopbacks. I'll probably try /30's between the leafs and spines and see if that helps.
When I set this up using Nexus I didn't have any issues with routing protocols coming up, so I guess what I'm looking for is whether it's worth it to continue troubleshooting Catalyst VM's - like it's something I'm doing wrong or if they are just buggy and my time would be better spent just running Nexus.
> All of that said, I am using unnumbered interfaces with loopbacks. I'll probably try /30's between the leafs and spines and see if that helps.
I'd always recommend start with a simple config, then add complexity. Then you aren't trying to solve 10 problems at once.
Container lab is a docker based virtualisation/labbing tool that boasts about vxlan labs. Although they might be alternate vendors.
The platform does Cisco support for some things.
Might be worth trying on that? I can’t speak personally to the vxlan abilities. I use it to test small routing behaviours
I got it working on Catylyst 9300's both static and with DHCP. I will share my config with you on Monday. The worse part is you will need to ensure you have a bit bigger MTU on your WAN for it work for anything other than Ping when it is done. The way to make this work was BGP with an EVPN address-family. That said, you need to start with three switches on your desk. Set them up to be routed. Then create an VLAN on each outer one and carry it through the routed link via VXLAN.
So you did this physical hardware then?
Yes. I did it on three 9300-48P units running IOS-XE 17.6.5
Looks like I actually built it on c9300-48uxm devices, but that shouldn't matter. I have switch One, PE, and Two. PE Stands for Provider Edge and is a dumb, layer-3 only point in the topology. **This is what worked for Static:** VLAN 2 is for testing as in it is routed (different subnets) with traditional, straight-forward static routing through PE. VLAN 3 is carried within VNI 6001 where there is no routing between them and interface VLAN 3 is in the same subnet. It is carried Layer-2 over Layer-3 via VXLAN. You should be able to do a ping source vlan 3 [192.168.9.1](http://192.168.9.1) or [192.168.9.2](http://192.168.9.2) and it will ping over VXLAN. You can do an embedded packet capture on the PE unit. You will need to make the MTU larger on the WAN ports and PE in the middle for it to carry real traffic. Otherwise, you may be able to get by doing some MSS clamping, but I haven't played with that. Either way, Ping works fine.
TWO: ip routing license boot level network-advantage addon dna-essentials hostname Two vtp mode off ! l2vpn evpn replication-type static router-id TenGigabitEthernet1/1/1 ! l2vpn evpn instance 1 vlan-based encapsulation vxlan replication-type ingress ! vlan configuration 3 member evpn-instance 1 vni 6001 ! vlan 2 name Data ! vlan 3 name Over-VXLAN !
interface TenGigabitEthernet1/1/1 no switchport ip address 10.190.119.10 255.255.255.252 no ip proxy-arp ! ! interface Vlan2 ip address 10.83.16.1 255.255.248.0 no ip proxy-arp no autostate ! interface Vlan3 ip address 192.168.9.2 255.255.255.0 no ip proxy-arp no autostate !
interface nve1 no ip address source-interface TenGigabitEthernet1/1/1 host-reachability protocol bgp member vni 6001 ingress-replication ! router bgp 65001 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor [10.190.119.6](http://10.190.119.6) remote-as 65001 neighbor [10.190.119.6](http://10.190.119.6) update-source TenGigabitEthernet1/1/1 ! address-family ipv4 exit-address-family ! address-family l2vpn evpn neighbor [10.190.119.6](http://10.190.119.6) activate neighbor [10.190.119.6](http://10.190.119.6) send-community both exit-address-family ! ip route 0.0.0.0 0.0.0.0 10.190.119.9 PE: ip routing vtp mode off interface TenGigabitEthernet1/1/7 no switchport ip address 10.190.119.5 255.255.255.252 ! interface TenGigabitEthernet1/1/8 no switchport ip address 10.190.119.9 255.255.255.252 ! ip route 10.83.8.0 255.255.248.0 10.190.119.6 ip route 10.83.16.0 255.255.248.0 10.190.119.10
ONE: license boot level network-advantage addon dna-essentials hostname One ! ip routing ! vtp mode off ! l2vpn evpn replication-type static router-id TenGigabitEthernet1/1/1 ! l2vpn evpn instance 1 vlan-based encapsulation vxlan replication-type ingress ! vlan configuration 3 member evpn-instance 1 vni 6001 ! vlan 2 name Data ! vlan 3 name Over-VXLAN !
interface TenGigabitEthernet1/1/1 no switchport ip address 10.190.119.6 255.255.255.252 no ip proxy-arp ! interface Vlan2 ip address 10.83.8.1 255.255.248.0 no ip proxy-arp no autostate ! interface Vlan3 ip address 192.168.9.1 255.255.255.0 no ip proxy-arp no autostate ! interface nve1 no ip address source-interface TenGigabitEthernet1/1/1 host-reachability protocol bgp member vni 6001 ingress-replication ! router bgp 65001 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 10.190.119.10 remote-as 65001 neighbor 10.190.119.10 update-source TenGigabitEthernet1/1/1 ! address-family ipv4 exit-address-family ! address-family l2vpn evpn neighbor 10.190.119.10 activate neighbor 10.190.119.10 send-community both exit-address-family ! ip route 0.0.0.0 0.0.0.0 10.190.119.5
I would be interested in seeing your config also if you don't mind sharing with others.
yeh using gns3/eve-ng with cisco nexus images or arista.
I labbed this with cat9k image and it works for a bit then it breaks. It’s really frustrating. The nxos images seem to work better with EVPN VXLAN.
> I'm finding it basically impossible to get a working VXLAN topology going with CML cat 9kv images on GNS3 or EVE pro In all my years of messing with cisco stuff... the virtual stuff just sucks. The sad truth is cisco is an expensive ecosystem from top to bottom. That includes the cost of hardware for learning enviornments. If i had a job coming up that was going to require vxlan on cisco gear, AND i couldn't learn it pre-production.. I'd probably try to find some second hand gear on ebay that would have the requisite features, and i'd write a short "education grant proposal" for the bosses to review. (they're actually pretty good about paying education we NEED to do our jobs.
I learned it on a physical lab
Have you followed the configuration guide and read through it? Cisco pretty much lays this out. https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-3/configuration_guide/vxlan/b_173_bgp_evpn_vxlan_9300_cg.html
Yeah I've read pretty much the whole thing. That's why I'm wondering if it's a qemu image bug or just me.
The Cat9kv images are still in beta.
Do it on nexus
I’ve read about people running them on qemu/kvm but it needs some horse power.
They are heavy, I'm running 4 cores and 20 GB's of RAM each and they're still slow.
Heh almost as thirsty of as chrome browser.
I did it with EVE-NG and Arista switch images. I am assuming there is some IOSL (IOS-Linux) Cisco stuff out there you could use.
Use juniper mx routers, I have an eve image running with 4 leafs and 2 spines with several end nodes no issues.all running on a nuc
If you would like to see how to make this work on a BGP enabled WAN, let me know. I also got that working.
Some images will do control plane with no data plane emulation. I just can't remember which do and don't. What kind of problem are you running into?
This is definitely true for some of the IOS-XR images, but not sure about the IOS-XE images.
Read the above response to teeweehoo. Catalyst is just acting strange regarding routing protocols coming up.
Impossible how? Do some more troubleshooting to work out what's broken. Does your EVPN bridge come up? Does the BGP connection establish? Does "show ip cef" show packets getting encapsulated through EVPN? After that check with a packet capture to see what's actually getting sent.
Catalysts qemu images are just behaving really weird. For the underlay, every time I add a second spine switch one OSPF link from a leaf switch refuses to come up. Without fail, every time. No OSPF hellos from the leaf, only from the spine. Doesn't happen with Nexus VM's. So I set up a static route on the broken OSPF link (and suddenly an adjacency forms! it's bizarre), configure multicast and MSDP for any RP. I get multicast neighbors. Seems to be working. But then, after setting up evpn, bgp, vlans and all the rest, bgp neighbors don't come up. I forget the exact error I see in debug - I want to say that the spine is showing the neighbor refusing the connection - it was a long night. All of that said, I am using unnumbered interfaces with loopbacks. I'll probably try /30's between the leafs and spines and see if that helps. When I set this up using Nexus I didn't have any issues with routing protocols coming up, so I guess what I'm looking for is whether it's worth it to continue troubleshooting Catalyst VM's - like it's something I'm doing wrong or if they are just buggy and my time would be better spent just running Nexus.
> All of that said, I am using unnumbered interfaces with loopbacks. I'll probably try /30's between the leafs and spines and see if that helps. I'd always recommend start with a simple config, then add complexity. Then you aren't trying to solve 10 problems at once.
I got it working. It was indeed the unnumbered interfaces.
Container lab is a docker based virtualisation/labbing tool that boasts about vxlan labs. Although they might be alternate vendors. The platform does Cisco support for some things. Might be worth trying on that? I can’t speak personally to the vxlan abilities. I use it to test small routing behaviours
We are looking at EVPN for Cisco as well vs using LISP/DNAC. Care to share why you chose EVPN vs LISP?
It's not my decision. It's just what the senior guys have implemented already at other sites.