VXLAN was introduced to virtual networking in 2011 with Cisco’s Nexus 1000V version 1.5 to facilitate cloud scale deployments. With VMware vCloud Suite 5.1 including full support for VXLAN, it is a good idea to start getting acquainted with how this technology can help you resolve networking concerns in your cloud deployments.
After some research & testing in the home lab, I got the opportunity to deploy VXLAN in a much larger test lab, and it is up and running now, even across different L3 networks….
Here is the lab topology:
Router\L3 Switch: Catalyst 3750
L2 Switch: Two Nexus 5548UP in vPC Configuration
Server Infrastructure: UCS
Virtual Networking: Nexus 1000V
Before I dive into the workflow, the obligatory legalese:
Disclaimer: Do not attempt enabling VXLAN, which requires multicast, without consulting your Network Team. Multicast traffic can increase the load on your routers and switches and potentially lead to performance problems. The content below is provided for educational purposes only.
With that out of the way, here is how I configured the components:
1. Enable Routing and Multicast (IPServices license will be required)
ip multicast-routing distributed ip routing
2. Create the VLANs for VXLAN functionality and the associated L3 VLAN Interfaces
I created two VLANs (140 and 141) and associated VLAN interfaces with \26 subnets (enough address space to support 62 hosts in each network).
vlan 140 name vxlan-subnet-01 interface vlan 140 ip address 172.16.66.1 255.255.255.192
NOTE: You do not need more than one VLAN for VXLAN, but I wanted to test L2 connectivity across L3 boundaries.
3. Enable either IGMP Querier or PIM Routing on the VLAN interfaces created in the previous step. I used the IGMP Querier at home. So, I opted to use PIM Routing for the test lab.
int vlan 140 ip pim sparse-dense-mode
4. Increase MTU to 1600
NOTE: The Catalyst switch requires a reload (aka REBOOT!) for the MTU changes to take effect
system mtu 1600 system mtu jumbo 1600 (may be larger if using iSCSI)
After the reload, you will need to increase the routing mtu as well (system mtu routing 1600). The routing MTU cannot be set larger than the system MTU, and until the reload, the switch still thinks that its MTU is 1500.
5. Trunk the VXLAN VLANs on the connection to the Nexus 5548s
int po10 switchport trunk allowed vlan add 140,141
Nexus 5548 (These steps must be completed on both Nexus switches)
1. Create the VXLAN VLANs (140 and 141)
vlan 140 name vxlan-subnet-01
2. Enable IGMP Snooping on the VLANs
vlan configuration 140 ip igmp snooping
3. Increase the Switch MTU to 1600
policy-map jumbo class class-default mtu 1600 system qos service-policy jumbo
4. Trunk the VLANs on the VPC peer-link and the VPCs connected to the L3 Switch and UCS Fabric Interconnects
int po5 (VPC Peer Link) switchport trunk allowed vlan add 140,141 int po10 (Catalyst 3750 Uplink) switchport trunk allowed vlan add 140,141 int po11 (UCS Fabric Interconnect A) switchport trunk allowed vlan add 140,141 int po12 (UCS Fabric Interconnect B) switchport trunk allowed vlan add 140,141
1. Create the VLANs (if they do not already exist)
2. Increase the MTU to 1600 on the vNICs or vNIC Templates (Requires a REBOOT for existing hosts, but you have a User-Ack Maintenance Policy anyway, right?…RIGHT?!?)
3. Add the VLANs to the approrpriate VNIC or VNIC Template (adding VLANs will not trigger a reboot)
4. (Optional) Add 2 vNICs (1 for each Fabric Interconnect) to the Service Profiles you will be testing with if you don’t want to modify your existing Nexus 1000V Uplink Port Profile. I followed this step in my implementation.
Nexus 1000V Setup
1. Enable the segmentation feature on your Nexus 1000V Switch
2. Create the VXLAN VLANs (Same syntax as Nexus 5548)
3. Create Uplink Port-Profiles for the VXLAN VLANs with MTU size of 1600
port-profile type ethernet Uplink_vxlan_subnet_01 vmware port-group switchport mode trunk switchport trunk allowed vlan 140 mtu 1600 channel-group auto mode on mac-pinning no shutdown system vlan 140 state enabled
4. Create the Vethernet Port-Profile for your vSphere VMkernel interfaces.
port-profile type vethernet vxlan-01-vmknic vmware port-group switchport mode access switchport access vlan 140 capability vxlan no shutdown state enabled
5. Create the bridge domain, which will specify the mutlicast group that the vSphere servers will join to exchange VXLAN data.
bridge-domain tenant-tmt segment id 5014 group 220.127.116.11
6. Create the Vethernet Port-Profile for the VM Traffic. Note that you specify the bridge-domain instead of a VLAN
port-profile type vethernet tenant-tmt-vxlan vmware port-group switchport mode access switchport access bridge-domain tenant-tmt no shutdown state enabled
1. Create a VMKernel interface on each of the vSphere servers that uses one of the VXLAN VLANs. The vSphere servers will use these VMKernel interfaces to join the VXLAN bridge domain’s multicast group. In my environment, one server uses the VLAN 140 address space, and the other server uses the VLAN 141 address space.
2. Create your VMs on each host and assign the tenant-tmt-vxlan portgroup to their network interfaces.
3. Within the VMs, make sure the VMs belong to the same IP subnet. The value does not matter. They could all belong to the 10.0.0.0\24 network, for example.
There you have it, your VXLAN setup is complete. In your production Cloud deployment, you would need to deploy an Edge Firewall solution such as vCloud Networking & Security (formerly vShield Edge) or Cisco’s ASA 1000V to provide NAT functionality to reach the outside world, if your application requires it
Check out these excellent blog articles for more details on VXLAN theory and implementation:
If you already have VXLAN running in production, feel free to share your tips and tricks in the comment section below.