VXLAN on UCS and vSphere: from L3 to Nexus 1000V

VXLAN was introduced to virtual networking in 2011 with Cisco’s Nexus 1000V version 1.5 to facilitate cloud scale deployments.  With VMware vCloud Suite 5.1 including full support for VXLAN, it is a good idea to start getting acquainted with how this technology can help you resolve networking concerns in your cloud deployments.

After some research & testing in the home lab, I got the opportunity to deploy VXLAN in a much larger test lab, and it is up and running now, even across different L3 networks….

Here is the lab topology:

Router\L3 Switch: Catalyst 3750
L2 Switch: Two Nexus 5548UP in vPC Configuration
Server Infrastructure: UCS
Virtual Networking: Nexus 1000V

Before I dive into the workflow, the obligatory legalese:

Disclaimer: Do not attempt enabling VXLAN, which requires multicast, without consulting your Network Team.  Multicast traffic can increase the load on your routers and switches and potentially lead to performance problems.  The content below is provided for educational purposes only.

With that out of the way, here is how I configured the components:

Catalyst 3750

1. Enable Routing and Multicast (IPServices license will be required)

ip multicast-routing distributed
ip routing

2. Create the VLANs for VXLAN functionality and the associated L3 VLAN Interfaces

I created two VLANs (140 and 141) and associated VLAN interfaces with \26 subnets (enough address space to support 62 hosts in each network).

vlan 140
name vxlan-subnet-01
interface vlan 140
ip address

NOTE: You do not need more than one VLAN for VXLAN, but I wanted to test L2 connectivity across L3 boundaries.

3. Enable either IGMP Querier or PIM Routing on the VLAN interfaces created in the previous step.  I used the IGMP Querier at home.  So, I opted to use PIM Routing for the test lab.

int vlan 140
ip pim sparse-dense-mode

4. Increase MTU to 1600
NOTE: The Catalyst switch requires a reload (aka REBOOT!) for the MTU changes to take effect

system mtu 1600
system mtu jumbo 1600 (may be larger if using iSCSI)

After the reload, you will need to increase the routing mtu as well (system mtu routing 1600).  The routing MTU cannot be set larger than the system MTU, and until the reload, the switch still thinks that its MTU is 1500.

5. Trunk the VXLAN VLANs on the connection to the Nexus 5548s

int po10
switchport trunk allowed vlan add 140,141

Nexus 5548 (These steps must be completed on both Nexus switches)

1. Create the VXLAN VLANs (140 and 141)

vlan 140
name vxlan-subnet-01

2. Enable IGMP Snooping on the VLANs

vlan configuration 140
ip igmp snooping

3. Increase the Switch MTU to 1600

policy-map jumbo
class class-default
mtu 1600
system qos
service-policy jumbo

4. Trunk the VLANs on the VPC peer-link and the VPCs connected to the L3 Switch and UCS Fabric Interconnects

int po5 (VPC Peer Link)
switchport trunk allowed vlan add 140,141
int po10 (Catalyst 3750 Uplink)
switchport trunk allowed vlan add 140,141
int po11 (UCS Fabric Interconnect A)
switchport trunk allowed vlan add 140,141
int po12 (UCS Fabric Interconnect B)
switchport trunk allowed vlan add 140,141

UCS Setup

1. Create the VLANs (if they do not already exist)

Create your VXLAN VLANs in UCS

2. Increase the MTU to 1600 on the vNICs or vNIC Templates (Requires a REBOOT for existing hosts, but you have a User-Ack Maintenance Policy anyway, right?…RIGHT?!?)

Increase your MTU to Accomodate VXLAN Header Info

3. Add the VLANs to the approrpriate VNIC or VNIC Template (adding VLANs will not trigger a reboot)

Don't forget to add your VXLAN VLANs to your vNIC\vNIC Template!

4. (Optional) Add 2 vNICs (1 for each Fabric Interconnect) to the Service Profiles you will be testing with if you don’t want to modify your existing Nexus 1000V Uplink Port Profile.  I followed this step in my implementation.

Nexus 1000V Setup

1. Enable the segmentation feature on your Nexus 1000V Switch

feature segmentation

2. Create the VXLAN VLANs (Same syntax as Nexus 5548)

3. Create Uplink Port-Profiles for the VXLAN VLANs with MTU size of 1600

port-profile type ethernet Uplink_vxlan_subnet_01
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 140
  mtu 1600
  channel-group auto mode on mac-pinning
  no shutdown
  system vlan 140
  state enabled

4. Create the Vethernet Port-Profile for your vSphere VMkernel interfaces.

port-profile type vethernet vxlan-01-vmknic
  vmware port-group
  switchport mode access
  switchport access vlan 140
  capability vxlan
  no shutdown
  state enabled

5. Create the bridge domain, which will specify the mutlicast group that the vSphere servers will join to exchange VXLAN data.

bridge-domain tenant-tmt
  segment id 5014

6. Create the Vethernet Port-Profile for the VM Traffic. Note that you specify the bridge-domain instead of a VLAN

port-profile type vethernet tenant-tmt-vxlan
  vmware port-group
  switchport mode access
  switchport access bridge-domain tenant-tmt
  no shutdown
  state enabled


1. Create a VMKernel interface on each of the vSphere servers that uses one of the VXLAN VLANs.  The vSphere servers will use these VMKernel interfaces to join the VXLAN bridge domain’s multicast group.  In my environment, one server uses the VLAN 140 address space, and the other server uses the VLAN 141 address space.

Server 1 VXLAN VMkernel Interface


Server 2 VXLAN VMkernel Interface (Remember: \26 subnet. So it is a different L3 network)

2. Create your VMs on each host and assign the tenant-tmt-vxlan portgroup to their network interfaces.

3. Within the VMs, make sure the VMs belong to the same IP subnet.  The value does not matter.  They could all belong to the\24 network, for example.

There you have it, your VXLAN setup is complete.  In your production Cloud deployment, you would need to deploy an Edge Firewall solution such as vCloud Networking & Security (formerly vShield Edge) or Cisco’s ASA 1000V to provide NAT functionality to reach the outside world, if your application requires it

Check out these excellent blog articles for more details on VXLAN theory and implementation:

Kamau Wanguhu
Duncan Epping
Rawlinson Rivera
Bobby Watson 

If you already have VXLAN running in production, feel free to share your tips and tricks in the comment section below.

1 thought on “VXLAN on UCS and vSphere: from L3 to Nexus 1000V”

Leave a Reply

Your email address will not be published. Required fields are marked *