Recently I did a “few” upgrades to the home lab. Besides an upgrade to enhance shared storage for vSphere (my old NAS was at 502 days uptime), I took the opportunity to enable jumbo packets on my Dell PowerConnect 5324 and the new fire-and-forget Thecus N7700PRO NAS. As the basis for new lab infrastructure to test VMware, Hyper-V and Xen, it’s a good improvement.

Since the first use was to test some of the new features of vSphere / vCenter 4.1, I also took the opportunity to change over to ESXi from ESX. According to VMware, 4.1 is the last release of ESX, so time to get cracking with ESXi, vMA, and the differences in managing the hosts.

I wanted to take advantage of jumbo frames on my ESXi systems. However, I didn’t decide this until I’d already installed the hosts (and didn’t see an advanced option to set the management interface MTU).

The Problem

I couldn’t provision my NAS storage into a different VLAN / physical network, which would be best practices for storage for the ESXi hosts. And I didn’t realize that with ESXi there is no way to set tell the hypervisor which port group / vmk to use for NFS access. It’s based on the configured network of the NAS device or based on the lowest numbered vmk of the machine. And since vmk0, Management Network, is created by default with an MTU of 1500 bytes, getting that to 9000 bytes is a problem.

The Solution

So what we’re going to do is use two vmk’s to set each to an MTU of 9000 while in maintenance mode. We’ll also change the port group names to reflect the one that deals with vMotion.

Prerequisites

So we have a cluster of hosts with multiple vmnics, but with managment, vMotion and storage all in the same network. The host we are going to change is in the following environment:

1. Switch and NFS storage both configured for jumbo frames and an MTU of 9000 bytes
2. vMA4.0.0 (or 4.1 — I’m using 4.0 for ghettovcbg2 compatibility) and vCenter 4.1 on installed dedicated systems (or one not affected by the host we are converting)
3. ESXi 4.1 configured with default MTU (1500 bytes) managed by vCenter
4. VLAN is the management and storage network, VMware hosts are trunked to the the switch
5. Ability to migrate guests to other cluster members
6. Full permissions to the vSphere
7. Working knowledge of making changes to networking and storage

The Process

Prep the ESXi Host

Our host, esx03.peanuts.local has two IP addresses, 172.16.200.90 assigned to vmk0 and 172.16.200.91 assigned to vmk1. Do the following to get the host ready:

• Migrate all running VM’s to other hosts
• Migrate powered down VM’s to another host
• Place the host into maintenance mode
• Optional – You may wish to disable HA. I found that the various connectivity changes forced HA scans that caused alerts to rise. No issues, but beware.

At this point we can do almost anything we wish to the host without affecting the rest of the environment.

Remove vmk1

At the start, vSwitch0 looks like this:

Edit the properties of the switch and remove the port group VMKernel Primary. At this point, there will be a single vmkernel, vmk0, that we’ll use to connect via the vMA.

Connect to the host via vMA

Login as the vi-admin and add the host via the IP address of vmk0, and connect. You will need the root account password for the ESXi box.

[vi-admin@vma ~]$ sudo vifp addserver 172.16.200.90
root@esx03.peanuts.local's password:
[vi-admin@vma ~]$ vifpinit 172.16.200.90

Convert vSwitch0 to Support Jumbo Frames

After connecting, modify the switch and verify the settings. Note, this may cause a connectivity outage. For one host, it took 30-40 seconds to complete the command.

[vi-admin@vma ~][172.16.200.90]$ esxcfg-vswitch -m 9000 vSwitch0
[vi-admin@vma ~][172.16.200.90]$ esxcfg-vswitch -l
Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks
vSwitch0        128             5               128                 9000    vmnic2,vmnic1,vmnic0

   PortGroup Name                VLAN ID   Used Ports      Uplinks
   ISP - Comcast                 502       0               vmnic2,vmnic1,vmnic0
   Development LAN               30        0               vmnic2,vmnic1,vmnic0
   VM Network                    10        0               vmnic2,vmnic1,vmnic0
   Management Network            10        1               vmnic0,vmnic1,vmnic2

Look to ensure this the vSwitch shows 9000 under MTU.

Create the New vmk1

Create a new port group and assign it to the proper VLAN. I used a name that would be more specific to the purpose once we’re done (vMotion).

[vi-admin@vma ~][172.16.200.90]$ esxcfg-vmknic -a -i 172.16.200.91 -n 255.255.255.0 -m 9000 "vMotion"
Added the VMkernel NIC successfully
[vi-admin@vma ~][172.16.200.90]$ esxcfg-vmknic -l
Interface  Port Group/DVPort             IP Family IP Address                        Netmask           MTU     Type
vmk0       Management Network            IPv4      172.16.200.90                     255.255.255.0     1500    STATIC
vmk0       Management Network            IPv6      fe80::21b:21ff:fe0f:82b           64                1500    STATIC
vmk1       vMotion                       IPv4      172.16.200.91                     255.255.255.0     9000    STATIC
vmk1       vMotion                       IPv6      fe80::250:56ff:fe77:3517          64                9000    STATIC

Verify that vmk1 has 9000 for the MTU (I removed the MAC address field so it would show here). Note that vmk0, how we’re connected and where NFS traffic transits, is still at 1500 bytes. Not for long!

Remove vmk0

Prior to removing vmk0, use the GUI to enabled Management traffic on the new vmk1 (vMotion), then select OK:

At the vSwitch0 properties page, select and Remove the Management Network port group (vmk0):

Note: when you hit Remove and confirm, connectivity to the host will be lost until the next step. What this does is starts the process of removing the vmkernel. However, since removal of that breaks connectivity, the actual removal of the port group doesn’t take place. At least it didn’t for me.

Create New vmk0

From the vMA, forcibly remove the host by IP address (vmk0′s), then add the host via the vmk1 IP address. Verify that the vSwitch has the port group Management Network still there and assigned to VLAN 10:

[vi-admin@vma ~][172.16.200.90]$ sudo vifp removeserver 172.16.200.90 --force
root@172.16.200.90's password:
[vi-admin@vma ~][172.16.200.90]$ sudo vifp addserver 172.16.200.91
root@esx03-vmk.peanuts.local's password:
[vi-admin@vma ~][172.16.200.90]$ vifpinit 172.16.200.91
[vi-admin@vma ~][172.16.200.91]$ esxcfg-vswitch -l
Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks
vSwitch0        128             5               128                 9000    vmnic2,vmnic1,vmnic0

   PortGroup Name                VLAN ID   Used Ports      Uplinks
   VM Network                    10        0               vmnic2,vmnic1,vmnic0
   ISP - Comcast                 502       0               vmnic2,vmnic1,vmnic0
   Development LAN               30        0               vmnic2,vmnic1,vmnic0
   vMotion                       10        1               vmnic2,vmnic1,vmnic0
   Management Network            10        0               vmnic0,vmnic1,vmnic2

If the port gorup isn’t there, use the steps above to create the port group and assign it to the VLAN. Now create the new vmk0 with jumbo frames and verify both vmk’s are correctly made:

[vi-admin@vma ~][172.16.200.91]$ esxcfg-vmknic -a -i 172.16.200.90 -n 255.255.255.0 -m 9000 "Management Network"
Added the VMkernel NIC successfully
[vi-admin@vma ~][172.16.200.91]$ esxcfg-vmknic -l
Interface  Port Group/DVPort             IP Family IP Address                        Netmask           MAC Address       MTU     Type
vmk1       vMotion                       IPv4      172.16.200.91                     255.255.255.0     00:50:56:77:35:17 9000    STATIC
vmk1       vMotion                       IPv6      fe80::250:56ff:fe77:3517          64                00:50:56:77:35:17 9000    STATIC
vmk0       Management Network            IPv4      172.16.200.90                     255.255.255.0     00:50:56:71:f8:b6 9000    STATIC
vmk0       Management Network            IPv6      fe80::250:56ff:fe71:f8b6          64                00:50:56:71:f8:b6 9000    STATIC

You can see that the MTU is correct for both vmk’s (scroll over, I  left the MAC address in this time).  At this point, vCenter will see the host.

Final Restart

One last reboot of the host will ensure everything is in proper order. I do this from vCenter once the host is reachable:

After the host comes back up in vCenter, we’ll clean up the networking config and storage.

Reassign Services to Proper vmk’s

Go into the vSwitch0 properties and set the port properties for each vmkernel:

Management Network - vmk0 – Management traffic - managed via vCenter / vMA, and as lowest vmk, used by NFS for storage on the same subnet.

vMotion - vmk1 – vMotion and FT (fault tolerance logging). Deselect Management traffic.

When you select OK, this will finalize the port group settings.

Clean Up

At this point, all NFS traffic (actually any traffic) can now use jumbo frames. I’d go over to Storage and refresh to make sure the store populates as expected. Finally, we’ll clear out the host entry in the vMA and add using the FQDN (a topic for another posting!):

[vi-admin@vma ~][172.16.200.91]$ sudo vifp removeserver 172.16.200.91
root@172.16.200.91's password:
[vi-admin@vma ~][172.16.200.91]$ sudo vifp addserver esx03.peanuts.local
root@esx03.peanuts.local's password:
[vi-admin@vma ~][172.16.200.91]$ vifpinit esx03
[vi-admin@vma ~][esx03]$ esxcfg-vmknic -l
Interface  Port Group/DVPort             IP Family IP Address                        Netmask           MAC Address       MTU     Type
vmk0       Management Network            IPv4      172.16.200.90                     255.255.255.0     00:50:56:71:f8:b6 9000    STATIC
vmk0       Management Network            IPv6      fe80::250:56ff:fe71:f8b6          64                00:50:56:71:f8:b6 9000    STATIC
vmk1       vMotion                       IPv4      172.16.200.91                     255.255.255.0     00:50:56:77:35:17 9000    STATIC
vmk1       vMotion                       IPv6      fe80::250:56ff:fe77:3517          64                00:50:56:77:35:17 9000    STATIC

And the server is now reachable via the FQDN and the vmk’s appear in proper order. If you have the ability, perform a netstat -na on the NAS to verify connections are coming from the vmk0 address.

Problems and Resolution

I’ll update the post with corrections, but with different configurations, we may run into different issues.

Observed Problems and Resolutions

Problem	Resolution
NAS shows connections from both vmk0 and vmk1 addresses	This may require one final reboot to clear. The reboot completed after creating vmk0 still had vmk1 as designated for management traffic. Those connections may live until no longer used or another reboot. This should only be an issue if NFS permissions are set to specific IP addresses.