Comments on: ESXi 4.1 and the 9000 Byte MTU (on vmk0)

By: Joseph P

Joseph P — Sat, 04 Feb 2012 14:40:05 +0000

Very simple and effective.

Thanks dude ! Help me a lot !

By: Lee

Lee — Tue, 19 Jul 2011 17:51:28 +0000

Great article, helped me alot

By: Craig

Craig — Mon, 27 Jun 2011 00:52:01 +0000

great write up dude. The only tip I give, depends on if you have the Lic for this, but it pays to dump the vswitch concept and create a distributed switch, under distributed switch you can manage all the VLan and port groups quite well.

as a personal favorite can create a direct link to the dedicated nic for man or NFS traffic into two separate port groups, one for service console the other for (the rest), talking home lab style here . Anyway, when you create the new vmkernel it asks for which port group to assign (in distributed switch).

And if you want to migrate it to a virtual switch its a one click job.

By: WS

WS — Fri, 24 Jun 2011 05:47:00 +0000

Great article. What if you have more than one vSwitch. Where will the new vmk be created? Is there a way to specify vSwitch in the CLI?

By: Dan

Dan — Wed, 04 May 2011 18:02:33 +0000

I’m running ESXi 4.1

I created my vSwitches and modified the MTU using the PowerCLI, executing the following:

esxcfg-vswitch.pl -m 9000 vSwitch2

Is that acceptable

By: VCAP-DCA Study guide – 6.3 Troubleshooting Network Performance and Connectivity | www.vExperienced.co.uk

Mon, 25 Apr 2011 09:59:59 +0000

[...] ping -s Here’s a good blogpost about MTU – check the [...]

By: Mohit

Mohit — Fri, 21 Jan 2011 01:15:24 +0000

Well i think i spoke too soon. When i deleted vmk0 and recreated it with mtu 9000 i could access the datastore. But after sometime i did not loose connectivity but i could not access data on the datastore.

her eis my config

vmk0 192.168.50.75 mask 255.255.255.0 m 9000 gw 192.168.50.1
vmk1 192.168.112.9 mask 255.255.255.0 m 9000 gw 192.168.50.1

VLAN 112 – 192.168.112.0/24 mtu =9216
VLAN 50 -192.168.50.0/24 mtu =1500

both on vSwitch0 with two 10G Nics.

When i recreated vmk1 I could access my datastore. however this is not stable. Atleast i feel so.

vSwitch has mtu 9000
nics have mtu 9000

I want my esx servers to use vmk1 to mount NFS Share. however it looks like its going through the .50.75 interface. I noticed this when i changed te right on the nfs share to only 112. it mounted the share but it could not write to it. I had to add 50.75 and then it could write to it.

What could be the problem???

By: me@gavinadams.org

me@gavinadams.org — Thu, 20 Jan 2011 19:42:20 +0000

Glad you got it working. My example is basically to help people test in a lab and to see what the process is for changing the MTU. Please follow VMware’s design guidelines for production environments.

Normally this have storage traffic onto a separate physical NIC, and do the same with host traffic. In my home config, I have 2 or 3 NICs teamed together for all use (vMotion, storage, and the various networks for the guests).

To verify, see if the command on the switch “sh controllers ethernet-controller iface” (where iface is the interface of the NAS or ESXi host) and see what shows up for different MTU sizes.

I don’t have a Cisco switch available to test jumbo frames on – poor only 2924′s here.

By: Mohit

Mohit — Thu, 20 Jan 2011 19:27:56 +0000

So i looked at my oncfiguratiosn and i saw the i had vmk0 for management on mtu 1500 and vmk1 for nfs on mtu 9000. The moment i chnaged the vmk0 to mtu 9000 i could access the datastore. however, i am still not sure if i am getting end to end jumbo frame traffic.

You mentioned that ” Management Network – vmk0 – Management traffic – managed via vCenter / vMA, and as lowest vmk, used by NFS for storage on the same subnet.”

I have my storage on a different subnet (vmk1) should me NFS traffic go through vmk1?? and not worry about vmk0.

By: me@gavinadams.org

me@gavinadams.org — Thu, 20 Jan 2011 12:40:09 +0000

Hi Mohit,

It sounds like there may still be a packet size problem somewhere between the ESXI host and the EMC. I’m not familiar with the Cisco switch or the EMS NAS, but slow performance can be indicative of mismatched MTUs.

The first thing I would try is to verify that the MTU is properly set, and operational, between the endpoints. If your NFS is UNIX/Linux based, use the ping command to the ESXi host to verify the MTU.

In Linux, it would be something like: ping -s 8500 -M do hostname
Where -s is the packet size and the “-M do” ensures the don’t fragment flag is set. You should get microsecond (less than 1ms) response times. If you see something like:

… icmp_seq=1 Frag needed and DF set (mtu = 9000)

That will indicate an issue.

If the NFS host doesn’t have a similar ping command, setup a host on the same VLAN as the ESXi vmkernel and NFS server. Make sure it also has a MTU of 9000 set, and ping the ESXi and NFS servers from there.

For Windows, the ping command would be: ping -l 8500 -f hostname

In know for some switches you may need to restart in order for the MTU to take affect. What version of IOS are you running, and what supervisor do you have?