Hi!, in the last post of the year, we will see hot to use the esxtop utility to analyze storage performance issues that affect HBAs, LUNs, and VMs.
We start the esxtop utility and view storage adapter (HBA) statistics on ESXi host to determine which adapter experiences the highest disk activity.
- Open an SSH session to some ESXi host (By default, we are presented with CPU statistics)
- In the esxtop display, enter d to view the disk adapter information.
The output should be similar to the example.
3. Enter f to display the Fields menu.
The asterisks next to the A, B, C, E, and G fields signify that statistics in these fields are shown in the disk statistics display. These fields act like a toggle. If we enter a, the A fields are shown in the display. We can turn on (and off) any of the fields by toggling the letter.
4. Press any key, such as Return, to get back to the disk statistics display. The table shows the statistics that you see when you select the D, E, and G fields.
First, we must interpret the statistics that are shown in the storage adapter display.
Which HBA might be the cause of slow storage performance?
vmhba65, because this HBA show high IOPS
What condition is degrading storage performance?
A high number of read commands are being issued from vmhba65
Monitor Performance by Storage Device
- In the esxtop utility, enter u to view information about the storage devices (LUNs).
We view storage device (LUN) activity on ESXi host.
2. Enter f to display the Fields menu.
3. Verify that only the A, F, G, and I fields are selected.
The table shows the statistics that you see when you select the F, G, and I fields.
4. Interpret the statistics that are shown in the storage device display.
Which storage device seems to be affected?
The device with the storage identifier naa.60003ff44dc75adcaf760d6a0ac8e3ef
5. Enter the command to view the datastore name of the affected storage device.
esxcli storage vmfs extent list
Monitor Storage Performance by VM
We use the VM disk view in esxtop to monitor the disk activity on ESXI host.
- In the esxtop display, enter v to view information about the VM disk activity.
We should see a similar output to the example.
2. In the Fields menu, verify that only the B, C, D, E, I, J, and K fields are selected.
The table shows the statistics that you see when you select the I, J, and K fields.
3. Interpret the statistics that are shown in the VM disk display.
Which VM or VMs might be contributing to slow storage performance?
Win-5, Win-5 and Win-6 are running several read commands per second. However, these VMs do not seem to be causing a significant amount of latency because the load is still less.
What possible solutions can help you get better performance?
Add another VMkernel port and vmnic for software iSCSI multipathing and set the multipathing policy to Round Robin. Also, check the DAVG value or latency values for the VM. Enable Storage I/O Control and set the value to 5 ms. Migrate one or two VMs to another datastore.
Closing Notes
For useful information about using the esxtop utility, see the following references.
Using esxtop to identify storage performance issues for ESX/ESXi (1008205)
Interpreting esxtop Statistics
Identifying disks when working with VMware ESXi/ESX (1014953)
Happy New Year! See you next…