VMware NSX 6.2: From zero to full deployment

VMware NSX is the SDDC technology of the future. What ESX was once for Servers, NSX is now for Networks. I highly encourage everyone to make yourselves familiar with this technology. NSX with all its features is quite complex, but the entry point is quite simple and requires only basic vSphere and networking skills. This beginners guide explains how to deploy NSX in your homelab even with limited physical ressources by downsizing NSX Manager and NSX Controller VMs. The guide starts at zero and quickly explains how to deploy NSX and connect your first Virtual Machine to a VXLAN based logical switch that is able to communicate to the physical world through an NSX Edge Gateway.

What do you need to create the Lab?

  • vCenter 6 with some physical ESXi Hosts
  • vSphere Distributed Switch (dvSwitch)
  • NSX Manager Appliance (Download: NSX 6.2.2)
  • There is no special physical Switch requirement

This guide is not intended as a comprehensive guide to fully understand all aspects of NSX. It’s just a quick deployment guide with some tweaks for low resource Homelabs.

  1. Prepare your vSphere Platform (vCenter Server Appliance, ESXi Hosts, Cluster and dvSwitch).  I recommend to update everything to the latest version available, which is currently vSphere 6.0 Update 2 and NSX 6.2.2.
  2. Configure a NTP Server for ESXi Hosts and the vCenter Server to ensure time consistency.
  3. Deploy the NSX Manager Appliance, provided as OVA (Download: NSX 6.2.2). The deployment wizard is pretty straightforward, nothing special here if you have ever deployed a template. Make sure to configure a NTP Server, and enable SSH.
  4. The NSX Manager is preconfigured with 4 vCPU and 16GB Memory. That might be a little oversized for a tiny homelab. If your hardware has limited resources, shutdown the NSX Manager and reduce the configured resources. 2 vCPU and 8 GB Memory should be possible without any impact (Of course, this is not supported by VMware). However, you can set it even lower but keep an eye on the memory consumption. You can check it by logging in to the NSX Manager with SSH (password set during ova deployment) and run the command “show system memory”:
  5. The NSX Manager needs to be registered with the vCenter Server. Open the NSX Manager with a browser and login as admin, with the password configured during the deployment.nsx-manager-webinterface-login
  6. Navigate to Manage vCenter Registration
  7. Configure the Lookup Service to the IP Address of your Platform Services Controller and the vCenter Server connection to your vCenter Server. For vCenters with an embedded PSC, the lookup service runs on the vCenter Server itself. The following information are required:
    – Lookup Service IP
    – Lookup Service Port (Default: 7444)
    – SSO Admin Credentials
    – vCenter Server Address (IP or DNS Name)
    – vCenter Server Admin CredentialsVerify that both Services are connected:
  8. Login to the vCenter Server Web Client. You should now see the Network & Security icon where NSX is configured. Open it:
  9. The first part of the configuration is to deploy NSX Controller nodes which are representing the Control Pane of NSX. NSX Controllers are Virtual Machines. At least 3 NSX Controllers are required for redundancy. Click the + symbol to deploy the first controller.
  10. Select the options for the NSX Controller until the IP Pool configuration and click Select to select or create an IP Pool.
  11. Click New IP Pool…
  12. Enter Network details. This should be the same layer 2 network as your vCenter Server, ESXi Hosts and NSX Manager. NSX Controllers are automatically configured with IP Addresses defined in the Static IP Pool. Configure the range with the number of IP Addresses you want to reserve for NSX Controllers.
  13. Finish the IP Pool configuration, select it and finish the Controller configuration:nsx-install-finish-controller-configuration
  14. Wait until the deployment has been finished.nsx-installation-deploying-nsx-controller
  15. Deploy 2 additional controllers in the same IP Pool and wait until all controller nodes are deployed and connected.
  16. NSX Controllers are preconfigured with 2 vCPU and 4GB Memory. For your tiny (unsupported) homelab you can also reduce NSX Controller resources, but this is a little bit more tricky because the “Edit settings” function is blocked by the vCenter. To disable vCenter Server protection you have to delete respective entries from the VPX_DISABLED_METHODS table. This method is also described by Tom Fojta.
    – SSH to the vCenter Server
    – Enable Bash
    – Connect to the vCenter Postgres Database
    – Identify Object IDs
    – Delete entries
    – Restart vCenter Server Service

    Command> shell.set --enabled True
    Command> shell
    vcsa:~ # /opt/vmware/vpostgres/current/bin/psql -U postgres
    psql.bin (9.3.9 (VMware Postgres release))
    Type "help" for help.
    postgres=# \connect VCDB
    You are now connected to database "VCDB" as user "postgres".
    VCDB=# select * from VPX_DISABLED_METHODS;
     entity_mo_id_val |          method_name           | source_id_val | reason_id_val
     vm-68            | vim.VirtualMachine.reconfigure | vShield_SVM   | vShield_SVM
     vm-81            | vim.VirtualMachine.reconfigure | vShield_SVM   | vShield_SVM 
     vm-82            | vim.VirtualMachine.reconfigure | vShield_SVM   | vShield_SVM 
    (1 rows)
    VCDB=# delete from VPX_DISABLED_METHODS where entity_mo_id_val = 'vm-68';
    DELETE 1
    VCDB=# delete from VPX_DISABLED_METHODS where entity_mo_id_val = 'vm-81';
    DELETE 1
    VCDB=# delete from VPX_DISABLED_METHODS where entity_mo_id_val = 'vm-82';
    DELETE 1
    VCDB=# select * from VPX_DISABLED_METHODS;
     entity_mo_id_val | method_name | source_id_val | reason_id_val
    (0 rows)
    postgres=# \q
    vcsa:~ # service-control --stop vmware-vpxd
    vcsa:~ # service-control --start vmware-vpxd


  17. After the vCenter Server service has been restarted you should be able to edit NSX Controller Resources. It is possible to resize NSX Controllers to 2 GB without any major issues. If you want to go lower, verify memory consumption by logging in to the NSX Controller with SSH and run the command “show system statistics graph memory/memory-used:value”. Memory Usage here is 800MB:nsx-controller-memory-consumption
  18. Now you have to prepare ESXi Hosts to be compatible with NSX. Navigate to Network and Security > Installation > Host Preparation, select your Cluster and click Install. This will install 2 VIB on all ESXi hosts in the Cluster. The installation is completely transparent to virtual machines. Maintenance Mode is not required.
  19. Refresh the vSphere Client to check the Status and wait until the installation has been finished.nsx-host-preparation-finished
  20. To allow ESXi hosts to talk to each other and to the physical network, they need Tunnel Endpoints (VTEP Interfaces). Click Not Configured in the VXLAN tab, configure VXLAN settings and add an IP Pool for VTEP interfaces. Make sure to size the IP Pool according to the number of ESXi Hosts:
  21. Navigate to Network and Security > Installation > Logical Network Preparation > Segment ID, and configure the Segment ID pool to the Number of VXLANs your are planning to use:
  22. To determine the replication boundaries of a VXLAN Network, a global transport zone is required. Navigate to Network and Security > Installation > Logical Network Preparation > Transport Zones, and click +
  23. Add a Global Transport Zone and set the Replication mode to Unicast. This setting allows NSX Controllers to talk to each other without any multicast requirements on physical switches. Select all clusters that needs to be part of the Transport Zone.nsx-installation-transport-zone-configuration
  24. Now the initial NSX configuration is finished and you are ready to configure the first Logical Switch. A Logical Switch is basically a VXLAN Network or Portgroup where Virtual Machines are connected to. Navigate to Network and Security > Logical Switches click +
  25. Name the Logical Switch and set the replication mode to Unicast.
  26. To connect the “virtual” Logical Switch beyond the VXLAN you need a NSX Edge. The Edge Gateway is a Virtual Machine with 2 network interfaces, one connected to the VXLAN and one connected to the outside network. The Edge Gateway acts as Router. Navigate to Network and Security > NSX Edge and click +nsx-add-edge-gateway
  27. Set the installation type to Edge Services Gateway and enter a namensx-edge-configuration-name
  28. Set Admin credentials and enable SSH.
  29. Set the Appliance Size and configure the NSX Edge Appliance placementnsx-edge-configuration-deploymentnsx-edge-configuration-placementnsx-edge-configuration-deployment-finished
  30. Configure Edge Interfaces. Place one interface to your physical connected portgroup and a second to the internal Logical Switch.
  31. Finish the deployment wizard and wait until the deployment has been finished.
  32. The VXLAN is now connected to the physical network. For ease of use I am going to enable DHCP on the logical switch. Doubleclick the new edge gateway and open the DHCP tab.nsx-edge-dhcp-service
  33. Add DHCP Pool configurationnsx-edge-add-dhcp-pool
  34. Activate DHCP and publish changes.nsx-edge-enable-dhcp
  35. Navigate to Network and Security > Logical Switches and connect a Virtual Machine to the Logical Switch

Congratulations! You have successfully deployed NSX in your Homelab.

Mastering VMware Snapshots

What is VMware Snapshot?

VMware Snapshot preserves the state and data of a virtual machine at a specific point in time.

Snapshot state includes the virtual machine’s power state (ex, powered-on, powered-off, suspended) and the data includes all of the files (disks, memory, and other devices, such as virtual network interface cards.) that make up the virtual machine.

Snapshots extremely simplify the Virtual Machine OS management and maintenance tasks by preserving the current state of the virtual machine.  So it allows you to revert to the state before the snapshot is taken.

Windows patching in Physical Server VS VMware Virtual machine

Physical workload

Before performing the OS patching or software upgrade, You need to take a backup such as OS Image level backup to restore the OS in case of any corruption is happened during the patching and also it is a time-consuming process.

Restore is not even guaranteed recovery if in case of the backup file is corrupted.

VMWare Virtual machine

In VMware Virtual Machine, Snapshots simplifies the process to capture the Point-in-time copy of the virtual machine without the need of any third party backup software.

It also allows you to take multiple snapshots to preserve the multiple states of the virtual machine. It also simplifies the restore process.

You just need to select the snapshot which is taken before patching to revert to the particular state of the virtual machine in case any issue happened during the patching.

How to Create VMware Snapshot

VMware Snapshot can be simply created using vSphere client connected to ESXi host or vCenter Server and through vSphere web client. Even VMware snapshots can be created using the command line and powerCLI scripts. Let’s take a look at how to create Virtual Machine Snapshot from vSphere Web Client.

Connect to your vCenter server using vSphere Web Client and Right-click the virtual machine -> Take Snapshot

VMware Snapshot

Name: Specify the name for the snapshot. Name of the snapshot should understandable easily from its name.

Description: Provide a description for this snapshot. You can even specify the Date and time to best identify its age.

As of now, Just go with the snapshot without Virtual Machine memory. I will explain the difference between the VMware snapshot with Virtual Machine’s memory and  Snapshot without Virtual Machine’s memory.  Click Ok to create the snapshot.

VMware Snapshot_2

You can validate that the “Create Virtual Machine Snapshot” tasks under your recent tasks in vSphere Web Client. It is completed and Snapshot is created.

VMware Snapshot_2.1

Difference between VM Snapshot with and without Virtual Machine Memory

Snapshot with Virtual Machine Memory

If you select the checkbox “Snapshot the virtual machine’s memory” during snapshot creation, Then memory flag will be set to 1 or true, a dump of memory state (internal state) of the virtual machine is also included in the snapshot.

It takes little longer time to create the memory snapshot as compared to snapshots with memory. Virtual Machine needs to be Powered on state to take the memory snapshot. If the virtual machine is in powered off state, Memory snapshot option will be grayed out.

This memory snapshot will allow you to restore the virtual machine to the same state as it was when a snapshot was taken. In Simple terms, it captures the live state of the virtual machine

For Example, you are modifying a Microsoft word Document file and memory snapshot is taken at that time. After some time VM goes into Blue screen. You want to restore the snapshot from the memory snapshot taken prior to the blue screen issue. Snapshot with memory will revert the VM into powered on state along with the opened MS word file.

VMware snapshot with Memory

Snapshot without Virtual Machine’s Memory option:

Snapshot taken without memory option will not capture the live stateof the virtual machine. Snapshot creates crash consistent disks, which you can be used to restore the virtual machine to the state prior to the snapshot but it won’t revert the VM into same power state as it during snapshot creation.

Virtual Machine can be either in Powered on or Powered off state to take VM snapshot without memory option.

Let’s say I have taken a snapshot when the virtual machine is powered on and before the patch upgrade. After I revert back to the snapshot taken without memory, the Virtual machine will be restored to the same data and state but power state of the virtual machine won’t be Powered on. It will be powered off. You need to manually power on the VM after reverting the snapshot.

VMware snapshot without Memory

VM Snapshot with Quiesce  Guest File System 

If you select the option “Quiesce guest file system” when taking snapshot, <quiesce> flag is 1 or true, It quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system’s in-memory cache to disk, or other higher-level application-specific tasks.

The virtual machine needs to be “powered on” State to take a snapshot with the quiesce file system option and also it requires VMware Tools to be installed on the Guest OS  to quiesce the file system in the virtual machine.

Note: Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup. Quiescing is not necessary for memory snapshots; it is used primarily for backups.

VMware Snapshot


How does VMware Snapshot work?

  1. When you initiate the request to create, delete or revert snapshot operation via client (Web Client, vSphere Client or Power CLI), Request will be sent from the client to Server using VMware API
  2. Snapshot create, delete or revert request will be forwarded to the ESXi host where that particular virtual machine is running. This process will happen only when initiating the snapshot create, delete or revert request from vCenter Server.  This will be skipped if you sent create snapshot request by directly connecting to the ESXi host.
  3. If you select Snapshot with Virtual machine’s memory option, ESXi hostwrites the memory of the virtual machine to the disk. the virtual machine is stunned throughout the duration of time the memory is being written.
  4. If you select snapshot with the quiesce guest file system option, the ESX host requests the guest operating system to quiesce the disks via VMware Tools. Depending on the guest operating system, the quiescing operation can be done by the sync driver, the vmsync module, or Microsoft’s Volume Shadow Copy (VSS) service.
  5.  ESXi host makes the modification to the virtual machine’s snapshot database file (.vmsd) and it reflects the changes in the snapshot manager of that virtual machine
  6. ESXi host calls Virtual DISK API functions to make changes to the child disks (-delta.vmdk and .vmdk) files and to the disk chain.

When you create a snapshot, The state of the virtual disk at the time of snapshot is preserved and writes to the VMDK file is not allowed. The system creates an additional VMDK file called delta disk for each VMDK disk in the datastore and allows to write any changes to that delta disk. The delta disk represents the difference between the current state of the virtual disk and the state that existed at the time the previous snapshot was taken.

If multiple snapshots are taken, Delta disks will be created for each of VM disk of the every snapshot and it can represent the difference between each snapshot.

When you delete the snapshot, The changes between snapshots and the previous disk states are merged and all the data from the delta disk that contains the information about the deleted snapshot is written to the parent VMDK disk. The amount of time it takes to commit or delete snapshots depends on how much data the guest operating system has written to the virtual disks since the last snapshot was taken.

Virtual Machine Snapshot Files

Virtual machine snapshot consists of multiple files. Virtual Machine snapshot mainly includes the following:

Settings state : Virtual machine’s settings (.nvram & .vmx) and power state

Disk State: state of the virtual machine’s associated disks

Memory state: Contents of virtual machine’s memory (Only if memory snapshot is selected)

Below are the some of the files which comprise of snapshot:







Below is the comparison of  Virtual machine files in the virtual machine directory before and after creating the first snapshot of the virtual machine.

VMware Snapshot

You can see the Delta disk information from the SSH session of ESXi host when browsing towards the virtual machine directory.

VMware Snapshot_4

Let’s discuss in detail about the what does each of the snapshot file means and purpose of the each file.


  • VM_name.vmsd is the snapshot list file and is created at the time that the virtual machine is created. It will be present in the Virtual machine directory regardless of the snapshot is present or not for that virtual machine.
  • The .vmsd file stores the names, description, and relationships for all of the virtual machine’s snapshots.
  • It maintains snapshot information for a virtual machine so that it can create a snapshot list in vSphere Web Client. This information includes the name of the snapshot .vmsn file and the name of the virtual disk file.
  • You can validate the size of the .vmsd file is grown after the snapshot is taken from the above screenshot which compares the files before and after VMware snapshot creation.

VMware Snapshot


  • VM_name-snasphot.vmsn is the snapshot state file and is used to store the state of the virtual machine when a snapshot is taken.
  • New .vmsn will be created for every snapshot and will be deleted when that snapshot is deleted
  • Size of the .vmsn varies based on the option selected during snapshot creation. For example, If you have selected Virtual machine’s memory during snapshot creation, it increases the Size of .vmsn file
  • I have 2 snapshots called “Snap_1″ & “Snap_2” for the virtual machine named “Redhat-1”. It has 2 .vmsn files (Redhat-1-snapshot1.vmsn & Redhat-1-snapshot2.vmsn) is created  in the virtual machine directory

VMware Snapshot



  •       VM_name-00000#.vmdk  is the disk descriptor file. This small text file contains information about the snapshot and snapshot disks.
  •        It gets created for each of the VMDK file, when you take the snapshot. For example, In the below screenshot, I have VM called “Redhat-1” has 2 VMDK’s (Redhat-1.vmdk & Redhat-1_1.vmdk), After taking first snapshot called “Snap_1”, It has created “Redhat-1-000001.vmdk & Redhat-1_1-000001.vmdk” and after second snapshot “Snap_1”, It created “Redhat-1-000002.vmdk & Redhat-1_1-000002.vmdk“


VMware Snapshot_7

Each VM_name-00000#.vmdk  updates the information about its Parent disk information. For example, For the First snapshot, “Redhat-1-000001.vmdk” updated its parentFileNameHint as the actual base disk (actual VMDK file).

After Second Snapshot, “Redhat-1-000002.vmdk” is updated with the ParentFileNameHint as “Redhat-1-000001.vmdk” which is the disk descriptor file created for the snapshot. So for the second Snapshot, Parent is the first snapshot and for the first snapshot, parent is base disk.

VMware Snapshot

When you checked from the virtual machine properties -> Click on Hard Disk -> you can see the Hard disk is mapped with the latest snapshot disk descriptor (VM_name-00000#.vmdk)

VMware Snapshots


  • VM_name-00000#-delta.vmdk is the delta disk file.
  • State of each virtual disk of the virtual machine is preserved when you take a snapshot of a virtual machine.
  • Virtual Machine stops writing to its VM_name-flat.vmdk file and all the writes will be redirected to delta disk “VM_name-00000#-delta.vmdk.
  • It gets created for each of the flat-vmdk file, when you take the snapshot. For example, In the below screenshot, I have VM called “Redhat-1” has 2 VMDK’s (Redhat-1-flat.vmdk & Redhat-1_1-flat.vmdk), After taking first snapshot called “Snap_1”, It has created “Redhat-1-000001-delta.vmdk & Redhat-1_1-000001-flat.vmdk” and after second snapshot “Snap_1”, It created “Redhat-1-000002-delta.vmdk & Redhat-1_1-000002-delta.vmdk“

VMware Snapshot


  • .vmem file will be created only you have selected the option “Snapshot Virtual machine’s memory” during the snapshot creation.
  • This file contains the entire contents of the virtual machine’s memory at the time of snapshot creation

VMware snapshot


Managing VMware Snapshots

Snapshots tab allows you to manage the virtual machine snapshots such as Revert to (Go To), Edit Snapshot, Delete Snapshot, Delete All Snapshots

Managing VMware Snapshots

In the Snapshot tab, you can perform the following actions:
• Edit Snapshot: Edit the snapshot name and description.
 Delete Snapshot: Removes the snapshot from the Snapshot Manager and consolidates the snapshot files to the parent snapshot disk and merge with the virtual machine base disk.
• Delete All Snapshots: Commits all the intermediate snapshots before the current-state icon (You Are Here) to the VM base VMDK file and removes all snapshots for that virtual machine.
• Revert to: Enables you to restore, or revert to, a particular snapshot. The snapshot that you restore becomes the current Snapshot.

We will discuss in In-depth Details of each of the above actions with various Scenarios.

Delete Snapshot

Delete Snapshot operation removes the snapshot from the Snapshot Manager and consolidates the snapshot files to the parent snapshot disk and merge with the virtual machine base disk. Let’s look into the detailed information of Delete Snapshot operation.

I have 2 VM snapshots “Snap_1″ & “Snap_2” for the virtual machine “winsvr”. If I delete the Snapshot “Snap_2”, It will consolidate the snapshot data to its parent snapshot  (Snap_1) disk. In this Example, “Snap_1” is the Parent of Snapshot “Snap_2”.

Before Deleting the Snapshot “Snap_2”, Below are the size of the each disks

Base Disk  (Winsvr-flat.vmdk) -> 21.4 GB

Snap_1 Delta disk (winsvr-000001-delta.vmdk) -> 2.7 GB

Snap_2 Delta disk (winsvr-000002-delta.vmdk) -> 3.4 GB

Delete VMware Snapshot

To delete the Snapshot “Snap_2” Select the particular snapshot and click on “Delete”

Delete Current Sansphot_2

Snapshot “Snap_2” is deleted and now VM has only one Snapshot “Snap_1”

Delete Current Sansphot_3

After the Snapshot removal, Snap_2 data (3.4 GB) is consolidated into its parent snapshot disk (Snap_1 delta disk) and Snap_1 Delta disk (winsvr-000001-delta.vmdk) is grown from 2.7 GB to 6.3 GB.

Delete VMware Snapshot

Delete Current Sansphot_4

Delete Intermediate Snapshot

When you delete a snapshot one or more levels above “You Are Here”, The snapshot state is deleted and that data will be committed to the virtual machine base disk. Refer the Below example:

If you delete the snapshot “Snap_1”, Snap_1 data (winsvr-000001-delta.vmdk) is committed into the Base disk (winsvr-flat.vmdk) and the foundation for the snapshot “Snap_2” will be retained

Deleting VMware Snapshot


Let’s take a look at the step by step demo of deleting Snapshot one or more level above “You are Here” state. I have currently 2 snapshots (Snap_1 & Snap_2) for the virtual machine “winsvr”.

I also have created Folder “File1” before taking snapshot “Snap_1” and also created Folders after creating each of the snapshots.

Delete Intermediate VMware Snapshots

In the below Screenshot, You can notice the 2 delta disks created as part of the Snapshot creation

Base disk -> winsvr-flat.vmdk

Snap_1  -> winsvr-000001-delta.vmdk

Snap_2 -> Winsvr-000002-delta.vmdk

I have also validated the actual usage of each of the above flat.vmdk and delta.vmdk using the below command

du -h servername-flat.vmdk
du -h servername-00000#-delta.vmdk

Delete Intermediate VMware Snapshots_2

To delete the Snap_1, Select the “Snap_1” and click on Delete.

Delete Intermediate VMware Snapshots_3

Once the Snapshot “Snap_1” is deleted, I don’t see the “Snap_1” delta disk “winsvr-000001-delta.vmdk”. As we stated earlier, When you delete a snapshot one or more levels above “You Are Here”, The snapshot state is deleted and that data will be committed into the virtual machine base disk.  

I have deleted the Snapshot “Snap_1” which had 2.8 GB of data. After the Snap_1 deletion, Snap_1 data is committed into the Base disk (winsvr-flat.vmdk) which was grown from 16 GB + 2.8 GB (Snap_1 data) = 18.7 GB

Delete Intermediate VMware Snapshots_5

Delete VM Snapshot below “You are Here”

When you delete a snapshot one or more levels below “You Are Here”, Subsequent snapshots are deleted and you can no longer return to those states. The snap_2 data is deleted.

For example, You have 2 Snapshots (Snap_1 & Snap_2) and your current snapshot is Snap_2, When you revert your virtual machine to snap_1, Snap_2 snapshot data will be deleted and the virtual machine can no longer return to the state of Snap_2.

Delete Snaphot one level above You are Here_3

VMware Snapshot


Delete All VM Snapshots

Prior to vSphere 4 update 2, Delete All Snapshots option from the Snapshot manager would require additional space to perform the operation, incase of committing multiple snapshots.  When using Delete All in the Snapshot Manager, the snapshot furthest from the base disk is committed to its parent, causing that parent snapshot to grow. When that commit is complete, that snapshot is removed and the process starts over on the newly updated snapshot to its parent. This continues until every snapshot has been committed. This can lead to an aggressive use of additional disk space if the snapshots are large.

VMware Updated the algorithm of Delete All Snapshots, Delete all snapshots operation to commit every snapshot of the chain directly to the Base Disk(s) of the virtual machine.  With this new Delete all algorithm,

  • If the Base Disk is thick provision (pre-allocated), no extra space is required for the Delete all operation. The Base Disk will not grow as it is preallocated or thick.
  • If the Base Disk is thin provision (non-preallocated), the base disk will grow only on committing information from the snapshots. Each thin provision disk may grow up to its maximum size as mentioned in the Provisioned Size option in the virtual machine settings for the disk.

All flat and delta files that are used by the chain of snapshots are locked.

Delete All VMware Snapshots_4


I have 2 snapshots called “Snap_1” & “Snap_2”. You can notice the delta files related to the snapshots. Before performing Delete All  opeartion of 2 snapshots. I have checked the actual usage of base disk and each delta disk of both snapshots

Base disk (winsvr-flat.vmdk)  -> 23.4 GB

Snap_1 delta disk (winsvr-000001-delta.vmdk) -> 6.8 GB

Snap_2 delta disk (winsvr-000002-delta.vmdk) -> 3.4 Gb

Delete All VMware Snapshots

To perform Delete All VMware Snapshot, Click on Delete All.

Delete All VMware Snapshots_2

Once Delete All operation is completed, I can notice that all the snapshot disks are committed to the virtual machine base disk (winsvr-flat.vmdk) and size of the base disk is grown from 23.4 GB to 33.2 GB (6.8 GB Snap_1 data + 3.4 GB Snap_2 data)

Delete All VMware Snapshots_3

Revert to Snapshot

Revert to snapshot will restore your virtual machine to the state that they were in at the time when you took the snapshot. If you revert to the VMware snapshot, your virtual machine will be in the powered off state, unless you took the Memory Snapshot. Reverting the memory snapshot will revert the virtual machine to state and it also maintains the live power state also.

Let’s see the step by step demo to understand the Revert to Snapshot option. I have created a  Folder called “File 1” created on the desktop of the virtual machine. File 1 folder is 6.48 GB of Size before taking the snapshot.

Revert to VMware Snapshot

I am creating the snapshot called “Snap_1”

Reverting VMware Snapshots_2

Specify the Snapshot name and description. I am taking the memory snapshot of the virtual machine to revert to live state of the virtual machine.

Reverting VMware Snapshots_3

Once the snapshot “Snap_1” is created for the virtual machine, I am deleting the folder “File 1” from the desktop.

Reverting VMware Snapshots_4

I have deleted the folder “File 1”. Let’s  revert the virtual machine to the snapshot “Snap_1”. To revert to the snapshot “Snap_1”, Select the snapshot “Snap_1” and  Click on “Revert To” to revert to the snapshot.

Reverting VMware Snapshots_5

Click on “Yes” to confirm to revert to snapshot “Snap_1”

Reverting VMware Snapshots_6

Revert to snapshot is completed successfully.

Reverting VMware Snapshots_7

Once Revert to the snapshot “Snap_1” is completed, Virtual machine “winsvr” is reverted to the state when you to took the snapshot. Since I have created a folder “File 1” prior to creating a snapshot. File 1 reappeared on the desktop.

Reverting VMware Snapshots_8

When you revert a virtual machine with snapshot, the current snapshot (delta disk) will be removed and a new one will be created linking to the parent disk.For Example, I have only one snapshot “Snap_1” and it has its delta file called “winsvr-000001-delta.vmdk”. After reverting the virtual machine to snapshot “Snap_1”, winsvr-000001-delta.vmdk  got deleted and new delta file called “winsvr-000002-delta.vmdk is created.

Revert to VMware Snapshot

Reverting VMware Snapshot One level above the Current Snapshot

Let’s take a look at the Scenario of Revert to the snapshot One level above the current snapshot. For Example, You have a virtual machine with 2 Snapshots “Snap_1” & “Snap_2”. Snap_2 is the current snapshot. This scenario talks about reverting your virtual machine to Snapshot 1 “Snap_1” instead of recent snapshot “Snap_2”.

Reverting VMware Snaphot One level above the Current Snapshot

Let’s revert to the Snapshot 1 “Snap_1”. Select the snapshot “Snap_1” and Click on “Go To” to revert the virtual machine to Snapshot 1.


Reverting VMware Snaphot One level above the Current Snapshot_2

When reverting to the Snapshot 1 “Snap_1”, All the files, folders and changes made after the Snapshot 1 will be lost. I don’t see any of the folders created after Snapshot 1.

Reverting VMware Snaphot One level above the Current Snapshot_3

An interesting fact is There is no longer “Winsvr-000002-delta.vmdk” delta file  present and New delta file called “Winsvr-000003-delta.vmdk” is created.

Reverting VMware Snaphot One level above the Current Snapshot_4

Virtual Machine VMDK also mapped with the newly created disk “winsvr-000003.vmdk”

Reverting VMware Snaphot One level above the Current Snapshot

All the active writes to the virtual machine are passed to the newly created virtual machine. You can notice the new delta file 000003-delta.vmdk. The new delta file is growing when new writes are coming. It has grown from 49 MB to 2.8 GB.

Snapshot Consolidation

Snapshot Consolidation is intended to resolve the problems that might occur with the snapshots. Snapshot consolidation is a method to commit a chain of snapshots to base disks when Snapshot manager shows no snapshots exist but the delta files are present in the virtual machine directory on the datatsore.

Snapshot consolidation is a way to clean unneeded snapshot delta files from a datastore. Left out delta files may continue to grow until the virtual machine runs out of datastore space.

With snapshot consolidation, vCenter Server displays a warning when the descriptor and the snapshot files do not match. After the warning is displayed, the user can use vSphere Web Client or vSphere Client to commit the snapshots.

Consolidate VMware Snapshot

To Perform consolidation, Click on Snapshot -> Consolidate

Virtual Machine Consolidation_1

Click on “Yes” to confirm the consolidated operation.

Virtual Machine Consolidation_2

Limitations of VMware Snapshot

Keeping snapshots for a longer duration than recommended can affect the virtual machine performance and also VM snapshot is not supported for some disk types and VM configured with bus sharing. Let’s understand what is the limitation of Virtual Machine Snapshots:

  • VMware Snapshot is not supported for raw disks and RDM Physical mode disks. Snapshot is supported for RDM with virtual compatibility mode.
  • VMware Snapshot does not support for independent disks. Virtual machines with independent disks must be powered off before you take a snapshot. Snapshots of powered-on or suspended virtual machines with independent disks are not supported.

Virtual Machine Snapshot Limitation -Independent Disk

  • Snapshots are not supported with PCI vSphere Direct Path I/O Devices
  • VMware does not support snapshot for the guest operating systems that use an iSCSI initiator in the guest.
  • VMware Snapshots are not supported for a Virtual machine configured withbus sharing.

Virtual Machine Snapshot Limitation - Bus Sharing

  • The virtual machine with VMDK larger than 2 TBs, snapshot operations can take significantly longer to finish.
  • A snapshot can negatively affect the virtual machine performance and Performance degradation is based on How long you keep the snapshot, depth of the snapshot tree and also based on the write rate of the virtual machine and its guest operating system have changed from the time you took the snapshot.
  • Snapshots are not meant for the method of backup and recovery. The snapshot only provides Point-in time image. If the files containing a virtual machine are lost, its snapshot files are also lost.

Best Practices for VMware Snapshots

  • VMware recommends only a maximum of 32 snapshots in a snapshot chain. However, for a better performance, use only 2 to 3 snapshots.
  • Do not use a single snapshot for more than 24-72 hours. Keeping VMware snapshot for longer duration will negatively affect the virtual machine performance
  • Snapshot file continuous to grow in size and when keeping snapshot for a longer duration, this can cause the datastore to grow and even sometimes it causes outage due to datatsore space issues.
  • Ensure Snapshots are deleted after every backup when using third-party backup softwares.
  • Effectively use vCenter Alarms to monitor the virtual machine snapshot and datastore space usage
  • You can also make use of PowerCLI script to report the virtual machine usage and also the age of the virtual machine snapshot.
  • Do Not attempt to increase the disk space of the virtual machine disk, when Snapshot is present. Increasing the disk size, when snapshots are still available can corrupt snapshots and result in data loss.
  • If you using vSphere version prior to vSphere 5.0, Ensure there are no snapshots prior to performing storage vMotion. Storage vMotion is supported for VM with snapshot after vSphere 5.0

VMware Snapshot Advanced Operations

Exclude Virtual Machine disk from VMware snapshot

We need to set the virtual disk to independent mode to exclude the disk from any snapshots. Power off the virtual machine and delete any existing snapshots before you change the virtual disk mode to Independent. Deleting a snapshot involves committing the existing data on the snapshot disk to the parent disk.

One of the real-time use cases  I can think is ORACLE RAC. Basically, in Oracle RAC have multiple disks and it will be shared between the cluster nodes and it will be kind of Multi-writer enabled virtual disks. When performing patching for the ORACLE RAC VM’s, We used to configure all the shared disks as Independent persistent to exclude from the snapshot. So only OS disk will be included as part of snapshot operation.

To configure the virtual disk as Independent disk, Right-click the virtual machine -> Edit Settings -> Virtual Hardware ->Expand the Hard Disk -> Select Independent -Persistent or Independent-Nonpersistent -> Click on OK.

Exclude VM disk from VMware Snapshot

Independent -Persistent: Changes to the disk are immediately and permanently written to the disk as same like a conventional disk in the phsyical computer.

Independent -NonPersistent: Changes to the disk are discarded when you power off or reset the virtual machine.  Changes to the disk are written to and read from a redo log file that is deleted when you power off or reset.


Exclude VM disk from VMware Snapshot_2

Even though I have virtual disks (Hard Disk 1 & Hard Disk 2) in the virtual machine “winsvr”, It created only one delta file for the Hard Disk 1 and no delta file is created for the Hard Disk 2 because it is configured as Independent disk.

Exclude VM disk from VMware Snapshot_3

Control Maximum of VMware Snapshot Per Virtual Machine

Even though The maximum supported amount of snapshots in a chain is 32,  It is recommended not to take more than 2 or 3 snapshots for production virtual machines. How do you control a maximum number of VMware snapshot per virtual machine? Currently, there is no way to control the maximum number of snapshot per virtual machine. There is an undocumented VMX entry as per William Lam’s article, which can control the maximum number of snapshots per virtual machine.

To add the advanced configuration entry to control the maximum number of VMware Snapshot per virtual Machine, Right-click the Virtual Machine -> VM Options ->Edit Configuration

control maximum number of VMware snapshots

Click on Add Row -> Enter “snapshot.maxSnapshots” in the Filed and Enter the number of snapshots in the unit in the Value column.

snapshot.maxSnapshots = n

In the below Screenshot, I have enter 3 to control the maximum number of the snapshot for the virtual machine to 3.

control maximum number of VMware snapshots_2

I already created 3 snapshots for the virtual machine “Winsvr”.

control maximum number of VMware snapshots_3

When i tried to created the 4 snapshot for the virtual machine “winsvr”. It throws the error “msg.snapsot.error-MAXSNAPSHOTS”

control maximum number of VMware snapshots_4

So we are clear that we are able to control the maximum number of VMware snapshots per virtual machine using this advanced paramenter.

Change default VMware Snapshot location

In ESXi 5.0 and later, virtual disk redolog (-delta.vmdk) files for snapshots are placed in the same directory as the parent virtual disk (.vmdk) file. There are some situations like your VMFS datastore don’t have enough storage to accommodate the vm Snapshot or you may not able to power on the vm due to insufficient space to hold your Swap file in the VMFS datastore. You may need to change the default location of your vm snapshot and point it to the different datastore, where it have enough space to store your snapshot related files (virtual disk redolog (-delta.vmdk)). Specifying the working directory for the virtual machine will ensures that subsequently created vm snapshots cause new virtual disk redolog (-delta.vmdk) files to be created in the defined datastore.

We need to make the advanced configuration entry in .VMX file of your virtual machine. I have written detailed article to change the default snapshot location.

Monitor & Report VMware Machine Snapshots

Creating vCenter Alarm to Monitor VMware Snapshot Usage

In production environment, it is necessary to monitor and report the virtual machine snapshots usage information. I will explain the procedure the configure the vCenter Server alarm to monitor the usage of VMware Snapshot and what actions need to perform when particular alarm is triggered.  Let’s see the step by step procedure how to configure vCenter alarm for snapshot usage.

To Configure vCenter Alarm ->Login to the vSphere Web client  ->Right-click the object you want to add the alarm  -> Navigate to Alarms > New Alarm Definition.

vCenter Alarm for VMware Snapshot Usage

Enter an alarm name and description. Select Virtual Machines from the Monitor drop-down. Select “specific conditions or state, for example CPU usage”  in the Monitor for option. Select the checkbox ” Enable this alarm” Click on Next.

vCenter Alarm for VMware Snapshot Usage_2

In the Trigger Type dropdown, click VM Snapshot Size (GB).

vCenter Alarm for VMware Snapshot Usage_3

Select “Is above” in the operator and Specify the Size in GB  for warning condition and critical condition. Click on Next.

vCenter Alarm for VMware Snapshot Usage_4

Select the action you want to occur when the alarm is triggered. I am specifying the action to “send a notification email”

vCenter Alarm for VMware Snapshot Usage_5

Specify the email address to send an notification email.  You can specify the frequency of email for each of the change of alarm condition.  Specify the duration of the repeat actions. Click on Finish to complete the alarm configuration.

vCenter Alarm for VMware Snapshot Usage_6

Power CLI Script – Report Virtual Machines with 3 days older VM Snapshots

I have written an excellent  article for the PowerCLI script which helps us toreport the Virtual Machines with 3 days older VM snapshots and export the output in Excel (.CSV) file. It is always recommended not to keep VM snapshots more than 24- 72 hours. The VM snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. This PowerCLI script could be really useful to identify the Virtual machines with 3 days older snapshot and delete them immediately or raise a concern to the respective team for the VMware snapshot deletion before VM snapshots causes outage to your production infrastructure.

PowerCLI script provides an output in CSV file in the below format. It provides the VM information, snapshot name and date it is created. It reports the virtual machine with snapshots which is older than 3 days.  As per VMware best practices, It is not recommended to keep the snapshots more than 24-72 hours. This script help you to achieve the best practices for VMware Snapshot.

Powercli to report snapshots older than 3 days

Managing VMware Snapshot from Command Line

vSphere Web Client and vSphere Client were always helpful but we need to also understand how to manage VMware snapshots operation from the ESXi host CLI. In this part, I will explain few of the general snapshot related operations from command line. All the below operations are based on “vim-cmd”. Before performing any operation on the virtual machine, we need to get the vmid of the virtual machine. To get the vmid of the virtual machine, Execute the below command

vim-cmd /vmsvc/getallvms

It will lists the vmid, name of the virtual machine, vmx file location, Guest OS, hardware version and annotation, if any. We need to make note of vmid to perform vmware snapshot related operation on that particular virtual machine.

Manage VMware Snapshot from ESXCLI

Get Snapshot Details

To get the Vmware snapshot details  of the virtual machine, execute the below command:

vim-cmd /vmsvc/snapshot.get <vmid>

vim-cmd/vmsvc/snapshot.get 7

It will list the Snapshot name, Snapshot ID, Snapshot description, Date created on and snapshot state. We need Snapshot ID to any operation related to that snapshot.

Manage VMware Snapshot from ESXCLI_2

Delete Snapshot

To delete the snapshot, first get the snapshot details using “snapshot.get” command and note down the snapshot ID. Execute the below command to delete the snapshot called “Snap_1” and Snapshot ID is “1”.

vim-cmd /vmsvc/snapshot.remove <vmid> <snapshot ID>

vim-cmd /vmsvc/snapshot.remove 7 1

Manage VMware Snapshot from ESXCLI_3

Create  Snapshot

We can create the snapshot from the command line. Below is the command line format. We need to specify the snapshotname, snapshot descritpion to create the snapshot. Optionally you can specify the snapshot option such as include memory snapshot and Quiesce file system. Use the below command to create the snapshot.

vim-cmd /vmsvc/snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

vim-cmd/vmsvc/snapshot.create 7 "snap_1" "snapbefore patching" 1 1

Manage VMware Snapshot from ESXCLI_4

I can also confirm from the vSphere web client that “snap_1” is created from the above command.

Manage VMware Snapshot from ESXCLI_5

Revert to Snapshot

You can also revert to the snapshot from the command line. Execute the below command to revert to the snapshot. You need to specify the vmid snapshot ID  and option to suppress the PowerOn of the VM.

vim-cmd vmsvc/snapshot.revert <vmid> <snapshot ID> <suppressPowerOn>

vim-vmd vmsvc/snapsot.revert 7 2 1 or 0

Manage VMware Snapshot from ESXCLI_6

Delete All Snapshots

You can perform the Delete All operation from command line as same as vSphere Web client and vsphere client. Just specify the VMID to delete all the snapshot of the virtual machine. No need to specify any snapshot related information.

vim-cmd /vmsvc/snapshot.removeall <vmid>

vim-cmd /vmsvc/snapshot.removeall 7

Thats’ it. We are done with Delete All option. It consolidates all the snapshot data to the base disk.

Manage VMware Snapshot from ESXCLI_7

What ‘s new with Snapshot in vSphere 6 & vSphere 6.5

Changes to Consolidation procedure in vSphere 6.0

Prior to vSphere 6.0, Consolidation of virtual machine snapshot will create an additional helper snapshot and all the new I/Os will be redirected to this newly created snapshot. Once all the changes are stored in the snapshot disk have been merged into the base disk, the helper snapshot was also committed. Helper snapshot ill also grow considerably during the consolidation operation. If the time to consolidate the helper is within a certain time-frame (12 seconds), Virtual machine will be stunned and consolidate the helper snapshot into the base disk. If it was outside the acceptable time-frame (12 seconds), then the process will be repeated to create new helper snapshot while consolidation of helper snapshot is happening until the helper could be committed to the base within the acceptable time-frame.

If the Virtual machine has more I/O, This process could never successfully consolidate the snapshot chain and helpers.

In vSphere 6.0, VMware introduced new process for consolidation process which uses the mirror driver (which is same as in the storage vMotion). With this mirror driver mechanism, changes to the virtual machine are written to the active VMDK and the base disk (wile protecting the write order) during consolidation. Snapshot consolidation will be completing now in 1 pass with minimal or indeed no helper disks. Consolidation duration is dramatically shorter stun time of the virtual machine. There is an excellent article from Cormac Hogan talking about this Consolidation changes in vSphere 6.0

Changes to Snapshot Format on VMFS 6 on vSphere 6.5

In vSphere 5.5 & vSphere 6.0, Snapshots taken on VMDK’s larger than 2 TB are Space Efficient Virtual Disk (SESPARSE) format. No user interaction is required. The redo logs are automatically created as SESPARSE instead of VMFSSPARSE (delta) when the base flat VMDK is larger than 2 TB. VMFS5 uses the VMFSsparse format for virtual disks smaller than 2 TB.

VMFSsparse is implemented on top of VMFS, and I/Os issued to a snapshot VM are processed by the VMFSsparse layer. Technically, VMFSsparse is a redo-log that starts empty, immediately after a VM snapshot is taken. The redo-log grows to the size of its base vmdk, when the entire vmdk is rewritten with new data after the VM snapshotting. This redo-log is just a file in the VMFS datastore. Upon snapshot creation, the base vmdk attached to the VM is changed to the newly created sparse vmdk. This is nothing but the Delta files.

From vSphere 6.5, SEsparse is a default format for all delta disks on the VMFS6 datastores. If you have VMFS 5 datastore on ESXi 6.5 hosts, Still it will be VMFSsparse (delta files). SEsparse is a format similar to VMFSsparse with some enhancements. This format is space efficient and supports space reclamation. With space reclamation, blocks that are deleted by the guest OS are marked and commands are issued to the SEsparse layer in the hypervisor to unmap those blocks. This helps to reclaim space allocated by SEsparse once the guest operating system has deleted that data.

When I create the Snapshot of the Virtual machine “vcsa-demo-1” which is running on ESXi 6.5 and VMFS 6, You can notice that delta disk format is “VMname-000001-sesparse.vmdk”. I cannot find any delta disk with “vmname-000001-delta.vmdk”

vSphere 6.5 VMware Snapshot Feature

I took the snapshot of the virtual machine “App-1” which is running on ESXi 6.5 but on datastore with VMFS 5. It has delta file with the format “vmname-000001-delta.vmdk”. For sesparse , We need to have VMFS 6 datastore even running on ESXi 6.5

vSphere 6.5 Snapshot Feature_2

Storage vMotion of VM with Snapshot from VMFS 5 to VMFS 6 Datastore

I took the snapshot of the virtual machine “App-1” which is running on ESXi 6.5 but on datastore with VMFS 5. It has delta file with the format “vmname-000001-delta.vmdk”

Sphere 6.5 Snapshot Feature_VM “App-1” is running on the Datastore with VMFS 5.

Storage vMotion of VMware Snapshot

Let’s migrate the vm with snapshot from VMFS 5 to VMFS 6 datastore.

Storage vMotion of VMware Snapshot_2

Storage vMotion of the virtual machine “App-1” is completed.

Storage vMotion of VMware Snapshot_3

Virtual machine with snapshot is running on VMFS 6

Storage vMotion of VMware Snapshot_4

Prior to Storage vMotion to VMFS 6 datastore, delta files was in VMFSsparse format (VMname-000001-delta.VMDK). After Storage vMotion to VMFS 6 datatsore, delta file is chnaged to SEsparse (VMname-000001-sesparse.VMDK)

Storage vMotion of VMware Snapshot_5

Similar to that, When a VM with a vmdk of the size smaller than 2 TB is migrated to VMFS5, the snapshot format changes to VMFSsparse. You cannot mix VMFSsparse redo-logs with SEsparse redo-logs in the same hierarchy.
That’s it. We are reaching the end to this deep dive article to master VMware Snapshot. I hope that I have covered almost all aspects of VMware Snapshots and its terminologies and will update the article regularly if any new changes to VMware Snapshot. I hope this is informative for you. Thanks for Reading !!! . Be social and share it in social media, if you feel worth sharing it.

vSphere 6.5 Blog posts:

What’s new in NSX 6.4.x ?

VMware’s latest version of NSX for vSphere has just been released, version 6.4.0, bringing with it a number of changes operations and security services, updates to the NSX interface as well as some enhancements to the Edge appliance.

For an incremental release, there has been a lot of new features added.

Security Services

  • Identity Firewall

Identity Firewall (IDFW) support for user sessions on remote desktop and application servers (RDSH).  NSX can now apply rules based on users on the same shared machine. Adding further micro segmentation capability to Horizon, RDSH and Citrix environments.

  • Distributed Firewall

NSX Distributed Firewall (DFW) previously worked with layer 2 to layer 4 rules, with this update some layer 7 functions have been added.  Once upgraded this allows NSX to inspect inside the traffic flows.  Now application context can be used for micro segmentation rules rather than just using the port.  New services have been added to make it easier to create rules based on applications.

Not all advanced firewall features have been added, for instance, no IDS or IPS features, a 3rdparty security product is still leveraged for these features.  NSX Enterprise licensing is required to be able to leverage layer 7 capabilities in the new release.

Other improvements include DFW rules can now be created in a stateless fashion, AD integration to allow for selective synchronisation speeding up AD updates and Application Rule Manager (ARM) can recommend security groups and policies based on new application ID.  ARM was introduced in 6.3, ARM can leverage real-time flow information to discover communications between applications to be able to help build a security model around that application.


  • User Interface       

With this version, a new HTML5 plugin has been introduced to support the vSphere HTML Client.  Features developed in HTML5 still remain compatible with the vSphere Web Client to provide a seamless experience.  This move aligns other VMware products developing HTML5 interfaces.

  • Upgrade Coordinator

The upgrade coordinator is a wizard driven upgrade review of the NSX system, the admin can choose the upgrade type and the wizard will guide the admin through the correct planning, reducing errors and increasing visibility into the upgrade process of each component.

  • System Scale

A new feature added to the client, this provides a dashboard with visibility into the current system scale capacity for various objects within NSX.  For instance, the dashboard will list the amount of currently configured hosts for NSX or the number of created Logical Switches then provide the maximum number of supported instances to easily show the admin how close they are to the limit.  Warnings and alerts can be configured when these limits are approaching or have been exceeded.

  • Central CLI

A central CLI has been added for logical switches, distributed firewall and logical router function to provide central access and troubleshooting.

  • API Improvements

REST API now supports JSON for API calls giving the admin a choice now between XML or JSON, XML does still remain the default.  JSON format support for POST, GET, PUT and DELETE operations.

Core Features

  • NSX Edge Enhancement

Enhancement to the Edge load balancer health check, adding three new health check monitors – DNS, LDAP and SQL as well as some further enhancement to existing monitors.  This provides deeper load balancing monitoring for more applications.

Generic Routing Encapsulation (GRE) is now supported in the Edge.  GRE must be configured via the API initially and is supported on the ESG uplink interface only.  Initially, only BGP over GRE is supported.

  • Support

A new Support Bundle tab has been added to the UI to be able to collect the required logs via a single click that can then be easily downloaded.  Multiple syslog servers can now be configured where previously only one could be, now up to five syslog servers can be configured.

  • NAT64

NAT64 facilitates communications between IPv6 and IPv4 hosts by using a form of NAT.  Stateful NAT64 is supported and only supported on ESG uplink interfaces

Upgrade Considerations

As ever with all VMware upgrades it’s critical the VMware Product Interoperability Matrix is referred to before proceeding with any upgrade.  Review the documentation from VMware including the published release notes in case a specific upgrade issue has been reported.  When upgrading NSX, a full NSX upgrade must be performed including the host upgrade which upgrades the NSX host VIBs.  Once upgraded, NSX cannot be downgraded, a backup of NSX manager is recommended before any upgrade.

vSphere 6.0 must be update 2 onwards and vSphere 6.5 must be 6.5a onwards, critically vSphere 5.5 is not supported with NSX 6.4. 


NSX 6.4 continues VMware’s NSX vision to cover the on-premises, EUC, support of for “New-App” framework and public cloud.  This upgrade brings advanced micro-segmentation, ease of use and scalability as well as bringing core feature enhancements all from an incremental upgrade.

Additional features can simply be added following an upgrade realising a true benefit of running a Software Defined Network such as NSX.

What’s New in VMware vSphere 6.5

The big virtualization news from the past week or so was definitely the release and General Availability of VMware vSphere version 6.5, available for download now.

New features include:

  • Improvements to vCenter Server 6.5
  • Transition to Web Client-only
  • Improved host management
  • Enhancements to VMware Tools
  • vRealize Operations Manager updated to 6.4
  • Enhancements to the API, specifically for developers and automation
  • Security enhancements (including VM-level encryption)
  • Improvements to VMware HA and DRS
  • Storage-related enhancements (including automated UNMAP)
  • Networking enhancements

I’ll review each of these areas and also go through some caveats — especially around compatibility with other VMware products.

Improvements to vCenter 6.5

vCenter Server 6.5 has several new features not available in previous releases, including support for more operating systems. vCenter Server 6.5 is now supported on Microsoft Windows, macOS, and Linux without the need for any plugins.

VMware is clearly favoring use of the vCenter Server Appliance over the Windows-installable version. There are some new features that are only available on the vCenter Server Appliance. These include:

  • Addition of a Migration Tool — The vCenter Server 6.5 Installer has a built-in Migration Tool. This tool not only makes it easier to migrate from vCenter Server 5.5 or 6.0 to 6.5, but also makes it easier to migrate to the vCenter Server Appliance 6.5 without the need to manage a separate Windows server for vSphere Update Manager. During a migration, the tool will migrate vCenter Server configuration, inventory, and alarm data by default.
  • Improved Appliance Management — vCenter Server Appliance 6.5 exposes additional configuration data for CPU, memory, network, database statistics, disk space usage, and health data, reducing reliance on the command-line interface for simple monitoring and operational tasks.
  • vCenter Native High Availability — vCenter Server Appliance 6.5 provides a built-in ability to cluster itself for high availability. It does this by creating active, passive, and witness nodes that are cloned from the existing vCenter Server Appliance instance. A vCenter HA cluster can be enabled, disabled, or destroyed at any time. A vCenter HA cluster keeps the active and passive nodes in sync using native Postgres SQL synchronous replication for the vCenter Server database and a separate asynchronous file system replication mechanism for data outside of the database.
    A screenshot of the vCenter HA configuration page is provided below.
  • Native Backup and Restore Functionality — This out-of-the-box feature allows users to back up vCenter Server and Platform Services Controller appliances directly. This feature supports vCenter Server Appliance instances with either embedded or external Platform Services Controller instances.
    A restore is launched from the same ISO from which the appliance was originally created or upgraded. A new vCenter Server Appliance instance is deployed and then ingests the backup files. The vCenter Server UUID and all configuration settings are retained.

vSphere Web Client Only

Back in May of this year, VMware announced that the vSphere Client application (many of us called it the “fat client”) would be phased out in favor of the HTML5 web client (what, in the past, many called the “thin client”). vSphere 6.5 makes the next step in this transition — no vSphere client ships with version 6.5.

While development is still continuing on the HTML5 web client, administrators will need to use the Adobe Flash-based web client for now.

A number of changes have been made to the web client, mostly to the locations of some settings and workflows. Theses changes bring the client more into line with the expectations of administrators, and make the UI more intuitive while reducing the number of clicks required to complete common administrative tasks.

Improved Host Management

vSphere 6.5 adds more capabilities to make it easier to patch, upgrade, and manage the configuration of ESXi hosts. Theses features include:

  • vSphere Update Manager — Generally viewed as the best way to keep ESXi hosts up to date, the vSphere Update Manager has been fully integrated into the vCenter Server appliance in 6.5. This integration eliminates the need for additional resources required for an additional virtual machine, and the database dependencies of previous versions. The new Update Manager information is managed by the vCenter vPostgress installation, although it is stored using a separate schema.
    In addition to patching and upgrading ESXi hosts, the vSphere Update Manager functionality in 6.5 can be used to update VMware Tools and VM compatibility (virtual hardware versions). Linux VMs are no longer needlessly rebooted after updating VMware Tools.
    Customers not ready to move off of the Windows version of vCenter server can still install Update Manager 6.5 in a separate Windows VM. This Update Manager VM can be connected to a vCenter Server instance on Windows, but not to a vCenter Server Appliance instance.
  • Host Profiles — This feature was first introduced in vSphere 4, and has matured with each successive version. 6.5 adds improvements that make it easier to manage and verify profiles and offer more informative compliance checks.
  • Auto Deploy — Auto Deploy allows ESXi hosts to use industry-standard PXE technology to boot from the network rather than from local disks. Auto Deploy enables quick patching and upgrading of hosts, as well as the rapid provisioning of new hosts in a cluster. Auto Deploy has been made easier to manage in 6.5 with the addition of a full-featured graphical user interface — in the past, administrators were required to use VMware PowerCLI to create and manage deploy rules, or to customize ESXi images. Additionally, several enhancements improve both the performance and resiliency of Auto Deploy in 6.5.

VMware Tools Enhancements

Several enhancements have been made to VMware Tools (a collection of in-guest drivers and agents that help optimize VM performance and increase manageability). These include:

  • Signed ISO Images — VMware Tools installers are distributed as ISO images that can be mounted to individual VMs to install or upgrade. ESXi 6.5 automatically verifies these OS images via their cryptographic signatures each time they’re read.
  • Tools Split for Current and Legacy OSes — VMware Tools 10.1 and future versions will be available for OEM-supported guest operating systems only. Guest OSes no longer supported by their vendors will be “frozen” at VMware Tools version 10.0.12. This frozen VMware Tools version will not receive future feature enhancements.
  • Improved Detection of Availability of Update Installers — vSphere 6.5 performs automatic checks for newer VMware Tools installation images. When found, VMs display an alert that an update is available.

vRealize Operations Manager updated to 6.4

With vSphere 6.5, vRealize Operations Manager has also been updated — to version 6.4. (I wish VMware would stick to the policy they said they would and keep version numbers in sync so as to make things less confusing…)

vRealize Operations Manager 6.4 adds three new dashboards:

  • Operations Overview — This shows environment-based information such as inventory summary, cluster update, overall alert volume, and widgets displaying the top 15 VMs experiencing CPU contention, memory contention, and disk latency.
  • Capacity Overview — This shows capacity totals, capacity in use for the CPU count, and storage-based metrics.
  • Troubleshoot a VM — This enables a view of individual VM-based information such as its alerts, relationships, and metrics based on demand, contentions, parent cluster contention, and parent datastore latency.

Developer and Automation Interfaces

Key enhancements have been made across the application program interfaces (APIs) and command-line interfaces (CLIs) in vSphere 6.5 to simplify interactions for developers and automation, giving customers choice of access with language bindings and automation tools.

The vCenter Server REST-based API has received new extensions that provide the ability to manage and configure the vCenter Server Appliance and enable basic VM management.

The 6.5 vCenter Server also adds an API Explorer, which provides a new way to discover which APIs are available for use. It also enables users to expand each API call, look at the required fields, understand the request body, and see the available filter information, as well as a list of response messages.

VMware PowerCLI has been updated to be completely module based in 6.5. The following new features have also been added:

  • The Move-VM cmdlet in the core vSphere module now supports Cross-vCenter vMotion.
  • The Open-VMConsoleWindow cmdlet now uses the latest version of the VMware Remote Client.
  • The storage module has been updated to include the ability to perform management tasks for VMware Virtual SAN (vSAN).
  • Additional cmdlets have been added to the storage module to interact with and manage vSphere Virtual Volumes (vVol) replication.
  • The VMware Horizon module has received the largest update, namely a complete rewrite. The module can now be run from anywhere, not just on the Horizon View connection server.

Security Enhancements

vSphere 6.5 adds a number of enhancements for increased security. These include:

  • Virtual Machine Encryption — vSphere 6.5 allows  for the encryption of VMs. Because the encryption occurs at the hypervisor level (as opposed to being a function of the guest OS running on the VM or a feature of the storage), VM encryption will work with any guest OS and datastore type. VM encryption is managed via policy. The policy framework being used integrates with vSphere Storage Policy Based Management (SPBM). Key management is based on industry-standard Key Management Interoperability Protocol (KMIP), offering customers choice and flexibility around key management. VM Encryption takes advantage of the latest CPU hardware advances in AES-NI encryption. The Advanced Encryption Standard Instruction Set is an extension of the x86 instruction set, and provides accelerated encryption and decryption functions.
  • Encrypted vMotion — Encrypted vMotion is enabled on a per-VM basis. It works by encrypting the data traveling over the network, rather than encrypting the network itself. This implements easier and is more flexible. Every packet is uniquely encrypted on the source host and is decrypted on the destination host.
  • Secure Boot Support — vSphere 6.5 introduces Secure Boot Support for both VMs and for the ESXi hypervisor. UEFI Secure Boot is a mechanism that only allows trusted code to be loaded by EFI firmware prior to OS handoff. This trust is determined by keys and certificates managed by the firmware. Implementing this feature for a VM enables secure boot of an EFI-aware OS in that VM.
  • Enhanced Logging — vSphere 6.5 added audit-quality logging. In prior versions, vSphere logs were more focused on troubleshooting rather than on IT operations or security use cases. Logs coming from vCenter Server 6.5 via syslog are now enriched with data from vCenter Server events. These logs clearly show “before” and “after” settings changes. allowing IT and security administrators an enhanced ability to track and troubleshoot issues. These changes do not require increasing the logging level beyond the standard “info” level. They also don’t add any noticeable load to the vCenter Service instance or add to the vCenter Server database, because this info has already been recorded as part of the existing vCenter Server event. The Enhanced Logging feature simply causes this information to be passed along to Syslog. Standard troubleshooting and support logs will be unaffected.
  • Security Automation — The VM Encryption, Encrypted vMotion, and Secure Boot features are all fully automatable via VMware PowerCLI and the vSphere API.

Improvements to VMware HA and DRS

Both VMware High Availability (HA) and Distributed Resource Scheduler (DRS) have been enhanced in vSphere 6.5. These enhancements include:

  • Proactive HA — Proactive HA integrates with select hardware partners to detect degraded components and evacuate VMs from affected hosts before an incident can cause an interruption of service. vCenter Server plug-ins from the hardware vendors provide the health status of hosts’ system memory, local storage, power supplies, cooling fans, and network adapters. If components become degraded, Proactive HA determines which hosts are at risk and places them into a new-to-6.5 state called “Quarantine Mode”. While in Quarantine Mode, VMs are migrated off of the host onto healthy hosts (as long as affinity and anti-affinity rules are not violated, and the performance of the VM will not be negatively affected). New VMs will not be added to hosts that are in Quarantine Mode.
  • HA Orchestrated Restart — Orchestrated Restart improves the recoverability of applications that run across more than one VM. This is done by creating dependency chains for VMs and using those to create VM restart rules. These rules enforce the restart order for VMs within the dependency chain, increasing the chances that an affected application will properly recover when vSphere HA restarts the VMs.
  • vSphere Fault Tolerance Improvements — In vSphere 6.5, Fault Tolerance (FT) integration with DRS has been improved to enable better placement decisions by ranking the hosts based on available network bandwidth and by recommending the datastore in which to to place the secondary VMDK files. Additionally, network latency between the primary and secondary VMs has been greatly decreased.
  • Improved DRS Load Balancing Algorithm — The load-balancing algorithm has worked well in most cases. However, with clusters becoming larger, distribution patterns become normalized, causing “outliers”. An outlier is a host whose utilization is greater than the average utilization of the cluster, but not far enough above that average to have a negative affect. The DRS algorithm has been improved to detect these outliers (by performing pairwise calculations between the most-utilized and least-utilized hosts) and make additional migration recommendations to reduce the variance, resulting in a better overall balance of cluster resources and individual VM performance.
  • Network-Aware DRS — In addition to the 25 metrics already used when making migration recommendations, vSphere DRS in 6.5 also takes network utilization into account. It monitors the Tx and Rx rates of the connected physical uplinks and will avoid placing VMs on hosts it considers to be network saturated (above 80% network utilization).

Storage-Related Enhancements

vSphere 6.5 offers a number of enhancements in its handling of storage. These include:

  • Support for Advanced Format Drives — The standard sector size for disk drives has been 512 bytes. In order to provide larger-capacity drives, the storage industry has been moving towards Advanced Format (AF) drives that use a larger physical sector size of 4,096 bytes (this is also referred to as “4K AF format”).
    vSphere 6.5 provides 512 emulation (512e) mode for VMFS datastores and RDMS. This allows it to work with legacy OSes and applications while still supporting the larger-capacity drives.
    Because it includes major changes that make VMFS metadata 4K-aligned, the 512e mode requires use of the new VMFS version 6 provided with vSphere 6.5.
  • Automated UNMAP — UNMAP is a VMware vSphere APIs for Array Integration (VAAI) primitive that enables the reclamation of dead or stranded space on thinly-provisioned VMFS volumes. In vSphere 6.0, this can be initiated manually by running an ESX CLI command. vSphere 6.5 automates the process by tracking the deleted VMFS blocks and reclaiming that space from the storage array in the background, ensuring a minimal affect on storage I/O. UNMAP will work at a Guest OS level with newer versions of Windows and Linux.
  • LUN Scalability — vSphere 6.5 supports a maximum of 512 LUNs (up from 256 in 6.0) and 2,000 storage paths (up from 1,024 in 6.0).
  • NFS v4.1 Support — While NFS v4.1 has been supported since vSphere 6.0, 6.5 adds a Kerberos integrity check (SEC_KRB5i) along with Kerberos authentication. NFS v4.1 with Kerberos is also supported with IPv6 in 6.5.
  • Support for Software iSCSI Static Routing — In previous versions, using the software iSCSI initiator required that the initiator and the target be on the same subnet. vSphere 6.5 allows for configuring statics routes between initiator and target subnets, and configuring multipathing without requiring use of the same network.

Networking Enhancements

vSphere 6.5 also adds several enhancements to its network handling. These include:

  • Enhancements for Nested ESXi — Prior to vSphere 6.5, running nested ESXi instances (installing ESXi as the Guest OS on a VM) required enabling promiscuous mode on virtual switches, sending all traffic on the “outer” virtual switch to the nested ESXi instance, resulting in unnecessary packet deliveries, high CPU usage, and low network throughput. vSphere 6.5 includes a MAC address learning capability to the outer switch that enables forwarding only the required packets to the nested instance, resulting in significant performance improvements in nested ESXi environments.
  • Dedicated Gateways for VMkernel Network Adapter – Prior to vSphere 6.5, DRS, vMotion, iSCSI, and provisioning have all used a single gateway, which has required adding static routes to all hosts. In vSphere 6.5 different VMkernel (vmk) services can use different default gateways, eliminating the need for static routes, making things more efficient and more scalable.
  • ERSPAN Support — vSphere 6.5 adds support for the ERSPAN protocol. ERSPAN mirrors traffic from one or more source ports and delivers the mirrored traffic to one or more destination ports on another switch.

VMware Product Compatibility Caveats

When I first started working on this post, I thought I’d include a list  of which other VMware products are not currently supported for use with vSphere 6.5. As it turns out, it will take surprisingly less time to list only those VMware products that are currently supported with vSphere 6.5.

As of the date of this writing, only the following versions of the below-listed products:

  • Horizon 7, version 7.0.2
  • Site Recovery Manager 6.5
  • Identity Manager 2.8 and 2.71
  • VMware PowerCLI 6.5.0
  • VMware Remote Console 9.0.0
  • VMware Tools 10.1.0, 10.0,12, 10.0.9, 10.0.8, 10.0.6, 10.0.5, 10.0.0
  • vRealize Operations Manager 6.4
  • vRealize Automation 7.2.0, 6.2.5
  • vRealize Log Insight 4.0.0, 3.3.2, 3.3.1, 3.3.0, 3.0.0
  • vSAN 6.5
  • vSphere Big Data Extensions 2.3.1
  • vSphere Data Protection 6.1.3
  • vSphere Replication 6.5

It’s worth noting that the VMware Product Interoperability Matrix lists three different status options:

  1. Compatible
  2. Not supported
  3. Incompatible

As of the time of this writing, the following VMware products are explicitly called out as being incompatible with vSphere 6.5:

  • VMware NSX, all versions
  • vRealize Configuration Manager
  • vCloud Director for Service Providers

Bottom line: If you’re running any VMware products that are not on the Compatible list above, you’ll want to hold off on upgrading to vSphere 6.5 until those products have been updated to be Compatible (supported) with it.

To check for the current compatibility with vSphere 6.5 status of a particular VMware product, go to the VMware Product Interoperability Matrix.

Another note — previous versions of vSphere will not work with VMFS6-formatted datastores. vSphere 6.5 can work with either VMFS5- or VMFS6-formatted datastores. There is no “in-place” conversion mechanism for updating a VMFS5 datastore to VMFS6. To migrate from VMFS5 to VMFS6, customers will need to create a new VMFS6 datastore and use Storage vMotion to move VMs to it from the VMFS5 datastore.

One Final Caveat

Before considering an upgrade to vSphere 6.5, make sure you read this VMware KnowledgeBase article titled Update sequence for vSphere 6.5 and its compatible VMware products. It provides details on which VMware products will need to be upgraded to which version and in which order.