How to migrate from Vcenter 5.5 to VCSA 6.5

Before you Start:

  • The migration supports going from 5.5 and 6.0 on Windows to the VCSA 6.5 appliance
  • You will need workstation not counting your Windows vCenter server (in case its physical)
  • Ensure you have the SSO password of your Windows vCenter deployment
  • If you are going to deploy the VCSA on a ESXi server make sure it has a Standard Switch port group that can talk on the same IP as your existing vCenter
  • Take a snapshot of your Windows vCenter VM before the upgrade
  • Rename your Windows vCenter VM with a -old at the end. When you deploy the VCSA you can use the original name
  • If your Windows vCenter is named after the installed version (Example: VCENTER55..my.lab) take a look at the last paragraph in VMware KB 2040354 (Blog post about this coming soon!)
  • Ensure your backup solution and vCenter plugins are compatibility with vCenter 6.5

The migration is a two stage process. The first stage consist of the deployment of the VCSA. The second stage is the migration of the Windows vCenter data to the newly deployed VCSA.

`

Let’s get started!

On your Windows vCenter Server:

Mount the VMware-VCSA-6.5.ISO and navigate to the migration-assistant folder. In there will be a executable called VMware-Migration-Assistant.exe. Right click and select Run as administrator:

It will launch a window like below. Enter the SSO password. The Migration Assistant window will run through a migration precheck. Once the prompt is at Waiting for migration to start you are ready to switch to your other workstation.

Switch over to your Workstation:

Mount the same VCSA 6.5 .ISO and navigate to the vcsa-ui-installer/win32 folder. Inside there will be an executable named installer.exeright click and select Run as administrator:

Welcome to the beautiful VCSA 6.5 Installer! Since our goal is a Windows 5.5 migration click the Migrate option:

Welcome to Stage 1! Click Next:

To continue the EULA must be accepted. Click Next:

Enter the information of your existing Windows vCenter server. Click Next:

If you are using the default self signed certificates you will receive the following message. Click Yes:

You can pick between a vCenter or an ESXi host for the appliance deployment target. I’m going to use a ESXi host. Enter your information and click Next:

If the target is using untrusted SSL certificates the following message will appear. Click Yes:

Enter what you want to name the VCSA virtual machine (as it will appear in vCenter inventory) and assign a Root password. Click Next:

Using the chart pick a deployment size that relates to your environment. For my deployment size I am going to select Tiny as its my home lab. Click Next:

Select a datastore for the VCSA to be deployed on.  Click Next:

For best performance do not use the Thin Provisioning checkbox. Since this is my home lab i’ll use it.

Pick a port group and enter a temporary IP address the VCSA can use for the migration. It will only use this IP for the migration and will take over the IP address of the source Windows vCenter server. Click Next:

Look through the Stage 1 Summary, if everything is correct click Finish:

You can monitor the console screens of both the Windows vCenter and the VCSA Installer window. They talk to each other 🙂

You should received a Deployment Complete message. Click Continue:

Note: If you exit the installed you can continue with the appliance setup by using the URL in the message.

Welcome to Stage 2! Where the Windows vCenter data is migrated over. Click Next:

The Installer will perform another pre-migration check. My results show some of the VMware Update Manager baselines and files may not be migrated over. No big deal! Click Close:

The installer will delete the computer account of the Windows vCenter and join itself to your domain. Enter an AD account with this permission and click Next:

Select what data you want to migrate over. I chose to migrate everything by selecting the third option. My vCenter was a recent and hasn’t had much time to collect a large amount of data. Once your selection is made click Next:

You can join the Customer Experience Improvement Program or deselect. Click Next:

Go over the Stage 2 Summary, ensure you have a backup/snapshot of your Windows vCenter, and click Finish:

You must select the check box in order to continue

During the data migration the Windows vCenter will be shutdown and the VCSA will be configured with its IP address. Click OK:

You can monitor the console screens of both the Windows vCenter and the VCSA Installer window. They talk to each other much more during Stage 2!

Stage 2 took the most time by far. Once it completes you should have the below screen with a URL to access the VCSA. Click the link then click Close:

On the VCSA 6.5 landing page you have two URLs, one to the Flash web client and the second to the new HTML5 vSphere Client.

Congratulations! You are now running on the vCenter Server Appliance 6.5!!!

Flash Web Client:

HTML5 vSphere Client:

POWERCLI: “INITIALIZE-POWERCLIENVIRONMENT.PS1 CANNOT BE LOADED BECAUSE THE EXECUTION OF SCRIPTS IS DISABLED ON THIS SYSTEM.”

Whenever you start PowerCLI, you see:

File C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1 cannot be loaded because the execution of scripts is disabled on this system. Please see “get-help about_signing” for more details.
At line:1 char:2
+ . <<<< “C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Script
s\Initialize-PowerCLIEnvironment.ps1”
+ CategoryInfo : NotSpecified: (:) [], PSSecurityException
+ FullyQualifiedErrorId : RuntimeException

PS C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI>

To permanently allow the execution of scripts, Right-click on PowerCLI > Run as Administrator:

PS C:\> Get-ExecutionPolicy
Restricted
PS C:\> Set-ExecutionPolicy RemoteSigned

Execution Policy Change
The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose you to the security risks described in the about_Execution_Policies help topic. Do you want to change the execution policy?
[Y] Yes [N] No [S] Suspend [?] Help (default is “Y”): y
PS C:\> Get-ExecutionPolicy
RemoteSigned

Introducing VMware 6.7

We are excited to share that today VMware is announcing vSphere 6.7, the latest release of the industry-leading virtualization and cloud platform. vSphere 6.7 is the efficient and secure platform for hybrid clouds, fueling digital transformation by delivering simple and efficient management at scale, comprehensive built-in security, a universal application platform, and seamless hybrid cloud experience.

vSphere 6.7 delivers key capabilities to enable IT organizations address the following notable trends that are putting new demands on their IT infrastructure:

  • Explosive growth in quantity and variety of applications, from business critical apps to new intelligent workloads.
  • Rapid growth of hybrid cloud environments and use cases.
  • On-premises data centers growing and expanding globally, including at the Edge.
  • Security of infrastructure and applications attaining paramount importance.

 

Let’s take a look at some of the key capabilities in vSphere 6.7:

 

Simple and Efficient Management, at Scale

vSphere 6.7 builds on the technological innovation delivered by vSphere 6.5, and elevates the customer experience to an entirely new level. It provides exceptional management simplicity, operational efficiency, and faster time to market, all at scale.

vSphere 6.7 delivers an exceptional experience for the user with an enhancedvCenter Server Appliance (vCSA). It introduces several new APIs that improve the efficiency and experience to deploy vCenter, to deploy multiple vCenters based on a template, to make management of vCenter Server Appliance significantly easier, as well as for backup and restore. It also significantly simplifies the vCenter Server topology through vCenter with embedded platform services controller in enhanced linked mode, enabling customers to link multiple vCenters and have seamless visibility across the environment without the need for an external platform services controller or load balancers.

Moreover, with vSphere 6.7 vCSA delivers phenomenal performance improvements (all metrics compared at cluster scale limits, versus vSphere 6.5):

  • 2X faster performance in vCenter operations per second
  • 3X reduction in memory usage
  • 3X faster DRS-related operations (e.g. power-on virtual machine)

These performance improvements ensure a blazing fast experience for vSphere users, and deliver significant value, as well as time and cost savings in a variety of use cases, such as VDI, Scale-out apps, Big Data, HPC, DevOps, distributed cloud native apps, etc.

vSphere 6.7 improves efficiency at scale when updating ESXi hosts, significantly reducing maintenance time by eliminating one of two reboots normally required for major version upgrades (Single Reboot). In addition to that, vSphere Quick Boot is a new innovation that restarts the ESXi hypervisor without rebooting the physical host, skipping time-consuming hardware initialization.

Another key component that allows vSphere 6.7 to deliver a simplified and efficient experience is the graphical user interface itself. The HTML5-based vSphere Client provides a modern user interface experience that is both responsive and easy to use. With vSphere 6.7, it includes added functionality to support not only the typical workflows customers need but also other key functionality like managing NSX, vSAN, VUM as well as third-party components.

 

Comprehensive Built-In Security

vSphere 6.7 builds on the security capabilities in vSphere 6.5 and leverages its unique position as the hypervisor to offer comprehensive security that starts at the core, via an operationally simple policy-driven model.

vSphere 6.7 adds support for Trusted Platform Module (TPM) 2.0 hardware devices and also introduces Virtual TPM 2.0, significantly enhancing protection and assuring integrity for both the hypervisor and the guest operating system. This capability helps prevent VMs and hosts from being tampered with, prevents the loading of unauthorized components and enables guest operating system security features security teams are asking for.

Data encryption was introduced with vSphere 6.5 and very well received.  With vSphere 6.7, VM Encryption is further enhanced and more operationally simple to manage.  vSphere 6.7 simplifies workflows for VM Encryption, designed to protect data at rest and in motion, making it as easy as a right-click while also increasing the security posture of encrypting the VM and giving the user a greater degree of control to protect against unauthorized data access.

vSphere 6.7 also enhances protection for data in motion by enabling encrypted vMotion across different vCenter instances as well as versions, making it easy to securely conduct data center migrations, move data across a hybrid cloud environment (between on-premises and public cloud), or across geographically distributed data centers.

vSphere 6.7 introduces support for the entire range of Microsoft’s Virtualization Based Security technologies. This is a result of close collaboration between VMware and Microsoft to ensure Windows VMs on vSphere support in-guest security features while continuing to run performant and secure on the vSphere platform.

vSphere 6.7 delivers comprehensive built-in security and is the heart of a secure SDDC. It has deep integration and works seamlessly with other VMware products such as vSAN, NSX and vRealize Suite to provide a complete security model for the data center.

 

Universal Application Platform

vSphere 6.7 is a universal application platform that supports new workloads (including 3D Graphics, Big Data, HPC, Machine Learning, In-Memory, and Cloud-Native) as well as existing mission critical applications. It also supports and leverages some of the latest hardware innovations in the industry, delivering exceptional performance for a variety of workloads.

vSphere 6.7 further enhances the support and capabilities introduced for GPUs through VMware’s collaboration with Nvidia, by virtualizing Nvidia GPUs even for non-VDI and non-general-purpose-computing use cases such as artificial intelligence, machine learning, big data and more. With enhancements to Nvidia GRID™ vGPU technology in vSphere 6.7, instead of having to power off workloads running on GPUs, customers can simply suspend and resume those VMs, allowing for better lifecycle management of the underlying host and significantly reducing disruption for end-users. VMware continues to invest in this area, with the goal of bringing the full vSphere experience to GPUs in future releases.

vSphere 6.7 continues to showcase VMware’s technological leadership and fruitful collaboration with our key partners by adding support for a key industry innovation poised to have a dramatic impact on the landscape, which is persistent memory. With vSphere Persistent Memory, customers using supported hardware modules, such as those available from Dell-EMC and HPE, can leverage them either as super-fast storage with high IOPS, or expose them to the guest operating system as non-volatile memory. This will significantly enhance performance of the OS as well as applications across a variety of use cases, making existing applications faster and more performant and enabling customers to create new high-performance applications that can leverage vSphere Persistent Memory.

Also check out the VirtualBlocks Core Storage 6.7 blog where you can find more information about new storage and network features such as Native 4Kn disk supportRDMA support, and Intel VMD for NVMethat further enhance Enterprise Applications running on vSphere.

 

Seamless Hybrid Cloud Experience

With the fast adoption of vSphere-based public clouds through VMware Cloud Provider Program partners, VMware Cloud on AWS, as well as other public cloud providers, VMware is committed to delivering a seamless hybrid cloud experience for customers.

vSphere 6.7 introduces vCenter Server Hybrid Linked Mode, which makes it easy and simple for customers to have unified visibility and manageability across an on-premises vSphere environment running on one version and a vSphere-based public cloud environment, such as VMware Cloud on AWS, running on a different version of vSphere. This ensures that the fast pace of innovation and introduction of new capabilities in vSphere-based public clouds does not force the customer to constantly update and upgrade their on-premises vSphere environment.

vSphere 6.7 also introduces Cross-Cloud Cold and Hot Migration, further enhancing the ease of management across and enabling a seamless and non-disruptive hybrid cloud experience for customers.

As virtual machines migrate between different data centers or from an on-premises data center to the cloud and back, they likely move across different CPU types. vSphere 6.7 delivers a new capability that is key for the hybrid cloud, called Per-VM EVC. Per-VM EVC enables the EVC (Enhanced vMotion Compatibility) mode to become an attribute of the VM rather than the specific processor generation it happens to be booted on in the cluster. This allows for seamless migration across different CPUs by persisting the EVC mode per-VM during migrations across clusters and during power cycles.

Previously, vSphere 6.0 introduced provisioning between vCenter instances. This is often called “cross-vCenter provisioning.” The use of two vCenter instances introduces the possibility that the instances are on different release versions. vSphere 6.7 enables customers to use different vCenter versions while allowing cross-vCenter, mixed-version provisioning operations (vMotion, Full Clone and cold migrate) to continue seamlessly. This is especially useful for customers leveraging VMware Cloud on AWS as part of their hybrid cloud.

 

Learn More

As the ideal, efficient, secure universal platform for hybrid cloud, supporting new and existing applications, serving the needs of IT and the business, vSphere 6.7 reinforces your investment in VMware. vSphere 6.7 is one of the core components of VMware’s SDDC and a fundamental building block of your cloud strategy. With vSphere 6.7, you can now run, manage, connect, and secure your applications in a common operating environment, across your hybrid cloud.

This article only touched upon the key highlights of this release, but there are many more new features. To learn more about vSphere 6.7, please see the following resources.

 

Note:

​As part of any new vSphere release, VMware expects to make compatible versions of dependent products available within one quarter of general availability in most cases. At vSphere 6.7 general availability, compatible versions of VMware NSX, VMware Integrated OpenStack and VMware vSphere Integrated Containers will not be available. Moreover, VMware Horizon 7.4 is not compatible with the Instant Clone API used in vSphere 6.7.  Instant Clone support for vSphere 6.7 will be available in an upcoming Horizon release. Existing NSX, VIC and VIO customers are advised not to upgrade to vSphere 6.7 until compatible versions become available. For additional information on NSX, VIC and VIO compatibility, please contact your VMware account team or reseller partner.

ElasticSearch Tutorial: Creating an Index and Querying

Elasticsearch is a highly-scalable document storage engine that specializes in search. Elasticsearch comes with reasonable default settings, but it will also easily scale to being able to search hundreds of millions of documents with sub-second latency. As you’ll see in this tutorial, the JSON-based nature of Elasticsearch, along with its simple REST API, make it easy to learn.

Installation

Elasticsearch is written in Java, so it should work on any operating system that can run Java. If you don’t have Java installed on your machine already, click here to download and install it. We’re going to do this tutorial with version 2.3.1 of Elasticsearch.

Elasticsearch is really simple to get up and running – you can have a one-node cluster running in just a few minutes with the following steps:

  1. Download and unzip Elasticsearch 2.3.1 found here.
  2. Navigate to the directory you unzipped it to, and run bin/elasticsearch if you are using Linux or OS X or bin/elasticsearch.bat if you are using Windows.
  3. In your browser, navigate to http://localhost:9200/, you should see a page that looks something like this:

elastic-ss-1

Basic Terms

Document

A document is the basic record you store in Elasticsearch. You can think of it being roughly similar to a row in a traditional database. Documents are structured as JSON objects and must belong to a type.

Type

A type is a set of documents with common fields. You can think of it as being similar to a table in a traditional database, but the definition is somewhat less strict.

Index

An index is a collection of related types of documents. It is somewhat similar in function to a database or schema in the traditional database world.

Shard

The example Elasticsearch index we build today will be really small, but many indexes can get quite large and it isn’t uncommon at all to have Elasticsearch index with multiple terabytes of data in them. Sharding helps you scale this data beyond one machine by breaking your index up into multiple parts and storing it on multiple nodes.

Beyond allowing for more storage, shards also allow for better performance, because data in the same index can be searched by multiple nodes at the same time.

Replica

A replica is simply a copy of a shard. This is useful for protecting against hardware failures – if all of your shards are replicated, then the failure of one node in your cluster will not impact availability. Replicas also improve throughput and latency by making your index’s data available to more nodes in the cluster.

The following diagram demonstrates how sharding and replication work:

elastic-diagram-1

Interacting with our cluster

As mentioned before, the interface to Elasticsearch is a REST API that you interact with over HTTP by sending certain URLs, and in some cases HTTP bodies composed of JSON objects that you use to give commands to the cluster. Some of these commands are simple GET requests and can be performed in your browser, but many others are POSTs with bodies, so we need a tool to help us make these requests to the cluster.

The are two easy options for accomplishing this: the Sense plugin for Chrome; or using an HTTP client such as cURL or Postman to compose requests yourself.  Following are the details for both options.

Sense plugin

The Sense plugin for Chrome can be found here, and is what we will use in this tutorial to interact with the cluster. It makes it easy for us to compose and send GETs, POSTs, or PUTs to the server to tell it to index data or retrieve results for us.

Once you’ve installed it, you’ll see a UI that looks like this:

elastic-ss-2

The top part (labeled ‘Server’) is the host and port where the server is located. If you used the default settings when starting Elasticsearch, you can leave this as localhost:9200.

The bottom part is broken up into two panes:

The left pane has your commands, which should be formatted with the HTTP action (GET, POST, PUT) and the URL (_search, /books/book/1). In the case of a GET, that will be all you need. In the case of a POST or PUT with a body, you’ll provide the body as JSON on subsequent lines. To run the command, simply click the green arrow next to it. You can have multiple commands in this view and run them separately. Here’s an example of an indexing operation:

elastic-ss-3

The right pane contains the result of your operation, in JSON form.

elastic-ss-4

Here is what an entire request looks like:

elastic-ss-5

Using an HTTP client

If you’d rather not install the Sense plugin, you can use an HTTP client such as cURL or Postman to compose requests yourself. When you see an action, simply add it to http://localhost:9200 to make the request (for example, if the Sense directions say ‘GET _search’, you’ll make a GET request to http://localhost:9200/_search.

Starting with ElasticSearch? Check this tutorial: ElasticSearch Tutorial – Creating an Index and QueryingCLICK TO TWEET

Loading our first documents

Let’s load some documents to give ourselves some data to work with. We’re going to create documents representing computer programming books, with the following fields:

title – The title of the book.
author – The author of the book.
language – The language the book is primarily about.
published – The year the book was published.
summary – A short summary of the book.

Additionally, our books will also have an ID to make it easier to look them up by that reference, but the ID won’t be part of the document we create.

So, let’s create our first book:

POST /books/book/1
{
  "title" : "Effective Java",
  "author" : "Josh Bloch",
  "language" : "Java",
  "publishYear" : 2008,
  "summary" : "Are you looking for a deeper understanding of the Java programming language so that you can write code that is clearer, more correct, more robust, and more reusable? Look no further! Effective Java, Second Edition, brings together seventy-eight indispensable programmer’s rules of thumb: working, best-practice solutions for the programming challenges you encounter every day."
}

You’ll notice we never defined an index or type, but it still worked – Elasticsearch will create these on the fly for you! As your documents get more complicated and your use of Elasticsearch grows, you’ll find situations where you want to customize the indexing behavior of Elasticsearch and will want to define a type by supplying a mapping, but we don’t need to do that for this tutorial.

Now, let’s make sure we can get our book back by ID:

GET /books/book/1

elastic-ss-6

Now that we understand how to create and view a document, let’s create a few more to give us some interesting stuff to search on:

POST /books/book/2
{
  "title" : "JavaScript: The Good Parts",
  "author" : "Douglas Crockford",
  "language" : "JavaScript",
  "publishYear" : 2008,
  "summary" : "Most programming languages contain good and bad parts, but JavaScript has more than its share of the bad, having been developed and released in a hurry before it could be refined. This authoritative book scrapes away these bad features to reveal a subset of JavaScript that's more reliable, readable, and maintainable than the language as a whole—a subset you can use to create truly extensible and efficient code."
}
POST /books/book/3
{
  "title" : "Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition",
  "author" : "Martin Odersky",
  "language" : "Scala",
  "publishYear" : 2011,
  "summary" : "Scala is an object-oriented programming language for the Java Virtual Machine. In addition to being object-oriented, Scala is also a functional language, and combines the best approaches to OO and functional programming."
}
POST /books/book/4
{
  "title" : "Hadoop: The Definitive Guide, 4th Edition",
  "author" : "Tom White",
  "language" : "Java",
  "publishYear" : 2015,
  "summary" : "Get ready to unlock the power of your data. With the fourth edition of this comprehensive guide, you’ll learn how to build and maintain reliable, scalable, distributed systems with Apache Hadoop. This book is ideal for programmers looking to analyze datasets of any size, and for administrators who want to set up and run Hadoop clusters."
}
POST /books/book/5
{
  "title" : "Learn You a Haskell for Great Good!: A Beginner's Guide",
  "author" : "Miran Lipovaca",
  "language" : "Haskell",
  "publishYear" : 2011,
  "summary" : "It's all in the name: Learn You a Haskell for Great Good! is a hilarious, illustrated guide to this complex functional language. Packed with the author's original artwork, pop culture references, and most importantly, useful example code, this book teaches functional fundamentals in a way you never thought possible."
}

Querying with the JSON DSL

Now we should have five documents in our index. We already saw one way to query for data in our index, by ID, so let’s look at another one: getting all of the documents in our index.

GET /books/book/_search

As you can see, there are a number of JSON elements in the result. The interesting ones are described below:

  1. “took” – This is how long (in ms) the search took.
  2. “timed_out” – You can tell Elasticsearch to stop querying and return results after a certain amount of time. We won’t cover how to do that in this tutorial, but this field says whether or not that timeout was reached.
  3. “hits” is our list of results.
  4. “_index” tells us what index the result came from. It is possible to search across multiple indexes.
  5. “_type” tells us what type the result came from. It is possible to search across multiple types.
  6. “_id” is the identifier for our document. In our case, we supplied them when creating our document, but it is also possible to let Elasticsearch assign them.
  7. “score” is telling us how well the document matched our query.
  8. “source” is the document itself.

Now that we know how to interpret our results, let’s try some more advanced queries.

Let’s find all books that are about Java. We’ll use a query string query to search all fields for ‘java’:

POST /books/book/_search
{
    "query" :
    {
        "query_string":
        {
           "query": "java"
        }
    }
}

elastic-ss-8

This searches all fields for any reference to Java. You’ll note that it matches three books: One with a reference to Java only in the language field, one that matches in the summary, language, and title fields, and another that only matches it in the summary.

We can also query on non-string fields (or perform exact matches on string fields) with a filter. Let’s combine a filter with a match_all query to find all books published in 2011 or later:

POST /books/book/_search
{
    "query" :
    {
        "filtered":
        {
            "query" :
            {
                "match_all": {}
            },
           "filter":
           {
               "range":
               {
                  "publishYear":
                  {
                     "from": 2011
                  }
               }
           }
        }
    }
}

elastic-ss-9

Now, let’s combine the two to find all Java books written in 2014 or earlier:

POST /books/book/_search
{
    "query" :
    {
        "filtered":
        {
            "query" :
            {
                "query_string":
                {
                    "query": "java"
                }
            },
           "filter":
           {
               "range":
               {
                  "publishYear":
                  {
                     "to": 2014
                  }
               }
           }
        }
    }
}

elastic-ss-10

Conclusion

You’ve learned how to index documents and query them with Elasticsearch with very little effort. This is just a small sample of the power of Elasticsearch, but it also shows you some of the power it has. Elasticsearch documentation is available to help you as you pursue more advanced usages of it.

How to reset the root password for VCSA 6.x

Resetting the VMware vCenter Inventory Database

Resetting the VMware vCenter Inventory Database

The call for help started like this, “The customer cannot connect to the Inventory Services page in order to build a Server 2012 Template.”  Upon talking to the customer, the problem was that they were unable to access anything in the vSphere web client; the error displayed stating that the web client was unable to authenticate to the VMware Inventory Service as shown below:

1_InvError

Though everything was fine when using the full-blown vSphere client, the error seen within the web client was the same regardless of the account used.  I had just upgraded their vCenter server a couple weeks ago and we did have a problem in that the server ran out of hard drive space while upgrading the vCenter database, thus my database became corrupted and had to be restored.  During the course of resolving the database corruption and disk space issues, we also cleared out old stats and historical data from the database and made the decision to P2V the vCenter server to increase the C:.

Well, when the vCenter server was ready once again, I started the upgrade process where we left off…Custom Install | vCenter Server since the SSO, Web Client, and Inventory Service tasks had been completed.  The installation of the vCenter Server continued without issue and all looked good.  Now I don’t know about you, but I’m having a hard time letting go of the full-blown vSphere client….that’s still my “go to” management tool for vSphere though we’ll all be forced to the web client soon.  Thus, I didn’t know there was a problem with the web client for a couple weeks.

To resolve the issue, I had to delete and recreate the Inventory Service databaseusing the steps below, found within the VMware vSphere Documentation Center:

1. Stop the VMware vCenter Inventory Service

2. Open a command prompt.  The documentation does say you have to, but I opened a CMD prompt with elevated privileges.

3. However you prefer (CMD or Explorer), delete the contents of the following directory (assuming vCenter was installed to the default location):

C:\Program Files\VMware\Infrastructure\Inventory_Service\data

4. In the CMD prompt, change directory to:

C:\Program Files\VMware\Infrastructure\Inventory_Service\scripts 

5. Execute the command createDB.bat, with no arguments, to reset the vCenter Inventory Service database.  **If you’re like me and are a “clicker”, you may want to back away from the computer here for a few moments.  It doesn’t need your help, it doesn’t need you clicking things to make sure its running, it just needs it time and space.  The second time I executed the command, I think it took about 5 minutes.  I’m not sure exactly since I left the office the second time.

2_CreateDB

6. When the command completes, start the VMware vCenter Inventory Service.

7. Back in the CMD prompt, run the is-change-sso.bat command to ensure the SSO information is correct.  An example command is shown here: is-change-sso.bat https://vcenter.ballblog.info:7444/lookupservice/sdk“administrator@vsphere.local” “SSO_Password_Here

8. In the CMD prompt, change directory to:

C:\Program Files\VMware\Infrastructure\VirtualCenter Server\isregtool

9. Run the register-is.bat command to update the stored configuration information of the Inventory Service:

register-is.bat https://vcenter.ballblog.info:443/sdkhttps://vcenter.ballblog.info:10443https://vcenter.ballblog.info:7444/lookupservice/sdk

In this example I am assuming the vCenter Server using the default ports of 443, 10443, and 7444 for vCenter Server, Inventory Service, and vCenter Single Sign On. If you use custom ports, replace the port numbers in the example with the port numbers you use.

10. Documentation says to restart the vCenter server, but I simply restarted the VMware VirtualCenter Server service and upon doing so, I was able to login to the vSphere web client with no further issues.

ESXi 6.0 reports “error code: 15” during Remediate update in VUM operation

 

Another vCenter another ESXi with problems applying last updates.

In this case is a HP DL360 G9 with ESXi 6.0 build 3568940.

Using VMware Update Manager to scan it shows 17 updates to install, stage is 7(the rest are older versions), when remediate the host we get this:

Remediate entity esxi721.localdomain. The host returns esxupdate error code: 15. The package manager transaction is not successful.
Check the Update Manager log files and esxupdate log files for more details.

Again in issue lot of troubleshooting to check were was the problem here.

Looking at the esxupdate.log there is some information about the locker folder:

2016-04-24T15:11:44Z esxupdate: downloader: DEBUG: Downloading from http://esxi721.localdomain:9084/vum/repository/hostupdate/vmw/vib20/tools-light/VMware_locker_tools-light_6.0.0-2.34.3620759.vib…
2016-04-24T15:12:48Z esxupdate: LockerInstaller: WARNING: There was an error in cleaning up product locker:

[Errno 2] No such file or directory: ‘/locker/packages/var/db/locker’
2016-04-24T15:12:48Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:

So need to investigate in the ESXi host. In VMware KB about this ‘error 15’ it says to double check the folder/link Locker > Store

I double check the link to see if the link exists and also the folder, and all is ok. Next check locker folder/link and if locker link is valid

Check if store location is correct

All is ok, so need to check locker/packages folder to see if Version(in this case folder 6.0.0) exists.

The folder doesn’t exist, and there is no floppies, vmtools folders that have all the files that ESXi and VUM needs for the updates. In the VMware KB recommends to delete old folder and links and recreate, in this case we don’t need to delete nothing, but to recreate and copy the necessary files(we will use another ESXi host with the same build).

Connecting to another host we will use SCP to copy all files to this ESXi host.

First if  you don’t have your SSH Client enable in the host firewall, you need to enabled to do the next task using SCP command.

To enable SSH Client in the source ESXi host:

Note: Don’t forget to disable SSH Client after do this tasks.

After you run SCP command you will be prompted for the root password of the remote host and once you have successfully authenticated the files will copy.

Only when trying to copy the files we find the real issue. Did not find anything in the logs related to this. Space issue to apply the updates.

So need to double check the root space.

Here I don’t see any issues with the space, but see big files from the Tardisk

Checking filesystems I see that the one is use for Locker is 100% full.

So next step is to find big files logs, and also inside /tmp if there is any dump files, or other big files that are contributing to this issue.

As we can see there is some big temp files in the list, so the next step is to delete some.

Note: Double check which files do you want to delete. Don’t delete any log files that you could need for any troubleshooting or audit.

After deleting the files that we will not need(and also deleted the files that we copy from the previous ESXi host), and also all folders inside Locker/Store folder, we can check the space.

We now have space around 0% and lot of free space.

Well copy the files again from the other ESXi host and finish 100%.

Now using VUM we will scan, stage and remediate the ESXi host and the problem is fix. After a final reboot(from remediate) the ESXi is fully updated.

Hope this can help.

Backup ASA config with PowerShell

During my years in the networking business one of my frustrations is that it is very hard to backup the configuration of an ASA. There are some commercial products like Solarwinds that can accomplish this goal, but it costs money. An open source alternative like Rancid is also available but is pretty hard to configure.
Determined to find a solution I started searching the internet and came across some PowerShell scripts.  I’m not a PowerShell specialist, but I do know how to put together the separate scripts. So to be clear, I did not invent the scipts I just put them together.

So let’s take a look at the script:

Read-Host  “Enter Password” -AsSecureString | ConvertFrom-SecureString | Out-File c:\<map>\cred01.txt
–I don’t want to sent the password of the ASA user plain over the network. So with the above line I make sure the password is encrypted. It is possible to convert the password back to plain text, but then you’ll need access to the server. So it is not rocksollid save, but safer then sending the password in plain text over the internet. If you make sure that the useraccount only has minimal rights on the ASA, there is minimal change of getting unwanted guests on your ASA. The line converts the plain password to an encrypted password and writes it to a .txt file.

$ASApw = Get-Content “c:\<map>\cred01.txt” | ConvertTo-SecureString #-AsPlainText #-Force
$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($ASApw)
$ASApw = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)
–The above three lines are needed to convert the encrypted password from the credentials file. This is needed because the ASA is unable to read an encrypted password.

$ASAIP = “<ip address>”
$ASAUser = “<username>”
$ASAEnablepw = $ASApw

#Modifies the ASA firewall
#Starts by writing a “commands” file#
echo en >>unicode.txt
echo $ASAEnablepw >>unicode.txt
echo “conf t” >>unicode.txt
echo “no pager” >>unicode.txt
echo “show run” >>unicode.txt
echo “pager 24” >>unicode.txt
echo “copy running-config startup-config” >>unicode.txt
echo “running-config” >>unicode.txt
echo exit >>unicode.txt
echo exit >>unicode.txt

#Converts the file to ASCII format (separate file)#
$lines = gc “unicode.txt
$lines | out-file -encoding Ascii -filepath commands.txt
–The above lines writes the actual ASA commands to the commands.txt file.

#Using the command file and plink.exe connects and runs the commands#
c:/Windows/System32/plink.exe -ssh -l $ASAUser -pw $ASApw $ASAIP -m commands.txt > “c:\<map>\ASA.txt
–To make things work you need to download the Plink tool. It is the command line version of Putty. It can be downloaded for free. I put the tool in de c:\windows\system32 folder, but you can place it everywhere you want. This line writes the configuration of the ASA to an .txt file.

#removes the files it created earlier#
del unicode.txt
del commands.txt

As you can see it’s actually a pretty easy script an above all it’s free.
To make a daily backup, create a task through “Task scheduler”.

VMware NSX 6.2: From zero to full deployment

VMware NSX is the SDDC technology of the future. What ESX was once for Servers, NSX is now for Networks. I highly encourage everyone to make yourselves familiar with this technology. NSX with all its features is quite complex, but the entry point is quite simple and requires only basic vSphere and networking skills. This beginners guide explains how to deploy NSX in your homelab even with limited physical ressources by downsizing NSX Manager and NSX Controller VMs. The guide starts at zero and quickly explains how to deploy NSX and connect your first Virtual Machine to a VXLAN based logical switch that is able to communicate to the physical world through an NSX Edge Gateway.

What do you need to create the Lab?

  • vCenter 6 with some physical ESXi Hosts
  • vSphere Distributed Switch (dvSwitch)
  • NSX Manager Appliance (Download: NSX 6.2.2)
  • There is no special physical Switch requirement

This guide is not intended as a comprehensive guide to fully understand all aspects of NSX. It’s just a quick deployment guide with some tweaks for low resource Homelabs.

  1. Prepare your vSphere Platform (vCenter Server Appliance, ESXi Hosts, Cluster and dvSwitch).  I recommend to update everything to the latest version available, which is currently vSphere 6.0 Update 2 and NSX 6.2.2.
  2. Configure a NTP Server for ESXi Hosts and the vCenter Server to ensure time consistency.
  3. Deploy the NSX Manager Appliance, provided as OVA (Download: NSX 6.2.2). The deployment wizard is pretty straightforward, nothing special here if you have ever deployed a template. Make sure to configure a NTP Server, and enable SSH.
  4. The NSX Manager is preconfigured with 4 vCPU and 16GB Memory. That might be a little oversized for a tiny homelab. If your hardware has limited resources, shutdown the NSX Manager and reduce the configured resources. 2 vCPU and 8 GB Memory should be possible without any impact (Of course, this is not supported by VMware). However, you can set it even lower but keep an eye on the memory consumption. You can check it by logging in to the NSX Manager with SSH (password set during ova deployment) and run the command “show system memory”:
    nsx-manager-show-system-memory
  5. The NSX Manager needs to be registered with the vCenter Server. Open the NSX Manager with a browser and login as admin, with the password configured during the deployment.nsx-manager-webinterface-login
  6. Navigate to Manage vCenter Registration
    nsx-manager-webinterface-homescreen
  7. Configure the Lookup Service to the IP Address of your Platform Services Controller and the vCenter Server connection to your vCenter Server. For vCenters with an embedded PSC, the lookup service runs on the vCenter Server itself. The following information are required:
    – Lookup Service IP
    – Lookup Service Port (Default: 7444)
    – SSO Admin Credentials
    – vCenter Server Address (IP or DNS Name)
    – vCenter Server Admin CredentialsVerify that both Services are connected:
    nsx-manager-webinterface-configure-vcenter-connection
  8. Login to the vCenter Server Web Client. You should now see the Network & Security icon where NSX is configured. Open it:
    nsx-webclient-network-and-security-icon
  9. The first part of the configuration is to deploy NSX Controller nodes which are representing the Control Pane of NSX. NSX Controllers are Virtual Machines. At least 3 NSX Controllers are required for redundancy. Click the + symbol to deploy the first controller.
    nsx-installation-add-nsx-controller
  10. Select the options for the NSX Controller until the IP Pool configuration and click Select to select or create an IP Pool.
    nsx-installation-first-controller-configuration
  11. Click New IP Pool…
    nsx-installation-add-ip-pool
  12. Enter Network details. This should be the same layer 2 network as your vCenter Server, ESXi Hosts and NSX Manager. NSX Controllers are automatically configured with IP Addresses defined in the Static IP Pool. Configure the range with the number of IP Addresses you want to reserve for NSX Controllers.
    nsx-installation-add-static-ip-pool
  13. Finish the IP Pool configuration, select it and finish the Controller configuration:nsx-install-finish-controller-configuration
  14. Wait until the deployment has been finished.nsx-installation-deploying-nsx-controller
  15. Deploy 2 additional controllers in the same IP Pool and wait until all controller nodes are deployed and connected.
    nsx-installation-3-controllers-deployed
  16. NSX Controllers are preconfigured with 2 vCPU and 4GB Memory. For your tiny (unsupported) homelab you can also reduce NSX Controller resources, but this is a little bit more tricky because the “Edit settings” function is blocked by the vCenter. To disable vCenter Server protection you have to delete respective entries from the VPX_DISABLED_METHODS table. This method is also described by Tom Fojta.
    – SSH to the vCenter Server
    – Enable Bash
    – Connect to the vCenter Postgres Database
    – Identify Object IDs
    – Delete entries
    – Restart vCenter Server Service

    Command> shell.set --enabled True
    Command> shell
    vcsa:~ # /opt/vmware/vpostgres/current/bin/psql -U postgres
    psql.bin (9.3.9 (VMware Postgres 9.3.9.0-2921310 release))
    Type "help" for help.
    
    postgres=# \connect VCDB
    You are now connected to database "VCDB" as user "postgres".
    VCDB=# select * from VPX_DISABLED_METHODS;
     entity_mo_id_val |          method_name           | source_id_val | reason_id_val
    ------------------+--------------------------------+---------------+---------------
     vm-68            | vim.VirtualMachine.reconfigure | vShield_SVM   | vShield_SVM
     vm-81            | vim.VirtualMachine.reconfigure | vShield_SVM   | vShield_SVM 
     vm-82            | vim.VirtualMachine.reconfigure | vShield_SVM   | vShield_SVM 
    (1 rows)
    
    VCDB=# delete from VPX_DISABLED_METHODS where entity_mo_id_val = 'vm-68';
    DELETE 1
    VCDB=# delete from VPX_DISABLED_METHODS where entity_mo_id_val = 'vm-81';
    DELETE 1
    VCDB=# delete from VPX_DISABLED_METHODS where entity_mo_id_val = 'vm-82';
    DELETE 1
    VCDB=# select * from VPX_DISABLED_METHODS;
     entity_mo_id_val | method_name | source_id_val | reason_id_val
    ------------------+-------------+---------------+---------------
    (0 rows)
    postgres=# \q
    vcsa:~ # service-control --stop vmware-vpxd
    vcsa:~ # service-control --start vmware-vpxd

    nsx-install-unlock-VPX_DISABLED_METHODS-vcsa

  17. After the vCenter Server service has been restarted you should be able to edit NSX Controller Resources. It is possible to resize NSX Controllers to 2 GB without any major issues. If you want to go lower, verify memory consumption by logging in to the NSX Controller with SSH and run the command “show system statistics graph memory/memory-used:value”. Memory Usage here is 800MB:nsx-controller-memory-consumption
  18. Now you have to prepare ESXi Hosts to be compatible with NSX. Navigate to Network and Security > Installation > Host Preparation, select your Cluster and click Install. This will install 2 VIB on all ESXi hosts in the Cluster. The installation is completely transparent to virtual machines. Maintenance Mode is not required.
    nsx-host-preparation
  19. Refresh the vSphere Client to check the Status and wait until the installation has been finished.nsx-host-preparation-finished
  20. To allow ESXi hosts to talk to each other and to the physical network, they need Tunnel Endpoints (VTEP Interfaces). Click Not Configured in the VXLAN tab, configure VXLAN settings and add an IP Pool for VTEP interfaces. Make sure to size the IP Pool according to the number of ESXi Hosts:
    nsx-configure-vxlannsx-add-static-pool-for-vtep-interfaces
  21. Navigate to Network and Security > Installation > Logical Network Preparation > Segment ID, and configure the Segment ID pool to the Number of VXLANs your are planning to use:
    nsx-installation-logical-network-preparation
    nsx-edit-segment-ids
  22. To determine the replication boundaries of a VXLAN Network, a global transport zone is required. Navigate to Network and Security > Installation > Logical Network Preparation > Transport Zones, and click +
    nsx-installation-add-transport-zone
  23. Add a Global Transport Zone and set the Replication mode to Unicast. This setting allows NSX Controllers to talk to each other without any multicast requirements on physical switches. Select all clusters that needs to be part of the Transport Zone.nsx-installation-transport-zone-configuration
  24. Now the initial NSX configuration is finished and you are ready to configure the first Logical Switch. A Logical Switch is basically a VXLAN Network or Portgroup where Virtual Machines are connected to. Navigate to Network and Security > Logical Switches click +
    nsx-add-logical-switch
  25. Name the Logical Switch and set the replication mode to Unicast.
    nsx-add-logical-switch-configuration
  26. To connect the “virtual” Logical Switch beyond the VXLAN you need a NSX Edge. The Edge Gateway is a Virtual Machine with 2 network interfaces, one connected to the VXLAN and one connected to the outside network. The Edge Gateway acts as Router. Navigate to Network and Security > NSX Edge and click +nsx-add-edge-gateway
  27. Set the installation type to Edge Services Gateway and enter a namensx-edge-configuration-name
  28. Set Admin credentials and enable SSH.
    nsx-edge-configuration-settings
  29. Set the Appliance Size and configure the NSX Edge Appliance placementnsx-edge-configuration-deploymentnsx-edge-configuration-placementnsx-edge-configuration-deployment-finished
  30. Configure Edge Interfaces. Place one interface to your physical connected portgroup and a second to the internal Logical Switch.
    nsx-edge-inferfaces
    nsx-edge-outside-interface
    nsx-edge-outside-portgroup
    nsx-edge-internal-interface
    esx-edge-internal-logical-switch
  31. Finish the deployment wizard and wait until the deployment has been finished.
  32. The VXLAN is now connected to the physical network. For ease of use I am going to enable DHCP on the logical switch. Doubleclick the new edge gateway and open the DHCP tab.nsx-edge-dhcp-service
  33. Add DHCP Pool configurationnsx-edge-add-dhcp-pool
  34. Activate DHCP and publish changes.nsx-edge-enable-dhcp
  35. Navigate to Network and Security > Logical Switches and connect a Virtual Machine to the Logical Switch
    nsx-add-vm-to-logical-switch

Congratulations! You have successfully deployed NSX in your Homelab.

Mastering VMware Snapshots

What is VMware Snapshot?

VMware Snapshot preserves the state and data of a virtual machine at a specific point in time.

Snapshot state includes the virtual machine’s power state (ex, powered-on, powered-off, suspended) and the data includes all of the files (disks, memory, and other devices, such as virtual network interface cards.) that make up the virtual machine.

Snapshots extremely simplify the Virtual Machine OS management and maintenance tasks by preserving the current state of the virtual machine.  So it allows you to revert to the state before the snapshot is taken.

Windows patching in Physical Server VS VMware Virtual machine

Physical workload

Before performing the OS patching or software upgrade, You need to take a backup such as OS Image level backup to restore the OS in case of any corruption is happened during the patching and also it is a time-consuming process.

Restore is not even guaranteed recovery if in case of the backup file is corrupted.

VMWare Virtual machine

In VMware Virtual Machine, Snapshots simplifies the process to capture the Point-in-time copy of the virtual machine without the need of any third party backup software.

It also allows you to take multiple snapshots to preserve the multiple states of the virtual machine. It also simplifies the restore process.

You just need to select the snapshot which is taken before patching to revert to the particular state of the virtual machine in case any issue happened during the patching.

How to Create VMware Snapshot

VMware Snapshot can be simply created using vSphere client connected to ESXi host or vCenter Server and through vSphere web client. Even VMware snapshots can be created using the command line and powerCLI scripts. Let’s take a look at how to create Virtual Machine Snapshot from vSphere Web Client.

Connect to your vCenter server using vSphere Web Client and Right-click the virtual machine -> Take Snapshot

VMware Snapshot

Name: Specify the name for the snapshot. Name of the snapshot should understandable easily from its name.

Description: Provide a description for this snapshot. You can even specify the Date and time to best identify its age.

As of now, Just go with the snapshot without Virtual Machine memory. I will explain the difference between the VMware snapshot with Virtual Machine’s memory and  Snapshot without Virtual Machine’s memory.  Click Ok to create the snapshot.

VMware Snapshot_2

You can validate that the “Create Virtual Machine Snapshot” tasks under your recent tasks in vSphere Web Client. It is completed and Snapshot is created.

VMware Snapshot_2.1

Difference between VM Snapshot with and without Virtual Machine Memory

Snapshot with Virtual Machine Memory

If you select the checkbox “Snapshot the virtual machine’s memory” during snapshot creation, Then memory flag will be set to 1 or true, a dump of memory state (internal state) of the virtual machine is also included in the snapshot.

It takes little longer time to create the memory snapshot as compared to snapshots with memory. Virtual Machine needs to be Powered on state to take the memory snapshot. If the virtual machine is in powered off state, Memory snapshot option will be grayed out.

This memory snapshot will allow you to restore the virtual machine to the same state as it was when a snapshot was taken. In Simple terms, it captures the live state of the virtual machine

For Example, you are modifying a Microsoft word Document file and memory snapshot is taken at that time. After some time VM goes into Blue screen. You want to restore the snapshot from the memory snapshot taken prior to the blue screen issue. Snapshot with memory will revert the VM into powered on state along with the opened MS word file.

VMware snapshot with Memory

Snapshot without Virtual Machine’s Memory option:

Snapshot taken without memory option will not capture the live stateof the virtual machine. Snapshot creates crash consistent disks, which you can be used to restore the virtual machine to the state prior to the snapshot but it won’t revert the VM into same power state as it during snapshot creation.

Virtual Machine can be either in Powered on or Powered off state to take VM snapshot without memory option.

Let’s say I have taken a snapshot when the virtual machine is powered on and before the patch upgrade. After I revert back to the snapshot taken without memory, the Virtual machine will be restored to the same data and state but power state of the virtual machine won’t be Powered on. It will be powered off. You need to manually power on the VM after reverting the snapshot.

VMware snapshot without Memory

VM Snapshot with Quiesce  Guest File System 

If you select the option “Quiesce guest file system” when taking snapshot, <quiesce> flag is 1 or true, It quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system’s in-memory cache to disk, or other higher-level application-specific tasks.

The virtual machine needs to be “powered on” State to take a snapshot with the quiesce file system option and also it requires VMware Tools to be installed on the Guest OS  to quiesce the file system in the virtual machine.

Note: Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup. Quiescing is not necessary for memory snapshots; it is used primarily for backups.

VMware Snapshot

 

How does VMware Snapshot work?

  1. When you initiate the request to create, delete or revert snapshot operation via client (Web Client, vSphere Client or Power CLI), Request will be sent from the client to Server using VMware API
  2. Snapshot create, delete or revert request will be forwarded to the ESXi host where that particular virtual machine is running. This process will happen only when initiating the snapshot create, delete or revert request from vCenter Server.  This will be skipped if you sent create snapshot request by directly connecting to the ESXi host.
  3. If you select Snapshot with Virtual machine’s memory option, ESXi hostwrites the memory of the virtual machine to the disk. the virtual machine is stunned throughout the duration of time the memory is being written.
  4. If you select snapshot with the quiesce guest file system option, the ESX host requests the guest operating system to quiesce the disks via VMware Tools. Depending on the guest operating system, the quiescing operation can be done by the sync driver, the vmsync module, or Microsoft’s Volume Shadow Copy (VSS) service.
  5.  ESXi host makes the modification to the virtual machine’s snapshot database file (.vmsd) and it reflects the changes in the snapshot manager of that virtual machine
  6. ESXi host calls Virtual DISK API functions to make changes to the child disks (-delta.vmdk and .vmdk) files and to the disk chain.

When you create a snapshot, The state of the virtual disk at the time of snapshot is preserved and writes to the VMDK file is not allowed. The system creates an additional VMDK file called delta disk for each VMDK disk in the datastore and allows to write any changes to that delta disk. The delta disk represents the difference between the current state of the virtual disk and the state that existed at the time the previous snapshot was taken.

If multiple snapshots are taken, Delta disks will be created for each of VM disk of the every snapshot and it can represent the difference between each snapshot.

When you delete the snapshot, The changes between snapshots and the previous disk states are merged and all the data from the delta disk that contains the information about the deleted snapshot is written to the parent VMDK disk. The amount of time it takes to commit or delete snapshots depends on how much data the guest operating system has written to the virtual disks since the last snapshot was taken.

Virtual Machine Snapshot Files

Virtual machine snapshot consists of multiple files. Virtual Machine snapshot mainly includes the following:

Settings state : Virtual machine’s settings (.nvram & .vmx) and power state

Disk State: state of the virtual machine’s associated disks

Memory state: Contents of virtual machine’s memory (Only if memory snapshot is selected)

Below are the some of the files which comprise of snapshot:

VM_name.vmsd

VM_name-snasphot#.vmsn

VM_name-00000#.vmdk

VM_name-00000#-delta.vmdk

VM_name-snapshot#.vmem

 

Below is the comparison of  Virtual machine files in the virtual machine directory before and after creating the first snapshot of the virtual machine.

VMware Snapshot

You can see the Delta disk information from the SSH session of ESXi host when browsing towards the virtual machine directory.

VMware Snapshot_4

Let’s discuss in detail about the what does each of the snapshot file means and purpose of the each file.

.vmsd

  • VM_name.vmsd is the snapshot list file and is created at the time that the virtual machine is created. It will be present in the Virtual machine directory regardless of the snapshot is present or not for that virtual machine.
  • The .vmsd file stores the names, description, and relationships for all of the virtual machine’s snapshots.
  • It maintains snapshot information for a virtual machine so that it can create a snapshot list in vSphere Web Client. This information includes the name of the snapshot .vmsn file and the name of the virtual disk file.
  • You can validate the size of the .vmsd file is grown after the snapshot is taken from the above screenshot which compares the files before and after VMware snapshot creation.

VMware Snapshot

-snasphot#.vmsn

  • VM_name-snasphot.vmsn is the snapshot state file and is used to store the state of the virtual machine when a snapshot is taken.
  • New .vmsn will be created for every snapshot and will be deleted when that snapshot is deleted
  • Size of the .vmsn varies based on the option selected during snapshot creation. For example, If you have selected Virtual machine’s memory during snapshot creation, it increases the Size of .vmsn file
  • I have 2 snapshots called “Snap_1″ & “Snap_2” for the virtual machine named “Redhat-1”. It has 2 .vmsn files (Redhat-1-snapshot1.vmsn & Redhat-1-snapshot2.vmsn) is created  in the virtual machine directory

VMware Snapshot

 

VM_name-00000#.vmdk

  •       VM_name-00000#.vmdk  is the disk descriptor file. This small text file contains information about the snapshot and snapshot disks.
  •        It gets created for each of the VMDK file, when you take the snapshot. For example, In the below screenshot, I have VM called “Redhat-1” has 2 VMDK’s (Redhat-1.vmdk & Redhat-1_1.vmdk), After taking first snapshot called “Snap_1”, It has created “Redhat-1-000001.vmdk & Redhat-1_1-000001.vmdk” and after second snapshot “Snap_1”, It created “Redhat-1-000002.vmdk & Redhat-1_1-000002.vmdk“

 

VMware Snapshot_7

Each VM_name-00000#.vmdk  updates the information about its Parent disk information. For example, For the First snapshot, “Redhat-1-000001.vmdk” updated its parentFileNameHint as the actual base disk (actual VMDK file).

After Second Snapshot, “Redhat-1-000002.vmdk” is updated with the ParentFileNameHint as “Redhat-1-000001.vmdk” which is the disk descriptor file created for the snapshot. So for the second Snapshot, Parent is the first snapshot and for the first snapshot, parent is base disk.

VMware Snapshot

When you checked from the virtual machine properties -> Click on Hard Disk -> you can see the Hard disk is mapped with the latest snapshot disk descriptor (VM_name-00000#.vmdk)

VMware Snapshots

VM_name-00000#-delta.vmdk

  • VM_name-00000#-delta.vmdk is the delta disk file.
  • State of each virtual disk of the virtual machine is preserved when you take a snapshot of a virtual machine.
  • Virtual Machine stops writing to its VM_name-flat.vmdk file and all the writes will be redirected to delta disk “VM_name-00000#-delta.vmdk.
  • It gets created for each of the flat-vmdk file, when you take the snapshot. For example, In the below screenshot, I have VM called “Redhat-1” has 2 VMDK’s (Redhat-1-flat.vmdk & Redhat-1_1-flat.vmdk), After taking first snapshot called “Snap_1”, It has created “Redhat-1-000001-delta.vmdk & Redhat-1_1-000001-flat.vmdk” and after second snapshot “Snap_1”, It created “Redhat-1-000002-delta.vmdk & Redhat-1_1-000002-delta.vmdk“

VMware Snapshot

VM_name-snapshot#.vmem

  • .vmem file will be created only you have selected the option “Snapshot Virtual machine’s memory” during the snapshot creation.
  • This file contains the entire contents of the virtual machine’s memory at the time of snapshot creation

VMware snapshot

 

Managing VMware Snapshots

Snapshots tab allows you to manage the virtual machine snapshots such as Revert to (Go To), Edit Snapshot, Delete Snapshot, Delete All Snapshots

Managing VMware Snapshots

In the Snapshot tab, you can perform the following actions:
• Edit Snapshot: Edit the snapshot name and description.
 Delete Snapshot: Removes the snapshot from the Snapshot Manager and consolidates the snapshot files to the parent snapshot disk and merge with the virtual machine base disk.
• Delete All Snapshots: Commits all the intermediate snapshots before the current-state icon (You Are Here) to the VM base VMDK file and removes all snapshots for that virtual machine.
• Revert to: Enables you to restore, or revert to, a particular snapshot. The snapshot that you restore becomes the current Snapshot.

We will discuss in In-depth Details of each of the above actions with various Scenarios.

Delete Snapshot

Delete Snapshot operation removes the snapshot from the Snapshot Manager and consolidates the snapshot files to the parent snapshot disk and merge with the virtual machine base disk. Let’s look into the detailed information of Delete Snapshot operation.

I have 2 VM snapshots “Snap_1″ & “Snap_2” for the virtual machine “winsvr”. If I delete the Snapshot “Snap_2”, It will consolidate the snapshot data to its parent snapshot  (Snap_1) disk. In this Example, “Snap_1” is the Parent of Snapshot “Snap_2”.

Before Deleting the Snapshot “Snap_2”, Below are the size of the each disks

Base Disk  (Winsvr-flat.vmdk) -> 21.4 GB

Snap_1 Delta disk (winsvr-000001-delta.vmdk) -> 2.7 GB

Snap_2 Delta disk (winsvr-000002-delta.vmdk) -> 3.4 GB

Delete VMware Snapshot

To delete the Snapshot “Snap_2” Select the particular snapshot and click on “Delete”

Delete Current Sansphot_2

Snapshot “Snap_2” is deleted and now VM has only one Snapshot “Snap_1”

Delete Current Sansphot_3

After the Snapshot removal, Snap_2 data (3.4 GB) is consolidated into its parent snapshot disk (Snap_1 delta disk) and Snap_1 Delta disk (winsvr-000001-delta.vmdk) is grown from 2.7 GB to 6.3 GB.

Delete VMware Snapshot

Delete Current Sansphot_4

Delete Intermediate Snapshot

When you delete a snapshot one or more levels above “You Are Here”, The snapshot state is deleted and that data will be committed to the virtual machine base disk. Refer the Below example:

If you delete the snapshot “Snap_1”, Snap_1 data (winsvr-000001-delta.vmdk) is committed into the Base disk (winsvr-flat.vmdk) and the foundation for the snapshot “Snap_2” will be retained

Deleting VMware Snapshot

 

Let’s take a look at the step by step demo of deleting Snapshot one or more level above “You are Here” state. I have currently 2 snapshots (Snap_1 & Snap_2) for the virtual machine “winsvr”.

I also have created Folder “File1” before taking snapshot “Snap_1” and also created Folders after creating each of the snapshots.

Delete Intermediate VMware Snapshots

In the below Screenshot, You can notice the 2 delta disks created as part of the Snapshot creation

Base disk -> winsvr-flat.vmdk

Snap_1  -> winsvr-000001-delta.vmdk

Snap_2 -> Winsvr-000002-delta.vmdk

I have also validated the actual usage of each of the above flat.vmdk and delta.vmdk using the below command

du -h servername-flat.vmdk
or 
du -h servername-00000#-delta.vmdk

Delete Intermediate VMware Snapshots_2

To delete the Snap_1, Select the “Snap_1” and click on Delete.

Delete Intermediate VMware Snapshots_3

Once the Snapshot “Snap_1” is deleted, I don’t see the “Snap_1” delta disk “winsvr-000001-delta.vmdk”. As we stated earlier, When you delete a snapshot one or more levels above “You Are Here”, The snapshot state is deleted and that data will be committed into the virtual machine base disk.  

I have deleted the Snapshot “Snap_1” which had 2.8 GB of data. After the Snap_1 deletion, Snap_1 data is committed into the Base disk (winsvr-flat.vmdk) which was grown from 16 GB + 2.8 GB (Snap_1 data) = 18.7 GB

Delete Intermediate VMware Snapshots_5

Delete VM Snapshot below “You are Here”

When you delete a snapshot one or more levels below “You Are Here”, Subsequent snapshots are deleted and you can no longer return to those states. The snap_2 data is deleted.

For example, You have 2 Snapshots (Snap_1 & Snap_2) and your current snapshot is Snap_2, When you revert your virtual machine to snap_1, Snap_2 snapshot data will be deleted and the virtual machine can no longer return to the state of Snap_2.

Delete Snaphot one level above You are Here_3

VMware Snapshot

 

Delete All VM Snapshots

Prior to vSphere 4 update 2, Delete All Snapshots option from the Snapshot manager would require additional space to perform the operation, incase of committing multiple snapshots.  When using Delete All in the Snapshot Manager, the snapshot furthest from the base disk is committed to its parent, causing that parent snapshot to grow. When that commit is complete, that snapshot is removed and the process starts over on the newly updated snapshot to its parent. This continues until every snapshot has been committed. This can lead to an aggressive use of additional disk space if the snapshots are large.

VMware Updated the algorithm of Delete All Snapshots, Delete all snapshots operation to commit every snapshot of the chain directly to the Base Disk(s) of the virtual machine.  With this new Delete all algorithm,

  • If the Base Disk is thick provision (pre-allocated), no extra space is required for the Delete all operation. The Base Disk will not grow as it is preallocated or thick.
  • If the Base Disk is thin provision (non-preallocated), the base disk will grow only on committing information from the snapshots. Each thin provision disk may grow up to its maximum size as mentioned in the Provisioned Size option in the virtual machine settings for the disk.

All flat and delta files that are used by the chain of snapshots are locked.

Delete All VMware Snapshots_4

 

I have 2 snapshots called “Snap_1” & “Snap_2”. You can notice the delta files related to the snapshots. Before performing Delete All  opeartion of 2 snapshots. I have checked the actual usage of base disk and each delta disk of both snapshots

Base disk (winsvr-flat.vmdk)  -> 23.4 GB

Snap_1 delta disk (winsvr-000001-delta.vmdk) -> 6.8 GB

Snap_2 delta disk (winsvr-000002-delta.vmdk) -> 3.4 Gb

Delete All VMware Snapshots

To perform Delete All VMware Snapshot, Click on Delete All.

Delete All VMware Snapshots_2

Once Delete All operation is completed, I can notice that all the snapshot disks are committed to the virtual machine base disk (winsvr-flat.vmdk) and size of the base disk is grown from 23.4 GB to 33.2 GB (6.8 GB Snap_1 data + 3.4 GB Snap_2 data)

Delete All VMware Snapshots_3

Revert to Snapshot

Revert to snapshot will restore your virtual machine to the state that they were in at the time when you took the snapshot. If you revert to the VMware snapshot, your virtual machine will be in the powered off state, unless you took the Memory Snapshot. Reverting the memory snapshot will revert the virtual machine to state and it also maintains the live power state also.

Let’s see the step by step demo to understand the Revert to Snapshot option. I have created a  Folder called “File 1” created on the desktop of the virtual machine. File 1 folder is 6.48 GB of Size before taking the snapshot.

Revert to VMware Snapshot

I am creating the snapshot called “Snap_1”

Reverting VMware Snapshots_2

Specify the Snapshot name and description. I am taking the memory snapshot of the virtual machine to revert to live state of the virtual machine.

Reverting VMware Snapshots_3

Once the snapshot “Snap_1” is created for the virtual machine, I am deleting the folder “File 1” from the desktop.

Reverting VMware Snapshots_4

I have deleted the folder “File 1”. Let’s  revert the virtual machine to the snapshot “Snap_1”. To revert to the snapshot “Snap_1”, Select the snapshot “Snap_1” and  Click on “Revert To” to revert to the snapshot.

Reverting VMware Snapshots_5

Click on “Yes” to confirm to revert to snapshot “Snap_1”

Reverting VMware Snapshots_6

Revert to snapshot is completed successfully.

Reverting VMware Snapshots_7

Once Revert to the snapshot “Snap_1” is completed, Virtual machine “winsvr” is reverted to the state when you to took the snapshot. Since I have created a folder “File 1” prior to creating a snapshot. File 1 reappeared on the desktop.

Reverting VMware Snapshots_8

When you revert a virtual machine with snapshot, the current snapshot (delta disk) will be removed and a new one will be created linking to the parent disk.For Example, I have only one snapshot “Snap_1” and it has its delta file called “winsvr-000001-delta.vmdk”. After reverting the virtual machine to snapshot “Snap_1”, winsvr-000001-delta.vmdk  got deleted and new delta file called “winsvr-000002-delta.vmdk is created.

Revert to VMware Snapshot

Reverting VMware Snapshot One level above the Current Snapshot

Let’s take a look at the Scenario of Revert to the snapshot One level above the current snapshot. For Example, You have a virtual machine with 2 Snapshots “Snap_1” & “Snap_2”. Snap_2 is the current snapshot. This scenario talks about reverting your virtual machine to Snapshot 1 “Snap_1” instead of recent snapshot “Snap_2”.

Reverting VMware Snaphot One level above the Current Snapshot

Let’s revert to the Snapshot 1 “Snap_1”. Select the snapshot “Snap_1” and Click on “Go To” to revert the virtual machine to Snapshot 1.

 

Reverting VMware Snaphot One level above the Current Snapshot_2

When reverting to the Snapshot 1 “Snap_1”, All the files, folders and changes made after the Snapshot 1 will be lost. I don’t see any of the folders created after Snapshot 1.

Reverting VMware Snaphot One level above the Current Snapshot_3

An interesting fact is There is no longer “Winsvr-000002-delta.vmdk” delta file  present and New delta file called “Winsvr-000003-delta.vmdk” is created.

Reverting VMware Snaphot One level above the Current Snapshot_4

Virtual Machine VMDK also mapped with the newly created disk “winsvr-000003.vmdk”

Reverting VMware Snaphot One level above the Current Snapshot

All the active writes to the virtual machine are passed to the newly created virtual machine. You can notice the new delta file 000003-delta.vmdk. The new delta file is growing when new writes are coming. It has grown from 49 MB to 2.8 GB.

Snapshot Consolidation

Snapshot Consolidation is intended to resolve the problems that might occur with the snapshots. Snapshot consolidation is a method to commit a chain of snapshots to base disks when Snapshot manager shows no snapshots exist but the delta files are present in the virtual machine directory on the datatsore.

Snapshot consolidation is a way to clean unneeded snapshot delta files from a datastore. Left out delta files may continue to grow until the virtual machine runs out of datastore space.

With snapshot consolidation, vCenter Server displays a warning when the descriptor and the snapshot files do not match. After the warning is displayed, the user can use vSphere Web Client or vSphere Client to commit the snapshots.

Consolidate VMware Snapshot

To Perform consolidation, Click on Snapshot -> Consolidate

Virtual Machine Consolidation_1

Click on “Yes” to confirm the consolidated operation.

Virtual Machine Consolidation_2

Limitations of VMware Snapshot

Keeping snapshots for a longer duration than recommended can affect the virtual machine performance and also VM snapshot is not supported for some disk types and VM configured with bus sharing. Let’s understand what is the limitation of Virtual Machine Snapshots:

  • VMware Snapshot is not supported for raw disks and RDM Physical mode disks. Snapshot is supported for RDM with virtual compatibility mode.
  • VMware Snapshot does not support for independent disks. Virtual machines with independent disks must be powered off before you take a snapshot. Snapshots of powered-on or suspended virtual machines with independent disks are not supported.

Virtual Machine Snapshot Limitation -Independent Disk

  • Snapshots are not supported with PCI vSphere Direct Path I/O Devices
  • VMware does not support snapshot for the guest operating systems that use an iSCSI initiator in the guest.
  • VMware Snapshots are not supported for a Virtual machine configured withbus sharing.

Virtual Machine Snapshot Limitation - Bus Sharing

  • The virtual machine with VMDK larger than 2 TBs, snapshot operations can take significantly longer to finish.
  • A snapshot can negatively affect the virtual machine performance and Performance degradation is based on How long you keep the snapshot, depth of the snapshot tree and also based on the write rate of the virtual machine and its guest operating system have changed from the time you took the snapshot.
  • Snapshots are not meant for the method of backup and recovery. The snapshot only provides Point-in time image. If the files containing a virtual machine are lost, its snapshot files are also lost.

Best Practices for VMware Snapshots

  • VMware recommends only a maximum of 32 snapshots in a snapshot chain. However, for a better performance, use only 2 to 3 snapshots.
  • Do not use a single snapshot for more than 24-72 hours. Keeping VMware snapshot for longer duration will negatively affect the virtual machine performance
  • Snapshot file continuous to grow in size and when keeping snapshot for a longer duration, this can cause the datastore to grow and even sometimes it causes outage due to datatsore space issues.
  • Ensure Snapshots are deleted after every backup when using third-party backup softwares.
  • Effectively use vCenter Alarms to monitor the virtual machine snapshot and datastore space usage
  • You can also make use of PowerCLI script to report the virtual machine usage and also the age of the virtual machine snapshot.
  • Do Not attempt to increase the disk space of the virtual machine disk, when Snapshot is present. Increasing the disk size, when snapshots are still available can corrupt snapshots and result in data loss.
  • If you using vSphere version prior to vSphere 5.0, Ensure there are no snapshots prior to performing storage vMotion. Storage vMotion is supported for VM with snapshot after vSphere 5.0

VMware Snapshot Advanced Operations

Exclude Virtual Machine disk from VMware snapshot

We need to set the virtual disk to independent mode to exclude the disk from any snapshots. Power off the virtual machine and delete any existing snapshots before you change the virtual disk mode to Independent. Deleting a snapshot involves committing the existing data on the snapshot disk to the parent disk.

One of the real-time use cases  I can think is ORACLE RAC. Basically, in Oracle RAC have multiple disks and it will be shared between the cluster nodes and it will be kind of Multi-writer enabled virtual disks. When performing patching for the ORACLE RAC VM’s, We used to configure all the shared disks as Independent persistent to exclude from the snapshot. So only OS disk will be included as part of snapshot operation.

To configure the virtual disk as Independent disk, Right-click the virtual machine -> Edit Settings -> Virtual Hardware ->Expand the Hard Disk -> Select Independent -Persistent or Independent-Nonpersistent -> Click on OK.

Exclude VM disk from VMware Snapshot

Independent -Persistent: Changes to the disk are immediately and permanently written to the disk as same like a conventional disk in the phsyical computer.

Independent -NonPersistent: Changes to the disk are discarded when you power off or reset the virtual machine.  Changes to the disk are written to and read from a redo log file that is deleted when you power off or reset.

 

Exclude VM disk from VMware Snapshot_2

Even though I have virtual disks (Hard Disk 1 & Hard Disk 2) in the virtual machine “winsvr”, It created only one delta file for the Hard Disk 1 and no delta file is created for the Hard Disk 2 because it is configured as Independent disk.

Exclude VM disk from VMware Snapshot_3

Control Maximum of VMware Snapshot Per Virtual Machine

Even though The maximum supported amount of snapshots in a chain is 32,  It is recommended not to take more than 2 or 3 snapshots for production virtual machines. How do you control a maximum number of VMware snapshot per virtual machine? Currently, there is no way to control the maximum number of snapshot per virtual machine. There is an undocumented VMX entry as per William Lam’s article, which can control the maximum number of snapshots per virtual machine.

To add the advanced configuration entry to control the maximum number of VMware Snapshot per virtual Machine, Right-click the Virtual Machine -> VM Options ->Edit Configuration

control maximum number of VMware snapshots

Click on Add Row -> Enter “snapshot.maxSnapshots” in the Filed and Enter the number of snapshots in the unit in the Value column.

snapshot.maxSnapshots = n

In the below Screenshot, I have enter 3 to control the maximum number of the snapshot for the virtual machine to 3.

control maximum number of VMware snapshots_2

I already created 3 snapshots for the virtual machine “Winsvr”.

control maximum number of VMware snapshots_3

When i tried to created the 4 snapshot for the virtual machine “winsvr”. It throws the error “msg.snapsot.error-MAXSNAPSHOTS”

control maximum number of VMware snapshots_4

So we are clear that we are able to control the maximum number of VMware snapshots per virtual machine using this advanced paramenter.

Change default VMware Snapshot location

In ESXi 5.0 and later, virtual disk redolog (-delta.vmdk) files for snapshots are placed in the same directory as the parent virtual disk (.vmdk) file. There are some situations like your VMFS datastore don’t have enough storage to accommodate the vm Snapshot or you may not able to power on the vm due to insufficient space to hold your Swap file in the VMFS datastore. You may need to change the default location of your vm snapshot and point it to the different datastore, where it have enough space to store your snapshot related files (virtual disk redolog (-delta.vmdk)). Specifying the working directory for the virtual machine will ensures that subsequently created vm snapshots cause new virtual disk redolog (-delta.vmdk) files to be created in the defined datastore.

We need to make the advanced configuration entry in .VMX file of your virtual machine. I have written detailed article to change the default snapshot location.

Monitor & Report VMware Machine Snapshots

Creating vCenter Alarm to Monitor VMware Snapshot Usage

In production environment, it is necessary to monitor and report the virtual machine snapshots usage information. I will explain the procedure the configure the vCenter Server alarm to monitor the usage of VMware Snapshot and what actions need to perform when particular alarm is triggered.  Let’s see the step by step procedure how to configure vCenter alarm for snapshot usage.

To Configure vCenter Alarm ->Login to the vSphere Web client  ->Right-click the object you want to add the alarm  -> Navigate to Alarms > New Alarm Definition.

vCenter Alarm for VMware Snapshot Usage

Enter an alarm name and description. Select Virtual Machines from the Monitor drop-down. Select “specific conditions or state, for example CPU usage”  in the Monitor for option. Select the checkbox ” Enable this alarm” Click on Next.

vCenter Alarm for VMware Snapshot Usage_2

In the Trigger Type dropdown, click VM Snapshot Size (GB).

vCenter Alarm for VMware Snapshot Usage_3

Select “Is above” in the operator and Specify the Size in GB  for warning condition and critical condition. Click on Next.

vCenter Alarm for VMware Snapshot Usage_4

Select the action you want to occur when the alarm is triggered. I am specifying the action to “send a notification email”

vCenter Alarm for VMware Snapshot Usage_5

Specify the email address to send an notification email.  You can specify the frequency of email for each of the change of alarm condition.  Specify the duration of the repeat actions. Click on Finish to complete the alarm configuration.

vCenter Alarm for VMware Snapshot Usage_6

Power CLI Script – Report Virtual Machines with 3 days older VM Snapshots

I have written an excellent  article for the PowerCLI script which helps us toreport the Virtual Machines with 3 days older VM snapshots and export the output in Excel (.CSV) file. It is always recommended not to keep VM snapshots more than 24- 72 hours. The VM snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. This PowerCLI script could be really useful to identify the Virtual machines with 3 days older snapshot and delete them immediately or raise a concern to the respective team for the VMware snapshot deletion before VM snapshots causes outage to your production infrastructure.

PowerCLI script provides an output in CSV file in the below format. It provides the VM information, snapshot name and date it is created. It reports the virtual machine with snapshots which is older than 3 days.  As per VMware best practices, It is not recommended to keep the snapshots more than 24-72 hours. This script help you to achieve the best practices for VMware Snapshot.

Powercli to report snapshots older than 3 days

Managing VMware Snapshot from Command Line

vSphere Web Client and vSphere Client were always helpful but we need to also understand how to manage VMware snapshots operation from the ESXi host CLI. In this part, I will explain few of the general snapshot related operations from command line. All the below operations are based on “vim-cmd”. Before performing any operation on the virtual machine, we need to get the vmid of the virtual machine. To get the vmid of the virtual machine, Execute the below command

vim-cmd /vmsvc/getallvms

It will lists the vmid, name of the virtual machine, vmx file location, Guest OS, hardware version and annotation, if any. We need to make note of vmid to perform vmware snapshot related operation on that particular virtual machine.

Manage VMware Snapshot from ESXCLI

Get Snapshot Details

To get the Vmware snapshot details  of the virtual machine, execute the below command:

vim-cmd /vmsvc/snapshot.get <vmid>

vim-cmd/vmsvc/snapshot.get 7

It will list the Snapshot name, Snapshot ID, Snapshot description, Date created on and snapshot state. We need Snapshot ID to any operation related to that snapshot.

Manage VMware Snapshot from ESXCLI_2

Delete Snapshot

To delete the snapshot, first get the snapshot details using “snapshot.get” command and note down the snapshot ID. Execute the below command to delete the snapshot called “Snap_1” and Snapshot ID is “1”.

vim-cmd /vmsvc/snapshot.remove <vmid> <snapshot ID>

vim-cmd /vmsvc/snapshot.remove 7 1

Manage VMware Snapshot from ESXCLI_3

Create  Snapshot

We can create the snapshot from the command line. Below is the command line format. We need to specify the snapshotname, snapshot descritpion to create the snapshot. Optionally you can specify the snapshot option such as include memory snapshot and Quiesce file system. Use the below command to create the snapshot.

vim-cmd /vmsvc/snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

vim-cmd/vmsvc/snapshot.create 7 "snap_1" "snapbefore patching" 1 1

Manage VMware Snapshot from ESXCLI_4

I can also confirm from the vSphere web client that “snap_1” is created from the above command.

Manage VMware Snapshot from ESXCLI_5

Revert to Snapshot

You can also revert to the snapshot from the command line. Execute the below command to revert to the snapshot. You need to specify the vmid snapshot ID  and option to suppress the PowerOn of the VM.

vim-cmd vmsvc/snapshot.revert <vmid> <snapshot ID> <suppressPowerOn>

vim-vmd vmsvc/snapsot.revert 7 2 1 or 0

Manage VMware Snapshot from ESXCLI_6

Delete All Snapshots

You can perform the Delete All operation from command line as same as vSphere Web client and vsphere client. Just specify the VMID to delete all the snapshot of the virtual machine. No need to specify any snapshot related information.

vim-cmd /vmsvc/snapshot.removeall <vmid>

vim-cmd /vmsvc/snapshot.removeall 7

Thats’ it. We are done with Delete All option. It consolidates all the snapshot data to the base disk.

Manage VMware Snapshot from ESXCLI_7

What ‘s new with Snapshot in vSphere 6 & vSphere 6.5

Changes to Consolidation procedure in vSphere 6.0

Prior to vSphere 6.0, Consolidation of virtual machine snapshot will create an additional helper snapshot and all the new I/Os will be redirected to this newly created snapshot. Once all the changes are stored in the snapshot disk have been merged into the base disk, the helper snapshot was also committed. Helper snapshot ill also grow considerably during the consolidation operation. If the time to consolidate the helper is within a certain time-frame (12 seconds), Virtual machine will be stunned and consolidate the helper snapshot into the base disk. If it was outside the acceptable time-frame (12 seconds), then the process will be repeated to create new helper snapshot while consolidation of helper snapshot is happening until the helper could be committed to the base within the acceptable time-frame.

If the Virtual machine has more I/O, This process could never successfully consolidate the snapshot chain and helpers.

In vSphere 6.0, VMware introduced new process for consolidation process which uses the mirror driver (which is same as in the storage vMotion). With this mirror driver mechanism, changes to the virtual machine are written to the active VMDK and the base disk (wile protecting the write order) during consolidation. Snapshot consolidation will be completing now in 1 pass with minimal or indeed no helper disks. Consolidation duration is dramatically shorter stun time of the virtual machine. There is an excellent article from Cormac Hogan talking about this Consolidation changes in vSphere 6.0

Changes to Snapshot Format on VMFS 6 on vSphere 6.5

In vSphere 5.5 & vSphere 6.0, Snapshots taken on VMDK’s larger than 2 TB are Space Efficient Virtual Disk (SESPARSE) format. No user interaction is required. The redo logs are automatically created as SESPARSE instead of VMFSSPARSE (delta) when the base flat VMDK is larger than 2 TB. VMFS5 uses the VMFSsparse format for virtual disks smaller than 2 TB.

VMFSsparse is implemented on top of VMFS, and I/Os issued to a snapshot VM are processed by the VMFSsparse layer. Technically, VMFSsparse is a redo-log that starts empty, immediately after a VM snapshot is taken. The redo-log grows to the size of its base vmdk, when the entire vmdk is rewritten with new data after the VM snapshotting. This redo-log is just a file in the VMFS datastore. Upon snapshot creation, the base vmdk attached to the VM is changed to the newly created sparse vmdk. This is nothing but the Delta files.

From vSphere 6.5, SEsparse is a default format for all delta disks on the VMFS6 datastores. If you have VMFS 5 datastore on ESXi 6.5 hosts, Still it will be VMFSsparse (delta files). SEsparse is a format similar to VMFSsparse with some enhancements. This format is space efficient and supports space reclamation. With space reclamation, blocks that are deleted by the guest OS are marked and commands are issued to the SEsparse layer in the hypervisor to unmap those blocks. This helps to reclaim space allocated by SEsparse once the guest operating system has deleted that data.

When I create the Snapshot of the Virtual machine “vcsa-demo-1” which is running on ESXi 6.5 and VMFS 6, You can notice that delta disk format is “VMname-000001-sesparse.vmdk”. I cannot find any delta disk with “vmname-000001-delta.vmdk”

vSphere 6.5 VMware Snapshot Feature

I took the snapshot of the virtual machine “App-1” which is running on ESXi 6.5 but on datastore with VMFS 5. It has delta file with the format “vmname-000001-delta.vmdk”. For sesparse , We need to have VMFS 6 datastore even running on ESXi 6.5

vSphere 6.5 Snapshot Feature_2

Storage vMotion of VM with Snapshot from VMFS 5 to VMFS 6 Datastore

I took the snapshot of the virtual machine “App-1” which is running on ESXi 6.5 but on datastore with VMFS 5. It has delta file with the format “vmname-000001-delta.vmdk”

Sphere 6.5 Snapshot Feature_VM “App-1” is running on the Datastore with VMFS 5.

Storage vMotion of VMware Snapshot

Let’s migrate the vm with snapshot from VMFS 5 to VMFS 6 datastore.

Storage vMotion of VMware Snapshot_2

Storage vMotion of the virtual machine “App-1” is completed.

Storage vMotion of VMware Snapshot_3

Virtual machine with snapshot is running on VMFS 6

Storage vMotion of VMware Snapshot_4

Prior to Storage vMotion to VMFS 6 datastore, delta files was in VMFSsparse format (VMname-000001-delta.VMDK). After Storage vMotion to VMFS 6 datatsore, delta file is chnaged to SEsparse (VMname-000001-sesparse.VMDK)

Storage vMotion of VMware Snapshot_5

Similar to that, When a VM with a vmdk of the size smaller than 2 TB is migrated to VMFS5, the snapshot format changes to VMFSsparse. You cannot mix VMFSsparse redo-logs with SEsparse redo-logs in the same hierarchy.
That’s it. We are reaching the end to this deep dive article to master VMware Snapshot. I hope that I have covered almost all aspects of VMware Snapshots and its terminologies and will update the article regularly if any new changes to VMware Snapshot. I hope this is informative for you. Thanks for Reading !!! . Be social and share it in social media, if you feel worth sharing it.

vSphere 6.5 Blog posts: