VMware Nested Virtualization Explained: Use Cases and Tutorial

When you run virtual machines in VMware vSphere, you should protect the VM data. VMware vSphere provides features that improve availability and reliability, including clustering features. However, a dedicated backup solution for VMware vSphere offers more reliable protection and more recovery options. Read about disaster recovery in this white paper and learn how to create an effective data protection strategy for environments implementing VMware virtualization technologies.

What Is Nested Virtualization

Nested virtualization refers to running a virtual hypervisor inside a physical hypervisor as a virtual machine (VM). This feature allows you to run VMs on the virtual hypervisor. Put simply, nested virtualization is when you run a VM inside a VM. Virtual machines running on a virtual hypervisor are called nested VMs. A virtual hypervisor runs on top of the virtual hardware of a VM and views this virtual hardware as real host hardware. Confused? Let’s take a closer look at all these terms.

• A host hypervisor is a hypervisor running on physical hardware.
• A guest hypervisor is a hypervisor running on virtual hardware (on a VM).
• An outer guest is a VM running on physical hardware. A guest OS is an operating system running on a VM.
• An inner guest is VMware’s term for a VM running within another VM (on virtual hardware).

The screenshot below shows different layers of VMware virtualization. The ESXi VM in Layer 2 is the outer guest, and the nested VMs in Layer 3 are inner guests. Technically, you can use deeper levels of nesting (Layer 4, Layer 5, etc.) if your computer has the resources for this high level of performance. Deeper levels of nesting are not illustrated in the image.

Hardware-assisted virtualization (also called hardware virtualization, HV, or native virtualization) uses the underlying physical hardware of a computer via software to run virtual machines. Hardware features help virtualize machines efficiently in this case compared with binary translation and paravirtualization that were used before. Intel and AMD include native virtualization support at the hardware level with their Intel VT-X and AMD-V features. Hardware virtualization is required to run 64-bit guests. Note that Intel VT-X or AMD-V must be enabled in UEFI/BIOS.

Supported hypervisors

Both type 1 and type 2 hypervisors can run nested VMs. Type 1 hypervisors run on top of physical hardware and are also called bare-metal hypervisors. Type 2 hypervisors are installed on underlying operating systems that are installed on the physical hardware (also called hosted hypervisor). In VMware environments, ESXi is a type 1 hypervisor. Type 2 hypervisors are VMware Workstation, VMware Player, and VMware Fusion.

Requirements:

• Virtualized hardware-assisted virtualization (virtualized HV) must be supported by a hypervisor to enable nested virtualization and run nested VMs. If virtualized HV is enabled for the outer guest, you can run any hypervisor that supports hardware virtualization within a VM.
• Hardware-assisted virtualization and virtualized HV are supported by VMware type 2 hypervisors starting from VMware Workstation 8, VMware Player 4, and VMware Fusion 4. As for ESXi, virtualized HV is supported starting from ESXi 5.0.
• Virtual hardware 9 or higher must be used for a VM (outer guest).

Available deployment configurations:

• VMware ESXi running as a VM on VMware Workstation, VMware Player, or VMware Fusion
• VMware ESXi running as a VM on VMware ESXi
• VMware ESXi running as a VM on other third-party hypervisors such as Microsoft Hyper-V or VirtualBox
• Running a non-VMware hypervisor as a VM on VMware ESXi, VMware Workstation, VMware Player, or VMware Fusion

VMware Support

Technically nested virtualization works on the VMware hypervisors mentioned above. At the same time, VMware doesn’t support running nested VMs in production environments. It means that you cannot request official technical support from VMware if you use a virtualized hypervisor and nested VMs. VMware approves only one exception for using a nested VM in VMware vSphere, and that is using the vSAN Witness Appliance, which is a type of nested ESXi installation.
VMware supports enabling Hyper-V on VMs for virtualization-based security (VBS). Microsoft Virtualization-Based Security is a new feature with Windows 10 and Windows Server 2016. VMware supports the use of VBS on virtual machines running Windows starting from VMware vSphere 6.7. Read about running VMware Workstation on a new Windows 10 build where VBS is enabled by default and how to fix related errors.

Licensing Options

If you install ESXi and other vSphere components as VMs for a nested environment, you must license these components as if you were using them on physical servers. This rule is true for installing nested VMware hypervisors on any hypervisors. You can use trial versions of ESXi, vCenter, and other VMware components or install Free ESXi. Read more about VMware vSphere licensing and vCenter licensing.

You can provision one CPU with multiple CPU cores to a VM running a virtual ESXi instead of provisioning multiple CPUs each with one CPU core as long as you comply with the licensing rules. Consider tuning the cpuid.coresPerSocket parameter.

Use Cases for VMware Nested Virtualization

The advantage of nested virtualization is that you can run multiple virtual hypervisors with nested VMs on one real hypervisor installed on a physical machine. Saving costs is another advantage. Nested virtualization can be used for many scenarios. Let’s look at the most common ones.

Education. You can install VMware ESXi on VMs to get to know the functionality of VMware vSphere. You can create a cluster by deploying a VM for shared storage and multiple ESXi VMs. Then configure a cluster, create nested VMs, perform VM live migration, and explore how HA, DRS, storage DRS, and other vSphere features work. The advantage of this approach is that you don’t need to buy SAS controllers, SAS disk drives, switches, or other equipment until you know enough to make a decision as to what hardware configuration you need to deploy VMware vSphere. If your personal computer meets the requirements, you can use it to build a VMware home lab.

Development. If you or your team develop an application for VMware vSphere, you can use nested VMs running on virtual ESXi servers.

Testing. You can test your applications for VMware vSphere running on virtual ESXi servers with nested VMs. However, run your final testing on vSphere in a real environment to uncover issues that can be dependent on physical hardware. Another use case is testing a new version of a VMware hypervisor in a virtual environment before making a decision about installing this hypervisor version on hardware or upgrading the existing version installed on the hardware. If you have never upgraded VMware vSphere components, it is better to experiment in a virtual environment. You can test different scenarios and check new features.

Performance of Nested VMs

When you run a virtual machine on a VMware hypervisor, there is one process related to the running VM on the hypervisor. The process consumes the host’s physical RAM (Random Access Memory) depending on the virtual RAM used by the VM. If you run five VMs, there are five processes, etc. The more processes are running on a host, the more time is required for context switching and physical processor scheduling. As a result, the ready time increases, and the performance level drops.

When using VMware nested virtualization, a process on a physical ESXi host contains a process of an ESXi VM, and the ESXi running on the VM contains multiple processes of nested VMs running inside (in the process of the virtual ESXi host that is the outer guest). For this reason, nested VMs work slower than regular VMs. The amount of performance degradation depends on your hardware performance and the layer of nesting.

VMware Tools for a Nested VMware Hypervisor

ESXi installed within a VM requires VMware Tools. VMware Tools is a set of drivers and utilities installed on a guest OS. These tools improve VM performance and user experience. VMware Tools running on a virtual ESXi (outer guest) provides the following:

• Information about nested ESXi, which is displayed in VMware vSphere Client: IP address, hostname, etc.
• The ability to shut down and restart a virtual ESXi properly via vSphere API and vSphere Client (when clicking the appropriate buttons in the interface)
• The ability to run scripts when the power state of an ESXi host changes (power on, power off, etc.)
• Guest Operations API support (former VIX API) for operations inside a VM (console of a virtual ESXi in this case)

VMware Tools for ESXi running on a VMware VM was first released for vSphere 5.0, 5.1, and 5.5. VMware Tools for manual installation are provided as a VIB package for ESXi 5.x versions. Starting from VMware vSphere 6.0, VMware Tools built-in ESX, and there is now no need to install VMware Tools manually on ESXi running as a VMware VM.

You can check the VMware Tools state on a virtual ESXi with the command:

/etc/init.d/vmtoolsd status

Networking

Network configuration can be a challenge when you configure ESXi nested virtualization. However, knowing the working principle of L2 and L3 networks, and virtual switches can simplify networking configuration for VMware nested virtualization.

If you want to install a nested hypervisor on ESXi, you need to enable Promiscuous mode and Forged transmits in the configuration of a vSwitch. Optionally you can also enable MAC address change. These three options are disabled by default (have the Reject status) for security reasons.

Promiscuous mode

Promiscuous mode is a security policy of virtual switches that can be configured on the vSwitch level or port group level. VMkernel and other network interfaces connected to a port group with Promiscuous mode enabled can see L2 network traffic from all network devices passing this vSwitch (a network adapter receives all IEEE 802.3 Ethernet frames despite the destination MAC address defined in the frames). By default, a network adapter receives only frames that are addressed to this adapter, while frames that have other destination MAC addresses are dropped. Promiscuous mode can be also used for network monitoring and traffic sniffing/analyzing for diagnostic purposes. When Promiscuous mode is enabled, a virtual switch acts like a hub. Settings on the port group level can override settings on the vSwitch level.

A hardware (physical) switch learns MAC addresses of the network devices connected to the switch. Unlike traditional physical switches, a virtual switch doesn’t learn MAC addresses of connected network interface controllers. This is because VMware vSphere (ESXi and vCenter) already knows the MAC addresses of the virtual machines that have virtual network adapters connected to a virtual switch. In this situation, a virtual switch forwards only frames to a VM if the destination MAC address is the MAC address of the ESXi physical or virtual network interface controller (NIC).

When using ESXi nested virtualization, the destination MAC address of a VM NIC (for a nested VM) differs from the MAC address of the virtual ESXi host on which the nested VMs are running. For this reason, a virtual switch on a physical ESXi host drops these frames (if Promiscuous mode is disabled).

Promiscuous mode and performance

When Promiscuous mode is enabled on a virtual switch or port group, network performance drops. You can notice this change if you use ESXi nested virtualization for network-intensive workloads that produce large amounts of network traffic.

MAC address change and security

MAC address spoofing refers to the use of false MAC addresses in ARP poisoning. For this reason, changing a MAC address by a guest OS is the point. The ability to bind the MAC address of a VM network adapter to the MAC address set in the VMX configuration file of the VM increases security and reduces the risk of ARP poisoning attacks. This VMware functionality enables you to achieve the security level that is not available in traditional physical environments. That’s why MAC address change is disabled by default.

Note: A vSwitch knows the MAC addresses of network adapters connected to the vSwitch based on the notification mechanism instead of learning the passing traffic as with traditional physical switches.

Forged transmits

Forged transmits is the security policy that has an impact on the outgoing traffic generated by a VM (from a virtual network adapter of a VM connected to the virtual switch). Forged transmits is a policy similar to the MAC address change policy but is applied for inbound not outbound traffic.

Once forged transmits are accepted, an ESXi host allows a VM to send frames if the effective MAC address is different from the source MAC address defined in the frame header (this fact is also called MAC impersonation). The effective address is the MAC address set by the guest OS. In the forged transmits Reject mode, a host doesn’t send frames to a VM if the effective MAC address used by this VM differs from the source MAC address defined in the header. In this case, a guest OS cannot identify that the network issue is caused by MAC address spoofing. By default, forged transmits are set to the Reject mode.

Let’s look at an example. There are three nested VMs running on a virtual ESXi host. Each machine has a unique MAC address.

ESXi – 00:50:56:DD:DD:DD

Nested VM 1 - 00:50:56:AA:AA:AA

Nested VM 2 - 00:50:56:BB:BB:BB

Nested VM 3 - 00:50:56:CC:CC:CC

When any of these VMs makes an attempt to connect outside the virtual switch of a virtual ESXi host via the network adapter of the virtual ESXi host, a virtual switch connected to this network adapter of the virtual ESXi host checks the source MAC address defined in the Ethernet frame sent by the VM. The source MAC address doesn’t match the effective MAC address of the virtual ESXi host. The frames sent from this VM are treated as Forged Transmits and dropped. The Forged Transmits policy is set to Reject in this case.

In VMware vSphere 6.7, a distributed virtual switch supports the MAC address learning functionality. It means that starting from vSphere 6.7, you can use a distributed virtual switch (VDS) v6.6 or higher instead of enabling the Promiscuous mode and Forged transmits on a standard virtual switch to configure VMware nested virtualization with networking. The macManagementPolicy must be enabled in the VDS configuration.

How to Configure VMware Nested Virtualization

With the theoretical part covered, we can now get to the practical part and configure VMware nested virtualization. In this walkthrough, I am going to deploy a virtual machine running ESXi 7.0.3 and run a nested Windows VM on that ESXi VM. I use vSphere 6.7 including ESXi 6.7 on a physical server and vCenter 6.7. This is an example of using ESXi nested virtualization when you need to check the latest version of VMware vSphere in the existing vSphere environment.

I use the following machines:

• 192.168.101.205 – a physical host running ESXi 6.7. There are multiple VMFS datastores attached to this host.
• 192.168.101.103 – VMware vCenter Server running on another host. The hostname is vcenter6-7 (vcenter6-7.localdomain). This vCenter Server is used to manage the ESXi host (192.168.101.205).
• 192.168.101.131 – A VM running ESXi 7.0 Update 3 (ESXi 7.0.3). The VM name is ESXi7-VM.
• 192.168.101.0/24 – is the address of the network used to connect physical and virtual machines.

You can use the same configuration or use your own configuration to deploy a virtual environment using VMware nested virtualization in your lab.

Uploading the ISO image

I need to upload the ESXi 7.0.3 installation ISO image to the datastore on ESXi 6.7, which is my physical ESXi host.

1. Open VMware vSphere client and go to Datastores.
2. Select a datastore to which you will upload the file. I select datastore 11 located on the ESXi 6.7 host (192.168.101.205).
3. Select the Files tab.
4. Click Upload Files.

5. Select the ESXi 7.0.3 installation ISO image file on your computer and hit Open.

Wait until the file is uploaded to the datastore.

Creating a VMware hypervisor VM

Now create a new VM on which you will install ESXI 7.0.3.

1. Go to Hosts and Clusters.
2. Select the needed ESXi host (192.168.101.205 in this case).
3. Click Actions > New Virtual Machine.

A New Virtual Machine wizard opens. Follow the steps to create the machine.

1) Select a creation type

1. Click Create a new virtual machine.
2. Click Next at each step of the wizard to continue.

2) Select a name and folder

Then specify the following parameters:

• Virtual machine name: ESXi7-VM
• VM location: Datacenter1

These parameters may be different in your infrastructure.

3) Select a compute resource

Select your physical ESXi host on which you want to deploy an ESXi 7 VM. I select 192.168.101.205.

4) Select storage

Select a datastore on which you will locate the VM files. Make sure that there is enough free space. I select datastore42 in this example.

5) Select compatibility

The compatibility selected defines the VM hardware version. The higher version provides more features, but the lower version allows you to run a VM on older versions of VMware ESXi and VMware Workstation. You may need to select a lower compatibility version if you plan to migrate a VM to hosts running an older hypervisor version. I am creating a VM on ESX 6.7 and selecting ESXi 6.5 and later for compatibility (VM version 13). Remember, nested virtualization is supported starting from VM hardware version 9.

6) Select a Guest OS

Select the following parameters.

• Guest OS Family: Other
• Guest OS version: VMware ESXi 6.5 or later

Once you select the version of ESXi, the warning message appears: This operating system is not supported. This is a reminder from VMware that you cannot request technical support when using ESXi on a VM if issues occur. Ignore this message and continue the installation.

7) Customize hardware

This step is important for the proper configuration of ESXi nested virtualization. You need to configure the virtual hardware for the new VM.

CPU. Select at least 2 CPUs. You can select multiple cores per CPU. Select the checkbox Expose hardware assisted virtualization to the guest OS to enable VMware nested virtualization for this VM.

Note: If you don’t expose hardware-assisted virtualization, you will get the following error when installing ESXi on a VM. <Hardware virtualization warning: Hardware virtualization is not a feature of the CPU, or is not enabled in the BIOS>. If you see this error, you will not be able to start a nested VM on a virtual ESXi host.

Memory. Select 8 GB of memory or more. Keep in mind that ESXi itself requires 4 GB of memory to run. You need to add more than 4 GB of memory to run a nested VM.

Hard Disk. Set the virtual hard disk options. Set the new virtual hard disk size, which must meet ESXi 7.0.3 storage requirements (at least 32 GB of disk space). Here, I create a 40-GB virtual disk for a VM to install ESXi 7.0.3.

Disk Provisioning. Select Thin provision. This option allows you to save storage space on the datastore on which the VM is located.

1. Network. Select the needed VM network. I use VMXNET3 as the network adapter for optimal performance.
2. New CD/DVD Drive. Expand the New CD/DVD Drive section.
3. Select Datastore ISO file in the drop-down menu.
4. In the CD/DVD Media line, click Browse and select the ISO file in the datastore.

5. Select the ESXi 7.0.3 installation ISO image file, which you previously uploaded, in the datastore.

8) Ready to complete

Check VM configuration summary and if everything is correct, hit Finish.

Starting the ESXi VM

Once the VM is created, start this VM. I select the ESXi7-VM in VMware vSphere Client, right-click the VM, and in the Actions menu, click Power > Power On.

When the VM has started, open the VM console in your web browser or in the VMware client (such as VMware Workstation or VMware Remote Console application) and install ESXi 7.0.3 as you usually install ESXi. Set the IP address and hostname on ESXi (192.168.101.131 is the IP address used by the ESXi7-VM in our walkthrough).

After you finish installing ESXi 7.0.3 on the VM, you can click the VM name in VMware vSphere Client and view the VM info. As you can see on the screenshot below, VMware Tools was automatically installed on the ESXi 7.0.3 VM.

Configuring a virtual switch

Before going to create a nested VM on the outer ESXi VM (192.168.101.131), we need to configure a virtual switch on the physical ESXi host (192.168.101.205). As explained above in the theoretical part of this blog post, Promiscuous mode and Forged transmits must be enabled on the virtual switch of the physical ESXi host to allow network connections for nested VMs outside the virtual ESXi host.

1. Go to Hosts and Clusters in VMware vSphere Client
2. Right-click the ESXi host on which you have installed the ESXi VM and click Settings to open ESXi configuration. The Configure tab opens.
3. Click Virtual switches in the Network category.
4. Then select the needed virtual switch (vSwitch0 in our case) and hit Edit.

5. Configure the security policies for the virtual switch. Select Accept for Promiscuous mode, MAC address changes, and Forged transmits. Hit OK to save the settings and close the window.

The virtual switch is now configured for ESXi nested virtualization.

Creating a datastore on an ESXi VM

When I created the ESXi 7.0.3 VM, I created one 40-GB virtual disk for ESX installation. Due to the new partition layout in ESXi 7, the VMFSL partition is created and there is no space for the VMFS partition that must be used to create a datastore on small disks (less than 128 GB). For this reason, I cannot store VMs on the virtual disk used to install ESXi 7.0.3 on this VM, and I need to create a second virtual disk for my ESX7 VM.

The VMFSL partition is used to store ESXi OS Data, the consolidation of the coredump, tools, and scratch. A VMFS datastore is created in the additional partition on the system disk if the disk size is larger than 128 GB.

Let’s create the second virtual disk for the ESXi7-VM. I will use this virtual disk for the VMFS datastore on the virtual ESXi host. To do this, shut down the ESXi7-VM before adding the virtual disk.

1. Go to VMware vSphere Client and select the ESXi7-VM.
2. Right-click the VM name and in the menu that opens, click Edit Settings.

3.  In the Edit Settings window, click Add new device and select Hard disk. Then set the new hard disk options.

I select the following configuration:

• Disk size: 30 GB
• Type: Thin provision

Hit OK to save the VM configuration and close the window.

Once the second virtual disk is created on ESXi7-VM, start the VM. Then connect to the ESXi 7.0.3 running on the VM (192.168.101.131) in a web browser and log in to VMware Host Client.

Go to Storage in the Navigator pane and click New datastore.

1) Datastore name

Select Create a new VMFS datastore and provide a name for the new datastore, for example, ds-nested01. Click Next at each step to continue.

2) Select partitioning options

As there is only one free virtual disk, this 30-GB disk is selected. Select Use full disk and VMFS 6 in the appropriate drop-down lists.

3) Ready to complete

Check the datastore configuration summary and hit Finish.

A datastore is created. You can use this datastore to store a nested VM (for example, a VM running Windows) in a virtual VMware hypervisor, that is, ESXi 7.0.3 in my case.

Uploading the ISO image for the nested VM

We have almost finished the preparations for creating a nested VM on a virtual ESXi host to explain how to configure VMware nested virtualization in practice. We are going to install Windows XP on the nested VM because this operating system is lightweight, and performance should be sufficient in a nested environment. Keep in mind the security aspects and use the latest version of Windows. You can install any other supported guest operating system (a guest OS for which VMware Tools are provided). To install a guest Windows OS, we need to use the Windows installation ISO image file. The idea is to boot the OS installer and install a guest OS on the nested VM.

There are two main routes for uploading the Windows installation ISO image.

1. Upload the ISO image to the datastore on the virtual ESXi host (the 30-GB datastore on the ESXi 7.0.3 VM which IP is 192.168.101.131 in my case). Select to use the ISO image from the datastore in the CD/DVD drive settings of the nested Windows VM.
2. Upload the ISO image to the datastore of a physical ESXi host. Mount this ISO image to the CD/DVD drive of the ESXi7 VM in the VM settings of ESXi7-VM. Select Host Device in the VM settings of the nested Windows XP VM when creating a new nested VM. The Host Device option for a CD/DVD drive used by a VM enables the pass-through mode and connects a VM to a physical CD/DVD drive of the physical ESXi host. As our ESXi host 7.0.3 is virtual in a case of VMware nested virtualization, the Host device option connects the virtual CD-DVD drive of the ESXi7-VM to the virtual CD/DVD drive of a nested VM.

The advantage of the second method is that you don’t need to use the datastore space on the virtual ESXi VM, you can store all ISO images in a single location, and overall performance should be slightly better. I use the second method in this walkthrough to explain the configuration of ESXi nested virtualization.

Upload the Windows XP installation ISO image file to the datastore (datastore11 in our case) just like you did for the ESXi 7.0.3 ISO image file in vSphere Client.

1. Go to Datastores.
2. Select the needed datastore.
3. Click the Files tab.
4. Click Upload files.
5. Select the needed ISO file, hit Open, and wait until the ISO file is uploaded to the datastore.

Edit settings of the ESXi7-VM (we use VMware vSphere Client for this purpose).

1. Select CD/DVD drive 1.
2. Select the Datastore ISO file option.
3. Click Browse in the CD/DVD Media section and select the Windows ISO image located on the datastore.
4. Make sure that these checkboxes are selected: Connected, Connect At Power On.

Creating a nested VM

Log in to the web interface of VMware Host Client for a virtual ESXi 7 host (192.168.101.131 in my case).
Go to Virtual Machines in the Navigator pane and click Create / Register VM.

A new virtual machine wizard opens in VMware Host Client.

1) Select creation type

Select Create a new virtual machine. Hit Next at each step of the wizard to continue.

2) Select a name and guest OS

Specify a unique name for a VM and OS. I select the following parameters.

• A VM name: WindowsXP
• Compatibility: ESXi 7.0 U2 virtual machine
• Guest OS family: Windows
• Guest OS version: Microsoft Windows XP Professional (32-bit)

If you want to install another guest OS on a VM, select the appropriate OS in the options.

3) Select storage

Select the datastore that you have created earlier on the virtual ESXi host. I select ds-nested01 to store VM files.

4) Customize settings

Configure the VM hardware and additional VM options.

I select:

• CPU: 1 CPU
• Memory: 512 MB of RAM (select up to 3.3 GB of RAM for Windows XP 32 bit)
• Hard disk: 8 GB
• Disk Provisioning: Thin provisioned

CD/DVD Drive 1: Host device

You can leave the default values for the other settings.

5) Ready to Complete

Check the configuration of the nested VM, and if everything is correct, hit Finish.

Power on the nested VM on ESXi 7.0.3. The VM should boot from the ISO image mounted to the virtual ESXi 7.0.3 VM. Install a guest operating system on the nested VM. You can click the VM screen preview to open a VM console and interact with the guest OS in the web interface.

Once you have installed a guest OS on the nested VM, install VMware Tools. If your ESXi distribution doesn’t contain ISO images with VMware Tools, you can download VMware Tools installation images from the official VMware website.

After installing VMware Tools, your network for a nested VM should work properly. Try to ping a nested VM from your physical ESXi host and vice versa.

Cloning ESXi VMs

After configuring ESXi on one VM you may need to deploy more ESXi VMs on your physical ESXi server. You can also deploy a nested vCenter Server Appliance and a VM with shared storage (for example, FreeNAS) to test diverse vSphere features including clustering features.

You can always install ESXi on a new VM from scratch, but you may need to do this operation routinely if you need to deploy multiple identical ESXi VMs. That’s why a better solution is to configure one ESXi VM and clone this VM. However, you need to take some additional steps to create properly working VM clones.

1. Change the VMkernel MAC address by running the command in ESXCLI.
   esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
2. Delete the UUID record from esx.conf because the UUID values must be unique on hosts.
   The system UUID record is stored in /etc/vmware/esx.conf
3. Open this file and delete the line starting with /system/uuid
4. Shut down the original ESXi VM. Now you can clone the ESXi VM. A new UUID is generated when you power on a VM (including VM clones) next time. Once you have powered on ESXi VM clones, change network settings on them (IP addresses, hostnames, etc.).

Now you know how to use VMware nested virtualization in practice and how to deploy a virtual environment with ESXi nested virtualization. You can use the same idea to deploy nested environments with other hypervisors.