Introduction

The purpose of this article is to describe some of the features of ESX 3.0 and VirtualCenter 2.0. Some basic knowledge of ESX 2.x and VirtualCenter 1.x is required to fully understand all topics described below. It is important to know that ESX 3.0 can be used on its own, without VirtualCenter. However, this is rarely done in reality. Most if not all enterprise customers use ESX in combination with VirtualCenter. VirtualCenter adds a management layer as well as extra features to ESX implementations.

Virtual Machine Features

ESX 3.0 makes it possible to put even more demanding workloads in a virtual machine (VM). A VM now supports up to 16GB of memory (3.6GB in ESX 2.x) and supports 4 virtual CPUs (2 in ESX 2.x).

An interesting feature is the ability to hot-add virtual disks. While a VM is running, disks can be added to the configuration which will then be picked up by the guest OS. In Windows Server 2003, you need to issue the Rescan Disks command to find these new disks. You need to make sure that the VM uses the correct virtual hardware for this feature to work properly. VMs that you migrated from GSX or ESX 2.x that use older virtual hardware will not allow you to hot-add virtual disks.

Like always, virtual machines need to use the latest version of the VMWare Tools to work properly. Some new features such as VCB (VMWare Consolidated Backup) require the new VMWare Tools package. In case of VCB, the new VMWare Tools will quiesce the virtual machine to ensure that all files on the file system are in a correct state before the VCB snapshot is taken. More information about VCB further in this document.

VirtualCenter Security

In ESX 2.x you can grant permissions in two ways. First, you can create accounts on the ESX server itself. This is just like creating accounts on Linux. You grant permissions to these accounts the Linux way. The second method requires VirtualCenter. VirtualCenter 1.x defines some roles and as an administrator you add user accounts to these roles. The user accounts can come from Active Directory. VirtualCenter 1.x roles are fixed.

In ESX 3.0, you can still create users and grant permissions on the ESX box directly. VirtualCenter 2.0 comes with an enhanced and more granular permissions model. In VirtualCenter 2.0, you can create custom roles and assign these roles granular permissions. For example: you can create a role called DepartmentX Operator and grant the right to power on/off specific virtual machines.

Management

In ESX 2.x, you normally use the MUI (Management User Interface) to make changes to the ESX configuration. The MUI is a web-based interface. There is no Win32 GUI available to change the configuration of ESX 2.x.

In ESX 3.0, the MUI is gone. You will use the VI Client (Virtual Infrastucture Client) to configure an ESX 3.0 box. Interestingly, the same VI Client is used to access VirtualCenter as well. This simplifies the amount of tools that you need to manage an ESX environment.

A web client (VI Web Client) is still available but is used to manage virtual machines, not the configuration of the ESX server itself.

Storage Options

In ESX 2.x, you basically have two options: local storage or SAN storage. ESX servers in a farm that need VMotion capability require the use of a SAN. In ESX 3.0, two extra storage options are added to the mix: iSCSI and NAS. This is a welcome addition for customers that might not want to invest in a SAN solution because of the cost and complexity.

iSCSI

iSCSI allows you to use storage devices over an IP network. An iSCSI initiator connects to an iSCSI target and presents disks (block devices) to the operating system. ESX 3.0 comes with a software initiator but also supports hardware initiators (e.g. QLogic QLA4010).

The software initiator is part of the vmkernel. You can compare it to Microsoft's iSCSI software initiator in that it is just some extra software in the OS that will connect to the iSCSI target and present the block devices. Naturally, a software initiator has less performance than a hardware initiator.

A hardware initiator is just an extra controller like any SCSI or FC (Fiber Channel) controller you add to ESX. ESX 3.0 comes with a limited list of supported controllers so check the list. The iSCSI controller contains a TCP/IP offload engine (or TOE) that takes the load of the operating system. Needless to say that a hardware initiator will yield the best performance.

iSCSI storage supports booting (hardware initiator only), VMFS/RDMs and VMotion.

NAS

ESX 3.0 supports NAS solutions as well but supports NFS only. You cannot use Windows shares (SMB). NFS volumes can be used for virtual disk storage (VMDK), ISOs and supports VMotion as well.

Distributed Resource Scheduler (DRS)

DRS is an optional feature of VirtualCenter 2.0 that is separately licensed. Basically, it allows for resource balancing across servers in a DRS cluster. Before going into what DRS actually does, we need to talk about the differences in resource management between ESX 2.x and ESX 3.0. I will only look at memory and CPU resources because only these two resources are used by DRS for load balancing purposes. I/O bandwidth is not taken into consideration.

ESX 2.x has the concept of shares, min and max values. A minimum setting for CPU or memory sets a guaranteed floor, for instance 20% of total CPU resources. A maximum setting is an upper limit like 80% of CPU resources. Shares come into play when there is resource contention. When VM A has 2000 shares and VM B has 1000 shares, A will get 66,33 % of what's available and B 33,33% (roughly).

ESX 3.0 also has these concepts although the names are a bit different. Minimum is now called reservation and maximum is called limit. ESX 3.0 introduces the concept of compute resources. A compute resource is an aggregation of resources (only CPU and memory) from one or more hosts. For example, a single host might have four 3Ghz CPU's and 16GB of memory totaling 12Ghz and 16GB. But when you have three hosts like that in a cluster, you have 36Ghz of CPU power and 48GB of memory at your disposal.

These compute resources can then be carved up into resource pools. Resource pools are defined and managed in the VI client. You just assign memory and CPU resources to the resource pool and you give the resource pool reservation, limit and share settings. You can then start creating VMs in the resource pools and even grant permissions on the resource pool level. You could give a group of people the right to create virtual machines in a resource pool. Of course, because of admission control, ESX will determine if it is allowed to power up a virtual machine based on the available resources in the pool. Note that virtual machines in a resource pool still have their own reservation, limit and shares settings.

It is clear from the short explanation above that resource scheduling in ESX 3.0 is a bit more complex than before. Hands-on experience with the product will reveal how easy it is to work with.

You should be aware of the fact that resource pools are a feature of ESX, not of DRS. You can create resource pools on a standalone ESX 3.0 box. Then what exactly does DRS do?

Based on analysis of available resources (CPU and memory), DRS will move (or recommend to move) virtual machines to hosts where the requested resources are available. DRS can be set to one of three modes: manual, automated and partially automated. In manual mode, DRS will provide recommendations to the administrator who can then take action. In automated mode, DRS automatically moves virtual machines when needed. In partially automated mode, DRS will automatically place the virtual machine on a host that has the required resources (initial placement) but will not move the machine afterwards. You can set placement constraints like "do not put VM A and VM B on the same host".

Distributed Availability Services (DAS)

Just like DRS, DAS is an additional feature of VirtualCenter 2.0 that is separately licensed. Basically, when an ESX node in a DAS cluster goes down, the virtual machines of that node are started on surviving nodes. Together with DRS, a decision can be made about where to place the virtual machines depending on the available resources in the cluster.

Virtual machines are just started on another host, so it is the same as starting a physical machine after a crash or hardware failure.

Please not that there are some other solutions for clustering ESX. For example, Veritas has a product out there that can do it now in ESX 2.x. More interestingly, a freeware clustering solution based on HA-Linux (heartbeat) is being developed. Check out the website for this project. 

VMWare Consolidated Backup (VCB)

Today, when you need to backup virtual machines, you typically place backup agents in all virtual machines on the ESX box. Needless to say, that puts extra load on the entire server, potentially impacting other virtual machines. Another option is snapshotting the virtual machine's disks and saving the disk files somewhere else.

VCB is a solution that allows you to take file level backups of virtual machine disks without requiring backup agents inside the virtual machine. How is that accomplished?

VCB requires a Windows Server 2003 machine that is called the backup proxy. This backup proxy has your regular backup software installed (Veritas, Legato, ...) and VMWare's VCB software. An integration module provides the link between the backup software and VCB. The backup proxy needs to see the same LUNs on the SAN as ESX. Those LUNs contain the vmdk files, the virtual machine's virtual disks. As you can see, VCB requires you to use a SAN solution.

The question here is: "How do you backup the files from inside the virtual machine's disks (vmdk files) that reside on a VMFS partition on a particular LUN?". The trick is that the VCB software on the proxy mounts a vmdk file on a junction (mount point) in Windows 2003.

The process in a nutshell:

1.   Use the backup software to perform a backup of the junction point. For example: D:\MOUNT\vmname.domain.com\letters\c.

2.   A pre-backup script will run. This script is part of the integration module for your backup software.

3.   The virtual machine that you are backing up will be quiesced. The new VMWare Tools will handle the quiescing. This basically means that we get a consistent file system because outstanding writes are written and new writes held off.

4.   The VMWare Tools will run a pre-freeze script. You can use that script to stop services etc...

5.   When the VM is quiesced, a snapshot can be created by adding a REDO file. After that, a post-thaw script is run where you can start services etc...

6.   The virtual disk file (vmdk) now contains a file system that is file consistent.

7.   VCB now mounts the vmdk file on the junction point in Windows 2003.

8.   The backup software backs up the files.

9.   Cleanup actions occur such as running post-backup scripts, removing the REDO file, etc...

That’s it for now, expect more posts about ESX 3.0 soon.