For a test lab with two servers we currently have running, I requested two test LUNs. One LUN coming from an IBM DS and the other from an HP EVA. The SAN team gave me the LUNs and I could see them on both servers.

The output from esxcfg-mpath -l, a new command in ESX 3 to list paths, shows the following for a server.

Disk vmhba0:0:0 /dev/cciss/c0d0 (69459MB) has 1 paths and policy of Fixed
 Local 4:1.0 vmhba0:0:0 On active preferred

Disk vmhba0:1:0 /dev/cciss/c0d1 (140006MB) has 1 paths and policy of Fixed
 Local 4:1.0 vmhba0:1:0 On active preferred

RAID Controller (SCSI-3) vmhba1:0:0  (0MB) has 2 paths and policy of Most Recently Used
 FC 12:1.0 10000000c9576168<->50001fe1500157fd vmhba1:0:0 On active preferred
 FC 12:1.0 10000000c9576168<->50001fe1500157f9 vmhba1:1:0 Standby
Disk vmhba1:0:1 /dev/sda (512000MB) has 2 paths and policy of Most Recently Used
 FC 12:1.0 10000000c9576168<->50001fe1500157fd vmhba1:0:1 Standby  preferred
 FC 12:1.0 10000000c9576168<->50001fe1500157f9 vmhba1:1:1 On active
Disk vmhba1:0:2 /dev/sdb (512000MB) has 2 paths and policy of Most Recently Used
 FC 12:1.0 10000000c9576168<->50001fe1500157fd vmhba1:0:2 Standby  preferred
 FC 12:1.0 10000000c9576168<->50001fe1500157f9 vmhba1:1:2 On active

No problem here so I created two VMFS volumes: ESXSAN01 and ESXSAN02.

When I refreshed the volumes on the second server, the VMFS volumes did not show. The vmkernel log file showed the following:

Sep 29 10:11:44 xs401551 vmkernel: 7:01:44:48.051 cpu3:1038)LVM: 5670: Device vmhba1:0:1:1 is a snapshot:
Sep 29 10:11:44 xs401551 vmkernel: 7:01:44:48.051 cpu3:1038)LVM: 5676: disk ID: <type 2, len 22, lun 1, devType 0, scsi 3, h(id) 14189849040130868952>
Sep 29 10:11:44 xs401551 vmkernel: 7:01:44:48.051 cpu3:1038)LVM: 5678: m/d disk ID: <type 2, len 22, lun 2, devType 0, scsi 3, h(id) 14189849040130868952>  

The second server refuses to mount the VMFS volume because it is detected as a snapshot. A volume is detected as a snapshot when the LUN IDs are different among servers. In this case, LUN1 on the first server was LUN2 on the second.

This is actually some new behaviour introduced with ESX 3.0. In ESX 2.x, having different LUN IDs was not an issue. To revert back to ESX 2.x behaviour, set LVM.DisallowSnapshotLUN=0. The option is set with the VI Client, in the Configuration page of a server, Advanced Settings.

From the moment you set the above option and refresh your storage, you will see the VMFS volumes on the second server.

This new behaviour was introduced because it is now possible to present a SAN snapshot of a LUN back to the same server. The new resignature feature will write a new signature to the snapshot LUN and give it a new volume name. This is handy in recovery scenarios but you have to set another advanced option to enable the feature. See the SAN guide for more info.

As a best practice, you should actually present the same LUN IDs to each server. How this is done depends entirely on your SAN and its configuration.

NOTE: when you revert to ESX 2.x behaviour using the option described above, do not present a snapshot LUN to ESX. Unless you want to test your DR procedures of course. :-)

Introduction

esXpress (www.esxpress.com) is software to backup virtual machines running on ESX. The current version (2.x) can be downloaded for free from their website. With the free version, you can take full backups of your running virtual machines. When you buy an additional license, you can also take differential backups and encryption is also included.

Version 2.x does not run on ESX 3.0. This is the case for many ESX 2.x versions of add-ons (e.g. Leostream P>V, vizioncore esxRanger, ...). For version 3.0, you will need esXpress 3.0. That version is currently in beta but you can register to try it out yourself.

This post discusses the beta version of esXpress 3.0. Not all features have been enabled yet but in its final form, it promises to be a very complete package with automated backups, many backup transports, easy restores, a web interface and more. Because it is a beta, this post does not discuss pros and cons but will show you how the product works.

How it works

VMware has recently published a whitepaper that talks about third-party software in the service console. Running applications with heavy load in the service console is not recommended. The esXpress developers followed VMware's advice and changed the architecture of esXpress to comply with VMware's recommendations.

Instead of performing the backup from within the service console, the backup is actually performed by a helper virtual machine called a Virtual Backup Appliance or VBA. When you backup a virtual machine, a VBA is started that backs up the disks of that virtual machine. From within the VBA, the backup is encrypted, compressed and sent to its backup destination. Multiple VBAs can run concurrently.

Of course, you still need some software in the console to actually initiate backup jobs and provide the user with an interface to backup and restore. However, that piece of software is very lightweight and has virtually no impact on the resources of the service console.

Let's get started showing you how the product is installed and how you perform a basic backup.

Installation

Installation is very simple. You need two files. At the time of this writing, the files were:

• esxpress-3-0Beta-8-esx-i386.rpm
• esxpressVBA-3.0Beta-1.esx.i386.rpm

Transfer those files to the ESX server, logon as root with a ssh client and install with “rpm –i”. First install esxpressVBA and then the other one.

Configuration

Configuration is straightforward using the phd command. When you run the phd command, you will see the text-based menu interface:

You first need to go into the Configuration Options and configure esXpress. As a basic minimum, you need to:

• Configure FTP settings
• Configure Virtual Helpers

FTP is the only transport supported in this beta. In subsequent betas, many more transports will be added like NFS, SMB, etc...

To configure FTP settings, in the Configuration Options menu, select Configure FTP.

You will see the FTP configuration screen. Just type the host name or IP address of an FTP server, the user name and password and the port and folder. The FTP server can of course be running on anything (e.g. Windows). Just make sure that writes (ftp put) are enabled.

Note that you can specify multiple FTP servers. If the first one is unavailable, the second one will be tried, and so on... Note that you do not need to enable ftpClient within the ESX firewall. This is because it is the helper VM (VBA) that is doing the FTPing, and not the console.

Next, configure the virtual helpers. From the Configuration Options, select Configure Virtual Helpers.

As stated earlier, the helpers are virtual machines. You need to tell esXpress the network that you want them to connect to. The helpers can get their IP settings using DHCP. Be sure to specify the HELPER_DEFAULT_VMFS because that is where the helper virtual machine will be created. The helpers do not need network connectivity to the virtual machines that you backup.

A helper virtual machine will show up in the ESX or VirtualCenter inventory.

Performing a backup

Manually performing a backup can be done from the menu. Start phd and select Backup ONE VM Now.

A list of VMs will be shown. Select a VM from the list.

Select OK. EsXpress will perform an INDEX backup (only the changes). If there is no previous backup, a FULL backup will be taken.

You are back in the main menu. Use the Tail Backup Log option to follow the process.

In the process, the VBA helper will be started and perform the backup. Below is a screenshot of the console of the helper.

As you can see in the above screenshot, the backup is taking place. Reading, compressing and copying with FTP is done at the same time to improve performance.

In the backup log, you can also follow the progress:

On your FTP server, the following directories are created:

The two folders for newtech-dc are from previous backups. The folder for newtech-SQL is the current backup. Inside the newtech-SQL folder, you will find:

When the backup is complete, the TMP extension will be removed.

Note that when your backup is complete, a dummy DELTA backup is taken of the same VM so, on your FTP server, you will end up with:

Restoring

Because automatic, menu based restores were not in the beta yet, I do not discuss them. It is possible to restore the disks manually.

Conclusion

esXpress 3.0 promises to be a welcome addition to the list of products that can do snapshot backups of virtual machines on VI3. It promises to be a very complete product and is well thought out. I especially like the use of helper VMs and their delta technology. Check it out for yourself and let me know what you think!

When you are doing a virtualization project, the hardware platform is always a tough choice. The customer I am working with at the moment has standardized on servers from HP and typically uses the DL380 and the DL580. For their virtualization project, they would like to use the DL580 because it is a 4-way system.

The DL580 has recently gone from G3 (generation 3) to G4. The G3 will be available to order until end 2006. From that moment, HP will just sell what they have left in stock.

The successor to the DL580 G3, the G4, can be equipped with an Intel Xeon 70xx series CPU or an Intel Xeon 71xx series. They are all 64-bit CPUs and have 2 cores. The top processor is the 7140M which runs at 3.4 Ghz and has a whopping 16MB L3 cache.

The problem with the DL580 G4 is that it is still based on Intel's NetBurst architecture which will be phased out. The shared front-side bus, slower access to memory, the power requirements, heat generation, and so forth make this architecture less than compelling. Especially when you compare this to the Opteron platform.

My choice for a good 4-way platform has always been Opteron. In HP's range, that is the DL585 G1. That one will soon be replaced by the G2 which introduces Opteron Rev. F, DDR2 memory and other improvements. This platform is more optimal because of its NUMA architecture, on-chip memory controllers, HyperTransport, etc... ESX has optimizations for NUMA which makes it even more interesting.

When you compare some benchmarks of Opteron versus Xeon on ESX 3.0, you can really see the advantage of Opteron. For example, check out these benchmarks. Quite a performance lead to say the least.

To make the choice harder though, Intel has published some benchmarks comparing their 7140M CPU with AMD's Opteron. You can check them out here. When you look at those benchmarks, the Intel 7140M has a lead over the AMD Opteron 885 (2.6 Ghz, dual core). It remains to be seen if Intel chose those benchmarks because they benefit the most from the 16MB L3 cache and what the 16MB L3 cache does for more heterogeneous workloads. Will there always be a performance lead (good for virtualization) or only for some specialized workloads? Difficult to say right now because there is almost no information about that.

Anyway, for 4-way systems, I would still go with Opteron and in this case the DL585 because it is a proven and stable platform. Especially with Rev. F (includes hardware virtualization support), DDR2 and later quad-core, you can't go wrong with that. Just my 2 cents!

From virtualization.info, I learnt that VMware has released a technology preview of a platform that allows you to run a paravirtualized OS. VMware uses its own VMI specification so it is not based on Xen or other initiatives.

Paravirtualization requires some changes to the guest OS in order to avoid overhead from virtualization. In general, a paravirtualized guest should see improvements in performance.

Because the technology preview is based on VMware Player for Linux, you should not see I/O performance improvements. You should see CPU improvements so CPU intensive applications should get a boost.

It will become more interesting when ESX 3.0 is able to run a paravirtualized OS because we should then see improvements from the I/O side as well. All in all, it looks very promising. Now if only the industry could agree on a common paravirtualization standard!

Get the technology preview here.