This is the second post in a series about XenSource XenEnterprise 3.1. The first post was about installation and basic management. This post is about virtual machine management. I will limit myself to Windows virtual machines because that is basically what I am interested in the most. I tinker a bit with Linux once in a while but our customers mainly use Windows.

Windows XenVMs

XenEnterprise 3.1 supports Windows Server 2003 and Windows XP. Version 3.2 will support Windows 2000 Server SP4.  Version 3.1 also has some limitations when compared to the upcoming version 3.2:

• Only 1 vCPU (4 in version 3.2)
• No suspend/resume (available in 3.2)
• 2GB of RAM (>2GB in 3.2)
• Unsigned paravirtualized drivers (should be signed in 3.2 with WHQL certification)

Version 3.2 has other improvements like better network performance and VLAN trunk support.

Installing a Windows Server 2003 XenVM

A new Windows virtual machine is installed from the XenServer management console by right clicking the XenServer and selecting Install XenVM. The virtual machine options can then be specified (click to enlarge):

I created a virtual machine with the maximum amount of memory (2047MB) and three virtual disks (the maximum). Windows Server 2003 can be installed from a cd image (iso) that is uploaded to /opt/xensource/packages/iso. I guess you can also use /var/opt/xen/iso_import. The iso can be uploaded with any scp client such as WinSCP or Bitvise Tunnelier. If you do not want to work with an iso, you can use the physical cd-rom of the XenServer host.

You should note that there is not a lot of free space for iso images in /. The XenSource website has more information about the creation of an iso repository. It is done by creating a new logical volume in the XenStorage volume group. There are other ways to do it as well.

When the Install button is clicked, the XenVM is started and you can take remote control from the management console (click image to enlarge):

The disks are presented as regular IDE devices. During installation, you do not need to load any device drivers (F6). Installation is fast as expected.

Let's take a look at the finished WIndows Server 2003 installation without the paravirtualized drivers. Here is a view of device manager (click to enlarge):

This Windows XenVM works without any additional drivers but that is not recommended. The paravirtualized drivers need to be installed from the supplied ISO image. To install the drivers, go the the Overview tab of the virtual machine and connect the xswindrivers.iso. When you connect the iso, autorun will be triggered to start the installation. During installation of the drivers, some warnings will popup because the drivers have not been signed. After installation of the drivers and a reboot, the Found New Hardware Wizard will appear to install the drivers for the network card. After that, device manager looks like:

After the drivers are installed, the XenEnterprise management console knows the ip address of the XenVM and will enable the remote desktop button. That button, when clicked, will launch mstsc.exe and connect to the XenVM with RDP. Note that you need to enable remote desktop in the vm. It is not like Windows Server Virtualization (Longhorn) where RDP will be used as the remote control protocol even during the boot up phase or during installation from cd.

Virtual Machine Storage

VMware ESX works with two types of storage: vmdk files on a vmfs volume or rdm (raw device mapping). When you use vmdk files on vmfs you simply encapsulate the disks of your virtual machine in a file on the host server. When you use rdm, you give a virtual machine more direct access to a SAN LUN for applications that require it. With XenEnterprise, don't expect your virtual machine disks to appear as files on the filesystem of the Xen host. Although Xen supports this, XenEnterprise uses LVM for virtual machine storage and each virtual machine's disk is a logical volume in the volume group that was created during installation. I believe that XenSource is planning to add support for other storage types, including the use of vhd files.

How does it actually work? Well, remember my server had one RAID5 volume with four local SAS disks (about 220GB in total). During installation, XenEnterprise creates three partitions: /dev/cciss/c0d0p1, /dev/cciss/c0d0p2 and /dev/cciss/c0d0p3 (it's an HP box). The first two are quite small and used by the system. The third is used to host an lvm 2 volume group. With pvdisplay, the physical volumes are shown and you can see to which volume group they belong. In my case /dev/cciss/c0d0p3 is used for volume group VG_XenStorage-1f64f92b-56f8-4e7f-adbb-812d62abe84f. To get more info about the volume group, use the vgdisplay command. To see the logical volumes created in the volume group (something like partitions on a regular disk), use lvdisplay. In my case, I see seven logical volumes. One is used by the system and the others correspond to the disks of the XenVMs.

A common requirement is the ability to resize a virtual machine's disks. With XenEnterprise that is easy. When a XenVM is shutdown, the new size of the disk can be set with the management console. No need to go to a command line!

Other possibilities

There are some other features that you might find useful:

• XenVM export and import to and from the system that runs the management console.
• XenVM cloning: simpler than VMware VirtualCenter but it does the job
• Mounting of virtual machine disks on the XenServer (command line).

Conclusion

Virtual machine management is pretty easy and straightforward with the management console. Installing a Windows XenVM is like installing a physical machine but you can also use cloning in combination with sysprep. XenEnterprise 3.1 has some serious limitations (vCPUs, memory, disks) for Windows XenVMs but this will improve with version 3.2 and higher.

My colleague Vincent has a post about the next version of esxReplicator. We are working at the same customer where we have implemented VI3 in their datacenters. At the moment, we are evaluating the use of ESX in remote sites. The infrastructure at those sites is rather basic. There is no SAN, no iSCSI so just local storage. To have some level of high availability, products such as vizioncore esxReplicator and Double-Take for VMware Infrastructure can be a solution.

I already posted a short review of the Double-Take product. Just like esxReplicator, it is easy to install and uses snapshots to perform replication. But I must say that esxReplicator has better scheduling options, a better GUI and reports its progress better. It can also send reports by e-mail so that you have some idea about what is going on.

As a follow-up to my previous post, I decided to install XenEnterprise and share my impressions with you. It is only right that when you feel you cannot recommend a product to a customer, you have some hard facts and real experience to back that up. So I took a DL380G5 with 8GB of RAM and 1 dual-core CPU and decided to give the installation a go. I installed XenEnterprise 3.1 because that is the latest release. Version 3.2 is in the works and has some features that you might want to wait for. Version 3.2 supports things like SMP for Windows guests (up to 4 vCPUs) and VLAN trunking support for virtual bridges.

Installation

Installation is very simple and straightforward. XenEnterprise 3.1 installs perfectly on the HP DL380G5. One thing to note is that you are asked to configure all the detected network interfaces. In my case, I was asked to configure both eth0 and eth1 (the 2 built-in connections). With ESX, you are asked to configure just one interface, the management interface for the console operating system and you can select the interface you want to use. I my test setup, I only configured eth0 and gave it an IP address on my private network (10.0.0.10). At the end of setup, you are asked for the date and time if you choose not to configure NTP.

I had four SAS disks in the DL380 and I configured them as one RAID5 volume. During setup, the volume is detected and partitioned for you. It is not yet clear to me how storage works with Xen and XenEnterprise because it is not very well explained in the documentation. With ESX, you just partition the system like any Linux configuration and then create some VMFS partitions for virtual machine storage. That concept is very clear and simple and well explained in the VMware documentation. I guess I need some more reading and experience to see how this all works with Xen.

After installation, you get a message to install the management software on a Linux or Windows system and connect to this XenEnterprise server.

Management Software

The installation cd contains a executable to install the management software on Windows. I installed the management software on a Windows XP SP2 machine and fired it up. The first time you start, you are asked to provide a password. That password is used to save host login information to disk in encrypted form. You are asked to provide this password when you start the management console:

After starting the console for the first time, you need to add the XenServer host you want to manage with the File / Add XenServer Host menu option. You then provide the host name or IP address and the password of root (set during installation). After the host is added, you can start to manage it. I created a Windows Server 2003 XenVM and a Debian XenVM. A later post will provide some details about working with XenVMs. The management console with these two virtual machines looks like this (click image to enlarge):

The two XenVMs are listed under the XenServer host I imaginatively called "xenhost". You will notice that both XenVMs have a small Xen logo in front of their status. That indicates that the XenVMs have been "enlightened" with paravirtualized drivers. Although not strictly needed, performance with these drivers is much better. This has also been discussed in the whitepaper published by XenSource where they compare their product with ESX 3.0.1.

XenServer Host Management

Management of the XenServer host from the management console is limited. When you right click the XenServer host, you get the following menu:

You can also get a text console to the XenServer host directly from the management console (click image to enlarge):

Some very basic performance graphs are also available (click image to enlarge):

Let's take a look at networking and storage. During installation, I only configured eth0 with an IP of 10.0.0.10. A network is created automatically called xenbr0 and it uses eth0 to "bridge" the virtual machine's virtual interfaces.

You can easily add and remove networks with the + and - buttons. In the screenshot below (click to enlarge), you see I added an internal network called vbridge0. Because I only have eth0 enabled, I can only create internal networks because eth0 is already in use.

What I miss here a bit is the wealth of options you can set with VMware's management tools. With VI Client, virtual networks are graphically represented as switches with ports for virtual machines, management, VMotion etc... With VI Client, you can also configure load balancing and set security options such as allowing promiscuous mode, permitting MAC address changes and so on. ESX also supports VLANs but that is coming with XenEnterprise 3.2. VI Client also allows you to set the IP address for your management interface, DNS servers and host name. XenSource's management tools only show this information. To change it you need to use the command line. That is no big deal but it should be added to the management console to make configuration as easy as possible.

Storage configuration with the XenEnterprise management console is simple. You can see what's in use and what is left:

Actual storage management needs to be done from the command line. You need to use XenServer sm commands and Linux lvm commands. To see what you have to do to add storage, take a look here. Storage management with ESX and VI Client is much simpler. Adding and removing storage (VMFS volumes), issuing SAN rescans and setting HBA failover policy can all be done from the management console. There are command line tools available but they are not required for general use.

Something that is really missing is good documentation for network and storage configuration. The user manual is very basic and does not explain the internals of the system. There is a knowledge base with quite some information and the forums are full of helpful posts but that is not enough.

Conclusion

Installation of XenEnterprise 3.1 is very simple on supported hardware. Installing the management console and performing basic management tasks is simple because of the Windows based management console. The bad news is that the management console only allows basic XenServer host management. You quickly need to use the command line to perform management tasks and managing storage and networking is not as straightforward as with VMware ESX. Most of these problems can be easily fixed however by building a better management GUI and writing better documentation. The basic technology works and is actually quite impressive if you think about it. In time, I fully expect this software to be a valuable alternative to ESX at a much lower price point. Well, maybe until Windows Server Virtualization arrives anyway.

In a later post, I will talk a bit about working with virtual machines on XenEnterprise.

XenSource have published their hypervisor performance comparison with ESX 3.0.1 data included. As you might recall, VMware started this by publishing a whitepaper comparing ESX 3.0.1 with the open source version of Xen 3.0.3 (not a good idea if you ask me). A bit later, XenSource answered by publishing a paper with their own results but without ESX 3.0.1 data as that is forbidden by VMware's EULA. XenSource compared ESX 3.0.1 to XenEnterprise, their commercial implementation of the Xen hypervisor.

The results are interesting because performance is very similar. But is this enough to conclude that the XenSource product is a valid  alternative to the VI3 offering? We think not and this for several reasons:

1. It is important for our customers to use a product that has established itself in the market. The XenSource product is not there yet and will need some time to mature.
2. The management tools have to be top-notch. Our experience shows that most customers virtualize Windows systems and are primarily running Windows in their environment. They do not want to mess with Linux configuration and management so it is important that the management products allow this. VI Client with VI3 comes very close to this need although some actions do require some command line knowledge.
3. Hardware compatibility and support. ESX 3.0.1 supports a wide array of hardware and storage devices. Installation and configuration is very straightforward. XenSource's products are not at that level yet.

There are probably other reasons but I currently do not see any good reason to recommend XenSource XenEnterprise to customers instead of VMware's VI3. Maybe you have another opinion? Feel free to comment! ;-)

If you modified the VMware VI3 server based license file and you want to have it checked, you can use the following web application: http://www.vmware.com/checklicense/.

Just paste your license file contents and let the tool check it. It will respond with the number of licenses for each feature and report errors.

We had a problem with a SharePoint implementation recently. All of a sudden, document icons were missing from document libraries, the New... button did not work, the option "Edit with Microsoft Word" was missing, and so on...

The reason for all of this? Incorrect permissions on docicon.xml on the web front-end server. That file contains definitions of file types, their icon and how these files should be opened. Make sure that the file inherits its permissions from the parent. If only Administrators and SYSTEM have rights, it will not work (obviously).

Head over to MSDN and check out the article about IIS 7.0. IIS 7.0 will be part of Longhorn but is already available on Windows Vista.

On the Microsoft SharePoint Products and Technologies Team Blog, Troy Starr is writing a series of blog posts about alternate access mappings (AAM). AAM is something particularly tricky in a SharePoint deployment that you need to get right. Depending on the complexity of your configuration (intranet, extranet, SSL, ...) you need to configure AAM appropriately.

Check out part 1 and part 2. Part 3 is coming later.

Arlindo Alves (Microsoft Belgium) is writing a series of blog posts about Longhorn. Now at part 6, it provides a good overview of the features and solutions that Longhorn can provide. Part 6 is here. You will find links to the other parts on the blog.

Personally I think the following features will provide a lot of benefits to customers:

• Windows Server Virtualization
  A virtualization solution that is very different from Virtual Server 2005 but that is expected to have quite a big impact on the virtualization market. Architecturally, it can be compared to Xen but the upgrade path from Virtual Server 2005 is expected to be smooth (vhd format). System Center Virtual Machine Manager (with PowerShell support) and MOM are expected to make administration, monitoring and reporting on the virtual infrastructure quite easy.
  
• Server Core
  I blogged about Server Core earlier and it is a great solution for many server roles (like domain controllers, DHCP and DNS etc...). Less installed components means a smaller attack surface and less patching. I expect the final release of server core to be a bit more administrator friendly than what you have now in the betas.
  
• PowerShell
  PowerShell is already available but I expect there will be some changes for Longhorn to make administration even simpler. Not much is known about those new features however.
  
• IIS 7
  Much more manageable than the current version, it will be important as a base for current and future applications.
  
• Active Directory
  A lot of new Active Directory features that will make life easier with Longhorn such as restartable Active Directory services, more granular security policies (as opposed to one per domain now), read only domain controller (RODC) and more.

Check out the new version of device emulator. It has some interesting new features (check the link for more info) and is supported on Vista. While you are at it, get the new emulator images for Windows Mobile 6 as well so you can see what's coming until you get your new device! ;-)

We are currently evaluating some virtual machine replication options for an ESX 3.0.1 branch office deployment. Without a SAN or iSCSI storage, we need to replicate virtual machines between ESX servers over the network. Currently, there are a few options:

• esxReplicator from vizioncore
• Double-Take for VMware Infrastructure

These options replicate the whole virtual machine from the outside. Of course, software exists to replicate virtual machines from within the guest operating system but we decided to exclude these for now to stay OS-agnostic.

At the moment, esxReplicator for VI3 is still in development with a public beta out soon. So we decided to take a look at Double-Take for VMware Infrastructure now and look at esxReplicator later.

Installation

The solution does not require any agents on the ESX servers . You only need a Windows Server 2003 machine (can be virtual) with the .NET Framework 2.0 installed. Installation is super simple and installs a Windows service that will be responsible for monitoring and licensing. You can connect to the service with a client that you install on your desktop or on the server itself.

Their software communicates with VirtualCenter to present you with ESX hosts and their virtual machines so you can choose which virtual machines to protect.

Note that it is highly recommended to use ESX 3.0.1 together with VirtualCenter 2.0.1. You also need to enable SSH logons with root and open the SSH client port on the ESX firewall.

How does it work?

After you start the Double-Take for VMware Infrastructure (DTVI) client, you need to provide credentials to logon to VirtualCenter. You will then be presented with a very simple interface to start creating protection jobs. Protecting a virtual machine is just a matter of launching a wizard and answering some basic questions:

• What is the source ESX host?
• What virtual machine you want to protect on the source ESX host?
• What is the target ESX host?
• What are the replication options? You can have compression, replicate when x MB of data has changed or replicate every x minutes regardless of the amount of changes.

DTVI uses standard virtual machine snapshots to do its work. When you protect a virtual machine for the first time, a snapshot is created and all vmdk files are transferred to the target ESX server. When that is finished, other files such as the vmx file are copied as well. To actually replicate the changes, DTVI creates a snapshot on the source machine and monitors it until the set treshold is reached. The snapshot is then copied from source ESX to target ESX using ssh. The source and target snapshot is then committed and a new snapshot is created on the source.

Note that the DTVI server is not involved in copying data between source and target ESX servers!

Failover

Failover is manual. If you want to use the replicated virtual machine you need to stop the replication job first. Then you simply register the target virtual machine with VirtualCenter and start it.

During our tests the software worked as expected and effectively replicated virtual machines between source and target. Note that the console network interface is used for copying.

Pitfalls

Because DTVI uses snapshots to easily capture changes and commit them to the target you should not create snapshots yourself. You should also stop using backup software that uses snapshots such as VCB. Naturally, you should not change the disk configuration of the source virtual machine while a protection is in place.

DTVI does not notify you when you do things that interfere with the protection job. It is perfectly possible to stop the source virtual machine and register/start the target without any complaint from DTVI. Administrators need to make sure that they stop or delete the protection job when they perform actions that interfere with DTVI. It is actually quite easy to corrupt your target virtual machine.

In my opinion, there are some things missing from DTVI (it is a first release after all):

• Monitoring and alerting for protection jobs.
• More intelligence for DTVI to notice things such as stopping and starting source and target virtual machines.
• Command line options to stop and start monitoring and create protection jobs.

But all things considered DTVI is simple to install and does what it is supposed to do quite well. I recommend you take a look at it if you need a simple replication option for VI3!

Microsoft released a new white paper about database mirroring with MOSS 2007 and WSS v3.0. The key point is that you need to perform some actions when a failover happens. You can do these actions manually or implement them using some scripts. You will also have to do some monitoring to know when a failover has happened.

If you are planning to use database mirroring with SharePoint, all I can say is test, test, test!!!

On pshell.info, I wrote an article about getting VMFS datastore information from VirtualCenter. The data is then handed over to PowerShell for further processing like free space calculations and so forth.

Check it out here.