The latest community preview (post beta 3) installs without issues on VMware ESX 3.0. Previously, you needed a driver for the cd-rom but that seems to have been fixed. When you create your virtual machine, be sure to select Windows Vista. This will ensure that the VMware Tools install properly. You need to install the tools because you won't have network connectivity without them.

I also installed Windows Server Core on ESX 3.0. To install the VMware Tools, use the following command:

msiexec.exe /i "<path to VMware Tools.msi>" /qn

The above command worked perfectly for me. It will reboot your server core machine automatically. After the reboot, you can configure your network interface with the netsh.exe command.