Managing multiple configurations using Virtual PC

We use Virtual PC extensively for testing in our lab. We have about 12 different configurations for each version of Windows we test. We've spent a lot of time learning how to create these configurations so that they work reliably and to create a reasonable balance between updates of individual configurations versus a global rebuild as the security updates and service packs pile up.

This blog entry describes how we create the VHDs and configure them in Virtual PC.

Layer Overview

Our VHD tree is built based on a set of defined layers. Layer 1 is comprised of single root VHD for each version of Windows, such as WinXP SP3. Layer 2 is built from differencing VHDs that use the root VHD as the parent. This means that the root VHD needs to be configured as far as possible, without requiring lower layer VHDs have to "undo" any configuration. Here is the VHD structure we use:

1. Root VHD - Contains base Windows install and common configurations

2. Configuration specific VHD - Includes apps, service packs, etc. that are different from the root. This is where configuration-dependent changes are done.

3. Working VHD - Includes temporary changes that are used over the course of several tests.

4. Undo file.

Layers are defined by how often they are changed, because when a layer changes, everything underneath it is invalidated. Layer 1 changes every six to eighteen months, usually when the number of security updates becomes overwhelming. Layer 2 changes every few months. Layer 3 is only used when a customized configuration needs to be tested for several days (or when you hit the wrong button and say "commit change.) Layer 4 changes on a minute to minute basis. It is discarded whenever a user selects "Discard changes" when a VPC is closed.

Roadblocks

Creating the VHD hierarchy is tricky. There are several problems we've seen that must be solved by making the changes in the correct order in the correct order. Some of these problems include:

• Duplicate Ethernet MAC addresses.
• Duplicate computer names.
• Domain disconnects.
• Accidental overwriting of a parent vhd file.
• Windows Update automatically overwriting desired configurations.

The MAC address will be updated automatically by Virtual PC as long as you create a new .vmc file from the menu. In other words, use the New Virtual Machine Wizard, don't make a copy of your existing .vmc file. If you insist on copying your .vmc file, you can manually remove the MAC address from the XML file and it will automatically be recreated.
Many people are also concerned about "SID duplication" and want to use something like NewSID or SysPrep. Recent reports indicate that this is no longer a concern. Please see NewSID Retirement and the Machine SID Duplication Myth.

Create the Layer 1 Root VPC

The first task in creating the hierarchy is to create the Layer 1 VHD, which contains the OS install and will be common to all dependent virtual machines.

1. Create the .vhd file using the Disk Wizard. Set to dynamic size, 20 to 32GB is usually a good size.
2. Create the Virtual PC .vmc using the New Virtual Machine Wizard under the File menu. Make sure you give it enough RAM, the defaults are almost always too small.
3. Install Windows. Install a version of Windows that includes the desired service pack. Set the system name to be something like RootWinXp so you know when you've forgotten to change it. Do not join a domain during installation.
4. Log in as an Administrator and activate Windows, if necessary. Don't do this in a child virtual machine or you'll need an additional activation key for every child.
5. Shut down Windows and make a backup of the VHD. Name it "Level 1 - Clean" This is a "clean machine" that's been activated. Use this VHD to create completely new configuration hierarchies.

Configure the Layer 1 Root VPC

1. Boot the virtual machine.
2. Install the Virtual PC Additions.
3. Set the vpc to be part of workgroup, if it's not already. Do NOT join the root vhd to a domain! That must be done in a lower layer.
4. Install desired Windows service pack, if any.
5. For Windows XP, go to Windows Update in Internet Explorer and enable Microsoft Update.
6. Install latest security updates through Windows Update. Pay attention to the following:
• For Windows XP, go to Windows Update in Internet Explorer and check High Priority and Optional updates. The task bar "Update" icon does not show these updates.
• Under Optional, I recommend you install Update for Root Certificates.
• Also under Optional, install Microsoft .NET Framework, if needed.
• Windows Update may automatically upgrade Internet Explorer if you don't do a "Custom" update and uncheck Internet Explorer. OTOH, IE8 for Windows XP is under Optional.
• After each reboot, go back to Windows Update and check again. You'll probably need to run through Windows Update several times to get all updates.
7. In Control Panel, set Automatic Updates to "Notify me", not to automatically download or to automatically install. Otherwise you end up fighting Windows Update every time you boot a configuration.
8. Disable the requirement for Ctrl-Alt-Del. This can save a lot of mousing around when running the virtual PC in a window. For Windows XP, this can be found under Control Panel / User Accounts / Advanced (or use Group Policy.)
9. Create any desired local users (as opposed to domain users.) Make sure you set them as Admin or Limited, appropriately. At a minimum, I recommend creating a "Limited" user for testing.
10. Disable the Internet Connection Wizard if you have Windows XP virtual machines. In Group Policy, it's under User Configuration\Administrative Templates\Windows Components\Internet Explorer\Internet Settings\Advanced Settings\Internet Connection Wizard\
11. Set firewall exceptions in Control Panel | Windows Firewall | Exceptions:
• File and Printer Sharing must be enabled to allow access the system by name on the network.
• Remote Desktop must be enabled to access the system remotely.
12. If desired, enable Remote Desktop under My Computer | Properties | Remote. Note that Remote Desktop is not supported in any of the "Home" versions of Windows. Local Administrators will automatically have rights to access the system via Remote Desktop.
13. If you use Remote Debugging in Visual Studio, start up MSVCMON.EXE and tell it unblock the firewall for your subnet. You will discover that this adds numerous exceptions to the firewall. MSVCMON.EXE can be found in:
    
    C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\Remote Debugger\x86\msvsmon.exe
14. Modify options in My Computer | Tools | Folder Options | View, such as "Display the contents of system folders", "Show hidden files and folders", "Hide extensions", and Hide protected operating system files". This can also be done in the domain logon script with:
    
    reg add hkcu\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v Hidden /t REG_DWORD /d 1 /f

reg add hkcu\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v HideFileExt /t REG_DWORD /d 0 /f

reg add hkcu\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v WebViewBarricade /t REG_DWORD /d 1 /f
15. Share any desired folders.
16. Shut down the root vpc using the "Shutdown" from the Start menu.
17. Commit all changes if you are using Undo disks.
18. Set the VHD to be read-only.
19. Create a backup of the file. Name it "Level 1 - Common".
20. Optionally, go to the Virtual PC Console and remove this virtual machine so you don't accidently try to use it. If you don't delete it, make sure you enable Undo disks so you don't accidentally change it (Virtual PC will helpfully remove the read-only attribute.)

Create a Layer 2 VPC

1. Use disk wizard to create a differencing hard disk to the root.
2. Make a backup of this vhd file. Name it "Level 2 - Empty".
3. Create the Virtual PC .vmc using the menus. Again, make sure you give it enough RAM. Turn off Undo for now.
4. Boot the new virtual PC.
5. Log in as an adminstrator.
6. Change the system name.
7. Join a domain, if desired. This can be done from the command line with the NetDom command, which is available in the Windows XP Support Tools. (This should all be on one line.)
   
   netdom join myvpc /domain:test.com
/userd:Administrator
/passwordd:<domain admin password>
8. Give any desired domain users or groups permission to login via Remote Desktop.
9. Log in as any domain users you plan to use frequently. This does the one-time configuration of the VPC for that user. Change Folder Options again for these users.
10. Install any configuration-dependent software.
11. Install updates and service packs for the software you just installed.
12. Map network shares with a domain logon script. If a computer on a domain must map a drive on a non-domain computer, make sure the non-domain computer has an account with the same name and password as the domain computer. Alternatively, you use this command to save the username/password on the domain computer, then say "net use x: \\acme\share" in the domain logon script.
    
net use x: \\acme\share
/savecred /persistent:yes
13. Check Windows Update one more time. On Vista and Windows 7, make sure you explicitly tell it to check again.
14. Once everything is configured to your satisfaction, shut down Windows from the Start menu so it shuts down cleanly.
15. Set the VHD to be read-only.
16. Create a backup of the file. Name it "Level 2 - Configured".

Create a Layer 3 VPC

1. Create a third vhd, differencing, whose parent is the vhd you just set to be read-only. I normally name it the same as the parent with the word "Child" appended. This is what I called the "Working VHD" earlier in this document. You will use this when you need to reuse a configuration a few times, then discard the configuration. Normally this vhd is empty.
2. Change the settings in Virtual PC for your virtual machine to point to this working file.
3. Enable Undo for this vhd.
4. Backup this third VHD. Label it "Level 3 - Empty". You'll need the backup whenever you discard your current working configuration.

At this point you should have five backup files. The files labeled "Empty" in the below should be small, between 44KB and 106KB:

• Level 1 - Clean root VHD, with a clean Windows install.
• Level 1 - Configured root VHD, with a mostly configured Windows install.
• Level 2 - Empty configuration VHD, which is an empty differencing disk whose parent is the Level 1 vhd.
• Level 2 - Configured VHD, which is your fully configured test environment.
• Level 3 - Empty VHD.

Create Remaining Configurations

1. Copy your Level 2 - Empty vhd to a new file that will be the new configuration. Under Windows 7, you can create all of them at once using the DISKPART utility. For example: (DiskPart requires elevated privileges and so must be run from a command prompt started with Run As Administrator.)
   
   DISKPART < Parts.txt
   
   And Parts.txt is one or more lines similar to this:
   create vdisk file="c:\vpc\ParentDisks\WinXP Office 2007.vhd" parent="c:\vpc\ParentDisks\WinXP SP3 Root.vhd"
2. Go to Step 3 under Configure a Layer 2 VPC and continue through Configure a Layer 3 VPCs.

Licensing

Licensing is also an issue. Strictly speaking, it appears that you need one Windows license per virtual PC. In practice, we have a dozen different variants of each root virtual PC, with only minor changes between each, such as whether it's a member of a workgroup or a member of a domain, or whether it's Service Pack 2 of WinXP or Service Pack 3. Since we rebuild these fairly routinely from scratch, it is hard to believe that we are supposed to own thirty different licenses and rebuy all of them every few months. Our solutions for Windows has been to use the volume licensing versions of Windows, which are available to Microsoft Certified Partners on MSDN Downloads.

NUnit Unit Testing with C++

I switch back and forth between C++ and C#. When doing C# development, NUnit rules the day for unit testing. Whether I'm doing automated tests from the command line or using the GUI to run selective tests (show below in a screenshot from SourceForge) NUnit is a pleasure to use.



If you've ever tried to run unit tests for C++, the landscape is much less appealing. C++ does not have a reflection API, nor does it have attributes that are embedded in the executable code, so it's much more tedious in C++ to do all of the housekeeping to initialize the framework and it's much more difficult to integrate external GUI tools. In short, unit test in C++ is a sub-par experience compared to more modern languages.

But I have good news for you - it's possible to use NUnit with C++. There are two minor caveats:

1. You must be using Microsoft Visual Studio 2005 or later (sorry MinGW and cygwin users.)
   
2. The code to be tested must either be a DLL or a static library. If you need to test code in an .EXE, you should factor that code out into its own static library.
   

The secret is to place your tests in a separate DLL compiled for C++/CLI, which you enable on the Visual Studio C++ properties page in the General properties, immediately under Configuration Properties. Under Project Defaults, set Common Language Runtime support to "/clr". Don't use any of the other variants - they won't work for this task.

If you haven't built this kind of project before, a C++/CLI project is a curious hybrid that contains all of the power of C++ as well as much of the power of .Net. (Access to certain features, like LINQ, is not available in C++/CLI.)

With a C++/CLI project, you can use NUnit attributes for classes and member functions, just like in C#. When you run NUnit, your target is your test DLL, which implicitly loads either your static library code or your DLL code.

This strategy can also be used in the test environment build into Visual Studio if you have the Professional edition or better.

Downloading symbols for .dmp files

We have an automated system for processing .dmp files received from the field, either our own collection or Microsoft's crash report system. We use cdb (the command line version of WinDbg) to automatically create a text file that contains the stack dump.

The batch file looks like this:

@set _NT_SYMBOL_PATH=SRV*c:\cache*http://msdl.microsoft.com/download/symbols
Cdb -lines -c "!analyze -v;q" -z %1

(Thanks to John Robbins for showing me how to use cdb in this manner.)

In theory, setting _NT_SYMBOL_PATH should provide cdb with enough information to automatically download symbols as needed. However, I wasn't seeing that happening. Without symbols, the debugger can't properly processes callstacks using FPO (Frame Pointer Omission), which means that the callstacks were often missing a lot of information.

Today I found a workaround. The symchk utility will examine a dmp file and verify that all of the required pdb and dbg files have been downloaded. Here is an example:

symchk /id Demo_000000.dmp /s SRV*c:\cache*http://msdl.microsoft.com/download/symbols


DreamWeaver CS4 - First Look

I finally broke down and bought DreamWeaver CS4 today. This is the first time I've used Dreamweaver since 1997. It works a lot better on a 4GB Core 2 Duo machine than it did on a Pentium II 350 :-)

Anyway, here are my initial impressions.

The Good

1. It doesn't crash! In the past eight years I don't think I ever used GoLive for more than 40 minutes straight without crashing. DW hasn't crashed yet.
   
2. Works under Windows Vista.
   
3. The upgrade registration was painless. I entered my version of GoLive and my GoLive serial #. No problems.
   
4. DW supports sftp.
   
5. I/O to the remote sftp server is fast, in direct contrast to every other web dev tool I've tried.
   
6. Template page updates are fast. GoLive was always very slow.
   
7. Live View allows you to see the browser view without going to the browser. Pretty slick.
   
8. Dual Monitor mode.
   

The Bad

1. It took me a long time to figure out how to activate the Import Golive Site command. It's not in any of the menus. Looking for "import golive" in Dreamweaver Help brings up numerous old articles about DW CS3, which worked completely differently. I think I finally found the answer by searching on "golive to dreamweaver cs4" (without the quotes) but the first several search results were still for CS3. The answer can be found at Dreamweaver for GoLive users, http://www.adobe.com/devnet/dreamweaver/golive_migration/
2. After going through the effort to find and install it, the GoLive import tool didn't work at all. It refused to open the site file. My workaround was to use the GL2DW tool from CS3, which is installed in GoLive instead of DW. GL2DW crashed halfway through, but it did enough that I was able to determine what to do myself.
   
3. I got stuck on how to open an existing site. Turns out you use "New Site..." even for an existing site. From what I've heard, this will just be the first in a long list of Dreamweaver-isms that I'll run into.
   
4. Where's the "remove whitespace" command? Every other tool I've ever used, even Microsoft Expression, does this automatically when the site is uploaded.
   
5. Only Subversion is supported for source control, which is a little strange since SourceSafe is supported as a test server. Not a big deal, but SourceSafe integration would have been handy.
   

Passive (PASV) ftp in Windows 7

I was controlling a customer's computer and trying to upload a file to our corporate ftp server. I could connect and log in, but any other command would cause the ftp client to hang. I was using the ftp command built in to XP because I didn't have privileges to install a 3rd party ftp client. It was frustrating, but I didn't have time to debug the problem.

Today I had the same problem on a system in the office when trying to connect to a new server. However, this time I was running a different ftp server (vsftpd) that gave me a helpful error message:

200 PORT command successful. Consider using PASV.

I tried the obvious, which was to restart ftp from the command prompt and then type PASV, but this gave the error "Invalid command." I found several posts that said that the ftp client built-in to Windows doesn't support passive mode. The good news is that they are wrong this is fixed in newer versions of Windows.

With Windows 7, enter this command to enter passive mode:

QUOTE PASV

With vsftpd, I was rewarded with this response, and everything started working:

227 Entering Passive Mode

 
As other have noted in the comments, this does not work on Windows XP.

Remote Debugging Managed Code

Last year I blogged about how to get Remote Debugging working. That article was for Visual Studio 2005, but it works fine with some minor modifications for directory and filenames for Visual Studio 2008.

Now I've had to take the next step, which is remote debugging managed code instead of native code. Turns out that debugging managed code is even more complicated than native code.

One of the key problems I originally had was that I was never able to get authenticated debugging working because one computer was on a workgroup and one was on a domain. After reading several articles, I was able to solve the problem by creating an account on the remote computer that had the same username and password as the account of the development computer. If your development system isn't on a domain, make sure that the remote account is created on the local computer and not the domain. You may need to use the "NET USER" command from the command line to create such an account.

If you are running Windows Firewall, the remote debug module msvcmon.exe will automatically unblock the right ports for you. At least that was easy.

The next trick is to convince .Net to run your application from a network drive. This technique is described in the .Net Security Blog.

By the way, the Remote Debugger Configuration Wizard was useless for me.

Choosing Web Development Software

For the past eight years I've been using Adobe GoLive. I got started with GoLive because it was recommended by people I trusted and it was a lot faster than Dreamweaver. GoLive also did a good job of editing PHP and had a nice, simple template tool that allowed me to do WYSIWYG editing without having to hand-maintain dozens of files.

Times have changed. The Windows version of GoLive is now horribly unstable and Adobe simply removes features instead of fixing them. I've started looking for a new tool. Our site has been PHP for several years, so it isn't worth the effort to switch it to ASP.net.

I found Visual Studio 2005 to be okay for quick HTML edits, but it's rather clunky editing HTML compared to more sophisticated tools. Out of the box, it also can't handle PHP. I tried VS.php, which is inexpensive, but the VS.php debugger never worked properly and their tech support was useless. After a year of trying, I gave up on VS.php in Sep 2007.

Instead I settled on PHPed from NuSphere. It is a fantastic tool that I'd recommend to any PHP developer. Its debugger is very reliable. However, PHPed is purely a PHP development tool and doesn't do HTML editing.

So I'm still looking for an HTML tool. Although Dreamweaver is the obvious choice, it's expensive to buy, expensive to support, and expensive to upgrade. So it wasn't my first choice. OTOH, I'm a Microsoft Certified Partner and so can use all of their development tools for free. When I saw Expression Web at the Professional Developers Conference in 2005, I initially had high hopes, but then I learned that it was restricted to Microsoft languages. Oh well.

In February, Microsoft shipped Expression Web 2 Beta. Among other things, this update supports PHP. It's also easily the fastest tool I've used. Unlike Visual Studio 2005 and Adobe GoLive, you don't need to take a coffee break to start it up and shut it down.

My initial reaction to Expression Web 2 was quite favorable. It's beta, so the WebDAV support doesn't work with Apache and there are other teething problems, but these are okay. However, the deal killer is that you can't define the root directory, so all URLs relative to the root don't work. Microsoft has indicated they don't plan to fix this problem in this release. WTF?

Concurrent to all of this, I tried iWeb on the Mac. It's a high-level tool designed for non-web designers. It generates completely unmaintainable code, but it works and the results look good. It also has the ability to do handy things like rotate images, add drop shadows, and other image operations that historically have taken twenty steps to perform using an external image editing application, such as Paint.Net or Photoshop. iWeb will also natively import PhotoShop and Illustrator files. This is a huge win compared to most tools under Windows. iWeb also has some pretty good templates, in stark contrast to any software I've ever seen under Windows. So I've seen a vision of what things could be like with the best of both worlds, but I haven't seen a solution.

So now I'm back to square one. I'll probably have to buy Dreamweaver. It's $1,000 for Creative Suite (CS3) Web Standard version or $1,600 for CS3 Web Premium. Ouch. To add insult to injury, there's no way to upgrade from GoLive to Dreamweaver, even though they're both made by Adobe. Thanks Adobe!

---

Update 5/13/2008: Adobe has formally announced that GoLive has been discontinued. The good news is that they are offering an upgrade price to switch to DreamWeaver as well as a migration kit for converting GoLive projects. Read the details at:
http://www.adobe.com/products/golive/pdfs/golive_eol_faq.pdf

---

Virtual Server 2007 versus VMware Server

With the advent of the Core 2 Duo processors, it's finally possible to build a server for virtual machines without breaking the bank. I have three computers in my office that run various versions of Exchange Server for testing. Most of the time, these boxes are little more than space heaters, so I was looking forward to integrating everything into a single box.

I started by installing Windows Server 2003 x64, which is required for Exchange 2007. My plan was to run Exchange 2007 on the host x64 operating system and the older versions of Exchange in virtual machines.

The Microsoft solution for creating virtual servers is Microsoft Virtual Server 2005 R2. I had tried to install the software before the R2 version and I found it to be difficult to install and even more difficult to use. The documentation, while extensive, didn't have any Getting Started section that made any sense. My experience installing the R2 version was only slightly better and I spent several hours figuring out how to do things that should have been obvious.

The next step was to convert my physical servers into virtual servers (P2V). I spent almost a day researching how to do this and finally gave up. Microsoft's tools are immature and very difficult to use. The P2V conversion process relies on having a Domain Controller, Microsoft Operations Manager (the 32-bit version), and other requirements, none of which I had the time or inclination to set up. I uninstalled Virtual Server 2005 R2.

The alternative was VMware's product, VMware Server, which just recently became free. I found this software to be very easy to use. I installed it, opened the desktop icon labeled "VMware Server Console," and was able to immediately get started. This was a stark contrast to my experience with Microsoft's product. (Note that I've used Virtual PC since before Microsoft purchased it, so I have no shortage of experience with the technology that Microsoft Virtual Server is based on.)

Converting my physical servers to virtual servers was equally easy. I downloaded VMware Converter and followed the simple wizard. Once the conversion process was finished, I went back to VMware Console, added the new virtual machine, booted it, and it worked. The only caveat was that you have to power down the old physical computer before booting the virtual machine, or you end up with duplicate names. Also, if you've set your DHCP server to hand out a particular IP address to the MAC on your old server, you need to update your DHCP server.

It's also possible to run the VMware console on other computers by downloading the Client installer from the VMware downloads page. Although this works well when you are managing multiple servers, the sluggish screen update over the LAN can be annoying. I use Remote Desktop when I need to work with one of the virtual servers for any period of time.

I'm really impressed with VMware Server. It works well and has some impressive upgrades if I need better manageability than I have now.

Parallel Builds - Success!

I've given up on the parallel build facility in VCBUILD (the /M option, such as /M2). It is almost useless for our build scenario. Instead I've switched to GNU make from the "cygwin" project. GNU make supports the -j command line parameter to enable parallel builds. GNU make also allows me to specify dependencies at whatever granularity level I need. Currently I specify dependencies on a per-project basis (one project = one .vcproj file), but I could just as easily go one level deeper and have multiple dependencies for each project, such as Debug/Release/Demo.

My makefile looks something like this (simplified for demonstration purposes):

all: sdk gui server

sdk:
vcbuild c:/proj/sdk/sdk.vcproj

gui: core
vcbuild c:/proj/gui/gui.vcproj

server: core
vcbuild c:/proj/svr/svr.vcproj

core:
vcbuild c:/proj/core/core.vcproj

This makefile is launched by a batch file that sets two very important environment variables:

set SolutionPath=c:\proj\proj.sln
set VCBUILD_DEFAULT_OPTIONS=/M1

By setting the "SolutionPath" environment variable to the solution file, we give VCBUILD the ability to set the $(SolutionDir) variable in the build scripts, which is typically required to build successfully.

When you point VCBUILD at a vcproj file, it automatically builds all configurations in that file. This way the makefile doesn't need to have knowledge of all the possible configurations. One feature you give up by specifying the vcproj file is that VCBUILD won't automatically add dependent libraries to the link line. For Visual Studio, this isn't a loss because the feature is largely broken anyway.

Using GNU make gives me the control that I need to keep my dual core processor busy. I'm finally happy :-)