WinSxS Breaks Old Libraries

After my previous experiences with handling Windows Side-by-Side Assemblies (WinSxS) in remote debugging and Isolated COM, I thought I was actually starting to get a handle on how it worked. Today I got stuck on another WinSxS problem, this time while porting our application to Visual Studio 2008.

The problem was that the application would fail to fail to start, with an error about the manifest being wrong. I used sxstrace, a handy tool under Vista, to try and determine what was happening. Sxstrace generated 180 lines of information about Vista's attempt to find and load the correct assemblies. It ended up being too much information. The only obvious problem I saw was that one of the DLLs being loaded was from Visual Studio 2005, not 2008.

I used Depends to look at the file and received the same errors. I looked in the Event Viewer and saw this error:

Activation context generation failed for "s:\csi\xsales\debug\XSALES.EXE". Dependent Assembly Microsoft.VC90.DebugCRT, processorArchitecture="x86", publicKeyToken="1fc8b3b9a1e18e3b", type="win32",version="9.0.20706.1" could not be found. Please use sxstrace.exe for detailed diagnosis.

This is even stranger, because 20706 was the version ID of Visual Studio 2008 Beta 2. I'm running the final release.

I used Visual Studio to open XSales.exe in Resource mode so I could look at the manifest itself. Here I found references to four versions of DebugCRT. The question was, where were they coming from? My code makes no explicit reference to assembly versions.

What I discovered was that the intermediate manifests generated by the compiler and linker include assembly information from all objects used by the linker, including objects from libraries, of which I had two. One of my .LIB files was generated by VS2005 and another was generated by VS2008 Beta 2, which is what caused the references to old assembly versions. Once I rebuilt those .LIB files with VS2008, the problem went away.

The lesson learned from all of this is that .LIB files are no longer easily portable between versions if they rely on any of the CRT or MFC DLLs. The painful part of this is that the problem doesn't show up until you try and run the software because none of the development tools warn about the inconsistency.

[Update 11/19/2008] You can find another solution here.

Corrupt \Boot\BCD in Windows Vista

Two nights ago, I started installing a video driver upgrade and went home for the night. The next morning I came in to discover that something had failed in the upgrade and the system was hung on a black screen. Even Remote Desktop access wasn't working.

I hit the Reset button. The system started to reboot. As per usual when I use the Reset button, the RAID array was incensed that I had interrupted its normal operation and took its revenge by taking four hours to validate the mirror. 89% of the way through validating the mirror, I received my first Vista Blue Screen of Death (BSOD). Something about a power driver not handling an event.

No problem. I hit the Reset button again. What's another four hours between friends?

Except this time, the system didn't reboot. It went through the BIOS check, started to boot Vista, and presented me with this error:

File: \Boot\BCD
Status: 0xc0000034
Info: The Windows Boot Configuration Data file is missing required information

The Microsoft Knowledgebase has a helpful article on this subject, which should have worked, but didn't. The error messages during the failed repair didn't clarify the cause of the problem.

I tried booting Vista from DVD and running recovery. Rebuilding the BCD file is a standard operation, except that it didn't work. The Recovery Manager said that it couldn't save the file. Curiouser and curiouser.

I restored the BCD file from backup (Acronis True Image pays for itself... again.)

I rebooted the system and got the exact same error. I went back in with the Recovery Manager and discovered that the file \Boot\BCD was gone, even though I had just restored it.

The final solution was to use Acronis to restore track zero and the Master Boot Record (MBR) from my last image backup. There never was a problem with the BCD file, that was just a red herring. I'm not sure how you'd solve this problem if you didn't have an image backup. It just goes to show the usefulness of image-level backups.

Update (11-15-2007): I discovered that the Vista repair utility created a file named C:\Temp\SrtTrail.txt. This file contained the list of tests that were performed. The last test showed this:

Root cause found: 
---------------------------
No OS files found on disk.

Repair action: Partition table repair
Result: Failed. Error code =  0x3bc3
Time taken = 154722 ms

If I'd seen this file earlier, I would have had a clear indicator to repair track zero instead of stumbling on the solution accidentally.

Legacy Code from ... CP/M?!?

Remember CP/M? Unless you are over 40, my guess is probably not. This week I ran into a bug in our software that traces its roots all the way back to CP/M in the 1970s. Over thirty years later, the code still exists in the Visual Studio 2005 C Runtime Library, waiting for the next innocent victim to run afoul of it.

Here's what happened. We had reverse engineered a data file format and had shipped the first pass to customers. Our only sample of the data file was quite small, about 10 records, but it was enough to determine the file format and make it work. The ten records in the file converted cleanly.

After the product shipped, several customers reported that only 26 records were being loaded. This was obviously incorrect as most of their files had hundreds of records. Our end-of-file handling code was common with numerous other modules and worked fine. The 26 records that did convert did so correctly.

Even when we instrumented our software to give more insight into what was happening at customer sites, we found nothing. Our software converted 26 records, detected end-of-file, and exited. WTF?

When we finally found a customer willing to share his data file, we ran our software in the debugger and got exactly the same results. 26 records were converted and the software exited cleanly, with no errors.

The lightbulb didn't go off over my head until I was looking at the data file with the cygwin "cat -v" command, which shows ASCII codes 0 through 31 as control characters. This particular data file had two bytes for each record ID and record numbering started at 0. The 27th record contained ID 26 (0x1a) which showed up as ^Z. Does that ring any bells? If you ever developed for CP/M (or for MS-DOS 1.0) it should.

Thirty years ago, CP/M only tracked the number of blocks in each file, not the number of bytes. By convention, Ctrl-Z was used to denote "end of file" for text files. MS-DOS 1.0, which bore a striking resemblance to CP/M internally, followed this same convention. At the time, the C Runtime Library understood this convention and automatically generated an "end of file" condition when ^Z was encountered. Today, the VS2005 C Runtime Library still contains that code and generates the end-of-file condition even if the exact length of the file extends beyond that point.

The bug in our software was that the file had been opened in text mode instead of binary mode. Normally this is easy to detect because the records become out of sync as they are read, but by some coincidence of the data layout, the data in this file was read in perfectly up until that ^Z.

So now I have one more reason to dislike CP/M, although (admittedly) in this day and age it seems somewhat pointless to carry a grudge against a dead operating system that was designed to run off of 8" floppy disks. Old habits are hard to break.

Static Code Analysis with Visual Studio 2005

Not too long ago I received a jarring reminder that "char" in Visual C++ is signed by default. The problem was related to the ctype functions, such as isalnum() and isspace(), all of which take an integer. When you sign-extend an eight-bit multibyte character (MBCS), you end up with a negative number. The ctype functions then do a table lookup on a negative number and your program crashes. This wouldn't be so interesting, except that this bug existed in our production code for six years before the software crashed for a customer in Europe. If the customer hadn't been willing to give me sample data to reproduce the problem, I'd still be scratching my head.

Apparently I'm not the only person who has run into this issue. Visual C++ 2005 includes a command line option named "/analyze" for finding this problem and many others. I had tried /analyze during the Whidbey Beta two years ago but didn't get useful results. Now the results were much more helpful.

The "/analyze" options enables the static code analysis feature in the C++ compiler. This feature is based on prefast, a technology that came out of Microsoft Research in 2001. Prefast knows about hundreds of common programming problems. I ran it on our codebase, which after eight years is quite mature and already compiled under warning level 4 with minimal problems.

What I learned was quite interesting. Prefast knows about the ctype problem I mentioned earlier and pointed out several lines I missed when I tried to fix the problem. All of those lines were crashes waiting to happen.

Prefast found several places where I expected one return type but was getting another. For example, at one point I checked for an HRESULT but was actually getting a bool, which meant that the sense of my error check was inverted.

Another error that prefast found was where I was calling sizeof on a variable that was defined as pointer instead of as an array, which meant that the length being handed to strncmp was wildly wrong. The code still worked, but that was a happy accident.

Prefast also had numerous warnings about the Boost C++ Library. It should be possible to suppress those warnings, but I haven't done so yet.

Prefast supports annotating your code with the Standard Annotation Language (SAL) to better describe the static behavior of the code. If you've looked at recent versions of the Windows SDK, there are numerous annotations such as __out_ecount(). These annotations provide additional information to prefast that allows for better analysis. You can learn more about these annotations at blogs.msdn.com.

Prefast also pointed out two problems with my code that I would never have found, even with a close code inspection. I'm sure you've seen stricmp, the case-insensitive version of strcmp. I was using stricmp to check for keywords. Turns out that this is a bad idea. Prefast gives warning C6400, which explains that some languages interpret combinations of letters as a single letter, which changes the behavior of stricmp. The correct solution is to use the Windows API call CompareString(), which should be set to LOCALE_INVARIANT or LANG_ENGLISH, depending on the version of Windows.

Prefast also warned me that using _alloca in a loop could cause a stack overflow. Normally I'd consider this obvious, but in this case the warning was being given about the W2A Unicode to MBCS conversion macro, which uses _alloca. I looked at the source code to W2A (in atlconv.h) and the problem seems to be handled properly, but it was a worthwhile exercise.

I'm adding prefast to my bag of recommended tricks. For more insight into the use of this tool, I invite you to read Scalable Defect Detection from the Center for Software Excellence at Microsoft.

Visual Studio 2008 Beta 2 Test Results

Today I tried building our product with Visual Studio 2008 Beta 2. The last two times I tested compiler upgrades (VC6 to VS.net 2003, then VS.net 2003 to VS2005), the upgrades were extremely painful and took days to complete.

I was pleasantly surprised that rebuilding our product in VS2008 Beta 2 went quite smoothly. The issues were minor:

• The switch /OPT:NOWIN98 option is no longer supported.
  Impact: None.
  
• Project could not be linked against a library originally built in VC6.
  Impact: Rebuild library.
  
• Visual Studio 2005 manifest that set requireAdministrator caused this error: manifest authoring error c1010001: Values of attribute "level" not equal in different manifest snippets.
  Impact: Removed the explicit manifest file and set the UAC level in the Linker node on the project properties.
  
• MT.EXE gives an error on a valid manifest.
  Impact: Move $(WindowsSdkDir)\bin to the top of the list for Executables files in VC++ Directories under Tools/Options.

In terms of the size of the generated executables, here are the stats:

Product	VS2005 size	VS2008 Size	Net Change
#1 (C++/MFC GUI)	1,198,592	1,180,672	-1.5%
#2 (C++/MFC GUI)	1,589,760	1,549,312	-2.5%
#3 (C++/MFC Console)	2,043,904	1,998,848	-2.2%

While not earth shattering, these numbers do show that they were able to contain bloat in MFC and the C runtime library. All of these applications are statically linked against MFC and the CRT.

The biggest surprise was that VC2008 worked fine with version 1.34.1 of the Boost library, which has not yet been tested on VC2008. (although, admittedly, our software only uses one or two of the Boost modules.)

That's it! So far, backwards compatibility is excellent.

Initial testing has shown no problems with the generated code. Common dialogs under Vista were automatically updated to Vista styling, as promised.

My only disappointment is that very little was done for the IDE for C++ developers in this version of Visual Studio. The tools for editing dialogs and other resources are still awful compared to VC6. For a look at the future of the IDE, take a look at Somasegar's blog. Attention is being renewed on the IDE for native C++ developers, but improvements won't ship until 2010.

Running the Visual Studio 2008 Beta 2 VHD on VMware Server

Microsoft has finally climbed on the bandwagon for using pre-configured virtual machines to distribute beta software. For anyone who tried to install earlier betas of Visual Studio 2005 Team System, you'll understand me when I say that these pre-built virtual machines will save you days of frustration.

I downloaded the VHD disk images for Visual Studio 2008 Beta 2. Although they work fine on the free download of Virtual PC 2007, I really wanted to run this image on my virtual machine server, which uses VMware Server (see my earlier comments on VMware Server versus Microsoft Virtual Server).

Before I describe the procedure, one BIG caveat: Once Windows is running in VMware, Windows will complain that it needs to be reactivated. If you are a Microsoft Partner you can get a key from MSDN Downloads. Otherwise you will need to use a new key, which basically means you need to buy Windows Server 2003 Enterprise. Therefore, if you don't have a ready supply of activation keys, this procedure won't work for you. It may be possible to call the activation people and have them honor the key built into the virtual machine, but I haven't tried.

Converting the Orcas VHD to VMware ended up being a lot more difficult than I'd hoped. The biggest problem was converting the virtual drives from .vhd format to .vmdk format. I found a nice utility named WinImage that could do this. WinImage converted the base Orcas image (2.8GB) without difficulty, but gave up with no error when I tried to convert the 11.8GB differencing disk. I didn't really want to go back and forth with the WinImage support for two days, so I looked for an alternative.

My final solution was to use Acronis to do a backup in Virtual PC, then use Acronis again to do a restore in VMware. To do this yourself, you'll need:

VMware Server (free)
Acronis TrueImage Home or better (commercial)
A Windows Server 2003 Enterprise installation CD
WinImage (shareware)

The solution was as follows:
1. Install Acronis TrueImage on any Windows box and create a Rescue CD.
2. Create a virtual machine in Virtual PC 2007 that contains Orcas Beta 2.
3. Set the virtual machine to connect to the CD you created in Step #1.
4. Boot the version machine, select Acronis and back up the Orcas virtual machine to any desired network drive.
5. Use WinImage to convert the Base01 image to VMDK. Make sure you create a dynamic disk and not a fixed disk.
6. In VMware Server, create a virtual machine that points at the file from #5.
7. Put the Acronis Rescue CD in a CD drive on that computer.
8. Start the virtual machine in VMware, press Esc, and boot from the Rescue CD.
9. Restore the Acronis backup to the current disk.
10. After restore completes, reboot the virtual machine. You'll get an error about a service that didn't start. Ignore it.
11. On the VM menu, select Send Ctrl-Alt-Del.
12. Enter the password from the Microsoft web page. You'll need to use the keyboard, your mouse probably won't work.
13. As the login completes, you'll be prompted for the path to install the Ethernet card. Put in the Windows Server 2003 Enterprise installation CD and click OK. Windows will complain that it can't find the driver, which is okay. Tell Windows you want to install from an alternative location.
14. Select the file Driver.cab on the Windows Server 2003 Enterprise installation CD. The network driver should be located automatically.
15. After installation of the initial driver completes, do not reboot or you won't be able to login again.
16. On the VM menu in the VMware Console, choose Install VMware tools.
17 At some point you'll be prompted for the file mouclass.sys. Tell the installer to use the copy of the file in C:\Windows\System32\Drivers.
18. Now you can reboot and everything should work.

Upgrading PEAR on Red Hat Enterprise ES 3

Recently we've upgraded our coding standards for our server-side PHP development to include unit tests, which required that we install PHPUnit for PHP 4. Normally installing a new pear module takes about ten seconds, like this:

pear install PHPUnit

Unfortunately, this time things didn't go so smoothly. I ended up with dozens of errors such as this one:

Warning: xml_parse() [http://www.php.net/function.xml-parse]: Unable to call handler _pkginfo_cdata_2_0() in Common.php on line 758

The list of errors ended with:

Notice: Undefined index: package in Common.php on line 1122
The following errors where found (use force option to install anyway):
missing package name
missing summary
missing description
missing license
missing version
missing release state
missing release date
missing release notes
no maintainer(s)
no files

If you look up these errors on Google, you get very few hits. Justin Patrin correctly identifies the problem: Red Hat ES 3 ships with version 1.1. This version of pear is considered ancient and doesn't support reading the latest packages.

You can check your version of pear with this command:

pear -V

Justin recommends the standard command for upgrading pear:

pear upgrade pear

Unfortunately, this command doesn't work when your current version of pear is so far out of date. In fact, the net result of this command is that the "pear" command is removed and not replaced. If you follow the link, you'll see a bug about this in the pear bug database, where the problem is considered "Bogus" (not a bug) because the author of the bug is running a "museum PEAR version (RHE3, pear 1.1)." Unfortunately, there is no discussion of how to actually perform the upgrade, since the command "pear upgrade pear" is completely broken in RHE3. I find it absurd that anyone who supports enterprise software would think that it's okay to simply orphan customers who are running software four years old.

I found several articles on resolving problems with old versions of pear. All of them recommended reinstalling pear from scratch, similar to this:

lynx -source http://go-pear.org/ php

Unfortunately, this didn't work for me. The Red Hat Package Manager (rpm) has very definite ideas as to where pear should be installed. However, rpm bundles pear with php, so you cannot simply uninstall Red Hat's pear and then install the standard version. In my case, I was unable to determine how to get go-pear to align its directory structure with rpm, and I ended up with a schizophrenic installation.

I removed all of /usr/share/pear and restored the directory from backup. I then double checked the installed packages:

pear list

Pear reported that no packages were installed. Since the Red Hat version of pear actually ships with several packages, it was clear that my installation was now completely broken.

After several hours of searching, I finally found on article by Patrick Berry titled Upgrading PEAR on RHEL 4. Although I was running RHEL 3 and not 4, the solution described gave me enough information to solve the problem.

The problem is that version 1.1 of pear cannot read the package to install the latest version, which is 1.67 as of this blog entry. Therefore, you have to go through some intermediate upgrades before eventually arriving at version 1.67.

I first tried following the steps under Patrick's Solution section. It didn't work for me - I received the error pear is not installed. This error appears to have been caused by my earlier attempt to install from go-pear. After much frustration, I solved this problem by removing the file /root/.pearrc.

However, Patrick's steps under Solution still didn't work for me. If I had bothered to read the Update AND the Comments, I would have seen the latest correct upgrade procedure, as described by Daniel a mere ten days ago:

pear upgrade pear-1.3.3
pear upgrade pear-1.4.11
pear upgrade

However, AGAIN this procedure didn't work for me. (Are you sensing a trend here?) When I tried to go from 1.3.3 to 1.4.11, I received an error about "getopt2 not found." It's not clear what caused this problem, but I saw a similar issue reported by others, again with no solution.

To resolve this problem, I had to manually download an interim version of Console_Getopt and copy it to /usr/share/pear/Console. The latest version was too new and version 1.21 was too old. One of the versions in the middle included getopt2 and didn't rely on any other unsupported functionality.

Once I put in place a newer version of Console_Getopt, I was able to upgrade to pear 1.4.11, and then to the latest version, with these commands:

pear upgrade pear-1.4.11
pear upgrade-all

The upgrade-all command also fixes the pear package manager so it shows the correct version of Console_Getopt.

After all of this effort, my installation of pear finally worked again, after wasting a mere two days figuring out all of the problems. (Begin Sarcasm) A big thank you to both Red Hat and the "pear" package manager maintainers for abandoning users of Red Hat Enterprise 3.(End Sarcasm)

Late-Breaking Information:
I just learned that the PEAR home page says that support for PEAR 1.3.6 and earlier will be dropped on January 1, 2008. Since today is still four months before that date, it seems that they've jumped the gun a little bit. However, they do give the definitive steps on how to upgrade your PEAR installation:

pear upgrade --force PEAR-1.3.6 Archive_Tar-1.3.1 Console_Getopt-1.2
pear upgrade --force PEAR-1.4.11
pear upgrade PEAR

However, it's impossible to find this from any search engine because they don't list the symptoms or the affected operating systems, nor do they refer to this procedure in Bug #5906 that I mentioned earlier.

Even More Late-Breaking Information:
Turns out that my pear installation still had issues because of all of the experimentation, so I had to wipe it and start over. This time I was able to get go-pear to work by following the instructions on this web page:
http://aspn.activestate.com/ASPN/Mail/Message/pear-general/1305645