Whither SQL Server Reporting Services?

Over the past several years, I've sometimes needed to run reports on a SQL Database. Given the ease with which reports can be created in Access, I figured that there had to be some reporting capabilities built into SQL Server. After some research, I was very surprised to learn that this wasn't true.

There were two obvious alternatives: stick with the free version of Crystal Reports that comes with Visual Studio (which is limited) or buy a commercial reporting tool. The old R&R Report Writer from dBase days was inexpensive and effective. Suffice to say that reporting tools today are dramatically more expensive.

Months later I learned about something called SQL Server Reporting Services. The only problem was finding it. Have you seen how many versions of SQL Server there are out there? MSDE, Express, Workgroup, Standard, Developer, Enterprise... Then you have to sort through Add-Ons such as Analysis Services, Reporting Services and Integration Services. Then you have to figure out what combinations of add-ons can be used with which versions of SQL Server. Finally, you have to figure out what obscure file to download to enable each of the various add-ons.

And did I mention the difficulty in sorting out the licensing rights to the various versions for us in development (non-production) environments?

And, to top it all of, it'll be a cold day in hell before you find any of this in Microsoft documentation. It was actually easier to find some of the information in Wikipedia than in Microsoft's docs!

So here I am, running Visual Studio 2008 on Windows Vista. Of course, VS2008 doesn't ship with SQL Server, you have to install SQL Server Express 2005. No problem there, but SQL Server Express 2005 doesn't ship with Reporting Services and you won't find any download labeled Reporting Services.

I thought I had struck gold when I found an article titled Introducing Business Intelligence Development Studio. That article says that Visual Studio 2005 has specific project types for creating reports and that you just need to select Add New Project. Fantastic! Of course, when I looked in VS2005 there was no such project type and the article had no troubleshooting tips.

Are you detecting a trend here?

I found a post where numerous people attempted to figure out how to get through this morass. The final conclusion was that you had to uninstall the Workstation components and install a new copy from a fresh download of the "SQL Express Toolkit." Of course, the person didn't post the link to that download and it isn't on MSDN Downloads.

I finally found it under the name Microsoft SQL Server 2005 Express Edition Toolkit. When I tried to install it on Vista I got a "compatibility warning." It remains to be seen what problems that causes. (Note that the Business Intelligence components under Workstation are not selected by default, so you have to make sure you check them manually.)

At this point I opened VS2008, tried to add a new project, and discovered that Business Intelligence Project was not one of the choices. (And yes, I did restart VS2008.) More digging on the web uncovered this little gem: Visual Studio 2008 Will NOT Support SQL Server 2005 Reporting Services projects.

So I started VS2005 and saw that Business Intelligence Projects was now visible at the top of the list in the Add New Project dialog in VS2005. Under that project type were two entries for Report Server Project. I also found the SQL Server Business Intelligence Development Studio (BIDS) in my Start menu under SQL Server 2005. I'm not yet sure exactly what the difference is between BIDS and the VS2005 projects.

If you need to install something other than Reporting Services, the instructions above won't help you much. After Setup is complete, you'll see this message:

The version of Business Intelligence Development Tools that is included in SQL Server 2005 Express Edition Toolkit does not include projects for SQL Server 2005 Integration Services or SQL Server 2005 Analysis Services. These projects are available only with editions of SQL Server 2005 that include Integration Services and Analysis Services. SQL Server 2005 Express Edition does not include Integration Services or Analysis Services.

You'll notice that this message does not tell you which versions of SQL Server will do what you need.

So I've gotten further than I was, but I still have several questions to resolve:

1. What is the relationship between BIDS and VS2005?
2. How do I install Analysis Services?
3. How do I install the server-side components of Reporting Services?
4. How will all of this change when SQL Server 2008 is released?

Maybe I should just use Microsoft Visual FoxPro, which is awful, but at least R&R Report Writer still supports it.

[Update 1/9/2008] Here are some initial answers to my questions:

1. BIDS runs in Visual Studio 2005. It's a "personality", just like C++ versus C#.
2. TBD
3. You have to install SQL Server 2005 Express Edition with Advanced Services SP2. In addition, when you do that installation, you have to specifically enable the features for Reporting Services. My recommendation is to enable everything. Don't forget to enable the subprojects. Make sure you install this BEFORE you install Visual Studio 2005 or 2008.
4. TBD

WinSxS Breaks Old Libraries

After my previous experiences with handling Windows Side-by-Side Assemblies (WinSxS) in remote debugging and Isolated COM, I thought I was actually starting to get a handle on how it worked. Today I got stuck on another WinSxS problem, this time while porting our application to Visual Studio 2008.

The problem was that the application would fail to fail to start, with an error about the manifest being wrong. I used sxstrace, a handy tool under Vista, to try and determine what was happening. Sxstrace generated 180 lines of information about Vista's attempt to find and load the correct assemblies. It ended up being too much information. The only obvious problem I saw was that one of the DLLs being loaded was from Visual Studio 2005, not 2008.

I used Depends to look at the file and received the same errors. I looked in the Event Viewer and saw this error:

Activation context generation failed for "s:\csi\xsales\debug\XSALES.EXE". Dependent Assembly Microsoft.VC90.DebugCRT, processorArchitecture="x86", publicKeyToken="1fc8b3b9a1e18e3b", type="win32",version="9.0.20706.1" could not be found. Please use sxstrace.exe for detailed diagnosis.

This is even stranger, because 20706 was the version ID of Visual Studio 2008 Beta 2. I'm running the final release.

I used Visual Studio to open XSales.exe in Resource mode so I could look at the manifest itself. Here I found references to four versions of DebugCRT. The question was, where were they coming from? My code makes no explicit reference to assembly versions.

What I discovered was that the intermediate manifests generated by the compiler and linker include assembly information from all objects used by the linker, including objects from libraries, of which I had two. One of my .LIB files was generated by VS2005 and another was generated by VS2008 Beta 2, which is what caused the references to old assembly versions. Once I rebuilt those .LIB files with VS2008, the problem went away.

The lesson learned from all of this is that .LIB files are no longer easily portable between versions if they rely on any of the CRT or MFC DLLs. The painful part of this is that the problem doesn't show up until you try and run the software because none of the development tools warn about the inconsistency.

[Update 11/19/2008] You can find another solution here.

Corrupt \Boot\BCD in Windows Vista

Two nights ago, I started installing a video driver upgrade and went home for the night. The next morning I came in to discover that something had failed in the upgrade and the system was hung on a black screen. Even Remote Desktop access wasn't working.

I hit the Reset button. The system started to reboot. As per usual when I use the Reset button, the RAID array was incensed that I had interrupted its normal operation and took its revenge by taking four hours to validate the mirror. 89% of the way through validating the mirror, I received my first Vista Blue Screen of Death (BSOD). Something about a power driver not handling an event.

No problem. I hit the Reset button again. What's another four hours between friends?

Except this time, the system didn't reboot. It went through the BIOS check, started to boot Vista, and presented me with this error:

File: \Boot\BCD
Status: 0xc0000034
Info: The Windows Boot Configuration Data file is missing required information

The Microsoft Knowledgebase has a helpful article on this subject, which should have worked, but didn't. The error messages during the failed repair didn't clarify the cause of the problem.

I tried booting Vista from DVD and running recovery. Rebuilding the BCD file is a standard operation, except that it didn't work. The Recovery Manager said that it couldn't save the file. Curiouser and curiouser.

I restored the BCD file from backup (Acronis True Image pays for itself... again.)

I rebooted the system and got the exact same error. I went back in with the Recovery Manager and discovered that the file \Boot\BCD was gone, even though I had just restored it.

The final solution was to use Acronis to restore track zero and the Master Boot Record (MBR) from my last image backup. There never was a problem with the BCD file, that was just a red herring. I'm not sure how you'd solve this problem if you didn't have an image backup. It just goes to show the usefulness of image-level backups.

Update (11-15-2007): I discovered that the Vista repair utility created a file named C:\Temp\SrtTrail.txt. This file contained the list of tests that were performed. The last test showed this:

Root cause found: 
---------------------------
No OS files found on disk.

Repair action: Partition table repair
Result: Failed. Error code =  0x3bc3
Time taken = 154722 ms

If I'd seen this file earlier, I would have had a clear indicator to repair track zero instead of stumbling on the solution accidentally.

Legacy Code from ... CP/M?!?

Remember CP/M? Unless you are over 40, my guess is probably not. This week I ran into a bug in our software that traces its roots all the way back to CP/M in the 1970s. Over thirty years later, the code still exists in the Visual Studio 2005 C Runtime Library, waiting for the next innocent victim to run afoul of it.

Here's what happened. We had reverse engineered a data file format and had shipped the first pass to customers. Our only sample of the data file was quite small, about 10 records, but it was enough to determine the file format and make it work. The ten records in the file converted cleanly.

After the product shipped, several customers reported that only 26 records were being loaded. This was obviously incorrect as most of their files had hundreds of records. Our end-of-file handling code was common with numerous other modules and worked fine. The 26 records that did convert did so correctly.

Even when we instrumented our software to give more insight into what was happening at customer sites, we found nothing. Our software converted 26 records, detected end-of-file, and exited. WTF?

When we finally found a customer willing to share his data file, we ran our software in the debugger and got exactly the same results. 26 records were converted and the software exited cleanly, with no errors.

The lightbulb didn't go off over my head until I was looking at the data file with the cygwin "cat -v" command, which shows ASCII codes 0 through 31 as control characters. This particular data file had two bytes for each record ID and record numbering started at 0. The 27th record contained ID 26 (0x1a) which showed up as ^Z. Does that ring any bells? If you ever developed for CP/M (or for MS-DOS 1.0) it should.

Thirty years ago, CP/M only tracked the number of blocks in each file, not the number of bytes. By convention, Ctrl-Z was used to denote "end of file" for text files. MS-DOS 1.0, which bore a striking resemblance to CP/M internally, followed this same convention. At the time, the C Runtime Library understood this convention and automatically generated an "end of file" condition when ^Z was encountered. Today, the VS2005 C Runtime Library still contains that code and generates the end-of-file condition even if the exact length of the file extends beyond that point.

The bug in our software was that the file had been opened in text mode instead of binary mode. Normally this is easy to detect because the records become out of sync as they are read, but by some coincidence of the data layout, the data in this file was read in perfectly up until that ^Z.

So now I have one more reason to dislike CP/M, although (admittedly) in this day and age it seems somewhat pointless to carry a grudge against a dead operating system that was designed to run off of 8" floppy disks. Old habits are hard to break.

Static Code Analysis with Visual Studio 2005

Not too long ago I received a jarring reminder that "char" in Visual C++ is signed by default. The problem was related to the ctype functions, such as isalnum() and isspace(), all of which take an integer. When you sign-extend an eight-bit multibyte character (MBCS), you end up with a negative number. The ctype functions then do a table lookup on a negative number and your program crashes. This wouldn't be so interesting, except that this bug existed in our production code for six years before the software crashed for a customer in Europe. If the customer hadn't been willing to give me sample data to reproduce the problem, I'd still be scratching my head.

Apparently I'm not the only person who has run into this issue. Visual C++ 2005 includes a command line option named "/analyze" for finding this problem and many others. I had tried /analyze during the Whidbey Beta two years ago but didn't get useful results. Now the results were much more helpful.

The "/analyze" options enables the static code analysis feature in the C++ compiler. This feature is based on prefast, a technology that came out of Microsoft Research in 2001. Prefast knows about hundreds of common programming problems. I ran it on our codebase, which after eight years is quite mature and already compiled under warning level 4 with minimal problems.

What I learned was quite interesting. Prefast knows about the ctype problem I mentioned earlier and pointed out several lines I missed when I tried to fix the problem. All of those lines were crashes waiting to happen.

Prefast found several places where I expected one return type but was getting another. For example, at one point I checked for an HRESULT but was actually getting a bool, which meant that the sense of my error check was inverted.

Another error that prefast found was where I was calling sizeof on a variable that was defined as pointer instead of as an array, which meant that the length being handed to strncmp was wildly wrong. The code still worked, but that was a happy accident.

Prefast also had numerous warnings about the Boost C++ Library. It should be possible to suppress those warnings, but I haven't done so yet.

Prefast supports annotating your code with the Standard Annotation Language (SAL) to better describe the static behavior of the code. If you've looked at recent versions of the Windows SDK, there are numerous annotations such as __out_ecount(). These annotations provide additional information to prefast that allows for better analysis. You can learn more about these annotations at blogs.msdn.com.

Prefast also pointed out two problems with my code that I would never have found, even with a close code inspection. I'm sure you've seen stricmp, the case-insensitive version of strcmp. I was using stricmp to check for keywords. Turns out that this is a bad idea. Prefast gives warning C6400, which explains that some languages interpret combinations of letters as a single letter, which changes the behavior of stricmp. The correct solution is to use the Windows API call CompareString(), which should be set to LOCALE_INVARIANT or LANG_ENGLISH, depending on the version of Windows.

Prefast also warned me that using _alloca in a loop could cause a stack overflow. Normally I'd consider this obvious, but in this case the warning was being given about the W2A Unicode to MBCS conversion macro, which uses _alloca. I looked at the source code to W2A (in atlconv.h) and the problem seems to be handled properly, but it was a worthwhile exercise.

I'm adding prefast to my bag of recommended tricks. For more insight into the use of this tool, I invite you to read Scalable Defect Detection from the Center for Software Excellence at Microsoft.

Visual Studio 2008 Beta 2 Test Results

Today I tried building our product with Visual Studio 2008 Beta 2. The last two times I tested compiler upgrades (VC6 to VS.net 2003, then VS.net 2003 to VS2005), the upgrades were extremely painful and took days to complete.

I was pleasantly surprised that rebuilding our product in VS2008 Beta 2 went quite smoothly. The issues were minor:

• The switch /OPT:NOWIN98 option is no longer supported.
  Impact: None.
  
• Project could not be linked against a library originally built in VC6.
  Impact: Rebuild library.
  
• Visual Studio 2005 manifest that set requireAdministrator caused this error: manifest authoring error c1010001: Values of attribute "level" not equal in different manifest snippets.
  Impact: Removed the explicit manifest file and set the UAC level in the Linker node on the project properties.
  
• MT.EXE gives an error on a valid manifest.
  Impact: Move $(WindowsSdkDir)\bin to the top of the list for Executables files in VC++ Directories under Tools/Options.

In terms of the size of the generated executables, here are the stats:

Product	VS2005 size	VS2008 Size	Net Change
#1 (C++/MFC GUI)	1,198,592	1,180,672	-1.5%
#2 (C++/MFC GUI)	1,589,760	1,549,312	-2.5%
#3 (C++/MFC Console)	2,043,904	1,998,848	-2.2%

While not earth shattering, these numbers do show that they were able to contain bloat in MFC and the C runtime library. All of these applications are statically linked against MFC and the CRT.

The biggest surprise was that VC2008 worked fine with version 1.34.1 of the Boost library, which has not yet been tested on VC2008. (although, admittedly, our software only uses one or two of the Boost modules.)

That's it! So far, backwards compatibility is excellent.

Initial testing has shown no problems with the generated code. Common dialogs under Vista were automatically updated to Vista styling, as promised.

My only disappointment is that very little was done for the IDE for C++ developers in this version of Visual Studio. The tools for editing dialogs and other resources are still awful compared to VC6. For a look at the future of the IDE, take a look at Somasegar's blog. Attention is being renewed on the IDE for native C++ developers, but improvements won't ship until 2010.

Running the Visual Studio 2008 Beta 2 VHD on VMware Server

Microsoft has finally climbed on the bandwagon for using pre-configured virtual machines to distribute beta software. For anyone who tried to install earlier betas of Visual Studio 2005 Team System, you'll understand me when I say that these pre-built virtual machines will save you days of frustration.

I downloaded the VHD disk images for Visual Studio 2008 Beta 2. Although they work fine on the free download of Virtual PC 2007, I really wanted to run this image on my virtual machine server, which uses VMware Server (see my earlier comments on VMware Server versus Microsoft Virtual Server).

Before I describe the procedure, one BIG caveat: Once Windows is running in VMware, Windows will complain that it needs to be reactivated. If you are a Microsoft Partner you can get a key from MSDN Downloads. Otherwise you will need to use a new key, which basically means you need to buy Windows Server 2003 Enterprise. Therefore, if you don't have a ready supply of activation keys, this procedure won't work for you. It may be possible to call the activation people and have them honor the key built into the virtual machine, but I haven't tried.

Converting the Orcas VHD to VMware ended up being a lot more difficult than I'd hoped. The biggest problem was converting the virtual drives from .vhd format to .vmdk format. I found a nice utility named WinImage that could do this. WinImage converted the base Orcas image (2.8GB) without difficulty, but gave up with no error when I tried to convert the 11.8GB differencing disk. I didn't really want to go back and forth with the WinImage support for two days, so I looked for an alternative.

My final solution was to use Acronis to do a backup in Virtual PC, then use Acronis again to do a restore in VMware. To do this yourself, you'll need:

VMware Server (free)
Acronis TrueImage Home or better (commercial)
A Windows Server 2003 Enterprise installation CD
WinImage (shareware)

The solution was as follows:
1. Install Acronis TrueImage on any Windows box and create a Rescue CD.
2. Create a virtual machine in Virtual PC 2007 that contains Orcas Beta 2.
3. Set the virtual machine to connect to the CD you created in Step #1.
4. Boot the version machine, select Acronis and back up the Orcas virtual machine to any desired network drive.
5. Use WinImage to convert the Base01 image to VMDK. Make sure you create a dynamic disk and not a fixed disk.
6. In VMware Server, create a virtual machine that points at the file from #5.
7. Put the Acronis Rescue CD in a CD drive on that computer.
8. Start the virtual machine in VMware, press Esc, and boot from the Rescue CD.
9. Restore the Acronis backup to the current disk.
10. After restore completes, reboot the virtual machine. You'll get an error about a service that didn't start. Ignore it.
11. On the VM menu, select Send Ctrl-Alt-Del.
12. Enter the password from the Microsoft web page. You'll need to use the keyboard, your mouse probably won't work.
13. As the login completes, you'll be prompted for the path to install the Ethernet card. Put in the Windows Server 2003 Enterprise installation CD and click OK. Windows will complain that it can't find the driver, which is okay. Tell Windows you want to install from an alternative location.
14. Select the file Driver.cab on the Windows Server 2003 Enterprise installation CD. The network driver should be located automatically.
15. After installation of the initial driver completes, do not reboot or you won't be able to login again.
16. On the VM menu in the VMware Console, choose Install VMware tools.
17 At some point you'll be prompted for the file mouclass.sys. Tell the installer to use the copy of the file in C:\Windows\System32\Drivers.
18. Now you can reboot and everything should work.

Upgrading PEAR on Red Hat Enterprise ES 3

Recently we've upgraded our coding standards for our server-side PHP development to include unit tests, which required that we install PHPUnit for PHP 4. Normally installing a new pear module takes about ten seconds, like this:

pear install PHPUnit

Unfortunately, this time things didn't go so smoothly. I ended up with dozens of errors such as this one:

Warning: xml_parse() [http://www.php.net/function.xml-parse]: Unable to call handler _pkginfo_cdata_2_0() in Common.php on line 758

The list of errors ended with:

Notice: Undefined index: package in Common.php on line 1122
The following errors where found (use force option to install anyway):
missing package name
missing summary
missing description
missing license
missing version
missing release state
missing release date
missing release notes
no maintainer(s)
no files

If you look up these errors on Google, you get very few hits. Justin Patrin correctly identifies the problem: Red Hat ES 3 ships with version 1.1. This version of pear is considered ancient and doesn't support reading the latest packages.

You can check your version of pear with this command:

pear -V

Justin recommends the standard command for upgrading pear:

pear upgrade pear

Unfortunately, this command doesn't work when your current version of pear is so far out of date. In fact, the net result of this command is that the "pear" command is removed and not replaced. If you follow the link, you'll see a bug about this in the pear bug database, where the problem is considered "Bogus" (not a bug) because the author of the bug is running a "museum PEAR version (RHE3, pear 1.1)." Unfortunately, there is no discussion of how to actually perform the upgrade, since the command "pear upgrade pear" is completely broken in RHE3. I find it absurd that anyone who supports enterprise software would think that it's okay to simply orphan customers who are running software four years old.

I found several articles on resolving problems with old versions of pear. All of them recommended reinstalling pear from scratch, similar to this:

lynx -source http://go-pear.org/ php

Unfortunately, this didn't work for me. The Red Hat Package Manager (rpm) has very definite ideas as to where pear should be installed. However, rpm bundles pear with php, so you cannot simply uninstall Red Hat's pear and then install the standard version. In my case, I was unable to determine how to get go-pear to align its directory structure with rpm, and I ended up with a schizophrenic installation.

I removed all of /usr/share/pear and restored the directory from backup. I then double checked the installed packages:

pear list

Pear reported that no packages were installed. Since the Red Hat version of pear actually ships with several packages, it was clear that my installation was now completely broken.

After several hours of searching, I finally found on article by Patrick Berry titled Upgrading PEAR on RHEL 4. Although I was running RHEL 3 and not 4, the solution described gave me enough information to solve the problem.

The problem is that version 1.1 of pear cannot read the package to install the latest version, which is 1.67 as of this blog entry. Therefore, you have to go through some intermediate upgrades before eventually arriving at version 1.67.

I first tried following the steps under Patrick's Solution section. It didn't work for me - I received the error pear is not installed. This error appears to have been caused by my earlier attempt to install from go-pear. After much frustration, I solved this problem by removing the file /root/.pearrc.

However, Patrick's steps under Solution still didn't work for me. If I had bothered to read the Update AND the Comments, I would have seen the latest correct upgrade procedure, as described by Daniel a mere ten days ago:

pear upgrade pear-1.3.3
pear upgrade pear-1.4.11
pear upgrade

However, AGAIN this procedure didn't work for me. (Are you sensing a trend here?) When I tried to go from 1.3.3 to 1.4.11, I received an error about "getopt2 not found." It's not clear what caused this problem, but I saw a similar issue reported by others, again with no solution.

To resolve this problem, I had to manually download an interim version of Console_Getopt and copy it to /usr/share/pear/Console. The latest version was too new and version 1.21 was too old. One of the versions in the middle included getopt2 and didn't rely on any other unsupported functionality.

Once I put in place a newer version of Console_Getopt, I was able to upgrade to pear 1.4.11, and then to the latest version, with these commands:

pear upgrade pear-1.4.11
pear upgrade-all

The upgrade-all command also fixes the pear package manager so it shows the correct version of Console_Getopt.

After all of this effort, my installation of pear finally worked again, after wasting a mere two days figuring out all of the problems. (Begin Sarcasm) A big thank you to both Red Hat and the "pear" package manager maintainers for abandoning users of Red Hat Enterprise 3.(End Sarcasm)

Late-Breaking Information:
I just learned that the PEAR home page says that support for PEAR 1.3.6 and earlier will be dropped on January 1, 2008. Since today is still four months before that date, it seems that they've jumped the gun a little bit. However, they do give the definitive steps on how to upgrade your PEAR installation:

pear upgrade --force PEAR-1.3.6 Archive_Tar-1.3.1 Console_Getopt-1.2
pear upgrade --force PEAR-1.4.11
pear upgrade PEAR

However, it's impossible to find this from any search engine because they don't list the symptoms or the affected operating systems, nor do they refer to this procedure in Bug #5906 that I mentioned earlier.

Even More Late-Breaking Information:
Turns out that my pear installation still had issues because of all of the experimentation, so I had to wipe it and start over. This time I was able to get go-pear to work by following the instructions on this web page:
http://aspn.activestate.com/ASPN/Mail/Message/pear-general/1305645

How To Create 32-bit Import Libraries Without .OBJs or Source

This article is intentionally titled the same as Microsoft KB article 131313 (was Q131313) because that KB article does not give enough information for you to create a .LIB file that works with Windows system DLLs.

There are some functions in the Win32 API that have no import library. One example is the function named RemoveControlByName() in OCCACHE.DLL. According to the documentation, the only way to use the function is to use LoadLibrary() and GetProcAddress(). This is error-prone, requires a lot of code to maintain, and isn't usable with the Delay-Load feature of VC++.

Obviously it would be preferable to have a .LIB to link against, but creating such a library is challenging. If you use Dependency Walker to look at OCCACHE.DLL, you'll see that the function is named simply RemoveControlByName. While that seems obvious, it shouldn't be possible to have this name because it doesn't include any notation for the calling convention.

If the function were __cdecl, then the function name should have started with an underline, such as _RemoveControlByName. If the function were __stdcall, then the underline should have been added as well as suffix to indicate the number of bytes in its call stack. RemoveControlByName has five 4-byte paramters, so a __stdcall signature should have looked like _RemoveControlByName@20. However, the function name has no decorations at all, which should be impossible according to Microsoft's discussion of name decoration.

The Q131313 article discusses the general case of manually creating a .LIB file for a .DLL file. The discussion under Creating a .DEF File says "The reason for this limitation is based on an assumption made by the LIB utility that all names are automatically exported without a leading underscore." This seems promising because we don't have a leading underscore. However, functions exported from Windows DLLs(almost) always using __stdcall, and the discussion under Creating a .DEF File is only applicable to __cdecl.

In spite of those warnings, I spent quite a bit of time trying to craft a .DEF file that described what I was trying to do. (You can use LIB.EXE to compile a .DEF file to a .LIB without any .OBJ files) Although most .DEF files simply list the undecorated function names under the EXPORTS section, the syntax of the .DEF file allows for creating aliases and other strange constructs. Some of the things I tried included:

; Raw function name
RemoveControlByName

; Alias the undecorated name to __stdcall
RemoveControlByName=RemoveControlByName@20

; Explicit reference to the necessary DLL
RemoveControlByName@20 = OCCACHE.RemoveControlByName

However, none of these generated a .LIB that worked. The first two I couldn't link against and the third would link but fail to run.

I thought I'd learn something by looking at the symbol that the parent program is trying to link against, but that was even worse:

__imp_?RemoveControlByName@20

I knew my header file was correct, so that was definitely the correct name. However, you'll notice the leading "__imp", which indicates that the symbol is being imported from a DLL. This was something else that apparently needed to be included in my hand-crafted .LIB file, and I hadn't seen any discussion anywhere on how to do that.

I've tried to solve this problem two other times in the last several years, and this was the point I gave up in both of those cases. However, this time, failure was not an option. I needed the solution.

I tried the second option in the KB article, described under Stubbing Out Functions. I painstakingly created a dozen functions that mimicked the signatures of the functions listed in the header file. If you are doing this for yourself, here's a tip for creating a Visual Studio project. The final file you need is a .LIB file, so it's tempting to use one of the Visual Studio projects for creating a .LIB. However, that's wrong. If you created a .LIB, then you'll link with the stub functions you created, which is useless. What you really want is a DLL project, which happens to create a .LIB as a by-product.

Anyway, I created the functions as described in the KB article. Since my project used .cpp files, all of the calls in my header file had to be declared extern "C". For example:

extern "C" {
#include "occache.h"
};

If you look in any of the standard Windows include files, such as winbase.h, you'll see this same declaration. Note that this declaration has no relation to _cdecl and therefore has no impact on the calling convention. In other words, it doesn't force all of the functions in occache.h to be called with _cdecl. What this declaration does is to modify all of the linker symbols so that they won't include the C++ name decorations, which encode all of the function's parameters into the function name.

I also updated all of the function signatures in the header file to include __declspec(dllexport).

I compiled it, linked my main application to the new .LIB file, and it linked! I thought I was done, but when I ran the application, I received the error "Entry Point Not Found: The procedure entry point _RemoveControlByName@20 could not be located in the dynamic link library OCCACHE.DLL"

I examined the .LIB with DUMPBIN. Under Public Symbols, I now see two definitions for RemoveControlByName:

_RemoveControlByName@20
__imp__RemoveControlByName@20

It appears that the "__imp" definition was a result of adding __declspec(dllexport), so that explained why the application linked successfully. One problem solved.

Continuing my examination of the DUMPBIN information, I saw that RemoveControlByName was defined as:

Archive member name at FE0: OCCACHE.DLL/
46D12949 time/date Sun Aug 26 00:18:33 2007
uid
gid
0 mode
38 size
correct header end

Version : 0
Machine : 14C (x86)
TimeDateStamp: 46D12949 Sun Aug 26 00:18:33 2007
SizeOfData : 00000024
DLL name : OCCACHE.DLL
Symbol name : _RemoveControlByName@20
Type : code
Name type : name
Hint : 9
Name : _RemoveControlByName@20

To determine whether or not this was correct, I used DUMPBIN to compare against known-good definitions in USER32.LIB, where I found that the "Name type" in the USER32 records was defined as "undecorate" instead of "name". Obviously, there was a magic incantation to set this flag, presumably in the .DEF file.

I added a .DEF file and spent several hours trying various alias combinations, none of which worked. Finally, in desperation, I created a .DEF file that contained just the raw function names. For example:

EXPORTS
RemoveControlByName

I built my application, it linked, and it ran. What happened?

I ran DUMPBIN again on my library. The record describing RemoveControlByName now contained the "undecorate" attribute:

Archive member name at FFC: OCCACHE.DLL/
46D12740 time/date Sun Aug 26 00:09:52 2007
uid
gid
0 mode
38 size
correct header end

Version : 0
Machine : 14C (x86)
TimeDateStamp: 46D12740 Sun Aug 26 00:09:52 2007
SizeOfData : 00000024
DLL name : OCCACHE.DLL
Symbol name : _RemoveControlByName@20
Type : code
Name type : undecorate
Hint : 6
Name : RemoveControlByName

Also, the very last line in the record showed that the "Name" was RemoveControlByName, with no decoration. Exactly what I needed.

It's clear that there's quite a bit of undocumented behavior here. Adding the entry to the .DEF file had a rather dramatic effect on the generated library. I couldn't find any mention of this behavior in the documentation on .DEF files. The only relevant reference I could find was in Microsoft's documentation under __cdecl, where it says "Underscore character (_) is prefixed to names, except when exporting __cdecl functions that use C linkage." This statement is true, but it's not the whole truth. To create a .LIB file that can link against such a construct, you also must have the function declared in a .DEF file.

In summary, to create a .LIB file that will let you link with Windows system DLLs, you need to:

1. Follow the instructions in Q131313 under Stubbing Out Functions. Make sure you name the project the same as the Windows DLL.
2. Make sure your dummy functions are defined with __declspec(dllexport) as well as __stdcall.
3. For the header file used by the parent application, make sure that your function declarations are surrounded with extern "C".
4. Add a .DEF file to your project that includes the function names with no decoration.

Media Player Slows Network in Vista

If you've read my earlier blog posts about Gigabit Ethernet, you know that I've had my share of difficulty getting good performance out of my GigE network. I've also posted that I've had problems with Vista that I didn't see in Windows XP or Windows Server 2003. Now I know why. Windows Media Player puts a big throttle on GigE network performance, even if it is paused and not playing anything.

The issue was reported at 2CPU.com and a response from Microsoft was reported at ZDNet. The problem is minor on 10/100 Ethernet, but on a GigE Ethernet the network performance can be throttled back to 100Mps levels. Apparently, there's no registry setting that will resolve the problem, the only solution is to shut down Windows Media Player. There are scattered reports that the problem happens with other media players, such as WinAmp, but I haven't confirmed these reports.

There's another problem I also ran into that caused GigE to throttle back to 100Mbps. This was a self-inflicted problem, but it took several weeks to resolve. I have a LinkSys WRT54G router that handles my gateway/firewall/NAT. All of my servers get their IP address with DHCP and the WRT54G is set to always hand out the same IP address to those servers. This was done so that the servers would get the DHCP settings from our provider. Anyway, one of the servers was migrated to a new motherboard with a different MAC address. The WRT54G started handing out a random DHCP address to that node but still reported that node name as having the other, assigned IP address, all of which made the network become schizophrenic over the IP address assigned to that particular node name. The result was that any traffic that was destined for that node ended up going to the gateway, which didn't have a GigE connection, so traffic was throttled back to 100Mbps speeds.

Update 8/28/2007 - Mark Russinovich has posted a detailed analysis of the network slowdown. It turns out that the more Network Interface Cards (NICs) you have in your system, the worse the problem gets. I have three NICs, including WiFi, which slows my performance to a theoretical maximum of 9MB/second.

ReadyBoost Damages USB Thumb Drives

Windows Vista ReadyBoost destroyed a second thumb drive yesterday. For the past three months I've just been leaving my computer on at night instead of trying to make ReadyBoost work with Sleep/Standby. Yesterday I was trying something new and I put the system in Sleep mode. I woke it up two hours later and the thumb drive had stopped working. I unplugged the thumb drive and replugged it and I got a "write protect" error. I tried the thumb drive on another computer and got the same result.

I downloaded the Format utility from the Apacer web site. I successfully reformatted the drive, enabled ReadyBoost, and was working again. For a few minutes. Then the light on the thumb drive turned on steady and the drive completely died. I tried it on three different computers and three different versions of Windows.

This is the second thumb drive that's failed when using ReadyBoost and resuming from Sleep. I've heard reports from readers of this blog that they have had a similar problem. How about you? Leave a comment.

Note that I don't believe that this has anything to do with the number of write cycles, as some journalists have complained about. In both cases the drive died immediately after resuming from Sleep.

Dell PowerConnect 2708 Performance

Just a quick note to say that the Dell PowerConnect 2708 is fantastic. I was able to get a sustained 112MB per second (that's 112 megabytes, not megabits) between two Core 2 Duo machines (Vista/Windows 2003 Server R2) using PCATTCP. That means that the connection is completely saturated and that the Dell box is switching at the maximum possible speed. Sweet.

In more real world numbers, I was able to get sustained 2.2GB/minute transfer rate using normal Windows file sharing copying from Vista to the server when the file copy was initiated on the Windows 2003 Server side. That's close to the maximum read speed for the source disk, so presumably the disk was the bottleneck and not the network.

Oddly enough, when the file copy was initiated on the Vista side, I only got about 1.2GB/minute with the same file being copied to the same destination. I can't explain that.

[Edited 8/27/2007 to add information on the read speed of the disk.]

Logo Certification - Success!

Last night we received word from Lionbridge that we had passed our Windows Vista Logo Certification test. Thus I've crossed the finish line of a journey that started in May of 2006, bringing to close a year-long project to make it all happen. It was a monumental effort to bring all of the pieces together because all of the vendors we rely on had to make their products Vista compliant before ours could pass certification.

There were several "firsts" to this project. I created an MSI installer for the first time. I installed a 64-bit version of Windows for the first time. We ported our code base to VC2005. I worked with Microsoft Consulting Services for the first time. And, not surprisingly, I did my first serious development work under and for Vista. So it's been quite an adventure.

The most important of these was our MSI install generator, Advanced Installer. You may wonder why we'd go with a relatively unknown install system instead of going with one of the "big boys" like InstallShield. The reason is that, between 1991 and 1998, I had to write several installers in InstallShield and and my company paid a lot of money for the privilege of doing so. I found InstallShield to be buggy and user abusive and The Stirling Group (the manufacturer) to be unresponsive. I stopped recommending them a long time ago.

The developers at Advanced Installer worked closely with all interested customers to make sure their installers could pass Vista certification. In several cases, they gave me information that the Vista Logo Certification tests were wrong and what they were doing was correct. In every one of those cases, Microsoft admitted that the Advanced Installer developers were correct.

Isolated COM

One of the worst documented features of Visual Studio is Isolated COM. Actually, there's lots of documentation on Isolated COM, but none of it is particularly useful. The mechanics of making Isolated COM work under Visual C++ are actually pretty easy, so this short article tells you what to do.

Isolated COM allows your application to use ActiveX components without having to register them. The original vision of this was to allow XCOPY deployment of the application, but Isolated COM has many benefits. You can have a private copy of the DLL without worrying that another application will install an older or newer copy that breaks your application. Isolated COM also allows you to successfully install and run on non-Administrator accounts.

A short history of how activation works: In Windows 9x and NT, ActiveX components were found by looking in HKEY_CLASSES_ROOT, which was really a link to HKEY_LOCAL_MACHINE\Software\Classes. Windows 2000 added support for also defining components in HKEY_CURRENT_USER\Software\Classes and HKEY_CLASSES_ROOT became a merged view of HKLM and HKCU. Finally, Windows XP and Windows Vista added support for looking for COM information in the manifest of the executable. This means that the manifest contains the DLL name, CLSID information, interface information, and type library pointers - most of the information that would normally appear in HKCR.

As you might guess, these definitions can be quite lengthy and creating them by hand is error prone. Visual Studio 2005, which supports manifests, includes support for automatically generating and embedding the correct XML declarations.

To switch from "old fashioned" ActiveX usage to Isolated COM, follow these steps:

1. First, make sure components are created with their CLSID and not the ProgId. In other words, don't use Word.Document, use {F4754C9B-64F5-4B40-8AF4-679732AC0607}. If you are using the #import command in Visual C++, you can use the __uuidof operator instead of a string. For example:

HRESULT hr = pDoc.CreateInstance("Word.Document");
HRESULT hr = pDoc.CreateInstance(__uuidof(Word::Document));

2. If you made any changes for #1, test your code with the DLL registered the normal way and make sure it works. Also make sure that your code checks and handles all HRESULT return values.

3. Unregister the ActiveX DLL using: regsvr32 /u waycool.dll

4. Copy the DLL to be in the same directory as your executable.

5. In Visual Studio 2005:

• Open the property sheet for the project.
• Click the plus sign next to "Manifest Tool"
• Click on "Isolated COM".
• Next to "Type Library File" enter "waycool.tlb"
• Next to "Component File Name" enter "waycool.dll"
• Build the project.

6. Run your application in the debugger and single step through all of the activation code to make sure it is working properly.

Once you've added support for Isolated COM, don't forget to update your installer so that ActiveX components aren't automatically registered under Windows XP and Vista. Also don't forget to retest your software under Win9x and Win2K if you support those versions of Windows.

Netgear GS108 GigE Switch Failure

I've had an unbelievably long run of hardware failures this year. First a Hitachi 500GB hard drive failed in my Windows 2003 Server, then the Apacer USB thumb drive I was using for ReadyBoost, and now my new GigE Ethernet switch, a Netgear GS108, is malfunctioning. Predictably, all of the failures have been after the 30 day mark, which means that I've had to go through an RMA process instead of a simple return.

The Netgear GS108 is particularly frustrating because it didn't do something simple like emit smoke and die. That would have been easy to diagnose. Instead it would alternate between a blinking failure mode and normal operation. In twenty years I've never seen an Ethernet switch or hub fail, so I wasted two days looking at everything except the switch.

The symptom is easy to see - all of the lights turn on for a second, then they all turn off, even on ports with nothing plugged in. Any PCs that are connected report that a network cable is unplugged. This may go on for a few seconds to a couple of minutes, then it will start working again, then it will fail again. The problem seemed to be most prevalent when 1000Mbps devices were mixed with 100Mbps devices. If all devices are 1000 or all devices are 100, then the GS108 worked fine.

Reviews on newegg.com show that a minority of other users have experienced the same problem I did.

I talked to Technical Support at Netgear. They immediately issued an RMA for the switch. Unfortunately, after waiting a week for the new GS108 to arrive, the new unit malfunctioned in the same way as the old unit. So I believe this problem is endemic to the GS108.

I've replaced the GS108 with a Dell PowerConnect 2708. It costs about 50% more than the GS108, but it is a fully managed switch that receives rave reviews from users. The Dell 2708 also includes a one year next-day parts warranty, as opposed to Netgear who charges you $50 for a next day exchange. You might as well order a new one from NewEgg.

I plugged in the Dell 2708 and it has worked flawlessly. I'm just running it in unmanaged mode since I have no need for VLANs or port aggregation.

The PowerConnect 2708 lists on Dell's web site for $158 if you just do a Google search, but that's the Medium/Large Business price. If you look in the Small Business section, the price is $109 with a $20 discount, or $89. Great deal:
http://www.dell.com/content/products/productdetails.aspx/pwcnt_2708?c=us&cs=04&l=en&s=bsd

Gigabit Ethernet

Backing up a 250GB hard drive over 100Mbit Ethernet takes a l..o..n..g time. At 10 megabytes per second (theoretically), that's 600 megabytes per minute, or about eight hours. Compression can cut this number down, but the backup rate is severely constrained by the speed of the link.

In contrast, backing up to an external hard drive over Firewire or USB2 yields about 30MB/second, which brings it down to a manageable 2 1/2 hours.

I decided to take the plunge to Gigabit Ethernet (GigE), something I've been wanting to do for quite a while but was waiting for prices to drop. I brought in a contractor and upgraded my eight year old Cat5 wiring to Cat6. Strictly speaking, GigE is supposed to run over Cat5 for short distances, but I was having difficulties and it wasn't worth taking the time to individually check each wire.

I've learned several things. First, the "jumbo frames" that everyone gets so excited about are useless on a general purpose network. If you turn on jumbo frames, then every device on the network must support them and be set to use the same jumbo frame size. Since devices like printers and Tivos don't support GigE, much less jumbo frames, it means that jumbo frames are a non-starter. The place that jumbo frames *can* be useful is for a dedicated connection between servers or a backbone link.

The next thing I learned is that today's network application protocols don't scale well to GigE. When you upgrade from 10Mb to 100Mb Ethernet, your performance typically jumps almost 10x. However, when you go from 100Mb to 1000Gb, your performance typically only jumps 2.5x, even if you've optimized your system configuration. Ugh. That's not what I was hoping for.

The primary optimization required for Microsoft Windows 2000 and XP is to update the receive window size in the registry. This change isn't required for Windows Vista because Vista automatically tunes its networking parameters for optimal performance.

You can see what the practical maximum speed of your link is using pcattcp.exe. You run it on a server and on a client and it reports the speed at which it was able to send data. Most importantly, you can set the write buffer size, which allows you to determine the effect of larger buffers. In my case I determined that a 64K buffer provides throughput at 90% of the link capacity.

I still have more research to do, but that's the beginning.

[Update 6/20/2007] After spending far too many hours testing various GigE configurations, I've determined that the original Cat5 wiring was probably working fine. The problems I was having were a combination of the malfunctioning GS108 and a configuration error in my router/switch.

There's a great article at Hardware Secrets that describes how Gigabit Ethernet works and why Cat5 cables are sufficient.

ReadyBoost Part IV

I finally ordered a 4GB Apacer HT203 and installed it. I also installed the Vista USB patch. The hang problem is now fixed (see my earlier blog entries about ReadyBoost.) I can browse the thumb drive after resuming from sleep without difficulty.

However, ReadyBoost still doesn't work reliably after resuming from Sleep. After the computer wakes up, ReadyBoost works for a while, then mysteriously turns off (as seen by opening the Properties for the drive and looking at the ReadyBoost tab.) If I reenable ReadyBoost, it just turns off again almost immediately.

At this point, I'm only slightly better off than I was before I applied the Vista USB fix. The computer no longer hangs, but ReadyBoost still becomes disabled after resuming from sleep. Doing a "Safe Remove" and then reinserting the thumb drive, even into another port, doesn't help.

Symbols for CRT and MFC Source Code

[12/19/2014: Updated to include Visual Studio 2013.]

I'm not sure why I keep getting stuck on the littlest things. Ever since I installed Visual Studio 2010, I haven't been able to debug into the C Runtime (CRT) or the MFC source code. This is an operation that never gave me any difficulties in years of using VC6, so it's been quite frustrating. I've made several attempts to fix the problem, but no success until today.

Reproducing the problem is straightforward. Set a breakpoint in one of your MFC message handler functions, go to the Call Stack window, and try to click on any of the functions in mfc80d.dll. Similarly, any attempt to set a breakpoint on an MFC function (such as CWinApp::CWinApp) simply fails to work.

Part 1

If you look at the Output window, you'll see a message such as this one:

'MyApp.exe': Loaded 'C:\WINDOWS\WinSxS\x86_microsoft.vc80.debugmfc_1fc8b3b9a1e18e3b_8.0.50727.762_none_29a8a38855141f6e\mfc80d.dll', Symbols loaded (source information stripped).

The key part is "source information stripped", which is strange, because the PDB files for MFC ship with Visual Studio (and the service pack) and those PDB files include source information.

I searched my system for mfc80d.pdb and I found two copies. The first copy is in the symbol cache at c:\cache\mfc80d.i386.pdb\A12C75C3E6A244E3B3BFBE577AB27642e. This file was about 2MB. The second copy of mfc80d.pdb was in c:\windows\symbols\dll. This file was 13.5MB, significantly larger and probably the version of the PDB file that I needed. The problem was making VC2005 use it. That's the hard part.

First go to Tools/Options, open Debugging, and select Symbols. Add the directory "C:\Windows\Symbols\dll" to the beginning of the list. This is different than how VC6 was configured, which required that you not include the "dll" part of the path.

Click OK. Now close Visual Studio. Go to Task Manager and make sure its gone. Now kill the process MSPDBSRV.EXE, if it's running.

Finally, the critical operation to making all of the other steps work is to remove the smaller copy of mfc80d.pdb from the cache. To do this, delete the directory c:\cache\mfc80d.i386.pdb. The path may be slightly different if you are on a 64-bit system.

At this point you can restart VC2005, load your project, and run your application. In the Modules window, you should see that the symbol file for mfc80d.dll is being loaded from C:\Windows\Symbols\dll.

Part 2

[This applies to Visual Studio 2010, 2012, and 2013.]

You may need to tell Visual Studio where the source code is installed. Right-click on a C++ solution and select Properties. Under Common Properties, select Debug Source Files. For Visual Studio 2013, it looks like this on my machine:

Problem solved.

Solution to Vista USB ReadyBoost Hang

Microsoft has released a patch for Vista that cures several problems with USB devices, including devices that hang when resuming from Sleep. Here's the solution:

Reliability update for the USB stack in Windows Vista
http://support.microsoft.com/kb/925528/en-us

Finding this patch isn't particularly easy if Google is your only search engine. The knowledgebase article doesn't mention ReadyBoost at all. It comes up as the first result if you search support.microsoft.com for vista and usb, but it's on page three if you search Google for vista usb hang.

My Apacer USB thumb drive is dead, so I can't test out the patch until I order a new one.

[Update 4/25/2007]
The Microsoft Knowledgebase article has been retitled Stop errors occur on a Windows-based computer that has 2GB or more of RAM and is using an NVIDIA nForce USB controller. My computer isn't based on the nVidia chipset, but the update still fixed my hang problem, so I'm not sure why Microsoft retitled the article. I do have 2GB of RAM.

Making VC2005 Remote Debugging Work

I use Virtual PC extensively for testing older operating systems and non-standard configurations. When I have to debug configuration-related problems, I use Remote Debugging to connect to the Virtual PC instance and debug in-situ.

Before I upgraded to VC2005, remote debugging with VC++ 6 was straightforward. I updated the PATH on the remote computer to include MSVCMON.EXE, the related debugger DLLs, and the debug versions of the C runtime library and MFC. Worked great.

Remote debugging with VC2005 is more challenging. Between security problems and Windows Side-by-Side assembles, it can take quite a few steps to make everything work. Here is what I had to do.

Your debugger and Windows firewall double-team you to make configuring remote debugging as painful as possible. My first problem was to fix the firewall, since I couldn't connect to the Windows XP SP2 computer at all. I fixed this by disabling the Windows Firewall (if you are following along, don't do this unless you are absolutely certain it's okay for your environment!) Probably overkill, but simple.

My next problem was debugger authentication. This proved to be a challenge because my Virtual PC test machine is on a domain and my development machine is not. After several hours of effort, I determined that they simply were not going to agree on a trust relationship, so I disabled authentication. (Handy tip - you can do this from the command line with /noauth. Also add /nosecuritywarn if you don't want to go crazy with security prompts.)

I thought I was golden at this point. I clicked F5 to debug my executable and was told, "Could not start App.exe". So I tried to start the application from Explorer. Same result. I ran Dependency Walker to check my DLL dependencies and was reminded that I needed to update my path to include the VC2005 debug libraries for the CRT and MFC. I did so, clicked F5 again, and... it still didn't work.

Several hours and much frustration later, I had received a solid education about Windows Side-by-Side Assemblies, otherwise known as WinSxS. In order to cure "DLL Hell," Windows XP and Windows Vista provide support for an executable's manifest to hardcode the desired version of dependent DLLs. For executables created with VC2005 that are dynamically linked to MFC and the CRT, a manifest is automatically inserted into the generated executable that includes this version information.

Once this is done, Windows XP and Windows Vista become absolutely hell bent on loading the correct version of the DLLs. Trying to work around this protection is futile. None of the following "obvious" solutions work:

• Putting mfc80d.dll in the same directory as the EXE, even if it's the correct version.
• Including mfc80d.dll on the user's path.
• Manually copying the appropriate directories from the \Windows\WinSxS on the development machine to \Windows\WinSxS on the remote debuggee.

After spending far too much time reading mostly useless documentation, I came across the correct solution. In your Visual Studio 2005 installation, you'll find a directory named:

\Program Files\Microsoft Visual Studio 8\VC\redist\Debug_NonRedist\x86

Inside this directory, you'll find two more directories named Microsoft.VC80.DebugCRT and Microsoft.VC80.DebugMFC. These directories must be copied to the same directory as your EXE and magically everything works right.

At this point you'd think the story is over, but it's not. Windows 2000, Windows NT and Win9x don't support WinSxS, so the manifest is ignored. To debug your application any of THOSE versions of Windows, you need to include the appropriate DLLs on your PATH (or copy the appopriate DLLs into the same directory as your EXE.) I set the path in Virtual PC to point directly to the WinSxS directories on my development machine, which were: ("duo" is the host name of the development computer, which drive C shared under the name "C")

\\duo\c\Windows\winsxs\x86_microsoft.vc80.debugmfc_1fc8b3b9a1e18e3b_8.0.50727.762_none_29a8a38855141f6e
\\duo\C\windows\winsxs\x86_microsoft.vc80.debugcrt_1fc8b3b9a1e18e3b_8.0.50727.762_none_24c8a196583ff03b

-
Finally, I used the following command in my batch file to start the debug monitor:

start "" "\\duo\c\Program Files\Microsoft Visual Studio 8\Common7\IDE\Remote Debugger\x86\msvsmon.exe" /noauth /anyuser /nosecuritywarn /timeout 7200

Finally, that made everything work properly.

Visual Studio 2005 SP1 Update for Windows Vista Released

A little bit later than I really needed it, but Visual Studio 2005 Service Pack 1 Update for Windows Vista has been released. Get it while it's hot!

http://www.microsoft.com/downloads/details.aspx?FamilyID=90e2942d-3ad1-4873-a2ee-4acc0aace5b6&DisplayLang=en

Virtual Server 2007 versus VMware Server

With the advent of the Core 2 Duo processors, it's finally possible to build a server for virtual machines without breaking the bank. I have three computers in my office that run various versions of Exchange Server for testing. Most of the time, these boxes are little more than space heaters, so I was looking forward to integrating everything into a single box.

I started by installing Windows Server 2003 x64, which is required for Exchange 2007. My plan was to run Exchange 2007 on the host x64 operating system and the older versions of Exchange in virtual machines.

The Microsoft solution for creating virtual servers is Microsoft Virtual Server 2005 R2. I had tried to install the software before the R2 version and I found it to be difficult to install and even more difficult to use. The documentation, while extensive, didn't have any Getting Started section that made any sense. My experience installing the R2 version was only slightly better and I spent several hours figuring out how to do things that should have been obvious.

The next step was to convert my physical servers into virtual servers (P2V). I spent almost a day researching how to do this and finally gave up. Microsoft's tools are immature and very difficult to use. The P2V conversion process relies on having a Domain Controller, Microsoft Operations Manager (the 32-bit version), and other requirements, none of which I had the time or inclination to set up. I uninstalled Virtual Server 2005 R2.

The alternative was VMware's product, VMware Server, which just recently became free. I found this software to be very easy to use. I installed it, opened the desktop icon labeled "VMware Server Console," and was able to immediately get started. This was a stark contrast to my experience with Microsoft's product. (Note that I've used Virtual PC since before Microsoft purchased it, so I have no shortage of experience with the technology that Microsoft Virtual Server is based on.)

Converting my physical servers to virtual servers was equally easy. I downloaded VMware Converter and followed the simple wizard. Once the conversion process was finished, I went back to VMware Console, added the new virtual machine, booted it, and it worked. The only caveat was that you have to power down the old physical computer before booting the virtual machine, or you end up with duplicate names. Also, if you've set your DHCP server to hand out a particular IP address to the MAC on your old server, you need to update your DHCP server.

It's also possible to run the VMware console on other computers by downloading the Client installer from the VMware downloads page. Although this works well when you are managing multiple servers, the sluggish screen update over the LAN can be annoying. I use Remote Desktop when I need to work with one of the virtual servers for any period of time.

I'm really impressed with VMware Server. It works well and has some impressive upgrades if I need better manageability than I have now.

ReadyBoost Redux

If you've followed my blog, you'll know that my experience with ReadyBoost has been painful. The most recent problem has been that the Apacer HT203 thumb drive isn't recognized as a USB 2.0 drive. Whenever I plug it in, Vista (or Windows XP on another system) gives the error that the "drive would perform better if plugged into a high speed port." Since Windows thought the device was USB 1.0, ReadyBoost wouldn't work at all. This fixed the hanging problem, but not the way I intended.

Previously, I had written Apacer to try and get help with the system hangs. Apacer technical support did finally respond. They hadn't heard of my problem before (predictably) but gave me a utility to low level format the drive, which I tried but it didn't help. (Note: it costs about $15 in postage, round trip, to send a thumb drive to Apacer for repairs. My recommendation - if the drive is 1GB or less, throw it away and buy a new one.)

Tonight I was tinkering with the drive again and tried to format the drive in Windows, just for chuckles. At which point I realized that Windows had previously formatted the drive as FAT instead of FAT32. I reformatted the device as FAT32, unplugged the device, plugged it back in, and suddenly it worked again as a USB 2.0 device!

Unfortunately the device failed again after a few hours, so I'm assuming it's a bad thumb drive and I'll replace it.

Fixing "Minimize CRT Use in ATL"

One of our products uses an ATL-based DLL as part of its installation. Under Visual Studio 6, the component was 39KB. When I rebuilt the DLL under VC2005, the size jumped to 108KB, a rather large increase for no additional functionality.

The "General" property sheet contains an item "Minimize CRT Use in ATL", which seems to be exactly what I want to solve the problem. However, getting it to work ended up being a big effort.

This problem had been discussed in several other places with no solution. Examples include the following sites. The last site concerned me the most, because Microsoft closed a bug filed against the problem saying "Won't Fix."

forums.microsoft.com
www.tutorials-ne.com
groups.google.com
connect.microsoft.com

The problem can be reproduced very easily:

1. Create an ATL DLL project using VC++ 2005 wizard
2. Switch projects to the Release configuration
3. Set "Use of ATL" to Static Link to ATL and set "Minimize CRT Use In ATL" to Yes.
4. Build the project.

You'll get the following errors:
LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already defined in atlmincrt.lib(atlinit.obj)
LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already defined in atlmincrt.lib(atlinit.obj)
LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already defined in atlmincrt.lib(atlinit.obj)
LIBCMT.lib(crt0dat.obj) : error LNK2005: __get_osplatform already defined in atlmincrt.lib(atlinit.obj)
LIBCMT.lib(crt0dat.obj) : error LNK2005: __osplatform already defined in atlmincrt.lib(atlinit.obj)
[etc.]

I first tried ignoring LIBCMT by adding it to "Ignore Specific Library" in the property sheet. This produced another set of errors, including:
atlmincrt.lib(atlloadcfg.obj) : error LNK2001: unresolved external symbol ___security_cookie
atls.lib(atlbase.obj) : error LNK2001: unresolved external symbol __except_handler4

I next tried excluding both ATLS and LIBCMT. This didn't work either. Some of the posts mentioned above tried excluding atlmincrt.lib, but since this appears to me to be the key library that makes everything work, I didn't consider excluding it to be a viable solution.

After several hours of trial and error, I determined that disabling exception handling allows the DLL to link successfully. You do this in the project properties under C/C++ | Code Generation | Enable C++ Exceptions. Set it to No.

At this point I was successful. My component was now 50KB. This was still not ideal, but much better than the 108KB I started with.

There is one important caveat to this approach. If you don't let ATL use the C run-time, then you can't use it either. Any attempt to call a function such as atoi, printf, or any other C run-time function will bring back all of the original link errors, such as error LNK2005: __encode_pointer already defined in atlmincrt.lib(atlinit.obj). Many C run-time functions have Windows equivalents, such as lstrcpy instead of strcpy, so it's possible to not use the C run-time, but it requires discipline.

If your project is generating the "__encode_pointer" link errors and you can't figure out what function you are calling that is causing the problem, then temporarily add LIBCMT to the Ignore Libraries list and build the project. Look for unresolved symbols in your object files. Ignore unresolved symbols in atls.lib, stdafx.obj, and atlmincrt.lib

Ms. Dewey Search Engine

Craziest twist on a search engine I've ever seen. Just try it.

http://www.msdewey.com/
(Requires Flash)

Update: Sadly, Ms. Dewey was shut down in early 2009. The Internet shall be a more somber place without her.

Reducing Executable Size

I’ve been trying to figure out how to reduce the size of executables in Visual Studio 2005. After I finished porting from Visual Studio 6 to Visual Studio 2005, the size of my Release executable went from 1.1 MB to 1.7 MB, a 50% increase in size. This was a problem because we have worked very hard to reduce the size of our executables.

I made several changes to the settings. The first change was to go to the Linker section of the project properties, under Optimization, and then set Optimize for Windows98 to “No (/OPT:NOWIN98)”. We don’t have enough win98 users to worry about a negligible performance hit for them. The effect of this was modest, but the change is recommended by several EXE optimization web sites I reviewed.

Next, we had custom optimization settings in VC6 because of a third party library that reacted poorly to being optimized. On the Optimization tab in the Properties for the project, I switched Optimization from “Custom” to “Minimize Size (/O1)” and also set Favor Size or Speed to “Favor Small Code (/Os)”. These two changes reduced the executable from 1.7MB to 1.25MB.

I was able to reduce the size by another 25KB by switching Inline Function Expansion from “Only __inline” to “Any suitable”. This seems counter-intuitive, but I've confirmed the result in several projects.

I had already enabled Whole Program Optimization, so it’s not clear what the net effect of that switch was. However, I discovered while debugging through the disassembly that the compiler will inline just about anything when "Inline Any suitable" and "Whole Program Optimization" are both set. In one case, the compiler determined that a function was only called from one place (presumably a determination made by Whole Program Optimization) and so inlined an entire function into another function, even though it wasn’t labeled __inline and it was a non-trivial function. This wouldn’t affect most situations, but it created very strange results when combined with our code obfuscation engine. I solved the problem by using #pragma to disable optimization for just those functions.

My final result was 1,151,023 bytes for VC6 and 1,260,544 bytes for VS2005, a net ten percent increase in size. My guess is that the increase is due to additional security checks put in by the compiler for buffer overruns.

[Revised 15-Feb-2007 - Fixed typos.]

ReadyBoost Update

---

Update 11/3/2008: The problems described in this post were resolved in Windows Vista Service Pack 1

---

I’ve done some experimenting to find the cause of my ReadyBoost problems. I found that the drive always hangs after the computer comes out of standby. After the computer wakes up, the thumb drive works for a little while and then hangs. The hang is not recoverable and prevents the computer from shutting down.

The solution I found is to select the drive in Explorer and choose Eject. It is not possible to choose Safely Remove Hardware because the device is in use. Then I pull out the thumb drive before putting the computer to sleep.

When the computer wakes up, I put the thumb drive back in. Sometimes Windows automatically starts using the drive again for ReadyBoost, but usually I have to go to the ReadyBoost tab in the Properties dialog and reenable ReadyBoost.

I've sent a technical support request to Apacer but have not heard anything back.