Reducing Executable Size

I’ve been trying to figure out how to reduce the size of executables in Visual Studio 2005. After I finished porting from Visual Studio 6 to Visual Studio 2005, the size of my Release executable went from 1.1 MB to 1.7 MB, a 50% increase in size. This was a problem because we have worked very hard to reduce the size of our executables.

I made several changes to the settings. The first change was to go to the Linker section of the project properties, under Optimization, and then set Optimize for Windows98 to “No (/OPT:NOWIN98)”. We don’t have enough win98 users to worry about a negligible performance hit for them. The effect of this was modest, but the change is recommended by several EXE optimization web sites I reviewed.

Next, we had custom optimization settings in VC6 because of a third party library that reacted poorly to being optimized. On the Optimization tab in the Properties for the project, I switched Optimization from “Custom” to “Minimize Size (/O1)” and also set Favor Size or Speed to “Favor Small Code (/Os)”. These two changes reduced the executable from 1.7MB to 1.25MB.

I was able to reduce the size by another 25KB by switching Inline Function Expansion from “Only __inline” to “Any suitable”. This seems counter-intuitive, but I've confirmed the result in several projects.

I had already enabled Whole Program Optimization, so it’s not clear what the net effect of that switch was. However, I discovered while debugging through the disassembly that the compiler will inline just about anything when "Inline Any suitable" and "Whole Program Optimization" are both set. In one case, the compiler determined that a function was only called from one place (presumably a determination made by Whole Program Optimization) and so inlined an entire function into another function, even though it wasn’t labeled __inline and it was a non-trivial function. This wouldn’t affect most situations, but it created very strange results when combined with our code obfuscation engine. I solved the problem by using #pragma to disable optimization for just those functions.

My final result was 1,151,023 bytes for VC6 and 1,260,544 bytes for VS2005, a net ten percent increase in size. My guess is that the increase is due to additional security checks put in by the compiler for buffer overruns.

[Revised 15-Feb-2007 - Fixed typos.]

ReadyBoost Update

---

Update 11/3/2008: The problems described in this post were resolved in Windows Vista Service Pack 1

---

I’ve done some experimenting to find the cause of my ReadyBoost problems. I found that the drive always hangs after the computer comes out of standby. After the computer wakes up, the thumb drive works for a little while and then hangs. The hang is not recoverable and prevents the computer from shutting down.

The solution I found is to select the drive in Explorer and choose Eject. It is not possible to choose Safely Remove Hardware because the device is in use. Then I pull out the thumb drive before putting the computer to sleep.

When the computer wakes up, I put the thumb drive back in. Sometimes Windows automatically starts using the drive again for ReadyBoost, but usually I have to go to the ReadyBoost tab in the Properties dialog and reenable ReadyBoost.

I've sent a technical support request to Apacer but have not heard anything back.

STLport on Visual Studio 2005

As part of my effort to port to VS2005, I updated to the latest version of STLport, 5.1.0. I've now reconsidered this and have switched to the STL built into VS2005. Here's why.

Years ago, when I initially switched from MFC collection classes to STL, I ran into one big problem - STL didn't have any debugging code to check iterators or array bounds. This caused random mysterious failures in the field in my first product that used STL when I didn't understand that certain operations invalidated iterators. I spent months working the problems out of that product. As part of that effort, I switched to STLport, which was much more robust than the STL implementation in Visual C++ 6.0 and STLport had debugging code in it to trap invalidated iterators, invalid array bounds, and other such programming errors.

The version of STL that ships with VS2005 benefits from Microsoft's security push and has had code added to check for problems like invalid array bounds. The VS2005 STL also has extensive debugging code that goes away in the release build. So my original need for STLport has ended. Even so, since my code was already thoroughly tested with STLport, I thought it best not to switch.

Today I was enduring my normal frustrations trying to debug an STLport set, which is effectively impossible to view in the debugger. I had the same problem with STL collection classes in VC6. Then I discovered that the VS2005 debugger actually understands the version of STL that ships with VS2005 and will automatically display the elements in collection classes such as set and vector. This is a huge timesaver and was enough justification for me to drop STLport.

Update 12/18/2007: A set of debugger visualizers has been published for STLport. Go to this link.

ReadyBoost Problems

I've had to disable ReadyBoost on my system. I've been using the Apacer HT202 Handy Steno 1GB drive, which at a 200x transfer rate is supposed to be one of the best drives on the market for ReadyBoost. I've run this configuration for two weeks now and have consistently run into the same problem. After using the system for several hours with dozens of windows open, the thumb drive hangs. It's not clear whether the drive itself hangs or whether it's a driver issue, but Vista is not able to recover. Explorer hangs, the DOS prompt hangs - anything that tries to access the drive locks up and can't be killed in Task Manager.

A hardware reset brings the system back to normal, except now the RAID array is unhappy because it's been interrupted and it takes two hours to validate the mirror, during which time the system is unusable. (Intel ICH8R chipset on a P5b Deluxe WiFi motherboard.)

Hopefully a solution will present itself after Vista is released to consumers.

The Windows SDK

If you want to use many of the new features in Windows Vista, you'll need to load the latest Platform SDK. Except it's no longer called the Platform SDK, it's now called the Windows SDK. You can find it at:

http://www.microsoft.com/downloads/details.aspx?familyid=7614FE22-8A64-4DFB-AA0C-DB53035F40A0&displaylang=en

This link points to the final release. Make sure you don't download one of the Release Candidates instead.

Using the latest SDK is a bit of a challenge. It's an all-or-nothing proposition because of new security tags that have been added. Once you've installed the SDK (1.2GB!!) you'll need to update Visual Studio 2005 to make it work. Go to Tools / Options / Projects and Solutions / VC++ directories and add the following:

• Add C:\Program Files\Microsoft SDKs\Windows\v6.0\bin to the list of directories for Executable files. The new entry should be above ALL of the VCInstallDir entries.
• Add C:\Program Files\Microsoft SDKs\Windows\v6.0\include to the list of directories for Include files. The new entry should be above $(VCInstallDir)PlatformSDK\include.
• Add C:\Program Files\Microsoft SDKs\Windows\v6.0\lib to the list of directories for Library files. The new entry should be above $(VCInstallDir)PlatformSDK\lib.

I also use STLport and the Boost library. I put the STLport and Boost directories at the top of the list. I then had to add #undef __in_range before including any STL files in order to make STLport build with the new SDK.

Finally, if you run into an error such as this one:

C:\Program Files\Microsoft SDKs\Windows\v6.0\Include\unknwn.idl(108) : error MIDL2025 : syntax error : expecting ] or , near "annotation"

This error is caused by using the Visual Studio 2005 MIDL compiler instead of the Windows SDK midl compiler. The Windows SDK midl compiler is thirteen months newer. The fix is to add the SDK bin directory to the list of directories as described above. Thanks to Mike Wasson for describing this fix in this article.

(Revised 12/27/2006)

Visual Studio Debugger Hangs

Ever since I first tried to run our software under Visual Studio 2003, I've had trouble with the debugger and the debuggee freezing solid. This problem cropped up again today in Visual Studio 2005. I knew the problem was something to do with the symbol server, but that was all I knew. Fortunately, the developer of VirtualDub has pinpointed the causes. You can find his discussion at:

http://www.virtualdub.org/blog/pivot/entry.php?id=118

The easiest solution to the second problem in the article is to "seed the cache." Visual Studio shows in its status bar (at the bottom) which DLL is being processed. In your application startup code you need to call LoadLibrary for each DLL that causes a hang, as shown in that status bar. This will force Visual Studio to download the symbols over the Internet while the process is in a "safe" state.

In my case, on a Windows Vista RTM system, I placed the following code in InitInstance() before any network calls were made:

LoadLibrary("SENSAPI.DLL");
LoadLibrary("SCHANNEL.DLL");
LoadLibrary("CREDSSP.DLL");
LoadLibrary("RASAPI32.DLL");

Note that these aren't all of the DLLs, just the DLLs at the top of the dependency hierarchy. The dependent DLLs will be loaded automatically. It's not a bad idea to just leave this code in your debug build because the problem will recur as soon as any of those DLLs are updated in a service pack. Make sure you add the related "FreeLibrary" calls if you do this.

Parallel Builds - Success!

I've given up on the parallel build facility in VCBUILD (the /M option, such as /M2). It is almost useless for our build scenario. Instead I've switched to GNU make from the "cygwin" project. GNU make supports the -j command line parameter to enable parallel builds. GNU make also allows me to specify dependencies at whatever granularity level I need. Currently I specify dependencies on a per-project basis (one project = one .vcproj file), but I could just as easily go one level deeper and have multiple dependencies for each project, such as Debug/Release/Demo.

My makefile looks something like this (simplified for demonstration purposes):

all: sdk gui server

sdk:
vcbuild c:/proj/sdk/sdk.vcproj

gui: core
vcbuild c:/proj/gui/gui.vcproj

server: core
vcbuild c:/proj/svr/svr.vcproj

core:
vcbuild c:/proj/core/core.vcproj

This makefile is launched by a batch file that sets two very important environment variables:

set SolutionPath=c:\proj\proj.sln
set VCBUILD_DEFAULT_OPTIONS=/M1

By setting the "SolutionPath" environment variable to the solution file, we give VCBUILD the ability to set the $(SolutionDir) variable in the build scripts, which is typically required to build successfully.

When you point VCBUILD at a vcproj file, it automatically builds all configurations in that file. This way the makefile doesn't need to have knowledge of all the possible configurations. One feature you give up by specifying the vcproj file is that VCBUILD won't automatically add dependent libraries to the link line. For Visual Studio, this isn't a loss because the feature is largely broken anyway.

Using GNU make gives me the control that I need to keep my dual core processor busy. I'm finally happy :-)