Technical Blog for Jim Beveridge

Windows 7 Forgets My Multiple Monitor Layout

2012-01-16T09:58:00.002-07:00

For the last several months I've had trouble where every time I rebooted, my second monitor would become the default and my primary monitor would stop working. I think this started when I upgraded drivers, but I'm not positive. I've searched and searched for a solution to this, including reversing cables, adjusting the multiple monitor settings in Windows 7, and various obscure registry entries - but nothing worked.

The really strange thing was that Windows showed the login screen on my main monitor and the problem didn't happen until after I logged in, when the secondary monitor would suddenly become my primary. This symptom provided the key to the problem. After you log in, the nVidia Control Panel overrides your Windows display settings, so you must make sure that everything is set the way you want in the nVidia Control Pane.

To solve the problem, I did this:

Opened nVidia Control Panel > Display > Set up multiple displays.
Dragged the displays to the proper location.
Checked all of the displays.
Right-clicked on the desired primary monitor.
Selected "Make this the Windows primary display."
Clicked Apply in the bottom right.

Base64 for Unicode UTF16

2011-11-29T11:32:00.001-07:00

Recently I needed to store a JPEG file in a Unicode UTF16 string. In UTF8 or ASCII, this is a trivial problem - use base64 encoding. But base64 encoding, which is 75% efficient for a single or multibyte character set, drops to 37.5% efficiency with double bytes - which means that the storage required for my JPEG data would almost triple. Since I had to store potentially thousands of images in memory simultaneously, this was a problem.

I ran across a great blog entry by Markus Scherer about a technique called Base16k. This seemed to be exactly what I needed, so I translated the JavaScript sample code into C++.

Note that this technique is designed particularly for UTF16. You will lose the efficiency if you write it to a file as UTF8. If you use UTF8, then don't use Base16k. Use base64 instead.

You can download the source code and sample project from:

Convert a Locale Name to LCID in C++

2011-10-28T15:50:00.004-07:00

I've spent most of the afternoon today figuring out how to convert a locale name, such as "en-US", into a locale identifier (LCID). Locale names are also called RFC 1766 language codes, or ISO 639 language names. As it turns out, this is easy to with the function LocaleNameToLCID(), but this function does not exist in Windows XP.

Microsoft provides a redistributable library that adds this functionality for XP, but it's 2MB and yet another module to install. Since my entire installation is 4MB, this isn't an option for me. You can read about that library in the article Mapping Locale Data on Downlevel Systems. In that article, they discuss the function DownlevelLocaleNameToLCID().

For handy reference, Microsoft provides a list of Language Identifier Constants and Strings on MSDN.

If you then need to convert the LCID to a CODEPAGEID, you can use this call to retrieve the code page for a particular LCID:

GetLocaleInfo(lcid, LOCALE_IDEFAULTCODEPAGE, ...) 

Additional information on this topic can be found on The Old New Thing blog, including a discussion of using the MLang library.

// This is a version of LocaleNameToLCID() that works on Windows XP.

// On Windows Vista or later, it automatically delegates to the

// built-in version of LocaleNameToLCID().

// This function is not case-sensitive.

// Tested with Unicode builds only, not tested under ANSI builds, but should be pretty close.

// Example value for szLocale is "en-US"

LCID CompatibleLocaleNameToLCID(LPCSTR szLocale)

CString strValue(szLocale);

CStringW wstrValue(strValue);

static  bool  bDone;

LCID (WINAPI * pfnLocaleNameToLCID)(LPCWSTR, DWORD);

if  (!bDone)

bDone = true ;

*(FARPROC*)&pfnLocaleNameToLCID = ::GetProcAddress(GetModuleHandleA("Kernel32" ), "LocaleNameToLCID" );

if  (pfnLocaleNameToLCID)

return  LocaleNameToLCID(wstrValue, 0);

CComPtr<IMultiLanguage> iLang;

HRESULT hr = iLang.CoCreateInstance(CLSID_CMultiLanguage);

LCID lcid;

if  (SUCCEEDED(hr) && SUCCEEDED(iLang->GetLcidFromRfc1766(&lcid, (BSTR)wstrValue.GetString()))) {

return  lcid;

return LOCALE_USER_DEFAULT;

Reading XML from C++

2011-10-25T14:02:00.000-07:00

Reading XML from a URL is easy with by MSXML::IXMLHttpRequest.

Sample code is available at:
http://msdn.microsoft.com/en-us/library/ms992615(v=exchg.65).aspx

However, there are some disadvantages to this interface, which does not give you access to the low level handles if you need to tweak anything. For example, some servers require you to authenticate with a certificate and IXMLHttpRequest provides no way to do this. This interface will also display errors to the user and prompt for required information, so it's not appropriate for an application that will run silently.

Therefore, Microsoft provides a second interface called IServerXMLHTTPRequesthttp://msdn.microsoft.com/en-us/library/windows/desktop/ms762278(v=VS.85).aspx. This interface inherits from IXMLHttpRequest, but it does not use WinInet and so is appropriate for use in servers. It also allows you to explicitly provide a certificate using SXH_OPTION_SELECT_CLIENT_SSL_CERT.

Google C++ Style Guide

2011-09-20T09:50:00.001-07:00

I've authored at least a couple of C++ style guides for companies over the years. Therefore, it was very interesting today for me to see Google's C++ style guide:

http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml

One point I saw that was interesting is Exceptions, where it says, "We do not use C++ exceptions." I find this interesting considering that some popular libraries (notably STL) do use exceptions - whether you like it or not. If you don't have exception handling, nasty things happen when things go wrong. I discuss one such debugging nightmare in an earlier blog post:

http://qualapps.blogspot.com/2009/11/handling-exceptions-from-stl.html

Display Unicode in a Windows console window with printf/wprintf

2011-01-19T17:25:00.003-07:00

Did you know that you can print Unicode to a console window under Windows? I didn't. Here are some blog entries that discuss how to do it. You need Visual Studio 2005 or later to display Unicode using the C run-time library. Thanks to Microsoft C++ MVPs Jochen Kalmbach and Igor Tandetnik for introducing me to these solutions:

http://blog.kalmbachnet.de/?postid=98

http://blogs.msdn.com/b/michkap/archive/2010/10/07/10072032.aspx

http://blogs.msdn.com/b/michkap/archive/2008/03/18/8306597.aspx

http://illegalargumentexception.blogspot.com/2009/04/i18n-unicode-at-windows-command-prompt.html

With the techniques in those articles, you can display both UTF-16 and UTF-8 Unicode characters in a console window.

There are some caveats:

The console window must be set to a TrueType font. By default it is set to a raster font, which does not support Unicode characters.
Not all languages are supported. Lucida Console, the most common console TrueType font, will display Russian characters, but not Arabic or Hebrew.

This article discusses the issues with Hebrew characters and provides some workarounds:
http://groups.google.com/group/microsoft.public.win32.programmer.international/browse_thread/thread/788eaf7082088b35

How to: Debug with Code Center Premium Source (In Detail)

2011-01-07T12:56:00.011-07:00

This article is an extensive update to the MSDN article of the same name. I saw that article two months ago and it was only with the help of Microsoft C++ MVP John Czopowik that I was able to make Windows Source Debugging work under Visual Studio 2010. I hope this article saves other people from the same frustration.

These instructions are for developers who have a Windows source code license and access to Code Center Premium.

To prepare for debugging with Code Center Premium

Close all copies of Visual Studio that are running (this is required or VS will overwrite your registry changes.)
Connect your SmartCard reader and insert the card you obtained from the Shared Source Initiative.
Windows Vista and Windows 7 will need to contact Windows Update to install a Gemalto driver. Make sure that you allow this to happen. (Windows XP: See the instructions that came with the card reader.)
Make sure your smart card certificate is working by opening Internet Explorer and going to https://codepremium.msdn.microsoft.com/. You should see prompts as shown in Steps 3 to 6 under Sample Walkthrough, below. This must work properly before you proceed with these instructions. Do not use Firefox because Firefox does not support smart card authentication!
Open the Registry Editor and go to: HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\10.0\Debugger
Set both SecureSourceLocalDirectory and SourceServerExtractToDirectory to the same directory. The directory does not have to already exist.
Launch Visual Studio.
On the Tools menu, click Options.
In the Options dialog box, open the Debugging node and click General.
Clear the Enable just my code (Managed only) check box.
Select Enable Enable source server support.
Select Print source server diagnostic messages to the Output window.
Clear Require source files to exactly match the original version.
Under the Debugging node, click Symbols.
In the Symbol file (.pdb) locations box, clear the Microsoft Server Symbols check box and add the following location:

https://codepremium.msdn.microsoft.com/symbols

Move this location to be second in the list to ensure that these symbols are loaded first. (You can't move it to the top. Microsoft Symbol Servers is always first, but you should have unchecked it.) Visual Studio 2005 and 2008 users: In the screen shot below, note the third entry in this list, C:\Windows\symbols\dll. This directory is where the MFC and C run-time symbols are installed by Visual Studio. These symbols are not part of Code Center, so you need to include them if you want to debug into MFC or the C run-time library. Visual Studio 2010 users: You don't need the reference to C:\Windows\symbols\dll. Visual Studio always looks there, even if it's not in this list.

In the Cache symbols in this directory box, enter a location such as C:\symbols where Code Center Premium can cache the symbols. This location should be DIFFERENT from the location used to store public symbols when Microsoft Symbol Servers is checked because symbols already downloaded from the Microsoft Symbol Servers will prevent symbols being downloaded from Code Center Premium.
Click OK.
Right-click the Solution under Solution Explorer and choose Properties:
Under Common Properties, select Debug Source Files.
Add a link for the desired operating system (complete list at the end of this article.) For example:

https://codepremium.msdn.microsoft.com/source/Windows%207/RTM

Note that spaces must be replaced with %20, and you cannot browse to this location in your browser.
In the same window, make sure that the list under Do not look for these source files is empty.
Click OK.
Close and restart Visual Studio to ensure that settings are persisted.

Sample Code

Here is sample code that I used for testing. It's a native code console application that can be tested under x86 or x64:


#include "stdafx.h"

#include <windows.h>
#include <shlwapi.h>

#pragma comment(lib, "shlwapi.lib")

int _tmain(int argc, _TCHAR* argv[])
{
  // Can't step into this function. You get a "not indexed" error.
  //char buf[256]={'\0'};
  //::PathRemoveExtensionA(buf);

  // Set a breakpoint on this function, then try to Step Into it.
  GetConsoleMode(0,0);

  // Stepping into this function should work too, but not on XP.
  //UnregisterApplicationRestart();

  return 0;
}

Sample Walkthrough

Set a breakpoint at GetConsoleMode in the sample code.
Start debugging (F5).
You should see the Confirm Certificate window. If not, your smart card isn't working.
Click OK.
You'll be asked to enter your PIN. You received this in your email confirmation that welcomed you to Code Center Premium.
Enter your PIN and click OK.
It may take a while to get started because the private symbols must be downloaded. Since they contain full debug information, they are much larger than the usual symbols.
When Visual Studio stops at the breakpoint, look at your Modules window. Here's how it appears on my Windows 7 64-bit machine: (click it to see it full size.)
Note that kernel32.dll shows "Symbols loaded", whereas KernelBase.dll shows "Symbols loaded (source information stripped). If kernel32.dll also shows "(source information stripped)", then you are using the wrong symbols. The most likely cause is if you didn't set a different cache directory for private symbols.
Also note that the timestamp for both kernel32.dll and KernelBase.dll is 7/13/2009. This means that they are RTM DLLs for Windows 7. The timestamp for ntdll.dll is 3/23/2010, which means that it's been updated and you can't step into it. So it is possible to debug on a patched system, but it's hit or miss because many DLLs have been patched and so can't be stepped into.
Now you will see a window asking to execute an untrusted command named SD.EXE:
This command will be executed by the Source Server at Code Center Premium. It will NOT be executed on your computer. You do not need to have SD.EXE on your computer.
Click Run.
You should now be looking at Windows source code.
If you look at the Output window, you should see messages such as the following: (some NDA information removed.)
SRCSRV: sd.exe -p windowsdepot.sys-ntgroup.ntdev.microsoft.com:2016 print -o "C:\Users\JimB\AppData\Local\SOURCE~2\WIN_WINDOWS\win7_rtm\windows\yyy\yyy\yyy\yyy.c\1\yyy.c" -q //depot/win7_rtm/windows/Core/xxx/xxx/xxxxx.c#1 SRCSRV: Source server cannot retrieve the source code for file 'd:\w7rtm\windows\zzz\zzz\zzz\zzz.c' in module 'C:\Windows\System32\kernel32.dll'. The system cannot find the file specified.
Note that Source Server is actually showing an error. In spite of this, debugging still works properly.

Things That Don't Work

You can't debug into 32-bit code running under a 64-bit operating system. In other words, you can't debug into any DLL that lives in \Windows\SysWOW64. (Remote debugging into a 32-bit virtual machine running under a 64-bit operating system is okay.)
You can't debug any hot fixes, security fixes or operating system updates. You can ONLY debug the RTM release or a service pack. In practice, this usually means that you are remote debugging a virtual machine that was installed from the DVD and Windows Update has been turned off.
You can't debug when running Visual Studio under Remote Desktop because the smart card doesn't work when you are logged in under Remote Desktop. (LogMeIn and GotoMyPC are not tested.)
The debugger won't step into some Windows functions. This usually means that you are trying to step into a DLL that's been updated since the original release.
Some functions are in files that Code Center Premium can't find. One example is PathRemoveExtensionA in shlwapi.dll. For files that can't be found, you'll see a "not indexed" error in your Output window:

SRCSRV: o:\w7rtm.obj.amd64fre\shell\shlwapi\srca\objfre\amd64\patha.cpp not indexed

Troubleshooting

Everything has to be exactly right for source code debugging to work properly. If things aren't working, here are some ideas:

Is your smartcard working? (Repeat Step 4 under Prepare for debugging, above.)
Did Visual Studio overwrite your registry entries? This usually leads to the error "SRCSRV: ... The system cannot find the file specified." (Repeat Step 5.)
Did the file get added to the list of Do not look for these source files? (See Steps 18 through 22.)
Are you debugging a 32-bit app on 64-bit Windows? You can't debug Windows source in this case. Run the 32-bit code on a 32-bit version of Windows in a virtual machine and use Remote Debugging.
Are you trying to step into the Windows kernel? Kernel debugging is different from user mode debugging. You must use WinDbg for kernel mode debugging.
Is the source code missing on Code Center Premium? This would be the case if you are getting the error SRCSRV: ... not indexed. Please see the section below titled ANSI versus Unicode Functions.
Are you seeing a warning about an "Untrusted Command"? This is normal. Just click Run.
Can't step into a Windows API function? Make sure you have symbols for the DLL. Look in the Modules list and make sure that it does NOT say "Source information stripped." If it does, clear your symbol cache and run your app again.

Running Visual Studio in a Virtual Machine

You can install Visual Studio 2010 on your Windows 7 clean machine and run Visual Studio 2010 in the virtual machine so you don't have to use remote debugging. However, you must use a virtual machine that supports USB devices, such as VMware or Virtual PC in Windows 7. Virtual PC 2007 will not work (not even with SP1.)

In Virtual PC on Windows 7, go to the USB menu and select USB SmartCard Reader. You will be prompted to install the drivers in the virtual machine. The smart card will be DISCONNECTED from the host system, so you will not be able to use it in the host machine until the virtual machine releases it.

You do not need to do this if you are using Remote Debugging to run your application in the virtual machine.

ANSI versus Unicode Functions

Debugging into the "A" and "W" versions of functions frequently doesn't work. Here is the explanation from C++ MVP Jochen Kalmbach:

The problem with " PathRemoveExtensionA" seems to be a "build feature". They implemented the file only once for A and W and then you uses this file without "A, W"... In this case, the source file is called "path.c" and is referenced as "patha.c" (A) and "pathw.c" (W).

If you want to debug into this files, you need just to copy the source from the CCP-Website for the "TCHAR" version and save it wherever you want with the name "patha.c" and/or "pathw.c". Then if VS askes you for the file, you can just use this file. It will exactly match.

It seems that this file is generated during the build-process, because it seems that it is also compiled from the output directory...

Source Server Paths

(Thanks again to C++ MVP Jochen Kalmbach for providing this list.)

Here is the list of all currently available source server paths. Links are not clickable because these links are only for use in Visual Studio, not in your browser.

Win7 RTM: https://codepremium.msdn.microsoft.com/source/Windows%207/RTM

W2k8 RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/RTM

W2k8 Hyper-V-RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/Hyper-V%20RTM

W2k8 SP2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/SP2

W2k8 R2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/R2

W2k3 RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/RTM

W2k3 R2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20R2

W2k3 SP1: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20SP1%20RTM

W2k3 SP2 RC: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20SP2%20RC

W2k3 SP2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20SP2%20RTM

Vista Beta1: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/Beta%201

Vista Beta2: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/Beta%202

Vista RC1: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/RC1

Vista RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/RTM

Vista SP1: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/SP1

Vista SP2: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/SP2

XP RTM: https://codepremium.msdn.microsoft.com/source/windows%20XP/RTM

XP SP1: https://codepremium.msdn.microsoft.com/source/windows%20XP/SP1

XP SP2: https://codepremium.msdn.microsoft.com/source/windows%20XP/SP2

XP SP3: https://codepremium.msdn.microsoft.com/source/windows%20XP/SP3

XP 64-bit: Not available.

W2k Datacenter RTM: https://codepremium.msdn.microsoft.com/source/Windows%202000/Datacenter%20RTM

W2k SP3: https://codepremium.msdn.microsoft.com/source/Windows%202000/SP3

W2k SP4: https://codepremium.msdn.microsoft.com/source/Windows%202000/SP4

Extracting Icons with the Windows SDK

2010-11-12T12:17:00.001-07:00

This is an article about the technical aspects of dealings with Windows Icons, including icon resources and HICON handles. But first, let me share with you how I feel about icons. I hate icons. I hate creating them. I hate working with them. I hate rendering them. I hate converting them. I hate dealing with transparency. About the only thing I like about icons is the artistry, and that's sadly lacking lately with the one and two letter icons now popularized with Adobe, Microsoft, Pandora, and Facebook. And let's not even talk about the new dark blue Visual Studio icon, that's pretty much invisible on the task bar.
Life got a little better about four years ago when I bought Axialis IconWorkshop. That's one handy piece of software, with free upgrades for life. If you are in the business of writing software, IconWorkshop will save you a lot of headaches.

But I digress.

Let me start out by pointing out the "official" documentation on icons. See the article on MSDN titled simply, Icons. Note the date of the article: 1995. Things have changed a lot in the last 15 years.

In the beginning, there were 16-color (4-bit) icons, with one color reserved for transparency, so there were really 15 colors available for use. As computers because more powerful, Microsoft introduced 256 color icons, then 24-bit truecolor icons, then 32-bit icons with 24-bit color and 8-bit transparency. Then, as a final coup-de-grace, Vista introduced 32-bit icons sized at 256x256 pixels. These took up so much space that they were stored in a completed different format (PNG). Ouch!

A good case study is the application icon for Outlook Express/Windows Mail in the file MSOERES.DLL, which has existed for most of the life of Microsoft Windows and has evolved as operating system support (and video card support) for icons has improved.

Vista and Windows 7: eight different sizes (256 pixel, 64, 48, 40, 32, 24, 22, 16) at three bit depths (32-bit, 8-bit and 4-bit.)
Outlook XP SP3: three sizes (48 pixel, 32, 16) at at three bit depths (32-bit, 8-bit and 4-bit.)
Windows 2000 SP4: three sizes (48 pixel, 32, 16) at at three bit depths (32-bit, 8-bit and 4-bit.)
Windows 95: two sizes (32 pixel and 16 pixel) at one bit depth (4-bit.)

The only surprise for me in this list is that the Windows 2000 DLL included 32-bit icons, which would have transparency. However, that DLL was part of Service Pack 4 in 2002 and so was probably shared with Windows XP. I'm not aware that Windows 2000 was able to handle transparent icons. Also, you might wonder why I list "dead" operating systems like Windows 95. The reason is that there are some APIs that are stuck in the days of Win9x and never improved, notably the toolbar code in MFC. The common controls also behave differently depending on whether the Common Controls 6 is enabled, either explicitly or with a manifest.

Here's an important point about dealing with icons that took me quite a while to understand. First, there's an icon resource with a DLL or EXE that contains the icon in multiple sizes and multiple bit depths. On the other hand, an HICON contains only a single one of those formats. If you want a different size or a different bit depth, you need to load a new HICON.

The most basic icon-handling function is LoadIcon. It loads an icon from a particular HINSTANCE whose "size conforms to the SM_CXICON and SM_CYICON system metric values." So let's say you call this function and get an icon back. What bit depth? I have no idea. Presumably the same as your desktop, but even that's a hazy concept, because your bit depth can change on the fly, especially when you start a Remote Desktop session. And there's no such thing as a 32-bit desktop - there's no transparency on your desktop. So a 32-bit icon kind of/sort of matches a 24-bit and a 16-bit desktop. But if the icon matches the bit depth of your desktop, how would the icon keep its transparency bits? Again, I don't know.

The documentation for LoadIcon says that "This function has been superseded by the LoadImage function." The LoadImage function takes the desired x and y size as parameters. That's helpful, now I can load those 256x256 bitmaps. But there's still no documentation on the bit depth.

What lead me down the path to this point is that I'm trying to extract some application icons for a web page that is being created for the local user. Those icons needs to be in PNG format to preserve transparency, so the goal was to load them from the original EXE as 32-bit icons, then save them as compressed PNG files.

Let me summarize my success for this project: Total Fail.

The first task was to get the icon for the particular file extension, such as ".doc". This problem is easy to solve and well documented - use SHGetFileInfo, which returns the HICON in either small or large (however those happen to be defined.) I spent several hours trying to convert that icon to a PNG file. The popular strategy seemed to be using OleCreatePictureIndirect, as described by DanRollins in the comments in this article and by neilsolent in this article The problem I saw was that my PNG file was always 16 colors. In later comments of that article, neilsolent said that he was seeing a similar problem, which no one was able to provide an answer to. I wasn't able to solve that problem.

Note that the algorithm offered in the initial question in that article is utterly wrong. An HICON is not an HGLOBAL and you can't access it using GlobalLock.

My fallback strategy was to just write a .ico file instead of automatically converting to a .png file. With a .ico file, converting to .png is trivial using IconWorks.

This was nowhere near as simple as I'd hoped. Remember that an icon resource contains multiple sizes and multiple bitdepths? A .ico file is really a container for manager all of those formats, and there is no Windows API for writing that file. Some documentation is given in that 1995 article I mentioned earlier, but it's a non-trivial problem to solve. There is a sample from Microsoft called IconPro (run the executable, open the .chm help file, look for IconPro, and it will give you the option to extract file sample code.) IconPro hasn't been updated in fifteen years. It knows how to write .ico files, so I was hoping that I could make some minor modifications and feed it the HICON from SHGetFileInfo. No dice. IconPro only reads the original EXE or DLL file.

No problem. I'd just retrieve to pointer to the original DLL or EXE and pass that information to the appropriate routine in IconPro.

Unfortunately, it's not that easy. In fact, it's really ugly. There's no Windows API that will automatically tell you the filename and index to retrieve an icon. SHGetFileInfo only returns an index to the system image list (aka the Icon Cache.) There's no hint of information about where the icon came from. I was surprised because I thought that this problem was simple - just look up the DefaultIcon key in the registry. Turns out that this is just one way of specifying icons. To even begin to understand the other methods, you have to delve into the shell namespace and the IExtractIcon interface. I didn't care that much.

I spent several more hours searching for a method that could take an HICON and write it to a .ico file. I found numerous other people asking this same question, but I found no answers that worked and would create a .ico file with 32-bit color depth.

At this point I gave up on an automated solution. I'll use IconWorks to manually extract the resources that I need from the relevant EXE and DLL files.

Norton Antivirus: The Clear Choice for SSDs

2010-11-02T23:56:00.000-07:00

Today I walked up to my computer and noticed that all eight cores were pegged. I thought that was odd, since the computer should have been doing absolutely nothing. Then I noticed the "Norton Antivirus" logo in the bottom right. "Could it be?" I thought to myself. Is there actually a consumer product on the market that is smart enough to use more than one core?!?

I tapped the keyboard, the Norton Antivirus banner disappeared, and all of the cores dropped back to idle.

I'm really impressed by this. I was trying to remember the last time I was impressed by antivirus software. I'm pretty sure the answer is "never."

Normally an antivirus application is limited by your hard drive's random access performance. Antivirus software processes files in a directory sequentially, but the files often aren't aren't laid out on the disk in that order. This means that the drive is mostly being accessed randomly, not sequentially, all of which is why it can take your antivirus software twelve hours to scan your hard drive, even with a fast hard drive and processor.

However, with an SSD, the system can maintain transfer rates in excess of 100MB/second, even with random access behavior. Crunching that much data per second is going to take more than one core, and, I'm impressed to say, Norton Antivirus appears to be up to the task.

Lightning Fast Builds with Visual Studio 2010 and an SSD

2010-09-11T22:19:00.007-07:00

I reduced my Visual Studio 2010 C++ build time from 21 minutes to 7 5 minutes! You can too. Here's how.

I'm a build performance junkie. If there's one thing I really hate in life, it's sitting around waiting for builds to complete. Fifteen years ago, the very first article I published was titled Speeding Up Visual C++. It was all about making Visual C++ 1.51 go faster on what was then a state of the art computer - an ISA bus Gateway DX2/50. Woo hoo! My recommendations were:

Use Precompiled Headers.
Upgrade to 16MB of memory.
Use 4 megabytes of Disk Cache.
Upgrade your Hard Drive to Fast SCSI or Enhanced IDE.
Turn off Browse Info.
32 Bit File Access.

Today computers are thousands of times faster, but rotating platter disk drives are still desperately slow. The seek time of 7200RPM drives has changed very little in the last ten years, although the transfer rate for sequential files has risen dramatically. That problem, combined with Visual Studio's desire to create hundreds of .tlog temporary files, quarter gigabyte .sdf files, and project size bloat means that the average build may be even slower today than it was fifteen years ago.

Historically, your CPU would be sitting idle most of the time waiting for the hard disk to keep up. Linking performance is based almost entirely on your disk's random access read/write performance. The highly rated Western Digital Caviar Black can only perform about 100 random access IOPS (I/O Operations Per Second.) It takes multiple I/O operations per OBJ file, so a significant fraction of the link time is waiting for the hard drive to do its job.

Enter the latest generation of SSDs driven by the Sandforce Controller, such as the OCZ Vertex 2. These drives can do over 40,000 IOPS - 400 times faster than a Caviar Black. And Visual Studio 2010 build performance is phenomenal. In fact, these drives are so fast that their impact on build time is negligible. This SSD will easily hit over 50MB/sec of 4KB random writes. In contrast, the ultra-zippy 10,000 RPM VelociRaptor can only do about 3.5MB/sec. (Note that disk striping or mirroring has minimal impact on build performance because the linker isn't smart enough to use scatter/gather to force the queue depth high enough to let the command queuing on the drive work its magic.)

Now that the hard disk performance no longer matters, our next bottleneck is the CPU. You can tell your boss you need one of those monster hyper-threaded quad core i7 processors such as the 875k or, for the money-is-no-object crowd, the hyper-threaded six core 980X. Visual Studio 2010 automatically uses parallel builds. My 875k pegs all eight cores simultaneously at 100%. Compiles proceed eight files at a time and the CPUs stay pegged until the compile is finished. I've never seen a project build so fast.

The next bottleneck is probably your RAM. If you have 4GB RAM running on a 32-bit OS, you are severely limited and you probably won't be able to run eight compiles (much less 24 compiles if you are using parallel MSBuild tasks, as I explain in Part 2.) Upgrade to Windows 7 64-bit with 8GB of RAM.

So it's interesting that the types of recommendations for build performance haven't changed much. Here is my updated list:

Use Precompiled Headers.
Upgrade to ~~16MB~~ 8GB of memory.
~~Use 4 megabytes of Disk Cache~~. Upgrade to 64-bit Windows.
Upgrade your Hard Drive to ~~Fast SCSI or Enhanced IDE~~ a Sandforce-based SSD.
Turn off Browse Info. (This is still true. Browse Info is different than Intellisense.)
~~32 Bit File Access~~. Check your motherboard's SATA implementation. At SSD speeds, not all controllers are created equal.

In Part 2 of this article, I'll talk about how to tune your build system to keep all that hardware busy.

The system used for this article was:

ASUS P7P55D-E Pro motherboard.
Intel 875k i7 processor.
OCZ Vertex 2 120GB SSD.
8GB RAM.
Thermaltake MUX-120 heatsink.

Do not use the SATA 3 Marvell controller on the ASUS motherboard. The standard ICH10 controller is much faster with SSDs.

The prior system that took 21 minutes was a Core 2 Duo 2.4 GHz with ASUS P5B Deluxe and Caviar Black 750GB hard drive.

Project is permanently out of date

2010-09-11T09:34:00.002-07:00

I've seen several cases with Visual Studio 2010 where my project would be permanently "out of date." I'd build the project, run it, and would immediately be told that my project was out of date. Solving this through trial and error is tedious at best.

The most common cause is a file that's in your project that you've deleted from the disk. These are easy to find in smaller projects by just looking through the project for a file with an X next to it. If you can't find the file causing the problem, enable debugging in Visual Studio 2010:
http://blogs.msdn.com/b/vsproject/archive/2009/07/21/enable-c-project-system-logging.aspx

I've also seen this problem when I renamed a file. The old filename is still embedded in the compiler's temporary files, even though the name no longer appears in the project. The solution is to delete all intermediate files in the project and rebuild.

Fixing Permissions on an External Hard Drive

2010-09-05T23:55:00.002-07:00

Today I pulled a hard drive from my old computer and hooked it up to my new computer, planning to move the data to the new drive and then use the old drive as a backup disk. Mostly this plan worked well, except that I wasn't allowed to delete many files and folders, even though I was an Administrator. Curious.

The problem turned out to be that I was on a workgroup, not a domain, and so the systems didn't have any common notion of "Administrator". Although Explorer knows how to request elevated permissions, this still isn't enough. You have to "Take Ownership" of the files in order to delete them, and there's no way to do this from Explorer.

I found a solution in the winmatrix forum, but it only works for files, not directories. You can't set the Full Control permission on a directory, so you end up being locked out of directories if you try and use these commands:

takeown /f filepath /r
icacls filepath /grant yourusername:f /t

Note that I've added /r and /t to the commands, which is required for the them to operate recursively.

Instead, I did the steps below. These steps assume that the new drive is on E:.

Open a Command Prompt with Run As Administrator.
Run this command: takeown /f e:\ /r
Right-Click on the root of the copied drive.
Select Properties.
Click the Security tab.
Click the Edit button.
Select Authenticated Users.
Click the checkbox under Allow for Full Control.
Run this command: icacls e:\*.* /reset /t
This command will force all permissions to mirror the permissions on the root of the drive that you set in #6. You must have the *.* or the root directory will be reset, which you don't want.

After these commands completed, I was able to delete all of the desired files. Executing these commands can take quite a while if you have many files on your disk.

Windows 7 Network Performance

2010-09-03T02:10:00.001-07:00

A few years back, after I installed Vista, I spent quite a bit of time trying to fix my GigE Ethernet performance with Windows Vista talking to Windows Server 2003 R2. My file copy performance hovered around 15 to 18 MB/sec, which was pretty dismal.

I've just built myself a new PC with a Core i7 CPU and Windows 7. I tried copying a file from the old Vista box (Core 2 Duo 2.4GHz on ASUS P5B Deluxe mobo) to the new Windows 7 box. The copy performance over the Ethernet went straight up to 50MB/sec! This was with a single WD Caviar Black drive on the Vista system, no RAID or striping.

When I tried copying from a Windows Server 2008 R2 system to the new Windows 7 system, I peaked at 112MB/sec for data that was cached, then backed off to 50 MB/sec for data that wasn't cached. Windows Server 2008 R2 was installed on the same hardware that used to be running Windows Server 2003 R2, so the performance increase was solely due to the OS upgrade.

I'm seeing similar performance gains over the internet. Talking to our corporate server from the Vista system, I maxed out at 1.5 MB/s. Under Windows 7 with i7, I'm able to max out the connection at 5 MB/s.

All of this leads me to believe that the GigE networking performance of Windows Server 2003 R2 was awful, given that I'm running Windows Server 2008 R2 on the exact same hardware with a 5x performance increase.

Net result: (no pun intended) I'm very happy with Windows 7.

Safely using erase() in STL

2010-07-25T01:24:00.006-07:00

There are few features of STL that have caused me more aggravation than the erase() functions.The problems with erase() are, first that it takes an iterator as an argument and second that erase() invalidates all iterators - sometimes. This makes for a catch-22 situation if you don't know "the secret."

Consider what happens if you want to erase each element of a collection that meets some condition. Deleting a single element is not an issue because you don't need the iterator afterward. Deleting all elements is similarly easy because you can simply call clear(). Deleting an arbitrary number of elements is generally written this way by most beginners:

set<int>coll;
/* ... */
for (set<int>::iterator itr=coll.begin(); itr!=coll.end(); ++itr)
   if (*itr % 2 == 0)
      coll.erase(itr);

Predictably, this code doesn't work. The iterator is invalidated when erase() is called and so the ++itr statement in the for() loop fails. Most modern versions of STL have debugging code to warn you if you use an invalidated iterator, but no such training wheels existed in the early days of STL. I got a lot of bug reports because of this mistake.

My first attempted workaround was for the vector class. I wanted to ignore the iterator completely and just step through the collection with an index:

for (unsigned int i=0; i<coll.size(); ++i)

When I tried to call coll.erase(), I discovered that I needed an iterator, which is what I wase trying to avoid in the first place. Years after I first tried to make this solution work, I finally learned the solution. Here's an example to delete all even values when using an index-based for loop;

vector<int> coll;
/* ... */
for (unsigned int i=0; i<coll.size(); ++i)
    if (coll[i]%2 == 0)
        coll.erase(coll.begin() + i--);

coll.begin() is always a valid iterator (even if it equals "end()"), and you are allowed to use array arithmetic on iterators for vector<> and for deque<>. Although this solutions works, it's a bit of a hack, especially because you have to "fix up" i after the call to erase() so that the index of the array won't change. This strategy also won't work for collections such as list<> and set<> that can't be indexed by a scalar.

So I was back to figuring out how to get a "valid" iterator after calling erase().

In some STL collections, the erase() function returns an iterator to the next element. Therefore, even though the original iterator was invalidated, you were given a new iterator that would work, so there was no conflict. I thought the code would be easy to write, once I learned about that return value:

set<int> coll;
/* ... */
for (set<int>::iterator itr=coll.begin(); itr!=coll.end(); ++itr)
   if (*itr%2 == 0)
      itr = coll.erase(itr);

I liked this solution. No tricks, easy to write, easy to read. But it crashed. Why? Because erase() could return coll.end(), which the for loop would try to increment, which wasn't allowed. So the solution looks like this:

set<int> coll;
/* ... */

set<int>::iterator itr=coll.begin();
while ( itr!=coll.end() )

{
   if (*itr%2 == 0)
      itr = coll.erase(itr);

else
++itr;
}

The Microsoft version of STL implements map::erase() by returning an iterator, but this is non-standard. I therefore ran into trouble when I switched to STLport and map::erase() no longer returned an iterator. I found the solution buried in the STL source code, where I learned that the iterators in some collections are stable if they are incremented before erase() is called, in which case you end up with this result:

set<int> coll;
/* ... */
set<int>::iterator itr=coll.begin();
while ( itr!=coll.end() )
{
   if (*itr%2 == 0)

   {
      set<int>::iterator next = itr;
      ++next;
      coll.erase(itr);
      itr = next;

   }
   else
   {
      ++itr;

}
}

Later I learned that the code could be simplified:

set<int> coll;
/* ... */

set<int>::iterator itr=coll.begin();
while ( itr!=coll.end())

{
   if (*itr%2 == 0)
      coll.erase(itr++);

else
++itr;
}

Note that we can't use the "hack" from the beginning of this article, which would allow us to write coll.erase(itr--) and then increment the iterator in a surrounding for loop. The problem is that a scalar can decrement below zero, but an iterator cannot decrement before begin().

Personally, I don't like the autoincrement code above as much as when erase() returns a value. It relies on the reader understanding the "trick" and it makes it more difficult for collection classes to know in advance what to do. For example, this technique can't be used with a vector, which is why vector does return an iterator. This creates a confusing inconsistency. (Why is why the Microsoft version of map::erase() is non-standard, but much more consistent.)

The question is, why does this trick work? If you've spent much time in C++, you've inevitably learned about the dangers of code like this:

int i = 5;
i = i++;

The code could end with i equal to either 5 or 6 - the result is undefined because the C++ standard only guarantees that ++ will be executed after i is read and before the semi-colon. It could be incremented before or after the assignment happens. (If you are wondering why, remember that assignments in C++ can be embedded in expressions, unlike most languages.)

To reword the guarantee in the language of the C++ Standard, the only guarantee is that ++ will be executed after i is read and before the next sequence point. The sequence point is the key to understanding our call to erase().

In the expression above, the semicolon is the sequence point. Now look at our code example:

coll.erase(itr++);

If you apply the same evaluation rules, the undefined timing of the execution of ++ would mean that the iterator could be evaluated after erase() returns and the iterator had been invalidated. This would crash, but experience shows that it doesn't. The reason is that the function call to erase() introduces a new sequence point. The code is guaranteed to be evaluated as:

The value of itr is loaded and pushed on the stack for erase().
itr is incremented.
erase() is called.

All of which gives the desired result. QED.

"Invalid pointer" error in Visual Studio 2010 in Configuration Manager

2010-07-16T14:36:00.000-07:00

I've been fighting against a problem I've seen in most of my project files that were upgraded from VS2005 or VS2008. Any attempt to delete a Configuration or a Platform results in the error "The operation could not be completed. Invalid pointer."

Today I was finally able to track down the problem, which is caused by some seemingly benign lines in the .vcxproj file. To fix the problem, open your .vcxproj file in Notepad, go to the end of the file, and you'll find some lines that look like this:

<ProjectExtensions>
  <VisualStudio>
    <UserProperties RESOURCE_FILE="Dot.rc" />
  </VisualStudio>
</ProjectExtensions>

Remove those lines, reload the project, and everything will start working fine. Those lines do not appear in project files that are created by the Visual Studio 2010 wizard, so I believe that they are not needed.

My complete bug report can be found on Microsoft Connect:
https://connect.microsoft.com/VisualStudio/feedback/details/575911/invalid-pointer-error-deleting-configurations-in-upgraded-vs2010-c-project-files#tabs

Debug "Just My Code" for C++ and MFC

2010-07-01T14:40:00.007-07:00

One of my biggest annoyances with debugging MFC code is constantly stepping through CString functions and COM object functions. In Visual C++ 6 there was some functionality in autoexp.dat for handing this, but I never got it to work to my satifaction.

This week I was able to solve the problem. Here's how (note that the 10.0 is for Visual Studio 2010. The value will be 9.0 for VS2008. For earlier versions, see the link at the end of this article.)

Open RegEdit
On 32-bit Windows, go to:
HKEY_LOCAL_MACHINE\Software\Microsoft\ VisualStudio\10.0\NativeDE\StepOver

On 64-bit Windows, go to:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\NativeDE\StepOver

At least for Visual Studio 2010, there are several default functions defined.
Add a new String key. Name it CStringT.
Edit the key and set the value to:

ATL\:\:CStringT.*

The colons must be backslashed as shown. CStringT is the template class used for CString, CStringA and CStringW. The CStringT class is shared with ATL, so that's why it's in the ATL namespace, even if you are using MFC. The "dot star" at the end matches anything for the rest of the line.
Here are a couple of other examples:

Key: CSimpleString
Value: ATL\:\:CSimpleStringT.*

Key: CComPtrBase
Value: ATL\:\:CComPtrBase.*

Key: vector
Value: std\:\:vector.*

Update 7/26/2010: There are additional features documented by Andy Pennell at Microsoft.

Configuring Mozy Pro for Visual Studio

2010-05-28T12:05:00.002-07:00

Earlier this week I revised my review of Mozy based on version 2.0 of the MozyPro client software. In short, MozyPro went from something I found acceptable to something that I am quite happy with.

One of my frustrations in earlier versions of MozyPro was figuring out how to exclude Visual Studio temporary files. In Visual Studio 2010, the IntelliSense data (the .sdf file) is 70MB to 300MB for each project. Since Visual Studio automatically rebuilds this file as necessary, it's a waste of time and money to back it up. There are numerous other temporary files, such as .ncb, .sbr, .bsc, and others, all of which are unnecessary to backup. This article tells how to set up MozyPro so that files with those extensions will not be backed up.

I found it best to create a global rule instead of modifying one of the predefined Backup Sets. Although MozyPro has a predefined Backup Set named "Visual Studio Projects", I've created other Backup Sets based on projects and customers. By creating a global rule, the temporary files will be ignored by all other projects.

To create a rule for Visual Studio, go to the Backup Sets page in the MozyPro configuration. Create a new backup set and call it something like Excluded Files. In the Backup Set Editor, put a checkmark next to C: so that this rule applies to the entire drive. Next check the box in the top right labeled "Files matching this set will be EXCLUDED." Under the Rules, the first rule should read:

Include / File Type / sdf pch ncb idb tlog sbr res dep obj ilk ipch bsc

This will exclude temporary and intermediate files for Visual Studio 2005 through 2010, as well as Virtual PC undo files. If you run Visual Studio 2010, also click the plus sign (+) on the right and create a second rule that reads:

Or / Include / Folder name / is / ipch / Files and Folders

Finally, if you use Virtual PC, create a third rule:

Or / Include / File Type / vud vsv

Note that the "Include" command in these rules really means "Exclude" because you checked the box "Files matching this set will be EXCLUDED..."

Understanding ReadDirectoryChangesW - Part 2

2010-05-19T20:13:00.019-07:00

The longest, most detailed description in the world of how to successfully use ReadDirectoryChangesW.

This is Part 2 of 2. Part 1 describes the theory and this part describes the implementation.

Download the sample code for this article.

Getting a Handle to the Directory

Now we'll look at the details of implementing the Balanced solution described in Part 1. When reading the declaration for ReadDirectoryChangesW, you'll notice that the first parameter is to a directory, and it's a HANDLE. Did you know that you can get a handle to a directory? There is no OpenDirectory function and the CreateDirectory function doesn't return a handle. Under the documentation for the first parameter, it says “This directory must be opened with the FILE_LIST_DIRECTORY access right.” Later, the Remarks section says, “To obtain a handle to a directory, use the CreateFile function with the FILE_FLAG_BACKUP_SEMANTICS flag.” The actual code looks like this:


HANDLE hDir = ::CreateFile(

strDirectory,

// pointer to the file name
    FILE_LIST_DIRECTORY,

// access (read/write) mode
    FILE_SHARE_READ

// share mode
        | FILE_SHARE_WRITE
        | FILE_SHARE_DELETE,
    NULL, // security descriptor
    OPEN_EXISTING,

// how to create
    FILE_FLAG_BACKUP_SEMANTICS // file attributes
        | FILE_FLAG_OVERLAPPED,
    NULL);

// file with attributes to copy

The first parameter, FILE_LIST_DIRECTORY, isn't even mentioned in the CreateFile() documentation. It's discussed in File Security and Access Rights, but not in any useful way.

Similarly, FILE_FLAG_BACKUP_SEMANTICS has this interesting note, "Appropriate security checks still apply when this flag is used without SE_BACKUP_NAME and SE_RESTORE_NAME privileges." In past dealings with this flag, it had been my impression that Administrator privileges were required, and the note seems to bear this out. However, attempting to enable these privileges on a Windows Vista system by adjusting the security token does not work if UAC is enabled. I'm not sure if the requirements have changed or if the documentation is simply ambiguous. Others are similarly confused.

The sharing mode also has pitfalls. I saw a few samples that left out FILE_SHARE_DELETE. You'd think that this would be fine since you do not expect the directory to be deleted. However, leaving out that permission prevents other processes from renaming or deleting files in that directory. Not a good result.

Another potential pitfall of this function is that the referenced directory itself is now "in use" and so can't be deleted. To monitor files in a directory and still allow the directory to be deleted, you would have to monitor the parent directory and its children.

Calling ReadDirectoryChangesW

The actual call to ReadDirectoryChangesW is the simplest part of the whole operation. Assuming you are using completion routines, the only tricky part is that the buffer must be DWORD-aligned.

The OVERLAPPED structure is supplied to indicate an overlapped operation, but none of the fields are actually used by ReadDirectoryChangesW. However, a little known secret of using Completion Routines is that you can supply your own pointer to the C++ object. How does this work? The documentation says that, "The hEvent member of the OVERLAPPED structure is not used by the system, so you can use it yourself." This means that you can put in a pointer to your object. You'll see this in my sample code below:


void CChangeHandler::BeginRead()
{
    ::ZeroMemory(&m_Overlapped, sizeof(m_Overlapped));
    m_Overlapped.hEvent = this;

    DWORD dwBytes=0;

    BOOL success = ::ReadDirectoryChangesW(
        m_hDirectory,
        &m_Buffer[0],
        m_Buffer.size(),
        FALSE, // monitor children?
        FILE_NOTIFY_CHANGE_LAST_WRITE
         | FILE_NOTIFY_CHANGE_CREATION
         | FILE_NOTIFY_CHANGE_FILE_NAME,
        &dwBytes,
        &m_Overlapped,
        &NotificationCompletion);
}

Since this call uses overlapped I/O, m_Buffer won't be filled in until the completion routine is called.

Dispatching Completion Routines

For the Balanced solution we've been discussing, there are only two ways to wait for Completion Routines to be called. If everything is being dispatched using Completion Routines, then SleepEx is all you need. If you need to wait on handles as well as to dispatch Completion Routines, then you want WaitForMultipleObjectsEx. The Ex version of the functions is required to put the thread in an "alertable" state, which means that completion routines will be called.

To terminate a thread that's waiting using SleepEx, you can write a Completion Routine that sets a flag in the SleepEx loop, causing it to exit. To call that Completion Routine, use QueueUserAPC, which allows one thread to call a completion routine in another thread.

Handling the Notifications

The notification routine should be easy. Just read the data and save it, right? Wrong. Writing the Completion Routine also has its complexities.

First, you need to check for and handle the error code ERROR_OPERATION_ABORTED, which means that CancelIo has been called, this is the final notification, and you should clean up appropriately. I describe CancelIo in more detail in the next section. In my implementation, I used InterlockedDecrement to decrease cOutstandingCalls, which tracks my count of active calls, then I returned. My objects were all managed by the MFC mainframe and so did not need to be deleted by the Completion Routine itself.

You can receive multiple notifications in a single call. Make sure you walk the data structure and check for each non-zero NextEntryOffset field to skip forward.

ReadDirectoryChangesW is a "W" routine, so it does everything in Unicode. There's no ANSI version of this routine. Therefore, the data buffer is also Unicode. The string is not NULL-terminated, so you can't just use wcscpy. If you are using the ATL or MFC CString class, you can instantiate a wide CString from a raw string with a given number of characters like this:

FILE_NOTIFY_INFORMATION* fni = (FILE_NOTIFY_INFORMATION*)buf;
CStringW wstr(fni.Data, fni.Length / sizeof(wchar_t));

Finally, you have to reissue the call to ReadDirectoryChangesW before you exit the completion routine.You can reuse the same OVERLAPPED structure. The documentation specifically says that the OVERLAPPED structure is not accessed again by Windows after the completion routine is called. However, you have to make sure that you use a different buffer than your current call or you will end up with a race condition.

One point that isn't clear to me is what happens to change notifications in between the time that your completion routine is called and the time you issue the new call to ReadDirectoryChangesW.

I'll also reiterate that you can still "lose" notifications if many files are changed in a short period of time. According to the documentation, if the buffer overflows, the entire contents of the buffer are discarded and the lpBytesReturned parameter contains zero. However, it's not clear to me if the completion routine will be called with dwNumberOfBytesTransfered equal to zero, and/or if there will be an error code specified in
dwNumberOfBytesTransfered.

There are some humorous examples of people trying (and failing) to write the completion routine correctly. My favorite is found on stackoverflow.com, where, after insulting the person asking for help, he presents his example of how to write the routine and concludes with, "It's not like this stuff is difficult." His code is missing error handling, he doesn't handle ERROR_OPERATION_ABORTED, he doesn't handle buffer overflow, and he doesn't reissue the call to ReadDirectoryChangesW. I guess it's not difficult when you just ignore all of the difficult stuff.

Using the Notifications

Once you receive and parse a notification, you need to figure out how to handle it. This isn't always easy. For one thing, you will often receive multiple duplicate notifications about changes, particularly when a long file is being written by its parent process. If you need the file to be complete, you should process each file after a timeout period has passed with no further updates. [Update: See the comment below by Wally The Walrus for details on the timeout.]

An article by Eric Gunnerson points out that the documentation for FILE_NOTIFY_INFORMATION contains a critical comment: If there is both a short and long name for the file, the function will return one of these names, but it is unspecified which one. Most of the time it's easy to convert back and forth between short and long filenames, but that's not possible if a file has been deleted. Therefore, if you are keeping a list of tracked files, you should probably track both the short and long filename. I was unable to reproduce this behavior on Windows Vista, but I only tried on one computer.

You will also receive some notifications that you may not expect. For example, even if you set the parameters of ReadDirectoryChangesW so you aren't notified about child directories, you will still get notifications about the child directories themselves. For example. Let's assume you have two directories, C:\A and C:\A\B. You move the file info.txt from the first directory to the second. You will receive FILE_ACTION_REMOVED for the file C:\A\info.txt and you will receive FILE_ACTION_MODIFIED for the directory C:\A\B. You will not receive any notifications about C:\A\B\info.txt.

There are some other surprises. Have you ever used hard links in NTFS? Hard links allow you to have multiple filenames that all reference the same physical file. If you have one reference in a monitored directory and a second reference in a second directory, you can edit the file in the second directory and a notification will be generated in the first directory. It's like magic.

On the other hand, if you are using symbolic links, which were introduced in Windows Vista, then no notification will be generated for the linked file. This makes sense when you think it through, but you have to be aware of these various possibilities.

There's yet a third possibility, which is junction points linking one partition to another. In that case, monitoring child directories won't monitor files in the linked partition. Again, this behavior makes sense, but it can be baffling when it's happening at a customer site and no notifications are being generated.

Shutting Down

I didn't find any articles or code (even in open source production code) that properly cleaned up the overlapped call. The documentation on MSDN for canceling overlapped I/O says to call CancelIo. That's easy. However, my application then crashed when exiting. The call stack showed that one of my third party libraries was putting the thread in an alertable state (which meant that Completion Routines could be called) and that my Completion Routine was being called even after I had called CancelIo, closed the handle, and deleted the OVERLAPPED structure.

As I was searching various web pages with sample code that called CancelIo, I found this page that included the code below:


CancelIo(pMonitor->hDir);

if (!HasOverlappedIoCompleted(&pMonitor->ol))
{

SleepEx(5, TRUE);
}

CloseHandle(pMonitor->ol.hEvent);
CloseHandle(pMonitor->hDir);

This looked promising. I faithfully copied it into my app. No effect.

I re-read the documentation for CancelIo, which makes the statement that "All I/O operations that are canceled complete with the error ERROR_OPERATION_ABORTED, and all completion notifications for the I/O operations occur normally." Decoded, this means that all Completion Routines will be called at least one final time after CancelIo is called. The call to SleepEx should have allowed that, but it wasn't happening. Eventually I determined that waiting for 5 milliseconds was simply too short. Maybe changing the "if" to a "while" would have solved the problem, but I chose to approach the problem differently since this solution requires polling every existing overlapped structure.

My final solution was to track the number of outstanding requests and to continue calling SleepEx until the count reached zero. In the sample code, the shutdown sequence works as follows:

The application calls CReadDirectoryChanges::Terminate (or simply allows the object to destruct.)
Terminate uses QueueUserAPC to send a message to CReadChangesServer in the worker thread, telling it to terminate.
CReadChangesServer::RequestTermination sets m_bTerminate to true and delegates the call to the CReadChangesRequest objects, each of which calls CancelIo on its directory handle and closes the directory handle.
Control is returned to CReadChangesServer::Run function. Note that nothing has actually terminated yet.

void Run()
{
    while (m_nOutstandingRequests || !m_bTerminate)
    {
        DWORD rc = ::SleepEx(INFINITE, true);
    }
}

CancelIo causes Windows to automatically call the Completion Routine for each CReadChangesRequest overlapped request. For each call, dwErrorCode is set to ERROR_OPERATION_ABORTED.
The Completion Routine deletes the CReadChangesRequest object, decrements nOutstandingRequests, and returns without queuing a new request.
SleepEx returns due to one or more APCs completing. nOutstandingRequests is now zero and m_bTerminate is true, so the function exits and the thread terminates cleanly.

In the unlikely event that shutdown doesn't proceed properly, there's a timeout in the primary thread when waiting for the worker thread to terminate. If the worker thread doesn't terminate in a timely fashion, we let Windows kill it during termination.

Network Drives

ReadDirectoryChangesW works with network drives, but only if the remote server supports the functionality. Drives shared from other Windows-based computers will correctly generate notifications. Samba servers may or may not generate notifications, depending on whether the underlying operating system supports the functionality. Network Attached Storage (NAS) devices usually run Linux, so won't support notifications. High-end SANs are anybody's guess.

ReadDirectoryChangesW fails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols.

Summary

If you've made it this far in the article, I applaud your can-do attitude. I hope I've given you a clear picture of the challenges of using ReadDirectoryChangesW and why you should be dubious of any sample code you see for using the function. Careful testing is critical, including performance testing.

Download the sample code for this article.

Understanding ReadDirectoryChangesW - Part 1

2010-05-19T11:16:00.072-07:00

The longest, most detailed description in the world of how to successfully use ReadDirectoryChangesW.

This is Part 1 of 2. This part describes the theory and Part 2 describes the implementation.

Download the sample code for this article.

I have spent this week digging into the barely-documented world of ReadDirectoryChangesW and I hope this article saves someone else some time. I believe I've read every article I could find on the subject, as well as numerous code samples. Almost all of the examples, including the one from Microsoft, either have significant shortcoming or have outright mistakes.

You'd think that this problem would have been a piece of cake for me, having been the author of Multithreading Applications in Win32, where I wrote a chapter about the differences between synchronous I/O, signaled handles, overlapped I/O, and I/O completion ports. Except that I only write overlapped I/O code about once every five years, which is just about long enough for me to forget how painful it was the last time. This endeavor was no exception.

Four Ways to Monitor Files and Directories

First, a brief overview of monitoring directories and files. In the beginning there was SHChangeNotifyRegister. It was implemented using Windows messages and so required a window handle. It was driven by notifications from the shell (Explorer), so your application was only notified about things that the shell cared about - which almost never aligned with what you cared about. It was useful for monitoring things that the user did in Explorer, but not much else.

SHChangeNotifyRegister was fixed in Windows Vista so it could report all changes to all files, but is was too late - there are still several hundred million Windows XP users and that's not going to change any time soon.

SHChangeNotifyRegister also had a performance problem, since it was based on Windows messages. If there were too many changes, your application would start receiving roll-up messages that just said "something changed" and you had to figure out for yourself what had really happened. Fine for some applications, rather painful for others.

Windows 2000 brought two new interfaces, FindFirstChangeNotification and ReadDirectoryChangesW. FindFirstChangeNotification is fairly easy to use but doesn't give any information about what changed. Even so, it can be useful for applications such as fax servers and SMTP servers that can accept queue submissions by dropping a file in a directory. ReadDirectoryChangesW does tell you what changed and how, at the cost of additional complexity.

Similar to SHChangeNotifyRegister, both of these new functions suffer from a performance problem. They can run significantly faster than shell notifications, but moving a thousand files from one directory to another will still cause you to lose some (or many) notifications. The exact cause of the missing notifications is complicated. Surprisingly, it apparently has little to do with how fast you process notifications.

Note that FindFirstChangeNotification and ReadDirectoryChangesW are mutually exclusive. You would use one or the other, but not both.

Windows XP brought the ultimate solution, the Change Journal, which could track in detail every single change, even if your software wasn't running. Great technology, but equally complicated to use.

The fourth and final solution is is to install a File System Filter, which was used in the popular SysInternals FileMon tool. There is a sample of this in the Windows Driver Kit (WDK). However, this solution is essentially a device driver and so potentially can cause system-wide stability problems if not implemented exactly correctly.

For my needs, ReadDirectoryChangesW was a good balance of performance versus complexity

The Puzzle

The biggest challenge to using ReadDirectoryChangesW is that there are several hundred possibilities for combinations of I/O mode, handle signaling, waiting methods, and threading models. Unless you're an expert on Win32 I/O, it's extremely unlikely that you'll get it right, even in the simplest of scenarios. (In the list below, when I say "call", I mean a call to ReadDirectoryChangesW.)

A. First, here are the I/O modes:

Blocking synchronous
Signaled synchronous
Overlapped asynchronous
Completion Routine (aka Asynchronous Procedure Call or APC)

B. When calling the WaitForXxx functions, you can:

Wait on the directory handle.
Wait on an event object in the OVERLAPPED structure.
Wait on nothing (for APCs.)

C. To handle notifications, you can use:

Blocking
WaitForSingleObject
WaitForMultipleObjects
WaitForMultipleObjectsEx
MsgWaitForMultipleObjectsEx
I/O Completion Ports

D. For threading models, you can use:

One call per worker thread.
Multiple calls per worker thread.
Multiple calls on the primary thread.
Multiple threads for multiple calls. (I/O Completion Ports)

Finally, when calling ReadDirectoryChangesW, you specify flags to choose what you want to monitor, including file creation, last modification date change, attribute changes, and other flags. You can use one flag per call and issue multiple calls or you can use use multiple flags in one call. Multiple flags is always the right solution. If you think you need to use multiple calls with one flag per call to make it easier to figure out what to do, then you need to read more about the data contained in the notification buffer returned by ReadDirectoryChangesW.

If your head is now swimming in information overload, you can easily see why so many people have trouble getting this right.

Wrong Techniques

As I was researching this solution, I saw a lot of recommendations that ranged from dubious to wrong to really, really wrong. Here's some commentary on what I saw.

If you are using the Simplicity solution above, don't use blocking calls because the only way to cancel it is with the undocumented technique of closing the handle or the Vista-only technique of CancelSynchronousIo. Instead, use the Signal Synchronous I/O mode by waiting on the directory handle. Also, to terminate threads, don't use TerminateThread, because that doesn't clean up resources and can cause all sorts of problems. Instead, create a manual-reset event object that is used as the the second handle in the call to WaitForMultipleObjects.When the event is set, exit the thread.

If you have dozens or hundreds of directories to monitor, don't use the Simplicity solution. Switch to the Balanced solution. Alternatively, monitor a root common directory and ignore files you don't care about.

If you have to monitor a whole drive, think twice (or three times) about this idea. You'll be notified about every single temporary file, every Internet cache file, every Application Data change - in short, you'll be getting an enormous number of notifications that could slow down the entire system. If you need to monitor an entire drive, you should probably use the Change Journal instead. This will also allow you to track changes even if your app is not running. Don't even think about monitoring the whole drive with FILE_NOTIFY_CHANGE_LAST_ACCESS.

If you are using overlapped I/O without using an I/O completion port, don't wait on handles. Use Completion Routines instead. This removes the 64 handle limitation, allows the operating system to handle call dispatch, and allows you to embed a pointer to your object in the OVERLAPPED structure. My example in a moment will show all of this.

If you are using worker threads, don't send results back to the primary thread with SendMessage. Use PostMessage instead. SendMessage is synchronous and will not return if the primary thread is busy. This would defeat the purpose of using a worker thread in the first place.

It's tempting to try and solve the issue of lost notifications by providing a huge buffer. However, this may not be the wisest course of action. For any given buffer size, a similarly-sized buffer has to be allocated from the kernel non-paged memory pool. If you allocate too many large buffers, this can lead to serious problems, including a Blue Screen of Death. Thanks to an anonymous contributor in the MSDN Community Content.

Jump to Part 2 of this article.

Download the sample code for this article.

Using MsgWaitForMultipleObjects in MFC

2010-05-17T20:06:00.002-07:00

One of the problems that I didn't solve in my Multithreading book was how to use MsgWaitForMultipleObjectsEx with MFC. I have always felt guilty about this because it was an important problem, but I simply ran out of time to do the implemenation. Recently I finally had a need to solve the problem. MFC has come a long way since 1996 and it's clear that this problem was planned for. With MFC in Visual Studio 2008 and 2010, I was able to solve the problem in minutes instead of days.

In short, the solution is to replace MFC's call to GetMessage with your own call to MsgWaitForMultipleObjectsEx. This will allow you to dispatch messages, handle signaled objects, and dispatch Completion Routines. Here's the code, which goes in your MFC App object:

// virtual
BOOL CMyApp::PumpMessage()
{    HANDLE hEvent = ...;
    HANDLE handles[] = { hEvent };
    DWORD const res =
      ::MsgWaitForMultipleObjectsEx(
        _countof(handles),
        handles,
        INFINITE,
        QS_ALLINPUT,
        MWMO_INPUTAVAILABLE | MWMO_ALERTABLE);
    switch (res)
    {
        case WAIT_OBJECT_0 + 0:
            // the event object was signaled...
            return true;
        case WAIT_OBJECT_0 + _countof(handles):
            return __super::PumpMessage();
        case WAIT_IO_COMPLETION:
            break;
    }

    return TRUE;
}

There are several obscure points in this code worth mentioning. Getting any of these wrong will cause it to break:

The MWMO_INPUTAVAILABLE is required to solve race conditions with the message queue.
The MWMO_ALERTABLE is required in order for Completion Routines to be called.
WAIT_IO_COMPLETION does not require any action on your part. It just indicates that a completion routine was called.
The handle array is empty. You do NOT wait on the directory handle. Doing so will prevent your completion routine from being called.
MsgWaitForMultipleObjectsEx indicates that a message is available (as opposed to a signaled object) by returning WAIT_OBJECT_0 plus the handle count.
The call to __super::PumpMessage() is for MFC when this is running in your application object. Outside of MFC, you should replace it with your own message loop.

Note that this strategy breaks whenever a dialog box is displayed because dialog boxes use their own message loop.

Visual Studio 2010 Tab Key Not Working

2010-05-12T21:25:00.001-07:00

Several days ago, my tab key stopped working when editing C++ code in Visual Studio 2010. Everything else worked fine, even Ctrl-Tab. But pressing Tab on the keyboard would not insert a tab into the file. I tried disabling add-ins. No change. I reset the keyboard in Tools / Options / Environment / Keyboard. Still no change. I tried running devenv /safemode. That didn't work either.

Finally I looked at the other command line switches for devenv.exe, where I saw devenv /ResetSettings. Finally, that fixed the problem.

Static Library Dependencies in Visual Studio 2010

2010-04-28T12:27:00.010-07:00

I've been porting my Visual Studio 2008 C++ application to Visual Studio 2010. One rough spot I've hit is the change in how static library dependencies works. In VS2005 and VS2008, we completely gave up on automated static library dependencies because they only worked in the IDE, not from automated builds with VCBUILD. VS2010 was supposed to fix this problem. (Note that MSBUILD replaces VCBUILD for command line builds in VS2010.)

The problem manifested itself during Batch Build with errors indicating that Debug libraries were being linked into Release builds. For example, these errors happen if you are using STL:

Core.lib(HttpHelpers.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in Activation.obj

MSVCRTD.lib(MSVCR100D.dll) : error LNK2005: _strcspn already defined in libcmt.lib(strcspn.obj)

I've also seen these errors for vanilla MFC projects:

msvcrtd.lib(ti_inst.obj) : error LNK2005: "private: __thiscall type_info::type_info(class type_info const &)" (??0type_info@@AAE@ABV0@@Z) already defined in libcmt.lib(typinfo.obj)

msvcrtd.lib(ti_inst.obj) : error LNK2005: "private: class type_info & __thiscall type_info::operator=(class type_info const &)" (??4type_info@@AAEAAV0@ABV0@@Z) already defined in libcmt.lib(typinfo.obj)

Visual Studio 2010 has multiple ways of specifying dependencies. The actual effect of each setting and settings' interdependence does not appear to be officially documented, although there is some information at http://msdnrss.thecoderblogs.com/2010/02/16/project-settings-changes-with-vs2010/ and at http://blogs.msdn.com/vcblog/archive/2010/03/02/visual-studio-2010-c-project-upgrade-guide.aspx.

The first way of specifying dependencies continues from previous versions of Visual Studio in Project | Project Dependencies. However, unlike earlier versions of Visual Studio, this only affects project build order, not dependency library linking. Also, this only affects project build order in the IDE. It does not affect project build order in MSBUILD. Therefore, you need to configure that page, but it's useless for command line builds. Changes on this page are reflected in the .sln file.

The real action is now on the Framework and References tab in the project properties. This tab used to be exclusively for .Net dependencies, but now it's been expanded for C++ dependencies. When I looked at this page for my project, it looked like this:

There are a couple of things to notice. First, the Configuration in the top left is grayed out. The Debug/Release settings do not apply to the Framework and References page. Second, you'll notice that the Full Path in the properties on the right points to the Debug build. In other words, the Debug library is always linked, even in the Release build. I've blogged about this problem in the past with .Net libraries, but this is pretty fundamental for C++ libraries.

I first thought that the solution was to set Reference Assembly Output to False. However, after reading the blog of one of the Microsoft Program Managers and following up with him, he said that the Reference Assembly Output option is only for managed libraries and does not do anything for native libraries. He said that the behavior I'm seeing is anamolous and that it shouldn't be happening.

[Update 7/18/2010] I have confirmed that this problem only happens with Batch Build. The problem doesn't happen if you explicitly choose a configuration, nor does the problem happen when running MSBUILD from the command line.

I have opened a bug report on Microsoft Connect. Please vote for it at:
https://connect.microsoft.com/VisualStudio/feedback/details/576146/link-library-dependencies-for-c-does-not-work-with-batch-build

[Update 7/18/2010] Microsoft has reproduced the problem, but has decided not to fix it. If you are reading this, PLEASE let them you that fixing this problem is important to you!

Visual C++ 2010 Apps Don't Support Windows 2000

2010-04-22T21:01:00.001-07:00

One of the rather important things missing in the list of Breaking Changes for Visual C++ in Visual Studio 2010 is that applications generated by VS2010 no longer support Windows 2000. Usage of W2K has fallen under 0.5%, but that's still a very large number of users. This was pointed out by Martin Richter in the Community Content. However, the actual story is a little more complex.

I built a C++ application to see what functions are actually being called. When examined in DEPENDS under Windows 2000, these are the functions that are undefined:

DecodePointer
EncodePointer
ReleaseActCtx
CreateActCtxW
ActivateActCtx
DeactivateActCtx

The interesting thing is the DecodePointer and EncodePointer are documented as only being available in Windows XP Service Pack 2 or later. This means that the minimum system requirements for an application generated by Visual C++ 2010 is WinXP SP2, or Windows Server 2003 SP1 for server versions of Windows.

In addition, DUMPBIN shows that the required version of Windows has been bumped from 5.00 to 5.01, which is Windows XP.

What's really annoying is that none of these calls are required under Windows 2000. These calls appear to be an artificial limitation purely for the purposes of preventing end-of-life versions of Windows from working properly.

Targeting C++/CLR v2.0 with Visual Studio 2010

2010-04-13T21:16:00.012-07:00

I just spent far too many hours recovering from trying to change the target .Net Framework version for a C++/CLR project in Visual Studio 20010 (final release.) I hope to save someone else the same frustation. If you don't need the features in .Net 4.0, there's a big advantage to targeting the old version since it is much more likely that your customers will have the old version installed.

If you go into the Framework and References page for a C++/CLR project in Visual Studio 2010, you will see that the Platform droplist is grayed out. There's nothing you can do to ungray it, so don't bother trying.

Underneath it says, "Targeted framework: .NETFramework, Version=4.0" There's no button you can click to change that. If you RTMF, the documentation says that, "The IDE does not support modifying the targeted framework, but you can change it manually." So that's what I tried to do. I went into the vcxproj file, changed the header to say ToolsVersion="2.0" and reloaded the project. Big mistake. The updated project file wouldn't load. I reverted back to the original file from source control and that wouldn't load either.

The first error I had to fix was:

The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

This turned out to be relatively easy to solve. I went into the vcxproj file and did a "Replace All" changing UserRootDir to VCTargetsPath.

The second error was something about ClCompile and not finding AssemblyInfo.cpp. I'm still not entirely clear what caused this because now I can't reproduce the problem. In the end, I removed the project from the solution, closed the solution, closed Visual Studio, deleted the .sdf file, deleted all of the build directories (Debug, Release, etc.), deleted the ipch directory, restarted Visual Studio, added the project back into the solution, and it worked.

None of which got me any closer to solving the original problem.

I created a new C++/CLR project from scratch, which pointed me at the answer. The project file I was using had been created by the Upgrade Wizard in Visual Studio 2010 and it turns out that an important line was left out. Underneath these two lines:

<PropertyGroup Label="Globals">
   <ProjectGuid>{96FD1FEF-8A07-44E6-9CC1-
D4284218B186}</ProjectGuid>

there should be another line that says:

<TargetFrameworkVersion>v4.0</TargetFrameworkVersion>

Change it from v4.0 to v2.0 and everything will magically switch to v2.0. The value v3.5 is also supported. There's some discussion of this at http://blogs.msdn.com/vcblog/archive/2010/03/02/visual-studio-2010-c-project-upgrade-guide.aspx halfway down under Known issues for conversion in VS2010.

Note that using v2.0 forces Visual Studio to use the Visual Studio 2008 compiler. Visual Studio 2010 can only compile v4.0 code.

Update 6/18/2010: To debug your code, you may have to explicitly tell Visual Studio 2010 that the .Net 2.0 CLR should be used. I ran into this when my EXE was native and my DLL was C++/CLR. Create a file named .exe.config and put the following information in the file:


<configuration>
    <startup>
        <supportedRuntime version="v2.0.50727"/>
    </startup>
</configuration>

More information can be found in this article on the Visual Studio Debugger Team Blog. Note that their sample did not work for me under Windows Vista until I removed the line that said:

<?xml version ="1.0"?>

Managing multiple configurations using Virtual PC

2010-03-18T15:09:00.054-07:00

We use Virtual PC extensively for testing in our lab. We have about 12 different configurations for each version of Windows we test. We've spent a lot of time learning how to create these configurations so that they work reliably and to create a reasonable balance between updates of individual configurations versus a global rebuild as the security updates and service packs pile up.

This blog entry describes how we create the VHDs and configure them in Virtual PC.

Layer Overview

Our VHD tree is built based on a set of defined layers. Layer 1 is comprised of single root VHD for each version of Windows, such as WinXP SP3. Layer 2 is built from differencing VHDs that use the root VHD as the parent. This means that the root VHD needs to be configured as far as possible, without requiring lower layer VHDs have to "undo" any configuration. Here is the VHD structure we use:

1. Root VHD - Contains base Windows install and common configurations

2. Configuration specific VHD - Includes apps, service packs, etc. that are different from the root. This is where configuration-dependent changes are done.

3. Working VHD - Includes temporary changes that are used over the course of several tests.

4. Undo file.

Layers are defined by how often they are changed, because when a layer changes, everything underneath it is invalidated. Layer 1 changes every six to eighteen months, usually when the number of security updates becomes overwhelming. Layer 2 changes every few months. Layer 3 is only used when a customized configuration needs to be tested for several days (or when you hit the wrong button and say "commit change.) Layer 4 changes on a minute to minute basis. It is discarded whenever a user selects "Discard changes" when a VPC is closed.

Roadblocks

Creating the VHD hierarchy is tricky. There are several problems we've seen that must be solved by making the changes in the correct order in the correct order. Some of these problems include:

Duplicate Ethernet MAC addresses.
Duplicate computer names.
Domain disconnects.
Accidental overwriting of a parent vhd file.
Windows Update automatically overwriting desired configurations.

The MAC address will be updated automatically by Virtual PC as long as you create a new .vmc file from the menu. In other words, use the New Virtual Machine Wizard, don't make a copy of your existing .vmc file. If you insist on copying your .vmc file, you can manually remove the MAC address from the XML file and it will automatically be recreated.
Many people are also concerned about "SID duplication" and want to use something like NewSID or SysPrep. Recent reports indicate that this is no longer a concern. Please see NewSID Retirement and the Machine SID Duplication Myth.

Create the Layer 1 Root VPC

The first task in creating the hierarchy is to create the Layer 1 VHD, which contains the OS install and will be common to all dependent virtual machines.

Create the .vhd file using the Disk Wizard. Set to dynamic size, 20 to 32GB is usually a good size.
Create the Virtual PC .vmc using the New Virtual Machine Wizard under the File menu. Make sure you give it enough RAM, the defaults are almost always too small.
Install Windows. Install a version of Windows that includes the desired service pack. Set the system name to be something like RootWinXp so you know when you've forgotten to change it. Do not join a domain during installation.
Log in as an Administrator and activate Windows, if necessary. Don't do this in a child virtual machine or you'll need an additional activation key for every child.
Shut down Windows and make a backup of the VHD. Name it "Level 1 - Clean" This is a "clean machine" that's been activated. Use this VHD to create completely new configuration hierarchies.

Configure the Layer 1 Root VPC

Boot the virtual machine.
Install the Virtual PC Additions.
Set the vpc to be part of workgroup, if it's not already. Do NOT join the root vhd to a domain! That must be done in a lower layer.
Install desired Windows service pack, if any.
For Windows XP, go to Windows Update in Internet Explorer and enable Microsoft Update.
Install latest security updates through Windows Update. Pay attention to the following:

For Windows XP, go to Windows Update in Internet Explorer and check High Priority and Optional updates. The task bar "Update" icon does not show these updates.
Under Optional, I recommend you install Update for Root Certificates.
Also under Optional, install Microsoft .NET Framework, if needed.
Windows Update may automatically upgrade Internet Explorer if you don't do a "Custom" update and uncheck Internet Explorer. OTOH, IE8 for Windows XP is under Optional.
After each reboot, go back to Windows Update and check again. You'll probably need to run through Windows Update several times to get all updates.

In Control Panel, set Automatic Updates to "Notify me", not to automatically download or to automatically install. Otherwise you end up fighting Windows Update every time you boot a configuration.
Disable the requirement for Ctrl-Alt-Del. This can save a lot of mousing around when running the virtual PC in a window. For Windows XP, this can be found under Control Panel / User Accounts / Advanced (or use Group Policy.)
Create any desired local users (as opposed to domain users.) Make sure you set them as Admin or Limited, appropriately. At a minimum, I recommend creating a "Limited" user for testing.
Disable the Internet Connection Wizard if you have Windows XP virtual machines. In Group Policy, it's under User Configuration\Administrative Templates\Windows Components\Internet Explorer\Internet Settings\Advanced Settings\Internet Connection Wizard\
Set firewall exceptions in Control Panel | Windows Firewall | Exceptions:

File and Printer Sharing must be enabled to allow access the system by name on the network.
Remote Desktop must be enabled to access the system remotely.

If desired, enable Remote Desktop under My Computer | Properties | Remote. Note that Remote Desktop is not supported in any of the "Home" versions of Windows. Local Administrators will automatically have rights to access the system via Remote Desktop.
If you use Remote Debugging in Visual Studio, start up MSVCMON.EXE and tell it unblock the firewall for your subnet. You will discover that this adds numerous exceptions to the firewall. MSVCMON.EXE can be found in:

C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\Remote Debugger\x86\msvsmon.exe
Modify options in My Computer | Tools | Folder Options | View, such as "Display the contents of system folders", "Show hidden files and folders", "Hide extensions", and Hide protected operating system files". This can also be done in the domain logon script with:

reg add hkcu\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v Hidden /t REG_DWORD /d 1 /f reg add hkcu\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v HideFileExt /t REG_DWORD /d 0 /f reg add hkcu\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v WebViewBarricade /t REG_DWORD /d 1 /f
Share any desired folders.
Shut down the root vpc using the "Shutdown" from the Start menu.
Commit all changes if you are using Undo disks.
Set the VHD to be read-only.
Create a backup of the file. Name it "Level 1 - Common".
Optionally, go to the Virtual PC Console and remove this virtual machine so you don't accidently try to use it. If you don't delete it, make sure you enable Undo disks so you don't accidentally change it (Virtual PC will helpfully remove the read-only attribute.)

Create a Layer 2 VPC

Use disk wizard to create a differencing hard disk to the root.
Make a backup of this vhd file. Name it "Level 2 - Empty".
Create the Virtual PC .vmc using the menus. Again, make sure you give it enough RAM. Turn off Undo for now.
Boot the new virtual PC.
Log in as an adminstrator.
Change the system name.
Join a domain, if desired. This can be done from the command line with the NetDom command, which is available in the Windows XP Support Tools. (This should all be on one line.)

netdom join myvpc /domain:test.com /userd:Administrator /passwordd:<domain admin password>
Give any desired domain users or groups permission to login via Remote Desktop.
Log in as any domain users you plan to use frequently. This does the one-time configuration of the VPC for that user. Change Folder Options again for these users.
Install any configuration-dependent software.
Install updates and service packs for the software you just installed.
Map network shares with a domain logon script. If a computer on a domain must map a drive on a non-domain computer, make sure the non-domain computer has an account with the same name and password as the domain computer. Alternatively, you use this command to save the username/password on the domain computer, then say "net use x: \\acme\share" in the domain logon script.
net use x: \\acme\share /savecred /persistent:yes
Check Windows Update one more time. On Vista and Windows 7, make sure you explicitly tell it to check again.
Once everything is configured to your satisfaction, shut down Windows from the Start menu so it shuts down cleanly.
Set the VHD to be read-only.
Create a backup of the file. Name it "Level 2 - Configured".

Create a Layer 3 VPC

Create a third vhd, differencing, whose parent is the vhd you just set to be read-only. I normally name it the same as the parent with the word "Child" appended. This is what I called the "Working VHD" earlier in this document. You will use this when you need to reuse a configuration a few times, then discard the configuration. Normally this vhd is empty.
Change the settings in Virtual PC for your virtual machine to point to this working file.
Enable Undo for this vhd.
Backup this third VHD. Label it "Level 3 - Empty". You'll need the backup whenever you discard your current working configuration.

At this point you should have five backup files. The files labeled "Empty" in the below should be small, between 44KB and 106KB:

Level 1 - Clean root VHD, with a clean Windows install.
Level 1 - Configured root VHD, with a mostly configured Windows install.
Level 2 - Empty configuration VHD, which is an empty differencing disk whose parent is the Level 1 vhd.
Level 2 - Configured VHD, which is your fully configured test environment.
Level 3 - Empty VHD.

Create Remaining Configurations

Copy your Level 2 - Empty vhd to a new file that will be the new configuration. Under Windows 7, you can create all of them at once using the DISKPART utility. For example: (DiskPart requires elevated privileges and so must be run from a command prompt started with Run As Administrator.)

DISKPART < Parts.txt

And Parts.txt is one or more lines similar to this:
create vdisk file="c:\vpc\ParentDisks\WinXP Office 2007.vhd" parent="c:\vpc\ParentDisks\WinXP SP3 Root.vhd"
Go to Step 3 under Configure a Layer 2 VPC and continue through Configure a Layer 3 VPCs.

Licensing

Licensing is also an issue. Strictly speaking, it appears that you need one Windows license per virtual PC. In practice, we have a dozen different variants of each root virtual PC, with only minor changes between each, such as whether it's a member of a workgroup or a member of a domain, or whether it's Service Pack 2 of WinXP or Service Pack 3. Since we rebuild these fairly routinely from scratch, it is hard to believe that we are supposed to own thirty different licenses and rebuy all of them every few months. Our solutions for Windows has been to use the volume licensing versions of Windows, which are available to Microsoft Certified Partners on MSDN Downloads.

Create separate references for Debug and Release DLLs in C++/CLR

2010-03-10T00:50:00.003-07:00

I'm one of the crazy people who has worked with C++/CLR to make native C++ code coexist with .Net code. It runs really well, with a minimum of frustration.

Except for one thing. We rely on some components supplied by a third party. These components have both Debug and Release versions. You'll notice that the References are listed under "Common Properties" in the Properties window, so you can't create separate references for Debug and Release by simply switching to the appropriate property set.

This has been annoying me now for at least two years, and I finally found a solution, thanks Marco Beninca in this post:

http://social.msdn.microsoft.com/Forums/en-US/clr/thread/9087fb4f-149f-4d66-b33e-f2f280c65fa6

The solution is to use #using in your source code instead of defining your assembly references on the "Framework and References" page. Then you can go to the General page for C/C++ and set different directories for Debug and for Release.

The other advantage to this solution is that you don't have to reset all of your references when you get a new version of the components.

One disadvantage to this strategy is that the compiler no longer takes care of copying the appropriate DLLs to the appropriate directories. This can easily cause you to build with a different set of DLLs than you are running against, which will certainly cause a crash. My solution was to create a Pre-Link step that copies the DLLs to $(OutDir).

Online Backups - The Good, The Bad, and the Ugly

2010-02-16T00:52:00.012-07:00

This is the third and last part in my series on doing backups for home and small office environments.

Online backup has come a long way in the past couple of years. Service has become more reliable and faster Internet connections have made backing up large quantities of data more palatable. My Internet connection is 5Mbps (30 MB/min) upstream, which is more than fast enough for overnight backup.

In spite of these advances, my opinion is that online backup is still best suited for last-ditch disaster recovery. Any online backup strategy should be paired with a primary backup strategy that includes an image and/or file backup to a USB drive or network drive. Buying 100-GB of online storage is quite expensive, much less 1-TB. The "unlimited" plans are attractive, but companies make significant compromises to make these unlimited plans affordable, including lower backup speeds, lack of geographic redundancy, and subpar technical support. My research indicates that cheap and reliable are mutually exclusive for online backup - the lower the price, the less reliable the service.

It should also be noted that it's the recovery that's most important, not the backup. Numerous people reported that Mozy could take several days (or more!) to prepare to restore a large number of files. Ordering a DVD was even worse. For me, this isn't acceptable.

Here's my experience with the various services.

Acronis. My first attempt at online backup was with Acronis, who introduced online backup in October 2009. My experience with it has been total failure. Acronis TrueImage 11 has never succeeded in creating a backup on my Windows Vista system. Acronis recently increased my backup space from 25GB to 250GB, all for $50/year. This seems like a safe choice for them, since I've never been able to backup more than 5GB to their servers.

Mozy I've been using Mozy Pro for about two months now. They charge 50 cents/GB/month for their Pro version. On the plus side, the nightly backup works quite reliably and I was easily able to select what I wanted to back up.

However, there are several downsides to Mozy. First, backup speed is only thirty to forty percent of the possible maximum. Mozy compresses data, then sends it. During compression, Mozy usually isn't transmitting data. When Mozy is transmitting, it only manages to max out my connection part of the time. So there's a lot of room for improvement in backup speed. [Updated 5/27/2010] The performance problems seem to have been resolved in Mozy Pro v2.0. I'm consistently maxing out my upstream connection at 5Mbps.

Second, numerous people have reported problems restoring data from Mozy, even from their Pro service. Former employees of Mozy confirm that Mozy has been having problems managing rapid growth.

Third, Mozy does not allow me to exclude specific extensions. While this isn't an issue for most people, it's a real problem when backing up development directories containing projects built by Microsoft Visual Studio.[Updated 5/27/2010] Mozy Pro v1.6 did allow excluding extensions, but it was difficult to use and the documentation was useless. Mozy Pro v2.0 makes it somewhat easier. See my article Configuring Mozy Pro for Visual Studio.

My final issue with Mozy is that it doesn't support multiple historical revisions. This is a feature that I consider very important in case local backups become corrupted (which, as I mentioned earlier, has been all too frequent with Acronis TrueImage.) [Updated 5/27/2010] Multiple historical revisions are definitely supported in Mozy Pro v2.0, but again the documentation is lacking. To view the historical revisions, go to My Computer, open MozyPro Remote Backup, open one of the drives, right-click on a folder, and select Change Time.

Carbonite I didn't try Carbonite, although I know several people who are happy with it.

Iron Mountain I will be trying Iron Mountain next. They are the most expensive, but, compared to the cost of losing the data, the price is cheap. They support multiple historical revisions, incremental block updates, Windows 7, and many other features. Iron Mountain makes backup systems for large companies, so I'm hopeful that their technology is more reliable. I'll keep you posted.

To recap, although online backup has been around for several years, most of the solutions still have significant shortcomings, especially with reliability. My research is that you get what you pay for, with no exceptions.

How I Backup My Windows System for Multiple Contingencies

2010-02-15T22:18:00.005-07:00

In my last blog posting, I described the events that have lead up to my paranoia about keeping my computer backed up. In this posting I'll talk about about how I actually keep my system backed up and the challenges involved in these methods.

These backup strategies are appropriate for individuals and for small offices. For larger environments, the reasons for doing various types of backups still hold true, but the implementation strategy will change significantly.

1. DVD Backup. My first strategy is mundane - my software development directories are backed up to DVD after every release. This is more complicated than it sounds. First, Visual Studio creates hundreds of megabytes of temporary files, including .ncb, .ilk, obj, and .pch files. There generally no point in backing up these files. In Roxio Creator I can exclude all of these file types, which makes it trivial to just drag the root directory of my development structure to the DVD. Other DVD burning applications, such as CDBurnerXP, won't automatically exclude file types, so it's somewhat more complex to use them for backups. [Update 12/21/2010: CDBurnerXP now supports excluding by extension, under Edit/Filter Files. However, you can't use this filter from the command line.]

I've been careful to use high quality CDs and DVDs for these backups. I have CDs I burned over ten years ago that I can still read without difficulty. My recordable DVDs are newer, so the jury is still out on how long they'll last. The important thing to remember about CD-R and DVD-R is that, unless you buy archive-grade media, they shouldn't be expected to last more than a few years.

DVD and CD backup requires a substantial amount of effort. You have to remember to do it, it's relatively time consumering (especially if you validate the media after burning), and you have to keep all of the media labeled and filed. So it's only a good streategy if you are quite disciplined. However, DVD backups are much easier to move offsite and you have a permanent record, especially if you need an audit trail.

2. Image Backup. Image backups are the cornerstone of my backup strategy. If my computer total fails, due to a virus, a lightning strike, or user error, the image backups are what allow me to recover in a few hours instead of a few days.

For the last eight years I've created my image backups with Acronis TrueImage. This has been a mixed blessing. On the plus side, Acronis can do incremental image backups, which saves a lot of disk space. Acronis has a standalone boot CD, so you can recover the system from a raw hard drive. Acronis can do backups over the network, even when booted from the recovery disk. All of these have been invaluable.

On the minus side, Acronis has had significant difficulty with consistent quality. Their latest release, Version 11, is a total disaster and has been completely unusable for many people. Acronis also has a nasty habit of creating corrupted backups, which means that a validation phase is required.

I do image backups once a week, supplemented by file backups that are done daily. I try to keep three months of image backups because some viruses can take weeks to trigger.

[Update 12/21/2010: Because of ongoing stability problems with Acronis, I have stopped using it.]

3. File Backup. Acronis TrueImage can also do file backups. I backup my development directories and documents on a daily basis. This usually comes out to less than 5-GB, so I can store a lot of them on a 1-TB backup disk. Again, file backups are done incrementally, so the baseline file is dramatically larger than the daily incremental files.

My big problem with Acronis is the poor support for rotation management. I want to keep backup snapshots every 30 days for the last six months, every week for the past month, and every day for the past week. Acronis provides several different options, none of which can do what I want, and all of which seem to cause the frequency of corrupted backups to increase dramatically. [Update 12/21/2010: Macrium Reflect also has better support for managing archive rotation.]

File backups provide protection from accidental deletion and provide the ability to revert to an older version of a document.

4. RAID 1 Mirror drives I mentioned this in my last blog post, but it bears repeating. I use mirrored drives to handle the case of catastrophic hard drive failure, which has happened to me multiple times before. One of the two mirrored hard drives can fail completely and it won't affect the performance of the system. [Update 5/30/2010 - This happened to me again today. A hard drive failed in my primary development system. Without the RAID 1 pair I'd be looking at restoring from backup right now.]

I use software RAID (Intel Matrix Storage), which definitely has its downsides. Every time you reboot without cleanly shutting down the system, (such as a power fialure or hitting the Reset button,) the drive mirrors have to be validated. This takes six to ten hours, during which the system is almost unusable.

Hardware RAID solutions with battery backup can fix this problem, but such RAID hardware is dramatically more expensive and generally is not compatible with image backup software such as Acronis.

Note that not all RAID strategies provides protection. RAID 0 (striping) actually DECREASES reliability, because if any drive fails, your data will be completely lost.

5. Online backup. My final strategy is for last ditch disaster recovery - theft, fire, natural disaster. I'll talk about it in the third and final part of my blogs about backup.

Six Myths About Backing Up Your Computer

2010-01-23T22:56:00.014-07:00

This is my tale about how I got religion about backups. I played a part in all of the stories below. The amazing part is not that any one or two of them happened around me - the amazing part is that they all happened around me.

Myth #1. I have a new hard drive, my data is safe.

Five years ago I helped a friend replace his hard drive after his original drive failed and he lost everything. So he bought the new drive, spent two or three days reloading everything, and went on his way. Three months later, the new drive failed too, and again he lost everything. My friend is an optimist. He believes he's had all the bad luck he's going to have and he still doesn't do backups.

A few years later, I was overseeing putting a new server in the office and had paid a consultant a lot of money to set it up to our satisfaction. The server was backed up after the OS was installed, but before the consultant made changes. After the consultant was finished, we went home for the weekend. When we came in on Monday, the hard disk had failed. All of the money we paid for the consultant was wasted. Even more embarrassing, the CD for the backup software was sitting on top of the server.

The Reality: Hard drives have their highest failure rate when they are new.

Myth #2. Backing up once or twice a month is enough.

My neighbor conscientiously made weekly backups of his laptop. He had decided that losing a few days of work wouldn't be a big deal. Inevitably, his hard drive died hours before a sizable document was due to his largest customer. He had a four day old backup, but he'd done well over thirty hours of work on the document in the interim. I was able to recover the data with low-level tools, but recovery companies charge hundreds to thousands of dollars for that kind of work.

The Reality: Murphy's Law should never be underestimated. Nightly backups are a minimum, and software for continuous backup is easily available.

Myth #3. Backing up documents to an external drive is good enough.

Two years after I helped my optimist friend above, the primary hard drive abruptly failed in our corporate ecommerce web server. We had a backup of the database and other data, but we still had to reconfigure the operating system from scratch. It took four days to get the server running again, which was basically four days of lost sales. I learned a harsh lesson, since it cost thousands of dollars in sales and made a lot of existing customers unappy.

The Reality: To quickly recover from a catastrophic failure, you must have image backups, not data backups.

Myth #4. Having an image backup on an external drive or a NAS is good enough.

I have now been in two different offices that were burglarized and the computers were stolen. If I'd had a USB hard drive sitting on the shelf, it would have been taken too. Even if you put the backup in the safe, it just takes one flood, fire, tornado, or hurricane to destroy the sensitive platters of a hard drive.

I've also known people who whose office was served with a search warrant. The content on computers is just assumed to be relevant, so officers often take *everything* that looks like a computer or a hard drive - including what's in the safe. If you don't have off-site backups, you're done because you may not ever get that equipment back.

The Reality: You must have off-site backups.

Myth #5. I use RAID hard drives so I don't have to do backups.

My primary computer had mirrored hard drives installed right from the start. Six months later, my computer contracted the second virus I've ever had. However, RAID is protection against disk failure, not against data loss that was intentionally caused by a virus. I recovered from the failure by reloading the entire system from image-level backups, as I described in #3.

As I became increasingly paranoid about hard drives failures, I upgraded our corporate web server with a hardware RAID card and enterprise-grade SCSI hard drives to allow RAID 1 drive mirroring. SCSI hard drives generally cost several times the price of a similarly-sized IDE (consumer) drive because they are expected to be substantially more reliable.

A week after the new hardware went live, I started seeing I/O errors. Our expensive new RAID controller board had malfunctioned. As a result, it corrupted the SQL database that contained the customer support system, which contained FIVE YEARS of customer support information, including an extensive knowledgebase. The primary backup was completely missing due to a configuration error in the backup script. I ended up staying up all night to restore the database from a third-level backup that I'd been paranoid enough to perform.

The Reality: If you don't have 100% redundancy, including spare computers ready to go, downtime and possible data loss are just one hardware failure away. Also, RAID provides zero protection again viruses and user error.

Myth #6. Paying experts to do backups is good enough.

After we upgraded to RAID hard drives, we purchased Managed Backup for our corporate server. This means that experts are responsible for keeping our server backed up, and they are responsible for recovery when things go wrong.

Last summer one of our web server log files was erased accidentally. These files are critical for financial analysis and prediction, so I was pleased that we had the experts managing our backup. Except for one thing. The "experts" considered the log files low value and a waste of space, so the file I needed had not been backed up. Damn.

The Reality: Backups that haven't been verified and tested aren't backups.

Conclusion

You can't be too paranoid about backup. I have four levels of redundancy in my backups, but I still worry that it won't be good enough. If your computer contains critical information, whether it's corporate documents, baby pictures, legal documents, graduate thesis, or financial records, do yourself a favor and back it up, preferably in multiple places.

In Part 2, I'll look at how I've implemented my tiered backup strategy and what software I use to make it happen.

Installing PHP on Windows 7

2009-11-19T11:15:00.001-07:00

If you've wrestled with installing PHP on IIS before, there's good news. Installing PHP in Windows 7 is easy. Super easy. Microsoft has created the Web Platform Installer (WPI), which supports installing many things, PHP included.

If you look in the IIS Manager, down at the bottom of the rows of icons you should see Web Platform Installer. If you don't see it there, go to your Start menu and search for "platform".

Check off PHP and let it install. There are a couple of dependencies that will be added automatically.

There's one more change you need to make after PHP is installed, which is to add "index.php" to the list of Default Documents.

There are some other things you should know:

WPI correctly installs PHP even on a 64-bit system. There's no need to use cscript to separately change configurations.
Apparently, there's still no 64-bit version of PHP for Windows. WPI installs the 32-bit version, which works correctly even under the 64-bit version of IIS.
There's no uninstall option. However, everything is installed to a single directory and removing the relevant FastCGI settings is straightforward.
The correct strategy for upgrading to new versions of PHP isn't obvious. I'm not sure if Microsoft used the stock build of PHP or if they created a custom one.

One small gotcha - WPI installs PHP 5. If this is your first encounter with PHP 5, you need to make sure you use <?php ... ?> as opposed to <? ... ?>.

Handling Exceptions from STL

2009-11-10T00:08:00.002-07:00

Recently I tracked down a crash to an out of bounds index in a STL vector. The strange thing was that the crash was being handled by Windows instead of by our error reporting code, which is designed to trap everything that the explicit exception handlers did not catch.

Obviously it wasn't.

I created some short test code:

vector vec;
vec[5] = 3;

When run under the debugger, I receive a helpful window that says "vector out of range" with the standard Abort/Retry/Ignore debugger options. All very well and good, but there was no exception being thrown.

I tried using set_unexpected(), which turns out didn't mean what I thought it meant. The function set_unexpected() is called when an exception is thrown through a function that doesn't declare that exception in the throw() statement in the function declaration. Visual C++ does not support these declarations, and so does not support set_unexpected(). This is discussed in the documentation.

So I tried set_terminate(), which is the only other function under Exception Handling Routines that seemed relevant. Unfortunately, my termination function was never called. Since my two line program certainly was not catching an exceptions, something else was happening.

I tried running the code again as a Release build instead of a Debug build. It still broke to the debugger, but in a separate place than the Debug build.

I tried reading the documentation, which makes no mention that operator[] can throw an exception.

Stranger and stranger. I tried walking the stack with the Release build, which looked like this:

msvcr90.dll!_crt_debugger_hook
msvcr90.dll!_invalid_parameter
msvcr90.dll!_invalid_parameter_noinfo
Exc.exe!wmain

The function _invalid_parameter() had this comment:
// No user handler is defined. Notify the debugger if attached.
So I tried running the Release build without the debugger. Which also yielded no useful information.

I don't remember exactly how I figured out what was happening, but it turns out that STL is throwing "out_of_range", which is derived from "logic_error", which is derived from class "exception". I've been using STL for twelve years. I've written articles about it. And I've never heard of these exception classes. Clearly, the authors of the documentation had never heard of them either.

Note that this behavior is different than other implementations of STL, which specifically document that only the at() functions throws an exception, not operator[]. At least "out_of_range" is defined in the standard.

[Update 8/13/2010 Visual Studio 2010 behaves according to the standard. In Release builds, vector<>::operator[] does not do bounds checking and does not throw the out_range_range exception. In Debug builds, an assertion is shown because _ITERATOR_DEBUG_LEVEL is set to 2 in yyvals.h if _DEBUG is defined.]

But the fun wasn't over. I still didn't know why set_terminate() wasn't working. The documentation for Unhandled C++ Exceptions only discusses set_terminate().

It turns out that unhandled logic_error exceptions are handled by invalid_parameter_handler and not by terminate_handler. This is broadly discussed in the documentation for Invalid Parameter Handler Routine. However, that documentation describes how errno is set, which doesn't appear to happen with STL.

The situation gets even more murky if you consider Windows SEH (Structured Exception Handling) as well as C++ exceptions. The standard way of dealing with SEH is to use SetUnhandledExceptionFilter(). However, Visual Studio 2008 changes this behavior. See a blog entry by Jochen Kalmbach. Related information about C-Runtime behavior is discussed in the MSDN forums.

Router for 50Mbps Broadband Service

2009-10-29T11:19:00.001-07:00

In my last blog entry I described how my WRT54G was the bottleneck in my new 50Mbps broadband service. Finding a replacement has ended up being substantially more difficult than I expected.

A point of confusion is that there are numerous different speeds. There's the speed of the wired ports, which is generally 10/100/1000. Then there's the speed of the wireless connection, generally 11/54/108/???. Then you have the WAN to LAN speed, which is the speed at which the router can actually route packets. This has *nothing* to do with the numbers above. Most older routers can't route more than 20 to 25Mbps, at which point they max out and become unresponsive. The routers that seem to be designed to have the best WAN to LAN performance are the "gaming" routers.

Very few routers include the specifications for the WAN to LAN performance. Of the ones that do, the specification is usually taken with important features like SPI (Stateful Packet Inspection) turned off. One Cisco router advertises 800Mbps WAN to LAN performance. What they don't tell you is that performance drops to 20Mbps (a 97% drop!) if you turn on IPS (Intrusion Protection System.)

The only reference I've found is the chart at smallnetbuilder.com. Make sure you select WAN to LAN Throughput. Less than 15% of the routers listed break the 150Mbps barrier. I chose this number because 100Mbps broadband is coming and I want some horsepower left over for features like SPI.

So if we look at the top routers, we learn that almost all of them are over $100 and many of them are over $150. This is a pretty big jump over the $40 routers that litter the bottom end of the list. A careful review of the routers on Amazon and NewEgg shows that many of these routers have serious problems. For example, the D-Link DIR-825 has been out for over a year and has achieved five stars on NewEgg with just 35% of reviewers. The somewhat better rated Linksys WRT600N is no longer for sale, and its replacement, the WRT610N, took a 20% performance hit and also falls to just 35% for five star ratings.

The D-Link DGL-4500, one of the "gaming" routers I mentioned earlier, has a more respectable 55% of five star ratings, has been out for two years, and costs $150. Personally, I've had several bad experiences with poor D-Link firmware in the past, so this isn't my first choice.

You might think I'm just being picky looking at the number of 5 star ratings, but the LinkSys WRT54GL router is a prime example of doing things right. After 2500 reviews, it has 84% five star ratings, and this router is prized by the hard-to-please hardcore techies.

At this point I'm leaning towards the Netgear WNDR3700. It's only been on the market for a couple of months, but has garnered 72% of five star ratings from the early adopters - impressive in an industry that usually requires a year of firmware updates to get things right. I've had troubles with Netgear in the past, but this whole situation seems to be be a matter of choosing the best of a dubious lot.

One router that hasn't shipped yet is the WNR3500L. It's based on open source and is generating a lot of buzz, but you can't get one yet. It's worth watching.

If you try to look at the cheaper units, the ratings become even more disparate. There are many more 1 star ratings for the LinkSys WRT120N than there are 5 star ratings. The Belkin N1 Vision F5D8232-4 suffers from similar ratings. The D-Link DGL-4300 is highly rated, but it runs hot. That makes sense - it's four years old, absolutely ancient technically.

Lastly, if you have Macs in-house, the AirPort Extreme seems to generate universal admiration. The MB053LL/A model scores well in the chart mentioned above, but there are several other models that aren't listed. The AirPort can be used with Windows, but it's much easier to configure with a Mac.

Cox Ultimate Broadband - 50Mbps

2009-10-27T23:20:00.004-07:00

I dumped my old ISP and signed up for Cox's "Ultimate" broadband package, with speeds up to 55Mbps downstream and 5Mbps upstream. This is the bleeding edge, and getting it working at full speed is tricky.

Amazingly, Cox was able to get the DOCSIS 3.0 modem installed and running on the first try. Given that this technology is new for Cox, I was quite surprised.

So I fired up Speedtest.net and obtained:
22 Mbps download
9 Mbps upstream

Astute readers will notice that this is half of what I was expecting downstream and almost twice what I was expecting upstream. Strange.

I'll save you the details of four hours of sleuthing and simply present my results:

The Router.The problem is almost entirely caused by my WRT54G router running DD-WRT v24-sp1. The router maxes out at 22 Mbps. If one IP connection is running at that speed, the WRT54G won't even accept wired http connections to the status page. I tried installing TOMATO, and that maxed out at 28.5Mbps. A respectable improvement, but far short of what I needed. Discussions on dslreports.com indicate that my results are typical and that the WRT54G is simply too slow to meet my needs.
Buffer management.You might think a fast Broadband connection would be just like a 100MB local Ethernet connection, but it's not. Local Ethernet connections typically have submillisecond latencies. Broadband connections can easily have 200ms latencies, which means that over a megabyte of data can be transmitted before an ACK is received. This is a major change in how buffers are managed in the Ethernet stack. I found that tuning for 50Mbps broadband required parameters very similar for max throughput for Gigabit Ethernet.
RWIN and MTU.Windows Vista and Windows 7 automatically tune IP parameters, so I didn't need to adjust the RWIN, MTU, or other parameters. However, Windows XP and earlier users will almost certainly need to hand tune parameters to make things work properly. Verizon has a Speed Optimizer tool that does this automatically, but Cox does not. Don't use Verizon's tool - Verizon uses an MTU of 1492 and Cox uses 1500.
Wireless Connections.If you are using 802.11g (as most people do), your connection maxes out at about 20Mbps. (Maybe 25Mbps if your wireless connection is perfect.) Buying 28Mbps or 50Mbps service is simply a waste of money. You need to upgrade to 802.11n to run full speed.
Testing. 50 Mbps is fairly slow for a LAN, but it's the bleeding edge for consumer WAN technology. Most web sites simply can't service data that fast. Speedtest.net won't go that fast if it's busy. I can only download at about 10Mbps from my corporate server, which has a 100Mbps connection to a backbone. This is probably a TCP tuning problem on the RedHat server, but this demonstrates that both Windows and Linux have default connections that are not well suited to this configuration.

[Update 2/26/2010] Here is the speed test result from the iMac in the office. This was performed without any tuning of OS X:

Porting C++ Applications to Visual Studio 2010

2009-10-24T13:44:00.007-07:00

[This blog entry was updated on 4/12/2010 to reflect the final release of Visual Studio 2010.]

I've now spent a couple of days porting my C++ applications to VS2010. Here are the top ten things I've learned.

1. STL
If your app makes significant use of STL, prepare to spend some time making it work in VS2010. The new version of STL has been significantly reworked to take advantage of rvalue references and static asserts. I spent several hours reworking some of my derived classes to update the supported operators.

2. Boost
Make sure you get version 1.42 of Boost, which includes support for VS2010. Thankfully, Boost 1.42 has not yet been updated with C++0x features, so porting was relatively straightforward.

3. Property Manager
Earlier versions of Visual Studio used the Property Manager to manage certain settings like optimization and DLL usage. Visual Studio 2010 dramatically expands the use of the Property Manager. If you created your own pages in the property manager in an existing project, you'll find that VS2010 migrates the information to .props files instead of .vsprops. The files use different formats, although they are both XML. Don't forget to check these new files into source control. I had trouble with parameters that were lost when the project was migrated, including Additional Include Directories, Additional Libraries, and Delay Load Libraries. Make sure you compare your VS2005 or VS2008 property pages with the equivalent pages in VS2010.

4. Target Name
VS2010 introduces a new "Target Name" property in the General page of Configuration Properties. I still haven't figured out the rationale, but you need to make sure your Target Name matches the Output File in the Linker or Library pages. (Note that Target Name should not have the extension included.) One symptom of this happening is that you try to run your application and the application is not found, even after building correctly.

5. Default Directories
For the last several releases, the default Include and Library search paths were found under Tools/Options. Now they are on the General page of the project Properties, under "VC++ Directories." This is all managed from the Property Manager, so you can override the default directories for a single solution by replacing Microsoft.Cpp.Win32.User (or Microsoft.Cpp.x64.User) with your own custom page. This is a big win over earlier versions of Visual Studio, where the default directories were a global setting that couldn't be overridden.

6. Platform Toolset
You can use the Visual Studio 2008 compiler/linker under the VS2010 IDE, which allows you to get the improved user experience without having to port your code. However, if you are still stuck on VS2003 or VS2005, this won't help you.

7. Target CLR Version
It's possible to target versions of the CLR other than 4.0, but you can't do it from the IDE. See my related blog post for instructions.

8. Show Inactive Code Blocks
The editor is now much more successful at detecting Inactive Code Blocks. Historically this has been a problem because commands such as "Go to declaration" don't work in an Inactive Code Block.

9. MFC Application Size
The size of all of my MFC-based applications (statically linked) has grown by about 1.5MB. I'm not very happy about this. I double-checked all of my build settings and they are correct. A similar question on the Visual C++ Team Blog was given the answer that "The increased capabilities of MFC have introduced a number of additional interdependencies that end up causing more of MFC to be pulled in when linking a minimal application that links with the static MFC libraries." So this problem may not be solvable.

10. Intellitrace
IntelliTrace is the "historical debugging" tool that records the guts of your application as it runs. Unfortunately, IntelliTrace doesn't work for C++. Bummer. It also requires the Visual Studio Ultimate, which isn't available to Microsoft Partners. Double bummer.

11. Range-based "for" loops
One of the features I was particularly looking forward to was support for a compiler-based "foreach" syntax, which made it much simpler to write loops based on STL containers. Unfortunately, Visual Studio 2010 doesn't include this feature because the committee standardized is too late in the Visual Studio beta process. More information can be found in the comments section from this blog entry on the Visual C++ Team Blog.

Visual Studio 2010: Review/First Impressions for C++

2009-10-21T01:38:00.003-07:00

I downloaded Visual Studio 2010 Ultimate Beta 2 and installed it on a system with Core 2 Duo 3GHz, Windows 7 RC, and 4GB RAM. Here are my first impressions.

First, the scope of this product is huge. There are so many features, it's like counting the stars in the sky (If you live in the city and can't see any stars, take a look at the Hubble Deep Field to see what I mean.) Once upon a time, you could classify a Visual Studio user as C#/VB/C++, with maybe some database work thrown in. Now we have XAML, Azure, SOAP, SharePoint, IIS, HTML, XML, and much more. The audience that this product caters to is diverse and far-reaching.

I'll limit my discussion to the C++ features, since I primarily do development in C++, MFC and ATL. The "gold standard" for developing in those technologies was Visual C++ 6.0 (VC6), which ran far better on a Pentium II 350 than Visual Studio 2008 ever did on a Core 2 Duo running ten times faster. VC6 was built by and for C++ development, and it was a pleasure to use. I didn't drop VC6 until I was finally forced to use VS2005 to support Vista.

So here are my first impressions:

VS2010 is much, much faster than VS2008. Opening the Server pane is now instant, instead of ten seconds or more of thrashing. Everyone was worried about the performance of the UI with the WPF rewrite. Everything I see related to performance is thumbs up.
Multiple core parallel compiles are now the default, instead of a hidden teaser like they were in VS2008.
The C++ editor is finally able to parse TODO and HACK tags in comments. The C# editor has been able to do this since VS2005.
The Help system works. Finally! The Help system last worked properly in VC6. Since VS2003, the help system has been bloated, insanely slow, and almost random in results that it returned. "Help 3.0" in VS2010 returns answers almost instantly. The help system is implemented as an http server running on the local system. When I search on Windows SDK calls, I actually get the result I want instead of useless Sharepoint, .Net and DDK results. This alone makes VS2010 worth the price of admission.
Help pages have been reformatted and are far more legible. However, it's not clear what will happen to the Community Content from VS2008.
The UI has some cosmetic changes, but the toolbars are basically unchanged. This is a relief after prior releases rearranged everything.
Many user interface stupidities have been fixed, like the resource browser closing every time you opened a resource for editing.
The resource editor allows you to put in an alpha-blended mockup for reference when editing dialogs. Cool.
The resource editor is still lame. Still no in-place editing of text.
The dialog editor has problems with locking up for ten seconds at a time. Hopefully this is just a Beta problem. Also, there is no context menu when right-clicking design elements.
Trying to search Help for #pragma still doesn't work.
New language constructs, like lambda functions and auto declarations, make functional programming much easier.

I haven't been able to test the size of generated code yet. I'm still trying to get my main projects to compile. There are breaking changes in STL, attributed ATL, and the build system that are causing me some rework.

I'm excited to try:

The Parallel Patterns Library (PPL). This will be a huge step forward in making use of multiple cores in C++. I've used the .Net Task Parallel Library in C# and was very impressed - it has some fantastic ideas behind its development.
The unit testing features, which appear to have been expanded since VS2008.
The "Basic" install of Team Foundation Server, which should let mere mortals use TFS without having the overhead of specialized servers.
Numerous other goodies that I'm still discovering.

The other piece of good news is the VS2010 Beta 2 is supposed to have a Go Live license, so you can ship code that it produces. Since the final release won't be until at least March, this makes it easier to start using new features.

NUnit Unit Testing with C++

2009-10-12T10:38:00.002-07:00

I switch back and forth between C++ and C#. When doing C# development, NUnit rules the day for unit testing. Whether I'm doing automated tests from the command line or using the GUI to run selective tests (show below in a screenshot from SourceForge) NUnit is a pleasure to use.

If you've ever tried to run unit tests for C++, the landscape is much less appealing. C++ does not have a reflection API, nor does it have attributes that are embedded in the executable code, so it's much more tedious in C++ to do all of the housekeeping to initialize the framework and it's much more difficult to integrate external GUI tools. In short, unit test in C++ is a sub-par experience compared to more modern languages.

But I have good news for you - it's possible to use NUnit with C++. There are two minor caveats:

You must be using Microsoft Visual Studio 2005 or later (sorry MinGW and cygwin users.)
The code to be tested must either be a DLL or a static library. If you need to test code in an .EXE, you should factor that code out into its own static library.

The secret is to place your tests in a separate DLL compiled for C++/CLI, which you enable on the Visual Studio C++ properties page in the General properties, immediately under Configuration Properties. Under Project Defaults, set Common Language Runtime support to "/clr". Don't use any of the other variants - they won't work for this task.

If you haven't built this kind of project before, a C++/CLI project is a curious hybrid that contains all of the power of C++ as well as much of the power of .Net. (Access to certain features, like LINQ, is not available in C++/CLI.)

With a C++/CLI project, you can use NUnit attributes for classes and member functions, just like in C#. When you run NUnit, your target is your test DLL, which implicitly loads either your static library code or your DLL code.

This strategy can also be used in the test environment build into Visual Studio if you have the Professional edition or better.

CPAN error: Recursive dependency detected

2009-09-25T11:19:00.002-07:00

Yesterday I updated my RedHat Enterprise system with the latest security fixes, after which my primary Perl scripts stopped working.

I went into the CPAN shell and started with 'install Bundle::CPAN', which yielded the following errors:


Recursive dependency detected:
Test::Harness
=> A/AN/ANDYA/Test-Harness-3.17.tar.gz
=> File::Spec
=> S/SM/SMUELLER/PathTools-3.30.tar.gz
=> Scalar::Util
=> G/GB/GBARR/Scalar-List-Utils-1.21.tar.gz
=> Test::More
=> M/MS/MSCHWERN/Test-Simple-0.94.tar.gz
=> Test::Harness.
Cannot continue.

I found several other references to this problem, none of which provided a solution for me.

I traced the problem back to List::Util. One low-level module required version 1.21 (which was installed), but gave an error that version 1.19 was installed. I traced the problem to the fact that my Perl installation had two separate directories where List::Util was stored:


/usr/lib/perl5/5.8.8/List/Util
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/List/Util

The first directory contained version 1.21, the second directory contained version 1.19.

I solved the problem by removing the multi-thread directories for List and for Scalar:


/usr/lib/perl5/5.8.8/i386-linux-thread-multi/List
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/Scalar

After this, I was able to install Bundle::CPAN.

PathMatchSpec Problems

2009-08-31T13:18:00.003-07:00

Today I was debugging a problem in my application where a wildcard name failed to match. I traced the problem to significant issues with the implementation of PathMatchSpec() in the Windows API.

In short, do not expect this function to work like the command interpreter Cmd.exe. This function does not handle many boundary conditions, nor does it properly handle empty extensions.

The simplest example is this command, which correctly finds the Windows directory on all versions of Windows and MS-DOS 6.x:

dir c:\windows.*

However, this API call returns false on Vista:

BOOL b = ::PathMatchSpec("C:\\Windows", "C:\\Windows.*");

Other inconsistencies are shown in the table below. The "right" answer to these scenarios is unclear because MS-DOS did not support long filenames or spaces in filenames. Although various versions of Windows are themselves inconsistent, PathMatchSpec() does not agree with any of them. I would argue that the correct behavior is what Windows Vista does.

Command	::PathMatchSpec("C:\\Windows", xxx);	MS-DOS	Win 9x	Win NT
dir c:\windows.*	False	Displays directory name	Displays directory name	Displays directory name
dir c:\windows.	False	Displays directory contents	Displays directory contents	Displays directory contents
dir c:\windows...	False	Displays directory name	Displays directory contents	Fail
dir "c:\windows " (Note the trailing space)	False	n/a	Displays directory contents	Fail
dir "c:\windows ." (Note the trailing space followed by a period.)	False	n/a	Displays directory contents	Fail

And yes, I did actually install MS-DOS to create this chart :-)

Setting the default Windows SDK

2009-08-27T09:23:00.005-07:00

Every time I install a new Windows SDK (or worse, a new copy of Visual Studio), I've gone through the painful process of updating all of the project directories for the Windows SDK include directory, lib directory, etc.

Visual Studio 2005 and 2008 are both smart enough to look in the version of the Windows SDK included with those compilers, but I'd never found a way to change the default version. Until now.

The Windows SDK comes with a utility called the Windows SDK Configuration Tool. You can find it in your Start menu.

When you run this, you can set the default SDK to be whichever version you want. Then Visual Studio will automatically reference that version of the SDK without any need to manually update project directories.

[Update 8/31/2011]
Visual Studio 2010 does not seem to pay attention to the Configuration Tool. Instead, this appears to be set on a project-by-project basis in Configuration Properties/Platform Toolset. After doing so, you may be able to fix some schizophrenic behavior by updating the MSBuild information too. Take a look at the following registry entries. (Thanks to the tip from http://geekswithblogs.net/rob/archive/2010/09/17/integrate-the-windows-sdk-v7.1-with-vs2010.aspx)

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\ToolsVersions\4.0

FrameworkSDKRoot (REG_SZ)
- $(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A@InstallationFolder)

SDK35ToolsPath (REG_SZ)
- $(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx35Tools-x86@InstallationFolder)

SDK40ToolsPath (REG_SZ)
- $(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx40Tools-x86@InstallationFolder)

Dangers of Buying Laptops Parts on eBay

2009-07-24T22:46:00.001-07:00

I've bought several laptops on eBay. All of them were Thinkpads coming off of lease, and all of them have worked well. My most recent purchase was a Thinkpad T42/p to run Windows 7, which, as I mentioned in an earlier post, has been very successful.

However, a five-year old Thinkpad T40 died on me recently. I bought some parts on eBay to try and fix the problem, and it's been an eye-opening experience.

Part #1 - New motherboard. Included one week warranty. Motherboard worked, but intermittently. By the time I figured out what was actually wrong, the warranty was expired. If I was more savvy fixing Thinkpad motherboards, I would have caught the problem earlier, but I ended up getting stuck with a $130 piece of junk.

Part #2 - New display. Billed as being in "Excellent Condition." Actual condition - display worked, but had significant damage to the bottom of the display that was visible once you turned the display on. However, these displays are relatively large (not like a DIMM module, for example) and paying the postage to send it back would have cost more than the display was worth. It turns out that PayPal doesn't care about fraudulent sellers - it's always the buyer's fault. Apparently, you can send somebody a 50 pound box of bricks instead of the laptop, and the buyer will still be responsible for paying the return postage in order to make a claim. PalPal refuses to talk to you unless you have the tracking number for the returned item.

Part #3 - Laptop, bought for parts, advertised as not booting. Result: seller lied about just about everything (except the part of the laptop not booting.) The battery wasn't new, it was five years old. The DVD was bad. The hard drive and the memory were both half of the advertised sizes.

All in all, I've had a 100% failure rate on all of the parts I ordered, even though all of these sellers had positive feedback of 99% or better.

A sample size of three isn't very large, but given my 100% success rate buying complete laptops and 100% failure rate buying parts, I'd place a strong warning on buying laptops parts via online auction.

Why I Custom Build My Computers

2009-05-17T22:10:00.018-07:00

For the last fifteen years I've built all of my own computers. The recipe has been straightforward:

Intel CPU at the "elbow" of the curve for price/performance.
ASUS motherboard with good fan control.
Hard drive recommended by StorageReview.com on their Leader Board.
Zalman CPU fan.
RAM with good reviews on NewEgg.
Namebrand power supply. For the last five years it's been a SeaSonic.
A fanless video card, usually an XFX nVidia card. (Nothing against ATI, I just know the nVidia quirks inside and out.)

The most important features to me are reliability, fan control and BIOS control. Reliability because I make my living based on these systems and BIOS control so that I can set the system up without compromise. I don't overclock, but BIOS control can be the critical difference when faced with non-Windows operating environments (which are often used by disaster recovery tools.)

Fan control is a biggy. I really, really hate fan noise. Only in the last few years of manufacturers started to work on this as computers have become home theater accessories. Fans have been made quieter and have been put under the control of the motherboard to slow them down as the cooling requirements drop.

I've ordered the last couple of new systems from EndPcNoise.com. They follow my "recipe" for everything except the hard drive. They also assemble it, test it, route and tie all of the cables, and add additional sound damping devices as requested. The systems I've bought from them have been the best I've ever owned. (I have no financial interest in this company, I'm just a happy customer.)

Last week I bought a used computer system for our test lab. Most of the lab computers are cast-offs from developers, so most of the lab systems follow my recipe. However, we needed a modern system for 64-bit testing, so I bought a used Gateway FX7026, a mid-range consumer system.

My expectations were appropriate for this system. I expected comparatively louder fan noise, poor documentation and shovelware installed with the operating system. I'm pleased to say that I was right on all of these. Unfortunately, things went downhill from there. I was reminded in no uncertain terms why I don't buy systems from vendors like Dell, HP and Gateway.

The first problem is figuring out what's in the computer. The documention doesn't really discuss it. As an example of someone who does this right, if you type a system's serial number into the IBM/Lenovo site, you'll be given all of the relevant build information for that particular system. Gateway doesn't give you any of that, so you have to find the components by searching the web or by tearing the system apart and trying to read part numbers.

The second problem is updating drivers. My definition of "simple reinstall" also comes from the IBM/Lenovo Thinkpad. Boot the "Restore CD." Walk away for 90 minutes. Come back, install the IBM System Update utility, let it install the latest required drivers. Install updates for Windows. Done.

Gateway used to have such a utility. They still recommend using it on all driver download pages. But the utility is no longer supported and does not work with any system built since 2004. So you have to manually go through the Downloads page, download each update, extract it, install it, reboot, and move on to the next update. Elapsed time - several hours. And you'd better have a second computer to help you with this because the drivers for the network chip are not built into XP.

I avoided most of these problems by installing Windows 7 RC, which has a remarkable inventory of drivers built into. There were only two red X's in the Device Manager after installation.

After Windows 7 booted, I was surprised to look in Task Manager and see that the network card maxed out at 100Mbps. This was a surprise because I had researched the G33 motherboard before purchasing the system and all models of the G33 include gigabit networking. I'd find out the cause shortly.

Next I tried to install the Windows 7 update that enabled Virtual PC and Windows XP. Except that the update refused to install, complaining that Virtualization extensions weren't supported. I knew that the processor supported them. This was a critical issue and I had read Intel's spec sheets before buying the computer.

I learned that Virtualization extensions required BIOS support. No problem, I'd get the latest BIOS from Intel. I had specifically bought this computer because it had a standard Intel motherboard. Gateway refers to the motherboard model as "Shroedoer Town" but doesn't list the exact model. I eventually found out from a handy article that it's a DG33SXG2. Shrewd readers will note that there is no such motherboard on Intel's site.

And here's the final insult. Gateway took Intel's bottom-of-the-line G33 motherboard - and detuned it with cheaper hardware! Gateway removed the GigE ethernet. They removed the ability to use standard Intel BIOS upgrades. They removed the support for virtualization extensions. And Gateway had the temerity to call it a "mid-level" system. I call it cheap.

So Gateway has now made sure that they will never, ever get more business from me. Poor documentation, poor hardware, poor driver updating - there really isn't a lot more to get wrong.

For me, this is a sad thing to see. The first computer I bought after I graduated college was a Gateway 486. I used the famous cow box as a coffee table in my apartment. The monitor was the very first 15" monitor on the market that was "affordable." There was a lot of innovation in that system and it lasted me for years.

P.S. I never did get the fan controls on the FX7026 to work. That's a standard feature of the G33 motherboard, so I can't tell if Gateway broke that too or if there were other factors at play. I also couldn't find any third party software, including SpeedFan, that supported the G33 fan and temperature controllers. Even Intel's software didn't work because I was running a 64-bit operating system.

AppCrash in StackHash_1703

2009-04-15T10:05:00.003-07:00

I've wasted an hour solving this one. Here's the symptom. Your application crashes on Windows Vista almost immediately after it starts. If you look at the "Details", you see something like:

Problem signature
Problem Event Name: APPCRASH
Application Name: xxxxx
...
Fault Module Name: StackHash_1703
Fault Module Version: 0.0.0.0

The problem is that there's no such module as "StackHash_1703", so this appears to be some special case in Windows. I tried turning off antivirus and enabling compatibility mode, but the application still would not work.

The problem was that I had DEP (Data Execution Protection) enabled. For whatever reason, the error message gave me the AppCrash error above instead of the standard message about "DEP has closed the program."

To solve the problem, I added the application to the DEP exclusion list and everything worked again.

If the application that's crashing is video related (such as MovieMaker or Media Player), then your problem is probably caused by an old versions of Nero or an old video Codec.

Dell 1815dn Printer Review: Stay Far Far Away

2009-03-09T14:26:00.006-07:00

I bought a Dell 1815dn multifunction laser printer a while back. I've absolutely hated the thing. Everything it does, it does the hard way. I would write more about how bad this printer is, but I've found someone else who beat me to it:

http://www.yelsew.com/dell-1815dn-review.html

I agree 110% with every single thing in this article. I could go on and on, except none of the language is printable in a family blog.

Linux vs Windows: Uptime

2009-03-01T19:05:00.008-07:00

If you haven't figured out from this blog, I'm a Windows Guy. But I have a dirty little secret: our corporate Internet presence is a Linux server. Recently I was talking to a Windows IIS developer at a medium-sized company and he was amazed that I would entrust our public face to such an "anarchy of developers." I asked him if he had ever administered a LAMP (Linux/Apache/MySQL/PHP) server and he admitted that he hadn't. I said to him that there were two key reasons that I use Linux: stability and maintainability.

Our corporate Linux server (Redhat Enterprise 3) has been rebooted once in the last five years and that was only because the power grid in the hosting facility was being upgraded. The current uptime count is 1,242 days, or almost three and a half years.

The pedants among you will point me to Netcraft's uptime list to show that Windows Servers can be kept up that long too. However, in my experience that's more than a rarity - it's almost a unique exception that requires Herculean efforts. In contrast, our corporate server was simply automatically updated as patches became available. All available patches have been installed (except kernel patches.) No special efforts were required to keep the Linux server running because of updates.

Much of this is possible because Linux does not have the concept of "in use" files. You'll never get an "Access denied" error because someone else has a file open. In practice, this means that shared libraries can be upgraded on the fly without affecting versions of those libraries that are in use by applications. Unix has allowed the replacement of open files for as long as I've used it, over 25 years.

So as you are sitting there thinking yourself smugly superior for your IIS server, take it from another Windows Guy: when your system absolutely, unquestionably has to stay running, Windows should probably not be your first choice.

You'll notice that the NetCraft list I cited earlier includes no LAMP servers. However, until fairly recently, most variants of Unix/Linux could not report more than 497 days of uptime. Thus the presence on the list of FreeBSD but not other variants. See the NetCraft uptime FAQ.

Postscript: Up until March 2007, the Uptime Project tracked who could keep a computer running the longest without rebooting. The winner was an OpenVMS system that had been running for nearly 12 years. Irish Rail allegedly had an OpenVMS machine up for 18 years. OpenVMS is also used by both the US Postal Service and by Amazon.com. Probably completely uninteresting for most of you, but I used VMS for about five years and it has a warm place in my heart.

XBOX Technical Support

2009-02-27T01:03:00.006-07:00

I spend part of every day working with customers, sort of "the buck stops here" technical support. So I have more than a little compassion for people who, day after day, must suffer through clueless customers. But Microsoft's XBOX Technical Support has set a new low.

It all started when my brand new XBOX 360 made grinding noises and ate the demo DVD that came with the system. The problem has been thoroughly documented by the press:
http://www.llamma.com/xbox360/news/Xbox-360-Game-Disc-Scratched.htm

A little one in the house is very attached to one of the games on the DVD and failure was not an option.

Scratched DVD. Easy problem, right? No, not in Microsoft land.

First I try to ask the question online in their support system at support.xbox.com. The only choices are for console hardware failures. There's not even a choice for "Other." So I call the 800 number.

After wading through menus for at least ten minutes, I finally reached a human. It takes half an hour to explain that my DVD is scratched. Her grasp of English is shaky at best. She says I have to send my console back for repair. Arguing is futile because she doesn't speak English well enough. So I get a service repair order.

After I hang up, I go to the web site to obtain the printing label. I try to go to the support registration system, but it won't let me because the "serial number is already registered." Well duh, she just did that on the phone. But she didn't associate the registration with my Live account.

I fill out a support form on support.xbox.com asking what to do. I use the category "Console will not power on" for lack of a better choice. I get a long form letter, starting with "I know how disappointing it is that you're unable to process a repair online. Please accept our apologies for any inconvenience you have felt regarding this unfortunate matter." The email takes eight paragraphs to say "call the 800 number." I tabled the registration problem and just printed the pre-paid shipping label from the web site.

I packaged the system as requested. I taped the DVD to the console with a yellow sticky. I write with my fat red Sharpie pen, "BAD DVD."

A few days later, I get back a new console and a second package with a DVD. Microsoft shipped back the same DVD to me, except they took off the label that said "BAD DVD." There are so many incompetent people involved it's hard to know who to be upset at.

I call back the 800 number again. Every time I ask a question, the support person spends five to ten minutes talking to her supervisor. She wants the serial number for the bad DVD. DVDs don't have serial numbers. She wants some information from the DVD's book. The XBOX Arcade DVD is a demo disc and has no book. It becomes clear that she's apparently never even seen an XBOX 360, much less used one.

After 45 minutes on the phone, she hasn't been able to figure out how to get me a new DVD or how to fix the registration with my Live account. She doesn't even know what a Live account is and tries to get me to use the License Transfer option on the web site. I point out that there's no console associated with my Live account, so I can't have any licenses. She insists that I try anyway, which immediately gives an error message.

She finally gives me an incident number and tells me to call back in an hour. I think that's what's called a "brush off."

I call back two hours later. The new person speaks English quite well and understands it at least better than the last two people. I tell her the problems. She says that she can fix my registration problem by just updating my record. At least she understands the problem and the fix. Her fix doesn't work, but I feel that I've at least obtained validation that I'm asking the right questions.

However, she also gives me the brush off about the DVD. She insists that she can do that for me, but says that "I have to call back tomorrow." At which point she will have gone home and I'll have to start this whole process over with someone else.

So this whole thing should have been one phone call that lasted less than five minutes. Instead, Microsoft's collateral damage is as follows:

Support reps on the phone for two hours.
Paid to ship console to and from customer.
Paid to replace customer's console (which probably had nothing wrong with it.)
Paid to ship bad DVD back to customer, in a separate envelope.
Public display of complete incompetence.

And the sad thing is, I still don't have a disc that works :-(

Windows 7 Beta on a Thinkpad T42/p

2009-02-23T23:31:00.009-07:00

This week I bought an old Thinkpad T42/p off of eBay. IMHO, this system was the pinnacle of the Thinkpad T series, with a great keyboard, a spacious 15" display running 1440x1050, Gigabit Ethernet, and most importantly an ATI Radeon Mobility 9600. This particular system has 1GB of RAM, which I'm told is the minimum required to run Aero. The only real downside to this laptop is that it does not support SATA drives, so you can't put in a 7200rpm SATA drive.

I did a clean install of Windows 7 and I have to say that it runs really well - far better than Windows XP ever did with a similar configuration. Superfetch makes a HUGE difference. I haven't tried to add ReadyBoost yet.

Although the system initially came up with the standard VGA display driver, the Radeon 9600 driver was automatically installed the first time I connected to Windows Update.

The Aero interface did not work at first. I'm not sure exactly what I did to get it working - maybe generating the Windows Experience Index under Control Panel / Performance Information and Tools. I also rebooted a few times. In any case, Aero is now working perfectly.

Here are the numbers reported by the Windows Experience Index:

Processor: 2.0
Memory (RAM): 4.1
Graphics: 2.0
Gaming graphics: 3.6
Hard drive transfer: 2.0

As near as I can tell, all features of the laptop are running properly. There are no missing drivers in the device list. Speedstep works properly. The wireless card works (although I had to choose WPA instead of WPA2 to connect properly to my WRT54G router running DD-WRT.) Display brightness works, volume works, even the ThinkLight. The only thing I installed from Lenovo is the Active Protection for the hard drive.

All in all, I'm quite happy with the T42/p and with Windows 7 experience running on this system.

[Update 5/21/2009] I wiped the system and installed Windows 7 RC today. I confirmed that Aero kicks in AFTER you generate the Windows Experience Index and reboot.

Here are the numbers reported by the Windows Experience Index for the RC. Compared to the Beta, Processor and Hard drive transfer went up and Gaming graphics went down.

Processor: 3.2
Memory (RAM): 4.1
Graphics: 2.0
Gaming graphics: 2.0
Hard drive transfer: 4.3

[Update 10/28/2009] Lenovo has published System Update version 4, which provides a few updates for Windows 7, including the HotKey manager for onscreen feedback of volume, brightness, etc. ActiveProtection is also supported. However, most of the other Thinkpad utilities are still not available for Windows 7.

Downloading symbols for .dmp files

2009-01-26T10:51:00.008-07:00

We have an automated system for processing .dmp files received from the field, either our own collection or Microsoft's crash report system. We use cdb (the command line version of WinDbg) to automatically create a text file that contains the stack dump.

The batch file looks like this:


@set _NT_SYMBOL_PATH=SRV*c:\cache*http://msdl.microsoft.com/download/symbols
Cdb -lines -c "!analyze -v;q" -z %1

(Thanks to John Robbins for showing me how to use cdb in this manner.)

In theory, setting _NT_SYMBOL_PATH should provide cdb with enough information to automatically download symbols as needed. However, I wasn't seeing that happening. Without symbols, the debugger can't properly processes callstacks using FPO (Frame Pointer Omission), which means that the callstacks were often missing a lot of information.

Today I found a workaround. The symchk utility will examine a dmp file and verify that all of the required pdb and dbg files have been downloaded. Here is an example:

symchk /id Demo_000000.dmp /s SRV*c:\cache*http://msdl.microsoft.com/download/symbols

List Control (CListCtrl) beeping

2009-01-16T14:27:00.002-07:00

For several months now I've been having problems with an MFC application where it would beep every time I changed selections in a CListView. I figured I was doing something wrong, but today I finally decided to track it down.

Thanks to VistaHeads for the answer.

Run Regedit and then delete the default value for
HKEY_CURRENT_USER\AppEvents\Schemes\Apps\.Default\ CCSelect\.current

Note that the offending value is blank, so it looks like you aren't doing anything by deleting the value, but go ahead and delete it anyway. (Delete the value, not the key.)

The other amusing thing I learned about this is that a worker thread is created to play the beep sound and the worker thread is created with the priority Time Critical. There are a lot of things that might be time critical, but beeping on a List View change is certainly not one of them. (Yeah, I know, interrupts, blah blah, buffer management, blah, blah. It's still silly.)

Unicode BOM Handling in the C Run-time Library

2009-01-12T12:09:00.004-07:00

Visual Studio 2005 and later include Unicode BOM (Byte-Order Mark) support, but I found the documentation somewhat lacking. Here are a few hints.

One of the primary points of confusion for me was what you were defining by setting the encoding. The answer is that you are providing a hint as to the encoding of the file (but only a hint. If the file has a BOM, then that BOM takes precedence.)

All calls you make to read or write the file must be with Unicode APIs. If you try to use an ANSI API, the C-Runtime library (CRT) will assert. This means that the CRT will do character set conversion between Unicode and the file's encoding, but won't do character set conversion between the local code page and the file's encoding. For example, you'll get an assertion if you open the file with ccs=utf-8 and then try to use fgets to read the data.

Other points:

The CRT will not perform any BOM handling if you do not specify a ccs= encoding. This means that backwards compatibility is retained because the CRT does not perform any behind-the-scenes processing if you don't ask it to.
Most BOM formats are not supported. For example, UTF-7, UTF-32 and especially UCS-16 big-endian are not handled.
If you specify a specify a ccs= encoding, then the BOM will be automatically removed from the data stream. However, you need to be careful of file positioning calls such as fseek and rewind because the bom will only be skipped when the file is first opened. For example, if you do fopen, fread, rewind, fread, then the second fread will read the BOM and the first fread will not.
The file encoding is respected when writing, so the number of characters actually written may be lesser or greater than the buffer size you wrote.
If you open the file in binary mode, then any ccs= specification will be ignored and no BOM handling will be performed.
Apparently the CRT does not provide a documented way to determine the encoding of the file.

How Not To Run A Web Site

2009-01-01T21:24:00.003-07:00

Last night, New Years Eve, I received the following error from Evite:

Sorry! Due to planned maintenance, certain pages may be unavailable. We should be back up within 30 minutes, so please check back shortly. 

Thanks for your patience!

As it turned out, "certain pages" included both the home page and the party I was attending.

What kind of complete idiot schedules "maintenance" on the biggest party night of the year for a web site that manages party invitations?

I could understand system overload, but scheduled maintenance?!?

So, with just six hours left in the year, Evite easily won my award for Best Example of How Not To Run A Web Site.

Calibrating Monitors with the Eye-One LT including dual monitors and dual LUTs

2008-12-17T10:12:00.020-07:00

I've spent the last four years staring at two monitors side by side that do not have matching colors. After a while, a minor irritant turned into a major irritant, and I bought the X-Rite Eye-One LT (aka the i1) to fix the problem.

As numerous others have noticed, the documentation leaves a great deal to be desired, so here is my addendum.

1. Don't bother installing from the CD. It's all out of date. Install the latest versions of the software from:

http://www.xrite.com/product_overview.aspx?ID=789&Action=Support

The only software required for calibration is i1Match, which is Vista-compatible as of v3.6.2. (You do not need to install the download labeled 32 & 64-Bit Drivers for Win2000, XP, and Vista. The drivers are included with i1Match)

2. Install the software before you plug in the USB cord. The manual is wrong when it says to do it the other way around.

3. Understand what your monitor is capable of. For the purposes of calibration, there are three classes of LCD monitors:

Monitors that do not allow you to adjust the contrast or whitepoint. My 5-year old Samsung 191T fell into this category. Eye-One works with these monitors, but the results are suboptimal. In general, it is almost impossible to calibrate these older monitors to be the same as other monitors.
Monitors that allow you to manually adjust the contrast, whitepoint, or RGB values. i1 walks you through making adjustments. The quality of the result depends on the number of adjustments available on the monitor.
Newer monitors where Eye-One is able to automatically control the monitor to set contrast, RGB and whitepoint. Such monitors are the easiest to calibrate and will have the highest level of success. Note that Eye-One only has built-in support for automatically controlling a small number of monitors. For other monitors, you'll need to manually make the adjustments.

4. Ambient light matters. The ambient light in your room has a significant impact on the result. Make sure you calibrate your monitor in the light where it's normally used. Don't do what I did and calibrate your monitor at night when the room is lit up with incandescent bulbs. The result is not satisfactory. (Note that the i1Match software actually warned me about this when I turned on "measure ambient light". The measurement bars shows that the light in my room was inappropriate for successful color correction.)

If you have Vista, you may also run into the problem of your color correction settings being lost every few minutes. Install Vista Service Pack 1.

5. Run the calibration. This is generally fairly painless. Make sure you read the help panel on the right in i1Match - it has useful information. Make sure you do the Contrast adjustment, if i1Match asks you to.

If you are calibrating an LCD monitor, you should use the counterweight instead of the suction cups built into the sensor. I found that the suction cups wouldn't stick to the matte screen of my LCD monitor.

6. Understand LUTs. The LUT, or "LookUp Table," is a feature of your video card that automatically performs color correction at all times. Without a video LUT, color correction only works in software like Photoshop and PaintShop Pro that can perform software-based color correction. Virtually all video cards made in recent years contain a hardware LUT.

You can verify that your video card has LUT support by downloading the LUT Tester.

When you run color calibration software such as i1Match, the result is an ICC file. ICC stands for Internation Color Consortium. The file contains information on how RGB values need to be adjusted to display properly on your monitor.

Here's the annoying part. Windows, even Windows Vista, does not understand how to load the hardware LUT with the ICC file. So Windows has all of the infrastructure to track ICC profiles for each monitor, but Windows doesn't actually do anything with that information!

The solution is software that loads the LUT when Windows starts. Such software reads the ICC profile for each monitor and loads the appropriate LUT. The most common example is Adobe Gamma Loader, but the Eye-One includes the Logo Calibration Loader, which should always be used to load the LUTs.

7. Dual monitors and Dual LUTs. This one is the 900-pound gorilla. The problem with dual monitors is that you need dual LUTs. Every monitor requires a unique calibration, even two monitors that are the identical model. Many video cards today have connections for two monitors. In order to show the correct colors in both of them, each must be calibrated separately AND your video card(s) must have one LUT for each monitor. Most low-end video cards (and most pre-2006 video cards) only have a single LUT, which means that either you need a second video card for your second monitor or you need to upgrade to a card with dual outputs and dual LUTs.

It can be tricky to determine if your system will be able to calibrate two monitors. There are several variables:

Operating system: If you are running Windows Vista SP1, then the OS includes the required support. If you are running Windows XP, then you need to install extra software.
Eye-One software. The i1Display 2 software apparently has built-in support for multiple monitors. The i1Display LT software does not support multiple monitors, but see my workaround below.
Dual LUTs. To determine whether your video card has dual LUTs, run through the calibration on the primary monitor. On the final screen, use the Calibration On/Off button to see how calibration affects your monitor. If you have dual LUTs, then only the primary monitor will be affected by the Calibration On/Off button. If you only have a single LUT, then both monitors will be affected.

I have verified that the Logo Calibration Loader supports multiple LUTs. It is intelligent enough to read the ICC profile for each monitor and set the appropriate LUT.

My system has an nVidia 7950GT video card. I was very happy to discover that this card has dual LUTs.

8. Remove conflicting software. When you install i1Match, your Startup group is updated to include the Logo Calibration Loader. This sofware updates the LUT(s)in your video card. Unfortunately, there are several other applications that try to do the same thing and must be removed. The most common is the Adobe Gamma Loader, which is installed with PhotoShop.

9 Advanced users. If you want to know more about what your monitor can do, download and the free sofware HCFR. Although this software will not create or manipulate ICC profiles, it will give you copious information about ambient light, your monitor's color space, and more. Just be prepared to spend some time figuring how it works. You'll also need to copy EyeOne.dll into HCFR's installation directory. Make sure you enable Eye-One support in HCFR.

10 Finding your ICC profiles in Vista. The ICC profiles are managed by Windows. To view them in Vista, go to Control Panel | Color Management. Note that this information is maintained on a per-monitor basis and you can switch between monitors using the droplist at the top of the window. Important! If you change the default ICC profile for a monitor, you'll notice that your screen colors do not change. You must run the Logo Calibration Loader to update your LUTs after you change the ICC profile. You can run this Loader from the Start menu or from C:\Program Files\GretagMacbeth\i1\Eye-One Match 3\CalibrationLoader\CalibrationLoader.exe.

11 Calibrating your second monitor with Eye-One LT. The cheaper LT version of Eye-One does not include built-in support for multiple monitors, but the workaround is easy. (This is for Vista, I haven't tested this in XP.) This process assumes that you have already completed the calibration for your primary monitor and that you've verified that you have dual LUTs. Here is how to calibrate your second monitor:

Open Control Panel
Open Personalization
Open Display Settings
Left-click the big #2.
Check the box labeled "This is my main monitor."
Left-click the big #1.
Uncheck the box labeled "Extend the desktop onto this monitor."
Click Apply
Your primary monitor is now disabled. You should be able to run i1Match and calibrate the second monitor without difficulty. The ICC profile for the second monitor will be set to the new ICC profile and the ICC profile for the first monitor should be unchanged.
Reenable your primary monitor by reversing the steps above for the "big #1" and the "big #2."
Run the Logo Calibration Loader to reload the LUTs for both monitors. This step is required.

Other Thoughts

I'm quite happy with my Eye-One. It calibrated my desktop PC, my laptop, and my Mac. It's considered to be the best colorimeter on the market that's "affordable." (The Spyder hardware also appears to score well, but Amazon Reviews are littered with complaints about their software.)

Another alternative is the Huey, which is half the price of the Eye-One. However, the Huey cannot calibrate luminence, which is critical for calibrating today's excessively bright LCD monitors. Also, luminence calibration is required in dual monitor setups to make both monitors appear the same.

After final calibration, I was not able to get good results calibrating my Samsung 191T or 191T+. However, both of these monitors are over four years old, both have over 15,000 hours on their backlights, and neither has an adjustable whitepoint. So the poor results are to be expected. (Both monitors looked better after calibration than before calibration, so there was definite improvement.) On the other hand, my brand new HP 2475w ended up with near-perfect calibration. After creating the ICC profile and loading the LUTs with Eye-One, I switched to HCFR to graph the results and the results were excellent. Luminance was a perfect match to theoretical and RGB was corrected within 5% for all values.

[Update 2/25/2009] I tested the latest version of i1Match (3.6.2) under Windows 7 Beta and it worked correctly. The Logo Calibration Loader also worked. One small hiccup was that it couldn't find the position of the sensor, but I just told it to continue and it worked fine.

[Update 5/21/2009] I upgraded the T42/p from Win7 Beta to Win7 RC. I copied the color profile from the Beta to the RC, set it to be the default, and the screen magically updated - even though I didn't start Logo Calibration Loader. This means that, as of Windows 7, Windows finally includes support for automatically loading the LUTs based on the color profile. Great news! (This means that you can remove the the Logo Calibration Loader from your Startup group if you have Windows 7.) Note that I was not able to test dual monitors to see if Windows 7 could handle multiple LUTs.

VMware Server 2.0 Review - What a disaster

2008-11-03T10:37:00.007-07:00

I've been a big fan of VMware Server, as I've described in earlier posts. Yesterday I tried to install VMware Server 2.0 and I have to say that they've screwed up the product so badly that they are making Microsoft Virtual Server look good in comparison.

First of all, the new VMware Server 2.0 is a 500MB download. Yes, that's HALF A GIGABYTE for a virtualization product. In contrast, Microsoft's download is a paltry 29MB.

Many installations of Windows can't handle an MSI file that's so large and so the product won't install. The instructions in the Release Notes, which are not accessible to Google, describe a fix that can't be done on a Primary Domain Controller, so I spent two hours trying to find an alternative. (The server is just a development test bed, so there's no issue with VMware hurting performance.)

When I tried the link on my desktop for connecting to a virtual machine, I got a 404 error. That's it. There's no link anywhere to documentation or support. Turns out that this 500MB download included Apache, Tomcat, and other resource-hogging applications. This is a far cry from the sleek install of the 1.0 product.

I did a little more reading and learned that all connections to VMware Server 2.0 are done through the browser. This is not good news. That's how Microsoft's product works, and it's miserable. One of the best features of VMware was its client interface, which was lightweight and fast.

To add insult to injury, the installation of VMware Server 2.0 broke my network connectivity. The network interface won't accept any incoming connections. File sharing, remote desktop - everything is broken.

So my review of VMware Server 2.0 is two thumbs down.

Soon I'll need to install Windows Server 2008, which isn't supported by VMware Server. As an alternative, I'll probably investigate Microsoft's new Hyper-V offering.

Postscript: I spent three hours trying to restore network connectivity for the server and finally gave up. I wiped the drive and restored from an image-level backup. Acronis TrueImage saves the day again. Happily, this had the side effect of reverting to the prior installation of VMware server.

DreamWeaver CS4 - First Look

2008-10-23T23:52:00.004-07:00

I finally broke down and bought DreamWeaver CS4 today. This is the first time I've used Dreamweaver since 1997. It works a lot better on a 4GB Core 2 Duo machine than it did on a Pentium II 350 :-)

Anyway, here are my initial impressions.

The Good

It doesn't crash! In the past eight years I don't think I ever used GoLive for more than 40 minutes straight without crashing. DW hasn't crashed yet.
Works under Windows Vista.
The upgrade registration was painless. I entered my version of GoLive and my GoLive serial #. No problems.
DW supports sftp.
I/O to the remote sftp server is fast, in direct contrast to every other web dev tool I've tried.
Template page updates are fast. GoLive was always very slow.
Live View allows you to see the browser view without going to the browser. Pretty slick.
Dual Monitor mode.

The Bad

It took me a long time to figure out how to activate the Import Golive Site command. It's not in any of the menus. Looking for "import golive" in Dreamweaver Help brings up numerous old articles about DW CS3, which worked completely differently. I think I finally found the answer by searching on "golive to dreamweaver cs4" (without the quotes) but the first several search results were still for CS3. The answer can be found at Dreamweaver for GoLive users, http://www.adobe.com/devnet/dreamweaver/golive_migration/
After going through the effort to find and install it, the GoLive import tool didn't work at all. It refused to open the site file. My workaround was to use the GL2DW tool from CS3, which is installed in GoLive instead of DW. GL2DW crashed halfway through, but it did enough that I was able to determine what to do myself.
I got stuck on how to open an existing site. Turns out you use "New Site..." even for an existing site. From what I've heard, this will just be the first in a long list of Dreamweaver-isms that I'll run into.
Where's the "remove whitespace" command? Every other tool I've ever used, even Microsoft Expression, does this automatically when the site is uploaded.
Only Subversion is supported for source control, which is a little strange since SourceSafe is supported as a test server. Not a big deal, but SourceSafe integration would have been handy.

How to load MSHTML with data

2008-10-17T23:12:00.038-07:00

Download the sample code for this article.

This week I needed to write code to parse HTML pages and find the IMG tags to extract the images. I saw that the IHTMLDocument2 interface includes the get_images() function, so I figured that this would be a trivial problem to solve, especially since MSHTML supports running outside of a browser window.

This "trivial problem" has now taken me the better part of a week to solve. Most of the solutions I found are buggy, wrong or incomplete, some of which are from Microsoft technical articles.

For this project, I was starting with a raw buffer that contained HTML data. The buffer was multibyte, not Unicode, but the encoding of the buffer was not known in advance, so I couldn't convert the data to Unicode without parsing the page to extract the charset tag. Since parsing is what MSHTML was supposed to do for me in the first place, this made the Unicode calls useless.

1. IMarkupServices

The first article I found was from CodeProject, titled How to identify the different elements in the selection of web pages? Although the article didn't solve my exact problem, it did lead me to IMarkupServices and ParseString(), with supporting information from CodeGuru in Lightweight HTML Parsing Using MSHTML.

~~Unfortunately, I was never able to make this work. As it turned out, this solution wouldn't have worked anyway because ParseString requires a Unicode string and I couldn't provide one.~~ ~~[Note: I now believe that I needed a message loop to make this work, as in #3 below, but I didn't verify this.]~~ Update 12/21/2010: I was finally able to make IMarkupServices work properly using sample code from MSDN. IMarkupServices does not require a message loop, and you can read HTML with any character set using ParseGlobal instead of ParseString. However, IMarkupServices does have some strange drawbacks. The IHtmlDocument2 object it returns is not fully functional. You cannot QI IPersistStreamNew to serialize it and IHtmlDocument3::get_documentElement simply fails. I've also found comments that IMarkupServices will simply fail on many web pages.

2. IHtmlDocument2::write()

My next try was to use IHtmlDocument2::write(). I found this in another article on CodeProject titled, Loading and parsing HTML using MSHTML. 3rd way. This was the easiest of the various solutions to use because it worked the first time and required no message loop (more on this in #3.) However, the write() function also requires the HTML to be passed as Unicode, which meant that this solution had the same problem as #1.

The only challenge to using IHtmlDocument2::write() is properly setting up the SAFEARRAY. Although the sample code in MSDN shows how to do this, it's complicated and easy to break.

3. IPersistStreamInit::Load()

My third try was to use IPersistStreamInit::Load(). This is the solution recommended by Microsoft in the article Loading HTML content from a Stream. This function caused me no end of aggravation. No matter what I tried, I couldn't get it to work. My call to Load() would return success, but my data wouldn't appear in the document.

I found other people with the same problem. It turns out that a few important details were omitted in the MSDN article. The first hint I found was in a Usenet post on microsoft.public.windows.inetexplorer.ie5.programming.components.webbrowser_ctl, where Mikko Noromaa recommended using CoInitializeEx(NULL,COINIT_MULTITHREADED) instead of CoInitialize(NULL).

My initial results with this change were perfect; for the first time my data could actually be retrieved from IHTMLDocument2. However, when I tried to change the data, I started seeing really weird crashes. Examining the call stack showed that my calls were being marshaled across COM apartments, which shouldn't have been necessary. Eventually, I determined that MSHTML wants to live in an STA (Single Thread Apartment.) When I declared my thread to be an MTA, COM automatically created a new thread to host MSHTML and marshaled my calls to that thread. Something was going wrong in those cross-thread calls and I didn't have the inclination to debug it.

I finally found a blog post by Scott Hanselman that described how to make Load() work properly. Recent versions of MSHTML require a message loop. Apparently, older versions of MSHTML did not. Without a message loop, Load() returns success but the actual work of loading the HTML is performed asynchronously. I had to add this code after Load() in order to make it work:


for (;;)
{
    CComBSTR bstrReady;
    hr = pDoc->get_readyState(&bstrReady);
    if (bstrReady != L"loading")
        break;
    //while( doc.readyState != "complete" )

    MSG msg;
    while (::PeekMessage( &msg, NULL, 0, 0, PM_NOREMOVE))
    {
        if ( !AfxGetApp()->PumpMessage( ) )
        {
        return;
        }
    }
}

Unfortunately, after going through all this time and effort to make Load() function, I discovered a small fact that made all of this effort useless: there's a bug in the MIME sniffing code used by IPersistStream::Load(). The bug is documented in this MSDN KB article:
BUG: PersistStreamInit::Load() Displays HTML Files as Text. You can read the discussion in Scott Hanselman's blog entry to understand why this is an issue. Update 12/21/2010: This bug was fixed in IE7 and the KB article is now marked as "retired content."

4. IPersistFile

My next try was to use IPersistFile, like this:

CComQIPtr<IPersistFile> pPersist(pDoc);
hr = pPersist->Load(L"C:\\abc.html", STGM_READ);
pPersist.Release();

As in #3, IPersistFile::Load() requires a message loop to function properly.

Unfortunately, using this function was not optimal for my situation because I already had the HTML document in memory. I didn't want to write it out to disk again and slow things down.

5. IPersistMoniker

I finally found the correct solution in a post by Craig Monro in microsoft.public.inetsdk.programming.html_objmodel.

The solution is to use IPersistMoniker to feed the stream to MSHTML. By using IPersistMoniker, you avoid the MIME sniffing bug, you don't need a Unicode buffer, and you can use in-memory data.

There is one problem with the solution posted by Mr. Monro. The SetHTML function in his example takes a Unicode string for the HTML data, but this isn't a requirement to use IPersistMoniker with MSHTML. I changed the function to use a byte buffer instead of a Unicode buffer and it worked fine. I also used SHCreateMemStream() to avoid having to make a second copy of the data buffer.

Preventing Execution [Added 12/21/2010]

According to the documentation in Microsoft's WalkAll sample, "If the loaded document is an HTML page which contains scripts, Java Applets and/or ActiveX controls, and those scripts are coded for immediate execution, MSHTML will execute them by default." This is very important to understand if the HTML code you are loading is not trusted because you begin executing the page as soon as it's loaded. This applies to all forms of loading described above except IMarkupServices.

The solution to this problem is not shown in my sample code, but it is shown in the WalkAll sample. Look in the comments at the beginning of the WalkAll sample for "IOleClientSite and IDispatch".

Updating the Document [Added 12/30/2010]

Once you've loaded the HTML document, you often want to update it and save the result. I found it necessary to set designMode to "On" in order to make changes "stick." Otherwise the change would appear to work, but would be discarded when I tried to save the document. You must set designMode to "On" after the document is loaded, because loading a document resets the designMode value.

If you want to save the document, QI for IPersistStreamInit and call Save(). (This doesn't work for documents loaded with IMarkupServices.) However, be aware that MSHTML cannot faithfully recreate the document that you loaded. In other words, if you do a load/save cycle, you can't diff the new file with the original and get a reasonable result. MSHTML normalizes tags, removes newlines, and makes many other changes. There does not appear to be any way to force MSHTML to save a byte-perfect form of the document.

As an alternative to IPersistStreamInit, you can get a pointer to the document element with IHTMLDocument3::get_documentElement(), and then call get_outerHTML(), but this strategy is imperfect for the following reasons:

Any DOCTYPE or xml declaration at the beginning of the document is discarded.
Any attributes on the BODY tag are discarded.
Character encoding is lost because you always end up with wide character Unicode. Even worse, any CHARSET declaration in the HEAD is preserved, so non-English documents can display as garbage.

Performance [Added 12/30/2010]

One of my concerns was the performance that would be offered by the MSHTML control, which is hardly a lightweght control. To better understand the behavior, I ran a series of performance tests on a 3GHz Core i7 processor. What I learned is that the time required to parse HTML is dwarfed by the time required to create an MSHTML object. In my tests, if the MSHTML object is created once and then reused, a thread can load a 2K file about 1000 times in one second. If the MSHTML object is created and destroyed on each iteration, performance drops to 25 loads per second, a 40x performance hit. The lesson is that HSHTML should be created once and reused. The reuse strategy is the fundamental reason IPersistStreamInit exists as a separate interface from IPersistStream.

Conclusion

This is one of the more difficult problems I've worked on lately. Between bad examples, bad documentation, bad 3rd party advice, and a plethora of different strategies, finding this solution was far more difficult than I expected. I hope this article saves some others from the same frustration.

Download the sample code for this article.

GigE File Sharing Performance - 96MB/sec!

2008-10-12T13:13:00.010-07:00

I wrote numerous blog entries last year about my difficulties getting GigE (Gigabit Ethernet) to work properly. Yesterday I upgraded both my Vista client and my Windows 2003 Server to 4GB RAM (see the end of this article for hardware configurations.) Suddenly my resource-constrained systems had lots of room to play. I reran my performance tests over my Gigabit Ethernet and came up with some very unexpected results.

First I used the DOS "copy" command on the Vista client to copy a 256MB file from the server to the client. The file was not cached on the server, and it transferred about 15MB/sec. This was the same performance I was getting before the RAM upgrade.

Next I repeated the same copy. The file was completely cached on the server and the file transferred at about 25MB/sec.

Finally, I again used the DOS "copy" command on the Vista client to copy the file from the client back to the server. The transfer peaked at 96MB/sec!! (A 256MB file copies VERY quickly at that speed.)

On the one hand, this is a contrived test - in the real world we almost never have the luxury of copying a file that's already cached in RAM. However, the tests lead me to several useful conclusions:

The tests show the peak performance of Windows file sharing when you take the disks out of the equation. At 96MB/sec, that's 85% of the practical maximum of 112MB/sec. Considering that the filesharing protocol itself has overhead, we are actually running at somewhere between 90% and 95% of the theoretical max. That's fantastic.
The file sharing protocol's latency is negligible. If it were significant, I wouldn't have reached the above performance numbers. Instead, the transfer rate of the hard drives is the primary performance constraint. Both test systems have SATA 7200RPM drives. I expect that if I had 10,000 RPM RAID 5 drives, my maximum performance when copying non-cached files would improve dramatically.
Something very strange is happening when copying from the server back down to the client. Why this operation is peaking at 25MB/sec is not clear.
Jumbo frames are completely unnecessary for peak performance. They might cut down on the CPU load, but even that's debatable when interrupt coalescing is enabled on the Ethernet card.

One set of datapoints that still need to be measured is to rerun from the command prompt on the server. In prior tests, it mattered which system initiated the file copy.

Finally, I accidentally performed the above tests with Virtual PC 2007 running on the Vista client. Virtual PC cut the transfer rates by 30 to 60%. The final copy peaked at 40MB/sec instead of 96MB/sec. Oddly enough, VMware Server was running on the Windows 2003 Server for all of the tests, and it had two virtual machines active. So while Virtual PC had a significant impact on network performance, it appears that VMware Server had no impact.

Client Configuration
Vista SP1
P5B Deluxe Wifi
Core 2 Duo 2.4 GHz
On-board Marvell Yukon Ethernet Card
  - Interrupt coalescing enabled
  - Jumbo frames disabled
4GB RAM
GigE registry update applied

Windows 2003 x64 Server Configuration
P5B Deluxe Wifi
Core 2 Duo 2.13 GHz
On-board Marvell Yukon Ethernet Card
  - Interrupt coalescing enabled
  - Jumbo frames disabled
4GB RAM
Ethernet adapters on the VMware virtual machines were running in Bridged mode.

Passive (PASV) ftp in Windows 7

2008-10-11T23:07:00.004-07:00

I was controlling a customer's computer and trying to upload a file to our corporate ftp server. I could connect and log in, but any other command would cause the ftp client to hang. I was using the ftp command built in to XP because I didn't have privileges to install a 3rd party ftp client. It was frustrating, but I didn't have time to debug the problem.

Today I had the same problem on a system in the office when trying to connect to a new server. However, this time I was running a different ftp server (vsftpd) that gave me a helpful error message:

200 PORT command successful. Consider using PASV.

I tried the obvious, which was to restart ftp from the command prompt and then type PASV, but this gave the error "Invalid command." I found several posts that said that the ftp client built-in to Windows doesn't support passive mode. The good news is that ~~they are wrong~~ this is fixed in newer versions of Windows.

With Windows 7, enter this command to enter passive mode:

QUOTE PASV

With vsftpd, I was rewarded with this response, and everything started working:

227 Entering Passive Mode

As other have noted in the comments, this does not work on Windows XP.

CString and Path functions in shlwapi.dll

2008-09-29T09:46:00.004-07:00

The "Path" functions in shlwapi.dll are examples of obscure APIs that are incredibly useful. There are dozens of useful functions in there, including PathAppend, PathFileExists, PathFindExtension, and PathFindFileName.

The only problem is that these functions don't play nicely with CString without going through the contortions of GetBuffer/ReleaseBuffer.

Turns out that Visual Studio 2005 and 2008 both include the atlpath.h header file, which includes wrappers for all of the Path functions to make them easy to use with CString. This makes those functions much easier (and much safer) to use.

Visual Studio 2008 C/C++ applications under Win9x

2008-09-28T08:28:00.004-07:00

On of the disadvantages of Visual Studio 2008 is that it does not create applications that support Windows 95/98/98SE/ME nor Windows NT. Although Windows XP has been shipping for seven years, there are still people out there who have never seen any reason to upgrade and are still running the old operating systems. (As of August 2008, www.w3counter.com shows that the combined market share of the Windows 9x variants is less than 0.9%.)

Windows 95 support was dropped in Visual Studio 2005, but working around the problem was easy.

Visual Studio 2008 is a thornier problem. VS2008 dropped support for all variants of Win9x and the C run-time depends on several functions that require Windows 2000 or later. Fortunately, a third party company has created a product named Legacy Extender that provides a drop-in solution. The caveat is that it only works with the statically linked C runtime libraries, not with the DLLs.

For a modest $29.95/developer, including unlimited upgrades, Legacy Extender is a great solution to a difficult problem.

Screenshots for technical support

2008-09-13T17:31:00.004-07:00

The bane of every support technician's extistence is trying to interpret what a customer is saying. I'm launching a new site named www.SendMyScreen.com that allows users to send technical support people a screenshot without requiring any expertise.

Check it out.

Remote Debugging Managed Code

2008-09-08T11:05:00.004-07:00

Last year I blogged about how to get Remote Debugging working. That article was for Visual Studio 2005, but it works fine with some minor modifications for directory and filenames for Visual Studio 2008.

Now I've had to take the next step, which is remote debugging managed code instead of native code. Turns out that debugging managed code is even more complicated than native code.

One of the key problems I originally had was that I was never able to get authenticated debugging working because one computer was on a workgroup and one was on a domain. After reading several articles, I was able to solve the problem by creating an account on the remote computer that had the same username and password as the account of the development computer. If your development system isn't on a domain, make sure that the remote account is created on the local computer and not the domain. You may need to use the "NET USER" command from the command line to create such an account.

If you are running Windows Firewall, the remote debug module msvcmon.exe will automatically unblock the right ports for you. At least that was easy.

The next trick is to convince .Net to run your application from a network drive. This technique is described in the .Net Security Blog.

By the way, the Remote Debugger Configuration Wizard was useless for me.

Testing Google APIs

2008-09-02T11:26:00.002-07:00

I blogged a little while ago about the problems with Google documentation. I've now completed my code that uses these APIs and I'm writing stress tests for the system. Except there's one problem: The server can generate all sorts of errors that aren't under your control, and there's no way to test these error conditions.

These errors are different than communication errors. It's easy to simulate a "link down" problem or similar TCP/IP failure. The problem is that the Google Data API servers can fail in strange and bizarre ways.

One example is the simple "Server Busy" command. The documentation says to "wait and retry." Unfortunately, there's no way to force this scenario, so there's no way to test this error.

Another example is error 403 "Operation could not be completed". What do you do with that? Even worse, if you are performing batch operations, you could have partial failure. Again, there's no way to test this scenario.

Please vote for my bug report asking Google to fix this problem:

http://code.google.com/p/gdata-issues/issues/detail?id=737

Visual Studio 2008 Feature Pack: Manifest Problems

2008-08-04T15:19:00.006-07:00

If you install the Visual Studio 2008 Feature Pack, be aware that the manifest handling in the release is buggy, to put it politely. It's very easy to end up with the manifest requiring both the RTM DLLs (9.00.21022.8) and the Feature Pack DLLs (9.00.30411.0). This can cause weird, untraceable errors at runtime. I blogged about this problem before, but in that case it was caused by changing from a Beta build to the Release build.

There's a helpful blog post about the problem in the Visual C++ Team Blog.

The important point is that there are two #defines that force the manifest to point to the newer DLLs:


#define _BIND_TO_CURRENT_CRT_VERSION 1
#define _BIND_TO_CURRENT_MFC_VERSION 1

There's a single #define that does the same thing, but it's broken in the Feature Pack. [9/7/2008 - It's fixed in Service Pack 1. Thanks "anonymous"!]


#define _BIND_TO_CURRENT_VCLIBS_VERSION 1

The blog points out that defining the explicit BIND macros is not always the right thing to do.

This problem bit me because I was following the directions in this blog post, which call for making copies of certain directories from Debug_NonRedist. In a system where the CRT and MFC DLLs have been installed, there are pointers in the registry from the old versions of the DLLs to the new versions, so everything "just works." However, when using the Debug_nonRedist directories, these redirect pointers don't exist. Since the manifest was calling for the old versions of the DLLs, the application wouldn't open.

If you get error c101008a, either Rebuild All or delete all of your "xxx.exe.embed.manifest" files and build the solution.

Installing a Code Signing Certificate

2008-07-25T08:57:00.013-07:00

Warning: DO NOT use a Windows Vista or Win7 computer to buy your certificate. View my earlier post to see why.

Every time I try and install a code signing certificate, I forget how I did it last time. You'd think that there would be a guide somewhere on how to do it, but if there is, I haven't found it. Both VeriSign and Thawte give tantalizing hints scattered among dozens of knowledgebase articles, but overall, it's rather poorly documented.

So here's how to do it: (Note: If you are using Comodo and saving to the CSP, you should skip to Step 3, then skip to Step 7. In this case, you don't use the PVK or SPC file.)

Prerequisites: You must have a PVK file and an SPC file. From VeriSign and Thawte, these are normally named mycert.spc and mykey.pvk. If you don't have both of these files, this article won't help you. You'll also need the password for the PVK file.
Install PVKIMPRT from Microsoft. You can download it here.
Remove your old certificate. If you are renewing an existing certificate, then keeping the old certificates installed isn't usually useful, and having multiple certificates will break SIGNTOOL if signtool is searching the certificate store. Go to Control Panel / Internet Options / Content, click Certificates, select your old certificate, and click Remove. The old certificate will probably be on the Personal page if you allowed PVKIMPRT to decide where to put it.
Import the certificate. Run PVKIMPRT to load the certificate into your cert store, like this (should all be entered on one line):

C:\Windows\PVKIMPRT.EXE -import c:\mycert.spc c:\mykey.pvk

You'll be prompted for your password, which you should already know. You'll also be asked which certificate store. You can let PVKIMPRT decide.
Verify the installation. Go to Control Panel / Internet Options / Content, click Certificates, select your old certificate, and click View in the bottom right. The certificate will probably be on the Personal page if you allowed PVKIMPRT to decide where to put it.
Install the intermediate certificate. When you view the certificate information, you'll probably get a message that says something like "Windows does not have enough information to verify this certificate."

Root Certificates

Make sure you have the private key. Go back to the Certificates Page on Internet Options and view your certificate. At the end of the information, right under the "Valid from" dates, you should see something that says "You have a private key that corresponds to this certificate." If this isn't there, delete the certificate, and repeat this procedure starting from Step 3. This happened to me when the intermediate certificate wasn't installed.
Export the PFX file. The PFX file can be used by SIGNTOOL. One advantage is that PFX files can be created without a password, which is handy in automated builds if you are using SIGNTOOL. You can see the complete process with pretty pictures at PentaWare. Here's the abridged version:
1. Go back to the Certificates Page on Internet Options
2. Select your certificate
3. Click Export.
4. Click Next.
5. Select "Yes, export the private key."
6. Click Next.
7. Select "Personal Information Exchange - PKCS #12 (.PFX)." (If this option is grayed out, then your private key was not imported).
8. Check the box labeled "Include all certificates in the certification path if possible. THIS IS VERY IMPORTANT.
9. Click Next.
10. On Windows XP, you can leave the password information blank if you plan to use this PFX file for automated builds. On Windows Vista, you must provide a password.
11. Click Next.
12. Enter a filename.
13. Click Next.
14. Click Finish.
Sign your code! If you need some hints on this, see my earlier post.

Visual Studio 2008 C++ Redistributable Components

2008-07-24T16:46:00.011-07:00

I've wasted two hours hunting down files that should be obvious, so I'm hoping this post helps someone else.

If you need to redistribute components from Visual Studio 2008, such as the C runtime or the MFC DLLs, there are four ways to do it:

1. Put the DLLs in your installer and install them into Windows\System32 directory yourself.

DON'T EVEN THINK OF DOING THIS. It's almost impossible to do it right because of WinSxS. If you want complete control over your installation, use option #2.

2. Use the redistributable directories Microsoft provides.

These directories provide copies of the DLLs that are only available to your application. ~~The directories should go in the same install directory as your EXE. This means that they are subdirectories of wherever your EXE is installed.~~ [Updated 11/19/2008] I no longer recommend copying the directory itself. Instead, copy the contents of the redistributable directory into the same directory as your app. Make sure you include the manifest file. I made this change because msvcm90.dll will not bind to msvcr90.dll if they are a subdirectory. As a bonus, this change makes this option work with Windows 2000.

Assuming you've installed Visual Studio 2008, you can find the directories at:

C:\Program Files\Microsoft Visual Studio 9.0\VC\redist\x86

Important: If you do this with Visual Studio 2008 SP1, make sure you put the following in your precompiled header:


#define _BIND_TO_CURRENT_VCLIBS_VERSION 1

Advantage: Doesn't require admin privileges. Works with XCOPY. Your app won't break if the system global version is updated by Microsoft (but you won't benefit from security fixes either.)
Disadvantage: Not viable if your EXEs and DLLs are installed across multiple directories.

3. Microsoft Visual C++ 2008 Redistributable Package:

Original distribution.

SP1 distribution. See caveats.

[Update 6/22/2009] Both of these distributions cause the the Windows 7 Logo Toolkit Beta to generate FAIL errors.

Advantage: All you have to do is run it. Permamently installs the components in the proper locations.
Disadvantage: Includes everything, so it's larger than the individual merge modules. If you are building your own installer, not as clean of a user experience as the merge modules.

4. Use the Merge Modules.

Assuming you've installed Visual Studio 2008, you can find the files at:

C:\Program Files\Common Files\Merge Modules

Advantage: Best user experience. Smallest download.
Disadvantage: Installation isn't permanent - the DLLs may be uninstalled when your application is uninstalled. Requires you to use Windows Installer. (This is only a disadvantage for a minority of developers. Windows Installer is a logo requirement for Vista.)

Google API Documentation

2008-07-16T23:03:00.003-07:00

One of my annoyances of dealing with many Open Source projects is the lack of documentation. nHibernate is a prime example - it's a fantastic technology with minimal documentation. The only book on the subject is a year late (and counting.)

I no longer find it entertaining to beat my head against a wall for days (or weeks) trying to figure something out that should have been documented in the first place. I don't mind paying for answers anymore. $200 is a small price to pay to get an answer that saves me a day of work.

I was therefore terribly disappointed by the tremendously poor quality of the documentation for the .Net library for the Google API. In the best tradition of F/OSS, the documentation for it is desperately lacking. Most functions have little more than a terse one-liner description, with no additional remarks. If it weren't for the fact that source code was included, trying to determine how to use the API might well be impossible.

A typical example is Captcha handling. The Google API will sometimes request the user to solve a Captcha to make sure that a human is really present. The problem is that there is no C# source code to demonstrate the correct response. Captcha is not handled by the sample code or the unit tests. It's mentioned in the newsgroups, but none of the responses actually contain code that works.

A more pathological example is documenation on the Contacts namespace. Try clicking on GroupMembershipCollection. No dice - broken link. Try looking at the code in the Samples directory. No help there, there's no sample code for contacts. Try looking at the Java library. Nope, it works differently from the .Net library (That's not a dig in this case. Both libraries are crafted appropriately for the style of each language.)

There's also no one you can call if you have a problem, even if you are willing to pay for the answer. Say what you will about Microsoft, but when they support something, they mean it. Monitored newsgroups, books, tech support, consultants, 3rd party training - you name it, the resources go on and on.

Google has a lot to learn about providing services to the enterprise market.

Fixing "ambiguous symbol" in a C++/CLR project

2008-07-11T12:05:00.005-07:00

I was trying to use a CString in a managed project today. This is supposed to be easy but it wasn't working. [Note: Microsoft factored CString from MFC several years ago, so CString can now be used standalone. Super handy class.]

I started with what I thought was a reasonable statement:


#include "cstringt.h"

This gave me numerous errors, starting with:

C:\Program Files\Microsoft SDKs\Windows\v6.0\Include\ocidl.h(6238) : error C2872: 'IServiceProvider' : ambiguous symbol
C:\Program Files\Microsoft Visual Studio 8\VC\atlmfc\include\atlcomcli.h(370) : error C2872: 'DISPPARAMS' : ambiguous symbol

With a little research, I determined I was using the wrong include file. So I switched to:


#include "atlstr.h"

When I tried to do that, I received similar errors:


C:\Program Files\Microsoft SDKs\Windows\v6.0\Include\ocidl.h(6238) : error C2872: 'IServiceProvider' : ambiguous symbol
C:\Program Files\Microsoft Visual Studio 8\VC\atlmfc\include\atlcomcli.h(370) : error C2872: 'DISPPARAMS' : ambiguous symbol

Several other posts inquiring about the same errors received no answers.

The problem is caused because the Microsoft SDK (aka Platform SDK) and mscorlib both have definitions for these classes.

The solution is to take the #include file that caused the problem and make sure it is placed before any "using namespace" statement. In my case, that meant moving #include near the top of stdafx.h.

Cheap Code Signing Certificates

2008-05-27T21:02:00.020-07:00

WARNING: if you are getting your code signing certificate from VeriSign or Thawte, DO NOT use a Windows Vista or Win7 computer to get your certificate. If you do, you will not be able to export the private key and so you won't be able to sign code on any other computer and you won't be able to back up your certificate. If this happens to you, Thawte will give you a free reissue (Thanks Thawte!) See https://www.thawte.com/ssl-digital-certificates/technical-support/ Make sure you do not use a Vista or Win7 computer for the reissue! [Added 7/13/2008]

Code signing certificates are expensive. VeriSign charges about $495/year. We've had a VeriSign certificate since 2001, originally because ActiveX controls must be signed to easily distribute them. Today, the Windows Vista Logo Requirements require that your installer, your EXEs, and all of your DLLs be signed with a code signing certificate.

This year I tried to find a cheaper solution than VeriSign. I thought I'd share my results.

VeriSign

VeriSign is arguably the King of Code Signing Certificates. They've established a strong brand and are widely recognized by consumers. Their certificates are universally recognized.

I have two frustrations with VeriSign. First, they use PVK files, which are aggravating to deal with, as I describe in an earlier blog entry.

Second, something breaks every other time we renew. The first time was when they started signing their code signing certs with an intermediate certificate instead of a top level cert. That caused us all sorts of problems. For a later renewal VeriSign suddenly decided that they couldn't support punctuation in the name, which meant that putting a comma before "Inc" was problematic. We had to change the company name in our build process, create a new cert from scratch instead of getting a renewal, and other problems.

When you are charging your customers that much money, you shouldn't be causing them this many problems.

Thawte

Thawte is about half the price of VeriSign. They appear to have equivalent compatibility (all the way back to Windows 95) but they don't have the brand recognition. ~~If I were launching a new product today, I'd get a Thawte certificate.~~ (see July 2010 update, below)

VeriSign now owns Thawte and continues to sell Thawte certs for half the price of VeriSign certs. It makes no sense to me.

Note that Thawte uses the VeriSign timestamping URL.

GlobalSign

GlobalSign also has compatibility going back to Windows 95, but I would not use them. GlobalSign limits you to 50 timestamps per year. Since we timestamp about 12 components every time we do a release build, we go through thousands of timestamps every year. Since timestamps are critical for a commercial product, I would not consider buying a certificate from GlobalSign.

Comodo

Comodo is the preferred "cheap solution." Both John Robbins and Keith Brown have blogged about this company.

A concern with Comodo certificates is that they don't work under Win9x unless the users have installed the Root Certificate Update. Since that update is an "optional" component under Windows Update, it's rare for unsophisticated users to install it. Therefore, if your application supports Windows 9x, I recommend you skip Comodo.

Conclusion

Here's my opinion if your audience is largely English speaking:

Internal use application - use your company's root certificate if you have one, otherwise use Comodo.
Commercial software application - buy VeriSign if you have a premium brand, otherwise buy Thawte.
Hobby or shareware developer - buy Comodo unless you need to support Win9x, in which case buy Thawte.

Update 7/19/2010

If you are using Visual Studio 2008 or later, Win9x isn't supported, so you might as well use Comodo.
If you buy your Comodo certifiate through Tucows, the price goes down to $75/year or less!
To join WinQual (required for Microsoft Partner status), you must have a VeriSign certificate. However, you don't have to spend $495. You can get a one year code signing certificate from VeriSign for $99! This offer is only valid for one year certificates. If you try and choose two or three year, the price goes back up to normal. At the end of that year, switch to Comodo.
Comodo does not try calling the registered phone number. Thawte does.
Comodo requires that the WhoIs Registrant for the company domain have the same postal address as the company paperwork (such as the DUNS listing.) If necessary, you can change your WhoIs listing until the cert is approved, then change it back. Changing the Administrative or Technical contact is not sufficient.
This article does not apply to code signing for device drivers.

Debugging .Net Framework Code

2008-05-20T18:06:00.004-07:00

When I started trying to write C# code, I spent most of my time with the Reflector trying to figure out what the .Net Framework was doing under the hood. My start routine was: start the debugger, see unexpected results, read source code in Reflector, repeat. I spent a lot of time searching for functions in the Reflector.

I spent years doing MFC programming. MFC shipped with complete source code from the very early days. Without that source code, it would have been almost impossible to solve many of the problems I ran into. (See my earlier blog about fixing problems with MFC symbols in Visual Studio 2005.)

I was tickled to discover today that Microsoft has made the source code of the Framework available for debugging purposes - even specialized code like Asp.Net. You must be running Visual Studio 2008. Take a look at Shawn Burke's Blog showing how to set it up.

Make sure you install the Visual Studio hotfix (QFE)!

What do developers want for working conditions?

2008-05-15T20:14:00.005-07:00

There's a great article by Joel Spolsky titled The Joel Test: 12 Steps to Better Code. This article talks about how to write better code, but it's equally relevant to building an environment where software developers want to work.

The article hits home for me because, in the mid-90s, I worked for a company that was developing software for Microsoft. By requirement, we adopted methodologies that Joel talks about, like the "zero defect methodology." This had an almost miraculous effect on our ability to ship stable software on time. Even today, I can tell with frightening accuracy when a project will ship by doing nothing more than looking at the bug database statistics. The last person to argue with me on the subject was the president of a $50M company. I said they'd ship in 2 to 3 years - if they were lucky. He said I had no idea what I was talking about because I was just a lowly engineer and he was certain that they'd ship in three months. Two years later, I was still waiting for the initial release.

We also adopted other Microsoft techniques, such as the 1 to 3 ratio of testers to developers. This change also had a dramatically positive effect, both on the ship schedule and on the morale of the developers. Consider that developers spend 80% of their time on the "normal" case. Testers spend 80% of their time on the boundary cases, thereby acting like customers who abuse the software right out of the box. If you really want to make a developer unhappy, tell him to spend 80% of his time testing (as opposed to developing) obscure boundary cases, like whether the software correctly handles 260 character filenames on multibyte character systems. Most developers will just read Slashdot instead.

Joel also talks about quiet working conditions, which is one of my pet peeves. The last five companies I worked for all had overhead intercom systems that would be used to page people throughout the day. Every time one of those pages came on, it interrupted six to twenty developers and broke them out of "the zone". Aggravated me no end.

The final point that Joel talks about is Netscape. Everyone says that Microsoft put Netscape out of business. However, Netscape did far more damage to themselves than Microsoft ever did. Netscape released version 4.5 in October, 1998. They didn't release a stable upgrade (7.0) until five long years later. This means that, for all intents, Netscape completely sat out the second half of the DotCom era. They tried to start fresh on an entirely new project, but their cadre of 20-something programmers simply didn't have the project management and large-scale software development experience to pull it off.

In summary, Joel's article is now eight years old, but many of the points discussed in the article, such as continuous builds, are now considered de riguer of any software development effort with any self respect. It's worth your time to read.

Fixing "PRJ0050 Failed: Failed to register output"

2008-05-02T11:52:00.012-07:00

I'm in the process of converting a COM DLL over to .Net using C++/CLR and I've been plagued with this error:

RegAsm : error RA0000 : An error occurred while writing the registration information to the registry. You must have administrative credentials to perform this task. Contact your system administrator for assistance
Project : error PRJ0050: Failed to register output. Please try enabling Per-user Redirection or register the component from a command prompt with elevated permissions.

I used Google to try and find a solution and found more confusion than solutions. The correct solution (for me) didn't appear anywhere. So here's my guide to solving this error.

There are three possible causes. Diagnosis is usually straightforward. Run Regsvr32 and try to manually register the DLL. If you see "entry-point DllRegisterServer was not found" then jump down to Disable Registration. If you see "access denied" then jump down to User Account Control. Finally, if you get "missing dependency," jump down to Missing Dependency.

User Account Control

This error is most likely to happen on Vista when you try to register a COM DLL. The reason is that, even if you are running as Administrator, you aren't really an Administrator because of User Account Control (UAC). This problem is easy to test. Enable per-user redirection in the project properties under Linker / General. Force the project to relink (make a minor change to a source file) and see what happens. If the error goes away, then you've found the problem. Amazingly enough, PROJ0050 is specifically mentioned in the documentation for the Linker Property Pages.

Other more intrusive solutions:

Close Visual Studio and restart it as an Administrator by right-clicking the Visual Studio icon and selecting Run as Administrator. The screen should blink and you should see the UAC prompt.
Always run Visual Studio as Administrator as follows. Right-click the Visual Studio icon, select Properties, go to the Compatibility tab, and enable "Run this program as an administrator."
Disable UAC (not recommended.)
Ignore the problem. It's not hurting anything.

No matter which solution you choose, make sure you test the registration of the dialog on a clean machine when you build your installer.

Disable Registration

This was the solution that applied to my problem. My project used to be a COM object written using MFC. The COM object required registration and so the project was set to register the DLL after linking. My fancy new .Net object did not require registration, so there was no "DllRegisterServer" entry point in the DLL. I fixed the problem in the project properties under Linker / General by setting Register Output to No. Make sure you do this for both Release and Debug builds.

Missing Dependency

This problem happens because your DLL is dependent on another DLL that can't be located. It's easy for this problem to happen if your DLL compiles to one directory and a dependent DLL compiles to another directory.

To diagnose this problem, use the Depends utility. Don't use the version that ships with Visual Studio, it's out of date. Download the latest version of Depends from http://www.dependencywalker.com/. When you run Depends, make sure you open the DLL that's in the target directory where the linker put it. Now look down the list in the middle pane and see what's marked with a yellow question mark. Ignore any modules with an hour glass, they are delay loaded and don't cause a problem if they are missing.

If the missing DLL is one of your DLLs, then update your path to include that DLL. Problem solved.

If the missing DLL is one of the MFC or C Runtime DLLs (which start with MSVC or MFC, respectively), then life is much more complicated. You might be having a problem with WinSxs, which means there's an error in your manifest. These problems can be nasty to fix, as I described in this post.

Finally, there was a red herring for my project when I ran Depends. The file MSVCR90D.DLL was marked as missing. I'm not sure what was causing this, but the DLL ran and loaded properly, so I'm suspicious that the problem was with Depends.

ReadyBoost - The Final Chapter?

2008-04-29T21:48:00.004-07:00

I'm very happy to say that Windows Vista SP1 finally cured the remainder of the problems with ReadyBoost that I'd been seeing. For the last 18 months I've been unable to put my system to sleep at night because it would kill the ReadyBoost drive, without which performance would suffer. This means that SP1 has fixes for USB above and beyond the USB patch from last year.

For those wondering why I care so much about putting my system to sleep at night - it's the principle of the matter. Leaving lights on is throwing money down the drain. Leaving your computer on is too.

Gigabit Ethernet and Vista SP1

2008-04-29T12:22:00.003-07:00

I've complained at length about the performance problems with Windows Vista and Gigabit Ethernet. In the most common scenario, Windows Vista severely throttles the performance of Gigabit Ethernet if you are listening to music or watching videos, such as WinAmp or Windows Media Player. The problem is even worse if you have multiple network cards (including wireless cards.)

Service Pack 1 for Vista contains a workaround for the problem, but you have to modify the registry directly with RegEdit - there's no user interface to control the throttling. You can find a complete description of the change in SP1 at AnandTech, including instructions:
http://www.anandtech.com/systems/showdoc.aspx?i=3233&p=2

Entrepreneur Conference

2008-03-20T22:03:00.002-07:00

I attend a couple of meetings of entrepreneur groups each month. I've learned a tremendous amount from these meetings, particularly those run by TIE. It's a hardcore business group that's light on socializing and heavy on running and funding your business. Presentations I've heard at TIE events have dramatically influenced the way I've built my business and market products.

Today I attended the Lucky Napkin conference, the first of its kind, by and for entrepreneurs. This conference was unabashedly pro-entreprenur, a breath of fresh air in a world that generally considers entrepreneurs to be on par with "crazy inventors." While the overwhelming majority of attendees were focused on consumer goods, it was great talking to other people who were also facing the challenges of running businesses with a lean budget and less than a handful of employees.

The founders of Lucky Napkin also consulted with with Danny Deutsch, the host of "The Big Idea" on CNBC. It's a great show, as long as you remember than Danny features the success stores and that there are a lot more failures than success stores.

DVDs of today's Lucky Napkin conference are available from their web site.

Choosing Web Development Software

2008-03-16T22:19:00.005-07:00

For the past eight years I've been using Adobe GoLive. I got started with GoLive because it was recommended by people I trusted and it was a lot faster than Dreamweaver. GoLive also did a good job of editing PHP and had a nice, simple template tool that allowed me to do WYSIWYG editing without having to hand-maintain dozens of files.

Times have changed. The Windows version of GoLive is now horribly unstable and Adobe simply removes features instead of fixing them. I've started looking for a new tool. Our site has been PHP for several years, so it isn't worth the effort to switch it to ASP.net.

I found Visual Studio 2005 to be okay for quick HTML edits, but it's rather clunky editing HTML compared to more sophisticated tools. Out of the box, it also can't handle PHP. I tried VS.php, which is inexpensive, but the VS.php debugger never worked properly and their tech support was useless. After a year of trying, I gave up on VS.php in Sep 2007.

Instead I settled on PHPed from NuSphere. It is a fantastic tool that I'd recommend to any PHP developer. Its debugger is very reliable. However, PHPed is purely a PHP development tool and doesn't do HTML editing.

So I'm still looking for an HTML tool. Although Dreamweaver is the obvious choice, it's expensive to buy, expensive to support, and expensive to upgrade. So it wasn't my first choice. OTOH, I'm a Microsoft Certified Partner and so can use all of their development tools for free. When I saw Expression Web at the Professional Developers Conference in 2005, I initially had high hopes, but then I learned that it was restricted to Microsoft languages. Oh well.

In February, Microsoft shipped Expression Web 2 Beta. Among other things, this update supports PHP. It's also easily the fastest tool I've used. Unlike Visual Studio 2005 and Adobe GoLive, you don't need to take a coffee break to start it up and shut it down.

My initial reaction to Expression Web 2 was quite favorable. It's beta, so the WebDAV support doesn't work with Apache and there are other teething problems, but these are okay. However, the deal killer is that you can't define the root directory, so all URLs relative to the root don't work. Microsoft has indicated they don't plan to fix this problem in this release. WTF?

Concurrent to all of this, I tried iWeb on the Mac. It's a high-level tool designed for non-web designers. It generates completely unmaintainable code, but it works and the results look good. It also has the ability to do handy things like rotate images, add drop shadows, and other image operations that historically have taken twenty steps to perform using an external image editing application, such as Paint.Net or Photoshop. iWeb will also natively import PhotoShop and Illustrator files. This is a huge win compared to most tools under Windows. iWeb also has some pretty good templates, in stark contrast to any software I've ever seen under Windows. So I've seen a vision of what things could be like with the best of both worlds, but I haven't seen a solution.

So now I'm back to square one. I'll probably have to buy Dreamweaver. It's $1,000 for Creative Suite (CS3) Web Standard version or $1,600 for CS3 Web Premium. Ouch. To add insult to injury, there's no way to upgrade from GoLive to Dreamweaver, even though they're both made by Adobe. Thanks Adobe!

Update 5/13/2008: Adobe has formally announced that GoLive has been discontinued. The good news is that they are offering an upgrade price to switch to DreamWeaver as well as a migration kit for converting GoLive projects. Read the details at:
http://www.adobe.com/products/golive/pdfs/golive_eol_faq.pdf

ReadyBoost with lots of RAM

2008-02-28T14:00:00.003-07:00

There's been quite a bit of discussion as to whether ReadyBoost does anything for you if you already have 2GB of RAM. On my system, I'm running 2GB of RAM and a 4GB thumb drive for ReadyBoost, and I believe ReadyBoost makes a dramatic difference.

I'm hardly a typical user. Normally I have open: Outlook 2007, Visual Studio 2005, Virtual PC 2007 running Windows XP, and Internet Explorer with a dozen pages open. The Performance tab in Task Manager currently shows I have 1GB cached, which isn't much considering how much is running.

Earlier this week my system started churning. Everything was taking a lot longer than it had the day before. It took me a while, but I finally realized that the lights weren't flashing on the ReadyBoost drive anymore. I rebooted, reset the ReadyBoost drive, and performance returned.

So I don't have any concrete, measurable results that I can hold up as evidence, but the subjective experience is pretty clear. Many reviewers have tried to do synthetic benchmarks, such as loading five applications in a row. These tests don't do a good job of measuring what ReadyBoost is doing. Loading apps measures SuperFetch, not ReadyBoost. IMHO, ReadyBoost seems to shine when the system is under heavy memory load and you switch applications task.

Visual Studio 2008 Unusable with C++

2008-01-07T14:49:00.003-07:00

Update 8/12/2008: Microsoft has released Service Pack 1 for Visual Studio 2008, which resolves these problems. See the last post on this page:
http://www.microsoft.com/downloads/details.aspx?FamilyID=27673c47-b3b5-4c67-bd99-84e525b5ce61

Update 3/30/2008: Microsoft has finally released two hotfixes that address these problems. See the last post on this page:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2799211&SiteID=1

I've now spent a couple of weeks using Visual Studio 2008 for C++ development, and my conclusion is that it's unusable. Those of you who follow this blog know that I'm generally supportive of Microsoft's errors, but this time there's no excuse.

Our project is about 250,000 lines of code, a relatively small project by some measures. Every few builds, you get this error:


LINK : fatal error LNK1000: Internal error during IncrBuildImage

And the build crashes. If you rebuild, it succeeds, but this doesn't help much for automated builds that rely on a deterministic compile result. The problem only appears if you have incremental linking turned on, but disabling incremental linking makes debugging unbearably slow and painful.

I'm not the only one having this problem:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=311732
If you are reading this blog, please go to that page and register your vote for an importance of five stars.

So I've just been living with that problem, since we're largely in a testing phase for VS2008 and automated builds aren't yet needed. Today I ran into another problem, and this one's a showstopper. The first time the compiler encounters a file that fails to compile because of a particular set of errors, the vc80.pdb program database file is corrupted and it's impossible to build any more files until you do a Rebuild All. Except that rebuilding everything causes the file to be rebuilt, which again corrupts the PDB file. Even if you try to compile that file by itself (using Build/Compile), the PDB file is still corrupted.

This problem was not only known, if was reported during the beta process, but never fixed:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=308775
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=309462

Both of the above problems have been open for 30 to 60 days. Perhaps a severity of "entire build fails catastrophically" isn't as serious as I think it is.

Whither SQL Server Reporting Services?

2007-12-25T22:18:00.000-07:00

Over the past several years, I've sometimes needed to run reports on a SQL Database. Given the ease with which reports can be created in Access, I figured that there had to be some reporting capabilities built into SQL Server. After some research, I was very surprised to learn that this wasn't true.

There were two obvious alternatives: stick with the free version of Crystal Reports that comes with Visual Studio (which is limited) or buy a commercial reporting tool. The old R&R Report Writer from dBase days was inexpensive and effective. Suffice to say that reporting tools today are dramatically more expensive.

Months later I learned about something called SQL Server Reporting Services. The only problem was finding it. Have you seen how many versions of SQL Server there are out there? MSDE, Express, Workgroup, Standard, Developer, Enterprise... Then you have to sort through Add-Ons such as Analysis Services, Reporting Services and Integration Services. Then you have to figure out what combinations of add-ons can be used with which versions of SQL Server. Finally, you have to figure out what obscure file to download to enable each of the various add-ons.

And did I mention the difficulty in sorting out the licensing rights to the various versions for us in development (non-production) environments?

And, to top it all of, it'll be a cold day in hell before you find any of this in Microsoft documentation. It was actually easier to find some of the information in Wikipedia than in Microsoft's docs!

So here I am, running Visual Studio 2008 on Windows Vista. Of course, VS2008 doesn't ship with SQL Server, you have to install SQL Server Express 2005. No problem there, but SQL Server Express 2005 doesn't ship with Reporting Services and you won't find any download labeled Reporting Services.

I thought I had struck gold when I found an article titled Introducing Business Intelligence Development Studio. That article says that Visual Studio 2005 has specific project types for creating reports and that you just need to select Add New Project. Fantastic! Of course, when I looked in VS2005 there was no such project type and the article had no troubleshooting tips.

Are you detecting a trend here?

I found a post where numerous people attempted to figure out how to get through this morass. The final conclusion was that you had to uninstall the Workstation components and install a new copy from a fresh download of the "SQL Express Toolkit." Of course, the person didn't post the link to that download and it isn't on MSDN Downloads.

I finally found it under the name Microsoft SQL Server 2005 Express Edition Toolkit. When I tried to install it on Vista I got a "compatibility warning." It remains to be seen what problems that causes. (Note that the Business Intelligence components under Workstation are not selected by default, so you have to make sure you check them manually.)

At this point I opened VS2008, tried to add a new project, and discovered that Business Intelligence Project was not one of the choices. (And yes, I did restart VS2008.) More digging on the web uncovered this little gem: Visual Studio 2008 Will NOT Support SQL Server 2005 Reporting Services projects.

So I started VS2005 and saw that Business Intelligence Projects was now visible at the top of the list in the Add New Project dialog in VS2005. Under that project type were two entries for Report Server Project. I also found the SQL Server Business Intelligence Development Studio (BIDS) in my Start menu under SQL Server 2005. I'm not yet sure exactly what the difference is between BIDS and the VS2005 projects.

If you need to install something other than Reporting Services, the instructions above won't help you much. After Setup is complete, you'll see this message:

The version of Business Intelligence Development Tools that is included in SQL Server 2005 Express Edition Toolkit does not include projects for SQL Server 2005 Integration Services or SQL Server 2005 Analysis Services. These projects are available only with editions of SQL Server 2005 that include Integration Services and Analysis Services. SQL Server 2005 Express Edition does not include Integration Services or Analysis Services.

You'll notice that this message does not tell you which versions of SQL Server will do what you need.

So I've gotten further than I was, but I still have several questions to resolve:

What is the relationship between BIDS and VS2005?
How do I install Analysis Services?
How do I install the server-side components of Reporting Services?
How will all of this change when SQL Server 2008 is released?

Maybe I should just use Microsoft Visual FoxPro, which is awful, but at least R&R Report Writer still supports it.

[Update 1/9/2008] Here are some initial answers to my questions:

BIDS runs in Visual Studio 2005. It's a "personality", just like C++ versus C#.
TBD
You have to install SQL Server 2005 Express Edition with Advanced Services SP2. In addition, when you do that installation, you have to specifically enable the features for Reporting Services. My recommendation is to enable everything. Don't forget to enable the subprojects. Make sure you install this BEFORE you install Visual Studio 2005 or 2008.
TBD

WinSxS Breaks Old Libraries

2007-12-07T15:32:00.003-07:00

After my previous experiences with handling Windows Side-by-Side Assemblies (WinSxS) in remote debugging and Isolated COM, I thought I was actually starting to get a handle on how it worked. Today I got stuck on another WinSxS problem, this time while porting our application to Visual Studio 2008.

The problem was that the application would fail to fail to start, with an error about the manifest being wrong. I used sxstrace, a handy tool under Vista, to try and determine what was happening. Sxstrace generated 180 lines of information about Vista's attempt to find and load the correct assemblies. It ended up being too much information. The only obvious problem I saw was that one of the DLLs being loaded was from Visual Studio 2005, not 2008.

I used Depends to look at the file and received the same errors. I looked in the Event Viewer and saw this error:

Activation context generation failed for "s:\csi\xsales\debug\XSALES.EXE". Dependent Assembly Microsoft.VC90.DebugCRT, processorArchitecture="x86", publicKeyToken="1fc8b3b9a1e18e3b", type="win32",version="9.0.20706.1" could not be found. Please use sxstrace.exe for detailed diagnosis.

This is even stranger, because 20706 was the version ID of Visual Studio 2008 Beta 2. I'm running the final release.

I used Visual Studio to open XSales.exe in Resource mode so I could look at the manifest itself. Here I found references to four versions of DebugCRT. The question was, where were they coming from? My code makes no explicit reference to assembly versions.

What I discovered was that the intermediate manifests generated by the compiler and linker include assembly information from all objects used by the linker, including objects from libraries, of which I had two. One of my .LIB files was generated by VS2005 and another was generated by VS2008 Beta 2, which is what caused the references to old assembly versions. Once I rebuilt those .LIB files with VS2008, the problem went away.

The lesson learned from all of this is that .LIB files are no longer easily portable between versions if they rely on any of the CRT or MFC DLLs. The painful part of this is that the problem doesn't show up until you try and run the software because none of the development tools warn about the inconsistency.

[Update 11/19/2008] You can find another solution here.

Corrupt \Boot\BCD in Windows Vista

2007-11-13T23:34:00.000-07:00

Two nights ago, I started installing a video driver upgrade and went home for the night. The next morning I came in to discover that something had failed in the upgrade and the system was hung on a black screen. Even Remote Desktop access wasn't working.

I hit the Reset button. The system started to reboot. As per usual when I use the Reset button, the RAID array was incensed that I had interrupted its normal operation and took its revenge by taking four hours to validate the mirror. 89% of the way through validating the mirror, I received my first Vista Blue Screen of Death (BSOD). Something about a power driver not handling an event.

No problem. I hit the Reset button again. What's another four hours between friends?

Except this time, the system didn't reboot. It went through the BIOS check, started to boot Vista, and presented me with this error:

File: \Boot\BCD
Status: 0xc0000034
Info: The Windows Boot Configuration Data file is missing required information

The Microsoft Knowledgebase has a helpful article on this subject, which should have worked, but didn't. The error messages during the failed repair didn't clarify the cause of the problem.

I tried booting Vista from DVD and running recovery. Rebuilding the BCD file is a standard operation, except that it didn't work. The Recovery Manager said that it couldn't save the file. Curiouser and curiouser.

I restored the BCD file from backup (Acronis True Image pays for itself... again.)

I rebooted the system and got the exact same error. I went back in with the Recovery Manager and discovered that the file \Boot\BCD was gone, even though I had just restored it.

The final solution was to use Acronis to restore track zero and the Master Boot Record (MBR) from my last image backup. There never was a problem with the BCD file, that was just a red herring. I'm not sure how you'd solve this problem if you didn't have an image backup. It just goes to show the usefulness of image-level backups.

Update (11-15-2007): I discovered that the Vista repair utility created a file named C:\Temp\SrtTrail.txt. This file contained the list of tests that were performed. The last test showed this:

Root cause found: 
---------------------------
No OS files found on disk.

Repair action: Partition table repair
Result: Failed. Error code =  0x3bc3
Time taken = 154722 ms

If I'd seen this file earlier, I would have had a clear indicator to repair track zero instead of stumbling on the solution accidentally.

Legacy Code from ... CP/M?!?

2007-11-13T22:51:00.001-07:00

Remember CP/M? Unless you are over 40, my guess is probably not. This week I ran into a bug in our software that traces its roots all the way back to CP/M in the 1970s. Over thirty years later, the code still exists in the Visual Studio 2005 C Runtime Library, waiting for the next innocent victim to run afoul of it.

Here's what happened. We had reverse engineered a data file format and had shipped the first pass to customers. Our only sample of the data file was quite small, about 10 records, but it was enough to determine the file format and make it work. The ten records in the file converted cleanly.

After the product shipped, several customers reported that only 26 records were being loaded. This was obviously incorrect as most of their files had hundreds of records. Our end-of-file handling code was common with numerous other modules and worked fine. The 26 records that did convert did so correctly.

Even when we instrumented our software to give more insight into what was happening at customer sites, we found nothing. Our software converted 26 records, detected end-of-file, and exited. WTF?

When we finally found a customer willing to share his data file, we ran our software in the debugger and got exactly the same results. 26 records were converted and the software exited cleanly, with no errors.

The lightbulb didn't go off over my head until I was looking at the data file with the cygwin "cat -v" command, which shows ASCII codes 0 through 31 as control characters. This particular data file had two bytes for each record ID and record numbering started at 0. The 27th record contained ID 26 (0x1a) which showed up as ^Z. Does that ring any bells? If you ever developed for CP/M (or for MS-DOS 1.0) it should.

Thirty years ago, CP/M only tracked the number of blocks in each file, not the number of bytes. By convention, Ctrl-Z was used to denote "end of file" for text files. MS-DOS 1.0, which bore a striking resemblance to CP/M internally, followed this same convention. At the time, the C Runtime Library understood this convention and automatically generated an "end of file" condition when ^Z was encountered. Today, the VS2005 C Runtime Library still contains that code and generates the end-of-file condition even if the exact length of the file extends beyond that point.

The bug in our software was that the file had been opened in text mode instead of binary mode. Normally this is easy to detect because the records become out of sync as they are read, but by some coincidence of the data layout, the data in this file was read in perfectly up until that ^Z.

So now I have one more reason to dislike CP/M, although (admittedly) in this day and age it seems somewhat pointless to carry a grudge against a dead operating system that was designed to run off of 8" floppy disks. Old habits are hard to break.

Static Code Analysis with Visual Studio 2005

2007-10-14T23:01:00.002-07:00

Not too long ago I received a jarring reminder that "char" in Visual C++ is signed by default. The problem was related to the ctype functions, such as isalnum() and isspace(), all of which take an integer. When you sign-extend an eight-bit multibyte character (MBCS), you end up with a negative number. The ctype functions then do a table lookup on a negative number and your program crashes. This wouldn't be so interesting, except that this bug existed in our production code for six years before the software crashed for a customer in Europe. If the customer hadn't been willing to give me sample data to reproduce the problem, I'd still be scratching my head.

Apparently I'm not the only person who has run into this issue. Visual C++ 2005 includes a command line option named "/analyze" for finding this problem and many others. I had tried /analyze during the Whidbey Beta two years ago but didn't get useful results. Now the results were much more helpful.

The "/analyze" options enables the static code analysis feature in the C++ compiler. This feature is based on prefast, a technology that came out of Microsoft Research in 2001. Prefast knows about hundreds of common programming problems. I ran it on our codebase, which after eight years is quite mature and already compiled under warning level 4 with minimal problems.

What I learned was quite interesting. Prefast knows about the ctype problem I mentioned earlier and pointed out several lines I missed when I tried to fix the problem. All of those lines were crashes waiting to happen.

Prefast found several places where I expected one return type but was getting another. For example, at one point I checked for an HRESULT but was actually getting a bool, which meant that the sense of my error check was inverted.

Another error that prefast found was where I was calling sizeof on a variable that was defined as pointer instead of as an array, which meant that the length being handed to strncmp was wildly wrong. The code still worked, but that was a happy accident.

Prefast also had numerous warnings about the Boost C++ Library. It should be possible to suppress those warnings, but I haven't done so yet.

Prefast supports annotating your code with the Standard Annotation Language (SAL) to better describe the static behavior of the code. If you've looked at recent versions of the Windows SDK, there are numerous annotations such as __out_ecount(). These annotations provide additional information to prefast that allows for better analysis. You can learn more about these annotations at blogs.msdn.com.

Prefast also pointed out two problems with my code that I would never have found, even with a close code inspection. I'm sure you've seen stricmp, the case-insensitive version of strcmp. I was using stricmp to check for keywords. Turns out that this is a bad idea. Prefast gives warning C6400, which explains that some languages interpret combinations of letters as a single letter, which changes the behavior of stricmp. The correct solution is to use the Windows API call CompareString(), which should be set to LOCALE_INVARIANT or LANG_ENGLISH, depending on the version of Windows.

Prefast also warned me that using _alloca in a loop could cause a stack overflow. Normally I'd consider this obvious, but in this case the warning was being given about the W2A Unicode to MBCS conversion macro, which uses _alloca. I looked at the source code to W2A (in atlconv.h) and the problem seems to be handled properly, but it was a worthwhile exercise.

I'm adding prefast to my bag of recommended tricks. For more insight into the use of this tool, I invite you to read Scalable Defect Detection from the Center for Software Excellence at Microsoft.

Visual Studio 2008 Beta 2 Test Results

2007-10-12T11:31:00.001-07:00

Today I tried building our product with Visual Studio 2008 Beta 2. The last two times I tested compiler upgrades (VC6 to VS.net 2003, then VS.net 2003 to VS2005), the upgrades were extremely painful and took days to complete.

I was pleasantly surprised that rebuilding our product in VS2008 Beta 2 went quite smoothly. The issues were minor:

The switch /OPT:NOWIN98 option is no longer supported.
Impact: None.
Project could not be linked against a library originally built in VC6.
Impact: Rebuild library.
Visual Studio 2005 manifest that set requireAdministrator caused this error: manifest authoring error c1010001: Values of attribute "level" not equal in different manifest snippets.
Impact: Removed the explicit manifest file and set the UAC level in the Linker node on the project properties.
MT.EXE gives an error on a valid manifest.
Impact: Move $(WindowsSdkDir)\bin to the top of the list for Executables files in VC++ Directories under Tools/Options.

In terms of the size of the generated executables, here are the stats:

Product	VS2005 size	VS2008 Size	Net Change
#1 (C++/MFC GUI)	1,198,592	1,180,672	-1.5%
#2 (C++/MFC GUI)	1,589,760	1,549,312	-2.5%
#3 (C++/MFC Console)	2,043,904	1,998,848	-2.2%

While not earth shattering, these numbers do show that they were able to contain bloat in MFC and the C runtime library. All of these applications are statically linked against MFC and the CRT.

The biggest surprise was that VC2008 worked fine with version 1.34.1 of the Boost library, which has not yet been tested on VC2008. (although, admittedly, our software only uses one or two of the Boost modules.)

That's it! So far, backwards compatibility is excellent.

Initial testing has shown no problems with the generated code. Common dialogs under Vista were automatically updated to Vista styling, as promised.

My only disappointment is that very little was done for the IDE for C++ developers in this version of Visual Studio. The tools for editing dialogs and other resources are still awful compared to VC6. For a look at the future of the IDE, take a look at Somasegar's blog. Attention is being renewed on the IDE for native C++ developers, but improvements won't ship until 2010.

Running the Visual Studio 2008 Beta 2 VHD on VMware Server

2007-10-11T10:07:00.000-07:00

Microsoft has finally climbed on the bandwagon for using pre-configured virtual machines to distribute beta software. For anyone who tried to install earlier betas of Visual Studio 2005 Team System, you'll understand me when I say that these pre-built virtual machines will save you days of frustration.

I downloaded the VHD disk images for Visual Studio 2008 Beta 2. Although they work fine on the free download of Virtual PC 2007, I really wanted to run this image on my virtual machine server, which uses VMware Server (see my earlier comments on VMware Server versus Microsoft Virtual Server).

Before I describe the procedure, one BIG caveat: Once Windows is running in VMware, Windows will complain that it needs to be reactivated. If you are a Microsoft Partner you can get a key from MSDN Downloads. Otherwise you will need to use a new key, which basically means you need to buy Windows Server 2003 Enterprise. Therefore, if you don't have a ready supply of activation keys, this procedure won't work for you. It may be possible to call the activation people and have them honor the key built into the virtual machine, but I haven't tried.

Converting the Orcas VHD to VMware ended up being a lot more difficult than I'd hoped. The biggest problem was converting the virtual drives from .vhd format to .vmdk format. I found a nice utility named WinImage that could do this. WinImage converted the base Orcas image (2.8GB) without difficulty, but gave up with no error when I tried to convert the 11.8GB differencing disk. I didn't really want to go back and forth with the WinImage support for two days, so I looked for an alternative.

My final solution was to use Acronis to do a backup in Virtual PC, then use Acronis again to do a restore in VMware. To do this yourself, you'll need:

VMware Server (free)
Acronis TrueImage Home or better (commercial)
A Windows Server 2003 Enterprise installation CD
WinImage (shareware)

The solution was as follows:
1. Install Acronis TrueImage on any Windows box and create a Rescue CD.
2. Create a virtual machine in Virtual PC 2007 that contains Orcas Beta 2.
3. Set the virtual machine to connect to the CD you created in Step #1.
4. Boot the version machine, select Acronis and back up the Orcas virtual machine to any desired network drive.
5. Use WinImage to convert the Base01 image to VMDK. Make sure you create a dynamic disk and not a fixed disk.
6. In VMware Server, create a virtual machine that points at the file from #5.
7. Put the Acronis Rescue CD in a CD drive on that computer.
8. Start the virtual machine in VMware, press Esc, and boot from the Rescue CD.
9. Restore the Acronis backup to the current disk.
10. After restore completes, reboot the virtual machine. You'll get an error about a service that didn't start. Ignore it.
11. On the VM menu, select Send Ctrl-Alt-Del.
12. Enter the password from the Microsoft web page. You'll need to use the keyboard, your mouse probably won't work.
13. As the login completes, you'll be prompted for the path to install the Ethernet card. Put in the Windows Server 2003 Enterprise installation CD and click OK. Windows will complain that it can't find the driver, which is okay. Tell Windows you want to install from an alternative location.
14. Select the file Driver.cab on the Windows Server 2003 Enterprise installation CD. The network driver should be located automatically.
15. After installation of the initial driver completes, do not reboot or you won't be able to login again.
16. On the VM menu in the VMware Console, choose Install VMware tools.
17 At some point you'll be prompted for the file mouclass.sys. Tell the installer to use the copy of the file in C:\Windows\System32\Drivers.
18. Now you can reboot and everything should work.

Upgrading PEAR on Red Hat Enterprise ES 3

2007-09-15T12:45:00.003-07:00

Recently we've upgraded our coding standards for our server-side PHP development to include unit tests, which required that we install PHPUnit for PHP 4. Normally installing a new pear module takes about ten seconds, like this:


pear install PHPUnit

Unfortunately, this time things didn't go so smoothly. I ended up with dozens of errors such as this one:


Warning: xml_parse() [http://www.php.net/function.xml-parse]: Unable to call handler _pkginfo_cdata_2_0() in Common.php on line 758

The list of errors ended with:

Notice: Undefined index: package in Common.php on line 1122
The following errors where found (use force option to install anyway):
missing package name
missing summary
missing description
missing license
missing version
missing release state
missing release date
missing release notes
no maintainer(s)
no files

If you look up these errors on Google, you get very few hits. Justin Patrin correctly identifies the problem: Red Hat ES 3 ships with version 1.1. This version of pear is considered ancient and doesn't support reading the latest packages.

You can check your version of pear with this command:


pear -V

Justin recommends the standard command for upgrading pear:


pear upgrade pear

Unfortunately, this command doesn't work when your current version of pear is so far out of date. In fact, the net result of this command is that the "pear" command is removed and not replaced. If you follow the link, you'll see a bug about this in the pear bug database, where the problem is considered "Bogus" (not a bug) because the author of the bug is running a "museum PEAR version (RHE3, pear 1.1)." Unfortunately, there is no discussion of how to actually perform the upgrade, since the command "pear upgrade pear" is completely broken in RHE3. I find it absurd that anyone who supports enterprise software would think that it's okay to simply orphan customers who are running software four years old.

I found several articles on resolving problems with old versions of pear. All of them recommended reinstalling pear from scratch, similar to this:


lynx -source http://go-pear.org/ php

Unfortunately, this didn't work for me. The Red Hat Package Manager (rpm) has very definite ideas as to where pear should be installed. However, rpm bundles pear with php, so you cannot simply uninstall Red Hat's pear and then install the standard version. In my case, I was unable to determine how to get go-pear to align its directory structure with rpm, and I ended up with a schizophrenic installation.

I removed all of /usr/share/pear and restored the directory from backup. I then double checked the installed packages:


pear list

Pear reported that no packages were installed. Since the Red Hat version of pear actually ships with several packages, it was clear that my installation was now completely broken.

After several hours of searching, I finally found on article by Patrick Berry titled Upgrading PEAR on RHEL 4. Although I was running RHEL 3 and not 4, the solution described gave me enough information to solve the problem.

The problem is that version 1.1 of pear cannot read the package to install the latest version, which is 1.67 as of this blog entry. Therefore, you have to go through some intermediate upgrades before eventually arriving at version 1.67.

I first tried following the steps under Patrick's Solution section. It didn't work for me - I received the error pear is not installed. This error appears to have been caused by my earlier attempt to install from go-pear. After much frustration, I solved this problem by removing the file /root/.pearrc.

However, Patrick's steps under Solution still didn't work for me. If I had bothered to read the Update AND the Comments, I would have seen the latest correct upgrade procedure, as described by Daniel a mere ten days ago:


pear upgrade pear-1.3.3
pear upgrade pear-1.4.11
pear upgrade

However, AGAIN this procedure didn't work for me. (Are you sensing a trend here?) When I tried to go from 1.3.3 to 1.4.11, I received an error about "getopt2 not found." It's not clear what caused this problem, but I saw a similar issue reported by others, again with no solution.

To resolve this problem, I had to manually download an interim version of Console_Getopt and copy it to /usr/share/pear/Console. The latest version was too new and version 1.21 was too old. One of the versions in the middle included getopt2 and didn't rely on any other unsupported functionality.

Once I put in place a newer version of Console_Getopt, I was able to upgrade to pear 1.4.11, and then to the latest version, with these commands:


pear upgrade pear-1.4.11
pear upgrade-all

The upgrade-all command also fixes the pear package manager so it shows the correct version of Console_Getopt.

After all of this effort, my installation of pear finally worked again, after wasting a mere two days figuring out all of the problems. (Begin Sarcasm) A big thank you to both Red Hat and the "pear" package manager maintainers for abandoning users of Red Hat Enterprise 3.(End Sarcasm)

Late-Breaking Information:
I just learned that the PEAR home page says that support for PEAR 1.3.6 and earlier will be dropped on January 1, 2008. Since today is still four months before that date, it seems that they've jumped the gun a little bit. However, they do give the definitive steps on how to upgrade your PEAR installation:


pear upgrade --force PEAR-1.3.6 Archive_Tar-1.3.1 Console_Getopt-1.2
pear upgrade --force PEAR-1.4.11
pear upgrade PEAR

However, it's impossible to find this from any search engine because they don't list the symptoms or the affected operating systems, nor do they refer to this procedure in Bug #5906 that I mentioned earlier.

Even More Late-Breaking Information:
Turns out that my pear installation still had issues because of all of the experimentation, so I had to wipe it and start over. This time I was able to get go-pear to work by following the instructions on this web page:
http://aspn.activestate.com/ASPN/Mail/Message/pear-general/1305645

How To Create 32-bit Import Libraries Without .OBJs or Source

2007-08-26T01:00:00.001-07:00

This article is intentionally titled the same as Microsoft KB article 131313 (was Q131313) because that KB article does not give enough information for you to create a .LIB file that works with Windows system DLLs.

There are some functions in the Win32 API that have no import library. One example is the function named RemoveControlByName() in OCCACHE.DLL. According to the documentation, the only way to use the function is to use LoadLibrary() and GetProcAddress(). This is error-prone, requires a lot of code to maintain, and isn't usable with the Delay-Load feature of VC++.

Obviously it would be preferable to have a .LIB to link against, but creating such a library is challenging. If you use Dependency Walker to look at OCCACHE.DLL, you'll see that the function is named simply RemoveControlByName. While that seems obvious, it shouldn't be possible to have this name because it doesn't include any notation for the calling convention.

If the function were __cdecl, then the function name should have started with an underline, such as _RemoveControlByName. If the function were __stdcall, then the underline should have been added as well as suffix to indicate the number of bytes in its call stack. RemoveControlByName has five 4-byte paramters, so a __stdcall signature should have looked like _RemoveControlByName@20. However, the function name has no decorations at all, which should be impossible according to Microsoft's discussion of name decoration.

The Q131313 article discusses the general case of manually creating a .LIB file for a .DLL file. The discussion under Creating a .DEF File says "The reason for this limitation is based on an assumption made by the LIB utility that all names are automatically exported without a leading underscore." This seems promising because we don't have a leading underscore. However, functions exported from Windows DLLs(almost) always using __stdcall, and the discussion under Creating a .DEF File is only applicable to __cdecl.

In spite of those warnings, I spent quite a bit of time trying to craft a .DEF file that described what I was trying to do. (You can use LIB.EXE to compile a .DEF file to a .LIB without any .OBJ files) Although most .DEF files simply list the undecorated function names under the EXPORTS section, the syntax of the .DEF file allows for creating aliases and other strange constructs. Some of the things I tried included:


; Raw function name
RemoveControlByName

; Alias the undecorated name to __stdcall
RemoveControlByName=RemoveControlByName@20

; Explicit reference to the necessary DLL
RemoveControlByName@20 = OCCACHE.RemoveControlByName

However, none of these generated a .LIB that worked. The first two I couldn't link against and the third would link but fail to run.

I thought I'd learn something by looking at the symbol that the parent program is trying to link against, but that was even worse:


__imp_?RemoveControlByName@20

I knew my header file was correct, so that was definitely the correct name. However, you'll notice the leading "__imp", which indicates that the symbol is being imported from a DLL. This was something else that apparently needed to be included in my hand-crafted .LIB file, and I hadn't seen any discussion anywhere on how to do that.

I've tried to solve this problem two other times in the last several years, and this was the point I gave up in both of those cases. However, this time, failure was not an option. I needed the solution.

I tried the second option in the KB article, described under Stubbing Out Functions. I painstakingly created a dozen functions that mimicked the signatures of the functions listed in the header file. If you are doing this for yourself, here's a tip for creating a Visual Studio project. The final file you need is a .LIB file, so it's tempting to use one of the Visual Studio projects for creating a .LIB. However, that's wrong. If you created a .LIB, then you'll link with the stub functions you created, which is useless. What you really want is a DLL project, which happens to create a .LIB as a by-product.

Anyway, I created the functions as described in the KB article. Since my project used .cpp files, all of the calls in my header file had to be declared extern "C". For example:


extern "C" {
#include "occache.h"
};

If you look in any of the standard Windows include files, such as winbase.h, you'll see this same declaration. Note that this declaration has no relation to _cdecl and therefore has no impact on the calling convention. In other words, it doesn't force all of the functions in occache.h to be called with _cdecl. What this declaration does is to modify all of the linker symbols so that they won't include the C++ name decorations, which encode all of the function's parameters into the function name.

I also updated all of the function signatures in the header file to include __declspec(dllexport).

I compiled it, linked my main application to the new .LIB file, and it linked! I thought I was done, but when I ran the application, I received the error "Entry Point Not Found: The procedure entry point _RemoveControlByName@20 could not be located in the dynamic link library OCCACHE.DLL"

I examined the .LIB with DUMPBIN. Under Public Symbols, I now see two definitions for RemoveControlByName:


_RemoveControlByName@20
__imp__RemoveControlByName@20

It appears that the "__imp" definition was a result of adding __declspec(dllexport), so that explained why the application linked successfully. One problem solved.

Continuing my examination of the DUMPBIN information, I saw that RemoveControlByName was defined as:


Archive member name at FE0: OCCACHE.DLL/
46D12949 time/date Sun Aug 26 00:18:33 2007
         uid
         gid
       0 mode
      38 size
correct header end

  Version      : 0
  Machine      : 14C (x86)
  TimeDateStamp: 46D12949 Sun Aug 26 00:18:33 2007
  SizeOfData   : 00000024
  DLL name     : OCCACHE.DLL
  Symbol name  : _RemoveControlByName@20
  Type         : code
  Name type    : name
  Hint         : 9
  Name         : _RemoveControlByName@20

To determine whether or not this was correct, I used DUMPBIN to compare against known-good definitions in USER32.LIB, where I found that the "Name type" in the USER32 records was defined as "undecorate" instead of "name". Obviously, there was a magic incantation to set this flag, presumably in the .DEF file.

I added a .DEF file and spent several hours trying various alias combinations, none of which worked. Finally, in desperation, I created a .DEF file that contained just the raw function names. For example:


EXPORTS
RemoveControlByName

I built my application, it linked, and it ran. What happened?

I ran DUMPBIN again on my library. The record describing RemoveControlByName now contained the "undecorate" attribute:


Archive member name at FFC: OCCACHE.DLL/
46D12740 time/date Sun Aug 26 00:09:52 2007
         uid
         gid
       0 mode
      38 size
correct header end

  Version      : 0
  Machine      : 14C (x86)
  TimeDateStamp: 46D12740 Sun Aug 26 00:09:52 2007
  SizeOfData   : 00000024
  DLL name     : OCCACHE.DLL
  Symbol name  : _RemoveControlByName@20
  Type         : code
  Name type    : undecorate
  Hint         : 6
  Name         : RemoveControlByName

Also, the very last line in the record showed that the "Name" was RemoveControlByName, with no decoration. Exactly what I needed.

It's clear that there's quite a bit of undocumented behavior here. Adding the entry to the .DEF file had a rather dramatic effect on the generated library. I couldn't find any mention of this behavior in the documentation on .DEF files. The only relevant reference I could find was in Microsoft's documentation under __cdecl, where it says "Underscore character (_) is prefixed to names, except when exporting __cdecl functions that use C linkage." This statement is true, but it's not the whole truth. To create a .LIB file that can link against such a construct, you also must have the function declared in a .DEF file.

In summary, to create a .LIB file that will let you link with Windows system DLLs, you need to:

Follow the instructions in Q131313 under Stubbing Out Functions. Make sure you name the project the same as the Windows DLL.
Make sure your dummy functions are defined with __declspec(dllexport) as well as __stdcall.
For the header file used by the parent application, make sure that your function declarations are surrounded with extern "C".
Add a .DEF file to your project that includes the function names with no decoration.

Media Player Slows Network in Vista

2007-08-25T22:01:00.002-07:00

If you've read my earlier blog posts about Gigabit Ethernet, you know that I've had my share of difficulty getting good performance out of my GigE network. I've also posted that I've had problems with Vista that I didn't see in Windows XP or Windows Server 2003. Now I know why. Windows Media Player puts a big throttle on GigE network performance, even if it is paused and not playing anything.

The issue was reported at 2CPU.com and a response from Microsoft was reported at ZDNet. The problem is minor on 10/100 Ethernet, but on a GigE Ethernet the network performance can be throttled back to 100Mps levels. Apparently, there's no registry setting that will resolve the problem, the only solution is to shut down Windows Media Player. There are scattered reports that the problem happens with other media players, such as WinAmp, but I haven't confirmed these reports.

There's another problem I also ran into that caused GigE to throttle back to 100Mbps. This was a self-inflicted problem, but it took several weeks to resolve. I have a LinkSys WRT54G router that handles my gateway/firewall/NAT. All of my servers get their IP address with DHCP and the WRT54G is set to always hand out the same IP address to those servers. This was done so that the servers would get the DHCP settings from our provider. Anyway, one of the servers was migrated to a new motherboard with a different MAC address. The WRT54G started handing out a random DHCP address to that node but still reported that node name as having the other, assigned IP address, all of which made the network become schizophrenic over the IP address assigned to that particular node name. The result was that any traffic that was destined for that node ended up going to the gateway, which didn't have a GigE connection, so traffic was throttled back to 100Mbps speeds.

Update 8/28/2007 - Mark Russinovich has posted a detailed analysis of the network slowdown. It turns out that the more Network Interface Cards (NICs) you have in your system, the worse the problem gets. I have three NICs, including WiFi, which slows my performance to a theoretical maximum of 9MB/second.

ReadyBoost Damages USB Thumb Drives

2007-06-20T09:05:00.001-07:00

Windows Vista ReadyBoost destroyed a second thumb drive yesterday. For the past three months I've just been leaving my computer on at night instead of trying to make ReadyBoost work with Sleep/Standby. Yesterday I was trying something new and I put the system in Sleep mode. I woke it up two hours later and the thumb drive had stopped working. I unplugged the thumb drive and replugged it and I got a "write protect" error. I tried the thumb drive on another computer and got the same result.

I downloaded the Format utility from the Apacer web site. I successfully reformatted the drive, enabled ReadyBoost, and was working again. For a few minutes. Then the light on the thumb drive turned on steady and the drive completely died. I tried it on three different computers and three different versions of Windows.

This is the second thumb drive that's failed when using ReadyBoost and resuming from Sleep. I've heard reports from readers of this blog that they have had a similar problem. How about you? Leave a comment.

Note that I don't believe that this has anything to do with the number of write cycles, as some journalists have complained about. In both cases the drive died immediately after resuming from Sleep.

Dell PowerConnect 2708 Performance

2007-06-06T22:18:00.001-07:00

Just a quick note to say that the Dell PowerConnect 2708 is fantastic. I was able to get a sustained 112MB per second (that's 112 megabytes, not megabits) between two Core 2 Duo machines (Vista/Windows 2003 Server R2) using PCATTCP. That means that the connection is completely saturated and that the Dell box is switching at the maximum possible speed. Sweet.

In more real world numbers, I was able to get sustained 2.2GB/minute transfer rate using normal Windows file sharing copying from Vista to the server when the file copy was initiated on the Windows 2003 Server side. That's close to the maximum read speed for the source disk, so presumably the disk was the bottleneck and not the network.

Oddly enough, when the file copy was initiated on the Vista side, I only got about 1.2GB/minute with the same file being copied to the same destination. I can't explain that.

[Edited 8/27/2007 to add information on the read speed of the disk.]

Logo Certification - Success!

2007-06-05T13:12:00.001-07:00

Last night we received word from Lionbridge that we had passed our Windows Vista Logo Certification test. Thus I've crossed the finish line of a journey that started in May of 2006, bringing to close a year-long project to make it all happen. It was a monumental effort to bring all of the pieces together because all of the vendors we rely on had to make their products Vista compliant before ours could pass certification.

There were several "firsts" to this project. I created an MSI installer for the first time. I installed a 64-bit version of Windows for the first time. We ported our code base to VC2005. I worked with Microsoft Consulting Services for the first time. And, not surprisingly, I did my first serious development work under and for Vista. So it's been quite an adventure.

The most important of these was our MSI install generator, Advanced Installer. You may wonder why we'd go with a relatively unknown install system instead of going with one of the "big boys" like InstallShield. The reason is that, between 1991 and 1998, I had to write several installers in InstallShield and and my company paid a lot of money for the privilege of doing so. I found InstallShield to be buggy and user abusive and The Stirling Group (the manufacturer) to be unresponsive. I stopped recommending them a long time ago.

The developers at Advanced Installer worked closely with all interested customers to make sure their installers could pass Vista certification. In several cases, they gave me information that the Vista Logo Certification tests were wrong and what they were doing was correct. In every one of those cases, Microsoft admitted that the Advanced Installer developers were correct.

Isolated COM

2007-06-04T21:41:00.000-07:00

One of the worst documented features of Visual Studio is Isolated COM. Actually, there's lots of documentation on Isolated COM, but none of it is particularly useful. The mechanics of making Isolated COM work under Visual C++ are actually pretty easy, so this short article tells you what to do.

Isolated COM allows your application to use ActiveX components without having to register them. The original vision of this was to allow XCOPY deployment of the application, but Isolated COM has many benefits. You can have a private copy of the DLL without worrying that another application will install an older or newer copy that breaks your application. Isolated COM also allows you to successfully install and run on non-Administrator accounts.

A short history of how activation works: In Windows 9x and NT, ActiveX components were found by looking in HKEY_CLASSES_ROOT, which was really a link to HKEY_LOCAL_MACHINE\Software\Classes. Windows 2000 added support for also defining components in HKEY_CURRENT_USER\Software\Classes and HKEY_CLASSES_ROOT became a merged view of HKLM and HKCU. Finally, Windows XP and Windows Vista added support for looking for COM information in the manifest of the executable. This means that the manifest contains the DLL name, CLSID information, interface information, and type library pointers - most of the information that would normally appear in HKCR.

As you might guess, these definitions can be quite lengthy and creating them by hand is error prone. Visual Studio 2005, which supports manifests, includes support for automatically generating and embedding the correct XML declarations.

To switch from "old fashioned" ActiveX usage to Isolated COM, follow these steps:

1. First, make sure components are created with their CLSID and not the ProgId. In other words, don't use Word.Document, use {F4754C9B-64F5-4B40-8AF4-679732AC0607}. If you are using the #import command in Visual C++, you can use the __uuidof operator instead of a string. For example:

~~HRESULT hr = pDoc.CreateInstance("Word.Document");~~
HRESULT hr = pDoc.CreateInstance(__uuidof(Word::Document));

2. If you made any changes for #1, test your code with the DLL registered the normal way and make sure it works. Also make sure that your code checks and handles all HRESULT return values.

3. Unregister the ActiveX DLL using: regsvr32 /u waycool.dll

4. Copy the DLL to be in the same directory as your executable.

5. In Visual Studio 2005:

Open the property sheet for the project.
Click the plus sign next to "Manifest Tool"
Click on "Isolated COM".
Next to "Type Library File" enter "waycool.tlb"
Next to "Component File Name" enter "waycool.dll"
Build the project.

6. Run your application in the debugger and single step through all of the activation code to make sure it is working properly.

Once you've added support for Isolated COM, don't forget to update your installer so that ActiveX components aren't automatically registered under Windows XP and Vista. Also don't forget to retest your software under Win9x and Win2K if you support those versions of Windows.

Netgear GS108 GigE Switch Failure

2007-05-30T21:16:00.002-07:00

I've had an unbelievably long run of hardware failures this year. First a Hitachi 500GB hard drive failed in my Windows 2003 Server, then the Apacer USB thumb drive I was using for ReadyBoost, and now my new GigE Ethernet switch, a Netgear GS108, is malfunctioning. Predictably, all of the failures have been after the 30 day mark, which means that I've had to go through an RMA process instead of a simple return.

The Netgear GS108 is particularly frustrating because it didn't do something simple like emit smoke and die. That would have been easy to diagnose. Instead it would alternate between a blinking failure mode and normal operation. In twenty years I've never seen an Ethernet switch or hub fail, so I wasted two days looking at everything except the switch.

The symptom is easy to see - all of the lights turn on for a second, then they all turn off, even on ports with nothing plugged in. Any PCs that are connected report that a network cable is unplugged. This may go on for a few seconds to a couple of minutes, then it will start working again, then it will fail again. The problem seemed to be most prevalent when 1000Mbps devices were mixed with 100Mbps devices. If all devices are 1000 or all devices are 100, then the GS108 worked fine.

Reviews on newegg.com show that a minority of other users have experienced the same problem I did.

I talked to Technical Support at Netgear. They immediately issued an RMA for the switch. Unfortunately, after waiting a week for the new GS108 to arrive, the new unit malfunctioned in the same way as the old unit. So I believe this problem is endemic to the GS108.

I've replaced the GS108 with a Dell PowerConnect 2708. It costs about 50% more than the GS108, but it is a fully managed switch that receives rave reviews from users. The Dell 2708 also includes a one year next-day parts warranty, as opposed to Netgear who charges you $50 for a next day exchange. You might as well order a new one from NewEgg.

I plugged in the Dell 2708 and it has worked flawlessly. I'm just running it in unmanaged mode since I have no need for VLANs or port aggregation.

The PowerConnect 2708 lists on Dell's web site for $158 if you just do a Google search, but that's the Medium/Large Business price. If you look in the Small Business section, the price is $109 with a $20 discount, or $89. Great deal:
http://www.dell.com/content/products/productdetails.aspx/pwcnt_2708?c=us&cs=04&l=en&s=bsd

Technical Blog for Jim Beveridge

Windows 7 Forgets My Multiple Monitor Layout

Base64 for Unicode UTF16

Convert a Locale Name to LCID in C++

Reading XML from C++

Google C++ Style Guide

Display Unicode in a Windows console window with printf/wprintf

How to: Debug with Code Center Premium Source (In Detail)

To prepare for debugging with Code Center Premium

Sample Code

Sample Walkthrough

Things That Don't Work

Troubleshooting

Running Visual Studio in a Virtual Machine

ANSI versus Unicode Functions

Source Server Paths

Extracting Icons with the Windows SDK

Norton Antivirus: The Clear Choice for SSDs

Lightning Fast Builds with Visual Studio 2010 and an SSD

Project is permanently out of date

Fixing Permissions on an External Hard Drive

Windows 7 Network Performance

Safely using erase() in STL

"Invalid pointer" error in Visual Studio 2010 in Configuration Manager

Debug "Just My Code" for C++ and MFC

Configuring Mozy Pro for Visual Studio

Understanding ReadDirectoryChangesW - Part 2

Getting a Handle to the Directory

Calling ReadDirectoryChangesW

Dispatching Completion Routines

Handling the Notifications

Using the Notifications

Shutting Down

Network Drives

Summary

Understanding ReadDirectoryChangesW - Part 1

Four Ways to Monitor Files and Directories

The Puzzle

Recommended Solutions

Wrong Techniques

Using MsgWaitForMultipleObjects in MFC

Visual Studio 2010 Tab Key Not Working

Static Library Dependencies in Visual Studio 2010

Visual C++ 2010 Apps Don't Support Windows 2000

Targeting C++/CLR v2.0 with Visual Studio 2010

Managing multiple configurations using Virtual PC

Layer Overview

Roadblocks

Create the Layer 1 Root VPC

Configure the Layer 1 Root VPC

Create a Layer 2 VPC

Create a Layer 3 VPC

Create Remaining Configurations

Licensing

Create separate references for Debug and Release DLLs in C++/CLR

Online Backups - The Good, The Bad, and the Ugly

How I Backup My Windows System for Multiple Contingencies

Six Myths About Backing Up Your Computer

Myth #1. I have a new hard drive, my data is safe.

Myth #2. Backing up once or twice a month is enough.

Myth #3. Backing up documents to an external drive is good enough.

Myth #4. Having an image backup on an external drive or a NAS is good enough.

Myth #5. I use RAID hard drives so I don't have to do backups.

Myth #6. Paying experts to do backups is good enough.

Conclusion

Installing PHP on Windows 7

Handling Exceptions from STL

Router for 50Mbps Broadband Service

Cox Ultimate Broadband - 50Mbps

Porting C++ Applications to Visual Studio 2010

Visual Studio 2010: Review/First Impressions for C++

NUnit Unit Testing with C++

CPAN error: Recursive dependency detected

PathMatchSpec Problems

Setting the default Windows SDK

Dangers of Buying Laptops Parts on eBay

Why I Custom Build My Computers

AppCrash in StackHash_1703

Dell 1815dn Printer Review: Stay Far Far Away

Linux vs Windows: Uptime