Tuesday, March 13, 2007

Making VC2005 Remote Debugging Work

I use Virtual PC extensively for testing older operating systems and non-standard configurations. When I have to debug configuration-related problems, I use Remote Debugging to connect to the Virtual PC instance and debug in-situ.

Before I upgraded to VC2005, remote debugging with VC++ 6 was straightforward. I updated the PATH on the remote computer to include MSVCMON.EXE, the related debugger DLLs, and the debug versions of the C runtime library and MFC. Worked great.

Remote debugging with VC2005 is more challenging. Between security problems and Windows Side-by-Side assembles, it can take quite a few steps to make everything work. Here is what I had to do.

Your debugger and Windows firewall double-team you to make configuring remote debugging as painful as possible. My first problem was to fix the firewall, since I couldn't connect to the Windows XP SP2 computer at all. I fixed this by disabling the Windows Firewall (if you are following along, don't do this unless you are absolutely certain it's okay for your environment!) Probably overkill, but simple.

My next problem was debugger authentication. This proved to be a challenge because my Virtual PC test machine is on a domain and my development machine is not. After several hours of effort, I determined that they simply were not going to agree on a trust relationship, so I disabled authentication. (Handy tip - you can do this from the command line with /noauth. Also add /nosecuritywarn if you don't want to go crazy with security prompts.)

I thought I was golden at this point. I clicked F5 to debug my executable and was told, "Could not start App.exe". So I tried to start the application from Explorer. Same result. I ran Dependency Walker to check my DLL dependencies and was reminded that I needed to update my path to include the VC2005 debug libraries for the CRT and MFC. I did so, clicked F5 again, and... it still didn't work.

Several hours and much frustration later, I had received a solid education about Windows Side-by-Side Assemblies, otherwise known as WinSxS. In order to cure "DLL Hell," Windows XP and Windows Vista provide support for an executable's manifest to hardcode the desired version of dependent DLLs. For executables created with VC2005 that are dynamically linked to MFC and the CRT, a manifest is automatically inserted into the generated executable that includes this version information.

Once this is done, Windows XP and Windows Vista become absolutely hell bent on loading the correct version of the DLLs. Trying to work around this protection is futile. None of the following "obvious" solutions work:

  • Putting mfc80d.dll in the same directory as the EXE, even if it's the correct version.
  • Including mfc80d.dll on the user's path.
  • Manually copying the appropriate directories from the \Windows\WinSxS on the development machine to \Windows\WinSxS on the remote debuggee.
After spending far too much time reading mostly useless documentation, I came across the correct solution. In your Visual Studio 2005 installation, you'll find a directory named:

\Program Files\Microsoft Visual Studio 8\VC\redist\Debug_NonRedist\x86

Inside this directory, you'll find two more directories named Microsoft.VC80.DebugCRT and Microsoft.VC80.DebugMFC. These directories must be copied to the same directory as your EXE and magically everything works right.

At this point you'd think the story is over, but it's not. Windows 2000, Windows NT and Win9x don't support WinSxS, so the manifest is ignored. To debug your application any of THOSE versions of Windows, you need to include the appropriate DLLs on your PATH (or copy the appopriate DLLs into the same directory as your EXE.) I set the path in Virtual PC to point directly to the WinSxS directories on my development machine, which were: ("duo" is the host name of the development computer, which drive C shared under the name "C")

\\duo\c\Windows\winsxs\x86_microsoft.vc80.debugmfc_1fc8b3b9a1e18e3b_8.0.50727.762_none_29a8a38855141f6e
\\duo\C\windows\winsxs\x86_microsoft.vc80.debugcrt_1fc8b3b9a1e18e3b_8.0.50727.762_none_24c8a196583ff03b

-
Finally, I used the following command in my batch file to start the debug monitor:

start "" "\\duo\c\Program Files\Microsoft Visual Studio 8\Common7\IDE\Remote Debugger\x86\msvsmon.exe" /noauth /anyuser /nosecuritywarn /timeout 7200

Finally, that made everything work properly.

4 comments:

  1. Thanks for the nice write-up. Clearer than most about how to solve the side-by-side issue.

    I really need to read up on the topic more.

    ReplyDelete
  2. Great clear explanation. The bit about copying the Debug_NonRedist\x86 folders fixed an issue that was blocking me.

    Thanks - Lee.

    ReplyDelete
  3. A reader wrote in to say that the steps in this article helped him get past the first hurdle, but he is now running into a new error message: "The application failed to initialize properly (0x80000003)."

    This is not a problem I've seen before, but I have a standard set of steps for debugging loader problems:

    - Run the Dependency Walker and look for files that fail to load. Make sure you download the latest version, the one that ships with VS/VC is out of date.
    - Enable the Fusion (WinSxS) log file and examine it.That link is a few years old, but I’ve used the tool on Vista, so I know it works.
    - I do much of my remote debugging with my debugger running in Vista and the application running on Windows 2000 just so I don’t have to fight some of these problems.
    - Set as many DLLs as possible to be delay load so that they don’t cause problems when the application loads. The C run-time library gives you extensive control over how delay-load libraries are loaded, which means you can put in error checking and logging. Look up PfnDliHook.
    - Run an API monitoring utility to track calls to LoadLibrary. The last call to LoadLibrary that fails is probably the culprit.
    - Narrow down the problem. Keep commenting out code (and thereby eliminating DLL dependencies) until it works, then add code back until it breaks. This can be a daunting task, but it’s better than beating your head against the wall for weeks.
    - Wild Guess – Will running on the debug version of Windows provide more information? I’ve never done this.

    ReplyDelete