- Open RegEdit
- On 32-bit Windows, go to:
HKEY_LOCAL_MACHINE\Software\Microsoft\
VisualStudio\10.0\NativeDE\StepOver
On 64-bit Windows, go to:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\NativeDE\StepOver
At least for Visual Studio 2010, there are several default functions defined. - Add a new String value. Name it CStringT.
- Edit the key and set the value to:
ATL\:\:CStringT.*
The colons must be backslashed as shown. CStringT is the template class used for CString, CStringA and CStringW. The CStringT class is shared with ATL, so that's why it's in the ATL namespace, even if you are using MFC. The "dot star" at the end matches anything for the rest of the line. - Here are a couple of other examples:
Key:CSimpleString
Value:ATL\:\:CSimpleStringT.*
Key:CComPtrBase
Value:ATL\:\:CComPtrBase.*
Key:vector
Value:std\:\:vector.*
Showing posts with label VS2008. Show all posts
Showing posts with label VS2008. Show all posts
Thursday, July 1, 2010
Debug "Just My Code" for C++ and MFC
One of my biggest annoyances with debugging MFC code is constantly stepping through CString functions and COM object functions. In Visual C++ 6 there was some functionality in autoexp.dat for handing this, but I never got it to work to my satifaction.
This week I was able to solve the problem. Here's how (note that the 10.0 is for Visual Studio 2010. The value will be 9.0 for VS2008. For earlier versions, see the link at the end of this article.)
Friday, May 28, 2010
Configuring Mozy Pro for Visual Studio
Earlier this week I revised my review of Mozy based on version 2.0 of the MozyPro client software. In short, MozyPro went from something I found acceptable to something that I am quite happy with.
One of my frustrations in earlier versions of MozyPro was figuring out how to exclude Visual Studio temporary files. In Visual Studio 2010, the IntelliSense data (the .sdf file) is 70MB to 300MB for each project. Since Visual Studio automatically rebuilds this file as necessary, it's a waste of time and money to back it up. There are numerous other temporary files, such as .ncb, .sbr, .bsc, and others, all of which are unnecessary to backup. This article tells how to set up MozyPro so that files with those extensions will not be backed up.
I found it best to create a global rule instead of modifying one of the predefined Backup Sets. Although MozyPro has a predefined Backup Set named "Visual Studio Projects", I've created other Backup Sets based on projects and customers. By creating a global rule, the temporary files will be ignored by all other projects.
To create a rule for Visual Studio, go to the Backup Sets page in the MozyPro configuration. Create a new backup set and call it something like Excluded Files. In the Backup Set Editor, put a checkmark next to C: so that this rule applies to the entire drive. Next check the box in the top right labeled "Files matching this set will be EXCLUDED." Under the Rules, the first rule should read:
This will exclude temporary and intermediate files for Visual Studio 2005 through 2010, as well as Virtual PC undo files. If you run Visual Studio 2010, also click the plus sign (+) on the right and create a second rule that reads:
Finally, if you use Virtual PC, create a third rule:
Note that the "Include" command in these rules really means "Exclude" because you checked the box "Files matching this set will be EXCLUDED..."
One of my frustrations in earlier versions of MozyPro was figuring out how to exclude Visual Studio temporary files. In Visual Studio 2010, the IntelliSense data (the .sdf file) is 70MB to 300MB for each project. Since Visual Studio automatically rebuilds this file as necessary, it's a waste of time and money to back it up. There are numerous other temporary files, such as .ncb, .sbr, .bsc, and others, all of which are unnecessary to backup. This article tells how to set up MozyPro so that files with those extensions will not be backed up.
I found it best to create a global rule instead of modifying one of the predefined Backup Sets. Although MozyPro has a predefined Backup Set named "Visual Studio Projects", I've created other Backup Sets based on projects and customers. By creating a global rule, the temporary files will be ignored by all other projects.
To create a rule for Visual Studio, go to the Backup Sets page in the MozyPro configuration. Create a new backup set and call it something like Excluded Files. In the Backup Set Editor, put a checkmark next to C: so that this rule applies to the entire drive. Next check the box in the top right labeled "Files matching this set will be EXCLUDED." Under the Rules, the first rule should read:
Include / File Type / sdf pch ncb idb tlog sbr res dep obj ilk ipch bscThis will exclude temporary and intermediate files for Visual Studio 2005 through 2010, as well as Virtual PC undo files. If you run Visual Studio 2010, also click the plus sign (+) on the right and create a second rule that reads:
Or / Include / Folder name / is / ipch / Files and FoldersFinally, if you use Virtual PC, create a third rule:
Or / Include / File Type / vud vsvNote that the "Include" command in these rules really means "Exclude" because you checked the box "Files matching this set will be EXCLUDED..."
Wednesday, May 19, 2010
Understanding ReadDirectoryChangesW - Part 2
The longest, most detailed description in the world of how to successfully use ReadDirectoryChangesW.
This is Part 2 of 2. Part 1 describes the theory and this part describes the implementation.
Go to the GitHub repo for this article or just download the sample code.
The first parameter, FILE_LIST_DIRECTORY, isn't even mentioned in the CreateFile() documentation. It's discussed in File Security and Access Rights, but not in any useful way.
Similarly, FILE_FLAG_BACKUP_SEMANTICS has this interesting note, "Appropriate security checks still apply when this flag is used without SE_BACKUP_NAME and SE_RESTORE_NAME privileges." In past dealings with this flag, it had been my impression that Administrator privileges were required, and the note seems to bear this out. However, attempting to enable these privileges on a Windows Vista system by adjusting the security token does not work if UAC is enabled. I'm not sure if the requirements have changed or if the documentation is simply ambiguous. Others are similarly confused.
The sharing mode also has pitfalls. I saw a few samples that left out FILE_SHARE_DELETE. You'd think that this would be fine since you do not expect the directory to be deleted. However, leaving out that permission prevents other processes from renaming or deleting files in that directory. Not a good result.
Another potential pitfall of this function is that the referenced directory itself is now “in use” and so can't be deleted. To monitor files in a directory and still allow the directory to be deleted, you would have to monitor the parent directory and its children.
The OVERLAPPED structure is supplied to indicate an overlapped operation, but none of the fields are actually used by ReadDirectoryChangesW. However, a little known secret of using Completion Routines is that you can supply your own pointer to the C++ object. How does this work? The documentation says that, "The hEvent member of the OVERLAPPED structure is not used by the system, so you can use it yourself." This means that you can put in a pointer to your object. You'll see this in my sample code below:
Since this call uses overlapped I/O, m_Buffer won't be filled in until the completion routine is called.
To terminate a thread that's waiting using SleepEx, you can write a Completion Routine that sets a flag in the SleepEx loop, causing it to exit. To call that Completion Routine, use QueueUserAPC, which allows one thread to call a completion routine in another thread.
First, you need to check for and handle the error code ERROR_OPERATION_ABORTED, which means that CancelIo has been called, this is the final notification, and you should clean up appropriately. I describe CancelIo in more detail in the next section. In my implementation, I used InterlockedDecrement to decrease cOutstandingCalls, which tracks my count of active calls, then I returned. My objects were all managed by the MFC mainframe and so did not need to be deleted by the Completion Routine itself.
You can receive multiple notifications in a single call. Make sure you walk the data structure and check for each non-zero NextEntryOffset field to skip forward.
ReadDirectoryChangesW is a "W" routine, so it does everything in Unicode. There's no ANSI version of this routine. Therefore, the data buffer is also Unicode. The string is not NULL-terminated, so you can't just use wcscpy. If you are using the ATL or MFC CString class, you can instantiate a wide CString from a raw string with a given number of characters like this:
Finally, you have to reissue the call to ReadDirectoryChangesW before you exit the completion routine.You can reuse the same OVERLAPPED structure. The documentation specifically says that the OVERLAPPED structure is not accessed again by Windows after the completion routine is called. However, you have to make sure that you use a different buffer than your current call or you will end up with a race condition.
One point that isn't clear to me is what happens to change notifications in between the time that your completion routine is called and the time you issue the new call to ReadDirectoryChangesW.
I'll also reiterate that you can still "lose" notifications if many files are changed in a short period of time. According to the documentation, if the buffer overflows, the entire contents of the buffer are discarded and the lpBytesReturned parameter contains zero. However, it's not clear to me if the completion routine will be called with dwNumberOfBytesTransfered equal to zero, and/or if there will be an error code specified in
dwNumberOfBytesTransfered.
There are some humorous examples of people trying (and failing) to write the completion routine correctly. My favorite is found on stackoverflow.com, where, after insulting the person asking for help, he presents his example of how to write the routine and concludes with, "It's not like this stuff is difficult." His code is missing error handling, he doesn't handle ERROR_OPERATION_ABORTED, he doesn't handle buffer overflow, and he doesn't reissue the call to ReadDirectoryChangesW. I guess it's not difficult when you just ignore all of the difficult stuff.
An article by Eric Gunnerson points out that the documentation for FILE_NOTIFY_INFORMATION contains a critical comment: If there is both a short and long name for the file, the function will return one of these names, but it is unspecified which one. Most of the time it's easy to convert back and forth between short and long filenames, but that's not possible if a file has been deleted. Therefore, if you are keeping a list of tracked files, you should probably track both the short and long filename. I was unable to reproduce this behavior on Windows Vista, but I only tried on one computer.
You will also receive some notifications that you may not expect. For example, even if you set the parameters of ReadDirectoryChangesW so you aren't notified about child directories, you will still get notifications about the child directories themselves. For example. Let's assume you have two directories, C:\A and C:\A\B. You move the file info.txt from the first directory to the second. You will receive FILE_ACTION_REMOVED for the file C:\A\info.txt and you will receive FILE_ACTION_MODIFIED for the directory C:\A\B. You will not receive any notifications about C:\A\B\info.txt.
There are some other surprises. Have you ever used hard links in NTFS? Hard links allow you to have multiple filenames that all reference the same physical file. If you have one reference in a monitored directory and a second reference in a second directory, you can edit the file in the second directory and a notification will be generated in the first directory. It's like magic.
On the other hand, if you are using symbolic links, which were introduced in Windows Vista, then no notification will be generated for the linked file. This makes sense when you think it through, but you have to be aware of these various possibilities.
There's yet a third possibility, which is junction points linking one partition to another. In that case, monitoring child directories won't monitor files in the linked partition. Again, this behavior makes sense, but it can be baffling when it's happening at a customer site and no notifications are being generated.
As I was searching various web pages with sample code that called CancelIo, I found this page that included the code below:
This looked promising. I faithfully copied it into my app. No effect.
I re-read the documentation for CancelIo, which makes the statement that "All I/O operations that are canceled complete with the error ERROR_OPERATION_ABORTED, and all completion notifications for the I/O operations occur normally." Decoded, this means that all Completion Routines will be called at least one final time after CancelIo is called. The call to SleepEx should have allowed that, but it wasn't happening. Eventually I determined that waiting for 5 milliseconds was simply too short. Maybe changing the "if" to a "while" would have solved the problem, but I chose to approach the problem differently since this solution requires polling every existing overlapped structure.
My final solution was to track the number of outstanding requests and to continue calling SleepEx until the count reached zero. In the sample code, the shutdown sequence works as follows:
ReadDirectoryChangesW fails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols.
Go to the GitHub repo for this article or just download the sample code.
This is Part 2 of 2. Part 1 describes the theory and this part describes the implementation.
Go to the GitHub repo for this article or just download the sample code.
Getting a Handle to the Directory
Now we'll look at the details of implementing the Balanced solution described in Part 1. When reading the declaration for ReadDirectoryChangesW, you'll notice that the first parameter is to a directory, and it's a HANDLE. Did you know that you can get a handle to a directory? There is no OpenDirectory function and the CreateDirectory function doesn't return a handle. Under the documentation for the first parameter, it says “This directory must be opened with the FILE_LIST_DIRECTORY access right.” Later, the Remarks section says, “To obtain a handle to a directory, use the CreateFile function with the FILE_FLAG_BACKUP_SEMANTICS flag.” The actual code looks like this:
HANDLE hDir = ::CreateFile( strDirectory, // pointer to the file name
FILE_LIST_DIRECTORY, // access (read/write) mode
FILE_SHARE_READ // share mode
| FILE_SHARE_WRITE
| FILE_SHARE_DELETE,
NULL, // security descriptor
OPEN_EXISTING, // how to create
FILE_FLAG_BACKUP_SEMANTICS // file attributes
| FILE_FLAG_OVERLAPPED,
NULL); // file with attributes to copyThe first parameter, FILE_LIST_DIRECTORY, isn't even mentioned in the CreateFile() documentation. It's discussed in File Security and Access Rights, but not in any useful way.
Similarly, FILE_FLAG_BACKUP_SEMANTICS has this interesting note, "Appropriate security checks still apply when this flag is used without SE_BACKUP_NAME and SE_RESTORE_NAME privileges." In past dealings with this flag, it had been my impression that Administrator privileges were required, and the note seems to bear this out. However, attempting to enable these privileges on a Windows Vista system by adjusting the security token does not work if UAC is enabled. I'm not sure if the requirements have changed or if the documentation is simply ambiguous. Others are similarly confused.
The sharing mode also has pitfalls. I saw a few samples that left out FILE_SHARE_DELETE. You'd think that this would be fine since you do not expect the directory to be deleted. However, leaving out that permission prevents other processes from renaming or deleting files in that directory. Not a good result.
Another potential pitfall of this function is that the referenced directory itself is now “in use” and so can't be deleted. To monitor files in a directory and still allow the directory to be deleted, you would have to monitor the parent directory and its children.
Calling ReadDirectoryChangesW
The actual call to ReadDirectoryChangesW is the simplest part of the whole operation. Assuming you are using completion routines, the only tricky part is that the buffer must be DWORD-aligned.The OVERLAPPED structure is supplied to indicate an overlapped operation, but none of the fields are actually used by ReadDirectoryChangesW. However, a little known secret of using Completion Routines is that you can supply your own pointer to the C++ object. How does this work? The documentation says that, "The hEvent member of the OVERLAPPED structure is not used by the system, so you can use it yourself." This means that you can put in a pointer to your object. You'll see this in my sample code below:
void CChangeHandler::BeginRead()
{
::ZeroMemory(&m_Overlapped, sizeof(m_Overlapped));
m_Overlapped.hEvent = this;
DWORD dwBytes=0;
BOOL success = ::ReadDirectoryChangesW(
m_hDirectory,
&m_Buffer[0],
m_Buffer.size(),
FALSE, // monitor children?
FILE_NOTIFY_CHANGE_LAST_WRITE
| FILE_NOTIFY_CHANGE_CREATION
| FILE_NOTIFY_CHANGE_FILE_NAME,
&dwBytes,
&m_Overlapped,
&NotificationCompletion);
}Since this call uses overlapped I/O, m_Buffer won't be filled in until the completion routine is called.
Dispatching Completion Routines
For the Balanced solution we've been discussing, there are only two ways to wait for Completion Routines to be called. If everything is being dispatched using Completion Routines, then SleepEx is all you need. If you need to wait on handles as well as to dispatch Completion Routines, then you want WaitForMultipleObjectsEx. The Ex version of the functions is required to put the thread in an “alertable” state, which means that completion routines will be called.To terminate a thread that's waiting using SleepEx, you can write a Completion Routine that sets a flag in the SleepEx loop, causing it to exit. To call that Completion Routine, use QueueUserAPC, which allows one thread to call a completion routine in another thread.
Handling the Notifications
The notification routine should be easy. Just read the data and save it, right? Wrong. Writing the Completion Routine also has its complexities.First, you need to check for and handle the error code ERROR_OPERATION_ABORTED, which means that CancelIo has been called, this is the final notification, and you should clean up appropriately. I describe CancelIo in more detail in the next section. In my implementation, I used InterlockedDecrement to decrease cOutstandingCalls, which tracks my count of active calls, then I returned. My objects were all managed by the MFC mainframe and so did not need to be deleted by the Completion Routine itself.
You can receive multiple notifications in a single call. Make sure you walk the data structure and check for each non-zero NextEntryOffset field to skip forward.
ReadDirectoryChangesW is a "W" routine, so it does everything in Unicode. There's no ANSI version of this routine. Therefore, the data buffer is also Unicode. The string is not NULL-terminated, so you can't just use wcscpy. If you are using the ATL or MFC CString class, you can instantiate a wide CString from a raw string with a given number of characters like this:
FILE_NOTIFY_INFORMATION* fni = (FILE_NOTIFY_INFORMATION*)buf;CStringW wstr(fni.Data, fni.Length / sizeof(wchar_t));Finally, you have to reissue the call to ReadDirectoryChangesW before you exit the completion routine.You can reuse the same OVERLAPPED structure. The documentation specifically says that the OVERLAPPED structure is not accessed again by Windows after the completion routine is called. However, you have to make sure that you use a different buffer than your current call or you will end up with a race condition.
One point that isn't clear to me is what happens to change notifications in between the time that your completion routine is called and the time you issue the new call to ReadDirectoryChangesW.
I'll also reiterate that you can still "lose" notifications if many files are changed in a short period of time. According to the documentation, if the buffer overflows, the entire contents of the buffer are discarded and the lpBytesReturned parameter contains zero. However, it's not clear to me if the completion routine will be called with dwNumberOfBytesTransfered equal to zero, and/or if there will be an error code specified in
dwNumberOfBytesTransfered.
There are some humorous examples of people trying (and failing) to write the completion routine correctly. My favorite is found on stackoverflow.com, where, after insulting the person asking for help, he presents his example of how to write the routine and concludes with, "It's not like this stuff is difficult." His code is missing error handling, he doesn't handle ERROR_OPERATION_ABORTED, he doesn't handle buffer overflow, and he doesn't reissue the call to ReadDirectoryChangesW. I guess it's not difficult when you just ignore all of the difficult stuff.
Using the Notifications
Once you receive and parse a notification, you need to figure out how to handle it. This isn't always easy. For one thing, you will often receive multiple duplicate notifications about changes, particularly when a long file is being written by its parent process. If you need the file to be complete, you should process each file after a timeout period has passed with no further updates. [Update: See the comment below by Wally The Walrus for details on the timeout.]An article by Eric Gunnerson points out that the documentation for FILE_NOTIFY_INFORMATION contains a critical comment: If there is both a short and long name for the file, the function will return one of these names, but it is unspecified which one. Most of the time it's easy to convert back and forth between short and long filenames, but that's not possible if a file has been deleted. Therefore, if you are keeping a list of tracked files, you should probably track both the short and long filename. I was unable to reproduce this behavior on Windows Vista, but I only tried on one computer.
You will also receive some notifications that you may not expect. For example, even if you set the parameters of ReadDirectoryChangesW so you aren't notified about child directories, you will still get notifications about the child directories themselves. For example. Let's assume you have two directories, C:\A and C:\A\B. You move the file info.txt from the first directory to the second. You will receive FILE_ACTION_REMOVED for the file C:\A\info.txt and you will receive FILE_ACTION_MODIFIED for the directory C:\A\B. You will not receive any notifications about C:\A\B\info.txt.
There are some other surprises. Have you ever used hard links in NTFS? Hard links allow you to have multiple filenames that all reference the same physical file. If you have one reference in a monitored directory and a second reference in a second directory, you can edit the file in the second directory and a notification will be generated in the first directory. It's like magic.
On the other hand, if you are using symbolic links, which were introduced in Windows Vista, then no notification will be generated for the linked file. This makes sense when you think it through, but you have to be aware of these various possibilities.
There's yet a third possibility, which is junction points linking one partition to another. In that case, monitoring child directories won't monitor files in the linked partition. Again, this behavior makes sense, but it can be baffling when it's happening at a customer site and no notifications are being generated.
Shutting Down
I didn't find any articles or code (even in open source production code) that properly cleaned up the overlapped call. The documentation on MSDN for canceling overlapped I/O says to call CancelIo. That's easy. However, my application then crashed when exiting. The call stack showed that one of my third party libraries was putting the thread in an alertable state (which meant that Completion Routines could be called) and that my Completion Routine was being called even after I had called CancelIo, closed the handle, and deleted the OVERLAPPED structure.As I was searching various web pages with sample code that called CancelIo, I found this page that included the code below:
CancelIo(pMonitor->hDir);
if (!HasOverlappedIoCompleted(&pMonitor->ol))
{
SleepEx(5, TRUE);
}
CloseHandle(pMonitor->ol.hEvent);
CloseHandle(pMonitor->hDir);This looked promising. I faithfully copied it into my app. No effect.
I re-read the documentation for CancelIo, which makes the statement that "All I/O operations that are canceled complete with the error ERROR_OPERATION_ABORTED, and all completion notifications for the I/O operations occur normally." Decoded, this means that all Completion Routines will be called at least one final time after CancelIo is called. The call to SleepEx should have allowed that, but it wasn't happening. Eventually I determined that waiting for 5 milliseconds was simply too short. Maybe changing the "if" to a "while" would have solved the problem, but I chose to approach the problem differently since this solution requires polling every existing overlapped structure.
My final solution was to track the number of outstanding requests and to continue calling SleepEx until the count reached zero. In the sample code, the shutdown sequence works as follows:
- The application calls CReadDirectoryChanges::Terminate (or simply allows the object to destruct.)
- Terminate uses QueueUserAPC to send a message to CReadChangesServer in the worker thread, telling it to terminate.
- CReadChangesServer::RequestTermination sets m_bTerminate to true and delegates the call to the CReadChangesRequest objects, each of which calls CancelIo on its directory handle and closes the directory handle.
- Control is returned to CReadChangesServer::Run function. Note that nothing has actually terminated yet.
void Run()
{
while (m_nOutstandingRequests || !m_bTerminate)
{
DWORD rc = ::SleepEx(INFINITE, true);
}
}- CancelIo causes Windows to automatically call the Completion Routine for each CReadChangesRequest overlapped request. For each call, dwErrorCode is set to ERROR_OPERATION_ABORTED.
- The Completion Routine deletes the CReadChangesRequest object, decrements nOutstandingRequests, and returns without queuing a new request.
- SleepEx returns due to one or more APCs completing. nOutstandingRequests is now zero and m_bTerminate is true, so the function exits and the thread terminates cleanly.
Network Drives
ReadDirectoryChangesW works with network drives, but only if the remote server supports the functionality. Drives shared from other Windows-based computers will correctly generate notifications. Samba servers may or may not generate notifications, depending on whether the underlying operating system supports the functionality. Network Attached Storage (NAS) devices usually run Linux, so won't support notifications. High-end SANs are anybody's guess.ReadDirectoryChangesW fails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols.
Summary
If you've made it this far in the article, I applaud your can-do attitude. I hope I've given you a clear picture of the challenges of using ReadDirectoryChangesW and why you should be dubious of any sample code you see for using the function. Careful testing is critical, including performance testing.Go to the GitHub repo for this article or just download the sample code.
Understanding ReadDirectoryChangesW - Part 1
The longest, most detailed description in the world of how to successfully use ReadDirectoryChangesW.
This is Part 1 of 2. This part describes the theory and Part 2 describes the implementation.
Go to the GitHub repo for this article or just download the sample code.
I have spent this week digging into the barely-documented world of ReadDirectoryChangesW and I hope this article saves someone else some time. I believe I've read every article I could find on the subject, as well as numerous code samples. Almost all of the examples, including the one from Microsoft, either have significant shortcoming or have outright mistakes.
You'd think that this problem would have been a piece of cake for me, having been the author of Multithreading Applications in Win32, where I wrote a chapter about the differences between synchronous I/O, signaled handles, overlapped I/O, and I/O completion ports. Except that I only write overlapped I/O code about once every five years, which is just about long enough for me to forget how painful it was the last time. This endeavor was no exception.
SHChangeNotifyRegister was fixed in Windows Vista so it could report all changes to all files, but is was too late - there are still several hundred million Windows XP users and that's not going to change any time soon.
SHChangeNotifyRegister also had a performance problem, since it was based on Windows messages. If there were too many changes, your application would start receiving roll-up messages that just said "something changed" and you had to figure out for yourself what had really happened. Fine for some applications, rather painful for others.
Windows 2000 brought two new interfaces, FindFirstChangeNotification and ReadDirectoryChangesW. FindFirstChangeNotification is fairly easy to use but doesn't give any information about what changed. Even so, it can be useful for applications such as fax servers and SMTP servers that can accept queue submissions by dropping a file in a directory. ReadDirectoryChangesW does tell you what changed and how, at the cost of additional complexity.
Similar to SHChangeNotifyRegister, both of these new functions suffer from a performance problem. They can run significantly faster than shell notifications, but moving a thousand files from one directory to another will still cause you to lose some (or many) notifications. The exact cause of the missing notifications is complicated. Surprisingly, it apparently has little to do with how fast you process notifications.
Note that FindFirstChangeNotification and ReadDirectoryChangesW are mutually exclusive. You would use one or the other, but not both.
Windows XP brought the ultimate solution, the Change Journal, which could track in detail every single change, even if your software wasn't running. Great technology, but equally complicated to use.
The fourth and final solution is is to install a File System Filter, which was used in the popular SysInternals FileMon tool. There is a sample of this in the Windows Driver Kit (WDK). However, this solution is essentially a device driver and so potentially can cause system-wide stability problems if not implemented exactly correctly.
For my needs, ReadDirectoryChangesW was a good balance of performance versus complexity.
A. First, here are the I/O modes:
If your head is now swimming in information overload, you can easily see why so many people have trouble getting this right.
Simplicity - A2C3D1 - Each call to ReadDirectoryChangesW runs in its own thread and sends the results to the primary thread with PostMessage. Most appropriate for GUI apps with minimal performance requirements. This is the strategy used in CDirectoryChangeWatcher on CodeProject. This is also the strategy used by Microsoft's FWATCH sample.
Performance - A4C6D4 - The highest performance solution is to use I/O completion ports, but, as an aggressively multithreaded solution, it's also a very complex solution that should be confined to servers. It's unlikely to be necessary in any GUI application. If you aren't a multithreading expert, stay away from this strategy.
Balanced - A4C5D3 - Do everything in one thread with Completion Routines. You can have as many outstanding calls to ReadDirectoryChangesW as you need. There are no handles to wait on, since Completion Routines are dispatched automatically. You embed the pointer to your object in the callback, so it's easy to keep callbacks matched up to their original data structure.
Originally I had thought that GUI applications could use MsgWaitForMultipleObjectsEx to intermingle change notifications with Windows messages. This turns out not to work because dialog boxes have their own message loop that's not alertable, so a dialog box being displayed would prevent notifications from being processed. Another good idea steamrolled by reality.
If you are using the Simplicity solution above, don't use blocking calls because the only way to cancel it is with the undocumented technique of closing the handle or the Vista-only technique of CancelSynchronousIo. Instead, use the Signal Synchronous I/O mode by waiting on the directory handle. Also, to terminate threads, don't use TerminateThread, because that doesn't clean up resources and can cause all sorts of problems. Instead, create a manual-reset event object that is used as the the second handle in the call to WaitForMultipleObjects.When the event is set, exit the thread.
If you have dozens or hundreds of directories to monitor, don't use the Simplicity solution. Switch to the Balanced solution. Alternatively, monitor a root common directory and ignore files you don't care about.
If you have to monitor a whole drive, think twice (or three times) about this idea. You'll be notified about every single temporary file, every Internet cache file, every Application Data change - in short, you'll be getting an enormous number of notifications that could slow down the entire system. If you need to monitor an entire drive, you should probably use the Change Journal instead. This will also allow you to track changes even if your app is not running. Don't even think about monitoring the whole drive with FILE_NOTIFY_CHANGE_LAST_ACCESS.
If you are using overlapped I/O without using an I/O completion port, don't wait on handles. Use Completion Routines instead. This removes the 64 handle limitation, allows the operating system to handle call dispatch, and allows you to embed a pointer to your object in the OVERLAPPED structure. My example in a moment will show all of this.
If you are using worker threads, don't send results back to the primary thread with SendMessage. Use PostMessage instead. SendMessage is synchronous and will not return if the primary thread is busy. This would defeat the purpose of using a worker thread in the first place.
It's tempting to try and solve the issue of lost notifications by providing a huge buffer. However, this may not be the wisest course of action. For any given buffer size, a similarly-sized buffer has to be allocated from the kernel non-paged memory pool. If you allocate too many large buffers, this can lead to serious problems, including a Blue Screen of Death. Thanks to an anonymous contributor in the MSDN Community Content.
Jump to Part 2 of this article.
Go to the GitHub repo for this article or just download the sample code.
This is Part 1 of 2. This part describes the theory and Part 2 describes the implementation.
Go to the GitHub repo for this article or just download the sample code.
I have spent this week digging into the barely-documented world of ReadDirectoryChangesW and I hope this article saves someone else some time. I believe I've read every article I could find on the subject, as well as numerous code samples. Almost all of the examples, including the one from Microsoft, either have significant shortcoming or have outright mistakes.
You'd think that this problem would have been a piece of cake for me, having been the author of Multithreading Applications in Win32, where I wrote a chapter about the differences between synchronous I/O, signaled handles, overlapped I/O, and I/O completion ports. Except that I only write overlapped I/O code about once every five years, which is just about long enough for me to forget how painful it was the last time. This endeavor was no exception.
Four Ways to Monitor Files and Directories
First, a brief overview of monitoring directories and files. In the beginning there was SHChangeNotifyRegister. It was implemented using Windows messages and so required a window handle. It was driven by notifications from the shell (Explorer), so your application was only notified about things that the shell cared about - which almost never aligned with what you cared about. It was useful for monitoring things that the user did in Explorer, but not much else.SHChangeNotifyRegister was fixed in Windows Vista so it could report all changes to all files, but is was too late - there are still several hundred million Windows XP users and that's not going to change any time soon.
SHChangeNotifyRegister also had a performance problem, since it was based on Windows messages. If there were too many changes, your application would start receiving roll-up messages that just said "something changed" and you had to figure out for yourself what had really happened. Fine for some applications, rather painful for others.
Windows 2000 brought two new interfaces, FindFirstChangeNotification and ReadDirectoryChangesW. FindFirstChangeNotification is fairly easy to use but doesn't give any information about what changed. Even so, it can be useful for applications such as fax servers and SMTP servers that can accept queue submissions by dropping a file in a directory. ReadDirectoryChangesW does tell you what changed and how, at the cost of additional complexity.
Similar to SHChangeNotifyRegister, both of these new functions suffer from a performance problem. They can run significantly faster than shell notifications, but moving a thousand files from one directory to another will still cause you to lose some (or many) notifications. The exact cause of the missing notifications is complicated. Surprisingly, it apparently has little to do with how fast you process notifications.
Note that FindFirstChangeNotification and ReadDirectoryChangesW are mutually exclusive. You would use one or the other, but not both.
Windows XP brought the ultimate solution, the Change Journal, which could track in detail every single change, even if your software wasn't running. Great technology, but equally complicated to use.
The fourth and final solution is is to install a File System Filter, which was used in the popular SysInternals FileMon tool. There is a sample of this in the Windows Driver Kit (WDK). However, this solution is essentially a device driver and so potentially can cause system-wide stability problems if not implemented exactly correctly.
For my needs, ReadDirectoryChangesW was a good balance of performance versus complexity.
The Puzzle
The biggest challenge to using ReadDirectoryChangesW is that there are several hundred possibilities for combinations of I/O mode, handle signaling, waiting methods, and threading models. Unless you're an expert on Win32 I/O, it's extremely unlikely that you'll get it right, even in the simplest of scenarios. (In the list below, when I say "call", I mean a call to ReadDirectoryChangesW.)A. First, here are the I/O modes:
- Blocking synchronous
- Signaled synchronous
- Overlapped asynchronous
- Completion Routine (aka Asynchronous Procedure Call or APC)
- Wait on the directory handle.
- Wait on an event object in the OVERLAPPED structure.
- Wait on nothing (for APCs.)
- Blocking
- WaitForSingleObject
- WaitForMultipleObjects
- WaitForMultipleObjectsEx
- MsgWaitForMultipleObjectsEx
- I/O Completion Ports
- One call per worker thread.
- Multiple calls per worker thread.
- Multiple calls on the primary thread.
- Multiple threads for multiple calls. (I/O Completion Ports)
If your head is now swimming in information overload, you can easily see why so many people have trouble getting this right.
Recommended Solutions
So what's the right answer? Here's my opinion, depending on what's most important:Simplicity - A2C3D1 - Each call to ReadDirectoryChangesW runs in its own thread and sends the results to the primary thread with PostMessage. Most appropriate for GUI apps with minimal performance requirements. This is the strategy used in CDirectoryChangeWatcher on CodeProject. This is also the strategy used by Microsoft's FWATCH sample.
Performance - A4C6D4 - The highest performance solution is to use I/O completion ports, but, as an aggressively multithreaded solution, it's also a very complex solution that should be confined to servers. It's unlikely to be necessary in any GUI application. If you aren't a multithreading expert, stay away from this strategy.
Balanced - A4C5D3 - Do everything in one thread with Completion Routines. You can have as many outstanding calls to ReadDirectoryChangesW as you need. There are no handles to wait on, since Completion Routines are dispatched automatically. You embed the pointer to your object in the callback, so it's easy to keep callbacks matched up to their original data structure.
Originally I had thought that GUI applications could use MsgWaitForMultipleObjectsEx to intermingle change notifications with Windows messages. This turns out not to work because dialog boxes have their own message loop that's not alertable, so a dialog box being displayed would prevent notifications from being processed. Another good idea steamrolled by reality.
Wrong Techniques
As I was researching this solution, I saw a lot of recommendations that ranged from dubious to wrong to really, really wrong. Here's some commentary on what I saw.If you are using the Simplicity solution above, don't use blocking calls because the only way to cancel it is with the undocumented technique of closing the handle or the Vista-only technique of CancelSynchronousIo. Instead, use the Signal Synchronous I/O mode by waiting on the directory handle. Also, to terminate threads, don't use TerminateThread, because that doesn't clean up resources and can cause all sorts of problems. Instead, create a manual-reset event object that is used as the the second handle in the call to WaitForMultipleObjects.When the event is set, exit the thread.
If you have dozens or hundreds of directories to monitor, don't use the Simplicity solution. Switch to the Balanced solution. Alternatively, monitor a root common directory and ignore files you don't care about.
If you have to monitor a whole drive, think twice (or three times) about this idea. You'll be notified about every single temporary file, every Internet cache file, every Application Data change - in short, you'll be getting an enormous number of notifications that could slow down the entire system. If you need to monitor an entire drive, you should probably use the Change Journal instead. This will also allow you to track changes even if your app is not running. Don't even think about monitoring the whole drive with FILE_NOTIFY_CHANGE_LAST_ACCESS.
If you are using overlapped I/O without using an I/O completion port, don't wait on handles. Use Completion Routines instead. This removes the 64 handle limitation, allows the operating system to handle call dispatch, and allows you to embed a pointer to your object in the OVERLAPPED structure. My example in a moment will show all of this.
If you are using worker threads, don't send results back to the primary thread with SendMessage. Use PostMessage instead. SendMessage is synchronous and will not return if the primary thread is busy. This would defeat the purpose of using a worker thread in the first place.
It's tempting to try and solve the issue of lost notifications by providing a huge buffer. However, this may not be the wisest course of action. For any given buffer size, a similarly-sized buffer has to be allocated from the kernel non-paged memory pool. If you allocate too many large buffers, this can lead to serious problems, including a Blue Screen of Death. Thanks to an anonymous contributor in the MSDN Community Content.
Jump to Part 2 of this article.
Go to the GitHub repo for this article or just download the sample code.
Monday, May 17, 2010
Using MsgWaitForMultipleObjects in MFC
One of the problems that I didn't solve in my Multithreading book was how to use MsgWaitForMultipleObjectsEx with MFC. I have always felt guilty about this because it was an important problem, but I simply ran out of time to do the implemenation. Recently I finally had a need to solve the problem. MFC has come a long way since 1996 and it's clear that this problem was planned for. With MFC in Visual Studio 2008 and 2010, I was able to solve the problem in minutes instead of days.
In short, the solution is to replace MFC's call to GetMessage with your own call to MsgWaitForMultipleObjectsEx. This will allow you to dispatch messages, handle signaled objects, and dispatch Completion Routines. Here's the code, which goes in your MFC App object:
There are several obscure points in this code worth mentioning. Getting any of these wrong will cause it to break:
In short, the solution is to replace MFC's call to GetMessage with your own call to MsgWaitForMultipleObjectsEx. This will allow you to dispatch messages, handle signaled objects, and dispatch Completion Routines. Here's the code, which goes in your MFC App object:
// virtual
BOOL CMyApp::PumpMessage()
{ HANDLE hEvent = ...;
HANDLE handles[] = { hEvent };
DWORD const res =
::MsgWaitForMultipleObjectsEx(
_countof(handles),
handles,
INFINITE,
QS_ALLINPUT,
MWMO_INPUTAVAILABLE | MWMO_ALERTABLE);
switch (res)
{
case WAIT_OBJECT_0 + 0:
// the event object was signaled...
return true;
case WAIT_OBJECT_0 + _countof(handles):
return __super::PumpMessage();
case WAIT_IO_COMPLETION:
break;
}
return TRUE;
}There are several obscure points in this code worth mentioning. Getting any of these wrong will cause it to break:
- The MWMO_INPUTAVAILABLE is required to solve race conditions with the message queue.
- The MWMO_ALERTABLE is required in order for Completion Routines to be called.
- WAIT_IO_COMPLETION does not require any action on your part. It just indicates that a completion routine was called.
- The handle array is empty. You do NOT wait on the directory handle. Doing so will prevent your completion routine from being called.
- MsgWaitForMultipleObjectsEx indicates that a message is available (as opposed to a signaled object) by returning WAIT_OBJECT_0 plus the handle count.
- The call to __super::PumpMessage() is for MFC when this is running in your application object. Outside of MFC, you should replace it with your own message loop.
Tuesday, March 9, 2010
Create separate references for Debug and Release DLLs in C++/CLR
I'm one of the crazy people who has worked with C++/CLR to make native C++ code coexist with .Net code. It runs really well, with a minimum of frustration.
Except for one thing. We rely on some components supplied by a third party. These components have both Debug and Release versions. You'll notice that the References are listed under "Common Properties" in the Properties window, so you can't create separate references for Debug and Release by simply switching to the appropriate property set.
This has been annoying me now for at least two years, and I finally found a solution, thanks Marco Beninca in this post:
http://social.msdn.microsoft.com/Forums/en-US/clr/thread/9087fb4f-149f-4d66-b33e-f2f280c65fa6
The solution is to use #using in your source code instead of defining your assembly references on the "Framework and References" page. Then you can go to the General page for C/C++ and set different directories for Debug and for Release.
The other advantage to this solution is that you don't have to reset all of your references when you get a new version of the components.
One disadvantage to this strategy is that the compiler no longer takes care of copying the appropriate DLLs to the appropriate directories. This can easily cause you to build with a different set of DLLs than you are running against, which will certainly cause a crash. My solution was to create a Pre-Link step that copies the DLLs to $(OutDir).
Except for one thing. We rely on some components supplied by a third party. These components have both Debug and Release versions. You'll notice that the References are listed under "Common Properties" in the Properties window, so you can't create separate references for Debug and Release by simply switching to the appropriate property set.
This has been annoying me now for at least two years, and I finally found a solution, thanks Marco Beninca in this post:
http://social.msdn.microsoft.com/Forums/en-US/clr/thread/9087fb4f-149f-4d66-b33e-f2f280c65fa6
The solution is to use #using in your source code instead of defining your assembly references on the "Framework and References" page. Then you can go to the General page for C/C++ and set different directories for Debug and for Release.
The other advantage to this solution is that you don't have to reset all of your references when you get a new version of the components.
One disadvantage to this strategy is that the compiler no longer takes care of copying the appropriate DLLs to the appropriate directories. This can easily cause you to build with a different set of DLLs than you are running against, which will certainly cause a crash. My solution was to create a Pre-Link step that copies the DLLs to $(OutDir).
Monday, October 12, 2009
NUnit Unit Testing with C++
I switch back and forth between C++ and C#. When doing C# development, NUnit rules the day for unit testing. Whether I'm doing automated tests from the command line or using the GUI to run selective tests (show below in a screenshot from SourceForge) NUnit is a pleasure to use.

If you've ever tried to run unit tests for C++, the landscape is much less appealing. C++ does not have a reflection API, nor does it have attributes that are embedded in the executable code, so it's much more tedious in C++ to do all of the housekeeping to initialize the framework and it's much more difficult to integrate external GUI tools. In short, unit test in C++ is a sub-par experience compared to more modern languages.
But I have good news for you - it's possible to use NUnit with C++. There are two minor caveats:
If you haven't built this kind of project before, a C++/CLI project is a curious hybrid that contains all of the power of C++ as well as much of the power of .Net. (Access to certain features, like LINQ, is not available in C++/CLI.)
With a C++/CLI project, you can use NUnit attributes for classes and member functions, just like in C#. When you run NUnit, your target is your test DLL, which implicitly loads either your static library code or your DLL code.
This strategy can also be used in the test environment build into Visual Studio if you have the Professional edition or better.
If you've ever tried to run unit tests for C++, the landscape is much less appealing. C++ does not have a reflection API, nor does it have attributes that are embedded in the executable code, so it's much more tedious in C++ to do all of the housekeeping to initialize the framework and it's much more difficult to integrate external GUI tools. In short, unit test in C++ is a sub-par experience compared to more modern languages.
But I have good news for you - it's possible to use NUnit with C++. There are two minor caveats:
- You must be using Microsoft Visual Studio 2005 or later (sorry MinGW and cygwin users.)
- The code to be tested must either be a DLL or a static library. If you need to test code in an .EXE, you should factor that code out into its own static library.
If you haven't built this kind of project before, a C++/CLI project is a curious hybrid that contains all of the power of C++ as well as much of the power of .Net. (Access to certain features, like LINQ, is not available in C++/CLI.)
With a C++/CLI project, you can use NUnit attributes for classes and member functions, just like in C#. When you run NUnit, your target is your test DLL, which implicitly loads either your static library code or your DLL code.
This strategy can also be used in the test environment build into Visual Studio if you have the Professional edition or better.
Labels:
C++,
Debug,
Release Engineering,
Tools,
VS2008
Thursday, August 27, 2009
Setting the default Windows SDK
Every time I install a new Windows SDK (or worse, a new copy of Visual Studio), I've gone through the painful process of updating all of the project directories for the Windows SDK include directory, lib directory, etc.
Visual Studio 2005 and 2008 are both smart enough to look in the version of the Windows SDK included with those compilers, but I'd never found a way to change the default version. Until now.
The Windows SDK comes with a utility called the Windows SDK Configuration Tool. You can find it in your Start menu.
When you run this, you can set the default SDK to be whichever version you want. Then Visual Studio will automatically reference that version of the SDK without any need to manually update project directories.
[Update 8/31/2011]
Visual Studio 2010 does not seem to pay attention to the Configuration Tool. Instead, this appears to be set on a project-by-project basis in Configuration Properties/Platform Toolset. After doing so, you may be able to fix some schizophrenic behavior by updating the MSBuild information too. Take a look at the following registry entries. (Thanks to the tip from http://geekswithblogs.net/rob/archive/2010/09/17/integrate-the-windows-sdk-v7.1-with-vs2010.aspx)
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\ToolsVersions\4.0
Visual Studio 2005 and 2008 are both smart enough to look in the version of the Windows SDK included with those compilers, but I'd never found a way to change the default version. Until now.
The Windows SDK comes with a utility called the Windows SDK Configuration Tool. You can find it in your Start menu.
When you run this, you can set the default SDK to be whichever version you want. Then Visual Studio will automatically reference that version of the SDK without any need to manually update project directories.
[Update 8/31/2011]
Visual Studio 2010 does not seem to pay attention to the Configuration Tool. Instead, this appears to be set on a project-by-project basis in Configuration Properties/Platform Toolset. After doing so, you may be able to fix some schizophrenic behavior by updating the MSBuild information too. Take a look at the following registry entries. (Thanks to the tip from http://geekswithblogs.net/rob/archive/2010/09/17/integrate-the-windows-sdk-v7.1-with-vs2010.aspx)
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\ToolsVersions\4.0
- FrameworkSDKRoot (REG_SZ)
- $(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A@InstallationFolder)
- SDK35ToolsPath (REG_SZ)
- $(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx35Tools-x86@InstallationFolder)
- SDK40ToolsPath (REG_SZ)
- $(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx40Tools-x86@InstallationFolder)
Monday, January 12, 2009
Unicode BOM Handling in the C Run-time Library
Visual Studio 2005 and later include Unicode BOM (Byte-Order Mark) support, but I found the documentation somewhat lacking. Here are a few hints.
One of the primary points of confusion for me was what you were defining by setting the encoding. The answer is that you are providing a hint as to the encoding of the file (but only a hint. If the file has a BOM, then that BOM takes precedence.)
All calls you make to read or write the file must be with Unicode APIs. If you try to use an ANSI API, the C-Runtime library (CRT) will assert. This means that the CRT will do character set conversion between Unicode and the file's encoding, but won't do character set conversion between the local code page and the file's encoding. For example, you'll get an assertion if you open the file with ccs=utf-8 and then try to use fgets to read the data.
Other points:
One of the primary points of confusion for me was what you were defining by setting the encoding. The answer is that you are providing a hint as to the encoding of the file (but only a hint. If the file has a BOM, then that BOM takes precedence.)
All calls you make to read or write the file must be with Unicode APIs. If you try to use an ANSI API, the C-Runtime library (CRT) will assert. This means that the CRT will do character set conversion between Unicode and the file's encoding, but won't do character set conversion between the local code page and the file's encoding. For example, you'll get an assertion if you open the file with ccs=utf-8 and then try to use fgets to read the data.
Other points:
- The CRT will not perform any BOM handling if you do not specify a ccs= encoding. This means that backwards compatibility is retained because the CRT does not perform any behind-the-scenes processing if you don't ask it to.
- Most BOM formats are not supported. For example, UTF-7, UTF-32 and especially UCS-16 big-endian are not handled.
- If you specify a specify a ccs= encoding, then the BOM will be automatically removed from the data stream. However, you need to be careful of file positioning calls such as fseek and rewind because the bom will only be skipped when the file is first opened. For example, if you do fopen, fread, rewind, fread, then the second fread will read the BOM and the first fread will not.
- The file encoding is respected when writing, so the number of characters actually written may be lesser or greater than the buffer size you wrote.
- If you open the file in binary mode, then any ccs= specification will be ignored and no BOM handling will be performed.
- Apparently the CRT does not provide a documented way to determine the encoding of the file.
Sunday, September 28, 2008
Visual Studio 2008 C/C++ applications under Win9x
On of the disadvantages of Visual Studio 2008 is that it does not create applications that support Windows 95/98/98SE/ME nor Windows NT. Although Windows XP has been shipping for seven years, there are still people out there who have never seen any reason to upgrade and are still running the old operating systems. (As of August 2008, www.w3counter.com shows that the combined market share of the Windows 9x variants is less than 0.9%.)
Windows 95 support was dropped in Visual Studio 2005, but working around the problem was easy.
Visual Studio 2008 is a thornier problem. VS2008 dropped support for all variants of Win9x and the C run-time depends on several functions that require Windows 2000 or later. Fortunately, a third party company has created a product named Legacy Extender that provides a drop-in solution. The caveat is that it only works with the statically linked C runtime libraries, not with the DLLs.
For a modest $29.95/developer, including unlimited upgrades, Legacy Extender is a great solution to a difficult problem.
Windows 95 support was dropped in Visual Studio 2005, but working around the problem was easy.
Visual Studio 2008 is a thornier problem. VS2008 dropped support for all variants of Win9x and the C run-time depends on several functions that require Windows 2000 or later. Fortunately, a third party company has created a product named Legacy Extender that provides a drop-in solution. The caveat is that it only works with the statically linked C runtime libraries, not with the DLLs.
For a modest $29.95/developer, including unlimited upgrades, Legacy Extender is a great solution to a difficult problem.
Monday, August 4, 2008
Visual Studio 2008 Feature Pack: Manifest Problems
If you install the Visual Studio 2008 Feature Pack, be aware that the manifest handling in the release is buggy, to put it politely. It's very easy to end up with the manifest requiring both the RTM DLLs (9.00.21022.8) and the Feature Pack DLLs (9.00.30411.0). This can cause weird, untraceable errors at runtime. I blogged about this problem before, but in that case it was caused by changing from a Beta build to the Release build.
There's a helpful blog post about the problem in the Visual C++ Team Blog.
The important point is that there are two #defines that force the manifest to point to the newer DLLs:
There's a single #define that does the same thing, but it's broken in the Feature Pack. [9/7/2008 - It's fixed in Service Pack 1. Thanks "anonymous"!]
The blog points out that defining the explicit BIND macros is not always the right thing to do.
This problem bit me because I was following the directions in this blog post, which call for making copies of certain directories from Debug_NonRedist. In a system where the CRT and MFC DLLs have been installed, there are pointers in the registry from the old versions of the DLLs to the new versions, so everything "just works." However, when using the Debug_nonRedist directories, these redirect pointers don't exist. Since the manifest was calling for the old versions of the DLLs, the application wouldn't open.
If you get error c101008a, either Rebuild All or delete all of your "xxx.exe.embed.manifest" files and build the solution.
There's a helpful blog post about the problem in the Visual C++ Team Blog.
The important point is that there are two #defines that force the manifest to point to the newer DLLs:
#define _BIND_TO_CURRENT_CRT_VERSION 1
#define _BIND_TO_CURRENT_MFC_VERSION 1There's a single #define that does the same thing, but it's broken in the Feature Pack. [9/7/2008 - It's fixed in Service Pack 1. Thanks "anonymous"!]
#define _BIND_TO_CURRENT_VCLIBS_VERSION 1The blog points out that defining the explicit BIND macros is not always the right thing to do.
This problem bit me because I was following the directions in this blog post, which call for making copies of certain directories from Debug_NonRedist. In a system where the CRT and MFC DLLs have been installed, there are pointers in the registry from the old versions of the DLLs to the new versions, so everything "just works." However, when using the Debug_nonRedist directories, these redirect pointers don't exist. Since the manifest was calling for the old versions of the DLLs, the application wouldn't open.
If you get error c101008a, either Rebuild All or delete all of your "xxx.exe.embed.manifest" files and build the solution.
Thursday, July 24, 2008
Visual Studio 2008 C++ Redistributable Components
I've wasted two hours hunting down files that should be obvious, so I'm hoping this post helps someone else.
If you need to redistribute components from Visual Studio 2008, such as the C runtime or the MFC DLLs, there are four ways to do it:
1. Put the DLLs in your installer and install them into Windows\System32 directory yourself.
DON'T EVEN THINK OF DOING THIS. It's almost impossible to do it right because of WinSxS. If you want complete control over your installation, use option #2.
2. Use the redistributable directories Microsoft provides.
These directories provide copies of the DLLs that are only available to your application.The directories should go in the same install directory as your EXE. This means that they are subdirectories of wherever your EXE is installed. [Updated 11/19/2008] I no longer recommend copying the directory itself. Instead, copy the contents of the redistributable directory into the same directory as your app. Make sure you include the manifest file. I made this change because msvcm90.dll will not bind to msvcr90.dll if they are a subdirectory. As a bonus, this change makes this option work with Windows 2000.
Assuming you've installed Visual Studio 2008, you can find the directories at:
Important: If you do this with Visual Studio 2008 SP1, make sure you put the following in your precompiled header:
Advantage: Doesn't require admin privileges. Works with XCOPY. Your app won't break if the system global version is updated by Microsoft (but you won't benefit from security fixes either.)
Disadvantage: Not viable if your EXEs and DLLs are installed across multiple directories.
3. Microsoft Visual C++ 2008 Redistributable Package:
Original distribution.
SP1 distribution. See caveats.
[Update 6/22/2009] Both of these distributions cause the the Windows 7 Logo Toolkit Beta to generate FAIL errors.
Advantage: All you have to do is run it. Permamently installs the components in the proper locations.
Disadvantage: Includes everything, so it's larger than the individual merge modules. If you are building your own installer, not as clean of a user experience as the merge modules.
4. Use the Merge Modules.
Assuming you've installed Visual Studio 2008, you can find the files at:
Advantage: Best user experience. Smallest download.
Disadvantage: Installation isn't permanent - the DLLs may be uninstalled when your application is uninstalled. Requires you to use Windows Installer. (This is only a disadvantage for a minority of developers. Windows Installer is a logo requirement for Vista.)
If you need to redistribute components from Visual Studio 2008, such as the C runtime or the MFC DLLs, there are four ways to do it:
1. Put the DLLs in your installer and install them into Windows\System32 directory yourself.
DON'T EVEN THINK OF DOING THIS. It's almost impossible to do it right because of WinSxS. If you want complete control over your installation, use option #2.
2. Use the redistributable directories Microsoft provides.
These directories provide copies of the DLLs that are only available to your application.
Assuming you've installed Visual Studio 2008, you can find the directories at:
C:\Program Files\Microsoft Visual Studio 9.0\VC\redist\x86Important: If you do this with Visual Studio 2008 SP1, make sure you put the following in your precompiled header:
#define _BIND_TO_CURRENT_VCLIBS_VERSION 1Advantage: Doesn't require admin privileges. Works with XCOPY. Your app won't break if the system global version is updated by Microsoft (but you won't benefit from security fixes either.)
Disadvantage: Not viable if your EXEs and DLLs are installed across multiple directories.
3. Microsoft Visual C++ 2008 Redistributable Package:
Original distribution.
SP1 distribution. See caveats.
[Update 6/22/2009] Both of these distributions cause the the Windows 7 Logo Toolkit Beta to generate FAIL errors.
Advantage: All you have to do is run it. Permamently installs the components in the proper locations.
Disadvantage: Includes everything, so it's larger than the individual merge modules. If you are building your own installer, not as clean of a user experience as the merge modules.
4. Use the Merge Modules.
Assuming you've installed Visual Studio 2008, you can find the files at:
C:\Program Files\Common Files\Merge ModulesAdvantage: Best user experience. Smallest download.
Disadvantage: Installation isn't permanent - the DLLs may be uninstalled when your application is uninstalled. Requires you to use Windows Installer. (This is only a disadvantage for a minority of developers. Windows Installer is a logo requirement for Vista.)
Tuesday, May 20, 2008
Debugging .Net Framework Code
When I started trying to write C# code, I spent most of my time with the Reflector trying to figure out what the .Net Framework was doing under the hood. My start routine was: start the debugger, see unexpected results, read source code in Reflector, repeat. I spent a lot of time searching for functions in the Reflector.
I spent years doing MFC programming. MFC shipped with complete source code from the very early days. Without that source code, it would have been almost impossible to solve many of the problems I ran into. (See my earlier blog about fixing problems with MFC symbols in Visual Studio 2005.)
I was tickled to discover today that Microsoft has made the source code of the Framework available for debugging purposes - even specialized code like Asp.Net. You must be running Visual Studio 2008. Take a look at Shawn Burke's Blog showing how to set it up.
Make sure you install the Visual Studio hotfix (QFE)!
I spent years doing MFC programming. MFC shipped with complete source code from the very early days. Without that source code, it would have been almost impossible to solve many of the problems I ran into. (See my earlier blog about fixing problems with MFC symbols in Visual Studio 2005.)
I was tickled to discover today that Microsoft has made the source code of the Framework available for debugging purposes - even specialized code like Asp.Net. You must be running Visual Studio 2008. Take a look at Shawn Burke's Blog showing how to set it up.
Make sure you install the Visual Studio hotfix (QFE)!
Friday, May 2, 2008
Fixing "PRJ0050 Failed: Failed to register output"
I'm in the process of converting a COM DLL over to .Net using C++/CLR and I've been plagued with this error:
I used Google to try and find a solution and found more confusion than solutions. The correct solution (for me) didn't appear anywhere. So here's my guide to solving this error.
There are three possible causes. Diagnosis is usually straightforward. Run Regsvr32 and try to manually register the DLL. If you see "entry-point DllRegisterServer was not found" then jump down to Disable Registration. If you see "access denied" then jump down to User Account Control. Finally, if you get "missing dependency," jump down to Missing Dependency.
User Account Control
This error is most likely to happen on Vista when you try to register a COM DLL. The reason is that, even if you are running as Administrator, you aren't really an Administrator because of User Account Control (UAC). This problem is easy to test. Enable per-user redirection in the project properties under Linker / General. Force the project to relink (make a minor change to a source file) and see what happens. If the error goes away, then you've found the problem. Amazingly enough, PROJ0050 is specifically mentioned in the documentation for the Linker Property Pages.
Other more intrusive solutions:
Disable Registration
This was the solution that applied to my problem. My project used to be a COM object written using MFC. The COM object required registration and so the project was set to register the DLL after linking. My fancy new .Net object did not require registration, so there was no "DllRegisterServer" entry point in the DLL. I fixed the problem in the project properties under Linker / General by setting Register Output to No. Make sure you do this for both Release and Debug builds.
To diagnose this problem, use the Depends utility. Don't use the version that ships with Visual Studio, it's out of date. Download the latest version of Depends from http://www.dependencywalker.com/. When you run Depends, make sure you open the DLL that's in the target directory where the linker put it. Now look down the list in the middle pane and see what's marked with a yellow question mark. Ignore any modules with an hour glass, they are delay loaded and don't cause a problem if they are missing.
If the missing DLL is one of your DLLs, then update your path to include that DLL. Problem solved.
If the missing DLL is one of the MFC or C Runtime DLLs (which start with MSVC or MFC, respectively), then life is much more complicated. You might be having a problem with WinSxs, which means there's an error in your manifest. These problems can be nasty to fix, as I described in this post.
Finally, there was a red herring for my project when I ran Depends. The file MSVCR90D.DLL was marked as missing. I'm not sure what was causing this, but the DLL ran and loaded properly, so I'm suspicious that the problem was with Depends.
RegAsm : error RA0000 : An error occurred while writing the registration information to the registry. You must have administrative credentials to perform this task. Contact your system administrator for assistance
Project : error PRJ0050: Failed to register output. Please try enabling Per-user Redirection or register the component from a command prompt with elevated permissions.I used Google to try and find a solution and found more confusion than solutions. The correct solution (for me) didn't appear anywhere. So here's my guide to solving this error.
There are three possible causes. Diagnosis is usually straightforward. Run Regsvr32 and try to manually register the DLL. If you see "entry-point DllRegisterServer was not found" then jump down to Disable Registration. If you see "access denied" then jump down to User Account Control. Finally, if you get "missing dependency," jump down to Missing Dependency.
User Account Control
This error is most likely to happen on Vista when you try to register a COM DLL. The reason is that, even if you are running as Administrator, you aren't really an Administrator because of User Account Control (UAC). This problem is easy to test. Enable per-user redirection in the project properties under Linker / General. Force the project to relink (make a minor change to a source file) and see what happens. If the error goes away, then you've found the problem. Amazingly enough, PROJ0050 is specifically mentioned in the documentation for the Linker Property Pages.Other more intrusive solutions:
- Close Visual Studio and restart it as an Administrator by right-clicking the Visual Studio icon and selecting Run as Administrator. The screen should blink and you should see the UAC prompt.
- Always run Visual Studio as Administrator as follows. Right-click the Visual Studio icon, select Properties, go to the Compatibility tab, and enable "Run this program as an administrator."
- Disable UAC (not recommended.)
- Ignore the problem. It's not hurting anything.
Disable Registration
This was the solution that applied to my problem. My project used to be a COM object written using MFC. The COM object required registration and so the project was set to register the DLL after linking. My fancy new .Net object did not require registration, so there was no "DllRegisterServer" entry point in the DLL. I fixed the problem in the project properties under Linker / General by setting Register Output to No. Make sure you do this for both Release and Debug builds.Missing Dependency
This problem happens because your DLL is dependent on another DLL that can't be located. It's easy for this problem to happen if your DLL compiles to one directory and a dependent DLL compiles to another directory.To diagnose this problem, use the Depends utility. Don't use the version that ships with Visual Studio, it's out of date. Download the latest version of Depends from http://www.dependencywalker.com/. When you run Depends, make sure you open the DLL that's in the target directory where the linker put it. Now look down the list in the middle pane and see what's marked with a yellow question mark. Ignore any modules with an hour glass, they are delay loaded and don't cause a problem if they are missing.
If the missing DLL is one of your DLLs, then update your path to include that DLL. Problem solved.
If the missing DLL is one of the MFC or C Runtime DLLs (which start with MSVC or MFC, respectively), then life is much more complicated. You might be having a problem with WinSxs, which means there's an error in your manifest. These problems can be nasty to fix, as I described in this post.
Finally, there was a red herring for my project when I ran Depends. The file MSVCR90D.DLL was marked as missing. I'm not sure what was causing this, but the DLL ran and loaded properly, so I'm suspicious that the problem was with Depends.
Monday, January 7, 2008
Visual Studio 2008 Unusable with C++
Update 8/12/2008: Microsoft has released Service Pack 1 for Visual Studio 2008, which resolves these problems. See the last post on this page:
http://www.microsoft.com/downloads/details.aspx?FamilyID=27673c47-b3b5-4c67-bd99-84e525b5ce61
Update 3/30/2008: Microsoft has finally released two hotfixes that address these problems. See the last post on this page:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2799211&SiteID=1
I've now spent a couple of weeks using Visual Studio 2008 for C++ development, and my conclusion is that it's unusable. Those of you who follow this blog know that I'm generally supportive of Microsoft's errors, but this time there's no excuse.
Our project is about 250,000 lines of code, a relatively small project by some measures. Every few builds, you get this error:
LINK : fatal error LNK1000: Internal error during IncrBuildImage
And the build crashes. If you rebuild, it succeeds, but this doesn't help much for automated builds that rely on a deterministic compile result. The problem only appears if you have incremental linking turned on, but disabling incremental linking makes debugging unbearably slow and painful.
I'm not the only one having this problem:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=311732
If you are reading this blog, please go to that page and register your vote for an importance of five stars.
So I've just been living with that problem, since we're largely in a testing phase for VS2008 and automated builds aren't yet needed. Today I ran into another problem, and this one's a showstopper. The first time the compiler encounters a file that fails to compile because of a particular set of errors, the vc80.pdb program database file is corrupted and it's impossible to build any more files until you do a Rebuild All. Except that rebuilding everything causes the file to be rebuilt, which again corrupts the PDB file. Even if you try to compile that file by itself (using Build/Compile), the PDB file is still corrupted.
This problem was not only known, if was reported during the beta process, but never fixed:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=308775
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=309462
Both of the above problems have been open for 30 to 60 days. Perhaps a severity of "entire build fails catastrophically" isn't as serious as I think it is.
Friday, December 7, 2007
WinSxS Breaks Old Libraries
After my previous experiences with handling Windows Side-by-Side Assemblies (WinSxS) in remote debugging and Isolated COM, I thought I was actually starting to get a handle on how it worked. Today I got stuck on another WinSxS problem, this time while porting our application to Visual Studio 2008.
The problem was that the application would fail to fail to start, with an error about the manifest being wrong. I used sxstrace, a handy tool under Vista, to try and determine what was happening. Sxstrace generated 180 lines of information about Vista's attempt to find and load the correct assemblies. It ended up being too much information. The only obvious problem I saw was that one of the DLLs being loaded was from Visual Studio 2005, not 2008.
I used Depends to look at the file and received the same errors. I looked in the Event Viewer and saw this error:
This is even stranger, because 20706 was the version ID of Visual Studio 2008 Beta 2. I'm running the final release.
I used Visual Studio to open XSales.exe in Resource mode so I could look at the manifest itself. Here I found references to four versions of DebugCRT. The question was, where were they coming from? My code makes no explicit reference to assembly versions.
What I discovered was that the intermediate manifests generated by the compiler and linker include assembly information from all objects used by the linker, including objects from libraries, of which I had two. One of my .LIB files was generated by VS2005 and another was generated by VS2008 Beta 2, which is what caused the references to old assembly versions. Once I rebuilt those .LIB files with VS2008, the problem went away.
The lesson learned from all of this is that .LIB files are no longer easily portable between versions if they rely on any of the CRT or MFC DLLs. The painful part of this is that the problem doesn't show up until you try and run the software because none of the development tools warn about the inconsistency.
[Update 11/19/2008] You can find another solution here.
The problem was that the application would fail to fail to start, with an error about the manifest being wrong. I used sxstrace, a handy tool under Vista, to try and determine what was happening. Sxstrace generated 180 lines of information about Vista's attempt to find and load the correct assemblies. It ended up being too much information. The only obvious problem I saw was that one of the DLLs being loaded was from Visual Studio 2005, not 2008.
I used Depends to look at the file and received the same errors. I looked in the Event Viewer and saw this error:
Activation context generation failed for "s:\csi\xsales\debug\XSALES.EXE". Dependent Assembly Microsoft.VC90.DebugCRT, processorArchitecture="x86", publicKeyToken="1fc8b3b9a1e18e3b", type="win32",version="9.0.20706.1" could not be found. Please use sxstrace.exe for detailed diagnosis.This is even stranger, because 20706 was the version ID of Visual Studio 2008 Beta 2. I'm running the final release.
I used Visual Studio to open XSales.exe in Resource mode so I could look at the manifest itself. Here I found references to four versions of DebugCRT. The question was, where were they coming from? My code makes no explicit reference to assembly versions.
What I discovered was that the intermediate manifests generated by the compiler and linker include assembly information from all objects used by the linker, including objects from libraries, of which I had two. One of my .LIB files was generated by VS2005 and another was generated by VS2008 Beta 2, which is what caused the references to old assembly versions. Once I rebuilt those .LIB files with VS2008, the problem went away.
The lesson learned from all of this is that .LIB files are no longer easily portable between versions if they rely on any of the CRT or MFC DLLs. The painful part of this is that the problem doesn't show up until you try and run the software because none of the development tools warn about the inconsistency.
[Update 11/19/2008] You can find another solution here.
Friday, October 12, 2007
Visual Studio 2008 Beta 2 Test Results
Today I tried building our product with Visual Studio 2008 Beta 2. The last two times I tested compiler upgrades (VC6 to VS.net 2003, then VS.net 2003 to VS2005), the upgrades were extremely painful and took days to complete.
I was pleasantly surprised that rebuilding our product in VS2008 Beta 2 went quite smoothly. The issues were minor:
In terms of the size of the generated executables, here are the stats:
While not earth shattering, these numbers do show that they were able to contain bloat in MFC and the C runtime library. All of these applications are statically linked against MFC and the CRT.
The biggest surprise was that VC2008 worked fine with version 1.34.1 of the Boost library, which has not yet been tested on VC2008. (although, admittedly, our software only uses one or two of the Boost modules.)
That's it! So far, backwards compatibility is excellent.
Initial testing has shown no problems with the generated code. Common dialogs under Vista were automatically updated to Vista styling, as promised.
My only disappointment is that very little was done for the IDE for C++ developers in this version of Visual Studio. The tools for editing dialogs and other resources are still awful compared to VC6. For a look at the future of the IDE, take a look at Somasegar's blog. Attention is being renewed on the IDE for native C++ developers, but improvements won't ship until 2010.
I was pleasantly surprised that rebuilding our product in VS2008 Beta 2 went quite smoothly. The issues were minor:
- The switch /OPT:NOWIN98 option is no longer supported.
Impact: None. - Project could not be linked against a library originally built in VC6.
Impact: Rebuild library. - Visual Studio 2005 manifest that set requireAdministrator caused this error: manifest authoring error c1010001: Values of attribute "level" not equal in different manifest snippets.
Impact: Removed the explicit manifest file and set the UAC level in the Linker node on the project properties. - MT.EXE gives an error on a valid manifest.
Impact: Move $(WindowsSdkDir)\bin to the top of the list for Executables files in VC++ Directories under Tools/Options.
In terms of the size of the generated executables, here are the stats:
| Product | VS2005 size | VS2008 Size | Net Change |
| #1 (C++/MFC GUI) | 1,198,592 | 1,180,672 | -1.5% |
| #2 (C++/MFC GUI) | 1,589,760 | 1,549,312 | -2.5% |
| #3 (C++/MFC Console) | 2,043,904 | 1,998,848 | -2.2% |
While not earth shattering, these numbers do show that they were able to contain bloat in MFC and the C runtime library. All of these applications are statically linked against MFC and the CRT.
The biggest surprise was that VC2008 worked fine with version 1.34.1 of the Boost library, which has not yet been tested on VC2008. (although, admittedly, our software only uses one or two of the Boost modules.)
That's it! So far, backwards compatibility is excellent.
Initial testing has shown no problems with the generated code. Common dialogs under Vista were automatically updated to Vista styling, as promised.
My only disappointment is that very little was done for the IDE for C++ developers in this version of Visual Studio. The tools for editing dialogs and other resources are still awful compared to VC6. For a look at the future of the IDE, take a look at Somasegar's blog. Attention is being renewed on the IDE for native C++ developers, but improvements won't ship until 2010.
Thursday, October 11, 2007
Running the Visual Studio 2008 Beta 2 VHD on VMware Server
Microsoft has finally climbed on the bandwagon for using pre-configured virtual machines to distribute beta software. For anyone who tried to install earlier betas of Visual Studio 2005 Team System, you'll understand me when I say that these pre-built virtual machines will save you days of frustration.
I downloaded the VHD disk images for Visual Studio 2008 Beta 2. Although they work fine on the free download of Virtual PC 2007, I really wanted to run this image on my virtual machine server, which uses VMware Server (see my earlier comments on VMware Server versus Microsoft Virtual Server).
Before I describe the procedure, one BIG caveat: Once Windows is running in VMware, Windows will complain that it needs to be reactivated. If you are a Microsoft Partner you can get a key from MSDN Downloads. Otherwise you will need to use a new key, which basically means you need to buy Windows Server 2003 Enterprise. Therefore, if you don't have a ready supply of activation keys, this procedure won't work for you. It may be possible to call the activation people and have them honor the key built into the virtual machine, but I haven't tried.
Converting the Orcas VHD to VMware ended up being a lot more difficult than I'd hoped. The biggest problem was converting the virtual drives from .vhd format to .vmdk format. I found a nice utility named WinImage that could do this. WinImage converted the base Orcas image (2.8GB) without difficulty, but gave up with no error when I tried to convert the 11.8GB differencing disk. I didn't really want to go back and forth with the WinImage support for two days, so I looked for an alternative.
My final solution was to use Acronis to do a backup in Virtual PC, then use Acronis again to do a restore in VMware. To do this yourself, you'll need:
VMware Server (free)
Acronis TrueImage Home or better (commercial)
A Windows Server 2003 Enterprise installation CD
WinImage (shareware)
The solution was as follows:
1. Install Acronis TrueImage on any Windows box and create a Rescue CD.
2. Create a virtual machine in Virtual PC 2007 that contains Orcas Beta 2.
3. Set the virtual machine to connect to the CD you created in Step #1.
4. Boot the version machine, select Acronis and back up the Orcas virtual machine to any desired network drive.
5. Use WinImage to convert the Base01 image to VMDK. Make sure you create a dynamic disk and not a fixed disk.
6. In VMware Server, create a virtual machine that points at the file from #5.
7. Put the Acronis Rescue CD in a CD drive on that computer.
8. Start the virtual machine in VMware, press Esc, and boot from the Rescue CD.
9. Restore the Acronis backup to the current disk.
10. After restore completes, reboot the virtual machine. You'll get an error about a service that didn't start. Ignore it.
11. On the VM menu, select Send Ctrl-Alt-Del.
12. Enter the password from the Microsoft web page. You'll need to use the keyboard, your mouse probably won't work.
13. As the login completes, you'll be prompted for the path to install the Ethernet card. Put in the Windows Server 2003 Enterprise installation CD and click OK. Windows will complain that it can't find the driver, which is okay. Tell Windows you want to install from an alternative location.
14. Select the file Driver.cab on the Windows Server 2003 Enterprise installation CD. The network driver should be located automatically.
15. After installation of the initial driver completes, do not reboot or you won't be able to login again.
16. On the VM menu in the VMware Console, choose Install VMware tools.
17 At some point you'll be prompted for the file mouclass.sys. Tell the installer to use the copy of the file in C:\Windows\System32\Drivers.
18. Now you can reboot and everything should work.
I downloaded the VHD disk images for Visual Studio 2008 Beta 2. Although they work fine on the free download of Virtual PC 2007, I really wanted to run this image on my virtual machine server, which uses VMware Server (see my earlier comments on VMware Server versus Microsoft Virtual Server).
Before I describe the procedure, one BIG caveat: Once Windows is running in VMware, Windows will complain that it needs to be reactivated. If you are a Microsoft Partner you can get a key from MSDN Downloads. Otherwise you will need to use a new key, which basically means you need to buy Windows Server 2003 Enterprise. Therefore, if you don't have a ready supply of activation keys, this procedure won't work for you. It may be possible to call the activation people and have them honor the key built into the virtual machine, but I haven't tried.
Converting the Orcas VHD to VMware ended up being a lot more difficult than I'd hoped. The biggest problem was converting the virtual drives from .vhd format to .vmdk format. I found a nice utility named WinImage that could do this. WinImage converted the base Orcas image (2.8GB) without difficulty, but gave up with no error when I tried to convert the 11.8GB differencing disk. I didn't really want to go back and forth with the WinImage support for two days, so I looked for an alternative.
My final solution was to use Acronis to do a backup in Virtual PC, then use Acronis again to do a restore in VMware. To do this yourself, you'll need:
VMware Server (free)
Acronis TrueImage Home or better (commercial)
A Windows Server 2003 Enterprise installation CD
WinImage (shareware)
The solution was as follows:
1. Install Acronis TrueImage on any Windows box and create a Rescue CD.
2. Create a virtual machine in Virtual PC 2007 that contains Orcas Beta 2.
3. Set the virtual machine to connect to the CD you created in Step #1.
4. Boot the version machine, select Acronis and back up the Orcas virtual machine to any desired network drive.
5. Use WinImage to convert the Base01 image to VMDK. Make sure you create a dynamic disk and not a fixed disk.
6. In VMware Server, create a virtual machine that points at the file from #5.
7. Put the Acronis Rescue CD in a CD drive on that computer.
8. Start the virtual machine in VMware, press Esc, and boot from the Rescue CD.
9. Restore the Acronis backup to the current disk.
10. After restore completes, reboot the virtual machine. You'll get an error about a service that didn't start. Ignore it.
11. On the VM menu, select Send Ctrl-Alt-Del.
12. Enter the password from the Microsoft web page. You'll need to use the keyboard, your mouse probably won't work.
13. As the login completes, you'll be prompted for the path to install the Ethernet card. Put in the Windows Server 2003 Enterprise installation CD and click OK. Windows will complain that it can't find the driver, which is okay. Tell Windows you want to install from an alternative location.
14. Select the file Driver.cab on the Windows Server 2003 Enterprise installation CD. The network driver should be located automatically.
15. After installation of the initial driver completes, do not reboot or you won't be able to login again.
16. On the VM menu in the VMware Console, choose Install VMware tools.
17 At some point you'll be prompted for the file mouclass.sys. Tell the installer to use the copy of the file in C:\Windows\System32\Drivers.
18. Now you can reboot and everything should work.
Subscribe to:
Posts (Atom)