Remember CP/M? Unless you are over 40, my guess is probably not. This week I ran into a bug in our software that traces its roots all the way back to CP/M in the 1970s. Over thirty years later, the code still exists in the Visual Studio 2005 C Runtime Library, waiting for the next innocent victim to run afoul of it.
Here's what happened. We had reverse engineered a data file format and had shipped the first pass to customers. Our only sample of the data file was quite small, about 10 records, but it was enough to determine the file format and make it work. The ten records in the file converted cleanly.
After the product shipped, several customers reported that only 26 records were being loaded. This was obviously incorrect as most of their files had hundreds of records. Our end-of-file handling code was common with numerous other modules and worked fine. The 26 records that did convert did so correctly.
Even when we instrumented our software to give more insight into what was happening at customer sites, we found nothing. Our software converted 26 records, detected end-of-file, and exited. WTF?
When we finally found a customer willing to share his data file, we ran our software in the debugger and got exactly the same results. 26 records were converted and the software exited cleanly, with no errors.
The lightbulb didn't go off over my head until I was looking at the data file with the cygwin "cat -v" command, which shows ASCII codes 0 through 31 as control characters. This particular data file had two bytes for each record ID and record numbering started at 0. The 27th record contained ID 26 (0x1a) which showed up as ^Z. Does that ring any bells? If you ever developed for CP/M (or for MS-DOS 1.0) it should.
Thirty years ago, CP/M only tracked the number of blocks in each file, not the number of bytes. By convention, Ctrl-Z was used to denote "end of file" for text files. MS-DOS 1.0, which bore a striking resemblance to CP/M internally, followed this same convention. At the time, the C Runtime Library understood this convention and automatically generated an "end of file" condition when ^Z was encountered. Today, the VS2005 C Runtime Library still contains that code and generates the end-of-file condition even if the exact length of the file extends beyond that point.
The bug in our software was that the file had been opened in text mode instead of binary mode. Normally this is easy to detect because the records become out of sync as they are read, but by some coincidence of the data layout, the data in this file was read in perfectly up until that ^Z.
So now I have one more reason to dislike CP/M, although (admittedly) in this day and age it seems somewhat pointless to carry a grudge against a dead operating system that was designed to run off of 8" floppy disks. Old habits are hard to break.
Tuesday, November 13, 2007
Sunday, October 14, 2007
Static Code Analysis with Visual Studio 2005
Not too long ago I received a jarring reminder that "char" in Visual C++ is signed by default. The problem was related to the ctype functions, such as isalnum() and isspace(), all of which take an integer. When you sign-extend an eight-bit multibyte character (MBCS), you end up with a negative number. The ctype functions then do a table lookup on a negative number and your program crashes. This wouldn't be so interesting, except that this bug existed in our production code for six years before the software crashed for a customer in Europe. If the customer hadn't been willing to give me sample data to reproduce the problem, I'd still be scratching my head.
Apparently I'm not the only person who has run into this issue. Visual C++ 2005 includes a command line option named "/analyze" for finding this problem and many others. I had tried /analyze during the Whidbey Beta two years ago but didn't get useful results. Now the results were much more helpful.
The "/analyze" options enables the static code analysis feature in the C++ compiler. This feature is based on prefast, a technology that came out of Microsoft Research in 2001. Prefast knows about hundreds of common programming problems. I ran it on our codebase, which after eight years is quite mature and already compiled under warning level 4 with minimal problems.
What I learned was quite interesting. Prefast knows about the ctype problem I mentioned earlier and pointed out several lines I missed when I tried to fix the problem. All of those lines were crashes waiting to happen.
Prefast found several places where I expected one return type but was getting another. For example, at one point I checked for an HRESULT but was actually getting a bool, which meant that the sense of my error check was inverted.
Another error that prefast found was where I was calling sizeof on a variable that was defined as pointer instead of as an array, which meant that the length being handed to strncmp was wildly wrong. The code still worked, but that was a happy accident.
Prefast also had numerous warnings about the Boost C++ Library. It should be possible to suppress those warnings, but I haven't done so yet.
Prefast supports annotating your code with the Standard Annotation Language (SAL) to better describe the static behavior of the code. If you've looked at recent versions of the Windows SDK, there are numerous annotations such as __out_ecount(). These annotations provide additional information to prefast that allows for better analysis. You can learn more about these annotations at blogs.msdn.com.
Prefast also pointed out two problems with my code that I would never have found, even with a close code inspection. I'm sure you've seen stricmp, the case-insensitive version of strcmp. I was using stricmp to check for keywords. Turns out that this is a bad idea. Prefast gives warning C6400, which explains that some languages interpret combinations of letters as a single letter, which changes the behavior of stricmp. The correct solution is to use the Windows API call CompareString(), which should be set to LOCALE_INVARIANT or LANG_ENGLISH, depending on the version of Windows.
Prefast also warned me that using _alloca in a loop could cause a stack overflow. Normally I'd consider this obvious, but in this case the warning was being given about the W2A Unicode to MBCS conversion macro, which uses _alloca. I looked at the source code to W2A (in atlconv.h) and the problem seems to be handled properly, but it was a worthwhile exercise.
I'm adding prefast to my bag of recommended tricks. For more insight into the use of this tool, I invite you to read Scalable Defect Detection from the Center for Software Excellence at Microsoft.
Apparently I'm not the only person who has run into this issue. Visual C++ 2005 includes a command line option named "/analyze" for finding this problem and many others. I had tried /analyze during the Whidbey Beta two years ago but didn't get useful results. Now the results were much more helpful.
The "/analyze" options enables the static code analysis feature in the C++ compiler. This feature is based on prefast, a technology that came out of Microsoft Research in 2001. Prefast knows about hundreds of common programming problems. I ran it on our codebase, which after eight years is quite mature and already compiled under warning level 4 with minimal problems.
What I learned was quite interesting. Prefast knows about the ctype problem I mentioned earlier and pointed out several lines I missed when I tried to fix the problem. All of those lines were crashes waiting to happen.
Prefast found several places where I expected one return type but was getting another. For example, at one point I checked for an HRESULT but was actually getting a bool, which meant that the sense of my error check was inverted.
Another error that prefast found was where I was calling sizeof on a variable that was defined as pointer instead of as an array, which meant that the length being handed to strncmp was wildly wrong. The code still worked, but that was a happy accident.
Prefast also had numerous warnings about the Boost C++ Library. It should be possible to suppress those warnings, but I haven't done so yet.
Prefast supports annotating your code with the Standard Annotation Language (SAL) to better describe the static behavior of the code. If you've looked at recent versions of the Windows SDK, there are numerous annotations such as __out_ecount(). These annotations provide additional information to prefast that allows for better analysis. You can learn more about these annotations at blogs.msdn.com.
Prefast also pointed out two problems with my code that I would never have found, even with a close code inspection. I'm sure you've seen stricmp, the case-insensitive version of strcmp. I was using stricmp to check for keywords. Turns out that this is a bad idea. Prefast gives warning C6400, which explains that some languages interpret combinations of letters as a single letter, which changes the behavior of stricmp. The correct solution is to use the Windows API call CompareString(), which should be set to LOCALE_INVARIANT or LANG_ENGLISH, depending on the version of Windows.
Prefast also warned me that using _alloca in a loop could cause a stack overflow. Normally I'd consider this obvious, but in this case the warning was being given about the W2A Unicode to MBCS conversion macro, which uses _alloca. I looked at the source code to W2A (in atlconv.h) and the problem seems to be handled properly, but it was a worthwhile exercise.
I'm adding prefast to my bag of recommended tricks. For more insight into the use of this tool, I invite you to read Scalable Defect Detection from the Center for Software Excellence at Microsoft.
Friday, October 12, 2007
Visual Studio 2008 Beta 2 Test Results
Today I tried building our product with Visual Studio 2008 Beta 2. The last two times I tested compiler upgrades (VC6 to VS.net 2003, then VS.net 2003 to VS2005), the upgrades were extremely painful and took days to complete.
I was pleasantly surprised that rebuilding our product in VS2008 Beta 2 went quite smoothly. The issues were minor:
In terms of the size of the generated executables, here are the stats:
While not earth shattering, these numbers do show that they were able to contain bloat in MFC and the C runtime library. All of these applications are statically linked against MFC and the CRT.
The biggest surprise was that VC2008 worked fine with version 1.34.1 of the Boost library, which has not yet been tested on VC2008. (although, admittedly, our software only uses one or two of the Boost modules.)
That's it! So far, backwards compatibility is excellent.
Initial testing has shown no problems with the generated code. Common dialogs under Vista were automatically updated to Vista styling, as promised.
My only disappointment is that very little was done for the IDE for C++ developers in this version of Visual Studio. The tools for editing dialogs and other resources are still awful compared to VC6. For a look at the future of the IDE, take a look at Somasegar's blog. Attention is being renewed on the IDE for native C++ developers, but improvements won't ship until 2010.
I was pleasantly surprised that rebuilding our product in VS2008 Beta 2 went quite smoothly. The issues were minor:
- The switch /OPT:NOWIN98 option is no longer supported.
Impact: None. - Project could not be linked against a library originally built in VC6.
Impact: Rebuild library. - Visual Studio 2005 manifest that set requireAdministrator caused this error: manifest authoring error c1010001: Values of attribute "level" not equal in different manifest snippets.
Impact: Removed the explicit manifest file and set the UAC level in the Linker node on the project properties. - MT.EXE gives an error on a valid manifest.
Impact: Move $(WindowsSdkDir)\bin to the top of the list for Executables files in VC++ Directories under Tools/Options.
In terms of the size of the generated executables, here are the stats:
| Product | VS2005 size | VS2008 Size | Net Change |
| #1 (C++/MFC GUI) | 1,198,592 | 1,180,672 | -1.5% |
| #2 (C++/MFC GUI) | 1,589,760 | 1,549,312 | -2.5% |
| #3 (C++/MFC Console) | 2,043,904 | 1,998,848 | -2.2% |
While not earth shattering, these numbers do show that they were able to contain bloat in MFC and the C runtime library. All of these applications are statically linked against MFC and the CRT.
The biggest surprise was that VC2008 worked fine with version 1.34.1 of the Boost library, which has not yet been tested on VC2008. (although, admittedly, our software only uses one or two of the Boost modules.)
That's it! So far, backwards compatibility is excellent.
Initial testing has shown no problems with the generated code. Common dialogs under Vista were automatically updated to Vista styling, as promised.
My only disappointment is that very little was done for the IDE for C++ developers in this version of Visual Studio. The tools for editing dialogs and other resources are still awful compared to VC6. For a look at the future of the IDE, take a look at Somasegar's blog. Attention is being renewed on the IDE for native C++ developers, but improvements won't ship until 2010.
Thursday, October 11, 2007
Running the Visual Studio 2008 Beta 2 VHD on VMware Server
Microsoft has finally climbed on the bandwagon for using pre-configured virtual machines to distribute beta software. For anyone who tried to install earlier betas of Visual Studio 2005 Team System, you'll understand me when I say that these pre-built virtual machines will save you days of frustration.
I downloaded the VHD disk images for Visual Studio 2008 Beta 2. Although they work fine on the free download of Virtual PC 2007, I really wanted to run this image on my virtual machine server, which uses VMware Server (see my earlier comments on VMware Server versus Microsoft Virtual Server).
Before I describe the procedure, one BIG caveat: Once Windows is running in VMware, Windows will complain that it needs to be reactivated. If you are a Microsoft Partner you can get a key from MSDN Downloads. Otherwise you will need to use a new key, which basically means you need to buy Windows Server 2003 Enterprise. Therefore, if you don't have a ready supply of activation keys, this procedure won't work for you. It may be possible to call the activation people and have them honor the key built into the virtual machine, but I haven't tried.
Converting the Orcas VHD to VMware ended up being a lot more difficult than I'd hoped. The biggest problem was converting the virtual drives from .vhd format to .vmdk format. I found a nice utility named WinImage that could do this. WinImage converted the base Orcas image (2.8GB) without difficulty, but gave up with no error when I tried to convert the 11.8GB differencing disk. I didn't really want to go back and forth with the WinImage support for two days, so I looked for an alternative.
My final solution was to use Acronis to do a backup in Virtual PC, then use Acronis again to do a restore in VMware. To do this yourself, you'll need:
VMware Server (free)
Acronis TrueImage Home or better (commercial)
A Windows Server 2003 Enterprise installation CD
WinImage (shareware)
The solution was as follows:
1. Install Acronis TrueImage on any Windows box and create a Rescue CD.
2. Create a virtual machine in Virtual PC 2007 that contains Orcas Beta 2.
3. Set the virtual machine to connect to the CD you created in Step #1.
4. Boot the version machine, select Acronis and back up the Orcas virtual machine to any desired network drive.
5. Use WinImage to convert the Base01 image to VMDK. Make sure you create a dynamic disk and not a fixed disk.
6. In VMware Server, create a virtual machine that points at the file from #5.
7. Put the Acronis Rescue CD in a CD drive on that computer.
8. Start the virtual machine in VMware, press Esc, and boot from the Rescue CD.
9. Restore the Acronis backup to the current disk.
10. After restore completes, reboot the virtual machine. You'll get an error about a service that didn't start. Ignore it.
11. On the VM menu, select Send Ctrl-Alt-Del.
12. Enter the password from the Microsoft web page. You'll need to use the keyboard, your mouse probably won't work.
13. As the login completes, you'll be prompted for the path to install the Ethernet card. Put in the Windows Server 2003 Enterprise installation CD and click OK. Windows will complain that it can't find the driver, which is okay. Tell Windows you want to install from an alternative location.
14. Select the file Driver.cab on the Windows Server 2003 Enterprise installation CD. The network driver should be located automatically.
15. After installation of the initial driver completes, do not reboot or you won't be able to login again.
16. On the VM menu in the VMware Console, choose Install VMware tools.
17 At some point you'll be prompted for the file mouclass.sys. Tell the installer to use the copy of the file in C:\Windows\System32\Drivers.
18. Now you can reboot and everything should work.
I downloaded the VHD disk images for Visual Studio 2008 Beta 2. Although they work fine on the free download of Virtual PC 2007, I really wanted to run this image on my virtual machine server, which uses VMware Server (see my earlier comments on VMware Server versus Microsoft Virtual Server).
Before I describe the procedure, one BIG caveat: Once Windows is running in VMware, Windows will complain that it needs to be reactivated. If you are a Microsoft Partner you can get a key from MSDN Downloads. Otherwise you will need to use a new key, which basically means you need to buy Windows Server 2003 Enterprise. Therefore, if you don't have a ready supply of activation keys, this procedure won't work for you. It may be possible to call the activation people and have them honor the key built into the virtual machine, but I haven't tried.
Converting the Orcas VHD to VMware ended up being a lot more difficult than I'd hoped. The biggest problem was converting the virtual drives from .vhd format to .vmdk format. I found a nice utility named WinImage that could do this. WinImage converted the base Orcas image (2.8GB) without difficulty, but gave up with no error when I tried to convert the 11.8GB differencing disk. I didn't really want to go back and forth with the WinImage support for two days, so I looked for an alternative.
My final solution was to use Acronis to do a backup in Virtual PC, then use Acronis again to do a restore in VMware. To do this yourself, you'll need:
VMware Server (free)
Acronis TrueImage Home or better (commercial)
A Windows Server 2003 Enterprise installation CD
WinImage (shareware)
The solution was as follows:
1. Install Acronis TrueImage on any Windows box and create a Rescue CD.
2. Create a virtual machine in Virtual PC 2007 that contains Orcas Beta 2.
3. Set the virtual machine to connect to the CD you created in Step #1.
4. Boot the version machine, select Acronis and back up the Orcas virtual machine to any desired network drive.
5. Use WinImage to convert the Base01 image to VMDK. Make sure you create a dynamic disk and not a fixed disk.
6. In VMware Server, create a virtual machine that points at the file from #5.
7. Put the Acronis Rescue CD in a CD drive on that computer.
8. Start the virtual machine in VMware, press Esc, and boot from the Rescue CD.
9. Restore the Acronis backup to the current disk.
10. After restore completes, reboot the virtual machine. You'll get an error about a service that didn't start. Ignore it.
11. On the VM menu, select Send Ctrl-Alt-Del.
12. Enter the password from the Microsoft web page. You'll need to use the keyboard, your mouse probably won't work.
13. As the login completes, you'll be prompted for the path to install the Ethernet card. Put in the Windows Server 2003 Enterprise installation CD and click OK. Windows will complain that it can't find the driver, which is okay. Tell Windows you want to install from an alternative location.
14. Select the file Driver.cab on the Windows Server 2003 Enterprise installation CD. The network driver should be located automatically.
15. After installation of the initial driver completes, do not reboot or you won't be able to login again.
16. On the VM menu in the VMware Console, choose Install VMware tools.
17 At some point you'll be prompted for the file mouclass.sys. Tell the installer to use the copy of the file in C:\Windows\System32\Drivers.
18. Now you can reboot and everything should work.
Saturday, September 15, 2007
Upgrading PEAR on Red Hat Enterprise ES 3
Recently we've upgraded our coding standards for our server-side PHP development to include unit tests, which required that we install PHPUnit for PHP 4. Normally installing a new pear module takes about ten seconds, like this:
Unfortunately, this time things didn't go so smoothly. I ended up with dozens of errors such as this one:
The list of errors ended with:
If you look up these errors on Google, you get very few hits. Justin Patrin correctly identifies the problem: Red Hat ES 3 ships with version 1.1. This version of pear is considered ancient and doesn't support reading the latest packages.
You can check your version of pear with this command:
Justin recommends the standard command for upgrading pear:
Unfortunately, this command doesn't work when your current version of pear is so far out of date. In fact, the net result of this command is that the "pear" command is removed and not replaced. If you follow the link, you'll see a bug about this in the pear bug database, where the problem is considered "Bogus" (not a bug) because the author of the bug is running a "museum PEAR version (RHE3, pear 1.1)." Unfortunately, there is no discussion of how to actually perform the upgrade, since the command "pear upgrade pear" is completely broken in RHE3. I find it absurd that anyone who supports enterprise software would think that it's okay to simply orphan customers who are running software four years old.
I found several articles on resolving problems with old versions of pear. All of them recommended reinstalling pear from scratch, similar to this:
Unfortunately, this didn't work for me. The Red Hat Package Manager (rpm) has very definite ideas as to where pear should be installed. However, rpm bundles pear with php, so you cannot simply uninstall Red Hat's pear and then install the standard version. In my case, I was unable to determine how to get go-pear to align its directory structure with rpm, and I ended up with a schizophrenic installation.
I removed all of /usr/share/pear and restored the directory from backup. I then double checked the installed packages:
Pear reported that no packages were installed. Since the Red Hat version of pear actually ships with several packages, it was clear that my installation was now completely broken.
After several hours of searching, I finally found on article by Patrick Berry titled Upgrading PEAR on RHEL 4. Although I was running RHEL 3 and not 4, the solution described gave me enough information to solve the problem.
The problem is that version 1.1 of pear cannot read the package to install the latest version, which is 1.67 as of this blog entry. Therefore, you have to go through some intermediate upgrades before eventually arriving at version 1.67.
I first tried following the steps under Patrick's Solution section. It didn't work for me - I received the error pear is not installed. This error appears to have been caused by my earlier attempt to install from go-pear. After much frustration, I solved this problem by removing the file
However, Patrick's steps under Solution still didn't work for me. If I had bothered to read the Update AND the Comments, I would have seen the latest correct upgrade procedure, as described by Daniel a mere ten days ago:
However, AGAIN this procedure didn't work for me. (Are you sensing a trend here?) When I tried to go from 1.3.3 to 1.4.11, I received an error about "getopt2 not found." It's not clear what caused this problem, but I saw a similar issue reported by others, again with no solution.
To resolve this problem, I had to manually download an interim version of Console_Getopt and copy it to
Once I put in place a newer version of Console_Getopt, I was able to upgrade to pear 1.4.11, and then to the latest version, with these commands:
The
After all of this effort, my installation of pear finally worked again, after wasting a mere two days figuring out all of the problems. (Begin Sarcasm) A big thank you to both Red Hat and the "pear" package manager maintainers for abandoning users of Red Hat Enterprise 3.(End Sarcasm)
Late-Breaking Information:
I just learned that the PEAR home page says that support for PEAR 1.3.6 and earlier will be dropped on January 1, 2008. Since today is still four months before that date, it seems that they've jumped the gun a little bit. However, they do give the definitive steps on how to upgrade your PEAR installation:
However, it's impossible to find this from any search engine because they don't list the symptoms or the affected operating systems, nor do they refer to this procedure in Bug #5906 that I mentioned earlier.
Even More Late-Breaking Information:
Turns out that my pear installation still had issues because of all of the experimentation, so I had to wipe it and start over. This time I was able to get go-pear to work by following the instructions on this web page:
http://aspn.activestate.com/ASPN/Mail/Message/pear-general/1305645
pear install PHPUnit
Unfortunately, this time things didn't go so smoothly. I ended up with dozens of errors such as this one:
Warning: xml_parse() [http://www.php.net/function.xml-parse]: Unable to call handler _pkginfo_cdata_2_0() in Common.php on line 758
The list of errors ended with:
Notice: Undefined index: package in Common.php on line 1122
The following errors where found (use force option to install anyway):
missing package name
missing summary
missing description
missing license
missing version
missing release state
missing release date
missing release notes
no maintainer(s)
no files
If you look up these errors on Google, you get very few hits. Justin Patrin correctly identifies the problem: Red Hat ES 3 ships with version 1.1. This version of pear is considered ancient and doesn't support reading the latest packages.
You can check your version of pear with this command:
pear -V
Justin recommends the standard command for upgrading pear:
pear upgrade pear
Unfortunately, this command doesn't work when your current version of pear is so far out of date. In fact, the net result of this command is that the "pear" command is removed and not replaced. If you follow the link, you'll see a bug about this in the pear bug database, where the problem is considered "Bogus" (not a bug) because the author of the bug is running a "museum PEAR version (RHE3, pear 1.1)." Unfortunately, there is no discussion of how to actually perform the upgrade, since the command "pear upgrade pear" is completely broken in RHE3. I find it absurd that anyone who supports enterprise software would think that it's okay to simply orphan customers who are running software four years old.
I found several articles on resolving problems with old versions of pear. All of them recommended reinstalling pear from scratch, similar to this:
lynx -source http://go-pear.org/ php
Unfortunately, this didn't work for me. The Red Hat Package Manager (rpm) has very definite ideas as to where pear should be installed. However, rpm bundles pear with php, so you cannot simply uninstall Red Hat's pear and then install the standard version. In my case, I was unable to determine how to get go-pear to align its directory structure with rpm, and I ended up with a schizophrenic installation.
I removed all of /usr/share/pear and restored the directory from backup. I then double checked the installed packages:
pear list
Pear reported that no packages were installed. Since the Red Hat version of pear actually ships with several packages, it was clear that my installation was now completely broken.
After several hours of searching, I finally found on article by Patrick Berry titled Upgrading PEAR on RHEL 4. Although I was running RHEL 3 and not 4, the solution described gave me enough information to solve the problem.
The problem is that version 1.1 of pear cannot read the package to install the latest version, which is 1.67 as of this blog entry. Therefore, you have to go through some intermediate upgrades before eventually arriving at version 1.67.
I first tried following the steps under Patrick's Solution section. It didn't work for me - I received the error pear is not installed. This error appears to have been caused by my earlier attempt to install from go-pear. After much frustration, I solved this problem by removing the file
/root/.pearrc.However, Patrick's steps under Solution still didn't work for me. If I had bothered to read the Update AND the Comments, I would have seen the latest correct upgrade procedure, as described by Daniel a mere ten days ago:
pear upgrade pear-1.3.3
pear upgrade pear-1.4.11
pear upgrade
However, AGAIN this procedure didn't work for me. (Are you sensing a trend here?) When I tried to go from 1.3.3 to 1.4.11, I received an error about "getopt2 not found." It's not clear what caused this problem, but I saw a similar issue reported by others, again with no solution.
To resolve this problem, I had to manually download an interim version of Console_Getopt and copy it to
/usr/share/pear/Console. The latest version was too new and version 1.21 was too old. One of the versions in the middle included getopt2 and didn't rely on any other unsupported functionality.Once I put in place a newer version of Console_Getopt, I was able to upgrade to pear 1.4.11, and then to the latest version, with these commands:
pear upgrade pear-1.4.11
pear upgrade-all
The
upgrade-all command also fixes the pear package manager so it shows the correct version of Console_Getopt.After all of this effort, my installation of pear finally worked again, after wasting a mere two days figuring out all of the problems. (Begin Sarcasm) A big thank you to both Red Hat and the "pear" package manager maintainers for abandoning users of Red Hat Enterprise 3.(End Sarcasm)
Late-Breaking Information:
I just learned that the PEAR home page says that support for PEAR 1.3.6 and earlier will be dropped on January 1, 2008. Since today is still four months before that date, it seems that they've jumped the gun a little bit. However, they do give the definitive steps on how to upgrade your PEAR installation:
pear upgrade --force PEAR-1.3.6 Archive_Tar-1.3.1 Console_Getopt-1.2
pear upgrade --force PEAR-1.4.11
pear upgrade PEAR
However, it's impossible to find this from any search engine because they don't list the symptoms or the affected operating systems, nor do they refer to this procedure in Bug #5906 that I mentioned earlier.
Even More Late-Breaking Information:
Turns out that my pear installation still had issues because of all of the experimentation, so I had to wipe it and start over. This time I was able to get go-pear to work by following the instructions on this web page:
http://aspn.activestate.com/ASPN/Mail/Message/pear-general/1305645
Sunday, August 26, 2007
How To Create 32-bit Import Libraries Without .OBJs or Source
This article is intentionally titled the same as Microsoft KB article 131313 (was Q131313) because that KB article does not give enough information for you to create a .LIB file that works with Windows system DLLs.
There are some functions in the Win32 API that have no import library. One example is the function named RemoveControlByName() in OCCACHE.DLL. According to the documentation, the only way to use the function is to use LoadLibrary() and GetProcAddress(). This is error-prone, requires a lot of code to maintain, and isn't usable with the Delay-Load feature of VC++.
Obviously it would be preferable to have a .LIB to link against, but creating such a library is challenging. If you use Dependency Walker to look at OCCACHE.DLL, you'll see that the function is named simply RemoveControlByName. While that seems obvious, it shouldn't be possible to have this name because it doesn't include any notation for the calling convention.
If the function were __cdecl, then the function name should have started with an underline, such as _RemoveControlByName. If the function were __stdcall, then the underline should have been added as well as suffix to indicate the number of bytes in its call stack. RemoveControlByName has five 4-byte paramters, so a __stdcall signature should have looked like _RemoveControlByName@20. However, the function name has no decorations at all, which should be impossible according to Microsoft's discussion of name decoration.
The Q131313 article discusses the general case of manually creating a .LIB file for a .DLL file. The discussion under Creating a .DEF File says "The reason for this limitation is based on an assumption made by the LIB utility that all names are automatically exported without a leading underscore." This seems promising because we don't have a leading underscore. However, functions exported from Windows DLLs(almost) always using __stdcall, and the discussion under Creating a .DEF File is only applicable to __cdecl.
In spite of those warnings, I spent quite a bit of time trying to craft a .DEF file that described what I was trying to do. (You can use LIB.EXE to compile a .DEF file to a .LIB without any .OBJ files) Although most .DEF files simply list the undecorated function names under the EXPORTS section, the syntax of the .DEF file allows for creating aliases and other strange constructs. Some of the things I tried included:
However, none of these generated a .LIB that worked. The first two I couldn't link against and the third would link but fail to run.
I thought I'd learn something by looking at the symbol that the parent program is trying to link against, but that was even worse:
I knew my header file was correct, so that was definitely the correct name. However, you'll notice the leading "__imp", which indicates that the symbol is being imported from a DLL. This was something else that apparently needed to be included in my hand-crafted .LIB file, and I hadn't seen any discussion anywhere on how to do that.
I've tried to solve this problem two other times in the last several years, and this was the point I gave up in both of those cases. However, this time, failure was not an option. I needed the solution.
I tried the second option in the KB article, described under Stubbing Out Functions. I painstakingly created a dozen functions that mimicked the signatures of the functions listed in the header file. If you are doing this for yourself, here's a tip for creating a Visual Studio project. The final file you need is a .LIB file, so it's tempting to use one of the Visual Studio projects for creating a .LIB. However, that's wrong. If you created a .LIB, then you'll link with the stub functions you created, which is useless. What you really want is a DLL project, which happens to create a .LIB as a by-product.
Anyway, I created the functions as described in the KB article. Since my project used .cpp files, all of the calls in my header file had to be declared extern "C". For example:
If you look in any of the standard Windows include files, such as winbase.h, you'll see this same declaration. Note that this declaration has no relation to _cdecl and therefore has no impact on the calling convention. In other words, it doesn't force all of the functions in occache.h to be called with _cdecl. What this declaration does is to modify all of the linker symbols so that they won't include the C++ name decorations, which encode all of the function's parameters into the function name.
I also updated all of the function signatures in the header file to include __declspec(dllexport).
I compiled it, linked my main application to the new .LIB file, and it linked! I thought I was done, but when I ran the application, I received the error "Entry Point Not Found: The procedure entry point _RemoveControlByName@20 could not be located in the dynamic link library OCCACHE.DLL"
I examined the .LIB with DUMPBIN. Under Public Symbols, I now see two definitions for RemoveControlByName:
It appears that the "__imp" definition was a result of adding __declspec(dllexport), so that explained why the application linked successfully. One problem solved.
Continuing my examination of the DUMPBIN information, I saw that RemoveControlByName was defined as:
To determine whether or not this was correct, I used DUMPBIN to compare against known-good definitions in USER32.LIB, where I found that the "Name type" in the USER32 records was defined as "undecorate" instead of "name". Obviously, there was a magic incantation to set this flag, presumably in the .DEF file.
I added a .DEF file and spent several hours trying various alias combinations, none of which worked. Finally, in desperation, I created a .DEF file that contained just the raw function names. For example:
I built my application, it linked, and it ran. What happened?
I ran DUMPBIN again on my library. The record describing RemoveControlByName now contained the "undecorate" attribute:
Also, the very last line in the record showed that the "Name" was RemoveControlByName, with no decoration. Exactly what I needed.
It's clear that there's quite a bit of undocumented behavior here. Adding the entry to the .DEF file had a rather dramatic effect on the generated library. I couldn't find any mention of this behavior in the documentation on .DEF files. The only relevant reference I could find was in Microsoft's documentation under __cdecl, where it says "Underscore character (_) is prefixed to names, except when exporting __cdecl functions that use C linkage." This statement is true, but it's not the whole truth. To create a .LIB file that can link against such a construct, you also must have the function declared in a .DEF file.
In summary, to create a .LIB file that will let you link with Windows system DLLs, you need to:
There are some functions in the Win32 API that have no import library. One example is the function named RemoveControlByName() in OCCACHE.DLL. According to the documentation, the only way to use the function is to use LoadLibrary() and GetProcAddress(). This is error-prone, requires a lot of code to maintain, and isn't usable with the Delay-Load feature of VC++.
Obviously it would be preferable to have a .LIB to link against, but creating such a library is challenging. If you use Dependency Walker to look at OCCACHE.DLL, you'll see that the function is named simply RemoveControlByName. While that seems obvious, it shouldn't be possible to have this name because it doesn't include any notation for the calling convention.
If the function were __cdecl, then the function name should have started with an underline, such as _RemoveControlByName. If the function were __stdcall, then the underline should have been added as well as suffix to indicate the number of bytes in its call stack. RemoveControlByName has five 4-byte paramters, so a __stdcall signature should have looked like _RemoveControlByName@20. However, the function name has no decorations at all, which should be impossible according to Microsoft's discussion of name decoration.
The Q131313 article discusses the general case of manually creating a .LIB file for a .DLL file. The discussion under Creating a .DEF File says "The reason for this limitation is based on an assumption made by the LIB utility that all names are automatically exported without a leading underscore." This seems promising because we don't have a leading underscore. However, functions exported from Windows DLLs(almost) always using __stdcall, and the discussion under Creating a .DEF File is only applicable to __cdecl.
In spite of those warnings, I spent quite a bit of time trying to craft a .DEF file that described what I was trying to do. (You can use LIB.EXE to compile a .DEF file to a .LIB without any .OBJ files) Although most .DEF files simply list the undecorated function names under the EXPORTS section, the syntax of the .DEF file allows for creating aliases and other strange constructs. Some of the things I tried included:
; Raw function name
RemoveControlByName
; Alias the undecorated name to __stdcall
RemoveControlByName=RemoveControlByName@20
; Explicit reference to the necessary DLL
RemoveControlByName@20 = OCCACHE.RemoveControlByName
However, none of these generated a .LIB that worked. The first two I couldn't link against and the third would link but fail to run.
I thought I'd learn something by looking at the symbol that the parent program is trying to link against, but that was even worse:
__imp_?RemoveControlByName@20
I knew my header file was correct, so that was definitely the correct name. However, you'll notice the leading "__imp", which indicates that the symbol is being imported from a DLL. This was something else that apparently needed to be included in my hand-crafted .LIB file, and I hadn't seen any discussion anywhere on how to do that.
I've tried to solve this problem two other times in the last several years, and this was the point I gave up in both of those cases. However, this time, failure was not an option. I needed the solution.
I tried the second option in the KB article, described under Stubbing Out Functions. I painstakingly created a dozen functions that mimicked the signatures of the functions listed in the header file. If you are doing this for yourself, here's a tip for creating a Visual Studio project. The final file you need is a .LIB file, so it's tempting to use one of the Visual Studio projects for creating a .LIB. However, that's wrong. If you created a .LIB, then you'll link with the stub functions you created, which is useless. What you really want is a DLL project, which happens to create a .LIB as a by-product.
Anyway, I created the functions as described in the KB article. Since my project used .cpp files, all of the calls in my header file had to be declared extern "C". For example:
extern "C" {
#include "occache.h"
};
If you look in any of the standard Windows include files, such as winbase.h, you'll see this same declaration. Note that this declaration has no relation to _cdecl and therefore has no impact on the calling convention. In other words, it doesn't force all of the functions in occache.h to be called with _cdecl. What this declaration does is to modify all of the linker symbols so that they won't include the C++ name decorations, which encode all of the function's parameters into the function name.
I also updated all of the function signatures in the header file to include __declspec(dllexport).
I compiled it, linked my main application to the new .LIB file, and it linked! I thought I was done, but when I ran the application, I received the error "Entry Point Not Found: The procedure entry point _RemoveControlByName@20 could not be located in the dynamic link library OCCACHE.DLL"
I examined the .LIB with DUMPBIN. Under Public Symbols, I now see two definitions for RemoveControlByName:
_RemoveControlByName@20
__imp__RemoveControlByName@20
It appears that the "__imp" definition was a result of adding __declspec(dllexport), so that explained why the application linked successfully. One problem solved.
Continuing my examination of the DUMPBIN information, I saw that RemoveControlByName was defined as:
Archive member name at FE0: OCCACHE.DLL/
46D12949 time/date Sun Aug 26 00:18:33 2007
uid
gid
0 mode
38 size
correct header end
Version : 0
Machine : 14C (x86)
TimeDateStamp: 46D12949 Sun Aug 26 00:18:33 2007
SizeOfData : 00000024
DLL name : OCCACHE.DLL
Symbol name : _RemoveControlByName@20
Type : code
Name type : name
Hint : 9
Name : _RemoveControlByName@20
To determine whether or not this was correct, I used DUMPBIN to compare against known-good definitions in USER32.LIB, where I found that the "Name type" in the USER32 records was defined as "undecorate" instead of "name". Obviously, there was a magic incantation to set this flag, presumably in the .DEF file.
I added a .DEF file and spent several hours trying various alias combinations, none of which worked. Finally, in desperation, I created a .DEF file that contained just the raw function names. For example:
EXPORTS
RemoveControlByName
I built my application, it linked, and it ran. What happened?
I ran DUMPBIN again on my library. The record describing RemoveControlByName now contained the "undecorate" attribute:
Archive member name at FFC: OCCACHE.DLL/
46D12740 time/date Sun Aug 26 00:09:52 2007
uid
gid
0 mode
38 size
correct header end
Version : 0
Machine : 14C (x86)
TimeDateStamp: 46D12740 Sun Aug 26 00:09:52 2007
SizeOfData : 00000024
DLL name : OCCACHE.DLL
Symbol name : _RemoveControlByName@20
Type : code
Name type : undecorate
Hint : 6
Name : RemoveControlByName
Also, the very last line in the record showed that the "Name" was RemoveControlByName, with no decoration. Exactly what I needed.
It's clear that there's quite a bit of undocumented behavior here. Adding the entry to the .DEF file had a rather dramatic effect on the generated library. I couldn't find any mention of this behavior in the documentation on .DEF files. The only relevant reference I could find was in Microsoft's documentation under __cdecl, where it says "Underscore character (_) is prefixed to names, except when exporting __cdecl functions that use C linkage." This statement is true, but it's not the whole truth. To create a .LIB file that can link against such a construct, you also must have the function declared in a .DEF file.
In summary, to create a .LIB file that will let you link with Windows system DLLs, you need to:
- Follow the instructions in Q131313 under Stubbing Out Functions. Make sure you name the project the same as the Windows DLL.
- Make sure your dummy functions are defined with __declspec(dllexport) as well as __stdcall.
- For the header file used by the parent application, make sure that your function declarations are surrounded with extern "C".
- Add a .DEF file to your project that includes the function names with no decoration.
Saturday, August 25, 2007
Media Player Slows Network in Vista
If you've read my earlier blog posts about Gigabit Ethernet, you know that I've had my share of difficulty getting good performance out of my GigE network. I've also posted that I've had problems with Vista that I didn't see in Windows XP or Windows Server 2003. Now I know why. Windows Media Player puts a big throttle on GigE network performance, even if it is paused and not playing anything.
The issue was reported at 2CPU.com and a response from Microsoft was reported at ZDNet. The problem is minor on 10/100 Ethernet, but on a GigE Ethernet the network performance can be throttled back to 100Mps levels. Apparently, there's no registry setting that will resolve the problem, the only solution is to shut down Windows Media Player. There are scattered reports that the problem happens with other media players, such as WinAmp, but I haven't confirmed these reports.
There's another problem I also ran into that caused GigE to throttle back to 100Mbps. This was a self-inflicted problem, but it took several weeks to resolve. I have a LinkSys WRT54G router that handles my gateway/firewall/NAT. All of my servers get their IP address with DHCP and the WRT54G is set to always hand out the same IP address to those servers. This was done so that the servers would get the DHCP settings from our provider. Anyway, one of the servers was migrated to a new motherboard with a different MAC address. The WRT54G started handing out a random DHCP address to that node but still reported that node name as having the other, assigned IP address, all of which made the network become schizophrenic over the IP address assigned to that particular node name. The result was that any traffic that was destined for that node ended up going to the gateway, which didn't have a GigE connection, so traffic was throttled back to 100Mbps speeds.
Update 8/28/2007 - Mark Russinovich has posted a detailed analysis of the network slowdown. It turns out that the more Network Interface Cards (NICs) you have in your system, the worse the problem gets. I have three NICs, including WiFi, which slows my performance to a theoretical maximum of 9MB/second.
The issue was reported at 2CPU.com and a response from Microsoft was reported at ZDNet. The problem is minor on 10/100 Ethernet, but on a GigE Ethernet the network performance can be throttled back to 100Mps levels. Apparently, there's no registry setting that will resolve the problem, the only solution is to shut down Windows Media Player. There are scattered reports that the problem happens with other media players, such as WinAmp, but I haven't confirmed these reports.
There's another problem I also ran into that caused GigE to throttle back to 100Mbps. This was a self-inflicted problem, but it took several weeks to resolve. I have a LinkSys WRT54G router that handles my gateway/firewall/NAT. All of my servers get their IP address with DHCP and the WRT54G is set to always hand out the same IP address to those servers. This was done so that the servers would get the DHCP settings from our provider. Anyway, one of the servers was migrated to a new motherboard with a different MAC address. The WRT54G started handing out a random DHCP address to that node but still reported that node name as having the other, assigned IP address, all of which made the network become schizophrenic over the IP address assigned to that particular node name. The result was that any traffic that was destined for that node ended up going to the gateway, which didn't have a GigE connection, so traffic was throttled back to 100Mbps speeds.
Update 8/28/2007 - Mark Russinovich has posted a detailed analysis of the network slowdown. It turns out that the more Network Interface Cards (NICs) you have in your system, the worse the problem gets. I have three NICs, including WiFi, which slows my performance to a theoretical maximum of 9MB/second.
Subscribe to:
Posts (Atom)