I've had an unbelievably long run of hardware failures this year. First a Hitachi 500GB hard drive failed in my Windows 2003 Server, then the Apacer USB thumb drive I was using for ReadyBoost, and now my new GigE Ethernet switch, a Netgear GS108, is malfunctioning. Predictably, all of the failures have been after the 30 day mark, which means that I've had to go through an RMA process instead of a simple return.
The Netgear GS108 is particularly frustrating because it didn't do something simple like emit smoke and die. That would have been easy to diagnose. Instead it would alternate between a blinking failure mode and normal operation. In twenty years I've never seen an Ethernet switch or hub fail, so I wasted two days looking at everything except the switch.
The symptom is easy to see - all of the lights turn on for a second, then they all turn off, even on ports with nothing plugged in. Any PCs that are connected report that a network cable is unplugged. This may go on for a few seconds to a couple of minutes, then it will start working again, then it will fail again. The problem seemed to be most prevalent when 1000Mbps devices were mixed with 100Mbps devices. If all devices are 1000 or all devices are 100, then the GS108 worked fine.
Reviews on newegg.com show that a minority of other users have experienced the same problem I did.
I talked to Technical Support at Netgear. They immediately issued an RMA for the switch. Unfortunately, after waiting a week for the new GS108 to arrive, the new unit malfunctioned in the same way as the old unit. So I believe this problem is endemic to the GS108.
I've replaced the GS108 with a Dell PowerConnect 2708. It costs about 50% more than the GS108, but it is a fully managed switch that receives rave reviews from users. The Dell 2708 also includes a one year next-day parts warranty, as opposed to Netgear who charges you $50 for a next day exchange. You might as well order a new one from NewEgg.
I plugged in the Dell 2708 and it has worked flawlessly. I'm just running it in unmanaged mode since I have no need for VLANs or port aggregation.
The PowerConnect 2708 lists on Dell's web site for $158 if you just do a Google search, but that's the Medium/Large Business price. If you look in the Small Business section, the price is $109 with a $20 discount, or $89. Great deal:
http://www.dell.com/content/products/productdetails.aspx/pwcnt_2708?c=us&cs=04&l=en&s=bsd
Wednesday, May 30, 2007
Wednesday, April 25, 2007
Gigabit Ethernet
Backing up a 250GB hard drive over 100Mbit Ethernet takes a l..o..n..g time. At 10 megabytes per second (theoretically), that's 600 megabytes per minute, or about eight hours. Compression can cut this number down, but the backup rate is severely constrained by the speed of the link.
In contrast, backing up to an external hard drive over Firewire or USB2 yields about 30MB/second, which brings it down to a manageable 2 1/2 hours.
I decided to take the plunge to Gigabit Ethernet (GigE), something I've been wanting to do for quite a while but was waiting for prices to drop. I brought in a contractor and upgraded my eight year old Cat5 wiring to Cat6. Strictly speaking, GigE is supposed to run over Cat5 for short distances, but I was having difficulties and it wasn't worth taking the time to individually check each wire.
I've learned several things. First, the "jumbo frames" that everyone gets so excited about are useless on a general purpose network. If you turn on jumbo frames, then every device on the network must support them and be set to use the same jumbo frame size. Since devices like printers and Tivos don't support GigE, much less jumbo frames, it means that jumbo frames are a non-starter. The place that jumbo frames *can* be useful is for a dedicated connection between servers or a backbone link.
The next thing I learned is that today's network application protocols don't scale well to GigE. When you upgrade from 10Mb to 100Mb Ethernet, your performance typically jumps almost 10x. However, when you go from 100Mb to 1000Gb, your performance typically only jumps 2.5x, even if you've optimized your system configuration. Ugh. That's not what I was hoping for.
The primary optimization required for Microsoft Windows 2000 and XP is to update the receive window size in the registry. This change isn't required for Windows Vista because Vista automatically tunes its networking parameters for optimal performance.
You can see what the practical maximum speed of your link is using pcattcp.exe. You run it on a server and on a client and it reports the speed at which it was able to send data. Most importantly, you can set the write buffer size, which allows you to determine the effect of larger buffers. In my case I determined that a 64K buffer provides throughput at 90% of the link capacity.
I still have more research to do, but that's the beginning.
[Update 6/20/2007] After spending far too many hours testing various GigE configurations, I've determined that the original Cat5 wiring was probably working fine. The problems I was having were a combination of the malfunctioning GS108 and a configuration error in my router/switch.
There's a great article at Hardware Secrets that describes how Gigabit Ethernet works and why Cat5 cables are sufficient.
In contrast, backing up to an external hard drive over Firewire or USB2 yields about 30MB/second, which brings it down to a manageable 2 1/2 hours.
I decided to take the plunge to Gigabit Ethernet (GigE), something I've been wanting to do for quite a while but was waiting for prices to drop. I brought in a contractor and upgraded my eight year old Cat5 wiring to Cat6. Strictly speaking, GigE is supposed to run over Cat5 for short distances, but I was having difficulties and it wasn't worth taking the time to individually check each wire.
I've learned several things. First, the "jumbo frames" that everyone gets so excited about are useless on a general purpose network. If you turn on jumbo frames, then every device on the network must support them and be set to use the same jumbo frame size. Since devices like printers and Tivos don't support GigE, much less jumbo frames, it means that jumbo frames are a non-starter. The place that jumbo frames *can* be useful is for a dedicated connection between servers or a backbone link.
The next thing I learned is that today's network application protocols don't scale well to GigE. When you upgrade from 10Mb to 100Mb Ethernet, your performance typically jumps almost 10x. However, when you go from 100Mb to 1000Gb, your performance typically only jumps 2.5x, even if you've optimized your system configuration. Ugh. That's not what I was hoping for.
The primary optimization required for Microsoft Windows 2000 and XP is to update the receive window size in the registry. This change isn't required for Windows Vista because Vista automatically tunes its networking parameters for optimal performance.
You can see what the practical maximum speed of your link is using pcattcp.exe. You run it on a server and on a client and it reports the speed at which it was able to send data. Most importantly, you can set the write buffer size, which allows you to determine the effect of larger buffers. In my case I determined that a 64K buffer provides throughput at 90% of the link capacity.
I still have more research to do, but that's the beginning.
[Update 6/20/2007] After spending far too many hours testing various GigE configurations, I've determined that the original Cat5 wiring was probably working fine. The problems I was having were a combination of the malfunctioning GS108 and a configuration error in my router/switch.
There's a great article at Hardware Secrets that describes how Gigabit Ethernet works and why Cat5 cables are sufficient.
Sunday, April 22, 2007
ReadyBoost Part IV
I finally ordered a 4GB Apacer HT203 and installed it. I also installed the Vista USB patch. The hang problem is now fixed (see my earlier blog entries about ReadyBoost.) I can browse the thumb drive after resuming from sleep without difficulty.
However, ReadyBoost still doesn't work reliably after resuming from Sleep. After the computer wakes up, ReadyBoost works for a while, then mysteriously turns off (as seen by opening the Properties for the drive and looking at the ReadyBoost tab.) If I reenable ReadyBoost, it just turns off again almost immediately.
At this point, I'm only slightly better off than I was before I applied the Vista USB fix. The computer no longer hangs, but ReadyBoost still becomes disabled after resuming from sleep. Doing a "Safe Remove" and then reinserting the thumb drive, even into another port, doesn't help.
However, ReadyBoost still doesn't work reliably after resuming from Sleep. After the computer wakes up, ReadyBoost works for a while, then mysteriously turns off (as seen by opening the Properties for the drive and looking at the ReadyBoost tab.) If I reenable ReadyBoost, it just turns off again almost immediately.
At this point, I'm only slightly better off than I was before I applied the Vista USB fix. The computer no longer hangs, but ReadyBoost still becomes disabled after resuming from sleep. Doing a "Safe Remove" and then reinserting the thumb drive, even into another port, doesn't help.
Saturday, April 21, 2007
Symbols for CRT and MFC Source Code
[12/19/2014: Updated to include Visual Studio 2013.]
I'm not sure why I keep getting stuck on the littlest things. Ever since I installed Visual Studio 2010, I haven't been able to debug into the C Runtime (CRT) or the MFC source code. This is an operation that never gave me any difficulties in years of using VC6, so it's been quite frustrating. I've made several attempts to fix the problem, but no success until today.
Reproducing the problem is straightforward. Set a breakpoint in one of your MFC message handler functions, go to the Call Stack window, and try to click on any of the functions in mfc80d.dll. Similarly, any attempt to set a breakpoint on an MFC function (such as CWinApp::CWinApp) simply fails to work.
'MyApp.exe': Loaded 'C:\WINDOWS\WinSxS\x86_microsoft.vc80.debugmfc_1fc8b3b9a1e18e3b_8.0.50727.762_none_29a8a38855141f6e\mfc80d.dll', Symbols loaded (source information stripped).
The key part is "source information stripped", which is strange, because the PDB files for MFC ship with Visual Studio (and the service pack) and those PDB files include source information.
I searched my system for mfc80d.pdb and I found two copies. The first copy is in the symbol cache at c:\cache\mfc80d.i386.pdb\A12C75C3E6A244E3B3BFBE577AB27642e. This file was about 2MB. The second copy of mfc80d.pdb was in c:\windows\symbols\dll. This file was 13.5MB, significantly larger and probably the version of the PDB file that I needed. The problem was making VC2005 use it. That's the hard part.
First go to Tools/Options, open Debugging, and select Symbols. Add the directory "C:\Windows\Symbols\dll" to the beginning of the list. This is different than how VC6 was configured, which required that you not include the "dll" part of the path.
Click OK. Now close Visual Studio. Go to Task Manager and make sure its gone. Now kill the process MSPDBSRV.EXE, if it's running.
Finally, the critical operation to making all of the other steps work is to remove the smaller copy of mfc80d.pdb from the cache. To do this, delete the directory c:\cache\mfc80d.i386.pdb. The path may be slightly different if you are on a 64-bit system.
At this point you can restart VC2005, load your project, and run your application. In the Modules window, you should see that the symbol file for mfc80d.dll is being loaded from C:\Windows\Symbols\dll.
I'm not sure why I keep getting stuck on the littlest things. Ever since I installed Visual Studio 2010, I haven't been able to debug into the C Runtime (CRT) or the MFC source code. This is an operation that never gave me any difficulties in years of using VC6, so it's been quite frustrating. I've made several attempts to fix the problem, but no success until today.
Reproducing the problem is straightforward. Set a breakpoint in one of your MFC message handler functions, go to the Call Stack window, and try to click on any of the functions in mfc80d.dll. Similarly, any attempt to set a breakpoint on an MFC function (such as CWinApp::CWinApp) simply fails to work.
Part 1
If you look at the Output window, you'll see a message such as this one:'MyApp.exe': Loaded 'C:\WINDOWS\WinSxS\x86_microsoft.vc80.debugmfc_1fc8b3b9a1e18e3b_8.0.50727.762_none_29a8a38855141f6e\mfc80d.dll', Symbols loaded (source information stripped).
The key part is "source information stripped", which is strange, because the PDB files for MFC ship with Visual Studio (and the service pack) and those PDB files include source information.
I searched my system for mfc80d.pdb and I found two copies. The first copy is in the symbol cache at c:\cache\mfc80d.i386.pdb\A12C75C3E6A244E3B3BFBE577AB27642e. This file was about 2MB. The second copy of mfc80d.pdb was in c:\windows\symbols\dll. This file was 13.5MB, significantly larger and probably the version of the PDB file that I needed. The problem was making VC2005 use it. That's the hard part.
First go to Tools/Options, open Debugging, and select Symbols. Add the directory "C:\Windows\Symbols\dll" to the beginning of the list. This is different than how VC6 was configured, which required that you not include the "dll" part of the path.
Click OK. Now close Visual Studio. Go to Task Manager and make sure its gone. Now kill the process MSPDBSRV.EXE, if it's running.
Finally, the critical operation to making all of the other steps work is to remove the smaller copy of mfc80d.pdb from the cache. To do this, delete the directory c:\cache\mfc80d.i386.pdb. The path may be slightly different if you are on a 64-bit system.
At this point you can restart VC2005, load your project, and run your application. In the Modules window, you should see that the symbol file for mfc80d.dll is being loaded from C:\Windows\Symbols\dll.
Part 2
[This applies to Visual Studio 2010, 2012, and 2013.]
You may need to tell Visual Studio where the source code is installed. Right-click on a C++ solution and select Properties. Under Common Properties, select Debug Source Files. For Visual Studio 2013, it looks like this on my machine:
Problem solved.
Tuesday, March 13, 2007
Solution to Vista USB ReadyBoost Hang
Microsoft has released a patch for Vista that cures several problems with USB devices, including devices that hang when resuming from Sleep. Here's the solution:
Reliability update for the USB stack in Windows Vista
http://support.microsoft.com/kb/925528/en-us
Finding this patch isn't particularly easy if Google is your only search engine. The knowledgebase article doesn't mention ReadyBoost at all. It comes up as the first result if you search support.microsoft.com for vista and usb, but it's on page three if you search Google for vista usb hang.
My Apacer USB thumb drive is dead, so I can't test out the patch until I order a new one.
[Update 4/25/2007]
The Microsoft Knowledgebase article has been retitled Stop errors occur on a Windows-based computer that has 2GB or more of RAM and is using an NVIDIA nForce USB controller. My computer isn't based on the nVidia chipset, but the update still fixed my hang problem, so I'm not sure why Microsoft retitled the article. I do have 2GB of RAM.
Reliability update for the USB stack in Windows Vista
http://support.microsoft.com/kb/925528/en-us
Finding this patch isn't particularly easy if Google is your only search engine. The knowledgebase article doesn't mention ReadyBoost at all. It comes up as the first result if you search support.microsoft.com for vista and usb, but it's on page three if you search Google for vista usb hang.
My Apacer USB thumb drive is dead, so I can't test out the patch until I order a new one.
[Update 4/25/2007]
The Microsoft Knowledgebase article has been retitled Stop errors occur on a Windows-based computer that has 2GB or more of RAM and is using an NVIDIA nForce USB controller. My computer isn't based on the nVidia chipset, but the update still fixed my hang problem, so I'm not sure why Microsoft retitled the article. I do have 2GB of RAM.
Making VC2005 Remote Debugging Work
I use Virtual PC extensively for testing older operating systems and non-standard configurations. When I have to debug configuration-related problems, I use Remote Debugging to connect to the Virtual PC instance and debug in-situ.
Before I upgraded to VC2005, remote debugging with VC++ 6 was straightforward. I updated the PATH on the remote computer to include MSVCMON.EXE, the related debugger DLLs, and the debug versions of the C runtime library and MFC. Worked great.
Remote debugging with VC2005 is more challenging. Between security problems and Windows Side-by-Side assembles, it can take quite a few steps to make everything work. Here is what I had to do.
Your debugger and Windows firewall double-team you to make configuring remote debugging as painful as possible. My first problem was to fix the firewall, since I couldn't connect to the Windows XP SP2 computer at all. I fixed this by disabling the Windows Firewall (if you are following along, don't do this unless you are absolutely certain it's okay for your environment!) Probably overkill, but simple.
My next problem was debugger authentication. This proved to be a challenge because my Virtual PC test machine is on a domain and my development machine is not. After several hours of effort, I determined that they simply were not going to agree on a trust relationship, so I disabled authentication. (Handy tip - you can do this from the command line with /noauth. Also add /nosecuritywarn if you don't want to go crazy with security prompts.)
I thought I was golden at this point. I clicked F5 to debug my executable and was told, "Could not start App.exe". So I tried to start the application from Explorer. Same result. I ran Dependency Walker to check my DLL dependencies and was reminded that I needed to update my path to include the VC2005 debug libraries for the CRT and MFC. I did so, clicked F5 again, and... it still didn't work.
Several hours and much frustration later, I had received a solid education about Windows Side-by-Side Assemblies, otherwise known as WinSxS. In order to cure "DLL Hell," Windows XP and Windows Vista provide support for an executable's manifest to hardcode the desired version of dependent DLLs. For executables created with VC2005 that are dynamically linked to MFC and the CRT, a manifest is automatically inserted into the generated executable that includes this version information.
Once this is done, Windows XP and Windows Vista become absolutely hell bent on loading the correct version of the DLLs. Trying to work around this protection is futile. None of the following "obvious" solutions work:
\Program Files\Microsoft Visual Studio 8\VC\redist\Debug_NonRedist\x86
Inside this directory, you'll find two more directories named Microsoft.VC80.DebugCRT and Microsoft.VC80.DebugMFC. These directories must be copied to the same directory as your EXE and magically everything works right.
At this point you'd think the story is over, but it's not. Windows 2000, Windows NT and Win9x don't support WinSxS, so the manifest is ignored. To debug your application any of THOSE versions of Windows, you need to include the appropriate DLLs on your PATH (or copy the appopriate DLLs into the same directory as your EXE.) I set the path in Virtual PC to point directly to the WinSxS directories on my development machine, which were: ("duo" is the host name of the development computer, which drive C shared under the name "C")
-
Finally, I used the following command in my batch file to start the debug monitor:
start "" "\\duo\c\Program Files\Microsoft Visual Studio 8\Common7\IDE\Remote Debugger\x86\msvsmon.exe" /noauth /anyuser /nosecuritywarn /timeout 7200
Finally, that made everything work properly.
Before I upgraded to VC2005, remote debugging with VC++ 6 was straightforward. I updated the PATH on the remote computer to include MSVCMON.EXE, the related debugger DLLs, and the debug versions of the C runtime library and MFC. Worked great.
Remote debugging with VC2005 is more challenging. Between security problems and Windows Side-by-Side assembles, it can take quite a few steps to make everything work. Here is what I had to do.
Your debugger and Windows firewall double-team you to make configuring remote debugging as painful as possible. My first problem was to fix the firewall, since I couldn't connect to the Windows XP SP2 computer at all. I fixed this by disabling the Windows Firewall (if you are following along, don't do this unless you are absolutely certain it's okay for your environment!) Probably overkill, but simple.
My next problem was debugger authentication. This proved to be a challenge because my Virtual PC test machine is on a domain and my development machine is not. After several hours of effort, I determined that they simply were not going to agree on a trust relationship, so I disabled authentication. (Handy tip - you can do this from the command line with /noauth. Also add /nosecuritywarn if you don't want to go crazy with security prompts.)
I thought I was golden at this point. I clicked F5 to debug my executable and was told, "Could not start App.exe". So I tried to start the application from Explorer. Same result. I ran Dependency Walker to check my DLL dependencies and was reminded that I needed to update my path to include the VC2005 debug libraries for the CRT and MFC. I did so, clicked F5 again, and... it still didn't work.
Several hours and much frustration later, I had received a solid education about Windows Side-by-Side Assemblies, otherwise known as WinSxS. In order to cure "DLL Hell," Windows XP and Windows Vista provide support for an executable's manifest to hardcode the desired version of dependent DLLs. For executables created with VC2005 that are dynamically linked to MFC and the CRT, a manifest is automatically inserted into the generated executable that includes this version information.
Once this is done, Windows XP and Windows Vista become absolutely hell bent on loading the correct version of the DLLs. Trying to work around this protection is futile. None of the following "obvious" solutions work:
- Putting mfc80d.dll in the same directory as the EXE, even if it's the correct version.
- Including mfc80d.dll on the user's path.
- Manually copying the appropriate directories from the \Windows\WinSxS on the development machine to \Windows\WinSxS on the remote debuggee.
\Program Files\Microsoft Visual Studio 8\VC\redist\Debug_NonRedist\x86
Inside this directory, you'll find two more directories named Microsoft.VC80.DebugCRT and Microsoft.VC80.DebugMFC. These directories must be copied to the same directory as your EXE and magically everything works right.
At this point you'd think the story is over, but it's not. Windows 2000, Windows NT and Win9x don't support WinSxS, so the manifest is ignored. To debug your application any of THOSE versions of Windows, you need to include the appropriate DLLs on your PATH (or copy the appopriate DLLs into the same directory as your EXE.) I set the path in Virtual PC to point directly to the WinSxS directories on my development machine, which were: ("duo" is the host name of the development computer, which drive C shared under the name "C")
\\duo\c\Windows\winsxs\x86_microsoft.vc80.debugmfc_1fc8b3b9a1e18e3b_8.0.50727.762_none_29a8a38855141f6e
\\duo\C\windows\winsxs\x86_microsoft.vc80.debugcrt_1fc8b3b9a1e18e3b_8.0.50727.762_none_24c8a196583ff03b
-
Finally, I used the following command in my batch file to start the debug monitor:
start "" "\\duo\c\Program Files\Microsoft Visual Studio 8\Common7\IDE\Remote Debugger\x86\msvsmon.exe" /noauth /anyuser /nosecuritywarn /timeout 7200
Finally, that made everything work properly.
Tuesday, March 6, 2007
Visual Studio 2005 SP1 Update for Windows Vista Released
A little bit later than I really needed it, but Visual Studio 2005 Service Pack 1 Update for Windows Vista has been released. Get it while it's hot!
http://www.microsoft.com/downloads/details.aspx?FamilyID=90e2942d-3ad1-4873-a2ee-4acc0aace5b6&DisplayLang=en
http://www.microsoft.com/downloads/details.aspx?FamilyID=90e2942d-3ad1-4873-a2ee-4acc0aace5b6&DisplayLang=en
Subscribe to:
Posts (Atom)