Linux vs Windows: Uptime

If you haven't figured out from this blog, I'm a Windows Guy. But I have a dirty little secret: our corporate Internet presence is a Linux server. Recently I was talking to a Windows IIS developer at a medium-sized company and he was amazed that I would entrust our public face to such an "anarchy of developers." I asked him if he had ever administered a LAMP (Linux/Apache/MySQL/PHP) server and he admitted that he hadn't. I said to him that there were two key reasons that I use Linux: stability and maintainability.

Our corporate Linux server (Redhat Enterprise 3) has been rebooted once in the last five years and that was only because the power grid in the hosting facility was being upgraded. The current uptime count is 1,242 days, or almost three and a half years.

The pedants among you will point me to Netcraft's uptime list to show that Windows Servers can be kept up that long too. However, in my experience that's more than a rarity - it's almost a unique exception that requires Herculean efforts. In contrast, our corporate server was simply automatically updated as patches became available. All available patches have been installed (except kernel patches.) No special efforts were required to keep the Linux server running because of updates.

Much of this is possible because Linux does not have the concept of "in use" files. You'll never get an "Access denied" error because someone else has a file open. In practice, this means that shared libraries can be upgraded on the fly without affecting versions of those libraries that are in use by applications. Unix has allowed the replacement of open files for as long as I've used it, over 25 years.

So as you are sitting there thinking yourself smugly superior for your IIS server, take it from another Windows Guy: when your system absolutely, unquestionably has to stay running, Windows should probably not be your first choice.

You'll notice that the NetCraft list I cited earlier includes no LAMP servers. However, until fairly recently, most variants of Unix/Linux could not report more than 497 days of uptime. Thus the presence on the list of FreeBSD but not other variants. See the NetCraft uptime FAQ.

Postscript: Up until March 2007, the Uptime Project tracked who could keep a computer running the longest without rebooting. The winner was an OpenVMS system that had been running for nearly 12 years. Irish Rail allegedly had an OpenVMS machine up for 18 years. OpenVMS is also used by both the US Postal Service and by Amazon.com. Probably completely uninteresting for most of you, but I used VMS for about five years and it has a warm place in my heart.

XBOX Technical Support

I spend part of every day working with customers, sort of "the buck stops here" technical support. So I have more than a little compassion for people who, day after day, must suffer through clueless customers. But Microsoft's XBOX Technical Support has set a new low.

It all started when my brand new XBOX 360 made grinding noises and ate the demo DVD that came with the system. The problem has been thoroughly documented by the press:
http://www.llamma.com/xbox360/news/Xbox-360-Game-Disc-Scratched.htm

A little one in the house is very attached to one of the games on the DVD and failure was not an option.

Scratched DVD. Easy problem, right? No, not in Microsoft land.

First I try to ask the question online in their support system at support.xbox.com. The only choices are for console hardware failures. There's not even a choice for "Other." So I call the 800 number.

After wading through menus for at least ten minutes, I finally reached a human. It takes half an hour to explain that my DVD is scratched. Her grasp of English is shaky at best. She says I have to send my console back for repair. Arguing is futile because she doesn't speak English well enough. So I get a service repair order.

After I hang up, I go to the web site to obtain the printing label. I try to go to the support registration system, but it won't let me because the "serial number is already registered." Well duh, she just did that on the phone. But she didn't associate the registration with my Live account.

I fill out a support form on support.xbox.com asking what to do. I use the category "Console will not power on" for lack of a better choice. I get a long form letter, starting with "I know how disappointing it is that you're unable to process a repair online. Please accept our apologies for any inconvenience you have felt regarding this unfortunate matter." The email takes eight paragraphs to say "call the 800 number." I tabled the registration problem and just printed the pre-paid shipping label from the web site.

I packaged the system as requested. I taped the DVD to the console with a yellow sticky. I write with my fat red Sharpie pen, "BAD DVD."

A few days later, I get back a new console and a second package with a DVD. Microsoft shipped back the same DVD to me, except they took off the label that said "BAD DVD." There are so many incompetent people involved it's hard to know who to be upset at.

I call back the 800 number again. Every time I ask a question, the support person spends five to ten minutes talking to her supervisor. She wants the serial number for the bad DVD. DVDs don't have serial numbers. She wants some information from the DVD's book. The XBOX Arcade DVD is a demo disc and has no book. It becomes clear that she's apparently never even seen an XBOX 360, much less used one.

After 45 minutes on the phone, she hasn't been able to figure out how to get me a new DVD or how to fix the registration with my Live account. She doesn't even know what a Live account is and tries to get me to use the License Transfer option on the web site. I point out that there's no console associated with my Live account, so I can't have any licenses. She insists that I try anyway, which immediately gives an error message.

She finally gives me an incident number and tells me to call back in an hour. I think that's what's called a "brush off."

I call back two hours later. The new person speaks English quite well and understands it at least better than the last two people. I tell her the problems. She says that she can fix my registration problem by just updating my record. At least she understands the problem and the fix. Her fix doesn't work, but I feel that I've at least obtained validation that I'm asking the right questions.

However, she also gives me the brush off about the DVD. She insists that she can do that for me, but says that "I have to call back tomorrow." At which point she will have gone home and I'll have to start this whole process over with someone else.

So this whole thing should have been one phone call that lasted less than five minutes. Instead, Microsoft's collateral damage is as follows:

• Support reps on the phone for two hours.
  
• Paid to ship console to and from customer.
  
• Paid to replace customer's console (which probably had nothing wrong with it.)
  
• Paid to ship bad DVD back to customer, in a separate envelope.
  
• Public display of complete incompetence.

And the sad thing is, I still don't have a disc that works :-(

Windows 7 Beta on a Thinkpad T42/p

This week I bought an old Thinkpad T42/p off of eBay. IMHO, this system was the pinnacle of the Thinkpad T series, with a great keyboard, a spacious 15" display running 1440x1050, Gigabit Ethernet, and most importantly an ATI Radeon Mobility 9600. This particular system has 1GB of RAM, which I'm told is the minimum required to run Aero. The only real downside to this laptop is that it does not support SATA drives, so you can't put in a 7200rpm SATA drive.

I did a clean install of Windows 7 and I have to say that it runs really well - far better than Windows XP ever did with a similar configuration. Superfetch makes a HUGE difference. I haven't tried to add ReadyBoost yet.

Although the system initially came up with the standard VGA display driver, the Radeon 9600 driver was automatically installed the first time I connected to Windows Update.

The Aero interface did not work at first. I'm not sure exactly what I did to get it working - maybe generating the Windows Experience Index under Control Panel / Performance Information and Tools. I also rebooted a few times. In any case, Aero is now working perfectly.

Here are the numbers reported by the Windows Experience Index:
Processor: 2.0
Memory (RAM): 4.1
Graphics: 2.0
Gaming graphics: 3.6
Hard drive transfer: 2.0

As near as I can tell, all features of the laptop are running properly. There are no missing drivers in the device list. Speedstep works properly. The wireless card works (although I had to choose WPA instead of WPA2 to connect properly to my WRT54G router running DD-WRT.) Display brightness works, volume works, even the ThinkLight. The only thing I installed from Lenovo is the Active Protection for the hard drive.

All in all, I'm quite happy with the T42/p and with Windows 7 experience running on this system.

[Update 5/21/2009] I wiped the system and installed Windows 7 RC today. I confirmed that Aero kicks in AFTER you generate the Windows Experience Index and reboot.

Here are the numbers reported by the Windows Experience Index for the RC. Compared to the Beta, Processor and Hard drive transfer went up and Gaming graphics went down.
Processor: 3.2
Memory (RAM): 4.1
Graphics: 2.0
Gaming graphics: 2.0
Hard drive transfer: 4.3

[Update 10/28/2009] Lenovo has published System Update version 4, which provides a few updates for Windows 7, including the HotKey manager for onscreen feedback of volume, brightness, etc. ActiveProtection is also supported. However, most of the other Thinkpad utilities are still not available for Windows 7.

Downloading symbols for .dmp files

We have an automated system for processing .dmp files received from the field, either our own collection or Microsoft's crash report system. We use cdb (the command line version of WinDbg) to automatically create a text file that contains the stack dump.

The batch file looks like this:

@set _NT_SYMBOL_PATH=SRV*c:\cache*http://msdl.microsoft.com/download/symbols
Cdb -lines -c "!analyze -v;q" -z %1

(Thanks to John Robbins for showing me how to use cdb in this manner.)

In theory, setting _NT_SYMBOL_PATH should provide cdb with enough information to automatically download symbols as needed. However, I wasn't seeing that happening. Without symbols, the debugger can't properly processes callstacks using FPO (Frame Pointer Omission), which means that the callstacks were often missing a lot of information.

Today I found a workaround. The symchk utility will examine a dmp file and verify that all of the required pdb and dbg files have been downloaded. Here is an example:

symchk /id Demo_000000.dmp /s SRV*c:\cache*http://msdl.microsoft.com/download/symbols


List Control (CListCtrl) beeping

For several months now I've been having problems with an MFC application where it would beep every time I changed selections in a CListView. I figured I was doing something wrong, but today I finally decided to track it down.

Thanks to VistaHeads for the answer.

Run Regedit and then delete the default value for
HKEY_CURRENT_USER\AppEvents\Schemes\Apps\.Default\ CCSelect\.current

Note that the offending value is blank, so it looks like you aren't doing anything by deleting the value, but go ahead and delete it anyway. (Delete the value, not the key.)

The other amusing thing I learned about this is that a worker thread is created to play the beep sound and the worker thread is created with the priority Time Critical. There are a lot of things that might be time critical, but beeping on a List View change is certainly not one of them. (Yeah, I know, interrupts, blah blah, buffer management, blah, blah. It's still silly.)

Unicode BOM Handling in the C Run-time Library

Visual Studio 2005 and later include Unicode BOM (Byte-Order Mark) support, but I found the documentation somewhat lacking. Here are a few hints.

One of the primary points of confusion for me was what you were defining by setting the encoding. The answer is that you are providing a hint as to the encoding of the file (but only a hint. If the file has a BOM, then that BOM takes precedence.)

All calls you make to read or write the file must be with Unicode APIs. If you try to use an ANSI API, the C-Runtime library (CRT) will assert. This means that the CRT will do character set conversion between Unicode and the file's encoding, but won't do character set conversion between the local code page and the file's encoding. For example, you'll get an assertion if you open the file with ccs=utf-8 and then try to use fgets to read the data.

Other points:

• The CRT will not perform any BOM handling if you do not specify a ccs= encoding. This means that backwards compatibility is retained because the CRT does not perform any behind-the-scenes processing if you don't ask it to.
  
• Most BOM formats are not supported. For example, UTF-7, UTF-32 and especially UCS-16 big-endian are not handled.
  
• If you specify a specify a ccs= encoding, then the BOM will be automatically removed from the data stream. However, you need to be careful of file positioning calls such as fseek and rewind because the bom will only be skipped when the file is first opened. For example, if you do fopen, fread, rewind, fread, then the second fread will read the BOM and the first fread will not.
  
• The file encoding is respected when writing, so the number of characters actually written may be lesser or greater than the buffer size you wrote.
  
• If you open the file in binary mode, then any ccs= specification will be ignored and no BOM handling will be performed.
  
• Apparently the CRT does not provide a documented way to determine the encoding of the file.
  

How Not To Run A Web Site

Last night, New Years Eve, I received the following error from Evite:

Sorry! Due to planned maintenance, certain pages may be unavailable. We should be back up within 30 minutes, so please check back shortly.

Thanks for your patience!

As it turned out, "certain pages" included both the home page and the party I was attending.

What kind of complete idiot schedules "maintenance" on the biggest party night of the year for a web site that manages party invitations?

I could understand system overload, but scheduled maintenance?!?

So, with just six hours left in the year, Evite easily won my award for Best Example of How Not To Run A Web Site.