How I Backup My Windows System for Multiple Contingencies

In my last blog posting, I described the events that have lead up to my paranoia about keeping my computer backed up. In this posting I'll talk about about how I actually keep my system backed up and the challenges involved in these methods.

These backup strategies are appropriate for individuals and for small offices. For larger environments, the reasons for doing various types of backups still hold true, but the implementation strategy will change significantly.

1. DVD Backup. My first strategy is mundane - my software development directories are backed up to DVD after every release. This is more complicated than it sounds. First, Visual Studio creates hundreds of megabytes of temporary files, including .ncb, .ilk, obj, and .pch files. There generally no point in backing up these files. In Roxio Creator I can exclude all of these file types, which makes it trivial to just drag the root directory of my development structure to the DVD. Other DVD burning applications, such as CDBurnerXP, won't automatically exclude file types, so it's somewhat more complex to use them for backups. [Update 12/21/2010: CDBurnerXP now supports excluding by extension, under Edit/Filter Files. However, you can't use this filter from the command line.]

I've been careful to use high quality CDs and DVDs for these backups. I have CDs I burned over ten years ago that I can still read without difficulty. My recordable DVDs are newer, so the jury is still out on how long they'll last. The important thing to remember about CD-R and DVD-R is that, unless you buy archive-grade media, they shouldn't be expected to last more than a few years.

DVD and CD backup requires a substantial amount of effort. You have to remember to do it, it's relatively time consumering (especially if you validate the media after burning), and you have to keep all of the media labeled and filed. So it's only a good streategy if you are quite disciplined. However, DVD backups are much easier to move offsite and you have a permanent record, especially if you need an audit trail.

2. Image Backup. Image backups are the cornerstone of my backup strategy. If my computer total fails, due to a virus, a lightning strike, or user error, the image backups are what allow me to recover in a few hours instead of a few days.

For the last eight years I've created my image backups with Acronis TrueImage. This has been a mixed blessing. On the plus side, Acronis can do incremental image backups, which saves a lot of disk space. Acronis has a standalone boot CD, so you can recover the system from a raw hard drive. Acronis can do backups over the network, even when booted from the recovery disk. All of these have been invaluable.

On the minus side, Acronis has had significant difficulty with consistent quality. Their latest release, Version 11, is a total disaster and has been completely unusable for many people. Acronis also has a nasty habit of creating corrupted backups, which means that a validation phase is required.

I do image backups once a week, supplemented by file backups that are done daily. I try to keep three months of image backups because some viruses can take weeks to trigger.

[Update 12/21/2010: Because of ongoing stability problems with Acronis, I have stopped using it.]

3. File Backup. Acronis TrueImage can also do file backups. I backup my development directories and documents on a daily basis. This usually comes out to less than 5-GB, so I can store a lot of them on a 1-TB backup disk. Again, file backups are done incrementally, so the baseline file is dramatically larger than the daily incremental files.

My big problem with Acronis is the poor support for rotation management. I want to keep backup snapshots every 30 days for the last six months, every week for the past month, and every day for the past week. Acronis provides several different options, none of which can do what I want, and all of which seem to cause the frequency of corrupted backups to increase dramatically. [Update 12/21/2010: Macrium Reflect also has better support for managing archive rotation.]

File backups provide protection from accidental deletion and provide the ability to revert to an older version of a document.

4. RAID 1 Mirror drives I mentioned this in my last blog post, but it bears repeating. I use mirrored drives to handle the case of catastrophic hard drive failure, which has happened to me multiple times before. One of the two mirrored hard drives can fail completely and it won't affect the performance of the system. [Update 5/30/2010 - This happened to me again today. A hard drive failed in my primary development system. Without the RAID 1 pair I'd be looking at restoring from backup right now.]

I use software RAID (Intel Matrix Storage), which definitely has its downsides. Every time you reboot without cleanly shutting down the system, (such as a power fialure or hitting the Reset button,) the drive mirrors have to be validated. This takes six to ten hours, during which the system is almost unusable.

Hardware RAID solutions with battery backup can fix this problem, but such RAID hardware is dramatically more expensive and generally is not compatible with image backup software such as Acronis.

Note that not all RAID strategies provides protection. RAID 0 (striping) actually DECREASES reliability, because if any drive fails, your data will be completely lost.

5. Online backup. My final strategy is for last ditch disaster recovery - theft, fire, natural disaster. I'll talk about it in the third and final part of my blogs about backup.

Six Myths About Backing Up Your Computer

This is my tale about how I got religion about backups. I played a part in all of the stories below. The amazing part is not that any one or two of them happened around me - the amazing part is that they all happened around me.

Myth #1. I have a new hard drive, my data is safe.

Five years ago I helped a friend replace his hard drive after his original drive failed and he lost everything. So he bought the new drive, spent two or three days reloading everything, and went on his way. Three months later, the new drive failed too, and again he lost everything. My friend is an optimist. He believes he's had all the bad luck he's going to have and he still doesn't do backups.

A few years later, I was overseeing putting a new server in the office and had paid a consultant a lot of money to set it up to our satisfaction. The server was backed up after the OS was installed, but before the consultant made changes. After the consultant was finished, we went home for the weekend. When we came in on Monday, the hard disk had failed. All of the money we paid for the consultant was wasted. Even more embarrassing, the CD for the backup software was sitting on top of the server.

The Reality: Hard drives have their highest failure rate when they are new.

Myth #2. Backing up once or twice a month is enough.

My neighbor conscientiously made weekly backups of his laptop. He had decided that losing a few days of work wouldn't be a big deal. Inevitably, his hard drive died hours before a sizable document was due to his largest customer. He had a four day old backup, but he'd done well over thirty hours of work on the document in the interim. I was able to recover the data with low-level tools, but recovery companies charge hundreds to thousands of dollars for that kind of work.

The Reality: Murphy's Law should never be underestimated. Nightly backups are a minimum, and software for continuous backup is easily available.

Myth #3. Backing up documents to an external drive is good enough.

Two years after I helped my optimist friend above, the primary hard drive abruptly failed in our corporate ecommerce web server. We had a backup of the database and other data, but we still had to reconfigure the operating system from scratch. It took four days to get the server running again, which was basically four days of lost sales. I learned a harsh lesson, since it cost thousands of dollars in sales and made a lot of existing customers unappy.

The Reality: To quickly recover from a catastrophic failure, you must have image backups, not data backups.

Myth #4. Having an image backup on an external drive or a NAS is good enough.

I have now been in two different offices that were burglarized and the computers were stolen. If I'd had a USB hard drive sitting on the shelf, it would have been taken too. Even if you put the backup in the safe, it just takes one flood, fire, tornado, or hurricane to destroy the sensitive platters of a hard drive.

I've also known people who whose office was served with a search warrant. The content on computers is just assumed to be relevant, so officers often take *everything* that looks like a computer or a hard drive - including what's in the safe. If you don't have off-site backups, you're done because you may not ever get that equipment back.

The Reality: You must have off-site backups.

Myth #5. I use RAID hard drives so I don't have to do backups.

My primary computer had mirrored hard drives installed right from the start. Six months later, my computer contracted the second virus I've ever had. However, RAID is protection against disk failure, not against data loss that was intentionally caused by a virus. I recovered from the failure by reloading the entire system from image-level backups, as I described in #3.

As I became increasingly paranoid about hard drives failures, I upgraded our corporate web server with a hardware RAID card and enterprise-grade SCSI hard drives to allow RAID 1 drive mirroring. SCSI hard drives generally cost several times the price of a similarly-sized IDE (consumer) drive because they are expected to be substantially more reliable.

A week after the new hardware went live, I started seeing I/O errors. Our expensive new RAID controller board had malfunctioned. As a result, it corrupted the SQL database that contained the customer support system, which contained FIVE YEARS of customer support information, including an extensive knowledgebase. The primary backup was completely missing due to a configuration error in the backup script. I ended up staying up all night to restore the database from a third-level backup that I'd been paranoid enough to perform.

The Reality: If you don't have 100% redundancy, including spare computers ready to go, downtime and possible data loss are just one hardware failure away. Also, RAID provides zero protection again viruses and user error.

Myth #6. Paying experts to do backups is good enough.

After we upgraded to RAID hard drives, we purchased Managed Backup for our corporate server. This means that experts are responsible for keeping our server backed up, and they are responsible for recovery when things go wrong.

Last summer one of our web server log files was erased accidentally. These files are critical for financial analysis and prediction, so I was pleased that we had the experts managing our backup. Except for one thing. The "experts" considered the log files low value and a waste of space, so the file I needed had not been backed up. Damn.

The Reality: Backups that haven't been verified and tested aren't backups.

Conclusion

You can't be too paranoid about backup. I have four levels of redundancy in my backups, but I still worry that it won't be good enough. If your computer contains critical information, whether it's corporate documents, baby pictures, legal documents, graduate thesis, or financial records, do yourself a favor and back it up, preferably in multiple places.

In Part 2, I'll look at how I've implemented my tiered backup strategy and what software I use to make it happen.

Installing PHP on Windows 7

If you've wrestled with installing PHP on IIS before, there's good news. Installing PHP in Windows 7 is easy. Super easy. Microsoft has created the Web Platform Installer (WPI), which supports installing many things, PHP included.

If you look in the IIS Manager, down at the bottom of the rows of icons you should see Web Platform Installer. If you don't see it there, go to your Start menu and search for "platform".

Check off PHP and let it install. There are a couple of dependencies that will be added automatically.

There's one more change you need to make after PHP is installed, which is to add "index.php" to the list of Default Documents.

There are some other things you should know:

• WPI correctly installs PHP even on a 64-bit system. There's no need to use cscript to separately change configurations.
  
• Apparently, there's still no 64-bit version of PHP for Windows. WPI installs the 32-bit version, which works correctly even under the 64-bit version of IIS.
  
• There's no uninstall option. However, everything is installed to a single directory and removing the relevant FastCGI settings is straightforward.
  
• The correct strategy for upgrading to new versions of PHP isn't obvious. I'm not sure if Microsoft used the stock build of PHP or if they created a custom one.
  

One small gotcha - WPI installs PHP 5. If this is your first encounter with PHP 5, you need to make sure you use <?php ... ?> as opposed to <? ... ?>.

Handling Exceptions from STL

Recently I tracked down a crash to an out of bounds index in a STL vector. The strange thing was that the crash was being handled by Windows instead of by our error reporting code, which is designed to trap everything that the explicit exception handlers did not catch.

Obviously it wasn't.

I created some short test code:
vector vec;
vec[5] = 3;

When run under the debugger, I receive a helpful window that says "vector out of range" with the standard Abort/Retry/Ignore debugger options. All very well and good, but there was no exception being thrown.

I tried using set_unexpected(), which turns out didn't mean what I thought it meant. The function set_unexpected() is called when an exception is thrown through a function that doesn't declare that exception in the throw() statement in the function declaration. Visual C++ does not support these declarations, and so does not support set_unexpected(). This is discussed in the documentation.

So I tried set_terminate(), which is the only other function under Exception Handling Routines that seemed relevant. Unfortunately, my termination function was never called. Since my two line program certainly was not catching an exceptions, something else was happening.

I tried running the code again as a Release build instead of a Debug build. It still broke to the debugger, but in a separate place than the Debug build.

I tried reading the documentation, which makes no mention that operator[] can throw an exception.

Stranger and stranger. I tried walking the stack with the Release build, which looked like this:
msvcr90.dll!_crt_debugger_hook
msvcr90.dll!_invalid_parameter
msvcr90.dll!_invalid_parameter_noinfo
Exc.exe!wmain

The function _invalid_parameter() had this comment:
// No user handler is defined. Notify the debugger if attached.
So I tried running the Release build without the debugger. Which also yielded no useful information.

I don't remember exactly how I figured out what was happening, but it turns out that STL is throwing "out_of_range", which is derived from "logic_error", which is derived from class "exception". I've been using STL for twelve years. I've written articles about it. And I've never heard of these exception classes. Clearly, the authors of the documentation had never heard of them either.

Note that this behavior is different than other implementations of STL, which specifically document that only the at() functions throws an exception, not operator[]. At least "out_of_range" is defined in the standard.

[Update 8/13/2010 Visual Studio 2010 behaves according to the standard. In Release builds, vector<>::operator[] does not do bounds checking and does not throw the out_range_range exception. In Debug builds, an assertion is shown because _ITERATOR_DEBUG_LEVEL is set to 2 in yyvals.h if _DEBUG is defined.]

But the fun wasn't over. I still didn't know why set_terminate() wasn't working. The documentation for Unhandled C++ Exceptions only discusses set_terminate().

It turns out that unhandled logic_error exceptions are handled by invalid_parameter_handler and not by terminate_handler. This is broadly discussed in the documentation for Invalid Parameter Handler Routine. However, that documentation describes how errno is set, which doesn't appear to happen with STL.

The situation gets even more murky if you consider Windows SEH (Structured Exception Handling) as well as C++ exceptions. The standard way of dealing with SEH is to use SetUnhandledExceptionFilter(). However, Visual Studio 2008 changes this behavior. See a blog entry by Jochen Kalmbach. Related information about C-Runtime behavior is discussed in the MSDN forums.

Router for 50Mbps Broadband Service

In my last blog entry I described how my WRT54G was the bottleneck in my new 50Mbps broadband service. Finding a replacement has ended up being substantially more difficult than I expected.

A point of confusion is that there are numerous different speeds. There's the speed of the wired ports, which is generally 10/100/1000. Then there's the speed of the wireless connection, generally 11/54/108/???. Then you have the WAN to LAN speed, which is the speed at which the router can actually route packets. This has *nothing* to do with the numbers above. Most older routers can't route more than 20 to 25Mbps, at which point they max out and become unresponsive. The routers that seem to be designed to have the best WAN to LAN performance are the "gaming" routers.

Very few routers include the specifications for the WAN to LAN performance. Of the ones that do, the specification is usually taken with important features like SPI (Stateful Packet Inspection) turned off. One Cisco router advertises 800Mbps WAN to LAN performance. What they don't tell you is that performance drops to 20Mbps (a 97% drop!) if you turn on IPS (Intrusion Protection System.)

The only reference I've found is the chart at smallnetbuilder.com. Make sure you select WAN to LAN Throughput. Less than 15% of the routers listed break the 150Mbps barrier. I chose this number because 100Mbps broadband is coming and I want some horsepower left over for features like SPI.

So if we look at the top routers, we learn that almost all of them are over $100 and many of them are over $150. This is a pretty big jump over the $40 routers that litter the bottom end of the list. A careful review of the routers on Amazon and NewEgg shows that many of these routers have serious problems. For example, the D-Link DIR-825 has been out for over a year and has achieved five stars on NewEgg with just 35% of reviewers. The somewhat better rated Linksys WRT600N is no longer for sale, and its replacement, the WRT610N, took a 20% performance hit and also falls to just 35% for five star ratings.

The D-Link DGL-4500, one of the "gaming" routers I mentioned earlier, has a more respectable 55% of five star ratings, has been out for two years, and costs $150. Personally, I've had several bad experiences with poor D-Link firmware in the past, so this isn't my first choice.

You might think I'm just being picky looking at the number of 5 star ratings, but the LinkSys WRT54GL router is a prime example of doing things right. After 2500 reviews, it has 84% five star ratings, and this router is prized by the hard-to-please hardcore techies.

At this point I'm leaning towards the Netgear WNDR3700. It's only been on the market for a couple of months, but has garnered 72% of five star ratings from the early adopters - impressive in an industry that usually requires a year of firmware updates to get things right. I've had troubles with Netgear in the past, but this whole situation seems to be be a matter of choosing the best of a dubious lot.

One router that hasn't shipped yet is the WNR3500L. It's based on open source and is generating a lot of buzz, but you can't get one yet. It's worth watching.

If you try to look at the cheaper units, the ratings become even more disparate. There are many more 1 star ratings for the LinkSys WRT120N than there are 5 star ratings. The Belkin N1 Vision F5D8232-4 suffers from similar ratings. The D-Link DGL-4300 is highly rated, but it runs hot. That makes sense - it's four years old, absolutely ancient technically.

Lastly, if you have Macs in-house, the AirPort Extreme seems to generate universal admiration. The MB053LL/A model scores well in the chart mentioned above, but there are several other models that aren't listed. The AirPort can be used with Windows, but it's much easier to configure with a Mac.

Cox Ultimate Broadband - 50Mbps

I dumped my old ISP and signed up for Cox's "Ultimate" broadband package, with speeds up to 55Mbps downstream and 5Mbps upstream. This is the bleeding edge, and getting it working at full speed is tricky.

Amazingly, Cox was able to get the DOCSIS 3.0 modem installed and running on the first try. Given that this technology is new for Cox, I was quite surprised.

So I fired up Speedtest.net and obtained:
22 Mbps download
9 Mbps upstream

Astute readers will notice that this is half of what I was expecting downstream and almost twice what I was expecting upstream. Strange.

I'll save you the details of four hours of sleuthing and simply present my results:

• The Router.The problem is almost entirely caused by my WRT54G router running DD-WRT v24-sp1. The router maxes out at 22 Mbps. If one IP connection is running at that speed, the WRT54G won't even accept wired http connections to the status page. I tried installing TOMATO, and that maxed out at 28.5Mbps. A respectable improvement, but far short of what I needed. Discussions on dslreports.com indicate that my results are typical and that the WRT54G is simply too slow to meet my needs.
  
• Buffer management.You might think a fast Broadband connection would be just like a 100MB local Ethernet connection, but it's not. Local Ethernet connections typically have submillisecond latencies. Broadband connections can easily have 200ms latencies, which means that over a megabyte of data can be transmitted before an ACK is received. This is a major change in how buffers are managed in the Ethernet stack. I found that tuning for 50Mbps broadband required parameters very similar for max throughput for Gigabit Ethernet.
  
• RWIN and MTU.Windows Vista and Windows 7 automatically tune IP parameters, so I didn't need to adjust the RWIN, MTU, or other parameters. However, Windows XP and earlier users will almost certainly need to hand tune parameters to make things work properly. Verizon has a Speed Optimizer tool that does this automatically, but Cox does not. Don't use Verizon's tool - Verizon uses an MTU of 1492 and Cox uses 1500.
  
• Wireless Connections.If you are using 802.11g (as most people do), your connection maxes out at about 20Mbps. (Maybe 25Mbps if your wireless connection is perfect.) Buying 28Mbps or 50Mbps service is simply a waste of money. You need to upgrade to 802.11n to run full speed.
  
• Testing. 50 Mbps is fairly slow for a LAN, but it's the bleeding edge for consumer WAN technology. Most web sites simply can't service data that fast. Speedtest.net won't go that fast if it's busy. I can only download at about 10Mbps from my corporate server, which has a 100Mbps connection to a backbone. This is probably a TCP tuning problem on the RedHat server, but this demonstrates that both Windows and Linux have default connections that are not well suited to this configuration.
  

[Update 2/26/2010] Here is the speed test result from the iMac in the office. This was performed without any tuning of OS X:

Porting C++ Applications to Visual Studio 2010

[This blog entry was updated on 4/12/2010 to reflect the final release of Visual Studio 2010.]

I've now spent a couple of days porting my C++ applications to VS2010. Here are the top ten things I've learned.

1. STL
If your app makes significant use of STL, prepare to spend some time making it work in VS2010. The new version of STL has been significantly reworked to take advantage of rvalue references and static asserts. I spent several hours reworking some of my derived classes to update the supported operators.

2. Boost
Make sure you get version 1.42 of Boost, which includes support for VS2010. Thankfully, Boost 1.42 has not yet been updated with C++0x features, so porting was relatively straightforward.

3. Property Manager
Earlier versions of Visual Studio used the Property Manager to manage certain settings like optimization and DLL usage. Visual Studio 2010 dramatically expands the use of the Property Manager. If you created your own pages in the property manager in an existing project, you'll find that VS2010 migrates the information to .props files instead of .vsprops. The files use different formats, although they are both XML. Don't forget to check these new files into source control. I had trouble with parameters that were lost when the project was migrated, including Additional Include Directories, Additional Libraries, and Delay Load Libraries. Make sure you compare your VS2005 or VS2008 property pages with the equivalent pages in VS2010.

4. Target Name
VS2010 introduces a new "Target Name" property in the General page of Configuration Properties. I still haven't figured out the rationale, but you need to make sure your Target Name matches the Output File in the Linker or Library pages. (Note that Target Name should not have the extension included.) One symptom of this happening is that you try to run your application and the application is not found, even after building correctly.

5. Default Directories
For the last several releases, the default Include and Library search paths were found under Tools/Options. Now they are on the General page of the project Properties, under "VC++ Directories." This is all managed from the Property Manager, so you can override the default directories for a single solution by replacing Microsoft.Cpp.Win32.User (or Microsoft.Cpp.x64.User) with your own custom page. This is a big win over earlier versions of Visual Studio, where the default directories were a global setting that couldn't be overridden.

6. Platform Toolset
You can use the Visual Studio 2008 compiler/linker under the VS2010 IDE, which allows you to get the improved user experience without having to port your code. However, if you are still stuck on VS2003 or VS2005, this won't help you.

7. Target CLR Version
It's possible to target versions of the CLR other than 4.0, but you can't do it from the IDE. See my related blog post for instructions.

8. Show Inactive Code Blocks
The editor is now much more successful at detecting Inactive Code Blocks. Historically this has been a problem because commands such as "Go to declaration" don't work in an Inactive Code Block.

9. MFC Application Size
The size of all of my MFC-based applications (statically linked) has grown by about 1.5MB. I'm not very happy about this. I double-checked all of my build settings and they are correct. A similar question on the Visual C++ Team Blog was given the answer that "The increased capabilities of MFC have introduced a number of additional interdependencies that end up causing more of MFC to be pulled in when linking a minimal application that links with the static MFC libraries." So this problem may not be solvable.

10. Intellitrace
IntelliTrace is the "historical debugging" tool that records the guts of your application as it runs. Unfortunately, IntelliTrace doesn't work for C++. Bummer. It also requires the Visual Studio Ultimate, which isn't available to Microsoft Partners. Double bummer.

11. Range-based "for" loops
One of the features I was particularly looking forward to was support for a compiler-based "foreach" syntax, which made it much simpler to write loops based on STL containers. Unfortunately, Visual Studio 2010 doesn't include this feature because the committee standardized is too late in the Visual Studio beta process. More information can be found in the comments section from this blog entry on the Visual C++ Team Blog.