Online Backups - The Good, The Bad, and the Ugly

This is the third and last part in my series on doing backups for home and small office environments.

Online backup has come a long way in the past couple of years. Service has become more reliable and faster Internet connections have made backing up large quantities of data more palatable. My Internet connection is 5Mbps (30 MB/min) upstream, which is more than fast enough for overnight backup.

In spite of these advances, my opinion is that online backup is still best suited for last-ditch disaster recovery. Any online backup strategy should be paired with a primary backup strategy that includes an image and/or file backup to a USB drive or network drive. Buying 100-GB of online storage is quite expensive, much less 1-TB. The "unlimited" plans are attractive, but companies make significant compromises to make these unlimited plans affordable, including lower backup speeds, lack of geographic redundancy, and subpar technical support. My research indicates that cheap and reliable are mutually exclusive for online backup - the lower the price, the less reliable the service.

It should also be noted that it's the recovery that's most important, not the backup. Numerous people reported that Mozy could take several days (or more!) to prepare to restore a large number of files. Ordering a DVD was even worse. For me, this isn't acceptable.

Here's my experience with the various services.

Acronis. My first attempt at online backup was with Acronis, who introduced online backup in October 2009. My experience with it has been total failure. Acronis TrueImage 11 has never succeeded in creating a backup on my Windows Vista system. Acronis recently increased my backup space from 25GB to 250GB, all for $50/year. This seems like a safe choice for them, since I've never been able to backup more than 5GB to their servers.

Mozy I've been using Mozy Pro for about two months now. They charge 50 cents/GB/month for their Pro version. On the plus side, the nightly backup works quite reliably and I was easily able to select what I wanted to back up.

However, there are several downsides to Mozy. First, backup speed is only thirty to forty percent of the possible maximum. Mozy compresses data, then sends it. During compression, Mozy usually isn't transmitting data. When Mozy is transmitting, it only manages to max out my connection part of the time. So there's a lot of room for improvement in backup speed. [Updated 5/27/2010] The performance problems seem to have been resolved in Mozy Pro v2.0. I'm consistently maxing out my upstream connection at 5Mbps.

Second, numerous people have reported problems restoring data from Mozy, even from their Pro service. Former employees of Mozy confirm that Mozy has been having problems managing rapid growth.

Third, Mozy does not allow me to exclude specific extensions. While this isn't an issue for most people, it's a real problem when backing up development directories containing projects built by Microsoft Visual Studio.[Updated 5/27/2010] Mozy Pro v1.6 did allow excluding extensions, but it was difficult to use and the documentation was useless. Mozy Pro v2.0 makes it somewhat easier. See my article Configuring Mozy Pro for Visual Studio.

My final issue with Mozy is that it doesn't support multiple historical revisions. This is a feature that I consider very important in case local backups become corrupted (which, as I mentioned earlier, has been all too frequent with Acronis TrueImage.) [Updated 5/27/2010] Multiple historical revisions are definitely supported in Mozy Pro v2.0, but again the documentation is lacking. To view the historical revisions, go to My Computer, open MozyPro Remote Backup, open one of the drives, right-click on a folder, and select Change Time.

Carbonite I didn't try Carbonite, although I know several people who are happy with it.

Iron Mountain I will be trying Iron Mountain next. They are the most expensive, but, compared to the cost of losing the data, the price is cheap. They support multiple historical revisions, incremental block updates, Windows 7, and many other features. Iron Mountain makes backup systems for large companies, so I'm hopeful that their technology is more reliable. I'll keep you posted.

To recap, although online backup has been around for several years, most of the solutions still have significant shortcomings, especially with reliability. My research is that you get what you pay for, with no exceptions.

How I Backup My Windows System for Multiple Contingencies

In my last blog posting, I described the events that have lead up to my paranoia about keeping my computer backed up. In this posting I'll talk about about how I actually keep my system backed up and the challenges involved in these methods.

These backup strategies are appropriate for individuals and for small offices. For larger environments, the reasons for doing various types of backups still hold true, but the implementation strategy will change significantly.

1. DVD Backup. My first strategy is mundane - my software development directories are backed up to DVD after every release. This is more complicated than it sounds. First, Visual Studio creates hundreds of megabytes of temporary files, including .ncb, .ilk, obj, and .pch files. There generally no point in backing up these files. In Roxio Creator I can exclude all of these file types, which makes it trivial to just drag the root directory of my development structure to the DVD. Other DVD burning applications, such as CDBurnerXP, won't automatically exclude file types, so it's somewhat more complex to use them for backups. [Update 12/21/2010: CDBurnerXP now supports excluding by extension, under Edit/Filter Files. However, you can't use this filter from the command line.]

I've been careful to use high quality CDs and DVDs for these backups. I have CDs I burned over ten years ago that I can still read without difficulty. My recordable DVDs are newer, so the jury is still out on how long they'll last. The important thing to remember about CD-R and DVD-R is that, unless you buy archive-grade media, they shouldn't be expected to last more than a few years.

DVD and CD backup requires a substantial amount of effort. You have to remember to do it, it's relatively time consumering (especially if you validate the media after burning), and you have to keep all of the media labeled and filed. So it's only a good streategy if you are quite disciplined. However, DVD backups are much easier to move offsite and you have a permanent record, especially if you need an audit trail.

2. Image Backup. Image backups are the cornerstone of my backup strategy. If my computer total fails, due to a virus, a lightning strike, or user error, the image backups are what allow me to recover in a few hours instead of a few days.

For the last eight years I've created my image backups with Acronis TrueImage. This has been a mixed blessing. On the plus side, Acronis can do incremental image backups, which saves a lot of disk space. Acronis has a standalone boot CD, so you can recover the system from a raw hard drive. Acronis can do backups over the network, even when booted from the recovery disk. All of these have been invaluable.

On the minus side, Acronis has had significant difficulty with consistent quality. Their latest release, Version 11, is a total disaster and has been completely unusable for many people. Acronis also has a nasty habit of creating corrupted backups, which means that a validation phase is required.

I do image backups once a week, supplemented by file backups that are done daily. I try to keep three months of image backups because some viruses can take weeks to trigger.

[Update 12/21/2010: Because of ongoing stability problems with Acronis, I have stopped using it.]

3. File Backup. Acronis TrueImage can also do file backups. I backup my development directories and documents on a daily basis. This usually comes out to less than 5-GB, so I can store a lot of them on a 1-TB backup disk. Again, file backups are done incrementally, so the baseline file is dramatically larger than the daily incremental files.

My big problem with Acronis is the poor support for rotation management. I want to keep backup snapshots every 30 days for the last six months, every week for the past month, and every day for the past week. Acronis provides several different options, none of which can do what I want, and all of which seem to cause the frequency of corrupted backups to increase dramatically. [Update 12/21/2010: Macrium Reflect also has better support for managing archive rotation.]

File backups provide protection from accidental deletion and provide the ability to revert to an older version of a document.

4. RAID 1 Mirror drives I mentioned this in my last blog post, but it bears repeating. I use mirrored drives to handle the case of catastrophic hard drive failure, which has happened to me multiple times before. One of the two mirrored hard drives can fail completely and it won't affect the performance of the system. [Update 5/30/2010 - This happened to me again today. A hard drive failed in my primary development system. Without the RAID 1 pair I'd be looking at restoring from backup right now.]

I use software RAID (Intel Matrix Storage), which definitely has its downsides. Every time you reboot without cleanly shutting down the system, (such as a power fialure or hitting the Reset button,) the drive mirrors have to be validated. This takes six to ten hours, during which the system is almost unusable.

Hardware RAID solutions with battery backup can fix this problem, but such RAID hardware is dramatically more expensive and generally is not compatible with image backup software such as Acronis.

Note that not all RAID strategies provides protection. RAID 0 (striping) actually DECREASES reliability, because if any drive fails, your data will be completely lost.

5. Online backup. My final strategy is for last ditch disaster recovery - theft, fire, natural disaster. I'll talk about it in the third and final part of my blogs about backup.

Six Myths About Backing Up Your Computer

This is my tale about how I got religion about backups. I played a part in all of the stories below. The amazing part is not that any one or two of them happened around me - the amazing part is that they all happened around me.

Myth #1. I have a new hard drive, my data is safe.

Five years ago I helped a friend replace his hard drive after his original drive failed and he lost everything. So he bought the new drive, spent two or three days reloading everything, and went on his way. Three months later, the new drive failed too, and again he lost everything. My friend is an optimist. He believes he's had all the bad luck he's going to have and he still doesn't do backups.

A few years later, I was overseeing putting a new server in the office and had paid a consultant a lot of money to set it up to our satisfaction. The server was backed up after the OS was installed, but before the consultant made changes. After the consultant was finished, we went home for the weekend. When we came in on Monday, the hard disk had failed. All of the money we paid for the consultant was wasted. Even more embarrassing, the CD for the backup software was sitting on top of the server.

The Reality: Hard drives have their highest failure rate when they are new.

Myth #2. Backing up once or twice a month is enough.

My neighbor conscientiously made weekly backups of his laptop. He had decided that losing a few days of work wouldn't be a big deal. Inevitably, his hard drive died hours before a sizable document was due to his largest customer. He had a four day old backup, but he'd done well over thirty hours of work on the document in the interim. I was able to recover the data with low-level tools, but recovery companies charge hundreds to thousands of dollars for that kind of work.

The Reality: Murphy's Law should never be underestimated. Nightly backups are a minimum, and software for continuous backup is easily available.

Myth #3. Backing up documents to an external drive is good enough.

Two years after I helped my optimist friend above, the primary hard drive abruptly failed in our corporate ecommerce web server. We had a backup of the database and other data, but we still had to reconfigure the operating system from scratch. It took four days to get the server running again, which was basically four days of lost sales. I learned a harsh lesson, since it cost thousands of dollars in sales and made a lot of existing customers unappy.

The Reality: To quickly recover from a catastrophic failure, you must have image backups, not data backups.

Myth #4. Having an image backup on an external drive or a NAS is good enough.

I have now been in two different offices that were burglarized and the computers were stolen. If I'd had a USB hard drive sitting on the shelf, it would have been taken too. Even if you put the backup in the safe, it just takes one flood, fire, tornado, or hurricane to destroy the sensitive platters of a hard drive.

I've also known people who whose office was served with a search warrant. The content on computers is just assumed to be relevant, so officers often take *everything* that looks like a computer or a hard drive - including what's in the safe. If you don't have off-site backups, you're done because you may not ever get that equipment back.

The Reality: You must have off-site backups.

Myth #5. I use RAID hard drives so I don't have to do backups.

My primary computer had mirrored hard drives installed right from the start. Six months later, my computer contracted the second virus I've ever had. However, RAID is protection against disk failure, not against data loss that was intentionally caused by a virus. I recovered from the failure by reloading the entire system from image-level backups, as I described in #3.

As I became increasingly paranoid about hard drives failures, I upgraded our corporate web server with a hardware RAID card and enterprise-grade SCSI hard drives to allow RAID 1 drive mirroring. SCSI hard drives generally cost several times the price of a similarly-sized IDE (consumer) drive because they are expected to be substantially more reliable.

A week after the new hardware went live, I started seeing I/O errors. Our expensive new RAID controller board had malfunctioned. As a result, it corrupted the SQL database that contained the customer support system, which contained FIVE YEARS of customer support information, including an extensive knowledgebase. The primary backup was completely missing due to a configuration error in the backup script. I ended up staying up all night to restore the database from a third-level backup that I'd been paranoid enough to perform.

The Reality: If you don't have 100% redundancy, including spare computers ready to go, downtime and possible data loss are just one hardware failure away. Also, RAID provides zero protection again viruses and user error.

Myth #6. Paying experts to do backups is good enough.

After we upgraded to RAID hard drives, we purchased Managed Backup for our corporate server. This means that experts are responsible for keeping our server backed up, and they are responsible for recovery when things go wrong.

Last summer one of our web server log files was erased accidentally. These files are critical for financial analysis and prediction, so I was pleased that we had the experts managing our backup. Except for one thing. The "experts" considered the log files low value and a waste of space, so the file I needed had not been backed up. Damn.

The Reality: Backups that haven't been verified and tested aren't backups.

Conclusion

You can't be too paranoid about backup. I have four levels of redundancy in my backups, but I still worry that it won't be good enough. If your computer contains critical information, whether it's corporate documents, baby pictures, legal documents, graduate thesis, or financial records, do yourself a favor and back it up, preferably in multiple places.

In Part 2, I'll look at how I've implemented my tiered backup strategy and what software I use to make it happen.

Installing PHP on Windows 7

If you've wrestled with installing PHP on IIS before, there's good news. Installing PHP in Windows 7 is easy. Super easy. Microsoft has created the Web Platform Installer (WPI), which supports installing many things, PHP included.

If you look in the IIS Manager, down at the bottom of the rows of icons you should see Web Platform Installer. If you don't see it there, go to your Start menu and search for "platform".

Check off PHP and let it install. There are a couple of dependencies that will be added automatically.

There's one more change you need to make after PHP is installed, which is to add "index.php" to the list of Default Documents.

There are some other things you should know:

• WPI correctly installs PHP even on a 64-bit system. There's no need to use cscript to separately change configurations.
  
• Apparently, there's still no 64-bit version of PHP for Windows. WPI installs the 32-bit version, which works correctly even under the 64-bit version of IIS.
  
• There's no uninstall option. However, everything is installed to a single directory and removing the relevant FastCGI settings is straightforward.
  
• The correct strategy for upgrading to new versions of PHP isn't obvious. I'm not sure if Microsoft used the stock build of PHP or if they created a custom one.
  

One small gotcha - WPI installs PHP 5. If this is your first encounter with PHP 5, you need to make sure you use <?php ... ?> as opposed to <? ... ?>.

Handling Exceptions from STL

Recently I tracked down a crash to an out of bounds index in a STL vector. The strange thing was that the crash was being handled by Windows instead of by our error reporting code, which is designed to trap everything that the explicit exception handlers did not catch.

Obviously it wasn't.

I created some short test code:
vector vec;
vec[5] = 3;

When run under the debugger, I receive a helpful window that says "vector out of range" with the standard Abort/Retry/Ignore debugger options. All very well and good, but there was no exception being thrown.

I tried using set_unexpected(), which turns out didn't mean what I thought it meant. The function set_unexpected() is called when an exception is thrown through a function that doesn't declare that exception in the throw() statement in the function declaration. Visual C++ does not support these declarations, and so does not support set_unexpected(). This is discussed in the documentation.

So I tried set_terminate(), which is the only other function under Exception Handling Routines that seemed relevant. Unfortunately, my termination function was never called. Since my two line program certainly was not catching an exceptions, something else was happening.

I tried running the code again as a Release build instead of a Debug build. It still broke to the debugger, but in a separate place than the Debug build.

I tried reading the documentation, which makes no mention that operator[] can throw an exception.

Stranger and stranger. I tried walking the stack with the Release build, which looked like this:
msvcr90.dll!_crt_debugger_hook
msvcr90.dll!_invalid_parameter
msvcr90.dll!_invalid_parameter_noinfo
Exc.exe!wmain

The function _invalid_parameter() had this comment:
// No user handler is defined. Notify the debugger if attached.
So I tried running the Release build without the debugger. Which also yielded no useful information.

I don't remember exactly how I figured out what was happening, but it turns out that STL is throwing "out_of_range", which is derived from "logic_error", which is derived from class "exception". I've been using STL for twelve years. I've written articles about it. And I've never heard of these exception classes. Clearly, the authors of the documentation had never heard of them either.

Note that this behavior is different than other implementations of STL, which specifically document that only the at() functions throws an exception, not operator[]. At least "out_of_range" is defined in the standard.

[Update 8/13/2010 Visual Studio 2010 behaves according to the standard. In Release builds, vector<>::operator[] does not do bounds checking and does not throw the out_range_range exception. In Debug builds, an assertion is shown because _ITERATOR_DEBUG_LEVEL is set to 2 in yyvals.h if _DEBUG is defined.]

But the fun wasn't over. I still didn't know why set_terminate() wasn't working. The documentation for Unhandled C++ Exceptions only discusses set_terminate().

It turns out that unhandled logic_error exceptions are handled by invalid_parameter_handler and not by terminate_handler. This is broadly discussed in the documentation for Invalid Parameter Handler Routine. However, that documentation describes how errno is set, which doesn't appear to happen with STL.

The situation gets even more murky if you consider Windows SEH (Structured Exception Handling) as well as C++ exceptions. The standard way of dealing with SEH is to use SetUnhandledExceptionFilter(). However, Visual Studio 2008 changes this behavior. See a blog entry by Jochen Kalmbach. Related information about C-Runtime behavior is discussed in the MSDN forums.

Router for 50Mbps Broadband Service

In my last blog entry I described how my WRT54G was the bottleneck in my new 50Mbps broadband service. Finding a replacement has ended up being substantially more difficult than I expected.

A point of confusion is that there are numerous different speeds. There's the speed of the wired ports, which is generally 10/100/1000. Then there's the speed of the wireless connection, generally 11/54/108/???. Then you have the WAN to LAN speed, which is the speed at which the router can actually route packets. This has *nothing* to do with the numbers above. Most older routers can't route more than 20 to 25Mbps, at which point they max out and become unresponsive. The routers that seem to be designed to have the best WAN to LAN performance are the "gaming" routers.

Very few routers include the specifications for the WAN to LAN performance. Of the ones that do, the specification is usually taken with important features like SPI (Stateful Packet Inspection) turned off. One Cisco router advertises 800Mbps WAN to LAN performance. What they don't tell you is that performance drops to 20Mbps (a 97% drop!) if you turn on IPS (Intrusion Protection System.)

The only reference I've found is the chart at smallnetbuilder.com. Make sure you select WAN to LAN Throughput. Less than 15% of the routers listed break the 150Mbps barrier. I chose this number because 100Mbps broadband is coming and I want some horsepower left over for features like SPI.

So if we look at the top routers, we learn that almost all of them are over $100 and many of them are over $150. This is a pretty big jump over the $40 routers that litter the bottom end of the list. A careful review of the routers on Amazon and NewEgg shows that many of these routers have serious problems. For example, the D-Link DIR-825 has been out for over a year and has achieved five stars on NewEgg with just 35% of reviewers. The somewhat better rated Linksys WRT600N is no longer for sale, and its replacement, the WRT610N, took a 20% performance hit and also falls to just 35% for five star ratings.

The D-Link DGL-4500, one of the "gaming" routers I mentioned earlier, has a more respectable 55% of five star ratings, has been out for two years, and costs $150. Personally, I've had several bad experiences with poor D-Link firmware in the past, so this isn't my first choice.

You might think I'm just being picky looking at the number of 5 star ratings, but the LinkSys WRT54GL router is a prime example of doing things right. After 2500 reviews, it has 84% five star ratings, and this router is prized by the hard-to-please hardcore techies.

At this point I'm leaning towards the Netgear WNDR3700. It's only been on the market for a couple of months, but has garnered 72% of five star ratings from the early adopters - impressive in an industry that usually requires a year of firmware updates to get things right. I've had troubles with Netgear in the past, but this whole situation seems to be be a matter of choosing the best of a dubious lot.

One router that hasn't shipped yet is the WNR3500L. It's based on open source and is generating a lot of buzz, but you can't get one yet. It's worth watching.

If you try to look at the cheaper units, the ratings become even more disparate. There are many more 1 star ratings for the LinkSys WRT120N than there are 5 star ratings. The Belkin N1 Vision F5D8232-4 suffers from similar ratings. The D-Link DGL-4300 is highly rated, but it runs hot. That makes sense - it's four years old, absolutely ancient technically.

Lastly, if you have Macs in-house, the AirPort Extreme seems to generate universal admiration. The MB053LL/A model scores well in the chart mentioned above, but there are several other models that aren't listed. The AirPort can be used with Windows, but it's much easier to configure with a Mac.

Cox Ultimate Broadband - 50Mbps

I dumped my old ISP and signed up for Cox's "Ultimate" broadband package, with speeds up to 55Mbps downstream and 5Mbps upstream. This is the bleeding edge, and getting it working at full speed is tricky.

Amazingly, Cox was able to get the DOCSIS 3.0 modem installed and running on the first try. Given that this technology is new for Cox, I was quite surprised.

So I fired up Speedtest.net and obtained:
22 Mbps download
9 Mbps upstream

Astute readers will notice that this is half of what I was expecting downstream and almost twice what I was expecting upstream. Strange.

I'll save you the details of four hours of sleuthing and simply present my results:

• The Router.The problem is almost entirely caused by my WRT54G router running DD-WRT v24-sp1. The router maxes out at 22 Mbps. If one IP connection is running at that speed, the WRT54G won't even accept wired http connections to the status page. I tried installing TOMATO, and that maxed out at 28.5Mbps. A respectable improvement, but far short of what I needed. Discussions on dslreports.com indicate that my results are typical and that the WRT54G is simply too slow to meet my needs.
  
• Buffer management.You might think a fast Broadband connection would be just like a 100MB local Ethernet connection, but it's not. Local Ethernet connections typically have submillisecond latencies. Broadband connections can easily have 200ms latencies, which means that over a megabyte of data can be transmitted before an ACK is received. This is a major change in how buffers are managed in the Ethernet stack. I found that tuning for 50Mbps broadband required parameters very similar for max throughput for Gigabit Ethernet.
  
• RWIN and MTU.Windows Vista and Windows 7 automatically tune IP parameters, so I didn't need to adjust the RWIN, MTU, or other parameters. However, Windows XP and earlier users will almost certainly need to hand tune parameters to make things work properly. Verizon has a Speed Optimizer tool that does this automatically, but Cox does not. Don't use Verizon's tool - Verizon uses an MTU of 1492 and Cox uses 1500.
  
• Wireless Connections.If you are using 802.11g (as most people do), your connection maxes out at about 20Mbps. (Maybe 25Mbps if your wireless connection is perfect.) Buying 28Mbps or 50Mbps service is simply a waste of money. You need to upgrade to 802.11n to run full speed.
  
• Testing. 50 Mbps is fairly slow for a LAN, but it's the bleeding edge for consumer WAN technology. Most web sites simply can't service data that fast. Speedtest.net won't go that fast if it's busy. I can only download at about 10Mbps from my corporate server, which has a 100Mbps connection to a backbone. This is probably a TCP tuning problem on the RedHat server, but this demonstrates that both Windows and Linux have default connections that are not well suited to this configuration.
  

[Update 2/26/2010] Here is the speed test result from the iMac in the office. This was performed without any tuning of OS X: