Norton Antivirus: The Clear Choice for SSDs

Today I walked up to my computer and noticed that all eight cores were pegged. I thought that was odd, since the computer should have been doing absolutely nothing. Then I noticed the "Norton Antivirus" logo in the bottom right. "Could it be?" I thought to myself.  Is there actually a consumer product on the market that is smart enough to use more than one core?!?

I tapped the keyboard, the Norton Antivirus banner disappeared, and all of the cores dropped back to idle.

I'm really impressed by this. I was trying to remember the last time I was impressed by antivirus software. I'm pretty sure the answer is "never."

Normally an antivirus application is limited by your hard drive's random access performance. Antivirus software processes files in a directory sequentially, but the files often aren't aren't laid out on the disk in that order. This means that the drive is mostly being accessed randomly, not sequentially, all of which is why it can take your antivirus software twelve hours to scan your hard drive, even with a fast hard drive and processor.

However, with an SSD, the system can maintain transfer rates in excess of 100MB/second, even with random access behavior. Crunching that much data per second is going to take more than one core, and, I'm impressed to say, Norton Antivirus appears to be up to the task.

Lightning Fast Builds with Visual Studio 2010 and an SSD

I reduced my Visual Studio 2010 C++ build time from 21 minutes to  7  5 minutes! You can too. Here's how.

I'm a build performance junkie. If there's one thing I really hate in life, it's sitting around waiting for builds to complete. Fifteen years ago, the very first article I published was titled Speeding Up Visual C++. It was all about making Visual C++ 1.51 go faster on what was then a state of the art computer - an ISA bus Gateway DX2/50. Woo hoo! My recommendations were:

1. Use Precompiled Headers.
2. Upgrade to 16MB of memory.
3. Use 4 megabytes of Disk Cache.
4. Upgrade your Hard Drive to Fast SCSI or Enhanced IDE.
5. Turn off Browse Info.
6. 32 Bit File Access.

Today computers are thousands of times faster, but rotating platter disk drives are still desperately slow. The seek time of 7200RPM drives has changed very little in the last ten years, although the transfer rate for sequential files has risen dramatically. That problem, combined with Visual Studio's desire to create hundreds of .tlog temporary files, quarter gigabyte .sdf files, and project size bloat means that the average build may be even slower today than it was fifteen years ago.

Historically, your CPU would be sitting idle most of the time waiting for the hard disk to keep up. Linking performance is based almost entirely on your disk's random access read/write performance. The highly rated Western Digital Caviar Black can only perform about 100 random access IOPS (I/O Operations Per Second.) It takes multiple I/O operations per OBJ file, so a significant fraction of the link time is waiting for the hard drive to do its job.

Enter the latest generation of SSDs driven by the Sandforce Controller, such as the OCZ Vertex 2. These drives can do over 40,000 IOPS - 400 times faster than a Caviar Black. And Visual Studio 2010 build performance is phenomenal. In fact, these drives are so fast that their impact on build time is negligible. This SSD will easily hit over 50MB/sec of 4KB random writes. In contrast, the ultra-zippy 10,000 RPM VelociRaptor can only do about 3.5MB/sec. (Note that disk striping or mirroring has minimal impact on build performance because the linker isn't smart enough to use scatter/gather to force the queue depth high enough to let the command queuing on the drive work its magic.)

Now that the hard disk performance no longer matters, our next bottleneck is the CPU. You can tell your boss you need one of those monster hyper-threaded quad core i7 processors such as the 875k or, for the money-is-no-object crowd, the hyper-threaded six core 980X. Visual Studio 2010 automatically uses parallel builds. My 875k pegs all eight cores simultaneously at 100%. Compiles proceed eight files at a time and the CPUs stay pegged until the compile is finished. I've never seen a project build so fast.

The next bottleneck is probably your RAM. If you have 4GB RAM running on a 32-bit OS, you are severely limited and you probably won't be able to run eight compiles (much less 24 compiles if you are using parallel MSBuild tasks, as I explain in Part 2.) Upgrade to Windows 7 64-bit with 8GB of RAM.

So it's interesting that the types of recommendations for build performance haven't changed much. Here is my updated list:

1. Use Precompiled Headers.
2. Upgrade to 16MB 8GB of memory.
3. Use 4 megabytes of Disk Cache. Upgrade to 64-bit Windows. 
4. Upgrade your Hard Drive to Fast SCSI or Enhanced IDE a Sandforce-based SSD. 
5. Turn off Browse Info. (This is still true. Browse Info is different than Intellisense.)
6. 32 Bit File Access. Check your motherboard's SATA implementation. At SSD speeds, not all controllers are created equal.

In Part 2 of this article, I'll talk about how to tune your build system to keep all that hardware busy.

The system used for this article was:

• ASUS P7P55D-E Pro motherboard.
• Intel 875k i7 processor.
• OCZ Vertex 2 120GB SSD.
• 8GB RAM.
• Thermaltake MUX-120 heatsink.

Do not use the SATA 3 Marvell controller on the ASUS motherboard. The standard ICH10 controller is much faster with SSDs.

The prior system that took 21 minutes was a Core 2 Duo 2.4 GHz with ASUS P5B Deluxe and Caviar Black 750GB hard drive.

Project is permanently out of date

I've seen several cases with Visual Studio 2010 and later where my project would be permanently "out of date." I'd build the project, run it, and would immediately be told that my project was out of date. Solving this through trial and error is tedious at best. This problem commonly is caused by two different problems:

Missing file in project

The most common cause is a file that's in your project that you've deleted from the disk. These are easy to find in smaller projects by just looking through the project for a file with an X next to it. If you can't find the file causing the problem, enable detailed Diagnostic output in MSBuild. In Visual Studio, open Tools | Options, select Projects and Solutions | Build and Run, and set MSBuild project build output verbosity to Detailed.

Alternatively, here's a handy Python script that will examine a .vcxproj file and look for any missing files:
http://stackoverflow.com/questions/2762930/vs2010-always-thinks-project-is-out-of-date-but-nothing-has-changed

I've also seen this problem when I renamed a file. The old filename is still embedded in the compiler's temporary files, even though the name no longer appears in the project. The solution is to delete all intermediate files in the project and rebuild.

Project upgraded from earlier version of Visual Studio

The StackOverflow page above also details a bug where MSBuild says, "Forcing rebuild of all source files due to a change in the command line since the last build."

The most likely cause is if you used /MP in the Additional Options box in project Properties | Configuration Properties | C/C++ | Command Line. This was often done in Visual Studio 2005. My project started building properly after I fixed this (although it was tricky, because I had set /MP1 on certain files due to bugs in multiprocessor support in VC2005.) This tip came from https://social.msdn.microsoft.com/Forums/vstudio/en-US/aca632fa-e1e0-4511-aa03-a309ae547a5b/how-to-see-the-command-line-forcing-rebuild-of-all-source-files-due-to-a-change-in-the-command?forum=msbuild.

The next problem I saw in the MSBuild log was the use of an old PDB filename. I was using Visual Studio 2013, which uses vc120.pdb. However, some of my project files used the name vc80.pdb, which was the wrong name and so was never found. In the property page for the project, set it to look something like this: (the part in bold is what you see after you set the value for the first time. The next time you reopen the page, it will show you the value that the compiler will use.)

Unhelpful recommendations

After enabling Detailed output for MSBuild, I saw this error. It didn't appear to affect the problem, but it should probably be fixed to prevent confusion in the future:
Project file contains ToolsVersion="4.0". This toolset may be unknown or missing, in which case you may be able to resolve this by installing the appropriate version of MSBuild, or the build may have been forced to a particular ToolsVersion for policy reasons. Treating the project as if it had ToolsVersion="12.0". For more information, please see http://go.microsoft.com/fwlink/?LinkId=293424.

Another suggestion which didn't work was that intermediate directories use a relative path. but I use an absolute path in my project and it works fine:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/36df2f40-f6d4-4123-8261-88394790f29e/project-always-needs-complete-rebuild-in-vs-2010?forum=vcgeneral

Fixing Permissions on an External Hard Drive

Today I pulled a hard drive from my old computer and hooked it up to my new computer, planning to move the data to the new drive and then use the old drive as a backup disk. Mostly this plan worked well, except that I wasn't allowed to delete many files and folders, even though I was an Administrator. Curious.

The problem turned out to be that I was on a workgroup, not a domain, and so the systems didn't have any common notion of "Administrator". Although Explorer knows how to request elevated permissions, this still isn't enough. You have to "Take Ownership" of the files in order to delete them, and there's no way to do this from Explorer.

I found a solution in the winmatrix forum, but it only works for files, not directories. You can't set the Full Control permission on a directory, so you end up being locked out of directories if you try and use these commands:

takeown /f filepath /r
icacls filepath /grant yourusername:f /t

(I've added /r and /t to the commands, which is required for the them to operate recursively.)

Instead, I did the steps below. These steps assume that the new drive is on E:.

1. Open a Command Prompt with Run As Administrator.
2. Run this command: takeown /f e:\ /r
3. Right-Click on the root of the copied drive.
4. Select Properties.
5. Click the Security tab.
6. Click the Edit button.
7. Select Authenticated Users.
8. Click the checkbox under Allow for Full Control.
9. Run this command: icacls e:\*.* /reset /t
   This command will force all permissions to mirror the permissions on the root of the drive that you set in #6. You must have the *.* or the root directory will be reset, which you don't want.

After these commands completed, I was able to delete all of the desired files. Executing these commands can take quite a while if you have many files on your disk.

Windows 7 Network Performance

A few years back, after I installed Vista, I spent quite a bit of time trying to fix my GigE Ethernet performance with Windows Vista talking to Windows Server 2003 R2. My file copy performance hovered around 15 to 18 MB/sec, which was pretty dismal.

I've just built myself a new PC with a Core i7 CPU and Windows 7. I tried copying a file from the old Vista box (Core 2 Duo 2.4GHz on ASUS P5B Deluxe mobo) to the new Windows 7 box. The copy performance over the Ethernet went straight up to 50MB/sec! This was with a single WD Caviar Black drive on the Vista system, no RAID or striping.

When I tried copying from a Windows Server 2008 R2 system to the new Windows 7 system, I peaked at 112MB/sec for data that was cached, then backed off to 50 MB/sec for data that wasn't cached. Windows Server 2008 R2 was installed on the same hardware that used to be running Windows Server 2003 R2, so the performance increase was solely due to the OS upgrade.

I'm seeing similar performance gains over the internet. Talking to our corporate server from the Vista system, I maxed out at 1.5 MB/s.  Under Windows 7 with i7, I'm able to max out the connection at 5 MB/s.

All of this leads me to believe that the GigE networking performance of Windows Server 2003 R2 was awful, given that I'm running Windows Server 2008 R2 on the exact same hardware with a 5x performance increase.

Net result: (no pun intended) I'm very happy with Windows 7.

Safely using erase() in STL

There are few features of STL that have caused me more aggravation than the erase() functions.The problems with erase() are, first that it takes an iterator as an argument and second that erase() invalidates all iterators - sometimes. This makes for a catch-22 situation if you don't know "the secret."

Consider what happens if you want to erase each element of a collection that meets some condition. Deleting a single element is not an issue because you don't need the iterator afterward. Deleting all elements is similarly easy because you can simply call clear(). Deleting an arbitrary number of elements is generally written this way by most beginners:

set<int>coll;
/* ... */
for (set<int>::iterator itr=coll.begin(); itr!=coll.end(); ++itr)
   if (*itr % 2 == 0)
      coll.erase(itr);

Predictably, this code doesn't work. The iterator is invalidated when erase() is called and so the ++itr statement in the for() loop fails. Most modern versions of STL have debugging code to warn you if you use an invalidated iterator, but no such training wheels existed in the early days of STL. I got a lot of bug reports because of this mistake.

My first attempted workaround was for the vector class. I wanted to ignore the iterator completely and just step through the collection with an index:

for (unsigned int i=0; i<coll.size(); ++i)

When I tried to call coll.erase(), I discovered that I needed an iterator, which is what I was trying to avoid in the first place. Years after I first tried to make this solution work, I finally learned the solution. Here's an example to delete all even values when using an index-based for loop;

vector<int> coll;
/* ... */
for (unsigned int i=0; i<coll.size(); ++i)
    if (coll[i]%2 == 0)
        coll.erase(coll.begin() + i--);

coll.begin() is always a valid iterator (even if it equals "end()"), and you are allowed to use array arithmetic on iterators for vector<> and for deque<>. Although this solutions works, it's a bit of a hack, especially because you have to "fix up" i after the call to erase() so that the index of the array won't change. This strategy also won't work for collections such as list<> and set<> that can't be indexed by a scalar.

So I was back to figuring out how to get a "valid" iterator after calling erase().

In some STL collections, the erase() function returns an iterator to the next element. Therefore, even though the original iterator was invalidated, you were given a new iterator that would work, so there was no conflict. I thought the code would be easy to write, once I learned about that return value:

set<int> coll;
/* ... */
for (set<int>::iterator itr=coll.begin(); itr!=coll.end(); ++itr)
   if (*itr%2 == 0)
      itr = coll.erase(itr);

I liked this solution. No tricks, easy to write, easy to read. But it crashed. Why? Because erase() could return coll.end(), which the for loop would try to increment, which wasn't allowed. So the solution looks like this:

set<int> coll;
/* ... */
set<int>::iterator itr=coll.begin();
while ( itr!=coll.end() )
{
   if (*itr%2 == 0)
      itr = coll.erase(itr);
   else
      ++itr;
}

The Microsoft version of STL implements map::erase() by returning an iterator, but this is non-standard. I therefore ran into trouble when I switched to STLport and map::erase() no longer returned an iterator. I found the solution buried in the STL source code, where I learned that the iterators in some collections are stable if they are incremented before erase() is called, in which case you end up with this result:

set<int> coll;
/* ... */
set<int>::iterator itr=coll.begin();
while ( itr!=coll.end() )
{
   if (*itr%2 == 0)
   {
      set<int>::iterator next = itr;
      ++next;
      coll.erase(itr);
      itr = next;
   }
   else
   {
      ++itr;
   }
}

Later I learned that the code could be simplified:

set<int> coll;
/* ... */
set<int>::iterator itr=coll.begin();
while ( itr!=coll.end())
{
   if (*itr%2 == 0)
      coll.erase(itr++);
   else
      ++itr;
}

Note that we can't use the "hack" from the beginning of this article, which would allow us to write coll.erase(itr--) and then increment the iterator in a surrounding for loop. The problem is that a scalar can decrement below zero, but an iterator cannot decrement before begin().

Personally, I don't like the autoincrement code above as much as when erase() returns a value. It relies on the reader understanding the "trick" and it makes it more difficult for collection classes to know in advance what to do. For example, this technique can't be used with a vector, which is why vector does return an iterator. This creates a confusing inconsistency. (Which is why the Microsoft version of map::erase() is non-standard, but much more consistent.)

The question is, why does this trick work? If you've spent much time in C++, you've inevitably learned about the dangers of code like this:

int i = 5;
i = i++;

The code could end with i equal to either 5 or 6 - the result is undefined because the C++ standard only guarantees that ++ will be executed after i is read and before the semi-colon. It could be incremented before or after the assignment happens. (If you are wondering why, remember that assignments in C++ can be embedded in expressions, unlike most languages.)

To reword the guarantee in the language of the C++ Standard, the only guarantee is that ++ will be executed after i is read and before the next sequence point. The sequence point is the key to understanding our call to erase().

In the expression above, the semicolon is the sequence point. Now look at our code example:

coll.erase(itr++);

If you apply the same evaluation rules, the undefined timing of the execution of ++ would mean that the iterator could be evaluated after erase() returns and the iterator had been invalidated. This would crash, but experience shows that it doesn't. The reason is that the function call to erase() introduces a new sequence point. The code is guaranteed to be evaluated as:

1. The value of itr is loaded and pushed on the stack for erase().
2. itr is incremented.
3. erase() is called.

All of which gives the desired result. QED.

"Invalid pointer" error in Visual Studio 2010 in Configuration Manager

I've been fighting against a problem I've seen in most of my project files that were upgraded from VS2005 or VS2008. Any attempt to delete a Configuration or a Platform results in the error "The operation could not be completed. Invalid pointer."

Today I was finally able to track down the problem, which is caused by some seemingly benign lines in the .vcxproj file. To fix the problem, open your .vcxproj file in Notepad, go to the end of the file, and you'll find some lines that look like this:

<ProjectExtensions>
  <VisualStudio>
    <UserProperties RESOURCE_FILE="Dot.rc" />
  </VisualStudio>
</ProjectExtensions>

Remove those lines, reload the project, and everything will start working fine. Those lines do not appear in project files that are created by the Visual Studio 2010 wizard, so I believe that they are not needed.
 
My complete bug report can be found on Microsoft Connect:
https://connect.microsoft.com/VisualStudio/feedback/details/575911/invalid-pointer-error-deleting-configurations-in-upgraded-vs2010-c-project-files#tabs