Display Unicode in a Windows console window with printf/wprintf

Did you know that you can print Unicode to a console window under Windows? I didn't. Here are some blog entries that discuss how to do it. You need Visual Studio 2005 or later to display Unicode using the C run-time library. Thanks to Microsoft C++ MVPs Jochen Kalmbach and Igor Tandetnik for introducing me to these solutions:

http://blog.kalmbachnet.de/?postid=98

http://blogs.msdn.com/b/michkap/archive/2010/10/07/10072032.aspx

http://blogs.msdn.com/b/michkap/archive/2008/03/18/8306597.aspx

http://illegalargumentexception.blogspot.com/2009/04/i18n-unicode-at-windows-command-prompt.html

With the techniques in those articles, you can display both UTF-16 and UTF-8 Unicode characters in a console window.

There are some caveats:

• The console window must be set to a TrueType font. By default it is set to a raster font, which does not support Unicode characters.
• Not all languages are supported. Lucida Console, the most common console TrueType font, will display Russian characters, but not Arabic or Hebrew.

This article discusses the issues with Hebrew characters and provides some workarounds:
http://groups.google.com/group/microsoft.public.win32.programmer.international/browse_thread/thread/788eaf7082088b35

How to: Debug with Code Center Premium Source (In Detail)

This article is an extensive update to the MSDN article of the same name. I saw that article two months ago and it was only with the help of Microsoft C++ MVP John Czopowik that I was able to make Windows Source Debugging work under Visual Studio 2010. I hope this article saves other people from the same frustration.

These instructions are for developers who have a Windows source code license and access to Code Center Premium.

To prepare for debugging with Code Center Premium

1. Close all copies of Visual Studio that are running (this is required or VS will overwrite your registry changes.)
2. Connect your SmartCard reader and insert the card you obtained from the Shared Source Initiative.
3. Windows Vista and Windows 7 will need to contact Windows Update to install a Gemalto driver. Make sure that you allow this to happen. (Windows XP: See the instructions that came with the card reader.)
4. Make sure your smart card certificate is working by opening Internet Explorer and going to https://codepremium.msdn.microsoft.com/. You should see prompts as shown in Steps 3 to 6 under Sample Walkthrough, below. This must work properly before you proceed with these instructions. Do not use Firefox because Firefox does not support smart card authentication!
5. Open the Registry Editor and go to: HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\10.0\Debugger
6. Set both SecureSourceLocalDirectory and SourceServerExtractToDirectory to the same directory. The directory does not have to already exist.
7. Launch Visual Studio.
8. On the Tools menu, click Options.
9. In the Options dialog box, open the Debugging node and click General.
10. Clear the Enable just my code (Managed only) check box.
11. Select Enable Enable source server support.
12. Select Print source server diagnostic messages to the Output window.
13. Clear Require source files to exactly match the original version.
14. Under the Debugging node, click Symbols.
15. In the Symbol file (.pdb) locations box, clear the Microsoft Server Symbols check box and add the following location:
    
    https://codepremium.msdn.microsoft.com/symbols
    
    Move this location to be second in the list to ensure that these symbols are loaded first. (You can't move it to the top. Microsoft Symbol Servers is always first, but you should have unchecked it.) Visual Studio 2005 and 2008 users: In the screen shot below, note the third entry in this list, C:\Windows\symbols\dll. This directory is where the MFC and C run-time symbols are installed by Visual Studio. These symbols are not part of Code Center, so you need to include them if you want to debug into MFC or the C run-time library. Visual Studio 2010 users: You don't need the reference to C:\Windows\symbols\dll. Visual Studio always looks there, even if it's not in this list.

16. In the Cache symbols in this directory box, enter a location such as C:\symbols where Code Center Premium can cache the symbols. This location should be DIFFERENT from the location used to store public symbols when Microsoft Symbol Servers is checked because symbols already downloaded from the Microsoft Symbol Servers will prevent symbols being downloaded from Code Center Premium.
17. Click OK.
18. Right-click the Solution under Solution Explorer and choose Properties:
    
    
19. Under Common Properties, select Debug Source Files.
20. Add a link for the desired operating system (complete list at the end of this article.) For example:
    
    https://codepremium.msdn.microsoft.com/source/Windows%207/RTM
    
    Note that spaces must be replaced with %20, and you cannot browse to this location in your browser.
    
    
21. In the same window, make sure that the list under Do not look for these source files is empty.
22. Click OK.
23. Close and restart Visual Studio to ensure that settings are persisted.

Sample Code

Here is sample code that I used for testing. It's a native code console application that can be tested under x86 or x64:

#include "stdafx.h"

#include <windows.h>
#include <shlwapi.h>

#pragma comment(lib, "shlwapi.lib")

int _tmain(int argc, _TCHAR* argv[])
{
  // Can't step into this function. You get a "not indexed" error.
  //char buf[256]={'\0'};
  //::PathRemoveExtensionA(buf);

  // Set a breakpoint on this function, then try to Step Into it.
  GetConsoleMode(0,0);

  // Stepping into this function should work too, but not on XP.
  //UnregisterApplicationRestart();

  return 0;
}

Sample Walkthrough

1. Set a breakpoint at GetConsoleMode in the sample code.
2. Start debugging (F5).
3. You should see the Confirm Certificate window. If not, your smart card isn't working.
   
4. Click OK.
5. You'll be asked to enter your PIN. You received this in your email confirmation that welcomed you to Code Center Premium.
   
6. Enter your PIN and click OK.
7. It may take a while to get started because the private symbols must be downloaded. Since they contain full debug information, they are much larger than the usual symbols.
8. When Visual Studio stops at the breakpoint, look at your Modules window. Here's how it appears on my Windows 7 64-bit machine: (click it to see it full size.)
   
   
9. Note that kernel32.dll shows "Symbols loaded", whereas KernelBase.dll shows "Symbols loaded (source information stripped). If kernel32.dll also shows "(source information stripped)", then you are using the wrong symbols. The most likely cause is if you didn't set a different cache directory for private symbols.
10. Also note that the timestamp for both kernel32.dll and KernelBase.dll is 7/13/2009. This means that they are RTM DLLs for Windows 7. The timestamp for ntdll.dll is 3/23/2010, which means that it's been updated and you can't step into it. So it is possible to debug on a patched system, but it's hit or miss because many DLLs have been patched and so can't be stepped into.
11. Now you will see a window asking to execute an untrusted command named SD.EXE:
    
    
12. This command will be executed by the Source Server at Code Center Premium. It will NOT be executed on your computer. You do not need to have SD.EXE on your computer.
13. Click Run.
14. You should now be looking at Windows source code.
15. If you look at the Output window, you should see messages such as the following: (some NDA information removed.)
    
SRCSRV: sd.exe -p windowsdepot.sys-ntgroup.ntdev.microsoft.com:2016 print -o "C:\Users\JimB\AppData\Local\SOURCE~2\WIN_WINDOWS\win7_rtm\windows\yyy\yyy\yyy\yyy.c\1\yyy.c" -q //depot/win7_rtm/windows/Core/xxx/xxx/xxxxx.c#1

SRCSRV: Source server cannot retrieve the source code for file 'd:\w7rtm\windows\zzz\zzz\zzz\zzz.c' in module 'C:\Windows\System32\kernel32.dll'. The system cannot find the file specified.
16. Note that Source Server is actually showing an error. In spite of this, debugging still works properly.

Things That Don't Work

• You can't debug into 32-bit code running under a 64-bit operating system. In other words, you can't debug into any DLL that lives in \Windows\SysWOW64. (Remote debugging into a 32-bit virtual machine running under a 64-bit operating system is okay.)
• You can't debug any hot fixes, security fixes or operating system updates. You can ONLY debug the RTM release or a service pack. In practice, this usually means that you are remote debugging a virtual machine that was installed from the DVD and Windows Update has been turned off.
• You can't debug when running Visual Studio under Remote Desktop because the smart card doesn't work when you are logged in under Remote Desktop. (LogMeIn and GotoMyPC are not tested.)
• The debugger won't step into some Windows functions. This usually means that you are trying to step into a DLL that's been updated since the original release.
• Some functions are in files that Code Center Premium can't find. One example is PathRemoveExtensionA in shlwapi.dll. For files that can't be found, you'll see a "not indexed" error in your Output window:
  
  SRCSRV: o:\w7rtm.obj.amd64fre\shell\shlwapi\srca\objfre\amd64\patha.cpp not indexed

Troubleshooting

Everything has to be exactly right for source code debugging to work properly. If things aren't working, here are some ideas:

• Is your smartcard working?  (Repeat Step 4 under Prepare for debugging, above.)
• Did Visual Studio overwrite your registry entries? This usually leads to the error "SRCSRV: ... The system cannot find the file specified." (Repeat Step 5.)
• Did the file get added to the list of Do not look for these source files? (See Steps 18 through 22.)
• Are you debugging a 32-bit app on 64-bit Windows? You can't debug Windows source in this case. Run the 32-bit code on a 32-bit version of Windows in a virtual machine and use Remote Debugging.
• Are you trying to step into the Windows kernel? Kernel debugging is different from user mode debugging. You must use WinDbg for kernel mode debugging.
• Is the source code missing on Code Center Premium? This would be the case if you are getting the error SRCSRV: ... not indexed. Please see the section below titled ANSI versus Unicode Functions.
• Are you seeing a warning about an "Untrusted Command"? This is normal. Just click Run.
• Can't step into a Windows API function? Make sure you have symbols for the DLL. Look in the Modules list and make sure that it does NOT say "Source information stripped." If it does, clear your symbol cache and run your app again.

Running Visual Studio in a Virtual Machine

You can install Visual Studio 2010 on your Windows 7 clean machine and run Visual Studio 2010 in the virtual machine so you don't have to use remote debugging. However, you must use a virtual machine that supports USB devices, such as VMware or Virtual PC in Windows 7. Virtual PC 2007 will not work (not even with SP1.)

In Virtual PC on Windows 7, go to the USB menu and select USB SmartCard Reader. You will be prompted to install the drivers in the virtual machine. The smart card will be DISCONNECTED from the host system, so you will not be able to use it in the host machine until the virtual machine releases it.

You do not need to do this if you are using Remote Debugging to run your application in the virtual machine.

ANSI versus Unicode Functions

Debugging into the "A" and "W" versions of functions frequently doesn't work. Here is the explanation from C++ MVP Jochen Kalmbach:

The problem with " PathRemoveExtensionA" seems to be a "build feature". They implemented the file only once for A and W and then you uses this file without "A, W"... In this case, the source file is called "path.c" and is referenced as "patha.c" (A) and "pathw.c" (W).

If you want to debug into this files, you need just to copy the source from the CCP-Website for the "TCHAR" version and save it wherever you want with the name "patha.c" and/or "pathw.c". Then if VS askes you for the file, you can just use this file. It will exactly match.

It seems that this file is generated during the build-process, because it seems that it is also compiled from the output directory...

Source Server Paths

(Thanks again to C++ MVP Jochen Kalmbach for providing this list.)

Here is the list of all currently available source server paths. Links are not clickable because these links are only for use in Visual Studio, not in your browser.

Win7 RTM: https://codepremium.msdn.microsoft.com/source/Windows%207/RTM


W2k8 RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/RTM

W2k8 Hyper-V-RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/Hyper-V%20RTM

W2k8 SP2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/SP2

W2k8 R2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202008/R2


W2k3 RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/RTM

W2k3 R2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20R2

W2k3 SP1: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20SP1%20RTM

W2k3 SP2 RC: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20SP2%20RC

W2k3 SP2: https://codepremium.msdn.microsoft.com/source/Windows%20Server%202003/Windows%20Server%202003%20SP2%20RTM


Vista Beta1: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/Beta%201

Vista Beta2: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/Beta%202

Vista RC1: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/RC1

Vista RTM: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/RTM

Vista SP1: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/SP1

Vista SP2: https://codepremium.msdn.microsoft.com/source/Windows%20Vista/SP2


XP RTM: https://codepremium.msdn.microsoft.com/source/windows%20XP/RTM

XP SP1: https://codepremium.msdn.microsoft.com/source/windows%20XP/SP1

XP SP2: https://codepremium.msdn.microsoft.com/source/windows%20XP/SP2

XP SP3: https://codepremium.msdn.microsoft.com/source/windows%20XP/SP3

XP 64-bit:  Not available.


W2k Datacenter RTM: https://codepremium.msdn.microsoft.com/source/Windows%202000/Datacenter%20RTM

W2k SP3: https://codepremium.msdn.microsoft.com/source/Windows%202000/SP3

W2k SP4: https://codepremium.msdn.microsoft.com/source/Windows%202000/SP4