As part of correcting issues in an isolated lab environment, it is often necessary for developers to connect to one or more machines within the isolated network.  While the concepts of remote debugging are pretty straight forward, I decided to go ahead and put together something that our developers could use as a guide to perform those types of activities.  Our codebase is 100% managed code, for whatever reason, it is more difficult to get attached and debugging across domains with managed code than it is with Native code.  Honestly, I don't understand why that is, but it can be a bit frustrating unless you fully understand how to get such an environment configured properly.

The first thing you need to do is install the Visual Studio Remote Debugging tools (x86 or x64) on the target machine.  The install is pretty straight forward, so I won't go into that directly.  We actually include the remote debugging monitor (MSVCMON) as part of our base lab images.  In our case we don't configure the tool to run as a service, instead, we place a short cut on the desktop so that it can be started when needed.

Once of the key aspects of remote debugging is that you have to use the same user account on your developer machine, as well as the target machine.  This means Local Accounts on both machines.  So the first step is to create a local account on your desktop, lets call it "Remote.Debugging" for our purposes.  This account doesn't necessarily need to be in any special local groups.  Create the same user name and password on the target machine.  This time, however, you need to put the account in the target machines Administrators group (the local administrators group, not the domains).

Once you have the accounts created, you need to update the permissions in the Visual Studio 2010 Remote Debugger. Once opened, use Tools->Permissions and add the account as part of the security.  Below is a screen shot of what you should see when you do this.

Now you are ready to attach and debug, should the need arise. Notice that the user signed into the desktop is from the isolated domain called "LAB", and the user name is Lab.Administrator.

When you need to debug, simple hold the shift key down and right-click on the Visual Studio 2010 Remote Debugger Icon on the desktop, you will get a pop up menu that looks something like this:

Notice that there is a option for 'Run as Different User'.  Select that, and you are presented with a credentials pop up window as follows:

It is important that you enter the local machine name in the windows credential dialog (because the account is local to this machine. A little known shortcut is that you can use the "." as the machine name for local accounts.

 Once you have successfully authenticated, you will have a Monitor window like the one below.  Notice that the user name now indicates Remote.Debugging@LabDesktop01.

This means that you are now ready to connect from Visual Studio.  On your workstation, follow the same process, hold the shift key down and right-click on the visual studio shortcut, you will see a pop up menu like this

 As before, select Run as Different user.  When prompted enter the credentials of the local account 'Remote.Debugging'.  If this is the first time you are performing this procedure, you will be asked the various initial-setup questions, such as your default environment settings, etc.

Generally, I open the solution that has the executable that I am debugging so that I have access to all of the source code within the environment.  If you are opening a project that is corrected to TFS, you will get a pop up asking you for your TFS credentials.  This is normal because you are running as the local user you created. Once you have opened the solution, and are ready to attach, select Debug->Attach to Process.

Enter the machine name (you know the long with that starts with LAB_), or the IP Address and press enter.  Once attached, within visual studio, you will get to select the process.

 You can select the "Show processes from all users" checkbox to see the other processes.  Once you find your process select it and click Attach.  Notice that the "Attach to" field currently indicates Automatic: Native Code; If you have connected as outlined here, this will change when you select your process to whatever is suitable.  Over in the Lab, the Visual Studio Remote Debugging monitor shows this

 Notice that both screens show that you are attaching as the user created specifically for this purpose.

And now a brief word about security...  Since many organizations have multiple developers, I recommend that you disable the account you created on your developer workstation until you need it for debugging.  This helps prevent unauthorized access to your computer.

After spending about a week working on some client issues, I was finally able to resume working on configuring a Lab environment to use multiple Isolated Networks for CI Builds (more on that in another future post once I am satifsfied I have it working properly).  When I restarted the Lab Environment (which was working last week), I received errors indicating that the TFS service account was now somehow invalid.

Much to my dismay, I spent several hours trying to figure out what the happened.  After searching around in all the usual places, I had come up empty.  I could tell it was going to be a long night...

Before I explain how I corrected this issue, a little additional background will probably be helpful.  The Hyper-V hosts in this particular environment are in a separate domain, lets call it 'XPServerFarm', and the "normal" servers all run in a separate domain; lets call it 'business.local'.  There is a two-way full trust between the two domains.  I suspect that it is a pretty normal configuration, although it does provide some challenges sometimes when it comes to permissions.

Reading the error message details, it looked to me like the TFS Service account did not have permissions on the host, I signed onto the host that has the VMs on it, and checked the local administrators group.  The account was in it.  Lots of things went through my mind at this point; but I'll refrain from sharing some of those thoughts and instead focus on some of the things I tried. 

Testing this involved shutting down the lab environment, and starting it multiple times; each time waiting for the environment to "settle" into a final state, often times this can take as long as 10 minutes depending on what is configured on the various lab VM machines.

I removed the account from the administrators group and re-added it; I don't know what I expected, but there was no change in the behavior.  I looked at the server and it had been running for about 70 days; I decided that I would reboot it (this server is only for testing so why not?); again no change.  Honestly, I really didn't expect the behavior to change...  Then I opened up Active Directory FSMO for XPServerFarm, which also happened to be the server that these VMs were on, and verified that the Trust itself was still intact.

At this point I was starting to get concerned... perhaps the problems were because it was on the AD FSMO, but I continued down the path of the verifying account issue itself.  Finally, I decided to log into the server as the TFS service account... there had to be something I was missing, configuration-wise, but why would it show up now?  I Remote Desktop-ed directly to the Hyper-V host computer and...

I reconnected as a local administrator on the Hyper-V host. Sure enough, the time had drifted to be about 12 minutes difference between the two domains.  Interestingly I had previously setup the time synchronization but apparently there was a problem.  I checked the configuration, and I found that the Time Remaining was 0 when I used the command

w32tm /query /peers

I waited a bit, and executed the command again... no change.  I updated the Time Service on the Hyper-V host as follows. I decided to include the details because I ran into an issue here as well (nothing is ever easy).

I unregistered the the time service using this command:

w32tm /unregister

Then I rebooted. When the host came back up, I re-registered the time service using this command

w32tm /register

Followed by attempting to start the time service 

Net start w32time

I could not get the service to start.  I was getting an error indicating that there was some sort of SID issue because the processes running in the svchost were using incompatible SIDs.  System Error 1290. I hunted around the internet and found some nonsense posts about changing the tapiserver's registry entry... instead I looked at all of the registry entries and discovered that they were all running with the same type of account (LocalSystem, LocalService or NetworkService); so that obviously wasn't it.  I rebooted because I remembered reading somewhere that that way svchost initializes itself includes some sort of registration for the processes running inside svchost.exe.  After rebooting, the time service started, and the problems were gone.

Now, I configured the time service to use the same time servers as the Business.Local domain using these commands

W32tm /config /syncfromflags:manual /manualpeerlist:”0.pool.ntp.org, 1.pool.ntp.org, 2.pool.ntp.org”
W32tm /config /reliable:yes
Net stop w32time
Net start w32time

Once completed, I issues these commands on both servers to make sure everything was working correctly.  

W32tm /config /update
W32tm /resync
Net stop w32time
Net start w32time

After waiting a few minutes, the time on the servers synced up and I was ready to continue. I went back to Lab Management Center and tried starting the environment.  After a period of time, the environment settled with this status:

After clicking Repair Testing Capability on the pull-down menu associated with the Testing Capabilities, I waited and was eventually greeted with a happy environment once again.