Now that the basics of MSDTC have been covered in Part 1 we can move on to troubleshooting more specific issues. Here I cover other tricks that can ruin your day. There was a period of time where it felt like all I did was diagnose and fix MSDTC issues. This is the outcome of that frustration – a guide that you may find helpful and prevent the stress I experienced. So without further ado…

General MSDTC Troubleshooting

Before deep diving let’s make sure the following basics are satisfied:

1. Make sure the DTC service is started and running as a network service on all nodes participating in the distributed transaction.
2. Confirm network DTC access is enabled – see Part 1
3. Verify NetBIOS and IP address of each computer
   • Ping the machines involved in the distributed transaction from each other
     • Verify that each machine involved in the distributed transaction can ping its partner by Host Name and the verified IP Address is returned.
     • Verify that reverse name lookup of the IP address associated with the NetBIOS name on each computer resolves to the correct computer name.
       
       Transact-SQL

       ping -a "IP Address associated with client computer NetBIOS name"

       1	ping -a "IP Address associated with client computer NetBIOS name"

   • Use a table like this to store the output

   

   • Additionally you can consider adding an entry in the HOSTS file for quick name resolution (default C:\Windows\System32\drivers\etc)
4. Check firewall isn’t blocking any of the systems engaged in the distributed transaction
   • Software like Symantec Endpoint Protection can sometimes block RCP traffic
   • To determine if the Windows Firewall is enabled, run the following from the command line
     
     Transact-SQL

     netsh advfirewall show allprofiles state

     1	netsh advfirewall show allprofiles state

   

   • If the firewall is on then add an exception for the MSDTC service
5. Check for restricted RPC ports
   1. Open Component Services > Expand to MyComputer > properties
   2. Choose the Default Protocols tab
   3. Choose properties for Connection Oriented TCP/IP
   4. If there are any entries in the dialogue box for COM Internet Services then RPC port restrictions are in place
      1. Take note of the range and adjust the firewall to allow them through

      

      

      1. NOTE: the RPC Endpoint Mapper runs on port 135, required for DTC – ensure a firewall exception exists for it
6. Conduct DTCPing tests
   1. DTCPing can determine if a client may communicate over RPC with another system running the RPC server component of the test
      1. NOTE: NetBIOS names should be used instead of FQDN names
   2. Start the DTCPing tool on both nodes
   3. After both processes are up you can select one system to start the test
   4. Enter the remote server’s NetBIOS name and select Ping
   5. If for any reason you enter something invalid or an exception is thrown you’ve got to restart the whole test. There is no refresh button!

No Transaction Is Active

Sometimes your application may return an error that looks like this:

Error Login failed for user ‘NT AUTHORITY\ANONYMOUS LOGON’. OLE DB provider “SQLNCLI10” for linked server “TheServer” returned message “No transaction is active.”

The Event Viewer logs will show something like this:

In my experience situations like this mean it is usually necessary to uninstall and reinstall the MSDTC service on the machines in the distributed transaction. On some occasions you may discover that ports necessary for MSDTC were closed off by the firewall and need to be corrected there. The Microsoft utility DTCPing can be used to identify this.

Cloned VMs

Sometimes when a VM is cloned from a template (VMware or Hyper-V) the CID can be duplicated between the two machines. Evidence of this can be seen in the Event Viewer (Event ID 4101). This means the two MSDTC services will not be able to communicate with each other.

A possible error message is:

The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction

The MSDTC feature of the Windows operating system requires unique CID values to ensure that MSDTC functionality between computers works correctly. Disk duplicate images of Windows installations must have unique CID values or MSDTC functionality may be impaired. This can occur when using virtual hard disks to deploy an operating system to a virtual machine.

Cannot Start MSDTC

After uninstalling and reinstalling MSDTC, the service may not start. It may throw an error like this:

Additionally, this error may occur:

An error occurred while processing the last operation.

Error code 8004e00F – Com+ was unable to talk to the Microsoft Distributed Transaction

Coordinator

The event log may contain additional troubleshooting information.

In this case some necessary registry values may be missing. Here’s an example of that scenario.

HKEY_LOCAL_MACHINE\Software\Microsoft\MSDTC\

• HKEY_LOCAL_MACHINE\Software\Microsoft\MSDTC\
• AllowOnlySecureRpcCalls[1] REG_DWORD
• FallbackToUnsecureRPCIfNecessary[0] REG_DWORD
• TurnOffRpcSecurity[0] REG_DWORD

HKEY_LOCAL_MACHINE\Software\Microsoft\MSDTC\Security

• HKEY_LOCAL_MACHINE\Software\Microsoft\MSDTC\Security
• AccountName[NT AUTHORITY\NetworkService] REG_SZ
• DomainControllerState[0] REG_DWORD
• NetworkDtcAccess[0] REG_DWORD
• NetworkDtcAccessAdmin[0] REG_DWORD
• NetworkDtcAccessClients[0] REG_DWORD
• NetworkDtcAccessTransactions[0] REG_DWORD
• NetworkDtcAccessInbound[0] REG_DWORD
• NetworkDtcAccessOutbound[0] REG_DWORD
• NetworkDtcAccessTip[0] REG_DWORD
• XaTransactions[0] REG_DWORD

If there are missing values then recreate them, set to the default value (it is in [brackets]), and run:

The Distributed Transaction role might need to be re-installed. In this case add the role for Distributed Transactions and check these boxes:

Afterwards continue to test, check, and troubleshoot the MSDTC service.

How to Add a Windows Firewall Exception for MSDTC

Should the need arise for a firewall exception this is how do it in Windows Server:

1. Click Start, click Run, type firewall.cpl, and then click OK to display the Windows Firewall dialog box.
2. Click Allow a program through the Windows Firewall to display the Windows Firewall Settings dialog.
3. Go to the Exceptions tab of the Windows Firewall Settings dialog box.
4. Add Program to display the Add a Program dialog box.
5. Browse and navigate to %system32%\msdtc.exe.
   1. Launch a command prompt, type echo %system32% and press Enter to determine the location of the \System32 directory on this computer.
6. Select msdtc.exe and click Open.
7. Change scope to specify the set of computers for which MSDTC communications should be allowed and click OK.
8. Click OK in the Add a Program dialog box and click OK in the Windows Firewall Settings dialog box.
9. Close the Windows Firewall dialog box.
10. Stop and restart the Distributed Transaction Coordinator service.
    1. Launch a command prompt, type net stop msdtc and press Enter.
    2. After the Distributed Transaction Coordinator service has stopped, type net start msdtc and press Enter.

How to Use MSDTCPing

Instructions for how to use MSDTCPing utility are here.

1. Download DTCPing.exe (see References)
2. Extract the files
3. Use the Readme.txt file that is included in the DTCPing.exe download to test Remote Procedure Call (RPC) and Distributed Transaction Coordinator (DTC) communication from Server1 to Server2. If this test is successful, run the test from Server2 to Server1
   1. Note that if RPC cannot flow in either direction, MS DTC communication fails in both directions. If RPC communication fails, the DTCPing window (on either server) displays this failure, which is also saved in the associated dtcping.log file. See the Readme.txt file for more information. If the test fails in either direction and the log indicates the failure is in RPC communication, continue to the next step. If the test fails in either direction and the log indicates the failure is in DTC communication, continue to step 9 below.
4. If RPC has failed in at least one direction (for example, from Server1 to Server2), direct your firewall administrator to make sure that the Internet Control Message Protocol (ICMP) is open in both directions
   1. NOTE: You can typically determine if RPC has failed by reading the dtcping.log file.
   2. By default, ICMP is port1. You can verify this in your protocol file, which is located in the %windir%\WinNT\System32\Drivers\ folder. Ping Server2 by NetBIOS name from Server1. If the ping fails, continue to the next step. Otherwise, continue to step 8.
5. Ping Server2 by IP address from Server1 to make sure that the correct port is open for a ping on the firewall. A Network Monitor trace can verify this. If the IP address ping succeeds and the NetBIOS name ping fails, there is a name resolution problem.
   1. NOTE: You can use the ipconfig /all command to retrieve the IP address or the IP addresses of a server.
   2. A quick way to test name resolution is to make an entry in the Hosts file of the client server. This is the server on which the NetBIOS name ping fails. You can model your entry after the sample entry that is included in the file
   3. NOTE: You must only make an entry in the Hosts file for troubleshooting purposes. If the new entry corrects the name resolution problem, remove the entry from the Hosts file, and make the entry you must in the DNS, the WINS server, or the LmHosts file.
6. If pinging Server2 from Server1 by NetBIOS name fails, or if pinging Server2 from Server1 by NetBIOS name succeeds but the DTCPing test shows RPC communication still fails, it is possible that Port 135 (the End Point Mapper, or EPM) has not been opened bi-directionally on the firewall. Check the firewall to make sure that the EPM is open in both directions. At this point, a Network Monitor trace may help to pinpoint the problem.
7. You only reach this step if the DTCPing test indicates RPC communication works in both directions. If DTCPing indicates no errors in either direction, then RPC and MS DTC communication is flowing properly.
8. If DTCPing indicates that DTC communication has failed in at least one direction (for example, from Server1 to Server2), direct firewall administrators to verify that the ports are open that the developer specified when the developer went through the MS DTC configuration article (see step 3). Additionally, some rules may be applied to the firewall that prohibits RPC callbacks for either (or both) servers. A Network Monitor trace may help to troubleshoot this particular scenario.
9. If DTCPing returns an error message similar to the following:
   

   Unexpected: My session guid is same as partner’s guid

   1. Check whether the current server has been duplicated or cloned from the other server. If so, locate the HKEY_CLASSES_ROOT\CID key in the registry. Under this key, you may notice more than one GUID. Locate the GUID whose underlying Description key is MSDTC. Note that this GUID is also listed in the DTCPing output window. If the other server has a GUID that is exactly the same for MS DTC in its registry, you must create a new GUID for MS DTC in one of the registries. You can use GuidGen to do this.
   2. After you add this new GUID, and also all of its underlying keys to HKEY_CLASSES_ROOT\CID, make sure to delete the old GUID that it is replacing.

Examples of a successful DTC Ping:

Reference Material

This is a bigger subject that I once thought. Below is a summary of some helpful links for troubleshooting MSDTC issues. May you have better luck that I did!

1. Enable Network DTC Access
2. Troubleshooting MSDTC Communication Checklist
3. Troubleshooting Problems with MSDTC
4. How to Troubleshoot MSDTC Firewall Issues
5. Configuring Microsoft Distributed Transaction Coordinator (DTC) to work through a firewall
6. Download DTCPing.exe
7. Troubleshooting MSDTC Issues with the DTCPing Tool
8. How to configure the MSDTC service to listen on a specific RPC server port
9. The hidden tool – MSDTC Transaction Tracing
10. View Trace Data
11. Download Windows Driver Kit 7.0
12. How to configure RPC dynamic port allocation to work with firewalls
13. Disaster Recovery for MSDTC on Windows Server 2003 and 2008
14. Troubleshooting RPC Endpoint Mapper errors using the Windows Server 2003 Support Tools from the product CD

If you have any horror stories from working with MSDTC I’d love to hear them!

 

---

Like what you are reading? Please subscribe!