Why Does the System Display Error Code 0x1104 When I Log In to a Windows ECS?

Symptom

The system displays an error message indicating that a protocol error (code: 0x1104) is detected when you use MSTSC to access an ECS running Windows Server 2008.

Figure 1 Protocol error (code: 0x1104)

Possible Causes

• Port 3389 of the security group on the ECS is disabled.
• The firewall on the ECS is disabled.
• Port 3389 on the ECS is used by other processes.
• The Remote Desktop Session Host is incorrectly configured.

Solution

1. Check security group settings.

   Check whether port 3389 is allowed in inbound direction. If it is allowed, go to 2.

2. Check whether the firewall is disabled:
   1. Log in to the Windows ECS.
   2. Click the Windows icon in the lower left corner of the desktop and choose Control Panel > Windows Firewall.

      

   3. Click Turn Windows Firewall on or off.

      View and set the firewall status.

      

   If the firewall is enabled, go to 3.

3. Log in to the ECS using VNC and check the port.
   1. Open the cmd window and run the following command:

      netstat -ano |findstr: 3389

      Figure 2 Checking port 3389
      

      As shown in Figure 2, port 3389 is used by the process with ID of 4.

   2. Open Task Manager and find the process with ID of 4 is the System process.
   3. Generally, the IIS and SQL Server run as the System process. Run the following HTTP command for further check.

      netsh http show servicestate

      Figure 3 Checking System process
      
   4. If port 3389 is used by HTTP protocols, it indicates that the port is used by IIS.
   5. Enter http://127.0.0.1:3389 in the address box of the browser and press Enter. Check whether the website can be visited normally.
   6. Change the port used by IIS and restart IIS.

4. If no error occurs during the preceding steps, go to step 5 to check whether error 0x1104 is caused by the configuration of Remote Desktop Session Host.
5. Check the remote desktop session host configuration.
   1. Log in to the ECS using VNC.
   2. Open the cmd window and enter gpedit.msc.
   3. Click OK to start Local Group Policy Editor.
   4. Choose Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services.
      Figure 4 Remote Desktop Services
      
   5. Remote Desktop Session Host > Security.
      Figure 5 Remote (RDP) Connection requires the use of the specified security layer
      
   6. Set Require use of specific security layer for remote (RDP) connections to Enabled and Security layer to RDP.
      Figure 6 Setting security layer
      
   7. Click OK.
   8. After the configuration is complete, open the cmd window.
   9. Run the following command to update the group policy:

      gpupdate

      Figure 7 Updating the group policy