Cygwin/X - Problems with XDMCP due to firewall software.

[Harold L Hunt II <huntharo at msu dot edu>]

Some users have been reporting that they cannot get an XDMCP login 
screen on a remote *nix box from their Windows XP machines.

I was unable to figure out what was going on here until I enabled the 
Windows Internet Connection Firewall (ICF) for my notebook when I was 
out of town.  Upon returning I could no longer get a login screen for my 
Linux box.  I then remembered that I had enabled ICF for the adapter 
that I was trying to use to connect to the Linux machine via XDMCP.

I then went into the ICF properties and enabled logging of dropped 
packets.  I tried again to get a login screen and got the following in 
my log file:

=======================================================================
#Verson: 1.0
#Software: Microsoft Internet Connection Firewall
#Time Format: Local
#Fields: date time action protocol src-ip dst-ip src-port dst-port size 
tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info

2003-12-23 01:52:08 DROP TCP 192.168.0.1 192.168.0.123 43195 6000 60 S 
2971670008 0 5840 - - -
=======================================================================

Now, for a little background on XDMCP.  XDMCP contacts the remote XDM 
server on UDP port 177.  Then, the remote XDM server attempts to make a 
connection back to your Windows box on TCP port 6000.

The problem here is that typical firewall software will expect an 
outgoing connection to respond back on the same port.  That is, an 
outbound connection to UDP port 177 would make the firewall allow an 
incoming connection from the remote host on UDP port 177.  However, XDM 
makes an incoming connection to your Windows box on TCP port 6000, which 
the firewall software (ICF or another firewall package) does not expect 
and does not allow.  So, the firewall software denies the incoming 
connection.

The error profile here is perfect.  Do the following to see if this is 
your problem:

1) Regardless of whether or not you think you have a firewall problem, 
launch XWin.exe with the proper -from and -query parameters.

2) Wait two or three minutes.

3) Cygwin/X will exit and you will see something like the following 
towards the end of /tmp/XWin.log:

=======================================================================
Fatal server error:
XDMCP fatal error: Session failed Session 225104017 failed for display 
windows-host:0: cannot open display

winDeinitClipboard - Noting shutdown in progress
winDeinitMultiWindowWM - Noting shutdown in progress
=======================================================================

4) You should see something like the following in your *DM log file on 
your *NIX machine, such as /var/log/kdm.log:

NOTE: This error message will not show up until XWin.exe has run for two 
or three minutes and shut itself down.

=======================================================================
Dec 23 02:10:40 *nix-host kdm[16484]: Hung in 
XOpenDisplay(windows-host:0), aborting
Dec 23 02:10:40 *nix-host kdm[16484]: server open failed for 
windows-host:0, giving up
Dec 23 02:10:40 *nix-host kdm[12545]: Display windows-host:0 cannot be 
opened
=======================================================================


In summary, it looks like the error message in /tmp/XWin.log actually 
comes from the *DM service running on the remote *NIX host when it is 
unable to make a return connection to your Windows host.


If you match this error profile you need to figure out if you have 
Internet Connection Firewall, another Windows firewall product, or a 
firewall in the network between your Windows host and your *NIX host. 
If you have a firewall product installed on your Windows host, try 
disabling it for just a few seconds to try making an XDMCP connection; 
if it works, you need to consult your firewall documentation to figure 
out how to allow incoming connections on TCP port 6000 from your remote 
*NIX host.  If you have a firewall box somewhere on the network path 
between your Windows host and your remote *NIX host, then you need to 
either configure it to allow connections as above, or work with your 
network administrator to fix the problem.

As a side note, this whole situation explains why I was able to get at 
least one user to be able to make a connection to the "echo" service 
running on UDP port 177 on his *NIX host.  That worked fine, but the 
return connection to TCP port 6000 on his Windows host was failing 
because of either a firewall on his Windows machine or somewhere in the 
network between his two machines.


I hope this helps somebody and this will go into the FAQ someday.  This 
is only going to get worse with Windows XP SP2, since it enables the 
Internet Connection Firewall by default; on the other hand, it does have 
some new features that sound like they may alleviate our troubles.  For 
example (I have to read this again, so don't quote me), there is a 
feature that allows incoming connectinos from a remote host for 3 
seconds after an outgoing connections is made; this may or may not allow 
our incoming connection on TCP port 6000 to be accepted after our 
outgoing connection on UDP port 177 is made.

Harold