We have a horrible, but crucial, application that our staff needs in order to be able to interface with a UNIX server. We previously had the UNIX server behind the firewall, and thus the application could connect directly to the server. However, a portion of the web catalog service would not work within the firewall no matter what we tried. so we decided to move the server outside.
Now... this application uses TCP Port 5100 for everything. When you start the program, it connects to the server, and if everything is ok, it asks for a username and password. when you type this stuff in, it sends it to the server. Now the mess begins. This application must download 4 or 5 .dat and .zip files that contain preferences. It creates a NEW connection for each download, but the connection is always to the remote server's TCP port 5100. These additional connections wait, and I believe eventually close, but the initial connection remains open, and the user does their work over this connection. These clients are FWC. So, I added a protocol definition allowing TCP 5100 Outbound, and added it to a protocol rule that any staff member can use. I have verified the FWC is using the protocol rule by looking in the FW log.
Now, what happens is, I open the program, it makes the initial connection, and i get the prompt for the username. I enter this, and it begins the login process. I notice it opens a second port to 5100 remote, and I see the progress bar that you should see when it begins to download the preference files, however, the progress bar stays at 0%, and the program sits there like that. You have to kill the program to exit. I look in the firewall logs, and nothing out of the ordinary shows up. it shows the right program, the right remote host, the right protocol, CONNECT is the operation, and 0 and 20000 are the statuses that you see (0 is completed successfully and 20000 is terminated normally), and it is using the appropriate rule.
so i tried adding some custom application configurations to the firewall mspclnt.ini:
And that didn't seem to have any effect. playing around, at the moment it looks like this:
Still no difference.
If i go to a workstation outside the firewall, it works fine. of course.
I did look in the packet filter logs, and i do see entries that are coming from the IP of this UNIX server, they are ICMP 3 (port unreachable) packets. obviously i'm sure that has something to do with it, however, they are being sent to the IPs of the DNS service running on my ISA server (I had to put external DNS on the same server as ISA cuz of budget constraints).
I've found that setting the workstation as a S-NAT client works, but I don't want to go that route unless there's a dire need.
Anyone have any ideas?