I'm running into a similar issue. We are running ISA 2004 SP1 on Win2K3. We currently have OWA, RPC over HTTP (for Outlook), SharePoint and a our ERP Web interface published through ISA. All of them work fine. ISA is not a member of the domain.
*Just a side note, SharePoint and the ERP Web app are using RSA authentication.*
I've set up an identical rule to SharePoint and the ERP Web app to publish an application called Changepoint. I've set up RSA auth as well.
*note, Changepoint only works with IE.*
*note, The odd thing about Changepoint, and it's inherent to the app, when you go to the URL it redirects you and brings up the application in a new window, therefore leaving you with 2 windows open. Popup blockers do get in the way, and in my testing all popup blockers are disabled.*
Now for the odd behavior...
With integrated auth set in IE:
Inside the network it works fine, no prompt.
Outside the network the integrated auth fails. I get the RSA auth, that works, but on the attempt to go to the web app, I get page cannot be displayed.
If I turn off integrated auth in IE, I get RSA, then I get windows prompt. Upon password entry it works fine.
In testing, even though Firefox is not supported for the app, when I set integrated auth in Firefox I get the RSA, then it actually works like it's supposed to. While I don't actually get the application, I get the first screen before the popup redirect. That tells me that it's authenticated.
Whew!....sorry for the long winded explanation.
So, my question is, am I missing a setting somewhere? Is this an ISA bug? Is it choking on the 2 browser windows that Changepoint uses upon login? Or, if anyone is familiar with Changepoint, is it something with that app?
Oh, not setting integrated auth is not really an option, since we have 2 other web apps that use it and it works. Our user base would complain...
Any insight into this would be very helpful.