If you are getting an authentication prompt, thenthe PIX is fine.
If you're using L2TP with IPSec, then UDP 500 is needed for IKE Negotiations and while Protocol 50 for Encapsulating Security Payload and/or Protocol 51 for Authentication Header is needed. User credentials are sent over the later protocols as data. If NAT is involved anywhere, then UDP 4500 is used for L2TP with IPSec.
For PPTP, the control channel is over TCP 1723 and user credentials are sent over Protocol 47 Generic Routing Encapsulation. If you are getting a challenge for credentials, this this means that the ISA Server is receving your credentials and rejecting them - it could not reject the credentials if it did not receive them.
What authentication method are you using on the ISA Server? RADIUS or regular Windows authentication?