Are you trying to publish ISA FBA or FBA from your web server? If ISA, then you need to ensure that the web listener configuration is correct and that either the web listener or the publishing rule requires the user to authenticate. You should also delegate authentication back to the web server using something like Basic Auth to give a good user experience.
If you are looking to use FBA from your web server, then you need to build the ISA listener and rule to not do the authentication. You also then need to publish the paths to the FBA pages to allow those to be sent to the client when the main site does the redirect to get authentication.