You'll struggle trying to get this to work while providing a nice user experience at the same time. Windows is going to want to inject its login dialog if the person doesn't have access to the page and will deny them access to the page altogether which denies you the ability to write code to react to the failure. It is just the nature of using windows authentication. And, if you don't have windows authentication enabled, the credentials are not captured and made available to your server side code.
What you can try is setting up a page that requires windows authentication but rather than have the visitor go to that page, you use a WebRequest class to attempt to access the page from the server side code of your Forms Authentication page. If the windows authentication fails, you should be able to trap the error and just prompt the user to enter via the forms authentication. I've not done this myself, so I'm not sure whether windows will force the dialog anyway. It shouldn't but I'm not sure.