When an unauthorized request comes in, the entire request is URL encoded, and added as a query string to the request to the authorization form, so I can see where this may result in a problem given your situation.
According to MSDN, the correct element to modify to reset maxQueryStringLength in web.config is the <httpRuntime> element inside the <system.web> element, see http://msdn.microsoft.com/en-us/library/e1f13641.aspx. Try modifying that element.