Windows Server - Active Directory DNS broken, NTDS Global Catalog error 1126

Asked By Ronny on 09-Jun-10 10:13 AM
Briefly: DNS is broken, and an inability to talk to an apparently healthy global catalog seems to be the core reason.

One of our 2 domain controllers started throwing disk errors earlier today, so I built a new DC, but could not add it to the domain (Logon Failure). On checking I found that the global Catalog was held by the remaining server, so turned off the dud box and used ntdsdiag to sieze control of the other FSMO roles and do a metadata cleanup. However, I still couldn't join the domain.

Investigation showed a handful of remaining references to the dead server (cleaned up AD Sites & Services and deleted the appropriate name servers from the interface DNS configuration). However, when I tried to delete the old server from DNS, I was refused. I believe this was because the old server had previously been domain primary.

In researching this I found a suggestion to uninstall and reinstall DNS, which I've done; however I find that now I cannot re-attach the DNS to Active Directory. Any attempt to do so gives an on-screen error saying "The zone cannot be created. The data is invalid" and refuses to create the zone.

At the same time Event Viewer tags the following errors (all NTDS Global Catalog errors):
1869 - Active Directory has located a global catalog in the following site....
1655 - Active Directory attempted to communicate with the following global catalog and the attempts were unsuccessful... Additional Data Error value: 5 Access is denied.
1126 - Active Directory was unable to establish a connection with the global catalog.... Additional Data Error value:
8430 The directory service encountered an internal failure. / Internal ID: 3200c89

The global catalog described here is the server itself, i.e.the local system. For a while I thought this was a replication issue (trying to replicate with the dead server) but I can't see any evidence of that.

dcdiag /v looks like this:

Domain Controller Diagnosis

Performing initial setup:
   * Verifying that the local machine ebitnswdc04, is a DC.
   * Connecting to directory service on server ebitnswdc04.
   * Collecting site info.
   * Identifying all servers.
   * Identifying all NC cross-refs.
   * Found 1 DC(s). Testing 1 of them.
   Done gathering initial info.

Doing initial required tests

   Testing server: SYD-51DruittSt\EBITNSWDC04
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         The host ee8bb430-9cee-4d41-ad3c-1d902fb063a2._msdcs.ebit.com.au could
not be resolved to an
         IP address.  Check the DNS server, DHCP, server name, etc
         Although the Guid DNS name
         (ee8bb430-9cee-4d41-ad3c-1d902fb063a2._msdcs.ebit.com.au) couldn't be
         resolved, the server name (ebitnswdc04.ebit.com.au) resolved to the IP
         address (10.3.3.82) and was pingable.  Check that the IP address is
         registered correctly with the DNS server.
         ......................... EBITNSWDC04 failed test Connectivity

Doing primary tests

   Testing server: SYD-51DruittSt\EBITNSWDC04
      Skipping all tests, because server EBITNSWDC04 is
      not responding to directory service requests
      Test omitted by user request: Topology
      Test omitted by user request: CutoffServers
      Test omitted by user request: OutboundSecureChannels
      Test omitted by user request: VerifyReplicas
      Test omitted by user request: VerifyEnterpriseReferences
      Test omitted by user request: CheckSecurityError

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : ebit
      Starting test: CrossRefValidation
         ......................... ebit passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ebit passed test CheckSDRefDom

   Running enterprise tests on : ebit.com.au
      Starting test: Intersite
         Skipping site SYD-51DruittSt, this site is outside the scope provided
         by the command line arguments provided.
         ......................... ebit.com.au passed test Intersite
      Starting test: FsmoCheck
         GC Name: \\ebitnswdc04.ebit.com.au
         Locator Flags: 0xe00003fd
         PDC Name: \\ebitnswdc04.ebit.com.au
         Locator Flags: 0xe00003fd
         Time Server Name: \\ebitnswdc04.ebit.com.au
         Locator Flags: 0xe00003fd
         Preferred Time Server Name: \\ebitnswdc04.ebit.com.au
         Locator Flags: 0xe00003fd
         KDC Name: \\ebitnswdc04.ebit.com.au
         Locator Flags: 0xe00003fd
         ......................... ebit.com.au passed test FsmoCheck
      Test omitted by user request: DNS
      Test omitted by user request: DNS

Results of dcdiag /test:dns follow...

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: SYD-51DruittSt\EBITNSWDC04
      Starting test: Connectivity
         The host ee8bb430-9cee-4d41-ad3c-1d902fb063a2._msdcs.ebit.com.au could
not be resolved to an
         IP address.  Check the DNS server, DHCP, server name, etc
         Although the Guid DNS name
         (ee8bb430-9cee-4d41-ad3c-1d902fb063a2._msdcs.ebit.com.au) couldn't be
         resolved, the server name (ebitnswdc04.ebit.com.au) resolved to the IP
         address (10.3.3.82) and was pingable.  Check that the IP address is
         registered correctly with the DNS server.
         ......................... EBITNSWDC04 failed test Connectivity

Doing primary tests

   Testing server: SYD-51DruittSt\EBITNSWDC04

DNS Tests are running and not hung. Please wait a few minutes...

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : ebit

   Running enterprise tests on : ebit.com.au
      Starting test: DNS
         Test results for domain controllers:

            DC: ebitnswdc04.ebit.com.au
            Domain: ebit.com.au


               TEST: Basic (Basc)
                  Error: No LDAP connectivity
                  Warning: adapter [00000007] Broadcom NetXtreme 5751 Gigabit Co
ntroller has invalid DNS server: 10.3.3.82 (<name unavailable>)
                  Error: all DNS servers are invalid
                  Error: The A record for this DC was not found
                  Warning: The Active Directory zone on this DC/DNS server was n
ot found (probably a misconfiguration)

               TEST: Forwarders/Root hints (Forw)
                  Error: Root hints list has invalid root hint server: b.root-se
rvers.net. (128.9.0.107)
                  Error: Root hints list has invalid root hint server: h.root-se
rvers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: l.root-se
rvers.net. (198.32.64.12)

            TEST: Records registration (RReg)
               Error: Record registrations cannot be found for all the network a
dapters

         Summary of test results for DNS servers used by the above domain contro
llers:

            DNS server: 10.3.3.82 (<name unavailable>)
               1 test failure on this DNS server
               Name resolution is not functional. _ldap._tcp.ebit.com.au. failed
 on the DNS server 10.3.3.82

            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.63.2.53

            DNS server: 128.9.0.107 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.9.0.107

            DNS server: 198.32.64.12 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 198.32.64.12

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
               ________________________________________________________________
            Domain: ebit.com.au
               ebitnswdc04                  PASS FAIL PASS n/a  PASS FAIL n/a

         ......................... ebit.com.au failed test DNS

ipconfig /all looks like this:

Windows IP Configuration

   Host Name . . . . . . . . . . . . : ebitnswdc04
   Primary Dns Suffix  . . . . . . . : ebit.com.au
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : ebit.com.au
                                       com.au

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom NetXtreme 5751 Gigabit Controller
   Physical Address. . . . . . . . . : 00-13-20-07-AF-A5
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 10.3.3.82
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.3.3.252
   DNS Servers . . . . . . . . . . . : 10.3.3.82
   Primary WINS Server . . . . . . . : 10.3.3.82

Any ideas of exactly why this (in particular, DNS registration) is failing would be most welcome. I suspect I may otherwise wind up hand-coding the DNS for our zone.

If it matters, we have a second domain in a trust relationship ("they" trust "us" but not vice-versa). However that domain seems to be unaffected except in that ebitnswdc04 is now knocking back their zone transfers.

Also if it matters, the authentication functions of the domain seem to be fine. I can log into that server (and other servers) without any problems. I should add that this was *not* the case before I yanked the server with the failed disk. However, the inability to resolve host names is going to be a problem... to the extent that netbios fails to fill the gap I suppose.

Also if you need additional diagnostics I'm happy to supply them. I tried to cover the high points here but I've undoubtedly missed something important. A couple of pointers to start would definitely be appreciated...

...Ronny Cook
Ronny replied to Ronny on 15-Jun-10 07:13 PM
This is fixed now per feedback from Microsoft; the problem was that the secure channel between DNS and AD needed to be reset. The command to do this is:

nltest /sc_change_pwd:domain.name

where domain.name is the full version of the domain, in our case ebit.com.au.

This requires the version of nltest on the Windows 2003 CD (in one of the CAB files under support/tools).

...Ronny
Blesson replied to Ronny on 18-Jun-10 06:16 AM
Thanks for that command.
We had the same problem and there was only one domain controller in our env. All the solutions posted in the other sites were to shift the services to backup domian which was not possible in our env.

But ur solution really worked and thanks a ton for posting it here

Regards,
Blesson,
Bahrain