Apr 25, 2010

What happen behind screen for HTTP authentication, Kerberos or NTLM?

It is well known that during client-server authentication negotiation if clients send Kerberos token, kerberos will be adopted, otherwise authentication will fall back to NTLM. But from where clients get kerberos token, and how NTLM is chosen as a secondary choice? With helps from WireShark, I have found answers for those questions:

If server IIS is set up as "Negotiate, NTLM", on the first session, client first attempt to make anonymous HTTP request (without credential), server responds with 401 deny. After ACK,client makes a request to its KDC or Domain Controller for service token. One of the followings can happen then:
  • if KDC respond back with a service token, 
    • client will cache this token for subsequent sessions before its expiry time (10 hours by default)

    • client will make another HTTP request with service token.
    • if the token is accepted by server, server send back 200, connection is established
    • if the token is invalid, server deny again with 401 (if client is IE, IE will pop up a window for password, authentication reach an impasse)
  • if KDC respond back with an KRB error( for example, SPN is not found), client will send NTLM negotiation to server,
    •  server respond with NTLM challenge back to client
    • client make HTTP request with NTLM token
  • if KDC respond back with a kerberos account token (like krbTGT, not requested service token), that means this KDC can't find the requested service token, instead it designates a subdomain KDC
    • with this subdomin TGT token, client can makes requests to sub domain KDC for the same  service token
    • negotiation continue then
On subsequent session, since client already has token cache, it won't ask for KDC again until the token expires. This is why kerberos is considered not only secure, but also efficient.
At first, clients use a special TGT token to make requests to its KDC for service token. Clients get this special TGT token at login time. This TGT token also has a expiry time, and on Window 2008 Domain, it is automatically renews 30 minutes before expiry time. In a rare case (UPDATE: this case only happens to some xp sp2, sp3 has fixed this problem), with client kerberos cache corrupted, clients just send NTLM directly without even querying KDC for kerberos token. As a workaround, loging off/on or reboot client computer can solve this problem.