- first it looks for the site listening to port 80 with header as sp.company.com. If it is found, it is selected;
- Otherwise, it looks for the site without any header and listening to any unsigned ip address at port 80. If there is one, it will be selected for authentication;
- If neither is found, IIS error out
Notice the URL doesn't have to point to the WFE. for example, in the case of Load Balancer, URL points to LB, not WFE.
After authentication succeed, It is up to SharePoint to serve the http request. SharePoint provides content solely based on its AAM configuration, i.e, it must have that specfic FQDN configured (in the prvious example, it is sharepoint.company.com), otherwise, it errors out.
Notice in some scenario, IIS site which does authentication can have a different FQDN than that in SharePoint AAM.