- Site permissions: Use Remote Interfaces. It will prevent web service (SOAP), WebDav and SPD connection to sharepoint sites;
- List permissions: View Application Pages. It will prevent anonymous users from view list form pages such as allitems.aspx, edititem.aspx etc
Anonymous users will be challenged or prompted when they try to access resources which are not granted by the "limited access" role. Anonymous users can access sharepoint resource only by server object model, not by web service (either asmx or wcf) regardless whether site is anonymous enabled or whether lockdown mode is turned on. The only workaround is for readonly web service as described in the following 2 posts:
exception: for readonly and SOAP version 1.1(asmx): modify SOAPAction Header:
http://mdasblog.wordpress.com/2010/03/18/allowing-anonymous-access-with-sharepoint-web-services-and-spservices/
http://weblogs.asp.net/jan/archive/2009/05/25/quot-the-security-validation-for-this-page-is-invalid-quot-when-calling-the-sharepoint-web-services.aspx
Otherwise web service call must carry a valid credential. However if web services doesn't involve any object model, anonymous users can call those kinds of web service even if those web services are hosted inside sharepoint.
WCF services add another level of security with security binding configurations for each endpoint. WCF endpoint security along with hosting IIS authentication setup together will determine if anonymous users can invoke WCF services. See here for details.
If the web service is custom web service and using sharepoint object model, the web service need to be in viti_bin, or sharepoint CAS policy need to change. As any web service with sharepoint object model involved, they are not open to anonymous users any way.
In sharepoint 2010, at web application level, there are 2 new policies: Users Policy and Anonymous Policy, which can deny individual uses or all anonymous user's access to an anonymous web application.
Anonymous users can't access to sharepoint application pages under _layouts folder as most sharepoint application pages inherit from LayoutPageBase which is a secure page, nevertheless some application pages inherit from UnsecuredLayoutPageBase such as searchresults.aspx, login.aspx etc, which are open to anonymous users.