SharePoint Content Deployment

Recently I have deployed sharepoint content from 32 bit window 2003 to 64 bit window 2008, here are some pitfall I ran into:

• Target SharePoint server has to be configred to accept "incoming content deployment jobs" and check "Do not require encryption" if SharePoint Central Admin site is not using SSL;
• Target SharePont site collection has to use blank template;
• Deploymentmanifest.xsd (under 12 hives\Template\xml on Target SharePoint Server) has to add the followings (this only requires for 32 to 64 bit mix)

<xs:attribute name="AllowAutomaticASPXPageIndexing" type="xs:boolean" use="optional" >
<xs:attribute name="ASPXPageIndexMode" type="xs:string" use="optional" > < xs:attribute name="NoCrawl" type="xs:boolean" use="optional" >
< xs:attribute name="CacheAllSchema" type="xs:boolean" use="optional" >

RSS Viewer web part bug

If you want to use RSS Viewer web part for private/authenticated feed, you need enable Sharepoint Kerberos authentication. Otherwise you get error: "The RSS webpart does not support authenticated feeds" even the feed from its own site.(update 03/31/2010:

• on window 2008, it can view authenticated feeds from its own site, but it is win2k8 only. on both window 2003 and window 2008 R2, it requires SPN registration (delegation not necessary) and Kerberos in order to view authenticated feeds;
• if both feeds and RSS Viewer on the same server, only consuming web application (RSS Viewer host) needs Kerberos even if feeds are from other web application with different application pool;(on window 2008, it only requires IIS kerberos setting, no SPN needed)
• the above apply for both moss and sharepoint 2010;

When you view a private feed, you may also get the following error:

"An unexpected error occured processing your request. Check the logs for details and correct the problem."


It happens when you use a non-default zone URL for sharepoint site: (update 03/31/2010: this appears not to happen on sharepoint 2010 beta2)



With AAM setting like this:






It works if using the default zone URL:
(update 03/31/2010) Reference: SharePoint 2010 and Kerberos by Spence Harbar

IIS 7 Kerberos authentication for SharePoint

IIS 7 has a new feature called Kernel Mode Authentication, it can be found off "Advance Settings.."

In order for SharePoint to use Kerberos authentication, it has to be disabled: (update 04/01/2010: sharepoint 2010 disable this by default!!)

This is necessary because Kernel Mode can't work with multi-server sysem where you can't register same SPN to multiple server accounts.

see here for IIS authentication negotiation process

NONONO In IIS 6, as long as NTAuthenticationProvider is set as "Negotiate, Kerberos", whether SPNs are registered or not, server granted Kerberos authentication. But IIS 7 seems to be of SPN awareness during negotiation regardless of Kernel Mode on or off: it only agree on Kerberos when the App Pool ID account has SPN registered, otherwise it falls back to NTLM.

add custom sharepoint web service in VS 2008

This article gives all you need to create a custom web serice in SharePoint 2007. But when adding a custom service in Visual Studio 2008, get the following error:



It turns out you have to append ?WSDL :



A custom sharepoint web service needs to be put in _vti_bin to be trusted. Otherwise either SharePoint trust level or CAS policy need to be modified.

SPGridView Paging and Filtering

A common issue with SPGridView is to enable paging: the PagerTemplate property of SPGridView needs to be set as null. If the SPGridView is declaratively defined in aspx page, the following code won't enable paging:

<sharepoint:spgridview id="SPGridView1" runat="server" >
<pagertemplate ></pagertemplate>
</SharePoint:SPGridView>

instead the following code works:

protected override void OnLoad(EventArgs e)
{
SPGridView1.PagerTemplate = null;
}

After paging is enabled, another common issue is, filter is off after navigating pages. I know filter is off acturally on every subseqent postback, like sorting. But to some degrees, it makes senses to turn it off since users are given no indication that a filter is on otherwise, which can certainly cause some confusion. But turning filters off on paging is very unsatifactory.

Inspired by the idea in this post, here is what I did to enable filtering across pages (based on .Net 3.5):

first cache filter settings :

protected override void OnPreRender(EventArgs e){

ViewState["FilterExpression"] = ObjectDataSource1.FilterExpression;

base.OnPreRender(e);

}

secondly, set filterexpression when postback is from pagings:

protected override void CreateChildControls() {

arg = (string)req.Form["__EVENTARGUMENT"];
if (arg.StartsWith("Page$") && ViewState["FilterExpression"] != null)

ObjectDataSource1.FilterExpression = ViewState["FilterExpression"].ToString();

}

related reference:

http://geekswithblogs.net/mnf/archive/2005/11/04/59081.aspx (.Net 2.0)

http://forums.asp.net/p/1067215/1067215.aspx (.Net 3.5)

Activating Sharepoint Timer Job

The best pratice to create a custom sharepoint timer job is to create a feature with web application scope and to instantiate a SPJobDefinition there. This is becuase it is application pool id's credentail that is used during feature activation, and only central admin pool id which is farm account has sufficient privilege.

When activating from other web application whose pool id is not farm account (by best practice, it should not), you may see one or both errors as follow:

• In browser, "unknown error" and in window event log, "EXECUTE permission denied on object 'proc_putObject' "

Cause: application pool id doesn;t (and should not) have write permission to config database

Workaround: assign application pool id as db_owner of sharepont config database

After that, you might get anohter error (if the app pool id is not sharepoint server local admin) when you try to activate (with permission on config database, you can now unactivate feature, but not activate)

• In browser, "HTTP 403 error (someone saying 404 error), and in ULS log, "...Microsoft\SharePoint\Config\bd189eb6-92d0-4ca5-87b0-770f542e3f0a\cache.ini' is denied"

Cause: Sharepoint need to cache timer jobs in WFE (so it doesn't need to get them from sql as often as every minute) and app pool id (in WSS_WPG group) doesn't have that file permission. In contrast, farm account in WSS_Admin_WPG has full permission on the folder of "C:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint".

Workaround: assign full control permission to application id for that folder or add app pool id into WSS_ADMIN_WPG group.

But the real solution is to activate timer job feature in the central admin, which means you need to create a feature of web application scope.

SharePoint databases part 2

part 1 outlines sharepoint databases and database backup. This part will focuse on secrity: what sharepoint accounts have access to sharepoint databases and in what roles.

SharePoint_config database:

• install account is its dbo
• farm account (and local admin) in db_owner role
• application pool account in WSS_Content_ApplicationPoolid role

Central_Admin database:

• same as config except that local admin is not in db_owner role

Content database:

• farm account is dbo
• app pool account and ssp service account are in db_owner role

SSP (and SSP Search DB):

• same as content database, plus search service account is in db_owner

Server Roles:

• Install account has dbcreator fixed server role & securityadmin fixed server role.
• Farm account has the same fixed server role, but it is automatically configured.
• other service only has public server role.

Understanding those and sharepoint application pool id (see this) can help to solve a lot sharepoint database permission issues such as :EXECUTE permission denied on object 'proc_putObject' in event log tells that the application pool id doesn't have write permission on configure database.

HTTP authentication on redirect

I recently had a hard way understanding Http authentication particularly involving redirect. After googling and conversing with a good friend, Alan, here is the picture on what will happen when server A (or browser)makes a http request to server B which redirects to server C:

• Server A: Request.Credentials = CredentialCache.DefaultCredentials Sent request to Sever B
• Server B: returns 401 – List of login schemes supported From Secured web page
• Server A: Sends Request + Request.Credentials (formatted according to the selected login scheme) to Server B
• Server B: returns 302 – "Redirect to Server C"
• Server A: Request sent again to Server C
• Server C: return 401 – List of login methods supported From New secured web page
• Server A: Request + Request.Credentials (formatted according to the selected login scheme) to Server C
• Server C: returns 200 – Content From New secured web page

Why auth handshake always happen even if the credential is set? This is by design, because Httprequest clients have no idea what auth schema the remote server is using. You can avoid first auth handshake by manually setting HttpRequest's header, like: req.Headers.Add("Authorization", "basic " + base64); But this needs to be done every time even for same Uri-Pref. Generally allowing handshake and setting PreAuthentication=true is better. See this post for details.

When making HttpRequest, HttpRequest object need to have a CookierContainer, which will be used by HttpResponse to fill in. It doesn't do anything with authentication though.

Another thing related is, in case of impersonating, Kerberos (both authentication and delegation) is required in order to forward default credentials (double hoppings).

core.js in sharepoint master pages

in sharepoint default.aspx file, core.js is loaded as deferred:
<sharepoint:scriptlink language="javascript" name="core.js" Defer="true" runat="server" / >

in layouts/application.aspx, it is loaded without deferred:
<sharepoint:scriptlink language="javascript" name="core.js" runat="server" >

The impact of this difference is, you can override javascript functions in application pages (layouts pages), but you can't do that to site pages.

SharePoint Page Provision

In SharePoint, there are 3 kinds of pages you can provision:

Application Page-- Pages in _layouts/ directory

Site Page-- ghosted site pages such as default.aspx

pages loaded in sharepoint document libraries


For pages in sharepoint document library, no server code is allowed by default, unless you change PageParserPaths in web.config. See this blog for details.

For both Application Page and Site Page, you can add server code in one of 3 ways:

• inline code (code render or declartion blocks)
• add user controls, for example: <%@ Register TagPrefix="XXX" TagName="lookup" Src="~/_controltemplates/LookupDefaultValue.ascx" % >
• inherits from code behind. Such as : <%@ Page inherits="CodeBehindDemo" % >. The trick is SharePoint dones't allow either CodeBehind or Src attributes. You have to use dll instead:

<%@ Assembly Name="NIAID.ListEventSetting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=a1c61a650ab8c476"% > <%@ Page Language="C#" Inherits="NIAID.ListEventSetting.ListEventPage" MasterPageFile="~/_layouts/application.master" % >

controls or methods used in aspx page need to be defined in the code behind with either protected or public access.

for site page, the dll also need to be in the safe control list in web.config.

Another option, less desired, is to use  CodeFile attribute in Web Site Project. It requires putting .cs source files into the layouts directory. Some people even developed user controls with CodeFile in layouts or controltemplate and use SmartPart or self-developed webpart loader (loadControl) for web part development.

Configure and Test SharePoint Incoming Email in Window Server 2008

I have built a Win2k8 based sharepoint 2007 farm (1 WFE, 1 Index and 1 SQL), and have its incoming email configured as follows:

1.install SMTP in both WFE( for incoming email) and Application Server (for outgoing email);

2.check services in CA to make sure WFE is running "incoming email service" and Application Server is running "outgoing email service";

3. Create a DNS MX record such as:
(same as parent folder) MX [10] spweb.sp.local or
moss MX [10]spweb.sp.local
this essentially tells spweb.sp.local is the mail server for the domain of sp.local or subdomain moss.sp.local. read here for more details.

4. Configure mail server's SMTP domain list to include an alias of sp.local (for normal user email address, used to test SMTP extension/pop3) and an alias of moss.sp.local for sharepoint email enabled lists:
5. Configure "incoming setting" in CA

• allow farm to receive email
• specify email address such as moss.sp.local

6. Configure list setting to allow list get emails

• specify email address for a sharepoint list, such as sharedDoc@moss.sp.local
• check "yes" on "Save original e-mail"

7. Configure an email client (outlook).

• install SMTP extension (pop3). This is a necessary (and hard) part since Win2k8 doesn't have POP3 built in.
• open window firewall at port 110 (for pop3)
• create an outlook account for testing:
  * Email address ends with one of SMTP domain alias list such as admin@sp.local
  * setting smtp server (not confused with sp outgoing eamil server) and pop3 server (iP address, not Netbios)
  * user password to logon to pop3 server (defined in smtp extension's configure.xml located in "c:\ProgramData\ppedv\visendosmtpextender")

8) Test first if sending and receiving email work okay in outlook by sending some email to itself, and then test sending email to a sharepoint list such as shareddoc@moss.sp.local.

Overall, the very simiplied picuture is: mail client asks its local SMTP to send an eamil to shareddoc@moss.sp.local)-> outlook's local SMTP server ask DNS,who is mail server for remote domain sp.local ? -> find MX record from DNS ->logon to remote mail server SMTP-> check for mail domain moss.sp.local: if it doesn't exisit, mail sending fails there. If it does exist, mail goes to SMTP's drop folder and stay there waiting for SharePoint Timer Job to pick up and put into the assocaited list. For regular user email, mail client using POP3 will do the pickup job, and the only difference is regular user email has its individual subfolder under the drop folder root.

Moss 2007 and Server 2008 window firewall

I have installed MOSS 2007 on a farm of 3 window 2008 servers:

• WFE: running web application,search query,and incoming email service
  
• Application: running Central Admin, Indexing (no query), outgoing email service and all other services
  
• SQL: sql 2008

There are serveral problems all related with window firewall:

• can't browse Central Admin from anywhere other than Application server
  

Window Firewall only open Http at port 80. open an exception at CA port in App server to fix this problem.

• Office Search Service or SSP can't be configured

Sharepoint SSP use web service call at port 56737. open exceptions on both WEF and App servers to fix this problem. If no query server is specified, configuration "looks like" it can go through, but "check service enabled in farm" disclose problems. Be sure to check it.

• Index Propagation fails

Sharepoint use SMB (File Shareing) protocol to push index from Index to Query server (WFE in this case). Open "File and Printer SharePoint" exception to fix this problem.

reference: http://blogs.msdn.com/joelo/archive/2007/02/13/protocols-ports-and-firewall-rules.aspx

http://blogs.msdn.com/uksharepoint/archive/2009/01/05/sharepoint-ports-proxies-and-protocols-an-overview-of-farm-communications.aspx

http://support.microsoft.com/kb/962928/en-us

Window Server 2008, Hyper-V and Wireless Connection

In preparation for SharePoint 2010, I need to convert a 64-bit Vista Workstation into a server runing window 2008 and then run Hyper-V to create a farm of 3 servers: a Domain Controller, a SQL server, and one Web server. My another goal is to setup all wireless network connection for all VMs and my physical server. Here are the outlines on some major steps:

1) Convert Vista to Win2k8.
Booting with Win2k8 CD. There are choices of either formating disk or replacing Vista. I chose the latter since I don't need to install drivers again.


2) Setup wireless connection
First need to enable wireless feature for win2k8:
Secondly need to update wireless driver: no Window 2k8 wireless driver availabe, but I can use Vista's driver: the win2k8 installation process save all vista drivers in Windows.old folder. I use "Update Driver" option in Device Manager:
3) enable RDP.
Since I build this as my work platform, I need to access anywher via RDP:

• need to Change "Remote" Option: Computer->Properties->Remote Setting->Remote

3) setup Hyper-V

• enable Hyper-V role of Win2k8. This adds some dependent feature as well.
• turn on Hardware Assisted Virtualization: On Intel platforms, Turn VT on in the BIOS (on AMD platforms, AMD-V is on all the time) http://www.microsoft.com/windows/virtual-pc/support/configure-bios.aspx  http://www.sysprobs.com/disable-enable-virtualization-technology-bios

4) Create VMs
First time when I created VMs, they don't have any network connection even if I have wireless connection for my phsical server. It turns out Hyper-V doesn't support wireless adapters. The workaround is:

• create a virtaul network in Hyper-V's Virtual Network Manager and then bridge it with wireless connection (details).
• Then run Hyper-V to create VMs with the virtual network connection created above.

For DC vm I chose mini requirement: 512MB memory and 32G disk.
Since I also want to RDP from my phsical server to VMs ( I can certainly connect to them, but if I RDP to my server first and then connect to VMs, window key combination can't go through to VMs) , I enabled RDP as I did to my physical server. But to my surprise that only allows me to use IP address! When i use VM BIOS, it said it can't find the server! I looked at my router's DNS, the VM BIOS name is not there. and I can't add it (get "entry already exist" error) for some reasons. As a workaround I open Window FireWall for PING (see here on how to ping), and I can then RDP its BIOS name.
5) promote Domain Controller

• run dcpromo.exe (detail)
• disable IP6 for DC vm's connection and configure IP4:
since my server is From DHCP, DC vm need a static IP address and its DNS server should point to itself details

6) join other VMs to Domain
I have create other 2 VMs also runing Win2k8, and need to join them to the Domain I just created. To do so, the only trick is to set each VM's DNS server to DC/DNS VM IP(otherwise it will go to my home router's DNS instead), and then change WorkGroup to Domain just like changing computer's BIOS name. DC will add VM server into AD once it is joined, and DNS server will dynamically add VM's name into dns table once VM is connected.

by now I have a farm of 3 virtual servers running on a sub domain (or virtual sub net). It was a long weekend, but I am so happy about Hyper-V. It really rocks!

SharePoint Managed Path and IIS ISAPI

In SharePoint 3.0 there is no more extra ISAPI extension such as the one in WSS 2.0. All requests are routed to aspnet_isapi.dll instead.

The aspnet_isapi.dll will serve a request if a physical IIS path is found as a match. For example, in IIS you can create a subfolder out of root directory of sharepoint site, and you can browse files inside the subfolder. Doing this actually blocks this subfolder name as sharepoint managed path, since IIS handle requests first.

If no physical IIS path found, Http handlers and Http Modules which are defined in the web.config will start to knick off to handle http requests. At this point, the managed path info will be used to resolve URL.

Update 04/05/2010: IIS will do authentication,and after that the request is yielded to the SharePoint14Module, which will contact file system first and then content database for appropriate page. See this blog for details.

There is an exception: if you put files (other than folders) under iis root directory of sharpoint site, those files can not be served unless they are defined as Explicit Inclusion type of managed path. This seems to be the wss 2.0 "excluded" concept.

Update: this exception only applies to MOSS and WSS 3.0, it is fixed in SharePoint 2010. great fix, no more confusing.

SharePoint and WebDav

Web-based Distributed Authoring and Versioning, or WebDav is to allow clients to access web-based documents as if they are in client's local driver (accessible via window explorer: yoursite/DavWWWRoot ). SharePoint features such as Explorer View is an example, saving office files into sharepoint server is another example. IIS has WebDav service, however SharePoint doesn't use it, instead, SharePoint has its own built in. So there is no need to enable IIS WebDav service in a sharepoint server.
On the client side, there are some requirements:

• need Office 2007 installed;
• Windows XP: Web Client is installed and enabled (by default).
• Windows Vista: The Web Folder service needs to be started.
• Windows 2008 (and R2): The Desktop Experience feature needs to be installed.

In case of FBA SharePoint sites, clients also need to

• enable "client integration"
• enable Cookies by checking "Auto sign me"
• window XP need to install hot fix Webfldrs-KB907306-ENU.exe

SharePoint Users also need the following permission (this is a part of "contribute" level permission, be default assigned to "member" shaerpoint group) to utilize any WebDav related feature:

SharePoint Authentication and IIS

We know SharePoint relies on IIS to do authentication (unless FBA), and provides contents based on AAM configuration. When a request such as http://sp.company.com/ hits a WFE, the IIS select a site to authenticate the request. The selection process works as follows:

1. first it looks for the site listening to port 80 with header as sp.company.com. If it is found, it is selected;
2. Otherwise, it looks for the site without any header and listening to any unsigned ip address at port 80. If there is one, it will be selected for authentication;
3. If neither is found, IIS error out

Notice the URL doesn't have to point to the WFE. for example, in the case of Load Balancer, URL points to LB, not WFE.

After authentication succeed, It is up to SharePoint to serve the http request. SharePoint provides content solely based on its AAM configuration, i.e, it must have that specfic FQDN configured (in the prvious example, it is sharepoint.company.com), otherwise, it errors out.

Notice in some scenario, IIS site which does authentication can have a different FQDN than that in SharePoint AAM.

AAM reference: http://blogs.msdn.com/sharepoint/archive/2007/03/06/what-every-sharepoint-administrator-needs-to-know-about-alternate-access-mappings-part-1.aspx

SharePoint databases part 1

It is long time since I started to wonder what databases SharePoint is using, and today I finally spend some time on this and get some clarifications:

1) SSP database and SSP search database. These 2 databases are tied with your individual SSP, and they are backup and restored with SSP. For backup and restore SSP, see This KB;

2)Content Database for SSP site and/or MySite: When you create your SSP,if you setup individual web applications for your SSP site and/or MySite , you will have 2 more extra content databases. But they need to be backuped and restored individually. They are content databases.

3) Config database: this is most importanat database, and if moved to a different SQL server, the target sql server must have exact the same version number (SPs and KBs) applied to backup/restore operation, seems Log Shipping can overcome this restriction.

4)Content Database for the Center Admin Site: This database name can't be modified and must be backup and restored with Config database. (Config DB name can be changed in registry: My Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\Secure\ConfigDB)

5)WSS search database. I am not sure what this datbase is used in MOSS since there is SSP Search database, is this for WSS Search as a legacy from WSS 2.0?
but anyway, this database need to be backuped and restored individually. Actually every time when you restart Search Help in the central Admin, you need to recreate this database.

6)Content database for portal sites. This is the database normally referred as "content databse".

You can pretty much manually move (via SQL backup and restore) all databases but SSP related dbs, which will end with SSP related dbs stay in the old SQL server, and all others live in a new SQL Server. To move all sharepoint database? see this KB http://technet.microsoft.com/en-us/library/cc512725.aspx for SSP backup/restore and SQL Server Connection Alias. Be cautious, SQL Connection Alias can cause a lot confusions in the future.

update: there is another db which draws more and more attentions as performance killer: TempDB. This db is particularly related with sharepoint indexing/crawling. For db related performance recommendation, see this post.

Part 2 will focuse on sharepoint database permission.

Sharepoint Kerberos, really?

when you change authentication type to Negotiate(Kerberos) in SharePoint central admin, all sharepoint do is to change NTAuthenticationProvider of sharepoint site in IIS to "Negotiate, NTML" from "NTLM". (how to find NTAuthenticationProvider value? see this post. If there are multiple WFEs in your farm, check each of them as I experienced a problem seeing the failure of setting this IIS meta data from SharePont Central Admin UI.).

What does that mean? is Kerberos guaranteed? not really. Kerberose will be selected only if clients support Kerberos (such as "intergrated window authentication" checked in IE), and  SPN is registered (which make sure Kerberos is selected) and registered correctly (which make sure authentication will not fail)
But how do you know for sure you have Kerberos functioning? I usually go to EvntView of server and in security log find event id = 540, and if you have logon type =3 and AuthenticationPackage = Kerberos. you are good to go.

Create PDF out of password protected web pages

Recently a user tried to use Adob 9.0 to convert one web page from their intranet site into a PDF file, and got an authorization failure error even though she can browse the site in IE. And since she can sucessfully do this on their internet site, it seems that we need to give Adobe some sort of permission to read sharepoint site. But how?

We know IE can browse sites without requiring users to enter password becuase of IE's credential passthrough, Adobe, unlike most of MS office product, doesn't have such functionality. But we can use IE's PDF boolbar to achieve credential passthrough:

1.Browser IE to the desired secure site;
2. IE view->toolbar->adobe PDF;
3. “Convert” appears in the toolbar as follows:

4. Choose "Convert Web Page to PDF.."

you should be all set to go!

very large SharePoint Database Log files

It is not unusual that you may find sharepoint databases have very large log files, sometimes they can be hundreds times of size of data file. The cause of this is:

• the databases in "Full" recovery model, and
• the log files have never been backed up ( this happens when doing sharepoint native backup, it only backups data, not log)

When databases in a Full recovery model, neither full nor differential backup will truncate inactive segments in log files. Those segments thus can't be reused in the future, which causes the log file to continue growing without bound.

The solution is,

• turn recovery model to 'Simple' (for non-prod environment only), or
• specifically backup Log files by using the following sql statement:
• BACKUP LOG WSS_Content_80 TO DISK='z:\temp\80.log.bak'. The result of this is: when log backup is done, sql server automatically truncates all inactive transactions and allow those segments to be reused in the future.\
• do Transaction Log back up on database. The T-log backup will automatically truncate log.

However, the log file size will not be reduced until you shrink the log file size by sql statement:

• DBCC SHRINKFILE ('Log_Logic_Name')

you can find the log logical name by issue sp_helpdb or from UI.

So to summary, the best practice is

• Turn on "Simple" recovery model for Dev, and Keep "Full" model for Prod
• do sql backup rather than sharepoint backup
• full backup complemented by daily partial backup and then by hourly T-log backup