Cannot connect to database master at SQL server at ..... The database might not exist, or the current user does not have permission to connect to it.This happened even after I opened firewall exceptions for sqlservr.exe or whatever port the named instance is running at! As that was what I did to the default instance and I never had problem connecting to the default instance. So what is the issue with the named instance?
In SQL server, I run sql profile and found no connection even being attempted from the sharepoint wizard, so it seems most likely a firewall issue. As a testing, I turned off window firewall, and it works immediately!
But I can't leave the firewall off, so I have to find out what else I need to open in the firewall. After some googling, I understand how sql named instance works differently from default instance: the named instances need handshaking when a connection is requested (as client requests normally don;t include port number, but just instance name), and it is SQL Browse Service which helps client find port number at which the named instance is running at. The service name is sqlbrowser.exe and it is running at UDP port 1434.
With that, opening firewall for either .exe or UDP port 1434, the problem goes away, and I got a first look at SharePoint 2010 RTM after so many months with Beta. It is so exciting!