this is a problem we had at a large social network i used to work with. launching a directory of users primarily for google's consumption was something that was difficult to scale for our huge size of database. The solution for us was node.js