INNOBLOG

This past weekend, Wikipedia founder Jimmy Wales new startup, Wikia, announced that it had acquired an innovative web crawler called Grub, moving it a step closer to offering an open-source search engine. But is the Grub acquisition a step in the right direction?

Grub works differently than the webcrawling systems used by search leaders such as Google or Yahoo!. Rather than maintain a giant bank of computers that "crawl the internet, indexing it as they go, Wikia can use Grub to spread out the computing burden. Anyone can download the client and contribute their own processing power and bandwidth to the effort.

The innovation isnt just a cost-cutting measure. According to Grubs website, "Almost 50% of the database a search engine uses is either out-of-date, or incomplete at any given time. For this reason, Jimmy Wales says that "search is broken.

I wont argue with that fact that 50% of a search engines database is either out-of-date, or incomplete. Im not a search guru and really dont know. But as a user of search engines like Google and Yahoo (and, occasionally Windows Live), I have some doubts as to whether the comprehensive index Wikia hopes to create is really the way to innovate in search.

Grubs stated goal of creating and maintaining an up-to-date, comprehensive internet index is an innovation that consumers cant absorb. Based on my own experience, I think that the Googles and Yahoos of the world do a fine job of indexing. The pages they leave out are probably not the ones most people want to find. Most peoples frustrations with search arent related to incomplete or out-of-date indexes, but probably have more to do with the difficulty of finding the right combination of keywords that bring up the desired search results.

Grubs index just isnt an innovation in the right direction, it wont be valuable to the end user. What Wikia needs for success is a better search experience for the end user. It might need to be better at semantic search, or maybe it needs to leverage an open community like Wikipedia does in order to give search results a human touch. The problem is that both have been tried, at least in part, before. Will Wikia be able to succeed in an area where so many others have failed?

Wikia is supposed to launch "the first stab of something by the end of year. I suppose well see how innovative it is then.