What "index" is this article asking Google to open? The index against which they run actual queries has to be tied to Google's core search algorithms, which I doubt they'd want to make public.
So would they open an "index" of web page contents? In this case, why would another search engine access Google's "index" rather than the original server? The original server is guaranteed to be up to date, and there's no single point of failure.
From my understanding, the premise is that the 'index' generated from each crawled site will be some set of metadata smaller than the site's actual content. So instead of many robots, each crawling through all the data on your site, there could be one bot, which updates a single (smaller) index that all search engines can access.
I agree that Google's index is probably optimized to work with their search algorithm. From what the author claims, though, this doesn't mean that Google would be losing anything by allowing other engines to use the index, as "all the value is in the analysis" of the index.
There's also significant value in knowing when to re-index sites due to changing conditions. For example, if some new iSomething is announced, re-indexing apple.com as well as a number of popular Apple-related news sites would be very helpful in keeping the index fresh. There's also feedback from the ranking algorithm in determining how often to re-index, how deep to index, etc etc.
There are two good reasons to access an index instead of the original server:
1) You won't DOS the host site.
2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time. Downloading a crawl dump from a central repository would be much easier.
> 2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time.
Yeah, consider a site like Hacker News, where the crawl delay is not 1 second but 30 seconds [1].
If you're trying to grab historical data like iHackerNews did [2], you might be better off hitting Google's webcache instead...except, to enumerate urls you have to scrape the search results pages, AFAIK.
This is why Google opening its webcache via an API is a GREAT idea.
I think the idea would be to have a global index of sites that sites use to mirror their data for searching.
This database would normally only accessible by search engines and the sites themselves could then disallow direct bot search in their robots.txt.
It occurs to me that this might have to be "invite only" - Google invites the sites they trust to put their data there but if they catch someone "cheating" in one way or another, they stop indexing. Plus they wouldn't have to invite really small sites.
So would they open an "index" of web page contents? In this case, why would another search engine access Google's "index" rather than the original server? The original server is guaranteed to be up to date, and there's no single point of failure.