Computing PageRank in a Distributed Internet Search Engine System Yuan Wang and David J. DeWitt   View Paper (PDF)   Return to RESEARCH SESSION 11:DISTRIBUTED SEARCH AND QUERY PROCESSING Abstract Existing Internet search engines use web crawlers to download data from the Web. Page quality is measured on central servers, where user queries are also processed. This paper argues that using crawlers has a list of disadvantages. Most importantly, crawlers do not scale. Even Google, the leading search engine, indexes less than 1% of the entire Web. This paper proposes a distributed search engine framework, in which every web server answers queries over its own data. Results from multiple web servers will be merged to generate a ranked hyperlink list on the submitting server. This paper presents a series of algorithms that compute PageRank in such framework. The preliminary experiments on a real data set demonstrate that the system achieves comparable accuracy on PageRank vectors to Google's well-known PageRank algorithm and, therefore, high quality of query results. ©2005 Association for Computing Machinery