|
|
|
                                          
|
Return to Session IIa: Test collections
Forming test collection relevance judgments from the pooled output of multiple retrieval systems has become the standard process for creating resources such as the TREC, CLEF, and NTCIR test collections. This paper presents a series of experiments examining three different ways of building test collections where no system pooling is used. First, a collection formation technique combining manual feedback and multiple systems is adapted to work with a single retrieval system. Second, an existing method based on pooling the output of multiple manual searches is re-examined: testing a wider range of searchers and retrieval systems than has been examined before. Third, a new approach is explored where the ranked output of a single automatic search on a single retrieval system is assessed for relevance: no pooling whatsoever. Using established techniques for evaluating the quality of relevance judgments, in all three cases, test collections are formed that are as good as TREC.
@inproceedings{1009001, author = {Mark Sanderson and Hideo Joho}, title = {Forming test collections with no system pooling}, booktitle = {SIGIR '04: Proceedings of the 27th annual international conference on Research and development in information retrieval}, year = {2004}, isbn = {1-58113-881-4}, pages = {33--40}, location = {Sheffield, United Kingdom}, doi = {http://doi.acm.org/10.1145/1008992.1009001}, publisher = {ACM Press}, }  ©2005 Association for Computing Machinery |
