|
|
|
     |
                                         |
Return to User studies
We describe a user study that examined the relationship between the quality of an Information Retrieval system and the effectiveness of its users in performing a task. The task involves finding answer facets of questions pertaining to a collection of newswire documents over a six month period. We artificially created sets of ranked lists at increasing levels of quality by blending the output of a state-of-the-art retrieval system with truth data created by annotators. Subjects performed the task by using these ranked lists to guide their labeling of answer passages in the retrieved articles. We found that as system accuracy improves, subject time on task and error rate decrease, and the rate of finding new correct answers increases. There is a large intermediary region in which the utility difference is not significant; our results suggest that there is some threshold of accuracy for this task beyond which user utility improves rapidly, but more experiments are needed to examine the area around that threshold closely.
@inproceedings{1076109, author = {James Allan and Ben Carterette and Joshua Lewis}, title = {When will information retrieval be "good enough"?}, booktitle = {SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval}, year = {2005}, isbn = {1-59593-034-5}, pages = {433--440}, location = {Salvador, Brazil}, doi = {http://doi.acm.org/10.1145/1076034.1076109}, publisher = {ACM Press}, address = {New York, NY, USA}, }  ©2006 Association for Computing Machinery |
