|
|
|
     |
                                         |
Return to Theory 2
When investigating alternative estimates for term discriminativeness, we discovered that relevance information and idf are much closer related than formulated in classical literature. Therefore, we revisited the justification of idf as it follows from the binary independent retrieval (BIR) model. The main result is a formal framework uncovering the close relationship of a generalised idf and the BIR model. The framework makes explicit how to incorporate relevance information into any retrieval function that involves an idf-component.In addition to the idf-based formulation of the BIR model, we propose Poisson-based estimates as an alternative to the classical estimates, this being motivated by the superiority of Poisson-based estimates for the within-document term frequencies. The main experimental finding is that a Poisson-based idf is superior to the classical idf, where the superiority is particularly evident for long queries.
@inproceedings{1076084, author = {Arjen P. de Vries and Thomas Roelleke}, title = {Relevance information: a loss of entropy but a gain for IDF?}, booktitle = {SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval}, year = {2005}, isbn = {1-59593-034-5}, pages = {282--289}, location = {Salvador, Brazil}, doi = {http://doi.acm.org/10.1145/1076034.1076084}, publisher = {ACM Press}, address = {New York, NY, USA}, }  ©2006 Association for Computing Machinery |
