|
|
|
                                          
|
Return to Session IXb: Natural Language Processing
This paper evaluates an extraction-based approach to answering definitional questions. Our system extracted useful linguistic constructs called linguistic features from raw text using information extraction tools and formulated answers based on such features. The features employed include appositives, copulas, structured patterns, relations, propositions and raw sentences. The features were ranked based on feature type and similarity to a question profile. Redundant features were detected using a simple heuristic-based strategy. The approach achieved state of the art performance at the TREC 2003 QA evaluation. Component analysis of the system was carried out using an automatic scoring function called Rouge (Lin and Hovy, 2003). Major findings include 1) answers using linguistic features are significantly better than those using raw sentences; 2) the most useful features are appositives and copulas; 3) question profiles, as a means of modeling user interests, can significantly improve system performance; 4) the Rouge scores are closely correlated with subjective evaluation results, indicating the suitability of using Rouge for evaluating definitional QA systems.
@inproceedings{1009064, author = {Jinxi Xu and Ralph Weischedel and Ana Licuanan}, title = {Evaluation of an extraction-based approach to answering definitional questions}, booktitle = {SIGIR '04: Proceedings of the 27th annual international conference on Research and development in information retrieval}, year = {2004}, isbn = {1-58113-881-4}, pages = {418--424}, location = {Sheffield, United Kingdom}, doi = {http://doi.acm.org/10.1145/1008992.1009064}, publisher = {ACM Press}, }  ©2005 Association for Computing Machinery |
