Matches in ScholarlyData for { <https://w3id.org/scholarlydata/inproceedings/lrec2008/papers/909> ?p ?o. }
Showing items 1 to 12 of
12
with 100 items per page.
- 909 creator olga-babko-malaya.
- 909 type InProceedings.
- 909 label "Annotation of Nuggets and Relevance in GALE Distillation Evaluation".
- 909 sameAs 909.
- 909 abstract "This paper presents an approach to annotation that BAE Systems has employed in the DARPA GALE Phase 2 Distillation evaluation. The purpose of the GALE Distillation evaluation is to quantify the amount of relevant and non-redundant information a distillation engine is able to produce in response to a specific, formatted query; and to compare that amount of information to the amount of information gathered by a bilingual human using commonly available state-of-the-art tools. As part of the evaluation, following NIST evaluation methodology of complex question answering (Voorhees, 2003), human annotators were asked to establish the relevancy of responses as well as the presence of atomic facts or information units, called nuggets of information. This paper discusses various challenges to the annotation of nuggets, called nuggetization, which include interaction between the granularity of nuggets and relevancy of these nuggets to the query in question. The approach proposed in the paper views nuggetization as a procedural task and allows annotators to revisit nuggetization based on the requirements imposed by the relevancy guidelines defined with a specific end-user in mind. This approach is shown in the paper to produce consistent annotations with high inter-annotator agreement scores.".
- 909 hasAuthorList authorList.
- 909 hasTopic Linguistics.
- 909 isPartOf proceedings.
- 909 keyword "Corpus (creation, annotation, etc.)".
- 909 keyword "Evaluation methodologies".
- 909 keyword "Information Extraction, Information Retrieval".
- 909 title "Annotation of Nuggets and Relevance in GALE Distillation Evaluation".