In this paper, we propose the use of multidocument sum- marization as a post-processing step in d... more In this paper, we propose the use of multidocument sum- marization as a post-processing step in document retrieval. We examine the use of the summary as a replacement to the standard ranked list. The form of the summary is novel because it has both informative and indicate elements, de- signed to help dieren t users perform their tasks better. Our
In this paper we present a quantitative and qualitative evaluation of DEFINDER, a rule-based syst... more In this paper we present a quantitative and qualitative evaluation of DEFINDER, a rule-based system that mines consumer-oriented full text articles in order to extract definitions and the terms they define. Two quantitative evaluations show that in terms of precision and recall as measured against human performance, DEFINDER obtained 87% and 75% respectively, thereby revealing the incompleteness of existing resources
The problem addressed in this paper concerns the automatic identification and extraction of medic... more The problem addressed in this paper concerns the automatic identification and extraction of medical terms along with their definitions and modifiers from full text consumer-oriented medical articles. The system, DEFINDER (Definition Finder), uses rule-based techniques. The output of our system can be used in several applications: creation and/or enhancement of on-line terminological resources, summarization and text categorization according to level
Meeting of the Association for Computational Linguistics, 1998
We present results of two methods for assessing the event profile of news articles as a function ... more We present results of two methods for assessing the event profile of news articles as a function of verb type. The unique contribution of this research is the focus on the role of verbs, rather than nouns. Two algorithms are presented and evaluated, one of which is shown to accurately discriminate documents by type and semantic properties, i.e. the event
Proceedings of the American Society for Information Science and Technology, 2008
In this paper we extend existing sensemaking models by adding detail on types of conceptual chang... more In this paper we extend existing sensemaking models by adding detail on types of conceptual change and cognitive mechanisms taken from theories of cognition and learning. Our extended model aims to offer a more complete picture of the cognitive processes of sense-making, including the underlying cognitive mechanisms and different types of conceptual changes so that we can link observations to theory. We conducted a preliminary test of the model against think-aloud protocols of users performing intelligence analysis tasks using a multilingual, multimedia information system. Their sense-making processes were consistent with our iterative model of search and sensemaking loops. Most of the cognitive mechanisms derived from the literature appeared, with a prevalence of data-driven mechanisms.
Proceedings of the first ACM/IEEE-CS joint conference on Digital libraries - JCDL '01, 2001
The needs of society have long been addressed through government resea rch support for new techno... more The needs of society have long been addressed through government resea rch support for new technologies-the Internet representing one example. Today, under the rubric of digital government, federal agencies as well as state and local units of governments at all levels have begun to leverage the fruits of these research investments to better serve the needs of their constituencies. Government
Proceedings of the 8th European workshop on Natural Language Generation - EWNLG '01, 2001
The task of creating indicative summaries that help a searcher decide whether to read a particula... more The task of creating indicative summaries that help a searcher decide whether to read a particular document is a difficult task. This paper examines the indicative summarization task from a generation perspective, by first analyzing its required content via published guidelines and corpus analysis. We show how these summaries can be factored into a set of document features, and how an implemented content planner uses the topicality document feature to create indicative multidocument query-based summaries.
We present a statistical similarity measuring and clustering tool, SIMFINDER, that organizes smal... more We present a statistical similarity measuring and clustering tool, SIMFINDER, that organizes small pieces of text from one or multiple documents into tight clusters. By placing highly related text units in the same cluster, SIMFINDER enables a subse- quent content selection/generation component to re- duce each cluster to a single sentence, either by ex- traction or by reformulation. We report
We discuss factors that affect human agreement on a semantic labeling task in the art history dom... more We discuss factors that affect human agreement on a semantic labeling task in the art history domain, based on the results of four experiments where we varied the number of labels annotators could assign, the number of annotators, the type and amount of training they received, and the size of the text span being labeled. Using the labelings from one experiment involving seven annotators, we investigate the relation between interannotator agreement and machine learning performance. We construct binary classifiers and vary the training and test data by swapping the labelings from the seven annotators. First, we find performance is often quite good despite lower than recommended interannotator agreement. Second, we find that on average, learning performance for a given functional semantic category correlates with the overall agreement among the seven annotators for that category. Third, we find that learning performance on the data from a given annotator does not correlate with the quality of that annotator's labeling. We offer recommendations for the use of labeled data in machine learning, and argue that learners should attempt to accommodate human variation. We also note implications for large scale corpus annotation projects that deal with similarly subjective phenomena.
The CLiMB project investigates semi-automatic methods to extract descriptive metadata from texts ... more The CLiMB project investigates semi-automatic methods to extract descriptive metadata from texts for indexing digital image collections. We devel- oped a set of functional semantic categories to classify text extracts that describe images. Each semantic category names a functional relation between an image depicting a work of art historical significance, and expository text associated with the image. This includes description
This paper presents a corpus-based system to expand multi-word index terms using a part-of-speech... more This paper presents a corpus-based system to expand multi-word index terms using a part-of-speech tagger and a full- fledged derivational morphological system, combined with a shallow parser. The unique contribution of the research is in using these linguistically based tools with filters in order to avoid the problems of semantic degradation typically associated with derivational analysis. The expansion and subsequent
... Effective use of natural language processing techniques for automatic conflation of multi-wor... more ... Effective use of natural language processing techniques for automatic conflation of multi-word terms: the role of derivational morphology, part of speech tagging, and shallow parsing. Authors:
Proceedings of the 35th annual meeting on Association for Computational Linguistics -, 1997
A system for the automatic production of controlled index terms is presented using linguistically... more A system for the automatic production of controlled index terms is presented using linguistically-motivated techniques. This includes a finite-state part of speech tagger, a derivational morphological processor for analysis and generation, and a unificationbased shallow-level parser using transformational rules over syntactic patterns. The contribution of this research is the successful combination of parsing over a seed term list coupled with derivational morphology to achieve greater coverage of multi-word terms for indexing and retrieval. Final results are evaluated for precision and recall, and implications for indexing and retrieval are discussed.
Proceedings of the 15th conference on Computational linguistics -, 1994
This paper presents the results of an experiment usiug machine-readable dictionaries (Mill)s) and... more This paper presents the results of an experiment usiug machine-readable dictionaries (Mill)s) and corpora for building concatenativc units for text to speech (T'PS) systems. Theoretical questions concerning the nature of t)honemic data in dictionaries are raised; phonemic dictionary data is viewed as a representative corpus over which to extract n-gram phonemic frequencies in the language. Dictionary data are compared to corpus data, and phoneme inventories arc evaluated for coverage. A methodology is defined to compute I)honemic n-grams for incorporation into a TTS system.
This paper describes the comparative evaluation of an experimental automated text summarization s... more This paper describes the comparative evaluation of an experimental automated text summarization system, Centrifuser and three conventional search engines - Google, Yahoo and About.com. Centrifuser provides information to patients and families relevant to their questions about specific health conditions. It then produces a multidocument summary of articles retrieved by a standard search engine, tailored to the user's question. Subjects, consisting of friends or family of hospitalized patients, were asked to "think aloud" as they interacted with the four systems. The evaluation involved audio- and video recording of subject interactions with the interfaces in situ at a hospital. Results of the evaluation show that subjects found Centrifuser's summarization capability useful and easy to understand. In comparing Centrifuser to the three search engines, subjects' ratings varied; however, specific interface features were deemed useful across interfaces. We conclude w...
We report on two corpora to be used in the evaluation of component systems for the tasks of (1) l... more We report on two corpora to be used in the evaluation of component systems for the tasks of (1) linear segmentation of text and (2) summary-directed sentence extraction. We present characteristics of the corpora, methods used in the collection of user judgments, and an overview of the application of the corpora to evaluating the component system. Finally, we discuss the
We present a new method for discovering a segmental discourse structure of a document while categ... more We present a new method for discovering a segmental discourse structure of a document while categorizing each segment's function and importance. Segments are determined by a zero-sum weighting scheme, used on occurrences of noun phrases and pronominal forms retrieved from the document. Segment roles are then calculated from the distribution of the terms in the segment. Finally, we present results
In this paper, we propose the use of multidocument sum- marization as a post-processing step in d... more In this paper, we propose the use of multidocument sum- marization as a post-processing step in document retrieval. We examine the use of the summary as a replacement to the standard ranked list. The form of the summary is novel because it has both informative and indicate elements, de- signed to help dieren t users perform their tasks better. Our
In this paper we present a quantitative and qualitative evaluation of DEFINDER, a rule-based syst... more In this paper we present a quantitative and qualitative evaluation of DEFINDER, a rule-based system that mines consumer-oriented full text articles in order to extract definitions and the terms they define. Two quantitative evaluations show that in terms of precision and recall as measured against human performance, DEFINDER obtained 87% and 75% respectively, thereby revealing the incompleteness of existing resources
The problem addressed in this paper concerns the automatic identification and extraction of medic... more The problem addressed in this paper concerns the automatic identification and extraction of medical terms along with their definitions and modifiers from full text consumer-oriented medical articles. The system, DEFINDER (Definition Finder), uses rule-based techniques. The output of our system can be used in several applications: creation and/or enhancement of on-line terminological resources, summarization and text categorization according to level
Meeting of the Association for Computational Linguistics, 1998
We present results of two methods for assessing the event profile of news articles as a function ... more We present results of two methods for assessing the event profile of news articles as a function of verb type. The unique contribution of this research is the focus on the role of verbs, rather than nouns. Two algorithms are presented and evaluated, one of which is shown to accurately discriminate documents by type and semantic properties, i.e. the event
Proceedings of the American Society for Information Science and Technology, 2008
In this paper we extend existing sensemaking models by adding detail on types of conceptual chang... more In this paper we extend existing sensemaking models by adding detail on types of conceptual change and cognitive mechanisms taken from theories of cognition and learning. Our extended model aims to offer a more complete picture of the cognitive processes of sense-making, including the underlying cognitive mechanisms and different types of conceptual changes so that we can link observations to theory. We conducted a preliminary test of the model against think-aloud protocols of users performing intelligence analysis tasks using a multilingual, multimedia information system. Their sense-making processes were consistent with our iterative model of search and sensemaking loops. Most of the cognitive mechanisms derived from the literature appeared, with a prevalence of data-driven mechanisms.
Proceedings of the first ACM/IEEE-CS joint conference on Digital libraries - JCDL '01, 2001
The needs of society have long been addressed through government resea rch support for new techno... more The needs of society have long been addressed through government resea rch support for new technologies-the Internet representing one example. Today, under the rubric of digital government, federal agencies as well as state and local units of governments at all levels have begun to leverage the fruits of these research investments to better serve the needs of their constituencies. Government
Proceedings of the 8th European workshop on Natural Language Generation - EWNLG '01, 2001
The task of creating indicative summaries that help a searcher decide whether to read a particula... more The task of creating indicative summaries that help a searcher decide whether to read a particular document is a difficult task. This paper examines the indicative summarization task from a generation perspective, by first analyzing its required content via published guidelines and corpus analysis. We show how these summaries can be factored into a set of document features, and how an implemented content planner uses the topicality document feature to create indicative multidocument query-based summaries.
We present a statistical similarity measuring and clustering tool, SIMFINDER, that organizes smal... more We present a statistical similarity measuring and clustering tool, SIMFINDER, that organizes small pieces of text from one or multiple documents into tight clusters. By placing highly related text units in the same cluster, SIMFINDER enables a subse- quent content selection/generation component to re- duce each cluster to a single sentence, either by ex- traction or by reformulation. We report
We discuss factors that affect human agreement on a semantic labeling task in the art history dom... more We discuss factors that affect human agreement on a semantic labeling task in the art history domain, based on the results of four experiments where we varied the number of labels annotators could assign, the number of annotators, the type and amount of training they received, and the size of the text span being labeled. Using the labelings from one experiment involving seven annotators, we investigate the relation between interannotator agreement and machine learning performance. We construct binary classifiers and vary the training and test data by swapping the labelings from the seven annotators. First, we find performance is often quite good despite lower than recommended interannotator agreement. Second, we find that on average, learning performance for a given functional semantic category correlates with the overall agreement among the seven annotators for that category. Third, we find that learning performance on the data from a given annotator does not correlate with the quality of that annotator's labeling. We offer recommendations for the use of labeled data in machine learning, and argue that learners should attempt to accommodate human variation. We also note implications for large scale corpus annotation projects that deal with similarly subjective phenomena.
The CLiMB project investigates semi-automatic methods to extract descriptive metadata from texts ... more The CLiMB project investigates semi-automatic methods to extract descriptive metadata from texts for indexing digital image collections. We devel- oped a set of functional semantic categories to classify text extracts that describe images. Each semantic category names a functional relation between an image depicting a work of art historical significance, and expository text associated with the image. This includes description
This paper presents a corpus-based system to expand multi-word index terms using a part-of-speech... more This paper presents a corpus-based system to expand multi-word index terms using a part-of-speech tagger and a full- fledged derivational morphological system, combined with a shallow parser. The unique contribution of the research is in using these linguistically based tools with filters in order to avoid the problems of semantic degradation typically associated with derivational analysis. The expansion and subsequent
... Effective use of natural language processing techniques for automatic conflation of multi-wor... more ... Effective use of natural language processing techniques for automatic conflation of multi-word terms: the role of derivational morphology, part of speech tagging, and shallow parsing. Authors:
Proceedings of the 35th annual meeting on Association for Computational Linguistics -, 1997
A system for the automatic production of controlled index terms is presented using linguistically... more A system for the automatic production of controlled index terms is presented using linguistically-motivated techniques. This includes a finite-state part of speech tagger, a derivational morphological processor for analysis and generation, and a unificationbased shallow-level parser using transformational rules over syntactic patterns. The contribution of this research is the successful combination of parsing over a seed term list coupled with derivational morphology to achieve greater coverage of multi-word terms for indexing and retrieval. Final results are evaluated for precision and recall, and implications for indexing and retrieval are discussed.
Proceedings of the 15th conference on Computational linguistics -, 1994
This paper presents the results of an experiment usiug machine-readable dictionaries (Mill)s) and... more This paper presents the results of an experiment usiug machine-readable dictionaries (Mill)s) and corpora for building concatenativc units for text to speech (T'PS) systems. Theoretical questions concerning the nature of t)honemic data in dictionaries are raised; phonemic dictionary data is viewed as a representative corpus over which to extract n-gram phonemic frequencies in the language. Dictionary data are compared to corpus data, and phoneme inventories arc evaluated for coverage. A methodology is defined to compute I)honemic n-grams for incorporation into a TTS system.
This paper describes the comparative evaluation of an experimental automated text summarization s... more This paper describes the comparative evaluation of an experimental automated text summarization system, Centrifuser and three conventional search engines - Google, Yahoo and About.com. Centrifuser provides information to patients and families relevant to their questions about specific health conditions. It then produces a multidocument summary of articles retrieved by a standard search engine, tailored to the user's question. Subjects, consisting of friends or family of hospitalized patients, were asked to "think aloud" as they interacted with the four systems. The evaluation involved audio- and video recording of subject interactions with the interfaces in situ at a hospital. Results of the evaluation show that subjects found Centrifuser's summarization capability useful and easy to understand. In comparing Centrifuser to the three search engines, subjects' ratings varied; however, specific interface features were deemed useful across interfaces. We conclude w...
We report on two corpora to be used in the evaluation of component systems for the tasks of (1) l... more We report on two corpora to be used in the evaluation of component systems for the tasks of (1) linear segmentation of text and (2) summary-directed sentence extraction. We present characteristics of the corpora, methods used in the collection of user judgments, and an overview of the application of the corpora to evaluating the component system. Finally, we discuss the
We present a new method for discovering a segmental discourse structure of a document while categ... more We present a new method for discovering a segmental discourse structure of a document while categorizing each segment's function and importance. Segments are determined by a zero-sum weighting scheme, used on occurrences of noun phrases and pronominal forms retrieved from the document. Segment roles are then calculated from the distribution of the terms in the segment. Finally, we present results
Uploads
Papers by Judith Klavans