Skip to main content

Dan Parvaz

MITRE, Human Languages Technology (E544), Department Member

Followers

116

Following

72

Mentions

1

Public Views

Innovative, knowledgeable computational linguist with over 10 years' experience. Strong background in both theoretical linguistics and computer science with specific knowledge of several Middle Eastern and AfPak languages. Experience in Machine Translation, Entity Extraction, Social Media analysis, and other areas of Natural Language Processing.

less

Interests

Uploads

Papers by Dan Parvaz

Performance Evaluation of Speech Translation Systems

Lrec, 2008

Low-Density Language Bootstrapping: the Case of Tajiki Persian

Language Resources and Evaluation, 2008

Low-density languages raise difficulties for standard approaches to natural language processing t... more Low-density languages raise difficulties for standard approaches to natural language processing that depend on large online corpora. Using Persian as a case study, we propose a novel method for bootstrapping MT capability for a low-density language in the case where it relates to a higher density variant. Tajiki Persian is a low-density language that uses the Cyrillic alphabet, while Iranian

Evaluation of 2-way Iraqi Arabic–English speech translation systems using automated metrics

Machine Translation, 2012

The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translati... more The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program (http://1.usa.gov/transtac) faced many challenges in applying automated measures of translation quality to Iraqi Arabic-English speech translation dialogues. Features of speech data in general and of Iraqi Arabic data in particular undermine basic assumptions of automated measures that depend on matching system outputs to reference translations. These features are described along with the challenges they present for evaluating machine translation quality using automated metrics. We show that scores for translation into Iraqi Arabic exhibit higher correlations with human judgments when they are computed from normalized system outputs and reference translations. Orthographic normalization, lexical normalization, and operations involving light stemming resulted in higher correlations with human judgments.

SpatialML: Annotation Scheme for Marking Spatial Expressions in Natural Language

October, 2009

We have developed a rich markup language called SpatialML for spatial locations, allowing potenti... more We have developed a rich markup language called SpatialML for spatial locations, allowing potentially better integration of text collections with resources such as databases that provide spatial information about a domain, including gazetteers, physical feature databases, mapping services, etc. ... Our focus is primarily on geography and culturally-relevant landmarks, rather than biology, cosmology, geology, or other regions of the domain of spatial language. However, we expect that these guidelines could be adapted to other such domains with ...

Machine Translation Errors

ACM Transactions on Asian Language Information Processing, 2011

Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different p... more Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different points in the systems" development using HTER methods to identify errors and human annotations to refine TER annotations. Although the frequencies of errors in the more mature systems were lower, the proportions of error types exhibited little change. Results include high frequencies of pronoun errors in translations to English, high frequencies of subject person inflection in translations to Iraqi Arabic, similar frequencies of word order errors in both translation directions, and very low frequencies of polarity errors. The problems with many errors can be generalized as the need to insert lexemes not present in the source or vice versa, which includes errors in multi-word expressions. Discourse context will be required to resolve some problems with deictic elements like pronouns.

Performance evaluation of speech translation systems

… of LREC 2008, Jan 1, 2008

One of the most challenging tasks for uniformed service personnel serving in foreign countries is... more One of the most challenging tasks for uniformed service personnel serving in foreign countries is effective verbal communication with the local population. To remedy this problem, several companies and academic institutions have been funded to develop machine translation systems as part of the DARPA TRANSTAC (Spoken Language Communication and Translation System for Tactical Use) program. The goal of this program is to demonstrate capabilities to rapidly develop and field free-form, two-way translation systems that would enable speakers of different languages to communicate with one another in real-world tactical situations. DARPA has mandated that each TRANSTAC technology be evaluated numerous times throughout the life of the program and has tasked the National Institute of Standards and Technology (NIST) to lead this effort. This paper describes the experimental design methodology and test procedures from the most recent evaluation, conducted in July 2007, which focused on English to/from Iraqi Arabic.

Applying Automated Metrics to Speech Translation Dialogs

… of LREC-2008, Jan 1, 2008

Over the past five years, the Defense Advanced Research Projects Agency (DARPA) has funded develo... more Over the past five years, the Defense Advanced Research Projects Agency (DARPA) has funded development of speech translation systems for tactical applications. A key component of the research program has been extensive system evaluation, with dual objectives of assessing progress overall and comparing among systems. This paper describes the methods used to obtain BLEU, TER, and METEOR scores for two-way English-Iraqi Arabic systems. We compare the scores with measures based on human judgments and demonstrate the effects of normalization operations on BLEU scores. Issues that are highlighted include the quality of test data and differential results of applying automated metrics to Arabic vs. English.

Normalization for automated metrics: English and Arabic speech translation

… Summit XII,(Ottawa, …, Jan 1, 2009

The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translati... more The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program has experimented with applying automated metrics to speech translation dialogues. For translations into English, BLEU, TER, and METEOR scores correlate well with human judgments, but scores for translation into Arabic correlate with human judgments less strongly. This paper provides evidence to support the hypothesis that automated measures of Arabic are lower due to variation and inflection in Arabic by demonstrating that normalization operations improve correlation between BLEU scores and Likert-type judgments of semantic adequacyas well as between BLEU scores and human judgments of the successful transfer of the meaning of individual content words from English to Arabic.

Low-Density Language Bootstrapping: The Case of Tajiki Persian

Proceedings of LREC 2008, Jan 1, 2008

Low-density languages raise difficulties for standard approaches to natural language processing t... more Low-density languages raise difficulties for standard approaches to natural language processing that depend on large online corpora. Using Persian as a case study, we propose a novel method for bootstrapping MT capability for a low-density language in the case where it relates to a higher density variant. Tajiki Persian is a low-density language that uses the Cyrillic alphabet, while Iranian Persian (Farsi) is written in an extended version of the Arabic script and has many computational resources available. Despite the orthographic differences, the two languages have literary written forms that are almost identical. The paper describes the development of a comprehensive finite-state transducer that converts Tajik text to Farsi script and runs the resulting transliterated document through an existing Persian-to-English MT system. Due to divergences that arise in mapping the two writing systems and phonological and lexical distinctions, the system uses contextual cues (such as the position of a phoneme in a word) as well as available Farsi resources (such as a morphological analyzer to deal with differences in the affixal structures and a lexicon to disambiguate the analyses) to control the potential combinatorial explosion. The results point to a valuable strategy for the rapid prototyping of MT packages for languages of similar uneven density.

Evaluation of Machine Translation Errors in English and Iraqi Arabic

Proceedings of the …, Jan 1, 2010

Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different p... more Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different points in the systems" development using HTER methods to identify errors and human annotations to refine TER annotations. Although the frequencies of errors in the more mature systems were lower, the proportions of error types exhibited little change. Results include high frequencies of pronoun errors in translations to English, high frequencies of subject person inflection in translations to Iraqi Arabic, similar frequencies of word order errors in both translation directions, and very low frequencies of polarity errors. The problems with many errors can be generalized as the need to insert lexemes not present in the source or vice versa, which includes errors in multi-word expressions. Discourse context will be required to resolve some problems with deictic elements like pronouns.

New wine in old skins

Automated metrics for speech translation

Proceedings of the …, Jan 1, 2009

In this paper, we describe automated measures used to evaluate machine translation quality in the... more In this paper, we describe automated measures used to evaluate machine translation quality in the Defense Advanced Research Projects Agency"s Spoken Language Communication and Translation System for Tactical Use program, which is developing speech translation systems for dialogue between English and Iraqi Arabic speakers in military contexts. Limitations of the automated measures are illustrated along with variants of the measures that seek to overcome those limitations. Both the dialogue structure of the data and the Iraqi Arabic language challenge these measures, and the paper presents some solutions adopted by MITRE and NIST to improve confidence in the scores.

Performance Evaluation of Speech Translation Systems

Lrec, 2008

Low-Density Language Bootstrapping: the Case of Tajiki Persian

Language Resources and Evaluation, 2008

Low-density languages raise difficulties for standard approaches to natural language processing t... more Low-density languages raise difficulties for standard approaches to natural language processing that depend on large online corpora. Using Persian as a case study, we propose a novel method for bootstrapping MT capability for a low-density language in the case where it relates to a higher density variant. Tajiki Persian is a low-density language that uses the Cyrillic alphabet, while Iranian

Evaluation of 2-way Iraqi Arabic–English speech translation systems using automated metrics

Machine Translation, 2012

The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translati... more The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program (http://1.usa.gov/transtac) faced many challenges in applying automated measures of translation quality to Iraqi Arabic-English speech translation dialogues. Features of speech data in general and of Iraqi Arabic data in particular undermine basic assumptions of automated measures that depend on matching system outputs to reference translations. These features are described along with the challenges they present for evaluating machine translation quality using automated metrics. We show that scores for translation into Iraqi Arabic exhibit higher correlations with human judgments when they are computed from normalized system outputs and reference translations. Orthographic normalization, lexical normalization, and operations involving light stemming resulted in higher correlations with human judgments.

SpatialML: Annotation Scheme for Marking Spatial Expressions in Natural Language

October, 2009

We have developed a rich markup language called SpatialML for spatial locations, allowing potenti... more We have developed a rich markup language called SpatialML for spatial locations, allowing potentially better integration of text collections with resources such as databases that provide spatial information about a domain, including gazetteers, physical feature databases, mapping services, etc. ... Our focus is primarily on geography and culturally-relevant landmarks, rather than biology, cosmology, geology, or other regions of the domain of spatial language. However, we expect that these guidelines could be adapted to other such domains with ...

Machine Translation Errors

ACM Transactions on Asian Language Information Processing, 2011

Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different p... more Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different points in the systems" development using HTER methods to identify errors and human annotations to refine TER annotations. Although the frequencies of errors in the more mature systems were lower, the proportions of error types exhibited little change. Results include high frequencies of pronoun errors in translations to English, high frequencies of subject person inflection in translations to Iraqi Arabic, similar frequencies of word order errors in both translation directions, and very low frequencies of polarity errors. The problems with many errors can be generalized as the need to insert lexemes not present in the source or vice versa, which includes errors in multi-word expressions. Discourse context will be required to resolve some problems with deictic elements like pronouns.

Performance evaluation of speech translation systems

… of LREC 2008, Jan 1, 2008

One of the most challenging tasks for uniformed service personnel serving in foreign countries is... more One of the most challenging tasks for uniformed service personnel serving in foreign countries is effective verbal communication with the local population. To remedy this problem, several companies and academic institutions have been funded to develop machine translation systems as part of the DARPA TRANSTAC (Spoken Language Communication and Translation System for Tactical Use) program. The goal of this program is to demonstrate capabilities to rapidly develop and field free-form, two-way translation systems that would enable speakers of different languages to communicate with one another in real-world tactical situations. DARPA has mandated that each TRANSTAC technology be evaluated numerous times throughout the life of the program and has tasked the National Institute of Standards and Technology (NIST) to lead this effort. This paper describes the experimental design methodology and test procedures from the most recent evaluation, conducted in July 2007, which focused on English to/from Iraqi Arabic.

Applying Automated Metrics to Speech Translation Dialogs

… of LREC-2008, Jan 1, 2008

Over the past five years, the Defense Advanced Research Projects Agency (DARPA) has funded develo... more Over the past five years, the Defense Advanced Research Projects Agency (DARPA) has funded development of speech translation systems for tactical applications. A key component of the research program has been extensive system evaluation, with dual objectives of assessing progress overall and comparing among systems. This paper describes the methods used to obtain BLEU, TER, and METEOR scores for two-way English-Iraqi Arabic systems. We compare the scores with measures based on human judgments and demonstrate the effects of normalization operations on BLEU scores. Issues that are highlighted include the quality of test data and differential results of applying automated metrics to Arabic vs. English.

Normalization for automated metrics: English and Arabic speech translation

… Summit XII,(Ottawa, …, Jan 1, 2009

The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translati... more The Defense Advanced Research Projects Agency (DARPA) Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program has experimented with applying automated metrics to speech translation dialogues. For translations into English, BLEU, TER, and METEOR scores correlate well with human judgments, but scores for translation into Arabic correlate with human judgments less strongly. This paper provides evidence to support the hypothesis that automated measures of Arabic are lower due to variation and inflection in Arabic by demonstrating that normalization operations improve correlation between BLEU scores and Likert-type judgments of semantic adequacyas well as between BLEU scores and human judgments of the successful transfer of the meaning of individual content words from English to Arabic.

Low-Density Language Bootstrapping: The Case of Tajiki Persian

Proceedings of LREC 2008, Jan 1, 2008

Low-density languages raise difficulties for standard approaches to natural language processing t... more Low-density languages raise difficulties for standard approaches to natural language processing that depend on large online corpora. Using Persian as a case study, we propose a novel method for bootstrapping MT capability for a low-density language in the case where it relates to a higher density variant. Tajiki Persian is a low-density language that uses the Cyrillic alphabet, while Iranian Persian (Farsi) is written in an extended version of the Arabic script and has many computational resources available. Despite the orthographic differences, the two languages have literary written forms that are almost identical. The paper describes the development of a comprehensive finite-state transducer that converts Tajik text to Farsi script and runs the resulting transliterated document through an existing Persian-to-English MT system. Due to divergences that arise in mapping the two writing systems and phonological and lexical distinctions, the system uses contextual cues (such as the position of a phoneme in a word) as well as available Farsi resources (such as a morphological analyzer to deal with differences in the affixal structures and a lexicon to disambiguate the analyses) to control the potential combinatorial explosion. The results point to a valuable strategy for the rapid prototyping of MT packages for languages of similar uneven density.

Evaluation of Machine Translation Errors in English and Iraqi Arabic

Proceedings of the …, Jan 1, 2010

Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different p... more Errors in machine translations of English-Iraqi Arabic dialogues were analyzed at two different points in the systems" development using HTER methods to identify errors and human annotations to refine TER annotations. Although the frequencies of errors in the more mature systems were lower, the proportions of error types exhibited little change. Results include high frequencies of pronoun errors in translations to English, high frequencies of subject person inflection in translations to Iraqi Arabic, similar frequencies of word order errors in both translation directions, and very low frequencies of polarity errors. The problems with many errors can be generalized as the need to insert lexemes not present in the source or vice versa, which includes errors in multi-word expressions. Discourse context will be required to resolve some problems with deictic elements like pronouns.

New wine in old skins

Automated metrics for speech translation

Proceedings of the …, Jan 1, 2009

In this paper, we describe automated measures used to evaluate machine translation quality in the... more In this paper, we describe automated measures used to evaluate machine translation quality in the Defense Advanced Research Projects Agency"s Spoken Language Communication and Translation System for Tactical Use program, which is developing speech translation systems for dialogue between English and Iraqi Arabic speakers in military contexts. Limitations of the automated measures are illustrated along with variants of the measures that seek to overcome those limitations. Both the dialogue structure of the data and the Iraqi Arabic language challenge these measures, and the paper presents some solutions adopted by MITRE and NIST to improve confidence in the scores.