Alegria (2), X. Artola (2), A. Diaz de Ilarraza (2), N. Ezeiza (2), A. Sologaistoa (2), A. Soroa ... more Alegria (2), X. Artola (2), A. Diaz de Ilarraza (2), N. Ezeiza (2), A. Sologaistoa (2), A. Soroa (2), A. Valverde (2) (1) Elhuyar R&D Zelai Haundi kalea,
Morphological query expansion and language-filtering words have proved to be valid methods when s... more Morphological query expansion and language-filtering words have proved to be valid methods when searching the web for content in Basque via APIs of commercial search engines, as the implementation of these methods in recent IR and web-as-corpus tools shows, but no real analysis has been carried out to ascertain the degree of improvement, apart from a comparison of recall and precision using a classical web search engine and measured in terms of hit counts. This paper deals with a more theoretical study that confirms the validity of the combination of both methods. We have measured the increase in recall obtained by morphological query expansion and the increase in precision and loss in recall produced by language-filtering-words, but not only by searching the web directly and looking at the hit counts-which are not considered to be very reliable at best-, but also using both a Basque web corpus and a classical lemmatised corpus, thus providing more exact quantitative results. Furthermore, we provide various corpora-extracted data to be used in the aforementioned methods, such as lists of the most frequent inflections and declinations (cases, persons, numbers, times, etc.) for each POS-the most interesting word forms for a morphologically expanded query-, or a list of the most used Basque words with their frequencies and document-frequencies-the ones that should be used as language-filtering words- .
Morphological query expansion and language-filtering words have proved to be valid methods when s... more Morphological query expansion and language-filtering words have proved to be valid methods when searching the web for content in Basque via APIs of commercial search engines, as the implementation of these methods in recent IR and web-as-corpus tools shows, but no real analysis has been carried out to ascertain the degree of improvement, apart from a comparison of recall and precision using a classical web search engine and measured in terms of hit counts. This paper deals with a more theoretical study that confirms the validity of the combination of both methods. We have measured the increase in recall obtained by morphological query expansion and the increase in precision and loss in recall produced by language-filtering-words, but not only by searching the web directly and looking at the hit counts –which are not considered to be very reliable at best–, but also using both a Basque web corpus and a classical lemmatised corpus, thus providing more exact quantitative results. Furth...
Maiz esan ohi da hizkuntza batek hizkuntza-teknologien garapenean duen maila lotuta dagoela hizku... more Maiz esan ohi da hizkuntza batek hizkuntza-teknologien garapenean duen maila lotuta dagoela hizkuntza horren 'osasunarekin', horren adierazle izateraino. Ez da aztia izan behar bi alderdi horien arteko lotura badagoela sumatzeko. Mundu zabalean posizio 'indartsuan' dauden hizkuntzak dira hizkuntza-teknologietan, eta, horien artean, corpusgintzan, baliabide gehien dituztenak. Artikulu honen helburua da egileek euskarazko eta beste hizkuntza batzuetako corpus-baliabideetan eta corpusgintzan egin duten datu-bilketa aurkeztea, eta datu horien azterketan oinarrituta gogoeta egitea, non gauden eta norantz jo behar genukeen hausnartzeko.
The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised tex... more The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised texts in Basque, which aims to be a major resource in research and development with respect to written technical Basque: terminology, syntax and style. It was released in December 2006 and can be queried at http://www.ztcorpusa.net.
ABSTRACT The experience of a user of major search engines or other web information retrieval serv... more ABSTRACT The experience of a user of major search engines or other web information retrieval services looking for information in the Basque language is far from satisfactory: they only return pages with exact matches but no inflections (necessary for an agglutinative language like Basque), many results in other languages (no search engine gives the option to restrict its results to Basque), etc. This paper proposes using morphological query expansion and language-filtering words in combination with the APIs of search engines as a very cost-effective solution to build appropriate web search services for Basque. The implementation details of the methodology (choosing the most appropriate language-filtering words, the number of them, the most frequent inflections for the morphological query expansion, etc.) have been specified by corpora-based studies. The improvements produced have been measured in terms of precision and recall both over corpora and real web searches. Morphological query expansion can improve recall up to 47 % and language-filtering words can raise precision from 15 % to around 90 %, although with a loss in recall of about 30–35 %. The proposed methodology has already been successfully used in the Basque search service Elebila (http://www.elebila.eu) and the web-as-corpus tool CorpEus (http://www.corpeus.org), and the approach could be applied to other morphologically rich or under-resourced languages as well.
The main goal of the OpenMT project is the development of open source machine translation archite... more The main goal of the OpenMT project is the development of open source machine translation architectures based on hybrid models and advanced syntactic-semantic processors. These architectures combine the three main Machine Translation (MT) frameworks, Rule-based (RBMT), Statistical (SMT) and Example-based (EBMT), into hybrid systems. Defined architectures and results will be open source, allow for a rapid development and adaptation of new advanced machine translation systems for other languages. The project deals with four different languages: English, Spanish, Catalan and Basque. The translation systems developed for all language pairs will be internally evaluated using a rich set of linguistic metrics, and in different international evaluation campaigns.
Corpusak azken urteotan hizkuntza-baliabide gisa hartu duen garrantzia inork gutxik uka lezake ga... more Corpusak azken urteotan hizkuntza-baliabide gisa hartu duen garrantzia inork gutxik uka lezake gaur egun. Corpusa hizkuntza aztertzeko ezinbesteko baliabidea da, hainbat alorretan erabiltzen dena: lexikografian, sintaxian, semantikan, diskurtsoaren analisian... Adibidez, gaur egun, mundu zabalean egiten diren lexikografia-lanetan hutsik egin gabe aipatzen den hitza corpus da. Dela hiztegigintzaren osagarri, dela hiztegia bera egiteko lehengai eta abiapuntu, corpusa hiztegigilearen tresna eta euskarri ezinbestekotzat jotzen da gero eta maizago, hainbesteraino, non corpusean oinarritua edo, gutxienez, corpusaren laguntzaz taxutua ez den hiztegiari nekez aitortzen baitzaio zehaztasuna, zorroztasuna, fidagarritasuna eta, oro har, kalitatea. Esan gabe doa, corpusak ez dira lexikografian soilik erabiltzen, diziplina oso bat da corpus--hizkuntzalaritza. Corpusek datu linguistikoak jasotzen dituzte eta baliozko lanabesak dira hizkuntzaren erabilera erreala aztertu nahi bada. Urte askoan, horrelako azterketa enpirikoak egitea hizkuntzalaritzaren korronte nagusitik kanpo egon bada ere, azken urteotan gero eta tresna estimatuagoak dira, ez noski gramatika sortzailearen alternatiba edo aurkari gisa, beste ikuspegi baten eta ebidentzien ekarle gisa baizik. Gainera, corpusetan bildutako datu-kopuru handien bidez, hizkuntza-teknologien alorreko behar eta eginkizun batzuei beste era batera erantzuteko modua egoten da (prozedura estatistikoetan oinarritutako desanbiguazio-teknikak, itzulpen-memoriak, etab.).
Zientzia eta Teknologiaren corpusa IXA taldearen eta Elhuyar Fundazioaren lankidetza-proiektua da... more Zientzia eta Teknologiaren corpusa IXA taldearen eta Elhuyar Fundazioaren lankidetza-proiektua da, eta Hizking21 ikerketa estrategikoko proiektuaren barnean ari da egiten. Artikulu honetan azalduko ditugu proiektu honen helburu eta ezaugarri nagusiak, erabili dugun metodologia, garatu ditugun baliabideak eta tresnak, eta aurkezteko moduan dauden lehen emaitzak. • Corpus Científico-Técnico del español (CCT).
Morphological query expansion and language-filtering words have proved to be valid methods when s... more Morphological query expansion and language-filtering words have proved to be valid methods when searching the web for content in Basque via APIs of commercial search engines, as the implementation of these methods in recent IR and web-as-corpus tools shows, but no real analysis has been carried out to ascertain the degree of improvement, apart from a comparison of recall and precision using a classical web search engine and measured in terms of hit counts. This paper deals with a more theoretical study that confirms the validity of the combination of both methods. We have measured the increase in recall obtained by morphological query expansion and the increase in precision and loss in recall produced by language-filtering-words, but not only by searching the web directly and looking at the hit counts -which are not considered to be very reliable at best-, but also using both a Basque web corpus and a classical lemmatised corpus, thus providing more exact quantitative results. Furthermore, we provide various corpora-extracted data to be used in the aforementioned methods, such as lists of the most frequent inflections and declinations (cases, persons, numbers, times, etc.) for each POS -the most interesting word forms for a morphologically expanded query-, or a list of the most used Basque words with their frequencies and document-frequencies -the ones that should be used as language-filtering words-.
The performance of major search engines for Basque is far from satisfactory, partly due to the ag... more The performance of major search engines for Basque is far from satisfactory, partly due to the agglutinative nature of the language -it is commonly known that search engines do not perform well with such languages- and partly because it is not a language to which search engines restrict their results. In this paper we present EusBila, a search service for
The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised tex... more The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised texts in Basque, which aims to be a major resource in research and development with respect to written technical Basque: terminology, syntax and style. It was released in December 2006 and can be queried at http://www.ztcorpusa.net.
The ZT corpus (Basque Corpus of Science and Technology) is a tagged collection of specialized tex... more The ZT corpus (Basque Corpus of Science and Technology) is a tagged collection of specialized texts in Basque, which wants to be a main resource in research and development about written technical Basque: terminology, syntax and style. It will be the first written corpus in Basque which will be distributed by ELDA (at the end of 2006) and it wants to be a methodological and functional reference for new projects in the future (i.e. a national corpus for Basque). We also present the technology and the tools to build this Corpus. These tools, Corpusgile and Eulia, provide a flexible and extensible infrastructure for creating, visualizing and managing corpora and for consulting, visualizing and modifying annotations generated by linguistic tools.
Alegria (2), X. Artola (2), A. Diaz de Ilarraza (2), N. Ezeiza (2), A. Sologaistoa (2), A. Soroa ... more Alegria (2), X. Artola (2), A. Diaz de Ilarraza (2), N. Ezeiza (2), A. Sologaistoa (2), A. Soroa (2), A. Valverde (2) (1) Elhuyar R&D Zelai Haundi kalea,
Morphological query expansion and language-filtering words have proved to be valid methods when s... more Morphological query expansion and language-filtering words have proved to be valid methods when searching the web for content in Basque via APIs of commercial search engines, as the implementation of these methods in recent IR and web-as-corpus tools shows, but no real analysis has been carried out to ascertain the degree of improvement, apart from a comparison of recall and precision using a classical web search engine and measured in terms of hit counts. This paper deals with a more theoretical study that confirms the validity of the combination of both methods. We have measured the increase in recall obtained by morphological query expansion and the increase in precision and loss in recall produced by language-filtering-words, but not only by searching the web directly and looking at the hit counts-which are not considered to be very reliable at best-, but also using both a Basque web corpus and a classical lemmatised corpus, thus providing more exact quantitative results. Furthermore, we provide various corpora-extracted data to be used in the aforementioned methods, such as lists of the most frequent inflections and declinations (cases, persons, numbers, times, etc.) for each POS-the most interesting word forms for a morphologically expanded query-, or a list of the most used Basque words with their frequencies and document-frequencies-the ones that should be used as language-filtering words- .
Morphological query expansion and language-filtering words have proved to be valid methods when s... more Morphological query expansion and language-filtering words have proved to be valid methods when searching the web for content in Basque via APIs of commercial search engines, as the implementation of these methods in recent IR and web-as-corpus tools shows, but no real analysis has been carried out to ascertain the degree of improvement, apart from a comparison of recall and precision using a classical web search engine and measured in terms of hit counts. This paper deals with a more theoretical study that confirms the validity of the combination of both methods. We have measured the increase in recall obtained by morphological query expansion and the increase in precision and loss in recall produced by language-filtering-words, but not only by searching the web directly and looking at the hit counts –which are not considered to be very reliable at best–, but also using both a Basque web corpus and a classical lemmatised corpus, thus providing more exact quantitative results. Furth...
Maiz esan ohi da hizkuntza batek hizkuntza-teknologien garapenean duen maila lotuta dagoela hizku... more Maiz esan ohi da hizkuntza batek hizkuntza-teknologien garapenean duen maila lotuta dagoela hizkuntza horren 'osasunarekin', horren adierazle izateraino. Ez da aztia izan behar bi alderdi horien arteko lotura badagoela sumatzeko. Mundu zabalean posizio 'indartsuan' dauden hizkuntzak dira hizkuntza-teknologietan, eta, horien artean, corpusgintzan, baliabide gehien dituztenak. Artikulu honen helburua da egileek euskarazko eta beste hizkuntza batzuetako corpus-baliabideetan eta corpusgintzan egin duten datu-bilketa aurkeztea, eta datu horien azterketan oinarrituta gogoeta egitea, non gauden eta norantz jo behar genukeen hausnartzeko.
The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised tex... more The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised texts in Basque, which aims to be a major resource in research and development with respect to written technical Basque: terminology, syntax and style. It was released in December 2006 and can be queried at http://www.ztcorpusa.net.
ABSTRACT The experience of a user of major search engines or other web information retrieval serv... more ABSTRACT The experience of a user of major search engines or other web information retrieval services looking for information in the Basque language is far from satisfactory: they only return pages with exact matches but no inflections (necessary for an agglutinative language like Basque), many results in other languages (no search engine gives the option to restrict its results to Basque), etc. This paper proposes using morphological query expansion and language-filtering words in combination with the APIs of search engines as a very cost-effective solution to build appropriate web search services for Basque. The implementation details of the methodology (choosing the most appropriate language-filtering words, the number of them, the most frequent inflections for the morphological query expansion, etc.) have been specified by corpora-based studies. The improvements produced have been measured in terms of precision and recall both over corpora and real web searches. Morphological query expansion can improve recall up to 47 % and language-filtering words can raise precision from 15 % to around 90 %, although with a loss in recall of about 30–35 %. The proposed methodology has already been successfully used in the Basque search service Elebila (http://www.elebila.eu) and the web-as-corpus tool CorpEus (http://www.corpeus.org), and the approach could be applied to other morphologically rich or under-resourced languages as well.
The main goal of the OpenMT project is the development of open source machine translation archite... more The main goal of the OpenMT project is the development of open source machine translation architectures based on hybrid models and advanced syntactic-semantic processors. These architectures combine the three main Machine Translation (MT) frameworks, Rule-based (RBMT), Statistical (SMT) and Example-based (EBMT), into hybrid systems. Defined architectures and results will be open source, allow for a rapid development and adaptation of new advanced machine translation systems for other languages. The project deals with four different languages: English, Spanish, Catalan and Basque. The translation systems developed for all language pairs will be internally evaluated using a rich set of linguistic metrics, and in different international evaluation campaigns.
Corpusak azken urteotan hizkuntza-baliabide gisa hartu duen garrantzia inork gutxik uka lezake ga... more Corpusak azken urteotan hizkuntza-baliabide gisa hartu duen garrantzia inork gutxik uka lezake gaur egun. Corpusa hizkuntza aztertzeko ezinbesteko baliabidea da, hainbat alorretan erabiltzen dena: lexikografian, sintaxian, semantikan, diskurtsoaren analisian... Adibidez, gaur egun, mundu zabalean egiten diren lexikografia-lanetan hutsik egin gabe aipatzen den hitza corpus da. Dela hiztegigintzaren osagarri, dela hiztegia bera egiteko lehengai eta abiapuntu, corpusa hiztegigilearen tresna eta euskarri ezinbestekotzat jotzen da gero eta maizago, hainbesteraino, non corpusean oinarritua edo, gutxienez, corpusaren laguntzaz taxutua ez den hiztegiari nekez aitortzen baitzaio zehaztasuna, zorroztasuna, fidagarritasuna eta, oro har, kalitatea. Esan gabe doa, corpusak ez dira lexikografian soilik erabiltzen, diziplina oso bat da corpus--hizkuntzalaritza. Corpusek datu linguistikoak jasotzen dituzte eta baliozko lanabesak dira hizkuntzaren erabilera erreala aztertu nahi bada. Urte askoan, horrelako azterketa enpirikoak egitea hizkuntzalaritzaren korronte nagusitik kanpo egon bada ere, azken urteotan gero eta tresna estimatuagoak dira, ez noski gramatika sortzailearen alternatiba edo aurkari gisa, beste ikuspegi baten eta ebidentzien ekarle gisa baizik. Gainera, corpusetan bildutako datu-kopuru handien bidez, hizkuntza-teknologien alorreko behar eta eginkizun batzuei beste era batera erantzuteko modua egoten da (prozedura estatistikoetan oinarritutako desanbiguazio-teknikak, itzulpen-memoriak, etab.).
Zientzia eta Teknologiaren corpusa IXA taldearen eta Elhuyar Fundazioaren lankidetza-proiektua da... more Zientzia eta Teknologiaren corpusa IXA taldearen eta Elhuyar Fundazioaren lankidetza-proiektua da, eta Hizking21 ikerketa estrategikoko proiektuaren barnean ari da egiten. Artikulu honetan azalduko ditugu proiektu honen helburu eta ezaugarri nagusiak, erabili dugun metodologia, garatu ditugun baliabideak eta tresnak, eta aurkezteko moduan dauden lehen emaitzak. • Corpus Científico-Técnico del español (CCT).
Morphological query expansion and language-filtering words have proved to be valid methods when s... more Morphological query expansion and language-filtering words have proved to be valid methods when searching the web for content in Basque via APIs of commercial search engines, as the implementation of these methods in recent IR and web-as-corpus tools shows, but no real analysis has been carried out to ascertain the degree of improvement, apart from a comparison of recall and precision using a classical web search engine and measured in terms of hit counts. This paper deals with a more theoretical study that confirms the validity of the combination of both methods. We have measured the increase in recall obtained by morphological query expansion and the increase in precision and loss in recall produced by language-filtering-words, but not only by searching the web directly and looking at the hit counts -which are not considered to be very reliable at best-, but also using both a Basque web corpus and a classical lemmatised corpus, thus providing more exact quantitative results. Furthermore, we provide various corpora-extracted data to be used in the aforementioned methods, such as lists of the most frequent inflections and declinations (cases, persons, numbers, times, etc.) for each POS -the most interesting word forms for a morphologically expanded query-, or a list of the most used Basque words with their frequencies and document-frequencies -the ones that should be used as language-filtering words-.
The performance of major search engines for Basque is far from satisfactory, partly due to the ag... more The performance of major search engines for Basque is far from satisfactory, partly due to the agglutinative nature of the language -it is commonly known that search engines do not perform well with such languages- and partly because it is not a language to which search engines restrict their results. In this paper we present EusBila, a search service for
The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised tex... more The ZT Corpus (Basque Corpus of Science and Technology) is a tagged collection of specialised texts in Basque, which aims to be a major resource in research and development with respect to written technical Basque: terminology, syntax and style. It was released in December 2006 and can be queried at http://www.ztcorpusa.net.
The ZT corpus (Basque Corpus of Science and Technology) is a tagged collection of specialized tex... more The ZT corpus (Basque Corpus of Science and Technology) is a tagged collection of specialized texts in Basque, which wants to be a main resource in research and development about written technical Basque: terminology, syntax and style. It will be the first written corpus in Basque which will be distributed by ELDA (at the end of 2006) and it wants to be a methodological and functional reference for new projects in the future (i.e. a national corpus for Basque). We also present the technology and the tools to build this Corpus. These tools, Corpusgile and Eulia, provide a flexible and extensible infrastructure for creating, visualizing and managing corpora and for consulting, visualizing and modifying annotations generated by linguistic tools.
Uploads
Papers by N. Areta