Word Segmentation

This paper presents a new approach to language model construction, learning a language model not from text, but directly from continuous speech. A phoneme lattice is created using acoustic model scores, and Bayesian techniques are used to robustly learn a language model from this noisy input. A novel sampling technique is devised that allows for the integrated learning of word boundaries and an n-gram language model with no prior linguistic knowledge. The proposed techniques were used to learn a language model directly from continuous, potentially large-vocabulary speech. This language model was able to significantly reduce the ASR phoneme error rate over a separate set of test data, and the proposed lattice processing and lexical acquisition techniques were found to be important factors in this improvement.

In this paper, we first describe an automatic table ground truth generation system which can efficiently generate a large amount of accurate table ground truth suitable for the development of table detection algorithms. Then a novel background-analysis-based, coarse-to-fine table identification algorithm and an X-Y cut table decomposition algorithm are described. We discuss an experimental protocol to evaluate the table detection algorithms. For a total of 1 125 document pages having 518 table entities and a total of 10 941 cell entities, our table detection algorithm takes line, word segmentation results as input and obtains around 90% cell correct detection rates.

Dzongkha, the national language of Bhutan, is continuous in written form and it fails to mark the word boundary. Dzongkha word segmentation is one of the fundamental problems and a prerequisite that needs to be solved before more advanced Dzongkha text processing and other natural language processing tools can be developed. This paper presents our initial attempt at segmenting Dzongkha sentences into words. The paper describes the implementation of Maximal Matching (Dictionary based Approach) followed by bigram techniques (Non-dictionary based Approach) in segmenting the Dzongkha scripts. Although the used techniques are basic and naive, it provides a baseline of the Dzongkha word segmentation task. Preliminary experimental results show percentage of segmentation accuracy. However, the segmentation accuracy is dependent on the type of document domain and size and quality of the lexicon and the corpus. Some of the related issues for future directions are also discussed.

Recent work on extracting features of gaps in handwritten text allows a classification into inter-word and intraword classes using suitable classification techniques. In this paper, we apply 5 different supervised classification algorithms from the machine learning field on both the original dataset and a dataset with the best features selected using mutual information. The classifiers are compared by employing McNemar's test. We find that SVMs and MLPs outperform the other classifiers and that preprocessing to select features works well.

Over the past couple of decades, research has established that infants are sensitive to the predominant stress pattern of their native language. However, the degree to which the stress pattern shapes infants' language development has yet to be fully determined. Whether stress is merely a cue to help organize the patterns of speech or whether it is an important part of the representation of speech sound sequences has still to be explored. Building on research in the areas of infant speech perception and segmentation, we asked how several months of exposure to the target language shapes infants' speech processing biases with respect to lexical stress. We hypothesize that infants represent stressed and unstressed syllables differently, and employed analyses of child-directed speech to show how this change to the representational landscape results in better distribution-based word segmentation as well as an advantage for stress-initial syllable sequences. A series of experiments then tested 9- and 7-month-old infants on their ability to use lexical stress without any other cues present to parse sequences from an artificial language. We found that infants adopted a stress-initial syllable strategy and that they appear to encode stress information as part of their proto-lexical representations. Together, the results of these studies suggest that stress information in the ambient language not only shapes how statistics are calculated over the speech input, but that it is also encoded in the representations of parsed speech sequences.

This chapter explores the hypothesis that diminutive usage in child-directed speech may provide multiple benefits for language acquisition. We summarize a series of experiments that exposed naïve English-speaking adults to Dutch or Russian diminutives, and tested their ability to ...

The current study examined the pattern of prosodic phrasing and the distribution of post-lexical pitch accent types in a Spanish-English bilingual child. We collected utterances from natural interactions between parents and the child at the age of 2;6 and 3;0, and analyzed them using MAE_ToBI and SP_ToBI. Then we compared prosodic development across ages within each language, and compared the child's speech production with monolingual English and Spanish parents' productions. Results revealed that both the child and parents divide their short utterances into smaller prosodic phrases and that most content words bear post-lexical pitch accent. The result suggests that there are abundant acoustic correlates of prosodic phrases and prominence in the language input, and it can make the word segmentation task easy for children.

The lack of distinct word delimiters, white space, between the words in a
sentence in written text is one the problems in performing Natural Language
Processing (NLP) tasks for Myanmar language. In order to determine the specific
meaning or context of each word in the text, a proper word segmentation process, is
needed as an essential basic component to operate further linguistic process. In this
paper, we proposed a method for word segmentation, which can detect and define
each word in a sentence accurately, written in Myanmar language, using statistical
approach. The accurate result of finding the correct word could help finding the
definite word from the sentence and its definition to carry out any other NLP
processing for documents written in Myanmar language.

One of the infant's first tasks in language acquisition is to discover the words embedded in a mostly continuous speech stream. This learning problem might be solved by using distributional cues to word boundaries-for example, by computing the transitional probabilities between sounds in the language input and using the relative strengths of these probabilities to hypothesize word boundaries. The learner might be further aided by language-specific prosodic cues correlated with word boundaries. As a first step in testing these hypotheses, we briefly exposed adults to an artificial language in which the only cues available for word segmentation were the transitional probabilities between syllables. Subjects were able to learn the words of this language. Furthermore, the addition of certain prosodic cues served to enhance performance. These results suggest that distributional cues may play an important role in the initial word segmentation of language learners. ᭧ sound structure of their native language. For Morgan, Seth Pollak, and Michael Spivey-Knowlton for example, 9-month-old infants prefer to listen helpful discussions, to Mark Seidenberg and two anonyto certain prosodic and phonotactic patterns mous reviewers for helpful comments on a previous draft, of their native language over those of other and to Tracey Lavonas, Denise Smith, Christa Tinari, languages, whereas 6-month-olds do not (Jusand Rachel Tunick for their assistance in conducting the experiments.

This paper attempts to participate in the ongoing discussion in search of a suitable model for the computational treatment of Greek morphology. Focusing on the unsupervised morphology learning technique, and particularly on the model of Linguistica by Goldsmith (2001), we attempt a computational treatment of specific word formation phenomena in Modern Greek (MG), such as suffixation and compounding with bound

Through this research the problem of Sindhi Word Segmentation has been addressed and various techniques have been discussed to solve this problem. Word Segmentation is the preliminary phase involved in any tool based on Natural Language Processing (NLP). For any system to understand the written text, it needs to be able to break it into individual tokens for processing. Sindhi being a cursive ligature based Persio-Arabic script, is quite complex and rich having large number of characters in its script with all characters having multiple glyph's based on its position in the text. In this paper Sindhi word Tokenization model has been proposed implementing various algorithms showing the process of tokenizing Sindhi text into individual words for corpus building and creating word repository for Sindhi Spell, grammar checker and other NLP applications. The problem of tokenization is resolved by first identifying the sentence boundaries and extracting each sentence into isolated list form, where each list element is a complete sentence. Then the segregated sentences are broken down into words with hard space character used as word boundaries and soft spaces are considered as part of word and thus ignored from segmenting. Finally each word is again filtered to remove special characters and then each word is converted and saved as token after validation.

This issue includes the following papers;
P1121653529,
AUTHOR="Sameh Magdy and Tarek Mahmoud and Mohamed Ibrahim",
TITLE="Improved Face Recognition based on Hidden Markov Model"

P1121705544,
AUTHOR="Afaf Abd Elkader Abd Elhafiz",
TITLE="Enhancing the Minimum Average Scheduling Algorithm (MASA) based on Makespan Minimizing",

P1121711553,
AUTHOR="Jimdelle C. Kintanar and Russel Roman L. Mallari and Jasmin D. Maullon and Ma. Camille H. Ragual and Julius M. Redilla and Roselito E. Tolentino"

P1121713558,
AUTHOR="Ammar Mohammed and Mohammed Karam and Hesham Hefny",
TITLE="GA-based Parameter Optimization for Word Segmentation"

P1121716566,
AUTHOR="Prof. Kamal ElDahshan and Khalid Walid and Mennatullah Walid and Mohamed Ali ",
TITLE="Automatic Conversion of Natural Language into Relational Database Queries"

Word segmentation: The role of distributional cues. Journal of Memory and Language, 35,[606][607][608][609][610][611][612][613][614][615][616][617][618][619][620][621] have shown that adults and infants can use the statistical properties of syllable sequences to extract words from continuous speech. They also showed that a similar learning mechanism operates with musical stimuli [Saffran, J. R., Johnson, R. E. K., Aslin, N., & Newport, E. L. (1999). Abstract Statistical learning of tone sequences by human infants and adults. Cognition, 70,. In this work we combined linguistic and musical information and we compared language learning based on speech sequences to language learning based on sung sequences. We hypothesized that, compared to speech sequences, a consistent mapping of linguistic and musical information would enhance learning. Results confirmed the hypothesis showing a strong learning facilitation of song compared to speech. Most importantly, the present results show that learning a new language, especially in the first learning phase wherein one needs to segment new words, may largely benefit of the motivational and structuring properties of music in song.

This study examined the changes in word segmentation and word writing from preschool through second grade in Hebrew and Spanish, two languages which use different scripts. Participants were 56 Spanish-speaking and 59 Hebrew-speaking children from middle-class public schools in Barcelona and Tel Aviv. Word segmentation developed with grade level from segmentation according to syllables to segmentation into subsyllabic elements. Very few children were able to completely segment words into consonants and vowels. The syllable was found to be the most important unit of segmentation across grade levels and in both language groups, although to a greater degree in Spanish. Two oral segmentation strategies were language specific: oral spelling in the Spanish group and pronunciation of consonants in isolation in the Hebrew group. Word writing evolved with grade level from formally constrained writing through syllabic mapping to alphabetic writing, according to the particular constraints of each script.

Cognitive neuroscience has focused on language acquisition as one of the main domains to test the respective roles of statistical vs. rule-like computation. Recent studies have uncovered that the brain of human neonates displays a typical signature in response to speech sounds even a few hours after birth. This suggests that neuroscience and linguistics converge on the view that, to a large extent, language acquisition arises due to our genetic endowment. Our research has also shown how statistical dependencies and the ability to draw structural generalizations are basic processes that interact intimately. First, we explore how the rhythmic properties of language bias word segmentation. Second, we demonstrate that natural speech categories play specific roles during language acquisition: some categories are optimally suited to compute statistical dependencies while other categories are optimally suited for the extraction of structural generalizations.

Document segmentation is one of the critical phases in machine recognition of any language. Correct segmentation of individual symbols decides the accuracy of character recognition technique. It is used to decompose image of a sequence of characters into sub images of individual symbols by segmenting lines and words. Devnagari is the most popular script in India. It is used for writing Hindi, Marathi, Sanskrit and Nepali languages. Moreover, Hindi is the third most popular language in the world. Devnagari documents consist of vowels, consonants and various modifiers. Hence proper segmentation of Devnagari word is challenging. A simple histogram based approach to segment Devnagari documents is proposed in this paper. Various challenges in segmentation of Devnagari script are also discussed.

The effect of spacing in relation to word segmentation was examined for four groups of non-native Chinese speakers (American, Korean, Japanese, and Thai) who were learning Chinese as second language. Chinese sentences with four types of spacing information were used: unspaced text, wordspaced text, character-spaced text, and nonword-spaced text. Also, participants' native languages were different in terms of their basic characteristics: English and Korean are spaced, whereas the other two are unspaced; Japanese is character based whereas the other three are alphabetic. Thus, we assessed whether any spacing effects were modulated by native language characteristics. Eye movement measures showed least disruption to reading for word-spaced text and longer reading times for unspaced than characterspaced text, with nonword-spaced text yielding the most disruption. These effects were uninfluenced by native language (though reading times differed between groups as a result of Chinese reading experience). Demarcation of word boundaries through spacing reduces non-native readers' uncertainty about the characters that constitute a word, thereby speeding lexical identification, and in turn, reading. More generally, the results indicate that words have psychological reality for those who are learning to read Chinese as a second language, and that segmentation of text into words is more beneficial to successful comprehension than is separating individual Chinese characters with spaces.

This paper proposes a novel of text document authorization based on SIFT and SURF. It mainly consists of two stages enrollment and identification. In all stages SIFT (scale invariant feature transform) descriptors are extracted the scale and orientation (SOs) of the each sentences. At the same time SURF (speed up robust feature) will extract the scale and orientation (SOs) of the same word. These feature values are stored in the code book. In the identification stage SOs of input handwriting text documents are extracted and classify the document on the basis of lib SVM classifier. Experimental result consist of different English data sets (Iam, Firemaker etc).And by measuring the classification accuracy a comparative study is done on SIFT and SURF algorithms.

Marathi and Hindi both being Indo-Aryan family members and using Devanagari script are similar to a great extent. Both follow SOV sentence structure and are equally liberal in word order. The translation for this language pair appears to be easy. But experiments show this to be a significantly difficult task, primarily due to the fact that Marathi is morphologically richer compared to Hindi. We propose a Marathi to Hindi Statistical Machine Translation (SMT) system which makes use of compound word splitting to tackle the morphological richness of Marathi.

It plays essential role in estimating and comparing research results achieved by different groups of researcher to create standard databases. Many handwritten databases reported are mostly based on Latin, Chinese, Arabic, and other languages, but none of them related with Uyghur handwriting. In this paper, a Uyghur handwritten text database written by multiple writers is introduced. It can be used for research in the recognition of Uyghur handwriting (word or text), word segmentation, document image processing (analysis or retrieval), and writer identification. This database contains 2016 handwritten document images written by 813 native writers of Uyghur. Each handwriting document is scanned, stored with .bmp file, and the noise in the handwriting image is removed. So far, this is the first Uyghur offline handwriting database that is available freely for worldwide researchers.

This paper discusses problems of word and sentence segmentation in Thai. Disagreements on word segmentation are caused mostly from compound words. To set a standard resource and tool of word segmentation, we suggest that only simple words and ...

This paper presents a text word extraction algorithm that takes a set of bounding boxes of glyphs and their associated text lines of a given document and partitions the glyphs into a set of text words, using only the geometric information of the input glyphs. The algorithm is probability based. An iterative, relaxation-like method is used to find the partitioning solution that maximizes the joint probability. To evaluate the performance of our text word extraction algorithm, we used a 3-fold validation method and developed a quantitative performance measure. The algorithm was evaluated on the UW-III database of some 1600 scanned document image pages. An area-overlap measure was used to find the correspondence between the detected entities and the ground-truth.

Chinese word segmentation (CWS), named entity recognition (NER) and part-ofspeech tagging is the lexical processing in Chinese language. This paper describes the work on these tasks done by France Telecom Team (Beijing) at the fourth International Chinese Language Processing Bakeoff. In particular, we employ Conditional Random Fields with different features for these tasks. In order to improve NER relatively low recall; we exploit non-local features and alleviate class imbalanced distribution on NER dataset to enhance the recall and keep its relatively high precision. Some other post-processing measures such as consistency checking and transformation-based error-driven learning are used to improve word segmentation performance. Our systems participated in most CWS and POS tagging evaluations and all the NER tracks. As a result, our NER system achieves the first ranks on MSRA open track and MSRA/CityU closed track. Our CWS system achieves the first rank on CityU open track, which means that our systems achieve state-of-the-art performance on Chinese lexical processing.

Recibido para revisión 12 de Julio de 2001, aceptado 12 de Septiembre de 2001, versión final recibida 30 de Septiembre de 2001 RESUMEN: El presente trabajo muestra los resultados obtenidos con los algoritmos desarrollados para la segmentación y caracterización de células del epitelio escamoso del cérvix mediante el procesamiento de imágenes. El propósito es construir una herramienta para el análisis citológico y que sirva para el prediagnóstico de anomalías que puedan presentarse a nivel del útero y cérvix como es el caso del cáncer de cuello uterino. Fue posible segmentar y caracterizar hasta un 85% de los núcleos en las imágenes analizadas.

Recibido para revisión 12 de Julio de 2001, aceptado 12 de Septiembre de 2001, versión final recibida 30 de Septiembre de 2001 RESUMEN: El presente trabajo muestra los resultados obtenidos con los algoritmos desarrollados para la segmentación y caracterización de células del epitelio escamoso del cérvix mediante el procesamiento de imágenes. El propósito es construir una herramienta para el análisis citológico y que sirva para el prediagnóstico de anomalías que puedan presentarse a nivel del útero y cérvix como es el caso del cáncer de cuello uterino. Fue posible segmentar y caracterizar hasta un 85% de los núcleos en las imágenes analizadas.

Authentication is the act of confirming the truth of an attribute of a datum or entity. This might involve confirming the identity of a person, tracing the origins of an artefact, ensuring that a product is what it's packaging and labelling claims to be, or assuring that a computer program is a trusted one. The authentication of information can pose special problems (especially man-in-the-middle attacks), and is often wrapped up with authenticating identity. Literary can involve imitating the style of a famous author. If an original manuscript, typewritten text, or recording is available, then the medium itself (or its packaging -anything from a box to e-mail headers) can help prove or disprove the authenticity of the document. The use of digital images of handwritten historical documents has become more popular in recent years. Volunteers around the world now read thousands of these images as part of their indexing process. Handwritten text images of old documents are sometimes difficult to read or noisy due to the preservation of the document and quality of the image . Handwritten text offers challenges that are rarely encountered in machine-printed text. In addition, most problems faced in reading machineprinted text (e.g., character recognition, word segmentation, letter segmentation, etc.) are more severe, in handwritten text. In this paper we Here in this paper we proposed a method for authenticating hand written text images using back propagation algorithm..

With growing interest in Chinese Language Processing, numerous NLP tools (e.g. word segmenters, part-of-speech taggers, and parsers) for Chinese have been developed all over the world. However, since no large-scale bracketed corpora are available to the public, these tools are trained on the corpora with different segmentation criteria, part-of-speech tagsets and bracketing guidelines, and therefore, comparisons are difficult. As a first step towards addressing this issue, we have been preparing a 100-thousand-word bracketed corpus since late 1998 and plan to release it to the public summer 2000. In this paper, we will address several challenges in building the corpus, namely, creating annotation guidelines, ensuring annotation accuracy and maintaining a high level of community involvement. 1. Introduction With growing interest in Chinese Language Processing, numerous NLP tools (e.g. word segmenters, part-of-speech taggers, and parsers) for Chinese have been developed all over the w...

This paper describes the state of the art in segmentation algorithms of aerial images. Different approaches and object classes are described and their advantages and limitations are shown. First the advantage of multiple input data (e.g., color, infrared, DEM) and the information that can be derived from these sources is discussed. Besides sensor data, "synthetic" input images (e.g., using texture filters) are generated to support the segmentation process. After an optional noise cleaning, primitives are extracted in scale space. This offers the possibility of selecting an optimal resolution depending on the size and shape of an object. Using this resolution, the raw segmentation will be stable and conflicts with other object classes will be reduced. Depending on the class of the object the final extraction has to be selected: Compact artificial objects can be segmented using primitives like areas, lines, or points. Linear objects like roads are similar but the borders are curves and the size is not limited. Arbitrary areas like meadows, forests, or fields have an arbitrary border and are mainly defined by their specific texture. Objects like trees or cars have to be treated in a very specific manner. Finally, different base algorithms for segmentation are discussed: Pixel classification is very simple but lacks the use of context. The extraction of primitives (egdes, lines, area, points) can be used as a basis for a wide class of objects. Texture analysis can be used for a rough segmentation of the image. Specialized operations are useful for the extraction of objects like single trees or to support the interpretation process.

We investigate whether adult learners' knowledge of phonotactic restrictions on word forms from their first language (L1) impact their word segmentation abilities in a new language. Adult learners were exposed to a speech stream in which language specific and non-language specific cues for word segmentation were pitted against one another. English rules about possible phonetic combinations (phonotactics) and transitional probabilities of syllables conflicted such that predictive transitional probabilities generated words that were phonotactically impossible in English. A control with phonotactically viable items was also run. At test, participants choose between words defined by deterministic transitional probabilities and words that are phonotactically possible in English, but have much lower transitional probabilities. A baseline of their abilities to track transitional probabilities in the stimuli was also collected. Results suggest that although participants are able to track the transitional probabilities in these stimuli, they are not using them to segment and extract words. Control subjects, however, do use transitional probabilities to segment words. This pattern of results is resilient, holding up with substantial increases in exposure and even when segmentation is encouraged by explicitly giving participants one of the words in the stream prior to exposure.

We present a series of experiments involving the machine translation of Zulu to English using a well-known statistical software system. Due to morphological complexity and relative scarcity of resources, the case of Zulu is challenging. Against a selection of baseline models, we show that a relatively naive approach of dividing Zulu words into syllables leads to a surprising improvement. We further improve on this model through manual configuration changes. Our best model significantly outperforms the baseline models (BLEU measure, at p < 0.001) even when they are optimised to a similar degree, only falling short of the well-known Morfessor morphological analyser that makes use of relatively sophisticated algorithms. These experiments suggest that even a simple optimisation procedure can improve the quality of this approach to a significant degree. This is promising particularly because it improves on a mostly language independent approach — at least within the same language family. Our work also drives the point home that sub-lexical alignment for Zulu is crucial for improved translation quality.

Vine-plot mapping and monitoring are crucial issues in land management, particularly for areas where vineyards are dominant, like in some French regions. In this context, the availability of an automatic tool for vineyard detection and characterization would be very useful. Due to the periodic patterns induced by this culture, frequency an al sis appears to be a very suited tool for vineyard detection in aerial images. A recursive process using Fast Fourier Transfor m algorithm has been developed to meet this need. This results in vine plot segmentation, with contours in polygonal form and characterization w ith accurate estimation of interrow width and row orientation. To foster large-scale applications, tests and validation have been carried out on standard very high spatial resolution remote-sensing data. More than 71% of adults, mechanically trained vines have been well d et cted with 44% having a good contour extraction and 27% beeing grouped by two or three.

Two novel approaches to extract text lines and words from handwritten document are presented. The line segmentation algorithm is based on locating the optimal succession of text and gap areas within vertical zones by applying Viterbi algorithm. Then, a text-line separator drawing technique is applied and finally the connected components are assigned to text lines. Word segmentation is based on a gap metric that exploits the objective function of a soft-margin linear SVM that separates successive connected components. The algorithms tested on the benchmarking datasets of ICDAR07 handwriting segmentation contest and outperformed the participating algorithms.

This study analyzes (speakers of English, Croatian, Austrian). Hypothesis 1 is that signers will show sensitivity to more simultaneous information and sign language (SL) specific phonological the word-segmentation strategies used in 3 groups of Deaf signers (ASL, HZJ, ÖGS) and 3 groups of hearing nonsigners constraints than nonsigners, such as distribution of handshapes (HS), places of articulation (POA), and movements

Question Answering (QA) is becoming an increasingly important research area in natural language processing. Since 1999, many international question answering contests have been held at conferences and workshops, such as TREC, CLEF, and NTCIR. Thus far, eleven languages – Bulgarian, Dutch, English, Finnish, French, German, Indonesian, Italian, Japanese, Portuguese, and Spanish – have been tested on monolingual or cross-lingual question answering tasks. Although Chinese is the second most popular language in the world, NTCIR only conducted the first QA contest in Chinese this year. The results reveal that there seems to be a performance gap between Chinese question answering systems and some systems of other languages. In this paper, we review previous works on Chinese question answering, including our two systems on frequently asked questions and factoid questions. Comparing Chinese with other languages, word segmentation is a key problem in Chinese question answering. We review stud...

Recibido para revisión 12 de Julio de 2001, aceptado 12 de Septiembre de 2001, versión final recibida 30 de Septiembre de 2001 RESUMEN: El presente trabajo muestra los resultados obtenidos con los algoritmos desarrollados para la segmentación y caracterización de células del epitelio escamoso del cérvix mediante el procesamiento de imágenes. El propósito es construir una herramienta para el análisis citológico y que sirva para el prediagnóstico de anomalías que puedan presentarse a nivel del útero y cérvix como es el caso del cáncer de cuello uterino. Fue posible segmentar y caracterizar hasta un 85% de los núcleos en las imágenes analizadas.

This paper describes the techniques we explored to improve the translation of news text in the German-English and Hungarian-English tracks of the WMT09 shared translation task. Beginning with a convention hierarchical phrase-based system, we found benefits for using word segmentation lattices as input, explicit generation of beginning and end of sentence markers, minimum Bayes risk decoding, and incorporation of a feature scoring the alignment of function words in the hypothesized translation. We also explored the use of monolingual paraphrases to improve coverage, as well as co-training to improve the quality of the segmentation lattices used, but these did not lead to improvements.

Uniform resource locators (URLs), which mark the address of a resource on the World Wide Web, are often human-readable and can hint at the category of the resource. This paper explores the use of URLs for web page categorization via a two-phase pipeline of word segmentation/expansion and classification. We quantify its performance against document-based methods, which require the retrieval of the source document.