Search

The absence of explicit word boundaries is a distinctive characteristic of Chinese script, setting it apart from most alphabetic scripts, leading to word boundary disagreement among readers. Previous studies have examined how this feature may influence reading performance. However, further investigations are required to generate more ecologically valid and generalizable findings. In order to advance our understanding of the impact of word boundaries in Chinese reading, we introduce the Chinese Word Segmentation Agreement (CWSA) corpus. This corpus consists of 500 sentences, comprising 9813 character tokens and 1590 character types, and provides data on word segmentation agreement at each character position. The data revealed a high level of overall segmentation agreement (92%). However, participants disagreed on the position of word boundaries in 8.96% of the cases. Moreover, about 85% of the sentences contained at least one ambiguous word boundary. The character strings with high levels of disagreement were tentatively classified into three categories, namely the morphosyntactic type (e.g., “反映–了”), modifier–head type (e.g., “科學–教育”), and others (e.g., “大力–支持”). Finally, the agreement scores also significantly influenced reading behaviors, as evidenced by analyses with published eye movement data. Specifically, a high level of disagreement was associated with longer single fixation durations. We discuss the implications of these results and highlight how the CWSA corpus can facilitate future research on word segmentation in Chinese reading. [ABSTRACT FROM AUTHOR]

Fuzzy logic is the core method for handling uncertainty and vagueness of information in agricultural natural language processing, and it also plays a crucial role in word segmentation and text classification algorithms using the neural network. Word segmentation is often the primary step in Chinese text classification tasks and has a profound effect on the generation ability of classification algorithm-based fuzzy logic. However, the high complexity of text classification models structure and specificity of agricultural data take a great challenge to studying the effect of word segmentation. Although there have been several attempts to resolve this issue, the main effort focuses on word segment Precision or the generalization performance of multiple word segment methods for the same classification algorithm and does not involve agricultural text. To solve this problem from the perspective of rational analysis and empirical analysis, a comprehensive analysis has been made to study the effect of Chinese word segmentation on fuzzy-based classification algorithms for agricultural questions. It initially discusses the characteristics of agricultural questions for the subsequent analysis of the field adaptability of word segmentation and classification algorithms, employs fuzzy logic to convert the Chinese word segmentation task into a sequence labeling problem, and then analyzes the characteristics, techniques, and performance disparities of the seven mainstream open-source Chinese word segmentation integration tools at the current stage. Subsequently, an exploration has been conducted into the impact of Chinese word segmentation on the generalization performance of classification algorithms under the proposed unified model framework for text classification based on fuzzy logic. Finally, many experiments have been performed on the actual data crawled from typical agricultural websites to empirically study the differences and robustness of the effect of different word segmentation tools on classification performance, as well as the contribution of the external dictionary. Comparative experimental results show which word segmentation tools have a solid effect on classification performance and a strong robust effect on the typical text feature extraction layer for classification tasks, and the external dictionary have no significant effect on classification performance. The research results have essential reference significance for how to select appropriate word segmentation tools to deal with Chinese natural language processing tasks in future. [ABSTRACT FROM AUTHOR]

We reported a large-scale Internet-based experiment to investigate the impact of emotion information on Chinese word segmentation, in which participants completed an overlapping ambiguous string (OAS) segmentation task and the Chinese version of Beck Depression Inventory-II in a counterbalanced order. OAS is a three-character string (ABC) in which the middle character can form a distinct word with both the character on its left side (word AB) and the character on its right side (word BC). Participants were presented with isolated OASs and were asked to report the word they identified first. Emotional OAS was constructed by a combination of a neutral word and an emotional word, with the neutral and emotional words sharing character B. We orthogonally manipulated the valence of the emotional words (positive vs. negative) and their position in the OAS (left-side vs. right-side). The results showed that compared with neutral words, both positive and negative words were more likely to be segmented, and this segmentation outcome was not affected by readers with different depression tendencies. These findings suggest that emotion information can influence word segmentation, and that both positive and negative words take precedence over neutral words in the word segmentation process. This study provides a new perspective and evidence to understand the impact of emotion information on word processing. [ABSTRACT FROM AUTHOR]

One of the first tasks in language acquisition is word segmentation, a process to extract word forms from continuous speech streams. Statistical approaches to word segmentation have been shown to be a powerful mechanism, in which word boundaries are inferred from sequence statistics. This approach requires the learner to represent the frequency of units from syllable sequences, though accounts differ on how much statistical exposure is required. In this study, we examined the computational limit with which words can be extracted from continuous sequences. First, we discussed why two occurrences of a word in a continuous sequence is the computational lower limit for this word to be statistically defined. Next, we created short syllable sequences that contained certain words either two or four times. Learners were presented with these syllable sequences one at a time, immediately followed by a test of the novel words from these sequences. We found that, with the computationally minimal amount of two exposures, words were successfully segmented from continuous sequences. Moreover, longer syllable sequences providing four exposures to words generated more robust learning results. The implications of these results are discussed in terms of how learners segment and store the word candidates from continuous sequences. [ABSTRACT FROM AUTHOR]

This paper focuses on the segmentation of unconstrained handwritten documents containing bilingual data at the word level. Most of the official documents available in India are bilingual, i.e., in the regional language as well as English language. For the purpose of current research work, the handwritten documents from the academic domain containing Punjabi from Gurumukhi script and English language from Latin script have been considered. A heuristic approach based on projection profiles, statistical and structural properties of script has been designed to segment the textual data of documents. The proposed approach can segment the closed, curved, skewed and touched text lines having large variations in pattern and size. For word segmentation, an end-point detection algorithm has been designed to segment the words with intra word gap. The proposed approaches have been evaluated by designing a domain-based bilingual handwritten dataset after having consultations with academicians, and experts in the field. For text line segmentation, an average accuracy of 95.89% and 92.95% has been achieved for IAM and bilingual dataset respectively. However, for word segmentation, there has been an accuracy of 89.74% and 92.25% respectively for IAM and bilingual dataset. As many as 280 documents with various writing styles and content have been selected for the purpose. [ABSTRACT FROM AUTHOR]

Two classes of cognitive mechanisms have been proposed to explain segmentation of continuous sensory input into discrete recurrent constituents: clustering and boundary-finding mechanisms. Clustering mechanisms are based on identifying frequently co-occurring elements and merging them together as parts that form a single constituent. Bracketing (or boundary-finding) mechanisms work by identifying rarely co-occurring elements that correspond to the boundaries between discrete constituents. In a series of behavioral experiments, I tested which mechanisms are at play in the visual modality both during segmentation of a continuous syllabic sequence into discrete word-like constituents and during recognition of segmented constituents. Additionally, I explored conscious awareness of the products of statistical learning—whole constituents versus merged clusters of smaller subunits. My results suggest that both online segmentation and offline recognition of extracted constituents rely on detecting frequently co-occurring elements, a process likely based on associative memory. However, people are more aware of having learnt whole tokens than of recurrent composite clusters. [ABSTRACT FROM AUTHOR]

One of the several challenges faced by neural machine translation systems is the lack of standard parallel corpora for several language pairs. Poor translation qualities often result from inadequate data. Aggravating this problem further are the issues of morphological complexity and agglutination, leading to unmanageable vocabulary size, rare words and data sparsity issues. Though this problem has been partly addressed by sub-word algorithms such as BPE, translation systems still lag in their ability to understand sentence and word structures associated with rich morphologies. This paper aims to address these issues by employing linguistically driven sub-word units into NMT systems. This approach is further enhanced by additional POS tag feature inputs. The proposed approach outperforms BPE driven machine translation models by several BLEU points and is also shown to have better recall measures from evaluation by ROUGE metric. The results have been evaluated upon a morphologically complex Dravidian language pair, Kannada and Telugu. [ABSTRACT FROM AUTHOR]

Sanskrit processing has seen a surge in the use of data-driven approaches over the past decade. Various tasks such as segmentation, morphological parsing, and dependency analysis have been tackled through the development of state-of-the-art models despite working with relatively limited datasets compared to other languages. However, a significant challenge lies in the availability of annotated datasets that are lexically, morphologically, syntactically, and semantically tagged. While syntactic and semantic tags are preferable for later stages of processing such as sentential parsing and disambiguation, lexical and morphological tags are crucial for low-level tasks of word segmentation and morphological parsing. The Digital Corpus of Sanskrit (DCS) is one notable effort that hosts over 650,000 lexically and morphologically tagged sentences from around 250 texts but also comes with its limitations at different levels of a sentence like chunk, segment, stem and morphological analysis. To overcome these limitations, we look at alternatives such as Sanskrit Heritage Segmenter (SH) and Saṃsādhanī tools, that provide information complementing DCS’ data. This work focuses on enriching the DCS dataset by incorporating analyses from SH, thereby creating a dataset that is rich in lexical and morphological information. Furthermore, this work also discusses the impact of such datasets on the performances of existing segmenters, specifically the Sanskrit Heritage Segmenter. [ABSTRACT FROM AUTHOR]

The study's aims were to (a) evaluate preschoolers' use of private speech (overt talk to themselves) during spelling; and (b) study how it is affected by the nature of orthography. Participants were 197 Hebrew speaking Israeli preschoolers (109 girls and 88 boys) (M = 5.6 years). Children spelled 12 words (N =44 letters) that represented one of three phonological word structures in terms of their consonants (C) and vowels (V): CV.CVC, CV.CV.CVC, and CVC.CVC. Children's private speech during spelling was documented and analysed. In this paper, we report the private speech units most frequently produced when spelling the letters correctly—CV, CVC, and letter name. When using private speech, children succeeded in correctly spelling a greater number of letters (30.69%) than when not using private speech (17.64%). The private speech across word structures primarily contained CV units and letter names. Children used private speech mostly for words' first letters and for CV.CVC words. The structure and position of the letters (first, second, last) had a combined effect on the production of private speech. CV units were used mostly in the first letter of CV.CVC words, CVC units in the second letter of a CVC.CVC word, and letter names in the first letter of CV.CVC words. These findings suggest that preschoolers are aware of the features of the orthography, as exemplified by their internal thought process when spelling words. Adults can support children's understanding of the written language by encouraging them to use private speech during word writing. [ABSTRACT FROM AUTHOR]

Language-specific features necessitate certain processes and skills in reading. The visually unmarked between-word boundaries in written Chinese render it critical that readers be able to segment words in the continuous texts. It may pose challenges for second language (L2) readers whose first language (L1) is word-spaced. In light of the lack of understanding of Chinese L2 readers' word segmentation, the present study investigated 100 L2 learners' word segmentation performance, the relationships between word segmentation and reading fluency and comprehension, and the differences in the aforementioned relationships among learners with different context-free word recognition abilities. Results demonstrated that L2 learners generally conducted word segmentation well and word segmentation contributed to reading fluency and comprehension beyond context-free word recognition. Word segmentation was the major predictor of reading fluency and comprehension among weak wordlist readers, whereas context-free word recognition made a larger contribution when learners have achieved stronger context-free word recognition ability. The findings suggested the importance of developing word segmentation skills and establishing high quality word representations in Chinese L2 reading instruction. [ABSTRACT FROM AUTHOR]

This paper presents the text line and word segmentation from unconstrained handwritten documents based on horizontal projection histogram (HPH) to detect mid-points and gap trailing between lines. The midpoints are estimated from the HPH for the first 100 to 200 columns of the whole document. Then, considering the mid-points, the gap is tracked between two consecutive lines from locally computed HPH for a block having k rows and j columns. The HPH block is examined for various cases to locate optimal rows that separate adjacent lines. The proposed method segments curve, touching and skew-lines and is robust to writing variation and language independent. Word segmentation is not treated as a separate problem and goes efficiently alongside the line segmentation. As the trailing of space between neighboring lines goes on, the vertical projection Histogram (VPH) of t columns is monitored between the above and below separator of a line and find the optimal word separator. The algorithm is evaluated on two isolated datasets of different languages (Meitei Mayek and English). Text-line and word segmentation on Meitei Mayek handwritten documents achieve 91.84% and 88.96% accuracy respectively. Similarly, the handwritten English document meets 94.18% and 87.73% accuracy for line and word segmentation. [ABSTRACT FROM AUTHOR]

This paper proposes a model which is used to recognize online handwritten Bangla words. This word recognition module comprises of different modules: preprocessing of the word samples, segmentation of words into basic strokes, recognizing the basic strokes using multilayer perceptron, followed by recognition of words using Hidden Markov Model (HMM) aided by Language Model (LM). For stroke recognition, two different feature extraction techniques (point-based and curvature-based procedures) are used using late fusion technique. Top 5 stroke recognition choices are used to construct HMM for the prediction of word sample. An N-gram LM is applied as a post-processing step to rectify the HMM outcomes if required. A total of 50 different word samples with 110 instances each are used to evaluate the proposed model. The overall stroke-level and word-level recognition accuracies obtained by this model are 95.4% and 90.3%, respectively. The proposed model can be extended to recognize online handwritten words written in other script like Devanagari, Assamese, and Gurumukhi, etc. The methodologies described in the manuscript can also be applied for offline word recognition purpose. [ABSTRACT FROM AUTHOR]

Two eye-movement experiments are reported in which a boundary paradigm was used to manipulate the presence versus absence of boundaries for high-frequency and low-frequency target words in the parafovea. In Experiment 1, this was done by introducing a blank space after the target words, whereas in Experiment 2 this was done by rendering the target words in red. In both experiments, higher frequency targets engendered longer saccades, whereas the presence of parafoveal word boundaries engendered shorter saccades. This pattern suggests the operation of two countermanding saccade-targeting mechanisms: one that uses parafoveal processing difficulty to adjust saccade lengths and a second that uses word boundaries to direct the eyes toward specific saccade targets. The implications of these findings for models of eye-movement control during reading are discussed, as are suggestions for integrating dynamic-adjustment and default-targeting accounts. [ABSTRACT FROM AUTHOR]

During sentence reading, low spatial frequency information afforded by spaces between words is the primary factor for eye guidance in spaced writing systems, whereas saccade generation for unspaced writing systems is less clear and under debate. In the present study, we investigated whether word-boundary information, provided by alternating colors (consistent or inconsistent with word-boundary information) influences saccade-target selection in Chinese. In Experiment 1, as compared to a baseline (i.e., uniform color) condition, word segmentation with alternating color shifted fixation location towards the center of words. In contrast, incorrect word segmentation shifted fixation location towards the beginning of words. In Experiment 2, we used a gaze-contingent paradigm to restrict the color manipulation only to the upcoming parafoveal words and replicated the results, including fixation location effects, as observed in Experiment 1. These results indicate that Chinese readers are capable of making use of parafoveal word-boundary knowledge for saccade generation, even if such information is unfamiliar to them. The present study provides novel support for the hypothesis that word segmentation is involved in the decision about where to fixate next during Chinese reading. [ABSTRACT FROM AUTHOR]

Automatic translation of Chinese text to Chinese Braille is important for blind people in China to acquire information using computers or smart phones. In this paper, a novel scheme of Chinese-Braille translation is proposed. Under the scheme, a Braille word segmentation model based on statistical machine learning is trained on a Braille corpus, and Braille word segmentation is carried out using the statistical model directly without the stage of Chinese word segmentation. This method avoids establishing rules concerning syntactic and semantic information and uses statistical model to learn the rules stealthily and automatically. To further improve the performance, an algorithm of fusing the results of Chinese word segmentation and Braille word segmentation is also proposed. Our results show that the proposed method achieves accuracy of 92.81% for Braille word segmentation and considerably outperforms current approaches using the segmentation-merging scheme. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the relative effect of two strategies for language resource addition for Japanese morphological analysis, a joint task of word segmentation and part-of-speech tagging. The first strategy is adding entries to the dictionary and the second is adding annotated sentences to the training corpus. The experimental results showed that addition of annotated sentences to the training corpus is better than the addition of entries to the dictionary. In particular, adding annotated sentences is especially efficient when we add new words with contexts of several real occurrences as partially annotated sentences, i.e. sentences in which only some words are annotated with word boundary information. According to this knowledge, we performed real annotation experiments on invention disclosure texts and observed word segmentation accuracy. Finally we investigated various language resource addition cases and introduced the notion of non-maleficence, asymmetricity, and additivity of language resources for a task. In the WS case, we found that language resource addition is non-maleficent (adding new resources causes no harm in other domains) and sometimes additive (adding new resources helps other domains). We conclude that it is reasonable for us, NLP tool providers, to distribute only one general-domain model trained from all the language resources we have. [ABSTRACT FROM AUTHOR]

The distal prosodic patterning established at the beginning of an utterance has been shown to influence downstream word segmentation and lexical access. In this study, we investigated whether distal prosody also affects word learning in a novel (artificial) language. Listeners were exposed to syllable sequences in which the embedded words were either congruent or incongruent with the distal prosody of a carrier phrase. Local segmentation cues, including the transitional probabilities between syllables, were held constant. During a test phase, listeners rated the items as either words or nonwords. Consistent with the perceptual grouping of syllables being predicted by distal prosody, congruent items were more likely to be judged as words than were incongruent items. The results provide the first evidence that perceptual grouping affects word learning in an unknown language, demonstrating that distal prosodic effects may be independent of lexical or other language-specific knowledge. [ABSTRACT FROM AUTHOR]

Statistical segmentation of continuous speech, i.e., the ability to utilise transitional probabilities between syllables in order to detect word boundaries, is reflected in the brain's auditory event-related potentials (ERPs). The N1 and N400 ERP components are typically enhanced for word onsets compared to random syllables during active listening. We used magnetoencephalography (MEG) to record event-related fields (ERFs) simultaneously with ERPs to syllables in a continuous sequence consisting of ten repeating tri-syllabic pseudowords and unexpected syllables presented between these pseudowords. We found the responses to differ between the syllables within the pseudowords and between the expected and unexpected syllables, reflecting an implicit process extracting the statistical characteristics of the sequence and monitoring for unexpected syllables. [ABSTRACT FROM AUTHOR]

The present study explored the influence of a new metrics of phonotactics on adults' use of transitional probabilities to segment artificial languages. We exposed French native adults to continuous streams of trisyllabic nonsense words. High-frequency words had either high or low congruence with French phonotactics, in the sense that their syllables had either high or low positional frequency in French trisyllabic words. At test, participants heard low-frequency words and part-words, which differed in their transitional probabilities (high for words, low for part-words) but were matched for frequency and phonotactic congruency. Participants' preference for words over part-words was found only in the high-congruence languages. These results establish that subtle phonotactic manipulations can influence adults' use of transitional probabilities to segment speech and unambiguously demonstrate that this prior knowledge interferes directly with segmentation processes, in addition to affecting subsequent lexical decisions. Implications for a hierarchical theory of segmentation cues are discussed. [ABSTRACT FROM AUTHOR]

The purpose of this study was to examine if the bilingual advantage of word awareness reported in an earlier study can last to the second year. 30 bilingual and 30 monolingual preschoolers (mean age 5) participated in this study. Three word awareness tasks were used to examine the marginal bilingual and monolingual children's word awareness skills: a word segmentation task (segmenting words and word count), word judgment task, e.g., word size judgment, "maomaochong" [caterpillar] (a long word but a creature small in size) vs. zhu [pig] (a short word but an animal large in size), and name manipulation task (declarative/procedural knowledge in label change). The results show that the bilingual group continued to significantly outperform their monolingual counterparts on the name manipulation task. In addition, they also outperformed their monolingual counterparts on word count and incongruous word size judgment tasks. These findings corroborate Bialystok's (1987) proposal that the more sophisticated development of metalinguistic skills by children learning a second language does not hinge on comparable competence in the two languages, as proposed by Cummins (1987). Rather, it demonstrates that the child's recognition of the coexistence of two separate language systems in representing the same set of concepts is the key to their enhanced word awareness (Bialystok, 1988). [ABSTRACT FROM AUTHOR]

Unsupervised morphological analysis is the task of segmenting words into prefixes, suffixes and stems without prior knowledge of language-specific morphotactics and morpho-phonological rules. This paper introduces a simple, yet highly effective algorithm for unsupervised morphological learning for Bengali, an Indo–Aryan language that is highly inflectional in nature. When evaluated on a set of 4,110 human-segmented Bengali words, our algorithm achieves an F-score of 83%, substantially outperforming Linguistica, one of the most widely-used unsupervised morphological parsers, by about 23%. [ABSTRACT FROM AUTHOR]

This article presents a combination of unsupervised and supervised learning techniques for the generation of word segmentation rules from a raw list of words. First, a language bias for word segmentation is introduced and a simple genetic algorithm is used in the search for a segmentation that corresponds to the best bias value. In the second phase, the words segmented by the genetic algorithm are used as an input for the first order decision list learner CLOG. The result is a set of first order rules which can be used for segmentation of unseen words. When applied on either the training data or unseen data, these rules produce segmentations which are linguistically meaningful, and to a large degree conforming to the annotation provided. [ABSTRACT FROM AUTHOR]

The previously proposed syllable-synchronous network search (SSNS) algorithm plays a very important role in the word decoding of the continuous Chinese speech recognition and achieves satisfying performance. Several related key factors that may affect the overall word decoding effect are carefully studied in this paper, including the perfecting of the vocabulary, the big-discount Turing re-estimating of the N-Gram probabilities, and the managing of the searching path buffers. Based on these discussions, corresponding approaches to improving the SSNS algorithm are proposed. Compared with the previous version of SSNS algorithm, the new version decreases the Chinese character error rate (CCER) in the word decoding by 42.1% across a database consisting of a large number of testing sentences (syllable strings). [ABSTRACT FROM AUTHOR]

In this paper a novel word-segmentation algorithm is presented to delimit words in Chinese natural language queries in NChiql system, a Chinese natural language query interface to databases. Although there are sizable literatures on Chinese segmentation, they cannot satisfy particular requirements in this system. The novel word-segmentation algorithm is based on the database semantics, namely Semantic Conceptual Model (SCM) for specific domain knowledge. Based on SCM, the segmenter labels the database semantics to words directly, which eases the disambiguation and translation (from natural language to database query) in NChiql. [ABSTRACT FROM AUTHOR]