Search

Handwritten documents are, as always, highly challenging for recognition tasks compared to printed documents. Rather than using isolated characters as elementary components for recognition, practical documents use words or character strings. In any handwritten recognition task, the segmentation of lines and words plays a pivotal role as the outputs produced at this stage can drastically affect the performance and the results of the recognition tasks. An approach combining two distinct techniques, namely horizontal projection profile and seam carving, for the segmentation of lines has been proposed in this paper. Using the horizontal projection profile method, a general idea of the location of lines in the document is obtained first, but since only using the horizontal projection profile method works better for printed documents, it is not enough for handwritten documents, so the seam carving method is applied to finely segment the lines, where line separation distance varies from writer to writer. Dynamic programming is used to create an energy matrix from the input image and determine the minimum energy paths from left to right. For word segmentation, contour points are traced before applying the seam carving algorithm to find possible paths, and paths that are intersecting with the characters of the text are removed. The standard publicly available IAM English handwritten dataset and the Bangla Writing dataset are used to analyse the text-line and line-word segmentation technique, and the results show promising recognition accuracy. more...

This article presents two massive datasets for central Kurdish handwriting digits and isolated characters named K-ZHMARA and K-PIT. The first dataset, named K-ZHMARA dataset, contains 70,000 images of Kurdish digits, 7000 images for each digit, and a printed A4 paper with a grid of 10 × 10 is used for data collection. Apart from digits, the K-PIT dataset includes 245,000 images of all Kurdish characters, 7000 images for each character; data was collected via a printed A4 paper with a grid of 12 × 10 for this dataset. Moreover, both datasets include 315,000 images. Python programming has been used to scan each piece of paper, segment, crop, resize, binarize, and invert the images via edge detection and image processing techniques. more...

Characters segmentation is a necessity and the most critical stage in Arabic OCR system. It has attracted the interest of a wide range of researchers. However, the nature of the Arabic cursive script poses extra challenges that need further investigation. Therefore, having a reliable and efficient Arabic OCR system that is independent of font variations is highly required. In this paper, an indirect, font-in dependent word and character segmentation algorithm for printed Arabic text investigated. The proposed algorithm takes a binary line image as an input and produces a set of binary images consisting of one character or ligature as an output. The segmentation performed at two levels: a word segmentation performed in the first level, by employing a vertical projection at the input line image along with using Interquartile Range (IQR) method to differentiate between word gaps and within word gaps. A projection profile method used as a second level of segmentation along with a set of statistical and topological features, which are font-independent, to identify the correct segmentation points from all potential points. The APTI dataset used to test the proposed algorithm with a variety of font type, size, and style. The algorithm experimented on 1800 lines (approximately 24,816 words) with an average accuracy of 97.7% for words segmentation and 97.51% for characters segmentation. more...

In analytical systems of Arabic text recognition, segmentation is a critical and decisive stage. In this stage, words after being extracted from the text document are segmented into individual characters to allow the feature extraction and classification. The absence of standard corpus for printed Arabic text does not allow a good and objective comparison between different segmentation systems. In this paper, we propose two contributions. The first one is a multi-font reference corpus of printed Arabic text, in which we grouped all segmentation problems and which can be used as a reference to compare different segmentation systems. The second is a hybrid method for segmentation of printed multi-font Arabic text. The proposed method is based on two of the most known techniques in the field: contour based and Template matching techniques. This method is insensitive to the font variability and the characters overlapping. To evaluate the results of this method, we have studied, implemented and tested other methods in literature by using the proposed corpus. The experimental results show that our method gives better segmentation rates. more...

Despite the essential role of statistical learning in shaping human behavior, there are still controversies concerning its measurement. In this paper, we present a novel online target-detection task in an acoustic word segmentation paradigm, which is able to track the process of learning and does not build on deliberation and decision making. Beside testing the novel online task, we also examined its relationship with two offline measures: the traditional two-alternative forced choice (2AFC) task, and the statistically-induced chunking recall (SICR) task (Isbilen et al., 2017). Participants showed a significant learning effect on the online task, reflected in the decrease of reaction times during training and in the differences between reaction times to predictable versus unpredictable targets. Online learning scores correlated with the 2AFC scores, but this association was only present when participants did not have explicit knowledge about stimuli. SICR scores were not associated with any of the other measures. The internal consistency was higher for online learning measures than for the other two tasks. These findings show that the online target detection task is a good tool for assessing statistical learning, and invite further research on its psychometric properties. more...

Background: There is growing evidence that COVID-19 brought changes that immediately affected early language development. Little is known for how long these COVID-19 related changes shaped development. The current study systematically and longitudinally addressed this issue, following up children's language development throughout the first 2.5 years., Method: The present study follows up on the sample from Frota et al. (2022), which demonstrated that 7-9-month-old infants born and raised during the pandemic do not segment words unlike pre-pandemic peers. Four studies were conducted: (1) word segmentation task at 12 months of age (N pandemic  = 15); (2) word learning task at 20 months of age (N pandemic  = 20); (3) language and communication development up to 30 months of age, via CDI and CSBS parental reports (N pandemic ranged 25-74); (4) overall development at 30 months of age using the Griffiths Developmental Scales (N pandemic  = 16)., Results: The pandemic sample consistently underperformed in all four studies in comparison to pre-pandemic data. There was no evidence of developed word segmentation abilities at 12 months of age, and no successful word learning at 20 months of age. Lexical development between 12 and 24 months of age was lower than in the pre-pandemic sample, while social communication did not seem to be affected. At 30 months of age, the pandemic sample showed lower scores and lower mental age on the Language and Communication Griffiths' subscale, in comparison to the pre-pandemic data., Conclusions: Infants born and raised during the pandemic have a poorer language development, that persists at least until 30 months of age., Competing Interests: Declaration of competing interest The authors declare no conflict of interest., (Copyright © 2024. Published by Elsevier B.V.) more...

This article presents a Bangla handwriting dataset named BanglaWriting that contains single-page handwritings of 260 individuals of different personalities and ages. Each page includes bounding-boxes that bounds each word, along with the unicode representation of the writing. This dataset contains 21,234 words and 32,787 characters in total. Moreover, this dataset includes 5,470 unique words of Bangla vocabulary. Apart from the usual words, the dataset comprises 261 comprehensible overwriting and 450 handwritten strikes and mistakes. All of the bounding-boxes and word labels are manually-generated. The dataset can be used for complex optical character/word recognition, writer identification, handwritten word segmentation, and word generation. Furthermore, this dataset is suitable for extracting age-based and gender-based variation of handwriting. more...

Over the first year, infants begin to learn the words of their language. Previous work suggests that certain statistical regularities in speech could help infants segment the speech stream into words, thereby forming a proto-lexicon that could support learning of the eventual vocabulary. However, computational models of word segmentation have typically been tested using language input that is much less variable than actual speech is. We show that using actual, transcribed pronunciations rather than dictionary pronunciations of the same speech leads to worse segmentation performance across models. We also find that phonologically variable input poses serious problems for lexicon building, because even correctly segmented word forms exhibit a complex, many-to-many relationship with speakers' intended words. Many phonologically distinct word forms were actually the same intended word, and many identical transcriptions came from different intended words. The fact that previous models appear to have substantially overestimated the utility of simple statistical heuristics suggests a need to consider the formation of the lexicon in infancy differently., Competing Interests: Declaration of Competing Interest None., (Copyright © 2023 Elsevier B.V. All rights reserved.) more...

While the languages of the world differ in many respects, they share certain commonalties, which can provide insight on our shared cognition. Here, we explore the learnability consequences of one of the striking commonalities between languages. Across languages, word frequencies follow a Zipfian distribution, showing a power law relation between a word's frequency and its rank. While their source in language has been studied extensively, less work has explored the learnability consequences of such distributions for language learners. We propose that the greater predictability of words in this distribution (relative to less skewed distributions) can facilitate word segmentation, a crucial aspect of early language acquisition. To explore this, we quantify word predictability using unigram entropy, assess it across languages using naturalistic corpora of child-directed speech and then ask whether similar unigram predictability facilitates word segmentation in the lab. We find similar unigram entropy in child-directed speech across 15 languages. We then use an auditory word segmentation task to show that the unigram predictability levels found in natural language are uniquely facilitative for word segmentation for both children and adults. These findings illustrate the facilitative impact of skewed input distributions on learning and raise questions about the possible role of cognitive pressures in the prevalence of Zipfian distributions in language., (Copyright © 2022 Elsevier B.V. All rights reserved.) more...

How can infants detect where words or morphemes start and end in the continuous stream of speech? Previous computational studies have investigated this question mainly for English, where morpheme and word boundaries are often isomorphic. Yet in many languages, words are often multimorphemic, such that word and morpheme boundaries do not align. Our study employed corpora of two languages that differ in the complexity of inflectional morphology, Chintang (Sino-Tibetan) and Japanese (in Experiment 1), as well as corpora of artificial languages ranging in morphological complexity, as measured by the ratio and distribution of morphemes per word (in Experiments 2 and 3). We used two baselines and three conceptually diverse word segmentation algorithms, two of which rely purely on sublexical information using distributional cues, and one that builds a lexicon. The algorithms' performance was evaluated on both word- and morpheme-level representations of the corpora. Segmentation results were better for the morphologically simpler languages than for the morphologically more complex languages, in line with the hypothesis that languages with greater inflectional complexity could be more difficult to segment into words. We further show that the effect of morphological complexity is relatively small, compared to that of algorithm and evaluation level. We therefore recommend that infant researchers look for signatures of the different segmentation algorithms and strategies, before looking for differences in infant segmentation landmarks across languages varying in complexity., (Copyright © 2021 Elsevier B.V. All rights reserved.) more...

Learners often need to identify and remember recurring units in continuous sequences, but the underlying mechanisms are debated. A particularly prominent candidate mechanism relies on distributional statistics such as Transitional Probabilities (TPs). However, it is unclear what the outputs of statistical segmentation mechanisms are, and if learners store these outputs as discrete chunks in memory. We critically review the evidence for the possibility that statistically coherent items are stored in memory and outline difficulties in interpreting past research. We use Slone and Johnson's (2018) experiments as a case study to show that it is difficult to delineate the different mechanisms learners might use to solve a learning problem. Slone and Johnson (2018) reported that 8-month-old infants learned coherent chunks of shapes in visual sequences. Here, we describe an alternate interpretation of their findings based on a multiple-cue integration perspective. First, when multiple cues to statistical structure were available, infants' looking behavior seemed to track with the strength of the strongest one - backward TPs, suggesting that infants process multiple cues simultaneously and select the strongest one. Second, like adults, infants are exquisitely sensitive to chunks, but may require multiple cues to extract them. In Slone and Johnson's (2018) experiments, these cues were provided by immediate chunk repetitions during familiarization. Accordingly, infants showed strongest evidence of chunking following familiarization sequences in which immediate repetitions were more frequent. These interpretations provide a strong argument for infants' processing of multiple cues and the potential importance of multiple cues for chunk recognition in infancy., (Copyright © 2020 Elsevier B.V. All rights reserved.) more...

Objective: This study aims at realizing unsupervised term discovery in Chinese electronic health records (EHRs) by using the word segmentation technique. The existing supervised algorithms do not perform satisfactorily in the case of EHRs, as annotated medical data are scarce. We propose an unsupervised segmentation method (GTS) based on the graph partition principle, whose multi-granular segmentation capability can help realize efficient term discovery., Methods: A sentence is converted to an undirected graph, with the edge weights based on n-gram statistics, and ratio cut is used to split the sentence into words. The graph partition is solved efficiently via dynamic programming, and multi-granularity is realized by setting different partition numbers. A BERT-based discriminator is trained using generated samples to verify the correctness of the word boundaries. The words that are not recorded in existing dictionaries are retained as potential medical terms., Results: We compared the GTS approach with mature segmentation systems for both word segmentation and term discovery. MD students manually segmented Chinese EHRs at fine and coarse granularity levels and reviewed the term discovery results. The proposed unsupervised method outperformed all the competing algorithms in the word segmentation task. In term discovery, GTS outperformed the best baseline by 17 percentage points (a 47% relative percentage of increment) on F1-score., Conclusion: In the absence of annotated training data, the graph partition technique can effectively use the corpus statistics and even expert knowledge to realize unsupervised word segmentation of EHRs. Multi-granular segmentation can be used to provide potential medical terms of various lengths with high accuracy., (Copyright © 2020 Elsevier Inc. All rights reserved.) more...

Infants' ability to segment words in fluent speech is affected by their language experience. In this study we investigated the conditions under which infants can segment words in a non-native language. Using the Head-turn Preference Procedure, we found that monolingual English-learning 8-month-olds can segment bisyllabic words in Spanish (trochees and iambs) but not French (iambs). Our results are incompatible with accounts that rely on distributional learning, language rhythm similarity, or target word prosodic shape alone. Instead, we show that monolingual English-learning infants are able to segment words in a non-native language as long as words have stress, as is the case in English. More specifically, we show that even in a rhythmically different non-native language, English-learning infants can find words by detecting stressed syllables and treating them as word onsets or offsets., (Copyright © 2018 Elsevier B.V. All rights reserved.) more...

A critical part of infants' ability to acquire any language involves segmenting continuous speech input into discrete word forms. Certain properties of words could provide infants with reliable cues to word boundaries. Here we investigate the potential utility of vowel harmony (VH), a phonological property whereby vowels within a word systematically exhibit similarity ("harmony") for some aspect of the way they are pronounced. We present evidence that infants with no experience of VH in their native language nevertheless actively use these patterns to generate hypotheses about where words begin and end in the speech stream. In two sets of experiments, we exposed infants learning English, a language without VH, to a continuous speech stream in which the only systematic patterns available to be used as cues to word boundaries came from syllable sequences that showed VH or those that showed vowel disharmony (dissimilarity). After hearing less than one minute of the streams, infants showed evidence of sensitivity to VH cues. These results suggest that infants have an experience-independent sensitivity to VH, and are predisposed to segment speech according to harmony patterns. We also found that when the VH patterns were more subtle (Experiment 2), infants required more exposure to the speech stream before they segmented based on VH, consistent with previous work on infants' preferences relating to processing load. Our findings evidence a previously unknown mechanism by which infants could discover the words of their language, and they shed light on the perceptual mechanisms that might be responsible for the emergence of vowel harmony as an organizing principle for the sound structure of words in many languages., (Copyright © 2017 Elsevier B.V. All rights reserved.) more...

Segmenting word units from running speech is a fundamental skill infants must develop in order to acquire language. Despite ample behavioral evidence of this skill, its neurocognitive basis remains unclear. Using behavioral testing and functional near-infrared spectroscopy, we aimed to uncover the neurocognitive substrates of word segmentation and its development. Of three age-groups of Japanese infants (5-6, 7-8, and 9-10months of age), the two older age-groups showed significantly larger temporo-parietal (particularly supramarginal gyrus) responses to target words repeatedly presented for training, than to control words. After the training, they also exhibited stronger inferior frontal responses to target words embedded in sentences. These findings suggest that word segmentation largely involves a cerebral circuit of phonological (phonetic) short-term memory. The dorsal pathway involved in encoding and decoding phonological representation may start to function stably at around 7months of age to facilitate the growth of the infant's vocabulary., (Copyright © 2017 Elsevier Inc. All rights reserved.) more...

While American English infants typically segment words from fluent speech by 7.5-months, studies of infants from other language backgrounds have difficulty replicating this finding. One possible explanation for this cross-linguistic difference is that the input infants from different language backgrounds receive is not as infant-directed as American English infant-directed speech (Floccia et al., 2016). Against this background, the current study investigates whether German 7.5- and 9-month-old infants segment words from fluent speech when the input is prosodically similar to American English IDS. While 9-month-olds showed successful segmentation of words from exaggerated IDS, 7.5-month-olds did not. These findings highlight (a) the beneficial impact of exaggerated IDS on infant speech segmentation, (b) cross-linguistic differences in word segmentation that are based not just on the kind of input available to children and suggest (c) developmental differences in the role of IDS as an attentional spotlight in speech segmentation., (Copyright © 2016 Elsevier B.V. All rights reserved.) more...

The division of labor hypothesis proposed by Nespor, Peña, and Mehler (2003) postulates that consonants are more important than vowels in lexical processing (when learning and recognizing words). This consonant bias (C-bias) is supported by many adult and toddler studies. However, some cross-linguistic variation has been found in toddlerhood, and various hypotheses have been proposed to account for the origin of the consonant bias, which make distinct predictions regarding its developmental trajectory during the first year of life. The present study evaluated these hypotheses by investigating the consonant bias in young French-learning infants, a language in which a consistent consonant bias is reported from 11months of age onward. Accordingly, in a series of word form segmentation experiments building on the fact that both 6- and 8-month-old French-learning infants can segment monosyllabic words, we investigated the relative impact of consonant and vowel mispronunciations on the recognition of segmented word forms at these two ages. Infants were familiarized with passages containing monosyllabic target words and then tested in different conditions all including consonant and/or vowel mispronunciations of the target words. Overall, our findings reveal a consonant bias at 8months, but an opposite vowel bias at 6months. These findings first establish that the consonant bias emerges between 6 and 8months of age in French-learning infants. Second, we discuss the factors that might explain such a developmental trajectory, highlighting the possible roles of pre-lexical and phonological acquisition., (Copyright © 2016 Elsevier B.V. All rights reserved.) more...

Speculations represent uncertainty toward certain facts. In clinical texts, identifying speculations is a critical step of natural language processing (NLP). While it is a nontrivial task in many languages, detecting speculations in Chinese clinical notes can be particularly challenging because word segmentation may be necessary as an upstream operation. The objective of this paper is to construct a state-of-the-art speculation detection system for Chinese clinical notes and to investigate whether embedding features and word segmentations are worth exploiting toward this overall task. We propose a sequence labeling based system for speculation detection, which relies on features from bag of characters, bag of words, character embedding, and word embedding. We experiment on a novel dataset of 36,828 clinical notes with 5103 gold-standard speculation annotations on 2000 notes, and compare the systems in which word embeddings are calculated based on word segmentations given by general and by domain specific segmenters respectively. Our systems are able to reach performance as high as 92.2% measured by F score. We demonstrate that word segmentation is critical to produce high quality word embedding to facilitate downstream information extraction applications, and suggest that a domain dependent word segmenter can be vital to such a clinical NLP task in Chinese language., (Copyright © 2016 Elsevier Inc. All rights reserved.) more...

The word segmentation paradigm originally designed by Jusczyk and Aslin (1995) has been widely used to examine how infants from the age of 7.5 months can extract novel words from continuous speech. Here we report a series of 13 studies conducted independently in two British laboratories, showing that British English-learning infants aged 8-10.5 months fail to show evidence of word segmentation when tested in this paradigm. In only one study did we find evidence of word segmentation at 10.5 months, when we used an exaggerated infant-directed speech style. We discuss the impact of variations in infant-directed style within and across languages in the course of language acquisition., (Crown Copyright © 2015. Published by Elsevier B.V. All rights reserved.) more...

Due to extensive variability in the phonetic realizations of words, there may be few or no proximal spectro-temporal cues that identify a word's onset or even its presence. Dilley and Pitt (2010) showed that the rate of context speech, distal from a to-be-recognized word, can have a sizeable effect on whether or not a word is perceived. This investigation considered whether there is a distinct role for distal rhythm in the disappearing word effect. Listeners heard sentences that had a grammatical interpretation with or without a critical function word (FW) and transcribed what they heard (e.g., are in Jill got quite mad when she heard there are birds can be removed and Jill got quite mad when she heard their birds is still grammatical). Consistent with a perceptual grouping hypothesis, participants were more likely to report critical FWs when distal rhythm (repeating ternary or binary pitch patterns) matched the rhythm in the FW-containing region than when it did not. Notably, effects of distal rhythm and distal rate were additive. Results demonstrate a novel effect of distal rhythm on the amount of lexical material listeners hear, highlighting the importance of distal timing information and providing new constraints for models of spoken word recognition., (Copyright © 2013 Elsevier B.V. All rights reserved.) more...

Functional magnetic resonance imaging (fMRI) was used to assess neural activation as participants learned to segment continuous streams of speech containing syllable sequences varying in their transitional probabilities. Speech streams were presented in four runs, each followed by a behavioral test to measure the extent of learning over time. Behavioral performance indicated that participants could discriminate statistically coherent sequences (words) from less coherent sequences (partwords). Individual rates of learning, defined as the difference in ratings for words and partwords, were used as predictors of neural activation to ask which brain areas showed activity associated with these measures. Results showed significant activity in the pars opercularis and pars triangularis regions of the left inferior frontal gyrus (LIFG). The relationship between these findings and prior work on the neural basis of statistical learning is discussed, and parallels to the frontal/subcortical network involved in other forms of implicit sequence learning are considered., (Copyright © 2012 Elsevier Inc. All rights reserved.) more...