Search

Some recent studies highlight that vehicular traffic and honking contribute to more than 50% of noise pollution in urban or sub-urban areas in developing countries, including Indian cities. Frequent honking has an adverse effect on health and hampers road safety, the environment, etc. Therefore, recognizing the various vehicle honks and classifying the honk of different vehicles can provide good insights into environmental noise pollution. Moreover, classifying honks based on vehicle types allows for the inference of contextual information of a location, area, or traffic. So far, the researchers have done outdoor sound classification and honk detection, where vehicular honks are collected in a controlled environment or in the absence of ambient noise. Such classification models fail to classify honk based on vehicle types. Therefore, it becomes imperative to design a system that can detect and classify honks of different types of vehicles to infer some contextual information. This paper presents a novel framework A C lassi H onk that performs raw vehicular honk sensing, data labeling, and classifies the honk into three major groups, i.e., light-weight vehicles, medium-weight vehicles, and heavy-weight vehicles. Raw audio samples of different vehicular honking are collected based on spatio-temporal characteristics and converted them into spectrogram images. A deep learning-based multi-label autoencoder model (MAE) is proposed for automated labeling of the unlabeled data samples, which provides 97.64% accuracy in contrast to existing deep learning-based data labeling methods. Further, various pre-trained models, namely Inception V3, ResNet50, MobileNet, and ShuffleNet are used and proposed an Ensembled Transfer Learning model (EnTL) for vehicle honks classification and performed comparative analysis. Results reveal that EnTL exhibits the best performance compared to pre-trained models and achieves 96.72% accuracy in our dataset. In addition, context of a location is identified based on these classified honk signatures in a city. [ABSTRACT FROM AUTHOR]

Data labeling systems are designed to facilitate the training and validation of machine learning algorithms under the umbrella of crowdsourcing practices. The current paper presents a novel approach for designing a customized data labeling system, emphasizing two key aspects: an innovative payment mechanism for users and an efficient configuration of output results. The main problem addressed is the labeling of datasets where golden items are utilized to verify user performance and assure the quality of the annotated outputs. Our proposed payment mechanism is enhanced through a modified skip-based golden-oriented function that balances user penalties and prevents spam activities. Additionally, we introduce a comprehensive reporting framework to measure aggregated results and accuracy levels, ensuring the reliability of the labeling output. Our findings indicate that the proposed solutions are pivotal in incentivizing user participation, thereby reinforcing the applicability and profitability of newly launched labeling systems. [ABSTRACT FROM AUTHOR]

Researchers often use machine learning and deep learning to predict price trends in the financial markets, aiming to achieve high returns. However, accurately predicting market prices is challenging due to their nonlinear and seemingly random nature. Improving the accuracy of the prediction model is the common focus of researchers, yet it is crucial to also consider the data used in training. Traditional labeling methods used in most price trend prediction studies are not robust as they are sensitive to small price changes, leading to inefficient model training. To address this issue, this study introduces a Dynamic Threshold Breakout (DTB) labeling system that labels data based on the price percentage change during a specific period. This proposed labeling system was then integrated into an automated trading system using LightGBM and evaluated using three different markets. The results showed that the DTB labeling method is effective for trading in financial markets in terms of winning ratio, payoff ratio, profit factor, accuracy and ROI in trading performance. [ABSTRACT FROM AUTHOR]

In this research, we introduce a novel open-source labeling tool, the Game-Based Assessment (GBA) Labeling Tool, specifically designed to address current challenges for data labeling in GBA scenarios. This web-based application facilitates the annotation of audio, video, and game event data, offering three different types of annotations – global, time instant, and time window annotations – to enhance accuracy in the labeling process. The tool also offers customizable labels and various types of visualizations to support different contexts and scenarios.

Purpose: To gain an understanding of data labeling requirements to train deep learning models for measurement of geographic atrophy (GA) with fundus autofluorescence (FAF) images. Design: Evaluation of artificial intelligence (AI) algorithms. Subjects: The Age-Related Eye Disease Study 2 (AREDS2) images were used for training and cross-validation, and GA clinical trial images were used for testing. Methods: Training data consisted of 2 sets of FAF images; 1 with area measurements only and no indication of GA location (Weakly labeled) and the second with GA segmentation masks (Strongly labeled). Main Outcome Measures: Bland–Altman plots and scatter plots were used to compare GA area measurement between ground truth and AI measurements. The Dice coefficient was used to compare accuracy of segmentation of the Strong model. Results: In the cross-validation AREDS2 data set (n = 601), the mean (standard deviation [SD]) area of GA measured by human grader, Weakly labeled AI model, and Strongly labeled AI model was 6.65 (6.3) mm2, 6.83 (6.29) mm2, and 6.58 (6.24) mm2, respectively. The mean difference between ground truth and AI was 0.18 mm2 (95% confidence interval, [CI], −7.57 to 7.92) for the Weakly labeled model and −0.07 mm2 (95% CI, −1.61 to 1.47) for the Strongly labeled model. With GlaxoSmithKline testing data (n = 156), the mean (SD) GA area was 9.79 (5.6) mm2, 8.82 (4.61) mm2, and 9.55 (5.66) mm2 for human grader, Strongly labeled AI model, and Weakly labeled AI model, respectively. The mean difference between ground truth and AI for the 2 models was −0.97 mm2 (95% CI, −4.36 to 2.41) and −0.24 mm2 (95% CI, −4.98 to 4.49), respectively. The Dice coefficient was 0.99 for intergrader agreement, 0.89 for the cross-validation data, and 0.92 for the testing data. Conclusions: Deep learning models can achieve reasonable accuracy even with Weakly labeled data. Training methods that integrate large volumes of Weakly labeled images with small number of Strongly labeled images offer a promising solution to overcome the burden of cost and time for data labeling. Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Research into the detection of human activities from wearable sensors is a highly active field, benefiting numerous applications, from ambulatory monitoring of healthcare patients via fitness coaching to streamlining manual work processes. We present an empirical study that evaluates and contrasts four commonly employed annotation methods in user studies focused on in-the-wild data collection. For both the user-driven, in situ annotations, where participants annotate their activities during the actual recording process, and the recall methods, where participants retrospectively annotate their data at the end of each day, the participants had the flexibility to select their own set of activity classes and corresponding labels. Our study illustrates that different labeling methodologies directly impact the annotations' quality, as well as the capabilities of a deep learning classifier trained with the data. We noticed that in situ methods produce less but more precise labels than recall methods. Furthermore, we combined an activity diary with a visualization tool that enables the participant to inspect and label their activity data. Due to the introduction of such a tool were able to decrease missing annotations and increase the annotation consistency, and therefore the F1-Score of the deep learning model by up to 8% (ranging between 82.1 and 90.4% F1-Score). Furthermore, we discuss the advantages and disadvantages of the methods compared in our study, the biases they could introduce, and the consequences of their usage on human activity recognition studies as well as possible solutions.

Recent advancements have significantly improved the field of segmentation-based change detection, particularly in the context of remote-sensing images. However, change detection datasets generally lack segmentation annotations, and the required labeling process is resource-intensive. We propose an improved change detection method based on segmentation to address this challenge. First, change detection annotations are converted to incomplete segmentation annotations through label matching. During segmentation, we utilize the focused information-guided segmentation method (FIGS) and a greenness index to provide prior information during training, guiding the model using accurately labeled regions. Finally, we generate a change map using pretrained features obtained from the segmentation stage. We demonstrate the robustness of our proposed label-matching process by comparing the results to a correctly matched dataset and show that incorporating FIGS and the greenness index improves the segmentation performance. Our method achieves effective change detection results even in scenarios associated with a shortage of annotations.

The integrity of data is crucial for the majority of existing data analysis methods. However, incomplete and unbalanced datasets from the collection and organization process will affect analysis accuracy. Existing interpolation algorithms often overlook feature importance, resulting in either cumbersome processes or underutilization of data information. This study introduces an interpolation method based on clustering algorithm focused on improving the accuracy and efficiency of missing data processing in datasets. First, in this paper, we clarify the problem to be solved about data interpolation, and we consider the importance of the information brought by the data itself to the interpolation, so we propose a scheme that combines clustering and interpolation. We propose a new method that uses the Lp norm as a similarity measure in the K-means clustering algorithm, and introduce a controllable weighting formula based on the current data segmentation. Methodologically, the clustering and interpolation are synchronized by iteratively updating the variable optimization cost function. The experimental results demonstrate significant improvements of the proposed interpolation algorithm over traditional techniques, particularly in tasks such as data labeling and classification within real datasets for clustering and classification.

Data labeling systems are designed to facilitate the training and validation of machine learning algorithms under the umbrella of crowdsourcing practices. The current paper presents a novel approach for designing a customized data labeling system, emphasizing two key aspects: an innovative payment mechanism for users and an efficient configuration of output results. The main problem addressed is the labeling of datasets where golden items are utilized to verify user performance and assure the quality of the annotated outputs. Our proposed payment mechanism is enhanced through a modified skip-based golden-oriented function that balances user penalties and prevents spam activities. Additionally, we introduce a comprehensive reporting framework to measure aggregated results and accuracy levels, ensuring the reliability of the labeling output. Our findings indicate that the proposed solutions are pivotal in incentivizing user participation, thereby reinforcing the applicability and profitability of newly launched labeling systems.

ABSTRACTBig Earth Data refers to the multidimensional integration and association of scientific data, including geography, resources, environment, ecology, and biology. An effective data classification system and label management strategy are important foundations for long-term management of data resources. The objective of this study was to construct a classification system and realize multidimensional semantic data label management for the Big Earth Data Science Engineering Program (CASEarth). This study constructed two sets of classification and coding systems that realize classification by mapping each other; namely, the geosphere-level and Sustainable Development Goals (SDGs) indicator classifications. This technique was based on natural language processing technology and solved problems with subject-word segmentation, weight calculation, and dynamic matching. A prototype system for classification and label management was constructed based on existing CASEarth datasets of more than 1,100. Furthermore, we expect our study to provide the methodology and technical support for user-oriented classification and label management services for Big Earth Data.

The fast expansion of the Internet of Things (IoT) networks raises the possibility of further network threats. In today's world, network traffic analysis has become an increasingly critical and useful tool for monitoring network traffic in general and analyzing attack patterns in particular. A few years ago, distributed denial-of-service attacks on IoT networks were considered the most pressing problem that needed to be addressed. The absence of high-quality datasets is one of the main obstacles to applying DDOS detection systems based on machine learning. Researchers have developed numerous methods to extract and analyze information from recorded files. From a literature review, it is clear that most of these tools share similar drawbacks. In this study, we proposed an intelligent raw network data extractor and labeler tool by incorporating the limitations of the tools that are available to transform PCAP to CSV. To generate and process a high-quality DDOS attack dataset suitable for machine learning models, we employed several data preprocessing operations on the selected network intrusion dataset. To confirm the validity and acceptability of the dataset, we tested different models. Among the models tested, the random forest was the most accurate in detecting the DDOS attack. [ABSTRACT FROM AUTHOR]

Human–robot collaboration has emerged as a prominent research topic in recent years. To enhance collaboration and ensure safety between humans and robots, researchers employ a variety of methods. One such method is physiological computing, which aims to estimate a human's psycho-physiological state by measuring various physiological signals such as galvanic skin response (GSR), electrocardiograph (ECG), heart rate variability (HRV), and electroencephalogram (EEG). This information is then used to provide feedback to the robot. In this paper, we present the latest state-of-the-art methods in physiological computing for human–robot collaboration. Our goal is to provide a comprehensive guide for new researchers to understand the commonly used physiological signals, data collection methods, and data labeling techniques. Additionally, we have categorized and tabulated relevant research to further aid in understanding this area of study. [ABSTRACT FROM AUTHOR]

Manual annotation for human action recognition with content semantics using 3D Point Cloud (3D-PC) in industrial environments consumes a lot of time and resources. This work aims to recognize, analyze, and model human actions to develop a framework for automatically extracting content semantics. Main Contributions of this work: 1. design a multi-layer structure of various DNN classifiers to detect and extract humans and dynamic objects using 3D-PC preciously, 2. empirical experiments with over 10 subjects for collecting datasets of human actions and activities in one industrial setting, 3. development of an intuitive GUI to verify human actions and its interaction activities with the environment, 4. design and implement a methodology for automatic sequence matching of human actions in 3D-PC. All these procedures are merged in the proposed framework and evaluated in one industrial Use-Case with flexible patch sizes. Comparing the new approach with standard methods has shown that the annotation process can be accelerated by 5.2 times through automation.

A major bottleneck preventing the extension of deep learning systems to new domains is the prohibitive cost of acquiring sufficient training labels. Alternatives such as weak supervision, active learning, and fine-tuning of pretrained models reduce this burden but require substantial human input to select a highly informative subset of instances or to curate labeling functions. REGAL (Rule-Enhanced Generative Active Learning) is an improved framework for weakly supervised text classification that performs active learning over labeling functions rather than individual instances. REGAL interactively creates high-quality labeling patterns from raw text, enabling a single annotator to accurately label an entire dataset after initialization with three keywords for each class. Experiments demonstrate that REGAL extracts up to 3 times as many high-accuracy labeling functions from text as current state-of-the-art methods for interactive weak supervision, enabling REGAL to dramatically reduce the annotation burden of writing labeling functions for weak supervision. Statistical analysis reveals REGAL performs equal or significantly better than interactive weak supervision for five of six commonly used natural language processing (NLP) baseline datasets.

Investing in the stock market involves high risks and equivalent returns. Many investors tend to be loss averse and suffer significant losses due to information asymmetry and emotion-based investing. In order to minimize such losses, various studies are attempting to predict the stock market's future and automate investment decisions. However, the stock market is affected by many factors and poses a challenge in prediction. Also, many researchers who develop stock trading systems are interested primarily in improving prediction model performance. Particularly, most studies set up-down labels in the data labeling phase. However, up-down labeling views all stock price fluctuations as identical, resulting in inefficient model training by including substantial noise in the learning data. Therefore, this study proposes SGA (Selective Genetic Algorithm) labeling to overcome such concerns. A trading system was developed using the proposed method, and a machine learning model and ensemble method supported the label prediction phase. The proposed trading system was analyzed empirically with 38 NASDAQ stocks and 33 KOSPI stocks, and a comparative experiment with up-down labeling method was conducted. In this study, SGA labeling performance exceeded that of up-down labeling. Additionally, instance selection and ensemble methods affected the trading system's performance. [ABSTRACT FROM AUTHOR]

Some recent studies highlight that vehicular traffic and honking contribute to more than 50% of noise pollution in urban or sub-urban areas in developing countries, including Indian cities. Frequent honking has an adverse effect on health and hampers road safety, the environment, etc. Therefore, recognizing the various vehicle honks and classifying the honk of different vehicles can provide good insights into environmental noise pollution. Moreover, classifying honks based on vehicle types allows for the inference of contextual information of a location, area, or traffic. So far, the researchers have done outdoor sound classification and honk detection, where vehicular honks are collected in a controlled environment or in the absence of ambient noise. Such classification models fail to classify honk based on vehicle types. Therefore, it becomes imperative to design a system that can detect and classify honks of different types of vehicles to infer some contextual information. This paper presents a novel framework A C lassi H onk that performs raw vehicular honk sensing, data labeling, and classifies the honk into three major groups, i.e., light-weight vehicles, medium-weight vehicles, and heavy-weight vehicles. Raw audio samples of different vehicular honking are collected based on spatio-temporal characteristics and converted them into spectrogram images. A deep learning-based multi-label autoencoder model (MAE) is proposed for automated labeling of the unlabeled data samples, which provides 97.64% accuracy in contrast to existing deep learning-based data labeling methods. Further, various pre-trained models, namely Inception V3, ResNet50, MobileNet, and ShuffleNet are used and proposed an Ensembled Transfer Learning model (EnTL) for vehicle honks classification and performed comparative analysis. Results reveal that EnTL exhibits the best performance compared to pre-trained models and achieves 96.72% accuracy in our dataset. In addition, context of a location is identified based on these classified honk signatures in a city., (© 2024. The Author(s), under exclusive licence to Springer Nature Switzerland AG.)

Human–robot collaboration has emerged as a prominent research topic in recent years. To enhance collaboration and ensure safety between humans and robots, researchers employ a variety of methods. One such method is physiological computing, which aims to estimate a human’s psycho-physiological state by measuring various physiological signals such as galvanic skin response (GSR), electrocardiograph (ECG), heart rate variability (HRV), and electroencephalogram (EEG). This information is then used to provide feedback to the robot. In this paper, we present the latest state-of-the-art methods in physiological computing for human–robot collaboration. Our goal is to provide a comprehensive guide for new researchers to understand the commonly used physiological signals, data collection methods, and data labeling techniques. Additionally, we have categorized and tabulated relevant research to further aid in understanding this area of study.

The main ionospheric trough (MIT) is a salient density feature in the mid‐latitude ionosphere and characterizing its structure is important for understanding Global Positioning System and HF signal propagation, and identifying geospace phenomena such as the plasmapause boundary layer. While a number of previous studies have statistically investigated the properties of the MIT utilizing low‐altitude satellite observations, they have been limited to latitudinal cross sections, and have not considered the inherent two‐dimensional structure of the MIT. In this work, we develop a regularized inversion method for identifying the two dimensional structure of the MIT in Total Electron Content maps. Because no ground truth labels exist for the MIT, we extensively characterize the behavior of the algorithm by comparing it to the method developed by Aa, Zou, et al. (2020, doi:https://doi.org/10.1029/2019JA027583). We show that statistics computed on the resulting labels are robust to our choice of algorithm parameters and that we are able to match the results of Aa, Zou, et al. (2020, doi:https://doi.org/10.1029/2019JA027583) with a particular selection of the parameters. In addition to enabling fundamentally different studies, our MIT labels are able to provide statistical MIT properties with higher resolution. Code to reproduce our data set is provided in a GitHub repository: https://github.com/gregstarr/trough. Plain Language Summary: The Earth's ionized outer atmosphere (the ionosphere) serves as the interface between our atmosphere and the near space environment. Signals from Global Positioning System satellites contain information about the total integrated plasma density (referred to as "Total Electron Content," (TEC)) between a given satellite and a given ground receiver. Measurements of TEC from large networks of ground receivers allow us to construct global images of the ionosphere, and thus study its variations in space and time. This paper uses such ionospheric images over a 20 year period to investigate a large‐scale coherent density depletion in the ionosphere known as the "mid‐latitude ionospheric trough" (MIT). In this work we have identified and labeled MIT regions in each images using computer vision techniques, and stored the results in a new image database. Using this "labeled data," we can gain insight into the average location, morphology, and occurrence rates of the MIT, an examine this phenomenon varies with season, time of day, geomagnetic activity, and other correlated parameters. Key Points: A method for identifying the main ionospheric trough (MIT) in global Total Electron Content images is developed and validatedStatistics extracted from a 10‐year data set are found to be in good agreement with a SWARM‐derived satellite modelOur technique enables statistical analyzes of two‐dimensional (lat, lon) MIT properties with high resolution [ABSTRACT FROM AUTHOR]

The advent of machine learning (ML) has significantly advanced the recognition and translation of sign languages, bridging communication gaps for hearing-impaired communities. At the heart of these technologies is data labeling, crucial for training ML algorithms on a huge amount of consistently labeled data to achieve models that generalize well. The adoption of language-agnostic annotations is essential to connect different sign languages, as single-language databases often provide limited lexicon examples, insufficient for training robust ML algorithms. This study critically examines the Hamburg Sign Language Notation System (HamNoSys), which describes the signer's initial position and body movements, in contrary to the meanings of glosses. Despite HamNoSys's utility in standardizing transcriptions across various sign languages, our investigation uncovers inconsistencies within HamNoSys that may negatively impact the development of accurate and reliable ML models. By analyzing HamNoSys labels across five sign languages, we identified a lack of standardized annotation procedures and the complexities within HamNoSys that introduce biases and errors. Our findings underscore the urgent need for unified, standardized data annotation guidelines to enhance the accuracy and efficiency of sign language recognition technologies. This research highlights the importance of addressing annotation challenges and advocates for a comprehensive, diversified database to improve the generalization of ML models. • HamNoSys notation system is not sufficient for ML labeling purposes • HamNoSys-based labels are often subjective and inconsistent • There is a need for a notation system useful in computer-aided systems [ABSTRACT FROM AUTHOR]

A major bottleneck preventing the extension of deep learning systems to new domains is the prohibitive cost of acquiring sufficient training labels. Alternatives such as weak supervision, active learning, and fine-tuning of pretrained models reduce this burden but require substantial human input to select a highly informative subset of instances or to curate labeling functions. REGAL (Rule-Enhanced Generative Active Learning) is an improved framework for weakly supervised text classification that performs active learning over labeling functions rather than individual instances. REGAL interactively creates high-quality labeling patterns from raw text, enabling a single annotator to accurately label an entire dataset after initialization with three keywords for each class. Experiments demonstrate that REGAL extracts up to 3 times as many high-accuracy labeling functions from text as current state-of-the-art methods for interactive weak supervision, enabling REGAL to dramatically reduce the annotation burden of writing labeling functions for weak supervision. Statistical analysis reveals REGAL performs equal or significantly better than interactive weak supervision for five of six commonly used natural language processing (NLP) baseline datasets. [ABSTRACT FROM AUTHOR]

Segmentation, or the classification of pixels (grid cells) in imagery, is ubiquitously applied in the natural sciences. Manual methods are often prohibitively time‐consuming, especially those images consisting of small objects and/or significant spatial heterogeneity of colors or textures. Labeling complicated regions of transition that in Earth surface imagery are represented by collections of mixed‐pixels, ‐textures, and ‐spectral signatures, can be especially error‐prone because it is difficult to reliably unmix, identify and delineate consistently. However, the success of supervised machine learning (ML) approaches is entirely dependent on good label data. We describe a fast, semi‐automated, method for interactive segmentation of N‐dimensional (x, y, N) images into two‐dimensional (x, y) label images. It uses human‐in‐the‐loop ML to achieve consensus between the labeler and a model in an iterative workflow. The technique is reproducible; the sequence of decisions made by human labeler and ML algorithms can be encoded to file, so the entire process can be played back and new outputs generated with alternative decisions and/or algorithms. We illustrate the scientific potential of segmentation of imagery of diverse settings and image types using six case studies from river, estuarine, and open coast environments. These photographic and non‐photographic imagery consist of 1‐ and 3‐bands on regular and irregular grids ranging from centimeters to tens of meters. We demonstrate high levels of agreement in label images generated by several labelers on the same imagery, and make suggestions to achieve consensus and measure uncertainty, ideal for widespread application in training supervised ML for image segmentation. Plain Language Summary: Labeling pixels in scientific images by hand is time‐consuming and error‐prone, so we would like to train computers to do that for us. We can use automated techniques from Artificial Intelligence or AI, like one called Deep Learning, but it needs a lot of example images and corresponding labels that have been made by hand. So, we still need to label quite a lot of images at the pixel level—called image segmentation. We made a computer program called Doodler that speeds up the process; you label some pixels, and it labels the rest. It is the fastest method we know of for image segmentation because it is semi‐automated. We also show that it produces accurate and precise labeling, as we demonstrated by having multiple people use this method to label the same images. Because it is so fast and accurate, it allows us to get enough data to train Deep Learning models to do segmentation on all the images we have, from the past and in the future. Doodler therefore enables geoscientists to use Artificial Intelligence to extract much more information from their imagery, in service of geoscience in general. Key Points: Methodology for image segmentation based on agreement between a labeler and a machine learning modelFaster and more accurate segmentation of interpretable imagery compared to traditional labelingLarge multi‐labeler consensus facilitates reproducible scientific inference from Earth surface images [ABSTRACT FROM AUTHOR]

An automatic classification of fine art images is limited by the scarcity of high-quality labels made by art experts. This study aims to provide meaningful automatic labeling of fine art paintings (machine labeling) without the need for human annotation. A new unsupervised Adversarial Clustering System (ACS) is proposed. The ACS is an adversarial learning approach comprising an unsupervised clustering module generating machine labels and a supervised classification module classifying the data based on the machine labels. Both modules are linked through an optimization algorithm iteratively improving the unsupervised clusters. The objective function driving the improvement consists of the within-cluster sum of squares (WCSS) error and the supervised classification accuracy. The proposed method was tested on three different fine-art datasets, including two sets of paintings previously categorized by art experts and one never categorized collection of Australian Aboriginal paintings. The unsupervised clusters were analyzed using standard unsupervised clustering metrics and a reliability measure between machine and human labeling. The ACS showed higher reliability compared to the classical k-means clustering method. The content analysis of unsupervised clusters indicated grouping based on scene composition, type, and shape of the object, edge sharpness and direction, and color palette.

Abstract Segmentation, or the classification of pixels (grid cells) in imagery, is ubiquitously applied in the natural sciences. Manual methods are often prohibitively time‐consuming, especially those images consisting of small objects and/or significant spatial heterogeneity of colors or textures. Labeling complicated regions of transition that in Earth surface imagery are represented by collections of mixed‐pixels, ‐textures, and ‐spectral signatures, can be especially error‐prone because it is difficult to reliably unmix, identify and delineate consistently. However, the success of supervised machine learning (ML) approaches is entirely dependent on good label data. We describe a fast, semi‐automated, method for interactive segmentation of N‐dimensional (x, y, N) images into two‐dimensional (x, y) label images. It uses human‐in‐the‐loop ML to achieve consensus between the labeler and a model in an iterative workflow. The technique is reproducible; the sequence of decisions made by human labeler and ML algorithms can be encoded to file, so the entire process can be played back and new outputs generated with alternative decisions and/or algorithms. We illustrate the scientific potential of segmentation of imagery of diverse settings and image types using six case studies from river, estuarine, and open coast environments. These photographic and non‐photographic imagery consist of 1‐ and 3‐bands on regular and irregular grids ranging from centimeters to tens of meters. We demonstrate high levels of agreement in label images generated by several labelers on the same imagery, and make suggestions to achieve consensus and measure uncertainty, ideal for widespread application in training supervised ML for image segmentation.

Various imaging parameters in X-ray computed tomography (CT) should be examined and optimized by task-based assessment of human observer performance. Recently, convolutional neural networks (CNNs) have been introduced as anthropomorphic model observers. However, when human-labeled data are not available or limited, CNNs with an existing training strategy do not produce good performance agreement with human observers. The purpose of this study is to propose new training strategies for a CNN-based anthropomorphic model observer without human-labeled data for signal-known-exactly and background-known-statistically detection tasks. We acquired cone-beam CT projection data of breast background volume and reconstructed the projection data using the Feldkamp-Davis-Kress algorithm with 12 different imaging conditions including viewing image planes. Training data for the CNN were labeled utilizing conventional model observers. We employed an early stopping rule to reflect internal noise during the CNN training. To examine the CNN performance, we used three different training-testing schemes. The performance agreement between the human and model observers was measured via a Bland-Altman plot, the root-mean-squared error (RMSE), and the Pearson's correlation coefficient ($r$ ) of their proportion correct values. Throughout the three different training-testing schemes, CNNs with the proposed training strategies yielded narrower limits of agreements (with a bias lower than 0.03) and higher scores in both RMSE and $r$ than the conventional model observers. This indicates the proposed training strategies enable the CNN-based anthropomorphic model observer to have good performance agreement with human observers and generalize better to different imaging conditions than conventional anthropomorphic model observers.

Classifying images using supervised machine learning (ML) relies on labeled training data—classes or text descriptions, for example, associated with each image. Data‐driven models are only as good as the data used for training, and this points to the importance of high‐quality labeled data for developing a ML model that has predictive skill. Labeling data is typically a time‐consuming, manual process. Here, we investigate the process of labeling data, with a specific focus on coastal aerial imagery captured in the wake of hurricanes that affected the Atlantic and Gulf Coasts of the United States. The imagery data set is a rich observational record of storm impacts and coastal change, but the imagery requires labeling to render that information accessible. We created an online interface that served labelers a stream of images and a fixed set of questions. A total of 1,600 images were labeled by at least two or as many as seven coastal scientists. We used the resulting data set to investigate interrater agreement: the extent to which labelers labeled each image similarly. Interrater agreement scores, assessed with percent agreement and Krippendorff's alpha, are higher when the questions posed to labelers are relatively simple, when the labelers are provided with a user manual, and when images are smaller. Experiments in interrater agreement point toward the benefit of multiple labelers for understanding the uncertainty in labeling data for machine learning research. Plain Language Summary: After hurricanes and storms, pictures taken from a plane can be used to observe how the coast was impacted. A single flight might take thousands of pictures. If a computer could automatically analyze the pictures, then a person would not need to look at them one‐by‐one. To teach a computer to analyze images, we need many pictures and many labels that describe what is visible in each picture. But where do we get those labels? Typically, a coastal scientist labels the pictures by sorting them into folders or typing codes into a spreadsheet. But does every coastal scientist label pictures the same way? Some labeling questions are easy to answer, and scientists mostly agree ("Is this image all water?"). Other labeling questions are harder to answer, and cause disagreement ("Was there damage to buildings?"). This paper is about how well scientists agree when labeling the same pictures, and how we can improve agreement among scientists. We try some experiments and offer a few ideas on how to improve agreement. We suggest writing very clear questions, using smaller images, and having a comprehensive manual. It turns out that having a manual with examples—and reading the manual!—really helps. Key Points: We measure agreement among coastal scientists labeling the same sets of poststorm imagesCoastal scientists agree more when rating landforms, less when labeling inferred processesIterating on questions, providing documentation, and using smaller image sizes all increase agreement [ABSTRACT FROM AUTHOR]

Abstract Classifying images using supervised machine learning (ML) relies on labeled training data—classes or text descriptions, for example, associated with each image. Data‐driven models are only as good as the data used for training, and this points to the importance of high‐quality labeled data for developing a ML model that has predictive skill. Labeling data is typically a time‐consuming, manual process. Here, we investigate the process of labeling data, with a specific focus on coastal aerial imagery captured in the wake of hurricanes that affected the Atlantic and Gulf Coasts of the United States. The imagery data set is a rich observational record of storm impacts and coastal change, but the imagery requires labeling to render that information accessible. We created an online interface that served labelers a stream of images and a fixed set of questions. A total of 1,600 images were labeled by at least two or as many as seven coastal scientists. We used the resulting data set to investigate interrater agreement: the extent to which labelers labeled each image similarly. Interrater agreement scores, assessed with percent agreement and Krippendorff's alpha, are higher when the questions posed to labelers are relatively simple, when the labelers are provided with a user manual, and when images are smaller. Experiments in interrater agreement point toward the benefit of multiple labelers for understanding the uncertainty in labeling data for machine learning research.

In the process of Unmanned Aerial Vehicle (UAV) flight testing, plenty of compound faults exist, which could be composed of concurrent single faults or over-limit states alarmed by Built-In-Test (BIT) equipment. At present, there still lacks a suitable automatic labeling approach for UAV flight data, effectively utilizing the information of the BIT record. The performance of the originally employed flight data-driven fault diagnosis models based on machine learning needs to be improved as well. A compound fault labeling and diagnosis method based on actual flight data and the BIT record of the UAV during flight test phase is proposed, through labeling the flight data with compound fault modes corresponding to concurrent single faults recorded by the BIT system, and upgrading the original diagnosis model based on Gradient Boosting Decision Tree (GBDT) and Fully Convolutional Network (FCNN), to eXtreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM) and modified Convolutional Neural Network (CNN). The experimental results based on actual test flight data show that the proposed method could effectively label the flight data and obtain a significant improvement in diagnostic performance, appearing to be practical in the UAV test flight process. [ABSTRACT FROM AUTHOR]

Data collection is a major bottleneck in machine learning and an active research topic in multiple communities. There are largely two reasons data collection has recently become a critical issue. First, as machine learning is becoming more widely-used, we are seeing new applications that do not necessarily have enough labeled data. Second, unlike traditional machine learning, deep learning techniques automatically generate features, which saves feature engineering costs, but in return may require larger amounts of labeled data. Interestingly, recent research in data collection comes not only from the machine learning, natural language, and computer vision communities, but also from the data management community due to the importance of handling large amounts of data. In this survey, we perform a comprehensive study of data collection from a data management point of view. Data collection largely consists of data acquisition, data labeling, and improvement of existing data or models. We provide a research landscape of these operations, provide guidelines on which technique to use when, and identify interesting research challenges. The integration of machine learning and data management for data collection is part of a larger trend of Big data and Artificial Intelligence (AI) integration and opens many opportunities for new research. [ABSTRACT FROM AUTHOR]

The applicability of deep learning to remote sensing is rapidly increasing in accordance with the improvement in spatiotemporal resolution of satellite images. However, unlike satellite images acquired in near-real-time over wide areas, there are limited amount of labeled data used for model training. In this article, three kinds of deep learning applications—data augmentation, semisupervised classification, and domain-adapted architecture—were tested in an effort to overcome the limitation of insufficient labeled data. Among the diverse tasks that can be used for classification, rice paddy detection in South Korea was performed for its ability to fully utilize the advantages of deep learning and high spatiotemporal image resolution. In the process of designing each application, the domain knowledge of remote sensing and rice phenology was integrated. Then, all possible combinations of the three applications were examined and evaluated with pixel-based comparisons in various environments and city-level comparisons using national statistics. The results of this article indicated that all combinations of the applications can contribute to increase classification performance, even though the uncertainty involved in imitating or utilizing unlabeled data remains. As the effectiveness of the proposed applications was experimentally confirmed, enhancement in the applicability of deep learning was expected in various remote sensing areas. In particular, the proposed applications would be significant when they are applied to a wide range of study areas and high-resolution images, as they tend to require a large amount of learning data from diverse environments, owing to high intra-class heterogeneity. [ABSTRACT FROM AUTHOR]

Data labeling, which assigns data with multiple classes, is indispensable for many applications, such as machine learning and data integration. However, existing labeling solutions either incur expensive cost for large datasets or produce noisy results. This paper introduces a cost-effective labeling approach and focuses on the labeling rule generation problem that aims to generate high-quality rules to largely reduce the labeling cost while preserving quality. To address the problem, we first generate candidate rules and then devise a game-based crowdsourcing approach CrowdGame to select high-quality rules by considering coverage and accuracy. CrowdGame employs two groups of crowd workers: One group answers rule validation tasks (whether a rule is valid) to play a role of rule generator, while the other group answers tuple checking tasks (whether the label of a data tuple is correct) to play a role of rule refuter. We let the two groups play a two-player game: Rule generator identifies high-quality rules with large coverage, while rule refuter tries to refute its opponent rule generator by checking some tuples that provide enough evidence to reject rules with low accuracy. This paper studies the challenges in CrowdGame. The first is to balance the trade-off between coverage and accuracy. We define the loss of a rule by considering the two factors. The second is rule accuracy estimation. We utilize Bayesian estimation to combine both rule validation and tuple checking tasks. The third is to select crowdsourcing tasks to fulfill the game-based framework for minimizing the loss. We introduce a minimax strategy and develop efficient task selection algorithms. We also develop a hybrid crowd-machine method for effective label assignment under budget-constrained crowdsourcing settings. We conduct experiments on entity matching and relation extraction, and the results show that our method outperforms state-of-the-art solutions. [ABSTRACT FROM AUTHOR]

Online learning engagement detection is a fundamental problem in educational information technology. Efficient detection of students' learning situations can provide information to teachers to help them identify students having trouble in real time. To improve the accuracy of learning engagement detection, we have collected two aspects of students' behavior data: face data (using adaptive weighted Local Gray Code Patterns for facial expression recognition) and mouse interaction. In this article, we propose a novel learning engagement detection algorithm based on the collected data (students' behavior), which come from the cameras and the mouse in the online learning environment. The cameras were utilized to capture students' face images, while the mouse movement data were captured simultaneously. In the process of image data labeling, we built two datasets for classifier training and testing. One took the mouse movement data as a reference, while the other did not. We performed experiments on two datasets using several methods and found that the classifier trained by the former dataset had a better performance, and its recognition rate is higher than that of the latter one (94.60% vs. 91.51%). [ABSTRACT FROM AUTHOR]

Online auctions have become one of the most convenient ways to commit fraud due to a large amount of money being traded every day. Shill bidding is the predominant form of auction fraud, and it is also the most difficult to detect because it so closely resembles normal bidding behavior. Furthermore, shill bidding does not leave behind any apparent evidence, and it is relatively easy to use to cheat innocent buyers. Our goal is to develop a classification model that is capable of efficiently differentiating between legitimate bidders and shill bidders. For our study, we employ an actual training dataset, but the data are unlabeled. First, we properly label the shill bidding samples by combining a robust hierarchical clustering technique and a semi-automated labeling approach. Since shill bidding datasets are imbalanced, we assess advanced over-sampling, under-sampling and hybrid-sampling methods and compare their performances based on several classification algorithms. The optimal shill bidding classifier displays high detection and low misclassification rates of fraudulent activities. [ABSTRACT FROM AUTHOR]

This paper addresses the challenges of model evaluation and optimization that arise from domain differences between the target and source domains during model deployment. Current methods for model accuracy evaluation require a fully annotated test set. However, obtaining additional human labels for every unique application scenario can be costly and time-intensive. To tackle this problem, this paper proposes an instance segmentation model evaluation method based on domain differences, which can give the prediction accuracy of the model on unlabeled test sets. Moreover, to enhance deployment accuracy cost-effectively, this paper proposes an "effective operation"-based labeling cost computation method and a weighted uncertainty sample selection method. The former accurately computes labeling costs for instance segmentation, while the latter selects the most valuable samples from the target domain for labeling and training. Model evaluation experiments demonstrate that this method's root mean square error (RMSE) on Cityscapes is approximately 4% less than that of other existing model evaluation methods. Model optimization experiments demonstrate that the proposed method achieves greater model accuracy than comparative methods under four distinct data partitioning protocols. The code is available at https://github.com/licongguan/Lamer. [ABSTRACT FROM AUTHOR]

Remote cameras ("trail cameras") are a popular tool for non-invasive, continuous wildlife monitoring, and as they become more prevalent in wildlife research, machine learning (ML) is increasingly used to automate or accelerate the labor-intensive process of labelling (i.e., tagging) photos. Human-machine hybrid tagging approaches have been shown to greatly increase tagging efficiency (i.e., time to tag a single image). However, those potential increases hinge on the extent to which an ML model makes correct vs. incorrect predictions. We performed an experiment using a ML model that produces bounding boxes around animals, people, and vehicles in remote camera imagery (MegaDetector) to consider the impact of a ML model's performance on its ability to accelerate human labeling. Six participants tagged trail camera images collected from 12 sites in Vermont and Maine, USA (January–September 2022) using three tagging methods (one with ML bounding box assistance and two without assistance). We used a generalized linear mixed model to examine the influence of ML model performance and tagging method on tagging efficiency. We found that ML bounding boxes offer significant improvement in tagging efficiency when labelling data compared to unassisted tagging. Additionally, the time taken to label with bounding boxes was not statistically different from an unassisted tagging approach. However, we found that gains in efficiency are contingent on the ML algorithm's performance and that incorrect ML predictions, particularly the 4.2% false positive and 3.6% false negative predictions, can slow the tagging process compared to a non-hybrid approach. These findings indicate that although practitioners usually forgo the production of bounding boxes when selecting a data labelling process due to the increased effort, ML bounding box-assisted tagging can offer an efficient method for labeling. More broadly, ML-assisted data labelling offers an opportunity to accelerate the analysis of trail camera imagery, but an assessment of the ML model's performance can illuminate whether the hybrid-tagging approach is ultimately a help or hinderance. • Human-Machine Learning (ML) tandem approaches can accelerate labelling of remote camera images. • ML-assisted tagging with bounding boxes was significantly faster than an unassisted approach. • Correct ML predictions sped up image tagging while incorrect predictions slowed it. • Examination of ML-performance is critical to assessing the efficacy of ML-assisted tagging. [ABSTRACT FROM AUTHOR]

In the process of Unmanned Aerial Vehicle (UAV) flight testing, plenty of compound faults exist, which could be composed of concurrent single faults or over-limit states alarmed by Built-In-Test (BIT) equipment. At present, there still lacks a suitable automatic labeling approach for UAV flight data, effectively utilizing the information of the BIT record. The performance of the originally employed flight data-driven fault diagnosis models based on machine learning needs to be improved as well. A compound fault labeling and diagnosis method based on actual flight data and the BIT record of the UAV during flight test phase is proposed, through labeling the flight data with compound fault modes corresponding to concurrent single faults recorded by the BIT system, and upgrading the original diagnosis model based on Gradient Boosting Decision Tree (GBDT) and Fully Convolutional Network (FCNN), to eXtreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM) and modified Convolutional Neural Network (CNN). The experimental results based on actual test flight data show that the proposed method could effectively label the flight data and obtain a significant improvement in diagnostic performance, appearing to be practical in the UAV test flight process.

Purpose: To gain an understanding of data labeling requirements to train deep learning models for measurement of geographic atrophy (GA) with fundus autofluorescence (FAF) images., Design: Evaluation of artificial intelligence (AI) algorithms., Subjects: The Age-Related Eye Disease Study 2 (AREDS2) images were used for training and cross-validation, and GA clinical trial images were used for testing., Methods: Training data consisted of 2 sets of FAF images; 1 with area measurements only and no indication of GA location (Weakly labeled) and the second with GA segmentation masks (Strongly labeled)., Main Outcome Measures: Bland-Altman plots and scatter plots were used to compare GA area measurement between ground truth and AI measurements. The Dice coefficient was used to compare accuracy of segmentation of the Strong model., Results: In the cross-validation AREDS2 data set (n = 601), the mean (standard deviation [SD]) area of GA measured by human grader, Weakly labeled AI model, and Strongly labeled AI model was 6.65 (6.3) mm 2 , 6.83 (6.29) mm 2 , and 6.58 (6.24) mm 2 , respectively. The mean difference between ground truth and AI was 0.18 mm 2 (95% confidence interval, [CI], -7.57 to 7.92) for the Weakly labeled model and -0.07 mm 2 (95% CI, -1.61 to 1.47) for the Strongly labeled model. With GlaxoSmithKline testing data (n = 156), the mean (SD) GA area was 9.79 (5.6) mm 2 , 8.82 (4.61) mm 2 , and 9.55 (5.66) mm 2 for human grader, Strongly labeled AI model, and Weakly labeled AI model, respectively. The mean difference between ground truth and AI for the 2 models was -0.97 mm 2 (95% CI, -4.36 to 2.41) and -0.24 mm 2 (95% CI, -4.98 to 4.49), respectively. The Dice coefficient was 0.99 for intergrader agreement, 0.89 for the cross-validation data, and 0.92 for the testing data., Conclusions: Deep learning models can achieve reasonable accuracy even with Weakly labeled data. Training methods that integrate large volumes of Weakly labeled images with small number of Strongly labeled images offer a promising solution to overcome the burden of cost and time for data labeling., Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article., (© 2024 by the American Academy of Ophthalmology.)

SMART is an open source web application designed to help data scientists and research teams efficiently build labeled training data sets for supervised machine learning tasks. SMART provides users with an intuitive interface for creating labeled data sets, supports active learning to help reduce the required amount of labeled data, and incorporates interrater reliability statistics to provide insight into label quality. SMART is designed to be platform agnostic and easily deployable to meet the needs of as many different research teams as possible. The project website1 contains links to the code repository and extensive user documentation. [ABSTRACT FROM AUTHOR]

Abstract The paper proposes an efficient method for training a neural network to count moving objects in a video, while another neural network concurrently prepares a labeled dataset for the first one. The detection, tracking, and counting of objects is crucial for effective Intelligence Transportation Systems (ITS), which should reduce congestion and recognize traffic offenders on highways and in urban areas. Creation of labeled data for training a neural network is one of the essential prerequisites for successful application of supervised machine learning. In this paper, the experimental results of the automatic labeling and counting of vehicles under real world conditions are shown. The method shows that by using the Convolutional Neural Network (CNN), the computing power and speed-up time for training a Recurrent Neural Network (RNN) with a Long Short-Term Memory (LSTM) cell for counting moving objects can be decreased. [ABSTRACT FROM AUTHOR]

Abstract Many predictive tasks require labeled data to induce classification models. The data labeling process may have a high cost. Several strategies have been proposed to optimize the selection of the most relevant examples, a process referred to as active learning. However, a lack of empirical studies comparing different active learning approaches across multiple datasets makes it difficult identifying the most promising strategies, or even assessing the relative gain of active learning over the trivial random selection of instances. In this study, a comprehensive comparison of active learning strategies is presented, with various instance selection criteria, different classification algorithms and a large number of datasets. The experimental results confirm the effectiveness of active learning and provide insights about the relationship between classification algorithms and active learning strategies. Additionally, ranking curves with bands are introduced as a means to summarize in a single chart the performance of each active learning strategy for different classification algorithms and datasets. [ABSTRACT FROM AUTHOR]

Abstract Large-scale data analysis is a challenging and relevant task for present-day research and industry. As a promising data analysis tool, clustering is becoming more important in the era of big data. In large-scale data clustering, sampling is an efficient and most widely used approximation technique. Recently, several sampling-based clustering algorithms have attracted considerable attention in large-scale data analysis owing to their efficiency. However, some of these existing algorithms have low clustering accuracy, whereas others have high computational complexity. To overcome these deficiencies, a stratified sampling based clustering algorithm for large-scale data is proposed in this paper. Its basic steps include: (1) obtaining a number of representative samples from different strata with a stratified sampling scheme, which are formed by locality sensitive hashing technique, (2) partitioning the chosen samples into different clusters using the fuzzy c -means clustering algorithm, (3) assigning the out-of-sample objects into their closest clusters via data labeling technique. The performance of the proposed algorithm is compared with the state-of-the-art sampling-based fuzzy c -means clustering algorithms on several large-scale data sets including synthetic and real ones. The experimental results show that the proposed algorithm outperforms the related algorithms in terms of clustering quality and computational efficiency for large-scale data sets. [ABSTRACT FROM AUTHOR]

[Display omitted] • Developed a multi-object localization algorithm by adapting an unsupervised single-object discovery algorithm. • Minimized human intervention in data annotation by proposing an AI-based automatic labeling method. • Achieved superior performance and generalization capacity by fine-tuning models previously trained with the automatically labeled data. • Accelerated the training of a competitive object detection and instance segmentation pipeline for detecting pears on trees with insufficient data. • Provided an automatic labeling method and training approach that can be implemented in arbitrary object detection applications. Accurate counting of pears in orchard environments is essential for crop management. However, due to the time and labor cost of manual counting, farmers rely on sampling a few trees and extending the count to the entire orchard, which overlooks the spatial variability of fruit load. This study proposes automatic labeling in case of insufficient manually labeled data to accelerate the adaptation of on-tree fruit counting solutions based on deep learning. Since the amount of training data needed can be one of the main limitations to deploying a robust solution in the field, here we present an approach to minimize human intervention in data labeling. We describe, first, how the latest advances in self-supervised learning and transformers can be used to create an automatic labeling procedure, and second, how the automatically labeled dataset can be exploited despite its noise. For the dataset creation using automatic labeling, we modified a single object discovery algorithm to localize multiple object instances, and we trained an image classifier to include only objects from our target class: pears. The automatically generated labels are regarded as 'noisy' because the quality of the annotations is lower than manually annotated data. We then pre-trained an object detection model with the automatically labeled data (121,038 annotations) and fine-tuned it with insufficient manually labeled data (500, 1000, and 2500 annotations). This allowed us to leverage knowledge from both datasets despite the quality and size restrictions. The performance comparison of object detection models trained this way to those trained only with insufficient manually labeled data showed average precision gains of 37.0, 21.4, and 1.0%. In addition, on unseen images taken with the same camera setup, the average precision gains were 35.9, 17.6, and 1.8%. Furthermore, gains were 38.5, 25.1, and 1.2% on unseen images taken with a different camera setup. These results suggest the substantial value of automatic labeling when having insufficient training data for fruit detection in pear orchards. Moreover, other object detection applications can replicate the presented methods and experiments, especially when resource constraints or rapid validation prevent extensive manual labeling. [ABSTRACT FROM AUTHOR]

Activity recognition, a key component in pervasive healthcare monitoring, relies on classification algorithms that require labeled data of individuals performing the activity of interest to train accurate models. Labeling data can be performed in a lab setting where an individual enacts the activity under controlled conditions. The ubiquity of mobile and wearable sensors allows the collection of large datasets from individuals performing activities in naturalistic conditions. Gathering accurate data labels for activity recognition is typically an expensive and time-consuming process. In this paper we present two novel approaches for semi-automated online data labeling performed by the individual executing the activity of interest. The approaches have been designed to address two of the limitations of self-annotation: (i) The burden on the user performing and annotating the activity, and (ii) the lack of accuracy due to the user labeling the data minutes or hours after the completion of an activity. The first approach is based on the recognition of subtle finger gestures performed in response to a data-labeling query. The second approach focuses on labeling activities that have an auditory manifestation and uses a classifier to have an initial estimation of the activity, and a conversational agent to ask the participant for clarification or for additional data. Both approaches are described, evaluated in controlled experiments to assess their feasibility and their advantages and limitations are discussed. Results show that while both studies have limitations, they achieve 80% to 90% precision.

High-quality labeled textual data are reported as an important type of research data in data-driven requirements engineering (RE), especially in automatic mining and analysis of massive textual data produced by software systems. Several tools have been designed to facilitate manual labeling of textual data at different levels of granularity. However, these tools neither aim to provide visualized statistics and analysis of labeled textual data, nor support collaboration among the coders to reduce the time cost in manual labeling and enhance the quality of labeling results. Besides, these tools seldom explicitly serve RE researchers. In this paper, we developed a Web-based labeling tool named CoolTeD (available at http://williamsriver.cn) for collaborative labeling of the textual datasets for RE purposes. Specifically, CoolTeD can be used to: (1) label textual data with the tag category based on ISO 25010 or other user-defined tag categories in a collaborative way; (2) review the labeling results with different confidence levels and contradictory labels, (3) identify contradictory labels and disagreements online; (4) automatically calculate the Cohen's Kappa coefficient of multiple coders, and (5) visualize the labeling results. The tool demo is available at https://youtu.be/KTVrLLenvLE. • CoolTeD supports the full spectrum of manual labeling of textual datasets for the RE purpose. • CoolTeD supports labeling one data item with multiple tags belonging to the default or user-defined tag categories • CoolTeD enables multiple coders to share and co-label textual datasets. • CoolTeD helps reviewers reach consistent labels efficiently by applying different checking strategies. [ABSTRACT FROM AUTHOR]