Search

This work was supported by project PID2020-119478GB-I00 granted by Ministerio de Ciencia, Innovacion y Univesidades, project P18-FR-4961 by Proyectos I+D+i Junta de Andalucia 2018 and the process no 2015/20606-6, FundacAo de Amparo a Pesquisa do Estado de SAo Paulo (FAPESP) ., Data in the real world is far from being perfect. The appearance of noise is a common issue that arises from the limitations of data acquisition mechanisms and human knowledge. In classification, label noise will hinder the performance of almost all classifiers, inducing a bias in the built model. While label noise has recently attracted researchers’ attention in standard classification, it has only recently begun to be studied in multiple instance classification. In this work, we propose the usage of filtering algorithms for multiple instance classification that are able to reduce the impact of negative instances within the bags. In order to do so, we decompose the bags to form a standard classification problem that can be efficiently treated by a specialized noise filter. Such a decomposition is tackled in different ways, with the aim of exploiting the knowledge offered by the examples from opposite bags. The bags are then rebuilt, without the identified noise instances. In our experiments, we show that by applying our approach we can diminish the impact of noise and even obtain better results at 0% noise level for several classifiers. Our approach sets out a promising approach to dealing with noise in the bags of multiple instance datasets and further improve the classification rate of the built models., Ministerio de Ciencia, Innovacion y Univesidades PID2020-119478GB-I00, Junta de Andalucia P18-FR-4961, Fundacao de Amparo a Pesquisa do Estado de Sao Paulo (FAPESP) 2015/20606-6

Causal classification concerns the estimation of the net effect of a treatment on an outcome of interest at the instance level, i.e., of the individual treatment effect (ITE). For binary treatment and outcome variables, causal classification models produce ITE estimates that essentially allow one to rank instances from a large positive effect to a large negative effect. Often, as in uplift modeling (UM), one is merely interested in this ranking, rather than in the ITE estimates themselves. In this regard, we investigate the potential of learning to rank (L2R) techniques to learn a ranking of the instances directly. We propose a unified formalization of different binary causal classification performance measures from the UM literature and explore how these can be integrated into the L2R framework. Additionally, we introduce a new metric for UM with L2R called the promoted cumulative gain (PCG). We employ the L2R technique LambdaMART to optimize the ranking according to PCG and show improved results over the use of standard L2R metrics and equal to improved results when compared with state-of-the-art UM. Finally, we show how L2R techniques can be used to specifically optimize for the top-k fraction of the ranking in a UM context, however, these results do not generalize to the test set. ispartof: IEEE Transactions on Knowledge and Data Engineering vol:34 issue:10 pages:4888-4904 status: published

The datasets that appear in publications are curated and have been split into training, testing and validation sub-datasets by domain experts. Consequently, machine learning models typically perform well on such split-by-hand prepared datasets. Whereas preparing real-world datasets into curated split training, testing and validation sub-dataset requires extensive effort. Usually, repetitive random splits are carried out and trained and evaluated on until reaching out a good score on the evaluation metrics. In this paper, an algorithmic method is proposed for preparing the sub-datasets splits for machine learning models. The objective of the proposed method is to achieve an evenly representative splits out of the dataset with standard and algorithmic way that reduce the perplexity of random splitting.

One of the critical aspects of building intelligent tutoring systems regards proper monitoring of student's activity and academic performance. This paper presents a continuous student knowledge tracing method implemented for Tesys e-Learning platform at the Faculty of Automation, Computers and Electronics in the University of Craiova. The student's knowledge level is continuously monitored and, after each recommended test by the SVD-based mechanism, a new set of knowledge weights are computed. We aim to achieve a comprehensive monitoring environment which can provide an accurate insight upon the student's knowledge level at any moment. In our approach, we added weights for both students and tests to improve the student's evolution monitoring and provide more accurate feedback. The setup for validation consisted of ten tests with eight questions per test and we used both current and past year tests data. Results revealed that assigning weights to questions, tests and students and using them in the recommendation process offers a better view of the student's evolution along with more accurate recommendations. Progress in this direction will provide more insight into available teaching materials and SVD-based recommender system such that the e-learning platform that integrates the presented mechanism will provide a better learning experience.

Many machine learning algorithms and almost all deep learning architectures are incapable of processing plain texts in their raw form. This means that their input to the algorithms must be numerical in order to solve classification or regression problems. Hence, it is necessary to encode these categorical variables into numerical values using encoding techniques. Categorical features are common and often of high cardinality. One-hot encoding in such circumstances leads to very high dimensional vector representations, raising memory and computability concerns for machine learning models. This paper proposes a deep-learned embedding technique for categorical features encoding on categorical datasets. Our technique is a distributed representation for categorical features where each category is mapped to a distinct vector, and the properties of the vector are learned while training a neural network. First, we create a data vocabulary that includes only categorical data, and then we use word tokenization to make each categorical data a single word. After that, feature learning is introduced to map all of the categorical data from the vocabulary to word vectors. Three different datasets provided by the University of California Irvine (UCI) are used for training. The experimental results show that the proposed deep-learned embedding technique for categorical data provides a higher F1 score of 89% than 71% of one-hot encoding, in the case of the Long short-term memory (LSTM) model. Moreover, the deep-learned embedding technique uses less memory and generates fewer features than one-hot encoding.

Due to the rapid developments in technology and the sudden expansion of social media use, Dialect Arabic has become an important source of data that needs to be addressed when building Arabic corpora. In this paper, thirty-three Arabic corpora are surveyed to show that despite all of the developments in the literature, Saudi dialect (SD) corpora still need further expansion. This paper contributes to the literature on SD corpora by creating the largest Saudi corpus – the King Saud University Saudi Corpus (KSUSC) – with +1B total words, including +119M SD words. The KSUSC not only is the newest and largest SD corpus but is also diverse, covering 26 domains in text collected from five different sources. This paper also contributes to the literature by developing a new incremental preprocessing system that is used to create relevant lexicons that are then used to clean and normalize the collected data. This incremental system is scalable and can be adapted for different resources and dialects. Moreover, the collection process for building the KSUSC is discussed in detail, and the challenges in collecting SD text with respect to each platform are highlighted. By the end of this paper, different design criteria are proposed and used with the KSUSC to conclude that the resulting corpus can be of great benefit to researchers who are interested in integrating the corpus with their own work or using its resulting lexicons with Saudi-based NLP tasks.

Optical motion capture (MOCAP) is a commonly used technology to record the motion of non-rigid objects with high accuracy in 3D space. However, the MOCAP data has to be processed further before it can be used. The scattered reconstructed motion data must constitute a human configuration by labelling process according to the predefined template, and the missing markers have to be reconstructed to produce a stable motion trajectory. In this work, we propose a novel labelling method for motion sequences. First, a novel graph matching method is employed to determine the connection relationship of the scattered motion data for a single frame. Then, Kalman filtering is used for tracking in the motion sequence. As for the challenge coming from missing markers, we propose a new motion data preprocessing method considering the bone length constraint, which represents the information of variation in the relative position of adjacent markers. The processed motion data is input into a Long-Short Term Memory (LSTM) model to recover the missing markers and de-noise the motion data. The experiment conducted on our own dataset proves that our labelling method achieves a similar effect to Cortex, which is a commonly used commercial motion data analysis software. The experiment on CMU dataset demonstrates that our missing marker reconstruction method can achieve an art-of-state result. The labelling code will be pulished on https://github.com/Lijianfang6930/Graph-Matching-for-Marker-Labelling

In the derived approach, an analysis is performed on Twitter data for World Cup soccer 2014 held in Brazil to detect the sentiment of the people throughout the world using machine learning techniques. By filtering and analyzing the data using natural language processing techniques, sentiment polarity was calculated based on the emotion words detected in the user tweets. The dataset is normalized to be used by machine learning algorithms and prepared using natural language processing techniques like word tokenization, stemming and lemmatization, part-of-speech (POS) tagger, name entity recognition (NER), and parser to extract emotions for the textual data from each tweet. This approach is implemented using Python programming language and Natural Language Toolkit (NLTK). A derived algorithm extracts emotional words using WordNet with its POS (part-of-speech) for the word in a sentence that has a meaning in the current context, and is assigned sentiment polarity using the SentiWordNet dictionary or using a lexicon-based method. The resultant polarity assigned is further analyzed using na&iuml, ve Bayes, support vector machine (SVM), K-nearest neighbor (KNN), and random forest machine learning algorithms and visualized on the Weka platform. Na&iuml, ve Bayes gives the best accuracy of 88.17% whereas random forest gives the best area under the receiver operating characteristics curve (AUC) of 0.97.

In this paper, for the first time, a novel discretization scheme is proposed aiming at enabling scalability but also at least three other strong challenges. It is based on a Left-to-Right (LR) scanning process, which partitions the input stream into intervals. This task can be implemented by an algorithm or by using a generator that builds automatically the discretization program. We focus especially on unsupervised discretization and design a method called Usupervised Left to Right Discretization (ULR-Discr). Extensive experiments were conducted using various cut-point functions on small, large and medical public datasets. First, ULR-Discr variants under different statistics are compared between themselves with the aim at observing the impact of the cut-point functions on accuracy and runtime. Then the proposed method is compared to traditional and recent techniques for classification. The result is that the classification accuracy is highly improved when using our method for discretization.

Tracking of eye movements is an established measurement for many types of experimental paradigms. More complex and more prolonged visual stimuli have made algorithmic approaches to eye movement event classification the most pragmatic option. A recent analysis revealed that many current algorithms are lackluster when it comes to data from viewing dynamic stimuli such as video sequences. Here we present an event classification algorithm—built on an existing velocity-based approach—that is suitable for both static and dynamic stimulation, and is capable of classifying saccades, post-saccadic oscillations, fixations, and smooth pursuit events. We validated classification performance and robustness on three public datasets: 1) manually annotated, trial-based gaze trajectories for viewing static images, moving dots, and short video sequences, 2) lab-quality gaze recordings for a feature length movie, and 3) gaze recordings acquired under suboptimal lighting conditions inside the bore of a magnetic resonance imaging (MRI) scanner for the same full-length movie. We found that the proposed algorithm performs on par or better compared to state-of-the-art alternatives for static stimulation. Moreover, it yields eye movement events with biologically plausible characteristics on prolonged dynamic recordings. Lastly, algorithm performance is robust on data acquired under suboptimal conditions that exhibit a temporally varying noise level. These results indicate that the proposed algorithm is a robust tool with improved classification accuracy across a range of use cases. The algorithm is cross-platform compatible, implemented using the Python programming language, and readily available as free and open source software from public sources.

A novel method of outlier detection for preprocessing stream data in the conditions of uncertainty when the data measurement means and their standard errors as the only available summary information about the initial data statistics is proposed. As neither the initial data samples nor their sample sizes are known, the classical methods of outlier detection including nonparametric methods of statistics cannot be applied in this case. The principal idea of the proposed approach to outlier detection is based on the use of the classical Gauss-Chebyshev type probability inequalities---the corresponding confidence intervals constructed on these inequalities allow to set up the problems of hypotheses testing similar to the classical settings as the problems of minimizing the upper bound of the Bayesian risk and maximizing the lower bound of the test power in the Neyman-Pearson sense. The results of the processing of the real-life data (Lunar Laser Ranging data) and the model data manifest unexpectedly good outlier detection performance.

In cytometry analysis, a large number of markers is measured for thousands or millions of cells, resulting in high-dimensional datasets. During the measurement of these samples, erroneous events can occur such as clogs, speed changes, slow uptake of the sample etc., which can influence the downstream analysis and can even lead to false discoveries. As these issues can be difficult to detect manually, an automated approach is recommended. In order to filter these erroneous events out, we created a novel quality control algorithm, Peak Extraction And Cleaning Oriented Quality Control (PeacoQC), that allows for automated cleaning of cytometry data. The algorithm will determine density peaks per channel on which it will remove low quality events based on their position in the isolation tree and on their mean absolute deviation distance to these density peaks. To evaluate PeacoQC's cleaning capability, it was compared to three other existing quality control algorithms (flowAI, flowClean and flowCut) on a wide variety of datasets. In comparison to the other algorithms, PeacoQC was able to filter out all different types of anomalies in flow, mass and spectral cytometry data, while the other methods struggled with at least one type. In the quantitative comparison, PeacoQC obtained the highest median balanced accuracy and a similar running time compared to the other algorithms while having a better scalability for large files. To ensure that the parameters chosen in the PeacoQC algorithm are robust, the cleaning tool was run on 16 public datasets. After inspection, only one sample was found where the parameters should be further optimized. The other 15 datasets were analyzed correctly indicating a robust parameter choice. Overall, we present a fast and accurate quality control algorithm that outperforms existing tools and ensures high-quality data that can be used for further downstream analysis. An R implementation is available.

Process Mining allows organizations to obtain actual business process models from event logs (discovery), to compare the event log or the resulting process model in the discovery task with the existing reference model of the same process (conformance), and to detect issues in the executed process to improve (enhancement). An essential element in the three tasks of process mining (discovery, conformance, and enhancement) is data cleaning, used to reduce the complexity inherent to real-world event data, to be easily interpreted, manipulated, and processed in process mining tasks. Thus, new techniques and algorithms for event data preprocessing have been of interest in the research community in business process. In this paper, we conduct a systematic literature review and provide, for the first time, a survey of relevant approaches of event data preprocessing for business process mining tasks. The aim of this work is to construct a categorization of techniques or methods related to event data preprocessing and to identify relevant challenges around these techniques. We present a quantitative and qualitative analysis of the most popular techniques for event log preprocessing. We also study and present findings about how a preprocessing technique can improve a process mining task. We also discuss the emerging future challenges in the domain of data preprocessing, in the context of process mining. The results of this study reveal that the preprocessing techniques in process mining have demonstrated a high impact on the performance of the process mining tasks. The data cleaning requirements are dependent on the characteristics of the event logs (voluminous, a high variability in the set of traces size, changes in the duration of the activities. In this scenario, most of the surveyed works use more than a single preprocessing technique to improve the quality of the event log. Trace-clustering and trace/event level filtering resulted in being the most commonly used preprocessing techniques due to easy of implementation, and they adequately manage noise and incompleteness in the event logs.

Intelligent dynamic spectrum resource management, which is based on vast amounts of sensing data from industrial IoT in the space–time and frequency domains, uses optimization algorithm-based decisions to minimize levels of interference, such as energy consumption, power control, idle channel allocation, time slot allocation, and spectrum handoff. However, these techniques make it difficult to allocate resources quickly and waste valuable solution information that is optimized according to the evolution of spectrum states in the space–time and frequency domains. Therefore, in this paper, we propose the implementation of intelligent dynamic real-time spectrum resource management through the application of data mining and case-based reasoning, which reduces the complexity of existing intelligent dynamic spectrum resource management and enables efficient real-time resource allocation. In this case, data mining and case-based reasoning analyze the activity patterns of incumbent users using vast amounts of sensing data from industrial IoT and enable rapid resource allocation, making use of case DB classified by case. In this study, we confirmed a number of optimization engine operations and spectrum resource management capabilities (spectrum handoff, handoff latency, energy consumption, and link maintenance) to prove the effectiveness of the proposed intelligent dynamic real-time spectrum resource management. These indicators prove that it is possible to minimize the complexity of existing intelligent dynamic spectrum resource management and maintain efficient real-time resource allocation and reliable communication; also, the above findings confirm that our method can achieve a superior performance to that of existing spectrum resource management techniques.

Because lithium-ion batteries are widely used for various purposes, it is important to estimate their state of health (SOH) to ensure their efficiency and safety. Despite the usefulness of model-based methods for SOH estimation, the difficulties of battery modeling have resulted in a greater emphasis on machine learning for SOH estimation. Furthermore, data preprocessing has received much attention because it is an important step in determining the efficiency of machine learning methods. In this paper, we propose a new preprocessing method for improving the efficiency of machine learning for SOH estimation. The proposed method consists of the relative state of charge (SOC) and data processing, which transforms time-domain data into SOC-domain data. According to the correlation analysis, SOC-domain data are more correlated with the usable capacity than time-domain data. Furthermore, we compare the estimation results of SOC-based data and time-based data in feedforward neural networks (FNNs), convolutional neural networks (CNNs), and long short-term memory (LSTM). The results show that the SOC-based preprocessing outperforms conventional time-domain data-based techniques. Furthermore, the accuracy of the simplest FNN model with the proposed method is higher than that of the CNN model and the LSTM model with a conventional method when training data are small.

Sensor monitoring networks and advances in big data analytics have guided the reliability engineering landscape to a new era of big machinery data. Low-cost sensors, along with the evolution of the internet of things and industry 4.0, have resulted in rich databases that can be analyzed through prognostics and health management (PHM) frameworks. Several da-ta-driven models (DDMs) have been proposed and applied for diagnostics and prognostics purposes in complex systems. However, many of these models are developed using simulated or experimental data sets, and there is still a knowledge gap for applications in real operating systems. Furthermore, little attention has been given to the required data preprocessing steps compared to the training processes of these DDMs. Up to date, research works do not follow a formal and consistent data preprocessing guideline for PHM applications. This paper presents a comprehensive, step-by-step pipeline for the preprocessing of monitoring data from complex systems aimed for DDMs. The importance of expert knowledge is discussed in the context of data selection and label generation. Two case studies are presented for validation, with the end goal of creating clean data sets with healthy and unhealthy labels that are then used to train machinery health state classifiers., 41 pages

International audience; With the emergence of Connected and Smart Cities, the need to predict traffic conditions has led to the development of a large variety of forecasting algorithms. In spite of various research efforts, the choice of models and techniques strongly depends on the use case, the highway infrastructure as well as the provided dataset. This study is launched as part of a project which aims to design an Intelligent Transport System (ITS) dedicated to highway supervisors to regulate traffic. This system needs to be supplied by continuous, real-time forecasting of short-term traffic congestions in order to make decisions accordingly. In this paper, we propose a general framework that, first, performs different data preprocessing techniques to improve data quality, and second, provides real-time multiple horizons predictions. Our framework uses different models combining Machine learning and Deep learning algorithms. Experiments results confirmed the necessity of the data preprocessing step, especially with highly dynamic data and heterogeneous mobility contexts. In addition, our methodology is tested in a real case study and shows very encouraging results.

In pattern recognition, irrelevant and redundant features together with a large number of noisy instances in the underlying dataset decrease performance of trained models and make the training process considerably slower, if not practically infeasible. In order to combat this so-called curse of dimensionality, one option is to resort to feature selection (FS) methods designed to select the features that contribute the most to the performance of the model, and one other option is to utilize feature extraction (FE) methods that map the original feature space into a new space with lower dimensionality. These two methods together are called feature reduction (FR) methods. On the other hand, deploying an FR method on a dataset with massive number of instances can become a major challenge, from both memory and run time perspectives, due to the complex numerical computations involved in the process. The research question we consider in this study is rather a simple, yet novel one: do these FR methods really need the whole set of instances (WSI) available for the best performance, or can we achieve similar performance levels with selecting a much smaller random subset of WSI prior to deploying an FR method? In this work, we provide empirical evidence based on comprehensive computational experiments that the answer to this critical research question is in the affirmative. Specifically, with simple random instance selection followed by FR, the amount of data needed for training a classifier can be drastically reduced with minimal impact on classification performance. We also provide recommendations on which FS/ FE method to use in conjunction with which classifier.

Rotating machinery plays a critical role in many significant fields. However, the unpredictable machinery faults may lead to the severe damage and losses. Hence, it is of great value to explore the precise approaches for fault diagnosis. With the development of the intelligent fault diagnosis methods based on deep learning, convolutional neural network (CNN) has aroused the attention of researchers in machinery fault diagnosis. In the light of the reduction of difficulty in feature learning and the improvement of final diagnosis accuracy, data preprocessing is necessary and crucial in CNN-based fault diagnosis methods. This review focuses on CNN-based fault diagnosis approaches in rotating machinery. Firstly, data preprocessing methods are overviewed. Then, we emphatically analyze and discuss several main techniques applied in CNN-based intelligent diagnosis, principally including the fast Fourier transform, wavelet transform, data augmentation, S-transform, and cyclic spectral analysis. Finally, the potential challenges and research objects are prospected on data preprocessing in intelligent fault diagnosis of rotary machinery.

Machine learning for author name disambiguation is usually conducted on the training and test subsets of labeled data created for a specific task. As a result, disambiguation models learned on heterogeneous labeled data are often inapplicable for other purposes that either do not use the same labeled data or do not make use of any labeled data at all. This article explores the idea of transfer learning in a new context, author name disambiguation. We focus on cases where a disambiguation task lacking labeled training data uses models trained on labeled data generated for other tasks. For this purpose, two labeled source datasets are used for training of disambiguation models to be applied to three test target datasets that are deficient of labeled training data. Our results show that transfer learning can produce disambiguation performances similar to those achievable by traditional machine learning in which training and test datasets come from the same labeled data source. The good performance through transfer learning are possible when training source datasets have similar feature distributions as test target datasets. This study suggests that through transfer learning, rich disambiguation models in previous studies can be retained and reused across ambiguous bibliographic data from different fields and data sources, motivating further research on how to correct feature distribution differences between source and target datasets to expand the application of transfer learning in author name disambiguation beyond the model sharing explored in this research.

Multistep wind speed forecasting is of great significance for wind energy utilization. To further improve the performance of multistep wind speed forecasting, a new multistep forecasting strategy is proposed based on the basic wind speed multistep forecasting strategy and the corresponding forecasting system is established. The system consists of three parts, including a wavelet decomposition preprocessing module, nonlinear autoregressive artificial neural network and nonlinear autoregressive exogenous artificial neural network composite prediction module, and support vector machine classifier postprocessing module. The system analyzes historical wind speed data from two different angles to obtain dual time series features. Then, the double time series feature is used to improve the accuracy and stability of the multipart forecast. The predictive performance of the system is verified through two experiments. Experiment I tests the prediction accuracy and computing resource consumption of the system. Experiment II tests the stability of the system prediction under different wind conditions. The results show that, compared with the traditional multistep prediction strategy system, the new system has a better prediction accuracy and more stable prediction performance.

A novel hybrid ensemble framework is developed to forecast the short-term wind speed, which consists of a data preprocessing technique, data-driven based forecasting algorithms, and an improved Jaya algorithm. In the data preprocessing process, the pauta criterion is employed to find out the outliers, and the variational mode decomposition algorithm decompose the original series to extract the trend and time-frequency information of the historical inputs. The data-driven forecasting algorithms, such as BP, LSSVM, ANFIS, and Elman, are exploited as the original predictor of the framework, while the weights of the predictors are computed by an improved optimization algorithm-CLSJaya. Based on the experimental results of two time-scale datasets from three sites, the proposed framework successfully overcomes the limitations of the individual forecasting models and achieves promising forecasting accuracy.

The instance reduction is one of the data preprocessing methods and aims to remove noises and (or) redundant instances from the training set. In the instance reduction, one of the most representative techniques is the edition method which can remove harmful instances from the training set to improve the prediction accuracy of k nearest neighbor (KNN). Nevertheless, most of existing edition methods still have some drawbacks, such as the parameter dependency, high computational time and relatively low accuracy. To solve these problems, we present a new fast parameter-free edition method based on local sets with natural neighbors (ELS). In ELS, we define a new concept of local sets by introducing natural neighbors. ELS can use the local sets to keep more reasonable class boundaries and effectively filtering out noisy instances (including global outliers). The main advantages of ELS are that (a) it is parameter-free; (b) it can remove global outliers and noisy instances; (c) it is relatively fast. Experiments clearly verify that (a) ELS outperforms existing representative edition methods in improving the prediction accuracy of KNN; (b) ELS can improve the performance of the condensation method and hybrid method in terms of both accuracy and reduction; (c) ELS consumes relatively low running time.

Smoking is associated with cancer, cardiovascular disease and premature death and can cause severe fire hazards. To assist with smoking cessation, this paper presents a wireless body area network-based system consisting of two off-the-shelf devices, one smartphone and one smartwatch, to detect smoking events by mining the inertial sensor data from both devices. In the system, an end-to-end trainable unified model is implemented by combining a variational autoencoder with a random forest to classify the collected data into smoking and nonsmoking after data preprocessing. The variational autoencoder is adopted to learn the feature representation and deal with the class imbalance problem, and the stochastic decision forest is adopted to guide the global optimization of the parameter learning process. Extensive validation of our scheme is performed on a large dataset we collected, and our scheme yields quite promising results in terms of accuracy and efficiency.

Imbalanced class distribution and missing data are two common problems and occurrences in water quality anomaly detection domain. Learning algorithms in an imbalanced dataset can yield an overrated classification accuracy driven by a bias towards the majority class at the expense of the minority class. On the other hand, missing values in data can induce complexity in the learning classifiers during data analysis. These two problems pose substantial challenges to the performance of learning algorithms in real-life water quality anomaly detection problems. Hence, the need for them to be carefully considered and addressed to achieve better performance. In this paper, the performance of a range of several combinations of techniques to deal with imbalanced classes in the context of binary-imbalanced water quality anomaly detection problem and the presence of missing values is extensively compare. The methods considered include seven missing data and eight resampling methods, on ten different learning state-of-the-art classifiers taking into account diversity in their learning philosophies. The different classifiers are evaluated using stratified 5-fold cross-validation, based on three performance evaluation metrics namely accuracy, ROC-AUC and F1-measure. Further experiments are carried out on nineteen variants of homogeneous and heterogeneous ensemble techniques embedded with resampling and missing value strategies during their training phase as well as an optimized deep neural network model. The experimental results show an improvement in the performance of the learning classifiers, especially when dealing with the class imbalance problem (on the one hand) and the incomplete data problem (on the other hand). Furthermore, the neural network model exhibit superior performance when dealing with both problems.

This work presents two methods that facilitate a 3D reconstruction of microscopic blood vessels in the volume slightly larger than 1 mm3. The source of the data are histological serial sections, i.e., microscopic images of probes, stained with immunohistochemistry. Odd and even sections have different stainings in our primary data set. Thus, firstly, an approach to register an alternately-stained series is presented. With image filtering and a feature-detection-based registration we obtain a registered stack of 148 serial sections. The series has missing sections, locally damaged sections, artifacts from acquisition. All these hinder correct connectivity of blood vessels. With our second approach we interpolate the missing information while maintaining the connectivity. We achieve this with deformations based on dense optical flow. The presented methodology is applicable to further histological series. A combination of both approaches allows us to reconstruct more than 76% larger volumes. An important detail was the composition mode of images. Summarizing, we use methods from image processing and computer vision to create large-scale 3D models from immunostained histological serial sections.

GPS trajectories are always embedded with errors, due to the weather or environmental variables. Existing trajectory repairing methods have employed Kalman filters or sequential data cleaning methods. Kalman filter or its variants change all observed measurements, while generally most measurements are originally accurate. Sequential data cleaning methods are mainly applied on one-dimensional data sequences, and when encountering multi-dimensional trajectories, their performance will be compromised due to that the features of multi-dimensional trajectories are not fully utilized. To address these issues, we propose to repair GPS trajectory with movement tendencies, speed change tendency, travel distance tendency and repair distance tendency. We formalize the tendency based trajectory repairing, and propose an exact solution to find the repair which minimize movement tendency score. Then we propose high quality candidate selection and dynamic error range estimation, to improve the efficiency and effectiveness of exact solution. Experiments on three data sets demonstrate the superiority of our proposal.

Human exposure induced by wireless communication systems increasingly draws the public attention. Here, an indoor down-link scenario is concerned and the exposure level is statistically analyzed. The electromagnetic field emitted by a WiFi box is measured and electromagnetic dosimetry features are evaluated from the whole-body specific absorption rate as computed with a finite-difference time-domain (a.k.a. FDTD) code. Due to computational cost, a statistical analysis is performed based on a surrogate model, which is constructed by means of so-called sparse polynomial chaos expansion, where the inner cross validation (ICV) is used to select the optimal hyperparameters during the model construction and assess the model performance. However, the model assessment based on ICV tends to be overly optimistic with small data sets. The method of cross-model validation is used and outer cross validation is carried out for the model assessment. The effects of the data preprocessing are investigated as well. On the basis of the surrogate model, the global sensitivity of the exposure to input parameters is analyzed from Sobol' indices., International Journal for Uncertainty Quantification, 10 (2), ISSN:2152-5080, ISSN:2152-5099

Missing data filling is a key step in power big data preprocessing, which helps to improve the quality and the utilization of electric power data. Due to the limitations of the traditional methods of filling missing data, an improved random forest filling algorithm is proposed. As a result of the horizontal and vertical directions of the electric power data are based on the characteristics of time series. Therefore, the method of improved random forest filling missing data combines the methods of linear interpolation, matrix combination and matrix transposition to solve the problem of filling large amount of electric power missing data. The filling results show that the improved random forest filling algorithm is applicable to filling electric power data in various missing forms. What's more, the accuracy of the filling results is high and the stability of the model is strong, which is beneficial in improving the quality of electric power data.

Data preprocessing techniques are devoted to correct or alleviate errors in data. Discretization and feature selection are two of the most extended data preprocessing techniques. Although we can find many proposals for static Big Data preprocessing, there is little research devoted to the continuous Big Data problem. Apache Flink is a recent and novel Big Data framework, following the MapReduce paradigm, focused on distributed stream and batch data processing. In this paper we propose a data stream library for Big Data preprocessing, named DPASF, under Apache Flink. We have implemented six of the most popular data preprocessing algorithms, three for discretization and the rest for feature selection. The algorithms have been tested using two Big Data datasets. Experimental results show that preprocessing can not only reduce the size of the data, but to maintain or even improve the original accuracy in a short time. DPASF contains useful algorithms when dealing with Big Data data streams. The preprocessing algorithms included in the library are able to tackle Big Datasets efficiently and to correct imperfections in the data., 19 pages

Background Although previous research showed that telehealth services can reduce the misuse of resources and urban–rural disparities, most healthcare insurers do not include telehealth services in their health insurance schemes. Therefore, no target variable exists for the classification approaches to learn from or train with. The problem of identifying the potential recipients of telehealth services when introducing telehealth services into health welfare or health insurance schemes becomes an unsupervised classification problem without a target variable. Methods We propose a HDTTCA approach, which is a systematic approach (the main process of HDTTCA involves (1) data set preprocessing, (2) decision tree model building, and (3) predicting and explaining of the most important attributes in the data set for patients who qualify for telehealth service) to identify those who are eligible for telehealth services. Results This work uses data from the NHIRD provided by the NHIA in Taiwan in 2012 as our research scope, which consist of 55,389 distinct hospitals and 653,209 distinct patients with 15,882,153 outpatient and 135,775 inpatient records. After HDTTCA produces the final version of the decision tree, the rules can be used to assign the values of the target variables in the entire NHIRD. Our data indicate that 3.56% (23,262 out of 653,209) of the patients are eligible for telehealth services in 2012. This study verifies the efficiency and validity of HDTTCA by using a large data set from the NHI of Taiwan. Conclusion This study conducts a series of experiments 30 times to compare the HDTTCA results with the logistic regression findings by measuring their average performance and determining which model addresses the telehealth patient classification problem better. Four important metrics are used to compare the results. In terms of sensitivity, the decision trees generated by HDTTCA and the logistic regression model are on equal grounds. In terms of accuracy, specificity, and precision, the decision tree generated by HDTTCA provides a better performance than that of the logistic regression model. When HDTTCA is applied, the decision tree model generates a competitive performance and provides clear, easily understandable rules. Therefore, HDTTCA is a suitable choice in solving telehealth service classification problems.

Data preprocessing is known to be essential to produce accurate data from which mining methods are able to extract valuable knowledge. When data constantly arrives from one or more sources, preprocessing techniques need to be adapted to efficiently handle these data streams. To help domain experts to define and execute preprocessing tasks for data streams, this paper proposes the use of active rule-based systems and, more specifically, complex event processing (CEP) languages and engines. The main contribution of our approach is the formulation of preprocessing procedures as event detection rules, expressed in an SQL-like language, that provide domain experts a simple way to manipulate temporal data. This idea is materialized into a publicly available solution that integrates a CEP engine with a library for online data mining. To evaluate our approach, we present three practical scenarios in which CEP rules preprocess data streams with the aim of adding temporal information, transforming features and handling missing values. Experiments show how CEP rules provide an effective language to express preprocessing tasks in a modular and high-level manner, without significant time and memory overheads. The resulting data streams do not only help improving the predictive accuracy of classification algorithms, but also allow reducing the complexity of the decision models and the time needed for learning in some cases.

The real-world data analysis and processing using data mining techniques often are facing observations that contain missing values. The main challenge of mining datasets is the existence of missing values. The missing values in a dataset should be imputed using the imputation method to improve the data mining methods’ accuracy and performance. There are existing techniques that use k-nearest neighbors algorithm for imputing the missing values but determining the appropriate k value can be a challenging task. There are other existing imputation techniques that are based on hard clustering algorithms. When records are not well-separated, as in the case of missing data, hard clustering provides a poor description tool in many cases. In general, the imputation depending on similar records is more accurate than the imputation depending on the entire dataset's records. Improving the similarity among records can result in improving the imputation performance. This paper proposes two numerical missing data imputation methods. A hybrid missing data imputation method is initially proposed, called KI, that incorporates k-nearest neighbors and iterative imputation algorithms. The best set of nearest neighbors for each missing record is discovered through the records similarity by using the k-nearest neighbors algorithm (kNN). To improve the similarity, a suitable k value is estimated automatically for the kNN. The iterative imputation method is then used to impute the missing values of the incomplete records by using the global correlation structure among the selected records. An enhanced hybrid missing data imputation method is then proposed, called FCKI, which is an extension of KI. It integrates fuzzy c-means, k-nearest neighbors, and iterative imputation algorithms to impute the missing data in a dataset. The fuzzy c-means algorithm is selected because the records can belong to multiple clusters at the same time. This can lead to further improvement for similarity. FCKI searches a cluster, instead of the whole dataset, to find the best k-nearest neighbors. It applies two levels of similarity to achieve a higher imputation accuracy. The performance of the proposed imputation techniques is assessed by using fifteen datasets with variant missing ratios for three types of missing data; MCAR, MAR, MNAR. These different missing data types are generated in this work. The datasets with different sizes are used in this paper to validate the model. Therefore, proposed imputation techniques are compared with other missing data imputation methods by means of three measures; the root mean square error (RMSE), the normalized root mean square error (NRMSE), and the mean absolute error (MAE). The results show that the proposed methods achieve better imputation accuracy and require significantly less time than other missing data imputation methods.

Topside piping is the most commonly failed equipment in the Petroleum and Maritime industry. The prominent degradation mechanism causing piping failure is fatigue which results in unnecessary hydrocarbon release from these assets. In order to avoid the unexpected fatigue failure of piping, it is essential to estimate the remaining fatigue life (RFL) of the aforementioned assets. Generally, engineering companies either rely on experimentally derived SN curves or on a probabilistic fracture mechanics approach to predict RFL. More recently, researchers have utilized surrogate models (classical ML models) to predict the same, and the results seem to be promising. In this manuscript, authors have tried to employ Deep Learning in order to predict the RFL of a topside piping in which crack has been detected. Firstly, different sources of uncertainty in the crack growth process are identified and quantified, with suitable distributions and parameters obtained from the literature. Thereafter, Monte Carlo Simulation is used to generate 5000 samples of the data consisting of 3 input parameters and 1 target feature (RFL class). Afterwards, the data is preprocessed, and feature importance criteria is applied. Finally, a custom Deep Learning model is developed to estimate the RFL class. An accuracy of 0.96 is achieved.

The rapid development in data science and the increasing availability of building operational data have provided great opportunities for developing data-driven solutions for intelligent building energy management. Data preprocessing serves as the foundation for valid data analyses. It is an indispensable step in building operational data analysis considering the intrinsic complexity of building operations and deficiencies in data quality. Data preprocessing refers to a set of techniques for enhancing the quality of the raw data, such as outlier removal and missing value imputation. This article serves as a comprehensive review of data preprocessing techniques for analysing massive building operational data. A wide variety of data preprocessing techniques are summarised in terms of their applications in missing value imputation, outlier detection, data reduction, data scaling, data transformation, and data partitioning. In addition, three state-of-the-art data science techniques are proposed to tackle practical data challenges in the building field, i.e., data augmentation, transfer learning, and semi-supervised learning. In-depth discussions have been presented to describe the pros and cons of existing preprocessing methods, possible directions for future research and potential applications in smart building energy management. The research outcomes are helpful for the development of data-driven research in the building field.

Patellofemoral pain syndrome (PFPS) is a common disease of the knee. Despite its high incidence rate, its specific cause remains unclear. The artificial neural network model can be used for computer-aided diagnosis. Traditional diagnostic methods usually only consider a single factor. However, PFPS involves different biomechanical characteristics of the lower limbs. Thus, multiple biomechanical characteristics must be considered in the neural network model. The data distribution between different characteristic dimensions is different. Thus, preprocessing is necessary to make the different characteristic dimensions comparable. However, a general rule to follow in the selection of biomechanical data preprocessing methods is lacking, and different preprocessing methods have their own advantages and disadvantages. Therefore, this paper proposes a multi-input convolutional neural network (MI-CNN) method that uses two input channels to mine the information of lower limb biomechanical data from two mainstream data preprocessing methods (standardization and normalization) to diagnose PFPS. Data were augmented by horizontally flipping the multi-dimensional time-series signal to prevent network overfitting and improve model accuracy. The proposed method was tested on the walking and running datasets of 41 subjects (26 patients with PFPS and 15 pain-free controls). Three joint angles of the lower limbs and surface electromyography signals of seven muscles around the knee joint were used as input. MI-CNN was used to automatically extract features to classify patients with PFPS and pain-free controls. Compared with the traditional single-input convolutional neural network (SI-CNN) model and previous methods, the proposed MI-CNN method achieved a higher detection sensitivity of 97.6%, a specificity of 76.0%, and an accuracy of 89.0% on the running dataset. The accuracy of SI-CNN in the running dataset was about 82.5%. The results prove that combining the appropriate neural network model and biomechanical analysis can establish an accurate, convenient, and real-time auxiliary diagnosis system for PFPS to prevent misdiagnosis.

54th CIRP Conference on Manufacturing Systems 2021 ��� "Towards Digitalized Manufacturing 4.0", CIRP-CMS 2021, online, 22 Sep 2021 - 24 Sep 2021; Procedia CIRP 104, 50-55 (2021). doi:10.1016/j.procir.2021.11.009 special issue: "54th CIRP CMS 2021 - Towards Digitalized Manufacturing 4.0 / Edited by Dimitris Mourtzis", Published by Elsevier, Amsterdam

Falls are dangerous for the elderly, often causing serious injuries especially when the fallen person stays on the ground for a long time without assistance. This paper extends our previous work on the development of a Fall Detection System (FDS) using an inertial measurement unit worn at the waist. Data come from SisFall, a publicly available dataset containing records of Activities of Daily Living and falls. We first applied a preprocessing and a feature extraction stage before using five Machine Learning algorithms, allowing us to compare them. Ensemble learning algorithms such as Random Forest and Gradient Boosting have the best performance, with a Sensitivity and Specificity both close to 99%. Our contribution is: a multi-class classification approach for fall detection combined with a study of the effect of the sensors’ sampling rate on the performance of the FDS. Our multi-class classification approach splits the fall into three phases: pre-fall, impact, post-fall. The extension to a multi-class problem is not trivial and we present a well-performing solution. We experimented sampling rates between 1 and 200 Hz. The results show that, while high sampling rates tend to improve performance, a sampling rate of 50 Hz is generally sufficient for an accurate detection.

In recent years, wind energy has become a competitively priced source of energy around the world, which has created increasing challenges for system operators. Accurate wind power generation forecasting plays an important role in power systems to improve the reliable and efficient operation. Therefore, numerous artificial intelligent methods such as machine learning and deep learning have been considered as solutions for accurate wind power forecasts. In addition to deterministic forecasting, the probabilistic forecasting becomes more important, because it indicates the level of uncertainty. In this paper, a hybrid forecasting model considering different Numerical Weather Prediction (NWP) models and the XGBoost training model is proposed for short-term wind power forecasting. The proposed forecasting algorithm includes data preprocessing, in which an autoencoder model is used to reduce the dimension of 20 NWP ensembles. The performance of the proposed method is investigated using historical wind power measurements and NWP results by the Taiwan Central Weather Bureau (CWB), the NWP includes spot wind speeds from WRFD, RWRF, and ensemble wind speeds from WEPS. Based on the forecasting results, the proposed model produces better performance and forecasting accuracy among other forecasting models, which reveals the importance of data preprocessing using autoencoders and the use of deep learning models in deterministic or probabilistic forecasts.

Currently, remote sensing is widely used in environmental monitoring applications, mostly air quality mapping and climate change supervision. However, satellite sensors occur massive volumes of data in near-real-time, stored in multiple formats and are provided with high velocity and variety. Besides, the processing of satellite big data is challenging. Thus, this study aims to approve that satellite data are big data and proposes a new big data architecture for satellite data processing. The developed software is enabling an efficient remote sensing big data ingestion and preprocessing. As a result, the experiment results show that 86 percent of the unnecessary daily files are discarded with a data cleansing of 20 percent of the erroneous and inaccurate plots. The final output is integrated into the Hadoop system, especially the HDFS, HBase, and Hive, for extra calculation and processing.

The scope of this paper is focused on the multidimensional poverty problem in Jordan. Household expenditure and income surveys provide data that are used for identifying and measuring the poverty status of Jordanian households. However, carrying out such surveys is hard, time consuming, and expensive. Machine learning could revolutionize this process. The contribution of this work is the proposal of an original machine learning approach to assess and monitor the poverty status of Jordanian households. This approach takes into account all the household expenditure and income surveys that took place since the early beginning of the new millennium. This approach is accurate, inexpensive, and makes poverty identification cheaper and much closer to real-time. Data preprocessing and handling imbalanced data are major parts of this work. Various machine learning classification models are applied. The LightGBM algorithm has achieved the best performance with 81% F1-Score. The final machine learning classification model could transform efforts to track and target poverty across the country. This work demonstrates how powerful and versatile machine learning can be, and hence, it promotes for adoption across many domains in both the private sector and government.

Methods to quantify the complexity of trajectory datasets are still a missing piece in benchmarking human trajectory prediction models. In order to gain a better understanding of the complexity of trajectory prediction tasks and following the intuition, that more complex datasets contain more information, an approach for quantifying the amount of information contained in a dataset from a prototype-based dataset representation is proposed. The dataset representation is obtained by first employing a non-trivial spatial sequence alignment, which enables a subsequent learning vector quantization (LVQ) stage. A large-scale complexity analysis is conducted on several human trajectory prediction benchmarking datasets, followed by a brief discussion on indications for human trajectory prediction and benchmarking.

Yere nüfuz eden radar sistemleri yaklaşık yirmi senedir arkeoloji, jeoloji, inşaat mühendisliği alanlarında yaygın olarak kullanılan bir teknolojidir. Yere nüfuz eden radar önemli bir uzaktan algılama teknolojisi olup, yüzey altındaki nesne ve katmanların algılanmasını ve konumunun belirlenmesini elektromanyetik yöntemlerle sağlamaktadır. Tüm metalik nesneler bir metal detektörü tarafından tespit edilip tanımlanmış olsa da plastik veya düşük metal içerikli kara mayınlarını tespit etmek ve tanımlamak için başka teknolojilere ihtiyaç bulunmaktadır. Yerin altının görüntülenmesine ihtiyaç duyulan tüm sivil ve askeri alanlardaki ihtiyaçların karşılanması kapsamında teknolojik gelişmeler doğrultusunda sistem geliştirme çalışmaları sürdürülmektedir. Bu çalışmada, radardan elde edilen verilerin işlenmesine yönelik görüntü ön işleme algoritma yapıları incelenmiştir. Aynı zamanda ön görüntüleme aşamasında yapılacak görüntü iyileştirmelerinin sistem başarımına etkisi irdelenmiştir. Önişleme aşamasındaki önerilen çapraz korelasyon yöntemi, yine önişleme yöntemi olarak kullanılan En Küçük Ortalama Kareler ve Kernel En Küçük Ortalama Kareler ile hız ve başarılı tespit açısından karşılaştırılmıştır. Sistemin gerçek zamanlı çalışmasına yönelik farklı yöntemler incelenmiş, çapraz korelasyon yöntemi ile daha hızlı ve başarılı tespitlerin yapılacağı gösterilmiştir. The Ground Penetrating Radar (GPR) systems have been widely used in the fields of archaeology, geology, civil engineering for about twenty years. The GPR is an important remote sensing technology that allows objects and layers under the surface to be perceived and positioned using electromagnetic techniques. Although all metallic objects are detected and identified by a metal detector, other technologies are needed to detect and identify landmines with plastic or low metal content. System development studies are continuing in line with technological developments within the scope of meeting the needs of all civil and military areas required to display under the surface. In this study, image pre-processing algorithms for processing the data obtained from the radar are examined. The effects of system improvements on the image enhancements to be made during the pre-screening phase have been studied. The proposed cross-correlation method in the pre-processing phase was compared with the Least Mean Squares and Kernel Least Mean Squares, also used as the pre-processing method, in terms of speed and successful detection. To run the system in real-time operations, these methods have been examined and as a result very fast and improved results have been obtained with the cross-correlation method compared to other methods.

In the construction industry, it is difficult to predict occupational accidents because various accident characteristics arise simultaneously and organically in different types of work. Furthermore, even when analyzing occupational accident data, it is difficult to deduce meaningful results because the data recorded by the incident investigator are qualitative and include a wide variety of data types and categories. Recently, numerous studies have used machine learning to analyze the correlations in such complex construction accident data, however, heretofore the focus has been on predicting severity with various variables, and several limitations remain when deriving the correlations between features from various variables. Thus, this paper proposes a data processing procedure that can efficiently manipulate accident data using optimal machine learning techniques and derive and systematize meaningful variables to rationally approach such complex problems. In particular, among the various variables, the most influential variables are derived through methods such as clustering, chi-square, Cramer&rsquo, s V, and predictor importance, then, the analysis is simplified by optimally grouping the variables. For accident data with optimal variables and elements, a predictive model is constructed between variables, using a support vector machine and decision-tree-based ensemble, then, the correlation between the dependent and independent variables is analyzed through an alluvial flow diagram for several cases. Therefore, a new processing procedure has been introduced in data preprocessing and accident prediction modelling to overcome difficulties from complex and diverse construction occupational accident data, and effective accident prevention is possible by deriving correlations of construction accidents using this process.

The new so-called COVID-19 virus is unfortunately founded to be highly transmissible across the globe. In this study, we propose a novel approach for estimating the spread level of the virus for each country for three different dates between April and May 2020. Unlike previous studies, this investigation does not process any historical data of spread but rather relies on the socio-economic indicators of each country. Actually, more than 1000 socio-economic indicators and more than 190 countries were processed in this study. Concretely, data preprocessing techniques and feature selection approaches were applied to extract relevant indicators for the classification process. Countries around the globe were assigned to 4 classes of spread. To find the class level of each country, many classifiers were proposed based especially on Support Vectors Machines (SVM), Multi-Layer Perceptrons (MLP) and Random Forests (RF). Obtained results show the relevance of our approach since many classifiers succeeded in capturing the spread level, especially the RF classifier, with an F-measure equal to 93.85% for April 15th, 2020. Moreover, a feature importance study is conducted to deduce the best indicators to build robust spread level classifiers. However, as pointed out in the discussion, classifiers may face some difficulties for future dates since the huge increase of cases and the lack of other relevant factors affecting this widespread.

The most common machine-learning methods solve supervised and unsupervised problems based on datasets where the problem&rsquo, s features belong to a numerical space. However, many problems often include data where numerical and categorical data coexist, which represents a challenge to manage them. To transform categorical data into a numeric form, preprocessing tasks are compulsory. Methods such as one-hot and feature-hashing have been the most widely used encoding approaches at the expense of a significant increase in the dimensionality of the dataset. This effect introduces unexpected challenges to deal with the overabundance of variables and/or noisy data. In this regard, in this paper we propose a novel encoding approach that maps mixed-type data into an information space using Shannon&rsquo, s Theory to model the amount of information contained in the original data. We evaluated our proposal with ten mixed-type datasets from the UCI repository and two datasets representing real-world problems obtaining promising results. For demonstrating the performance of our proposal, this was applied for preparing these datasets for classification, regression, and clustering tasks. We demonstrate that our encoding proposal is remarkably superior to one-hot and feature-hashing encoding in terms of memory efficiency. Our proposal can preserve the information conveyed by the original data.

The dramatic increase in devices and services that has characterized modern societies in recent decades, boosted by the exponential growth of ever faster network connections and the predominant use of wireless connection technologies, has materialized a very crucial challenge in terms of security. The anomaly-based intrusion detection systems, which for a long time have represented some of the most efficient solutions to detect intrusion attempts on a network, have to face this new and more complicated scenario. Well-known problems, such as the difficulty of distinguishing legitimate activities from illegitimate ones due to their similar characteristics and their high degree of heterogeneity, today have become even more complex, considering the increase in the network activity. After providing an extensive overview of the scenario under consideration, this work proposes a Local Feature Engineering (LFE) strategy aimed to face such problems through the adoption of a data preprocessing strategy that reduces the number of possible network event patterns, increasing at the same time their characterization. Unlike the canonical feature engineering approaches, which take into account the entire dataset, it operates locally in the feature space of each single event. The experiments conducted on real-world data showed that this strategy, which is based on the introduction of new features and the discretization of their values, improves the performance of the canonical state-of-the-art solutions.

Simple Summary Many studies are working with GPS sensors to track farm animals in order to gather information related to the animals’ health, the use of space and resources provided to the animals or their social contacts. When a herd of animals is tracked uninterruptedly, a huge amount of data is generated. Furthermore, problems like the limited battery life of wireless sensors have to be dealt with. This article presents the software “HerdGPS-Preprocessor” for the preparation of GPS data collected from an animal herd. The GPS data is provided cleaned and organized, such that the output files enable direct statistical analysis. Additionally, contact lists are output to enable network analyses. This study also delivers an example analysis with the “HerdGPS-Preprocessor”, in which GPS data of forty horses kept in a loose-housing barn were used to visualise and analyse contact networks. From the calculation of network parameters (density, diameter) and metrics associated to the clique structure of the horses, resting and activity phases could be distinguished. Furthermore, by network analysis it was shown that pasture opening and the visiting times of horse owners affect the contact structure. Thus, “HerdGPS-Preprocessor” is a valuable tool in studies collecting GPS data from herd animals. Abstract Sensors delivering information on the position of farm animals have been widely used in precision livestock farming. Global Positioning System (GPS) sensors are already known from applications in military, private and commercial environments, and their application in animal science is increasing. However, as trade-offs between sensor cost, battery life and sensor weight have to be made, GPS based studies scheduling long data collection periods and including a high number of animals, have to deal with problems like high hardware costs and data disruption during recharging of sensors. Furthermore, human–animal interaction due to sensor changing at the end of battery life interferes with the animal behaviour under analysis. The present study thus proposes a setting to deal with these challenges and offers the software tool “HerdGPS-Preprocessor”, because collecting position data from multiple animals nonstop for several weeks produces a high amount of raw data which needs to be sorted, preprocessed and provided in a suitable format per animal and day. The software tool “HerdGPS-Preprocessor” additionally outputs contact lists to enable a straight analysis of animal contacts. The software tool was exemplarily deployed for one month of daily and continuous GPS data of 40 horses in a loose-housing boarding facility in northern Germany. Contact lists were used to generate separate networks for every hour, which are then analysed with regard to the network parameter density, diameter and clique structure. Differences depending on the day and the day time could be observed. More dense networks with more and larger cliques were determined in the hours prior to the opening of additional pasture.