Search

Temporal proteomics data sets are often confounded by the challenges of missing values. These missing data points, in a time-series context, can lead to fluctuations in measurements or the omission of critical events, thus hindering the ability to fully comprehend the underlying biomedical processes. We introduce a Data Multiple Imputation (DMI) pipeline designed to address this challenge in temporal data set turnover rate quantifications, enabling robust downstream analysis to gain novel discoveries. To demonstrate its utility and generalizability, we applied this pipeline to two use cases: a murine cardiac temporal proteomics data set and a human plasma temporal proteomics data set, both aimed at examining protein turnover rates. This DMI pipeline significantly enhanced the detection of protein turnover rate in both data sets, and furthermore, the imputed data sets captured new representation of proteins, leading to an augmented view of biological pathways, protein complex dynamics, as well as biomarker-disease associations. Importantly, DMI exhibited superior performance in benchmark data sets compared to single imputation methods (DSI). In summary, we have demonstrated that this DMI pipeline is effective at overcoming challenges introduced by missing values in temporal proteome dynamics studies.

During hydrogen refueling, the data values determining the state of charge (SoC) of a vehicle can be missing due to internal and external factors. This causes inaccurate SoC estimation, resulting in oversupply or undersupply. To overcome this issue, an attention-based hydrogen refueling imputation (AHRI) model, which restores missing values, is proposed in this paper. In particular, considering that data variables can vary depending on the environmental conditions and equipment in a hydrogen refueling station (HRS), we use the attention mechanism. It determines the primary features, which improves the predictive performance and helps adapt to new conditions. Using the observed data during hydrogen refueling, we train the proposed AHRI model and verify its efficacy. Experimental results show that the proposed AHRI model outperforms existing imputation models significantly. Here, AHRI achieves 0.95 and 0.82 in terms of R 2 when 20% and 40% of the values are missing, respectively. These results indicate that the proposed model can be used to solve the data missing problems in HSRs. [ABSTRACT FROM AUTHOR]

The completeness of precipitation observation data is a crucial foundation for hydrological simulation, water resource analysis, and environmental assessment. Traditional data imputation methods suffer from poor adaptability, lack of precision, and limited model diversity. Rapid and accurate imputation using available data is a key challenge in precipitation monitoring. This study selected precipitation data from the Jiaojiang River basin in the southeastern Zhejiang Province of China from 1991 to 2020. The data were categorized based on various missing rates and scenarios, namely MCR (Missing Completely Random), MR (Missing Random), and MNR (Missing Not Random). Imputation of precipitation data was conducted using three types of Artificial Intelligence (AI) methods (Backpropagation Neural Network (BPNN), Random Forest (RF), and Support Vector Regression (SVR)), along with a novel Multiple Linear Regression (MLR) imputation method built upon these algorithms. The results indicate that the constructed MLR imputation method achieves an average Pearson's correlation coefficient (PCC) of 0.9455, an average Nash–Sutcliffe Efficiency (NSE) of 0.8329, and an average Percent Bias (Pbias) of 10.5043% across different missing rates. MLR simulation results in higher NSE and lower Pbias than the other three single AI models, thus effectively improving the estimation performance. The proposed methods in this study can be applied to other river basins to improve the quality of precipitation data and support water resource management. [ABSTRACT FROM AUTHOR]

The timely and accurate prediction of maize (Zea mays L.) yields prior to harvest is critical for food security and agricultural policy development. Currently, many researchers are using machine learning and deep learning to predict maize yields in specific regions with high accuracy. However, existing methods typically have two limitations. One is that they ignore the extensive correlation in maize planting data, such as the association of maize yields between adjacent planting locations and the combined effect of meteorological features and maize traits on maize yields. The other issue is that the performance of existing models may suffer significantly when some data in maize planting records is missing, or the samples are unbalanced. Therefore, this paper proposes an end-to-end bipartite graph neural network-based model for trait data imputation and yield prediction. The maize planting data is initially converted to a bipartite graph data structure. Then, a yield prediction model based on a bipartite graph neural network is developed to impute missing trait data and predict maize yield. This model can mine correlations between different samples of data, correlations between different meteorological features and traits, and correlations between different traits. Finally, to address the issue of unbalanced sample size at each planting location, we propose a loss function based on the gradient balancing mechanism that effectively reduces the impact of data imbalance on the prediction model. When compared to other data imputation and prediction models, our method achieves the best yield prediction result even when missing data is not pre-processed. [ABSTRACT FROM AUTHOR]

Experimental blood flow measurement techniques are invaluable for a better understanding of cardiovascular disease formation, progression, and treatment. One of the emerging methods is time‐resolved three‐dimensional phase‐contrast magnetic resonance imaging (4D flow MRI), which enables noninvasive time‐dependent velocity measurements within large vessels. However, several limitations hinder the usability of 4D flow MRI and other experimental methods for quantitative hemodynamics analysis. These mainly include measurement noise, corrupt or missing data, low spatiotemporal resolution, and other artifacts. Traditional filtering is routinely applied for denoising experimental blood flow data without any detailed discussion on why it is preferred over other methods. In this study, filtering is compared to different singular value decomposition (SVD)‐based machine learning and autoencoder‐type deep learning methods for denoising and filling in missing data (imputation). An artificially corrupted and voxelized computational fluid dynamics (CFD) simulation as well as in vitro 4D flow MRI data are used to test the methods. SVD‐based algorithms achieve excellent results for the idealized case but severely struggle when applied to in vitro data. The autoencoders are shown to be versatile and applicable to all investigated cases. For denoising, the in vitro 4D flow MRI data, the denoising autoencoder (DAE), and the Noise2Noise (N2N) autoencoder produced better reconstructions than filtering both qualitatively and quantitatively. Deep learning methods such as N2N can result in noise‐free velocity fields even though they did not use clean data during training. This work presents one of the first comprehensive assessments and comparisons of various classical and modern machine‐learning methods for enhancing corrupt cardiovascular flow data in diseased arteries for both synthetic and experimental test cases. [ABSTRACT FROM AUTHOR]

Over the past decade, the prevalence of inflammatory bowel disease (IBD) has significantly increased, making early detection crucial for improving patient survival rates. Medical research suggests that changes in the human gut microbiome are closely linked to IBD onset, playing a critical role in its prediction. However, the current gut microbiome data often exhibit missing values and high dimensionality, posing challenges to the accuracy of predictive algorithms. To address these issues, we proposed the comprehensive data optimization and risk prediction framework (CDORPF), an ensemble learning framework designed to predict IBD risk based on the human gut microbiome, aiding early diagnosis. The framework comprised two main components: data optimization and risk prediction. The data optimization module first employed triple optimization imputation (TOI) to impute missing data while preserving the biological characteristics of the microbiome. It then utilized importance-weighted variational autoencoder (IWVAE) to reduce redundant information from the high-dimensional microbiome data. This process resulted in a complete, low-dimensional representation of the data, laying the foundation for improved algorithm efficiency and accuracy. In the risk prediction module, the optimized data was classified using a random forest (RF) model, and hyperparameters were globally optimized using improved aquila optimizer (IAO), which incorporated multiple strategies. Experimental results on IBD-related gut microbiome datasets showed that the proposed framework achieved classification accuracy, recall, and F1 scores exceeding 0.9, outperforming comparison models and serving as a valuable tool for predicting IBD onset risk. [ABSTRACT FROM AUTHOR]

Metal powder contributes to the environmental burdens of additive manufacturing (AM) substantially. Current life cycle assessments (LCAs) of metal powders present considerable variations of lifecycle environmental inventory due to process divergence, spatial heterogeneity, or temporal fluctuation. Most importantly, the amounts of LCA studies on metal powder are limited and primarily confined to partial material types. To this end, based on the data surveyed from a metal powder supplier, this study conducted an LCA of titanium and nickel alloy produced by electrode-inducted and vacuum-inducted melting gas atomization, respectively. Given that energy consumption dominates the environmental burden of powder production and is influenced by metal materials' physical properties, we proposed a Bayesian stochastic Kriging model to estimate the energy consumption during the gas atomization process. This model considered the inherent uncertainties of training data and adaptively updated the parameters of interest when new environmental data on gas atomization were available. With the predicted energy use information of specific powder, the corresponding lifecycle environmental impacts can be further autonomously estimated in conjunction with the other surveyed powder production stages. Results indicated the environmental impact of titanium alloy powder is slightly higher than that of nickel alloy powder and their lifecycle carbon emissions are around 20 kg CO2 equivalency. The proposed Bayesian stochastic Kriging model showed more accurate predictions of energy consumption compared with conventional Kriging and stochastic Kriging models. This study enables data imputation of energy consumption during gas atomization given the physical properties and producing technique of powder materials. [ABSTRACT FROM AUTHOR]

The timely and accurate prediction of maize (Zea mays L.) yields prior to harvest is critical for food security and agricultural policy development. Currently, many researchers are using machine learning and deep learning to predict maize yields in specific regions with high accuracy. However, existing methods typically have two limitations. One is that they ignore the extensive correlation in maize planting data, such as the association of maize yields between adjacent planting locations and the combined effect of meteorological features and maize traits on maize yields. The other issue is that the performance of existing models may suffer significantly when some data in maize planting records is missing, or the samples are unbalanced. Therefore, this paper proposes an end-to-end bipartite graph neural network-based model for trait data imputation and yield prediction. The maize planting data is initially converted to a bipartite graph data structure. Then, a yield prediction model based on a bipartite graph neural network is developed to impute missing trait data and predict maize yield. This model can mine correlations between different samples of data, correlations between different meteorological features and traits, and correlations between different traits. Finally, to address the issue of unbalanced sample size at each planting location, we propose a loss function based on the gradient balancing mechanism that effectively reduces the impact of data imbalance on the prediction model. When compared to other data imputation and prediction models, our method achieves the best yield prediction result even when missing data is not pre-processed. [ABSTRACT FROM AUTHOR]

Missing pixel imputation presents a critical challenge in image processing and computer vision, particularly in applications such as image restoration and inpainting. The primary objective of this paper is to accurately estimate and reconstruct missing pixel values to restore complete visual information. This paper introduces a novel model called the Enhanced Connected Pixel Identity GAN with Neutrosophic (ECP-IGANN), which is designed to address two fundamental issues inherent in existing GAN architectures for missing pixel generation: (1) mode collapse, which leads to a lack of diversity in generated pixels, and (2) the preservation of pixel integrity within the reconstructed images. ECP-IGANN incorporates two key innovations to improve missing pixel imputation. First, an identity block is integrated into the generation process to facilitate the retention of existing pixel values and ensure consistency. Second, the model calculates the values of the 8-connected neighbouring pixels around each missing pixel, thereby enhancing the coherence and integrity of the imputed pixels. The efficacy of ECP-IGANN was rigorously evaluated through extensive experimentation across five diverse datasets: BigGAN-ImageNet, the 2024 Medical Imaging Challenge Dataset, the Autonomous Vehicles Dataset, the 2024 Satellite Imagery Dataset, and the Fashion and Apparel Dataset 2024. These experiments assessed the model's performance in terms of diversity, pixel imputation accuracy, and mode collapse mitigation, with results demonstrating significant improvements in the Inception Score (IS) and Fréchet Inception Distance (FID). ECP-IGANN markedly enhanced image segmentation performance in the validation phase across all datasets. Key metrics, such as Dice Score, Accuracy, Precision, and Recall, were improved substantially for various segmentation models, including Spatial Attention U-Net, Dense U-Net, and Residual Attention U-Net. For example, in the 2024 Medical Imaging Challenge Dataset, the Residual Attention U-Net's Dice Score increased from 0.84 to 0.90, while accuracy improved from 0.88 to 0.93 following the application of ECP-IGANN. Similar performance enhancements were observed with the other datasets, highlighting the model's robust generalizability across diverse imaging domains. [ABSTRACT FROM AUTHOR]

Missing data in tabular dataset is a common issue as the performance of downstream tasks usually depends on the completeness of the training dataset. Previous missing data imputation methods focus on numeric and categorical columns, but we propose a novel end-to-end approach called Table Transformers for Imputing Textual Attributes (TTITA) based on the transformer to impute unstructured textual columns using other columns in the table. We conduct extensive experiments on three datasets, and our approach shows competitive performance outperforming baseline models such as recurrent neural networks and Llama2. The performance improvement is more significant when the target sequence has a longer length. Additionally, we incorporate multi-task learning to simultaneously impute for heterogeneous columns, boosting the performance for text imputation. We also qualitatively compare with ChatGPT for realistic applications. • Proposed TTITA to impute text attributes given other heterogeneous tabular columns. • Encoded inputs into a context vector for cross-attention in the transformer decoder. • Outperformed baseline models including the GRU and Llama2 on real-world datasets. • Incorporated multi-task learning for multi-column imputation and boosting performance. • Prepared the software as an open-source package for custom applications. [ABSTRACT FROM AUTHOR]

This study presents an improved data-centric approach to short-term water demand forecasting using univariate time series from water reservoir levels. The dataset comprises water level recordings from 21 reservoirs in Eastern Thessaloniki collected over 15 months via a SCADA system provided by the water company EYATH S.A. The methodology involves data preprocessing, anomaly detection, data imputation, and the application of predictive models. Techniques such as the Interquartile Range method and moving standard deviation are employed to identify and handle anomalies. Missing values are imputed using LSTM networks optimized through the Optuna framework. This study emphasizes a data-centric approach in deep learning, focusing on improving data quality before model application, which has proven to enhance prediction accuracy. This strategy is crucial, especially in regions where reservoirs are the primary water source, and demand distribution cannot be solely determined by flow meter readings. LSTM, Random Forest Regressor, ARIMA, and SARIMA models are utilized to extract and analyze water level trends, enabling more accurate future water demand predictions. Results indicate that combining deep learning techniques with traditional statistical models significantly improves the accuracy and reliability of water demand predictions, providing a robust framework for optimizing water resource management. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel methodology for estimating missing data in energy consumption datasets. Conventional data imputation methods are not suitable for these datasets, because they are time series with special characteristics and because, for some applications, it is quite important to preserve the shape of the daily energy profile. Our answer to this need is the use of autoencoders. First, we split the problem into two subproblems: how to estimate the total amount of daily energy, and how to estimate the shape of the daily energy profile. We encode the shape as a new feature that can be modeled and predicted using autoencoders. In this way, the problem of imputation of profile data are reduced to two relatively simple problems on which conventional methods can be applied. However, the two predictions are related, so special care should be taken when reconstructing the profile. We show that, as a result, our data imputation methodology produces plausible profiles where other methods fail. We tested it on a highly corrupted dataset, outperforming conventional methods by a factor of 3.7. [ABSTRACT FROM AUTHOR]

Environmental parameter data collected by sensors for monitoring the environment of agricultural facility operations are usually incomplete due to external environmental disturbances and device failures. And the missing of collected data is completely at random. In practice, missing data could create biased estimations and make multivariate time series predictions of environmental parameters difficult, leading to imprecise environmental control. A multivariate time series imputation model based on generative adversarial networks and multi-head attention (ATTN-GAN) is proposed in this work to reducing the negative consequence of missing data. ATTN-GAN can capture the temporal and spatial correlation of time series, and has a good capacity to learn data distribution. In the downstream experiments, we used ATTN-GAN and baseline models for data imputation, and predicted the imputed data, respectively. For the imputation of missing data, over the 20%, 50% and 80% missing rate, ATTN-GAN had the lowest RMSE, 0.1593, 0.2012 and 0.2688 respectively. For water temperature prediction, data processed with ATTN-GAN over MLP, LSTM, DA-RNN prediction methods had the lowest MSE, 0.6816, 0.8375 and 0.3736 respectively. Those results revealed that ATTN-GAN outperformed all baseline models in terms of data imputation accuracy. The data processed by ATTN-GAN is the best for time series prediction.

Abstract Missing pixel imputation presents a critical challenge in image processing and computer vision, particularly in applications such as image restoration and inpainting. The primary objective of this paper is to accurately estimate and reconstruct missing pixel values to restore complete visual information. This paper introduces a novel model called the Enhanced Connected Pixel Identity GAN with Neutrosophic (ECP-IGANN), which is designed to address two fundamental issues inherent in existing GAN architectures for missing pixel generation: (1) mode collapse, which leads to a lack of diversity in generated pixels, and (2) the preservation of pixel integrity within the reconstructed images. ECP-IGANN incorporates two key innovations to improve missing pixel imputation. First, an identity block is integrated into the generation process to facilitate the retention of existing pixel values and ensure consistency. Second, the model calculates the values of the 8-connected neighbouring pixels around each missing pixel, thereby enhancing the coherence and integrity of the imputed pixels. The efficacy of ECP-IGANN was rigorously evaluated through extensive experimentation across five diverse datasets: BigGAN-ImageNet, the 2024 Medical Imaging Challenge Dataset, the Autonomous Vehicles Dataset, the 2024 Satellite Imagery Dataset, and the Fashion and Apparel Dataset 2024. These experiments assessed the model’s performance in terms of diversity, pixel imputation accuracy, and mode collapse mitigation, with results demonstrating significant improvements in the Inception Score (IS) and Fréchet Inception Distance (FID). ECP-IGANN markedly enhanced image segmentation performance in the validation phase across all datasets. Key metrics, such as Dice Score, Accuracy, Precision, and Recall, were improved substantially for various segmentation models, including Spatial Attention U-Net, Dense U-Net, and Residual Attention U-Net. For example, in the 2024 Medical Imaging Challenge Dataset, the Residual Attention U-Net’s Dice Score increased from 0.84 to 0.90, while accuracy improved from 0.88 to 0.93 following the application of ECP-IGANN. Similar performance enhancements were observed with the other datasets, highlighting the model’s robust generalizability across diverse imaging domains.

Abstract Metal powder contributes to the environmental burdens of additive manufacturing (AM) substantially. Current life cycle assessments (LCAs) of metal powders present considerable variations of lifecycle environmental inventory due to process divergence, spatial heterogeneity, or temporal fluctuation. Most importantly, the amounts of LCA studies on metal powder are limited and primarily confined to partial material types. To this end, based on the data surveyed from a metal powder supplier, this study conducted an LCA of titanium and nickel alloy produced by electrode-inducted and vacuum-inducted melting gas atomization, respectively. Given that energy consumption dominates the environmental burden of powder production and is influenced by metal materials’ physical properties, we proposed a Bayesian stochastic Kriging model to estimate the energy consumption during the gas atomization process. This model considered the inherent uncertainties of training data and adaptively updated the parameters of interest when new environmental data on gas atomization were available. With the predicted energy use information of specific powder, the corresponding lifecycle environmental impacts can be further autonomously estimated in conjunction with the other surveyed powder production stages. Results indicated the environmental impact of titanium alloy powder is slightly higher than that of nickel alloy powder and their lifecycle carbon emissions are around 20 kg CO2 equivalency. The proposed Bayesian stochastic Kriging model showed more accurate predictions of energy consumption compared with conventional Kriging and stochastic Kriging models. This study enables data imputation of energy consumption during gas atomization given the physical properties and producing technique of powder materials.

A neural network‐based model is proposed to estimate missing values of incomplete datasets to augment their size. An autoassociative neural network (AANN), for which the output vector is identical to the input vector, was built for a styrene production process dataset. The proposed model was used to investigate the ability of an AANN to estimate one to three missing variables, evaluating the impact of the size of the datasets used and the level of correlation of the missing values with other process variables. Results show that the proposed AANN model can predict the process data even when the number of records used is relatively small. Moreover, the AANN method is suitable for estimating missing variables with an accuracy that depends on the correlation coefficient of the missing values with other process variables, keeping acceptable estimation for weakly‐correlated variables. Moreover, the model was tested on noisy data, and it is shown that the model trained on noisy data can also predict missing values in an acceptable estimation range. [ABSTRACT FROM AUTHOR]

Accurate load forecasting, especially in the short term, is crucial for the safe and stable operation of power systems and their market participants. However, as modern power systems become increasingly complex, the challenges of short-term load forecasting are also intensifying. To address this challenge, data-driven deep learning techniques and load aggregation technologies have gradually been introduced into the field of load forecasting. However, data quality issues persist due to various factors such as sensor failures, unstable communication, and susceptibility to network attacks, leading to data gaps. Furthermore, in the domain of aggregated load forecasting, considering the potential interactions among aggregated loads can help market participants engage in cross-market transactions. However, aggregated loads often lack clear geographical locations, making it difficult to predefine graph structures. To address the issue of data quality, this study proposes a model named adversarial graph convolutional imputation network (AGCIN), combined with local and global correlations for imputation. To tackle the problem of the difficulty in predefining graph structures for aggregated loads, this study proposes a learnable adjacency matrix, which generates an adaptive adjacency matrix based on the relationships between different sequences without the need for geographical information. The experimental results demonstrate that the proposed imputation method outperforms other imputation methods in scenarios with random and continuous missing data. Additionally, the prediction accuracy of the proposed method exceeds that of several baseline methods, affirming the effectiveness of our approach in imputation and prediction, ultimately enhancing the accuracy of aggregated load forecasting. [ABSTRACT FROM AUTHOR]

For modern Intelligent Transportation System (ITS), data missing during traffic raster acquisition can be inevitable because of the loop detector malfunction or signal interference. Nevertheless, missing data imputation is meaningful due to the periodic spatio-temporal characteristics and individual randomness of traffic raster data. In this paper, traffic raster data collected from all spatial regions at each time interval are considered as a multiple channel image. Accordingly, the traffic raster data over a period of time can be regarded as video, on which an unsupervised generative neural network called MSST-VAE (Multiple Streams Spatial Temporal-VAE) is proposed for traffic raster data imputation, and this model can even robustly performs at varied missing rates while many other approaches fail to conduct. Two major innovations can be summarized in MSSTVAE: Firstly, it uses multiple periodic streams of Variational Auto-Encoders (VAEs) with Sylvester Normalizing Flows (SNFs), which shows strong generalization ability. Secondly, after the traffic raster data are transferred into videos, an ECB (Extraction-and-Calibration Block) consisting of dilated P3D gated convolution and multi-horizon attention mechanism is employed to learn global-local-granularity spatial features and long-short-term temporal features. Extensive experiments on three real traffic flow datasets validate that MSST-VAE outperforms other classical traffic imputation models with the least imputation error. [ABSTRACT FROM AUTHOR]

Purpose: Crime data analysis has gained significant interest due to its peculiarities. One key characteristic of property crimes is the uncertainty surrounding their exact temporal location, often limited to a time window. Methods: This study introduces a spatio-temporal logistic regression model that addresses the challenges posed by temporal uncertainty in crime data analysis. Inspired by the aoristic method, our Bayesian approach allows for the inclusion of temporal uncertainty in the model. Results: To demonstrate the effectiveness of our proposed model, we apply it to both simulated datasets and a dataset of residential burglaries recorded in Valencia, Spain. We compare our proposal with a complete cases model, which excludes temporally-uncertain events, and also with alternative models that rely on imputation procedures. Our model exhibits superior performance in terms of recovering the true underlying crime risk. Conclusions: The proposed modeling framework effectively handles interval-censored temporal observations while incorporating covariate and space–time effects. This flexible model can be implemented to analyze crime data with uncertainty in temporal locations, providing valuable insights for crime prevention and law enforcement strategies. [ABSTRACT FROM AUTHOR]

Forecasting the future levels of air pollution provides valuable information that holds importance for the general public, vulnerable populations, and policymakers. High-quality data are essential for precise and reliable forecasts and investigations of air pollution. Missing observations arise when the sensors utilized for assessing air quality parameters experience malfunctions, which result in erroneous measurements or gaps in the dataset and hinder the data quality. This research paper presents a novel approach for imputing missing values in air quality data in a univariate approach. The algorithm employs the random forest (RF) algorithm to impute missing observations in a bi-directional (forward and reverse in time) manner for air quality (particulate matter less than 2.5 μm (PM2.5)) data from the Republic of Serbia. The algorithm was evaluated against simple methods, such as the mean and median imputation methods, for missing observations over durations of 24, 48, and 72 h. The results indicate that our algorithm yielded comparable error rates to the median imputation method for all periods when imputing the PM2.5 data. Ultimately, the algorithm's higher computational complexity proved itself as not justified considering the minimal error decrease it achieved compared with the simpler methods. However, for future improvement, additional research is needed, such as utilizing low-code machine learning libraries and time-series forecasting techniques. [ABSTRACT FROM AUTHOR]

Missing data imputation is a key topic in healthcare that covers the issues and strategies involved in dealing with partial data in medical records, clinical trials, and health surveys. Data in healthcare might be missing for a variety of reasons, including non-response in surveys, data entry problems, or unrecorded information during therapeutic appointments. This paper introduces a novel approach to impute missing data utilizing a hybrid model that integrates denoising autoencoders with generative adversarial networks (GANs). We begin by highlighting the prevalence of missing data in health care datasets and the potential impact on analytical outcomes. The proposed methodology leverages the denoising autoencoder's ability to reconstruct data from noisy inputs, coupled with the GAN's proficiency in generating synthetic data that is indistinguishable from real data. By combining these two neural network architectures, our model demonstrates an enhanced capability to predict and fill in missing data points effectively. To validate our approach, we conducted experiments on several large-scale health care datasets with varying degrees of artificially introduced missingness. The performance of our model was benchmarked against traditional imputation methods such as mean imputation and k-nearest neighbors, as well as against standalone denoising autoencoders and GANs. Our results indicate a significant improvement in imputation accuracy, as measured by root mean square error (RMSE) and mean absolute error (MAE), confirming the efficacy of the hybrid model in handling missing data in a robust manner. [ABSTRACT FROM AUTHOR]

As smart grid advance, Power Load Forecasting (PLF) has become a research hotspot. As the foundation of the forecasting model, the Power Load Monitoring (PLM) data takes on great importance due to its completeness, reliability and accuracy. However, monitoring equipment failures, transmission channel congestion and anomalies result in missing PLM data, which directly affects the performance of the PLF model. To address this issue, this paper proposes an L1-and-L2-Regularized Nonnegative Tensor Factorization (LNTF) model to impute PLM missing data. Its main idea is threefold: (1) combining L1 and L2 norms to achieve effective feature extraction and improve the model's robustness; (2) incorporating two temporal-dependent linear biases to describe the fluctuations of PLM data; (3) adding nonnegative constraints to precisely define the nonnegativity of PLM data. Extensive empirical studies on two publicly real-world PLM datasets with 1,569,491 and 413,357 known entries and missing rates of 93.35% and 96.75% demonstrate that the proposed LNTF improves 14.04%, 59.31%, and 71.43% on average over the state-of-the-art imputation models in terms of imputation error, convergence rounds, and time cos, respectively. Its high computational efficiency and low imputation error make practical sense for PLM data imputation. [ABSTRACT FROM AUTHOR]

Data missing is a ubiquitous problem in real-world systems that adversely affects the performance of machine learning algorithms. Although many useful imputation methods are available to address this issue, they often fail to consider the information provided by both features and labels. As a result, the performance of these methods might be constrained. Furthermore, feature selection as a data quality improvement technique has been widely used and has demonstrated its efficiency. To overcome the limitation of imputation methods, we propose a novel algorithm that combines data imputation and feature selection to tackle classification problems for mixed data. Based on the mean and standard deviation of quantitative features and the selecting probabilities of unique values of categorical features, our algorithm constructs different imputation models for quantitative and categorical features. Particle swarm optimization is used to optimize the parameters of the imputation models and select feature subsets simultaneously. Additionally, we introduce a legacy learning mechanism to enhance the optimization capability of our method. To evaluate the performance of the proposed method, seven algorithms and twelve datasets are used for comparison. The results show that our algorithm outperforms other algorithms in terms of accuracy and F 1 score and has reasonable time overhead. [ABSTRACT FROM AUTHOR]

Numerous clinical trials have established that various biologic and oral small-molecule therapies are efficacious in patients with psoriasis. However, as there are limited head-to-head trials, healthcare providers may compare results across multiple trials when providing treatment recommendations. Direct comparisons among agents are challenging because psoriasis trials differ in terms of study design, patient population, and data analysis methodologies. Longterm clinical trials present additional challenges because the number of patients enrolled generally declines over time. The missing patient data that might occur, coupled with the speciffc approach used to substitute or impute that missing data, might introduce bias and skew efficacy results. In this review, we discuss how variations in study design and analytical methodologies affect effcacy outcomes in clinical trials. We also review published trials of biologic and oral small-molecule therapies for psoriasis to illustrate how issues related to missing data and choices in data imputation methodologies can affect the interpretation of effcacy outcomes. Imputation methodologies discussed include nonresponder imputation, modified nonresponder imputation, treatment failure rules, last observation carried forward, modified baseline observation carried forward, and multiple imputation. This review provides a foundation for the healthcare provider's critical evaluation of the psoriasis literature and emphasizes the importance of considering the level of evidence provided in a clinical trial when making treatment decisions. [ABSTRACT FROM AUTHOR]

Clinical rating scales can be used to assess the severity of bipolar disorder; however, their use involves clinician–patient interactions, which is labor-intensive. Therefore, this study proposes a digital-phenotyping-based system that provides clinical ratings of bipolar disorder severity using global positioning system, self-scale, daily mood, user emotion, sleep time, and multimedia data; these ratings are given on Hamilton Depression Rating Scale (HAM-D) and Young Mania Rating Scale (YMRS). A K-nearest-neighbor-based imputation method was used to handle missing data. In this method, missing data points are filled in with the multiple correlations between different features. Furthermore, the Least Absolute Shrinkage and Selection Operator (Lasso)-regression-based multilayer perceptron (Lasso-MLP) method was adopted to predict the total and factor scores on the HAM-D and YMRS. Five-fold cross-validation were used in evaluation experiments. When the designed data imputation method was used with Lasso-MLP, the mean square errors of the total score and average factor score on HAM-D (the YMRS) were 0.56 (0.38) and 1.88 (0.98), respectively, which were smaller than the corresponding values obtained through Lasso regression (by 0.12 and 0.05, respectively, for HAM-D and by 0.12 and 0.10, respectively, for the YMRS). The experimental results also indicated that the models trained with the imputed data outperformed those trained without imputed data. Thus, the developed approaches can eliminate the missing data problem and provide accurate clinical ratings. [ABSTRACT FROM AUTHOR]

Missing values are a common issue in quantitative social researches. One of the ways to handle missing data is by data imputation. This article outlines the challenges of traditional data imputa- tion methods, which often introduce biases, and presents an advanced approach that features inte- gration of paradata auxiliary information collected during surveys into the imputation process, using the European Social Survey (ESS) as its dataset. It is proposed that the usage of paradata could enhance predictive models used for imputation. It discusses the practical applications of data imputation, particularly through the lens of sensitive topics such as LGBT issues in socially conser- vative countries, where missingness could be heavily skewed due to social inacceptability of certain answers. To evaluate the effectiveness of the proposed approach towards imputation, the research employs the approach of using the 'ideal dataset', which is a subset of the original dataset with no missing vales, and then introduces artificial missing values that are not MCAR (Missing Completely at Random) to simulate the real case of missing data. Having artificial missingness allows for evaluation of the imputation procedure by comparing it with the original dataset. The study uses a novel approach towards creation of realistic missing data patterns through clustering based on response patterns. The research uses advanced statistical methods to handle missing data, and incorporates paradata from the survey process to improve the accuracy of predictive models. By comparing statistical metrics such as RMSE, MAE, and R-squared, the article evaluates the effectiveness of these methods in mimicking the original dataset's variability [ABSTRACT FROM AUTHOR]

With the wide adoption of building automation system, and the advancement of data, sensing, and machine learning techniques, data-driven fault detection and diagnostics (FDD) for building heating, ventilation, and air conditioning systems has gained increasing attention. In this paper, data-driven FDD is defined as those that are built or trained from data via machine learning or multivariate statistical analysis methods. Following this definition, this paper reviews and summarizes the literature on data-driven FDD from three aspects: process, systems studied (including the systems being investigated, the faults being identified, and the associated data sources), and evaluation metrics. A data-driven FDD process is further divided into the following steps: data collection, data cleansing, data preprocessing, baseline establishment, fault detection, fault diagnostics, and potential fault prognostics. Literature reported data-driven methods used in each step of an FDD process are firstly discussed. Applications of data-driven FDD in various HVAC systems/components and commonly used data source for FDD development are reviewed secondly, followed by a summary of typical metrics for evaluating FDD methods. Finally, this literature review concludes that despite the promising performance reported in the literature, data-driven FDD methods still face many challenges, such as real-building deployment, performance evaluation and benchmarking, scalability and transferability, interpretability, cyber security and data privacy, user experience, etc. Addressing these challenges is critical for a broad real-building adoption of data-driven FDD.

Data in the Internet of Things (IoT) domain may be missing due to connectivity errors, environmental extremes, sensor malfunctions, and human errors. Despite the many approaches for imputing missing values, the most significant difficulty in terms of imputation precision or compute complexity for larger missing sub-sequences in uni-variate series is still being explored. This work introduced IMD-MP (Imputation of Missing Data using Matrix Profile), a new technique that improves imputation accuracy for big data analysis in IoT applications based on spatial-temporal correlations using a novel distance metric Matrix Profile Distance (MPD). Our method preserves spatial correlation by grouping the sensors present in the network (using grouping algorithm-GA) to impute the missing data of the failed sensor node. After grouping, similar sensor nodes to the failed sensor node are identified using the Node Similarity Algorithm (NSF). From its similar sensor data, a certain number of sub-sequences that are most similar to the one preceding the failed node’s missing values are gathered. These sub-sequences heights are optimized to ensure temporal correlation in the imputed data. To find the optimal imputation sequence, the current research uses MPD and similarity scores. Numerical findings using sensor data from real-time environmental mon-itoring and Intel data sets demonstrate the algorithm’s effectiveness compared to other benchmarks.

Abstract Background Extracellular vesicle-derived (EV)-miRNAs have potential to serve as biomarkers for the diagnosis of various diseases. miRNA microarrays are widely used to quantify circulating EV-miRNA levels, and the preprocessing of miRNA microarray data is critical for analytical accuracy and reliability. Thus, although microarray data have been used in various studies, the effects of preprocessing have not been studied for Toray’s 3D-Gene chip, a widely used measurement method. We aimed to evaluate batch effect, missing value imputation accuracy, and the influence of preprocessing on measured values in 18 different preprocessing pipelines for EV-miRNA microarray data from two cohorts with amyotrophic lateral sclerosis using 3D-Gene technology. Results Eighteen different pipelines with different types and orders of missing value completion and normalization were used to preprocess the 3D-Gene microarray EV-miRNA data. Notable results were suppressed in the batch effects in all pipelines using the batch effect correction method ComBat. Furthermore, pipelines utilizing missForest for missing value imputation showed high agreement with measured values. In contrast, imputation using constant values for missing data exhibited low agreement. Conclusions This study highlights the importance of selecting the appropriate preprocessing strategy for EV-miRNA microarray data when using 3D-Gene technology. These findings emphasize the importance of validating preprocessing approaches, particularly in the context of batch effect correction and missing value imputation, for reliably analyzing data in biomarker discovery and disease research.

Abstract In modern healthcare, integrating Artificial Intelligence (AI) and Internet of Medical Things (IoMT) is highly beneficial and has made it possible to effectively control disease using networks of interconnected sensors worn by individuals. The purpose of this work is to develop an AI-IoMT framework for identifying several of chronic diseases form the patients’ medical record. For that, the Deep Auto-Optimized Collaborative Learning (DACL) Model, a brand-new AI-IoMT framework, has been developed for rapid diagnosis of chronic diseases like heart disease, diabetes, and stroke. Then, a Deep Auto-Encoder Model (DAEM) is used in the proposed framework to formulate the imputed and preprocessed data by determining the fields of characteristics or information that are lacking. To speed up classification training and testing, the Golden Flower Search (GFS) approach is then utilized to choose the best features from the imputed data. In addition, the cutting-edge Collaborative Bias Integrated GAN (ColBGaN) model has been created for precisely recognizing and classifying the types of chronic diseases from the medical records of patients. The loss function is optimally estimated during classification using the Water Drop Optimization (WDO) technique, reducing the classifier’s error rate. Using some of the well-known benchmarking datasets and performance measures, the proposed DACL’s effectiveness and efficiency in identifying diseases is evaluated and compared.

Autoencoders are the backbones of many imputation methods that aim to relieve the sparsity issue in single-cell RNA sequencing (scRNA-seq) data. The imputation performance of an autoencoder relies on both the neural network architecture and the hyperparameter choice. So far, literature in the single-cell field lacks a formal discussion on how to design the neural network and choose the hyperparameters. Here, we conducted an empirical study to answer this question. Our study used many real and simulated scRNA-seq datasets to examine the impacts of the neural network architecture, the activation function, and the regularization strategy on imputation accuracy and downstream analyses. Our results show that (i) deeper and narrower autoencoders generally lead to better imputation performance; (ii) the sigmoid and tanh activation functions consistently outperform other commonly used functions including ReLU; (iii) regularization improves the accuracy of imputation and downstream cell clustering and DE gene analyses. Notably, our results differ from common practices in the computer vision field regarding the activation function and the regularization strategy. Overall, our study offers practical guidance on how to optimize the autoencoder design for scRNA-seq data imputation.

Background: Extracellular vesicle-derived (EV)-miRNAs have potential to serve as biomarkers for the diagnosis of various diseases. miRNA microarrays are widely used to quantify circulating EV-miRNA levels, and the preprocessing of miRNA microarray data is critical for analytical accuracy and reliability. Thus, although microarray data have been used in various studies, the effects of preprocessing have not been studied for Toray's 3D-Gene chip, a widely used measurement method. We aimed to evaluate batch effect, missing value imputation accuracy, and the influence of preprocessing on measured values in 18 different preprocessing pipelines for EV-miRNA microarray data from two cohorts with amyotrophic lateral sclerosis using 3D-Gene technology. Results: Eighteen different pipelines with different types and orders of missing value completion and normalization were used to preprocess the 3D-Gene microarray EV-miRNA data. Notable results were suppressed in the batch effects in all pipelines using the batch effect correction method ComBat. Furthermore, pipelines utilizing missForest for missing value imputation showed high agreement with measured values. In contrast, imputation using constant values for missing data exhibited low agreement. Conclusions: This study highlights the importance of selecting the appropriate preprocessing strategy for EV-miRNA microarray data when using 3D-Gene technology. These findings emphasize the importance of validating preprocessing approaches, particularly in the context of batch effect correction and missing value imputation, for reliably analyzing data in biomarker discovery and disease research. [ABSTRACT FROM AUTHOR]

Smart grids commonly rely on analyzing sensor data to monitor power equipment. However, these sensor data can suffer varying levels of loss or corruption due to complex interferences, leading to a pressing need for precise missing value imputation in power equipment data. We propose a data imputation model for power equipment based on generative adversarial networks with an auxiliary encoder, named PEDI-GAN. In particular, the auxiliary encoder is designed and integrated into the GAN structure to optimize random vectors for the generator. Through data masking, we pinpoint missing data locations, enabling the generator to focus on generating accurate values for those points. Additionally, we address gradient disappearance and model collapse in GAN training by using the gradient penalty to redesign the loss function for PEDI-GAN. Experimental results demonstrate PEDI-GAN's superiority in accuracy and generalization compared to baseline methods, with notable reductions in mean absolute error and root mean square error by an average of 16.75% and 11.09%, respectively. [ABSTRACT FROM AUTHOR]

Accurate climatic data, especially precipitation measurements, play a critical role in various studies concerning the water cycle, particularly in modeling flood and drought risks. Unfortunately, these datasets often suffer from temporary gaps that are randomly dispersed over time. This study aims to assess the effectiveness of three imputation methods: KNN, MICE, and missForest, in impute missing values in climate series. The evaluation is conducted in two distinct rainfall regimes: the Moulouya basin and the Sous Massa basin. The performance analysis considers the percentage of missing data across the entire dataset. The imputed datasets are used to estimate annual precipitation, which are then subjected to statistical tests to identify potential trends and detect changepoints. The analysis focuses on the precipitation series within the Souss Massa watershed, encompassing 27 rainfall stations. Results indicate that data imputation has a highly positive impact on the study of rainfall series trends and change point detection. The study found that studying trends without data imputation could lead to questionable conclusions. The most significant breakpoints detected in the analyzed rainfall series were in the years 1988, 1991, 1997, 2007, and 2010. The decrease in precipitation at stations showing a downward trend varies between -60 mm and -137 mm using the MICE method, and between -40 mm and 186 mm using the missForest method. [ABSTRACT FROM AUTHOR]

In the relam of the Internet of Things (IoT), prevalence of missing data due to continuous data collection by smart devices necessitates the essential preliminary step of data imputation before engaging in information mining activities. IoT data exhibit robust interconnections in both spatial and temporal dimensions, surpassing the limitations of Euclidean space. Yet, prevailing machine learning and deep learning approaches often focus solely on temporal attributes or capture spatial features exclusively within a Euclidean framework. To address these challenges, this paper introduces a novel network named ST-Bi-LSTM-VAE (Spatio-Temporal Bidirectional Long Short-Term Memory based Variational Auto-Encoder). The architecture of ST-Bi-LSTM-VAE is primarily grounded in the Variational Auto-Encoder (VAE) framework. This innovative approach incorporates two distinct types of VAEs. The first type is dedicated to computing the adjacent matrix of the device network, a crucial input for the Graph Convolutional Network (GCN) essential in capturing intricate spatial relationships among devices. The second type of VAE is specifically tailored for data imputation, leveraging both global spatial and temporal dependencies. Empirical experiments conducted on diverse publicly available datasets substantiate the efficacy of ST-Bi-LSTM-VAE. The results obtained consistently demonstrate that proposed method surpasses baseline techniques in maintaining pattern, structure, and trend across datasets even at 50% missing gap for imputation task with 4.91% performance improvement in case of Intel Berkley Research Laboratory (IBRL) dataset and 3.5% on PRSA dataset. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) and, more broadly, activities of daily life recognition using wearable devices have the potential to transform a number of applications, including mobile healthcare, smart homes, and fitness monitoring. Recent approaches for HAR use multiple sensors on various locations on the body to achieve higher accuracy for complex activities. While multiple sensors increase the accuracy, they are also susceptible to reliability issues when one or more sensors are unable to provide data to the application due to sensor malfunction, user error, or energy limitations. Training multiple activity classifiers that use a subset of sensors is not desirable, since it may lead to reduced accuracy for applications. To handle these limitations, we propose a novel generative approach that recovers the missing data of sensors using data available from other sensors. The recovered data are then used to seamlessly classify activities. Experiments using three publicly available activity datasets show that with data missing from one sensor, the proposed approach achieves accuracy that is within 10% of the accuracy with no missing data. Moreover, implementation on a wearable device prototype shows that the proposed approach takes about 1.5 ms for recovering data in the w-HAR dataset, which results in an energy consumption of 606 μJ. The low-energy consumption ensures that SensorGAN is suitable for effectively recovering data in tinyML applications on energy-constrained devices. [ABSTRACT FROM AUTHOR]