Search

With the proliferation of the Internet, network complexities for both commercial and state organizations have significantly increased, leading to more sophisticated and harder-to-detect network attacks. This evolution poses substantial challenges for intrusion detection systems, threatening the cybersecurity of organizations and national infrastructure alike. Although numerous deep learning techniques such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs) have been applied to detect various network attacks, they face limitations due to the lack of standardized input data, affecting model accuracy and performance. This paper proposes a novel preprocessing method for flow data from network intrusion detection systems (NIDSs), enhancing the efficacy of a graph neural network model in malicious flow detection. Our approach initializes graph nodes with data derived from flow features and constructs graph edges through the analysis of IP relationships within the system. Additionally, we propose a new graph model based on the combination of the graph neural network (GCN) model and SAGEConv, a variant of the GraphSAGE model. The proposed model leverages the strengths while addressing the limitations encountered by the previous models. Evaluations on two IDS datasets, CICIDS-2017 and UNSW-NB15, demonstrate that our model outperforms existing methods, offering a significant advancement in the detection of network threats. This work not only addresses a critical gap in the standardization of input data for deep learning models in cybersecurity but also proposes a scalable solution for improving the intrusion detection accuracy. [ABSTRACT FROM AUTHOR]

With the development of smart grids and new power systems, the combination of non-intrusive load identification technology and smart home technology can provide users with the operating conditions of home appliances and equipment, thus reducing home energy loss and improving users' ability to demand a response. This paper proposes a non-intrusive load decomposition model with a parallel multiscale attention mechanism (PMAM). The model can extract both local and global feature information and fuse it through a parallel multiscale network. This improves the attention mechanism's ability to capture feature information over long time periods. To validate the model's decomposition ability, we combined the PMAM model with four benchmark models: the Long Short-Term Memory (LSTM) recurrent neural network model, the Time Pooling-based Load Disaggregation Model (TPNILM), the Extreme Learning Machine (ELM), and the Load Disaggregation Model without Parallel Multi-scalar Attention Mechanisms (UNPMAM). The model was trained on the publicly available UK-DALE dataset and tested. The models' test results were quantitatively evaluated using a confusion matrix. This involved calculating the F1 score of the load decomposition. A higher F1 score indicates better model decomposition performance. The results indicate that the PMAM model proposed in this paper maintains an F1 score above 0.9 for the decomposition of three types of electrical equipment under the same household user, which is 3% higher than that of the other benchmark models on average. In the cross-household test, the PMAM also demonstrated a better decomposition ability, with the F1 score maintained above 0.85, and the mean absolute error (MAE) decreased by 5.3% on average compared with that of the UNPMAM. [ABSTRACT FROM AUTHOR]

Understanding road conditions is essential for implementing effective road safety measures and driving solutions. Road situations encompass the day-to-day conditions of roads, including the presence of vehicles and pedestrians. Surveillance cameras strategically placed along streets have been instrumental in monitoring road situations and providing valuable information on pedestrians, moving vehicles, and objects within road environments. However, these video data and information are stored in large volumes, making analysis tedious and time-consuming. Deep learning models are increasingly utilized to monitor vehicles and identify and evaluate road and driving comfort situations. However, the current neural network model requires the recognition of situations using time-series video data. In this paper, we introduced a multi-directional detection model for road situations to uphold high accuracy. Deep learning methods often integrate long short-term memory (LSTM) into long-term recurrent network architectures. This approach effectively combines recurrent neural networks to capture temporal dependencies and convolutional neural networks (CNNs) to extract features from extensive video data. In our proposed method, we form a multi-directional long-term recurrent convolutional network approach with two groups equipped with CNN and two layers of LSTM. Additionally, we compare road situation recognition using convolutional neural networks, long short-term networks, and long-term recurrent convolutional networks. The paper presents a method for detecting and recognizing multi-directional road contexts using a modified LRCN. After balancing the dataset through data augmentation, the number of video files increased, resulting in our model achieving 91% accuracy, a significant improvement from the original dataset. [ABSTRACT FROM AUTHOR]

Short-term load forecasting is a challenging research problem and has a tremendous impact on electricity generation, transmission, and distribution. A robust forecasting algorithm can help power system operators to better tackle the ever-changing electric power demand. This paper presents a novel deep neural network for short-term electric load forecasting for the St. John's campus of Memorial University of Newfoundland (MUN). The electric load data are obtained from the Memorial University of Newfoundland and combined with metrological data from St. John's. This dataset is used to formulate a multivariate time-series forecasting problem. A novel deep learning algorithm is presented, consisting of a 1D Convolutional Neural Network, which is followed by an encoder–decoder-based network with attention. The input used for this model is the electric load consumption and metrological data, while the output is the hourly prediction of the next day. The model is compared with Gated Recurrent Unit (GRU) and Long Short Term Memory (LSTM)-based Recurrent Neural Network. A CNN-based encoder–decoder model without attention is also tested. The proposed model shows a lower mean absolute error (MAE), mean squared error (MSE), mean absolute percentage error (MAPE), and higher R2 score. These evaluation metrics show an improved performance compared to GRU and LSTM-based RNNs as well as the CNN encoder–decoder model without attention. [ABSTRACT FROM AUTHOR]

The supervision of security risk level of carbofuran pesticide residues can guarantee the food quality and security of residents effectively. In order to predict the potential key risk vegetables and regions, this paper constructs a security risk assessment model, combined with the k-means++ algorithm, to establish the risk security level. Then the evaluation index value of the security risk model is predicted to determine the security risk level based on the deep learning model. The model consists of a convolutional neural network (CNN) and a long short-term memory network (LSTM) optimized by an arithmetic optimization algorithm (AOA), namely, CNN-AOA-LSTM. In this paper, a comparative experiment is conducted on a small sample data set of independently constructed security risk assessment indicators. Experimental results show that the accuracy of the CNN-AOA-LSTM prediction model based on attention mechanism is 6.12% to 18.99% higher than several commonly used deep neural network models (gated recurrent unit, LSTM, and recurrent neural networks). The prediction model proposed in this paper provides scientific reference to establish the priority order of supervision, and provides forward-looking supervision for the government. [ABSTRACT FROM AUTHOR]

This paper presents a comprehensive approach to predicting short-term (for the upcoming 2 weeks) changes in estuarine dissolved oxygen concentrations via machine learning models that integrate historical water sampling, historical and upcoming 2-week meteorological data, and river discharge and discharge metrics. Dissolved oxygen is a critical indicator of ecosystem health, and this approach is implemented for the Neuse River Estuary, North Carolina, U.S.A., which has a long history of hypoxia-related habitat degradation. Through meticulous data preprocessing and feature selection, this research evaluates the predictions of dissolved oxygen concentrations by comparing a recurrent neural network with four other models, including a Multilayer Perceptron, Long Short-Term Memory, Gradient Boosting, and AutoKeras, through sensitivity experiments. The input predictors to our prediction models include water temperature, turbidity, chlorophyll-a, aggregated river discharge, and aggregated wind based on eight directions. By emphasizing the most impactful predictors, we streamlined the model-building processes and built a hindcast system from 2015 to 2019. We found that the recurrent neural network model was most effective in predicting the dissolved oxygen concentrations, with an R2 value of 0.99 at multiple stations. Different from our machine learning hindcast models that used observed upcoming meteorological and discharge data, an actual forecast system would use forecasted meteorological and discharge data. Therefore, an actual operational forecast may have lower accuracy than the hindcast, as determined by the accuracy of the predicted meteorological and discharge data. Nevertheless, our studies enhance our understanding of the factors influencing dissolved oxygen variability and set the basis for the implementation of a predictive tool for environmental monitoring and management. We also emphasized the importance of building station-specific models to improve the prediction results. [ABSTRACT FROM AUTHOR]

The management of large enterprises influences their efficiency and profitability. One of the important aspects is the appropriate management of electricity consumption used for production and daily operation. The problem becomes more complicated when you need to manage not one but a large complex of buildings with heterogeneous purposes. In the paper, we examine real-time series data of electrical energy consumption in a complex of heterogeneous buildings, including offices and warehouses, using time series analysis methods such as the Holt–Winters model and ARIMA/SARIMA model, and neural networks (Deep Neural Network, Recurrent Neural Network, and Long Short-Term Memory). Experimental research was performed on a dataset obtained from an energy consumption meter placed in the building complex, built in different periods, and equipped with a variety of automation devices. The data were collected over a period of four years 2018–2021 in the form of time series. Results show that classic models are good at predicting energy consumption in the mentioned type of buildings. The ARIMA model gave the best results—for buildings characterized by seasonality and trends the forecasts were almost perfect with actual values. [ABSTRACT FROM AUTHOR]

Accidents due to defective railway lines and derailments are common disasters that are observed frequently in Southeast Asian countries. It is imperative to run proper diagnosis over the detection of such faults to prevent such accidents. However, manual detection of such faults periodically can be both time-consuming and costly. In this paper, we have proposed a Deep Learning (DL)-based algorithm for automatic fault detection in railway tracks, which we termed an Ensembled Convolutional Autoencoder ResNet-based Recurrent Neural Network (ECARRNet). We compared its output with existing DL techniques in the form of several pre-trained DL models to investigate railway tracks and determine whether they are defective or not while considering commonly prevalent faults such as—defects in rails and fasteners. Moreover, we manually collected the images from different railway tracks situated in Bangladesh and made our dataset. After comparing our proposed model with the existing models, we found that our proposed architecture has produced the highest accuracy among all the previously existing state-of-the-art (SOTA) architecture, with an accuracy of 93.28% on the full dataset. Additionally, we split our dataset into two parts having two different types of faults, which are fasteners and rails. We ran the models on those two separate datasets, obtaining accuracies of 98.59% and 92.06% on rail and fastener, respectively. Model explainability techniques like Grad-CAM and LIME were used to validate the result of the models, where our proposed model ECARRNet was seen to correctly classify and detect the regions of faulty railways effectively compared to the previously existing transfer learning models. [ABSTRACT FROM AUTHOR]

The degradation of the cutting tool and its optimal replacement is a major problem in machining given the variability in this degradation even under constant cutting conditions. Therefore, monitoring the degradation of cutting tools is an important part of the process in order to replace the tool at the optimal time and thus reduce operating costs. In this paper, a cutting tool degradation monitoring technique is proposed using bootstrap-based artificial neural networks. Different indicators from the turning operation are used as input to the approach: the RMS value of the cutting force and torque, the machining duration, and the total machined length. They are used by the approach to estimate the size of the flank wear (VB). Different neural networks are tested but the best results are achieved with an architecture containing two hidden layers: the first one containing six neurons with a Tanh activation function and the second one containing six neurons with an ReLu activation function. The novelty of the approach makes it possible, by using the bootstrap approach, to determine a confidence interval around the prediction. The results show that the networks are able to accurately track the degradation and detect the end of life of the cutting tools in a timely manner, but also that the confidence interval allows an estimate of the possible variation of the prediction to be made, thus helping in the decision for optimal tool replacement policies. [ABSTRACT FROM AUTHOR]

A lack of labeled examples is a problem in different domains, such as text and image processing, medicine, and static reservoir characterization, because supervised learning relies on vast volumes of these data to perform successfully, but this is quite expensive. However, large amounts of unlabeled data exist in these domains. The deep semi-supervised learning (DSSL) approach leverages unlabeled data to improve supervised learning performance using deep neural networks. This approach has succeeded in image recognition, text classification, and speech recognition. Nevertheless, there have been few works on pre-stack seismic reservoir characterization, in which knowledge of rock and fluid properties is fundamental for oil exploration. This paper proposes a methodology to estimate acoustic impedance using pre-stack seismic data and DSSL with a recurrent neural network. The few labeled datasets for training were pre-processed from raw seismic and acoustic impedance data from five borehole logs. The results showed that the acoustic impedance estimation at the well location and outside it was better predicted by the DSSL compared to the supervised version of the same neural network. Therefore, employing a large amount of unlabeled data can be helpful in the development of seismic data interpretation systems. [ABSTRACT FROM AUTHOR]

Maritime traffic is increasing more and more, creating more complex navigation environments for ships. Ship trajectory prediction based on historical AIS data is a vital method of reducing navigation risks and enhancing the efficiency of maritime traffic control. At present, employing machine learning or deep learning techniques to construct predictive models based on AIS data has become a focal point in ship trajectory prediction research. This paper systematically evaluates various trajectory prediction methods, spanning classical machine learning approaches and emerging deep learning techniques, to uncover their respective merits and drawbacks. In this work, a variety of studies were investigated that applied different algorithms in ship trajectory prediction, including regression models (RMs), artificial neural networks (ANNs), Kalman filtering (KF), and random forests (RFs) in machine learning, along with deep learning such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM), gate recurrent unit (GRU) networks, and sequence-to-sequence (Seq2seq) networks. The performance of predictive models based on different algorithms in trajectory prediction tasks was graded and analyzed. Among the existing studies, deep learning methods exhibit significant performance and considerable potential application value for maritime traffic systems, which can be assessed by future work on ship trajectory prediction research. [ABSTRACT FROM AUTHOR]

Real-time prediction of hull girder loads is of great significance for the safety of ship structures. Some scholars have used neural network technology to investigate hull girder load real-time prediction methods based on motion monitoring data. With the development of deep learning technology, a variety of recurrent neural networks have been proposed; however, there is still a lack of systematic comparative analysis on the prediction performance of different networks. In addition, the real motion monitoring data inevitably contains noise, and the effect of data noise has not been fully considered in previous studies. In this paper, four different recurrent neural network models are comparatively investigated, and the effect of different levels of noise on the prediction accuracy of various load components is systematically analyzed. It is found that the GRU network is suitable for predicting the torsional moment and horizontal bending moment, and the LSTM network is suitable for predicting the vertical bending moment. Although filtering has been applied to the original noise data, the prediction accuracy still decreased as the noise level increased. The prediction accuracy of the vertical bending moment and horizontal bending moment is higher than that of the torsional moment. [ABSTRACT FROM AUTHOR]

Dysarthric speech has several pathological characteristics, such as discontinuous pronunciation, uncontrolled volume, slow speech, explosive pronunciation, improper pauses, excessive nasal sounds, and air-flow noise during pronunciation, which differ from healthy speech. Automatic speech recognition (ASR) can be very helpful for speakers with dysarthria. Our research aims to provide a scoping review of ASR for dysarthric speech, covering papers in this field from 1990 to 2022. Our survey found that the development of research studies about the acoustic features and acoustic models of dysarthric speech is nearly synchronous. During the 2010s, deep learning technologies were widely applied to improve the performance of ASR systems. In the era of deep learning, many advanced methods (such as convolutional neural networks, deep neural networks, and recurrent neural networks) are being applied to design acoustic models and lexical and language models for dysarthric-speech-recognition tasks. Deep learning methods are also used to extract acoustic features from dysarthric speech. Additionally, this scoping review found that speaker-dependent problems seriously limit the generalization applicability of the acoustic model. The scarce available speech data cannot satisfy the amount required to train models using big data. [ABSTRACT FROM AUTHOR]

In this paper, a novel approach involving a fractional recurrent neural network (RNN) is proposed to achieve the observer-based synchronization of a cancer cell model. According to the properties of recurrent neural networks, our proposed framework serves as a predictive method for the behavior of fractional-order chaotic cancer systems with uncertain orders. Through a stability analysis of weight updating laws, we design a fractional-order Nonlinear Autoregressive with Exogenous Inputs (NARX) network, in which its learning algorithm demonstrates admissible and faster convergence. The main contribution of this paper lies in the development of a fractional neural observer for the fractional-order cancer systems, which is robust in the presence of uncertain orders. The proposed fractional-order model for cancer can capture complex and nonlinear behaviors more accurately than traditional integer-order models. This improved accuracy can provide a more realistic representation of cancer dynamics. Simulation results are presented to demonstrate the effectiveness of the proposed method, where mean square errors of synchronization by applying integer and fractional weight matrix laws are calculated. The density of tumor cell, density of healthy host cell and density of effector immune cell errors for the observer-based synchronization of fractional-order (OSFO) cancer system are less than 0.0.0048, 0.0062 and 0.0068, respectively. Comparative tables are provided to validate the improved accuracy achieved by the proposed framework. [ABSTRACT FROM AUTHOR]

Recent technological advances have enabled the development of sophisticated robotic and sensor systems monitored and controlled by algorithms based on computational intelligence. The deeply intertwined and cooperating devices connected to the Internet and local networks, usually through wireless communication, are increasingly used in systems deployed among people in public spaces. The challenge is to ensure that physical and digital components work together securely, especially as the impact of cyberattacks is significantly increasing. The paper addresses cybersecurity issues of mobile service robots with distributed control architectures. The focus is on automatically detecting anomalous behaviors possibly caused by cyberattacks on onboard and external sensors measuring the robot and environmental parameters. We provide an overview of the methods and techniques for protecting robotic systems. Particular attention is paid to our technique for anomaly detection in a service robot's operation based on sensor readings and deep recurrent neural networks, assuming that attacks result in the robot behaving inconsistently. The paper presents the architecture of two artificial neural networks, their parameters, and attributes based on which the potential attacks are identified. The solution was validated on the PAL Robotics TIAGo robot operating in the laboratory and replicating a home environment. The results confirm that the proposed system can effectively support the detection of computer threats affecting the sensors' measurements and, consequently, the functioning of a service robotic system. [ABSTRACT FROM AUTHOR]

Facial emotion recognition (FER) is a computer vision process aimed at detecting and classifying human emotional expressions. FER systems are currently used in a vast range of applications from areas such as education, healthcare, or public safety; therefore, detection and recognition accuracies are very important. Similar to any computer vision task based on image analyses, FER solutions are also suitable for integration with artificial intelligence solutions represented by different neural network varieties, especially deep neural networks that have shown great potential in the last years due to their feature extraction capabilities and computational efficiency over large datasets. In this context, this paper reviews the latest developments in the FER area, with a focus on recent neural network models that implement specific facial image analysis algorithms to detect and recognize facial emotions. This paper's scope is to present from historical and conceptual perspectives the evolution of the neural network architectures that proved significant results in the FER area. This paper endorses convolutional neural network (CNN)-based architectures against other neural network architectures, such as recurrent neural networks or generative adversarial networks, highlighting the key elements and performance of each architecture, and the advantages and limitations of the proposed models in the analyzed papers. Additionally, this paper presents the available datasets that are currently used for emotion recognition from facial expressions and micro-expressions. The usage of FER systems is also highlighted in various domains such as healthcare, education, security, or social IoT. Finally, open issues and future possible developments in the FER area are identified. [ABSTRACT FROM AUTHOR]

Forecasting energy consumption models allow for improvements in building performance and reduce energy consumption. Energy efficiency has become a pressing concern in recent years due to the increasing energy demand and concerns over climate change. This paper addresses the energy consumption forecast as a crucial ingredient in the technology to optimize building system operations and identifies energy efficiency upgrades. The work proposes a modified multi-head transformer model focused on multi-variable time series through a learnable weighting feature attention matrix to combine all input variables and forecast building energy consumption properly. The proposed multivariate transformer-based model is compared with two other recurrent neural network models, showing a robust performance while exhibiting a lower mean absolute percentage error. Overall, this paper highlights the superior performance of the modified transformer-based model for the energy consumption forecast in a multivariate step, allowing it to be incorporated in future forecasting tasks, allowing for the tracing of future energy consumption scenarios according to the current building usage, playing a significant role in creating a more sustainable and energy-efficient building usage. [ABSTRACT FROM AUTHOR]

Real-time and accurate state-of-charge estimation performs an important role in the smooth operation of various electric vehicle battery management systems. Neural network theory represents one of the most effective and commonly used methods of SOC prediction. However, traditional neural network methods are disadvantaged by such issues as the limited range of application, limited generalization ability, and low accuracy, which makes it difficult to meet the increasing safety requirements on electric vehicles. In view of these problems, an ensemble learning algorithm based on the AdaBoost.Rt is proposed in this paper. AdaBoost.Rt recurrent neural network model is purposed to ensure the accurate prediction of lithium battery SOC. Relying on a chain-connected recurrent neural network model, this method enables the correlation adaptability of sample data in the spatio-temporal dimension. The ensemble learning method was adopted to devise a method of multi-RNN model integration, with the RNN model as the base learner, thus constructing the AdaBoost.Rt-RNN strong learner model. According to the results of simulation and experimental comparisons, the integrated algorithm proposed in this paper is applicable to improve the accuracy of SOC prediction and the generalization performance of the model. [ABSTRACT FROM AUTHOR]

Unemployment, a significant economic and social challenge, triggers repercussions that affect individual workers and companies, generating a national economic impact. Forecasting the unemployment rate becomes essential for policymakers, allowing them to make short-term estimates, assess economic health, and make informed monetary policy decisions. This paper proposes the innovative GA-LSTM method, which fuses an LSTM neural network with a genetic algorithm to address challenges in unemployment prediction. Effective parameter determination in recurrent neural networks is crucial and a well-known challenge. The research uses the LSTM neural network to overcome complexities and nonlinearities in unemployment predictions, complementing it with a genetic algorithm to optimize the parameters. The central objective is to evaluate recurrent neural network models by comparing them with GA-LSTM to identify the most appropriate model for predicting unemployment in Ecuador using monthly data collected by various organizations. The results demonstrate that the hybrid GA-LSTM model outperforms traditional approaches, such as BiLSTM and GRU, on various performance metrics. This finding suggests that the combination of the predictive power of LSTM with the optimization capacity of the genetic algorithm offers a robust and effective solution to address the complexity of predicting unemployment in Ecuador. [ABSTRACT FROM AUTHOR]

As the architecture of logic devices is evolving towards gate-all-around (GAA) structure, research efforts on advanced transistors are increasingly desired. In order to rapidly perform accurate compact modeling for these ultra-scaled transistors with the capability to cover dimensional variations, neural networks are considered. In this paper, a compact model generation methodology based on artificial neural network (ANN) is developed for GAA nanosheet FETs (NSFETs) at advanced technology nodes. The DC and AC characteristics of GAA NSFETs with various physical gate lengths (Lg), nanosheet widths (Wsh) and thicknesses (Tsh), as well as different gate voltages (Vgs) and drain voltages (Vds) are obtained through TCAD simulations. Subsequently, a high-precision ANN model architecture is evaluated. A systematical study on the impacts of ANN size, activation function, learning rate, and epoch (the times of complete pass through the entire training dataset) on the accuracy of ANN models is conducted, and a shallow neural network configuration for generating optimal ANN models is proposed. The results clearly show that the optimized ANN model can reproduce the DC and AC characteristics of NSFETs very accurately with a fitting error (MSE) of 0.01. [ABSTRACT FROM AUTHOR]

This paper reviews the application of artificial neural network (ANN) models to time series prediction tasks. We begin by briefly introducing some basic concepts and terms related to time series analysis, and by outlining some of the most popular ANN architectures considered in the literature for time series forecasting purposes: feedforward neural networks, radial basis function networks, recurrent neural networks, and self-organizing maps. We analyze the strengths and weaknesses of these architectures in the context of time series modeling. We then summarize some recent time series ANN modeling applications found in the literature, focusing mainly on the previously outlined architectures. In our opinion, these summarized techniques constitute a representative sample of the research and development efforts made in this field. We aim to provide the general reader with a good perspective on how ANNs have been employed for time series modeling and forecasting tasks. Finally, we comment on possible new research directions in this area. [ABSTRACT FROM AUTHOR]

Commercial buildings in hot and humid tropical climates rely significantly on cooling systems to maintain optimal occupant comfort. A well-accurate day-ahead load profile prediction plays a pivotal role in planning the energy requirements of cooling systems. Despite the pressing need for effective day-ahead cooling load predictions, current methodologies have not fully harnessed the potential of advanced deep-learning techniques. This paper aims to address this gap by investigating the application of innovative deep-learning models in day-ahead hourly cooling load prediction for commercial buildings in tropical climates. A range of multi-output deep learning techniques, including Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory networks (LSTMs), are employed to enhance prediction accuracy. Furthermore, these individual deep learning techniques are synergistically integrated to create hybrid models, such as CNN-LSTM and Sequence-to-Sequence models. Experiments are conducted to choose the time horizons from the past that can serve as input to the models. In addition, the influence of various categories of input parameters on prediction performance has been assessed. Historical cooling load, calendar features, and outdoor weather parameters are found in decreasing order of influence on prediction accuracy. This research focuses on buildings located in Singapore and presents a comprehensive case study to validate the proposed models and methodologies. The sequence-to-sequence model provided better performance than all the other models. It offered a CV-RMSE of 7.4%, 10%, and 6% for SIT@Dover, SIT@NYP, and the simulated datasets, which were 2.3%, 3%, and 1% less, respectively, than the base Deep Neural Network model. [ABSTRACT FROM AUTHOR]

In the numerical analysis of manufacturing processes of metal parts, many material properties depending on, for example, the temperature or stress state, must be taken into account. Often these data are dependent on the temperature changes over time. Strongly non-linear material property relationships are usually represented using diagrams. In numerical calculations, these diagrams are analyzed in order to take into account the coupling between the properties. An example of these types of material properties is the dependence of the kinetics of phase transformations in the solid state on the rate and history of temperature change. In literature, these data are visualized Continuous Heating Transformation (CHT) and Continuous Cooling Transformation (CCT) diagrams. Therefore, it can be concluded that time series analysis is important in numerical modeling. This analysis can also be performed using neural networks. This work presents a new approach to storing and analyzing the data contained in the discussed CCT diagrams. The application of Long-Short-Term Memory (LSTM) neural networks and their architecture to determine the correct values of phase fractions depending on the history of temperature change was analyzed. Moreover, an area of research was elements that determine what type of information should be stored by LSTM network coefficients, e.g., whether the network should store information about changes of single phase transformations, or whether it would be better to extract data from differences between several networks with similar architecture. The purpose of the studied network is strongly different from typical applications of artificial neural networks. The main goal of the network was to store information (even by overfitting the network) rather than some form of generalization that allows computation for unknown cases. Therefore, the authors primarily investigated in the ability of the layer-based LSTM network to store nonlinear time series data. The analyses presented in this paper are an extension of the issues presented in the paper entitled "Model of the Austenite Decomposition during Cooling of the Medium Carbon Steel Using LSTM Recurrent Neural Network". [ABSTRACT FROM AUTHOR]

Dropout is one of the most popular regularization methods in the scholarly domain for preventing a neural network model from overfitting in the training phase. Developing an effective dropout regularization technique that complies with the model architecture is crucial in deep learning-related tasks because various neural network architectures have been proposed, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs), and they have exhibited reasonable performance in their specialized areas. In this paper, we provide a comprehensive and novel review of the state-of-the-art (SOTA) in dropout regularization. We explain various dropout methods, from standard random dropout to AutoDrop dropout (from the original to the advanced), and also discuss their performance and experimental capabilities. This paper provides a summary of the latest research on various dropout regularization techniques for achieving improved performance through "Internal Structure Changes", "Data Augmentation", and "Input Information". We can see that proper regularization with respect to structural constraints of network architecture is a critical factor to facilitate overfitting avoidance. We discuss the strengths and limitations of the methods presented in this work, which can serve as valuable references for future research and the development of new approaches. We also pay attention to the scholarly domain in the discussion in order to meet the overwhelming increase of scientific research outcomes by providing an analysis of several important academic scholarly issues of neural networks. [ABSTRACT FROM AUTHOR]

Dynamic neural networks (DNNs) are a type of artificial neural network (ANN) designed to work with sequential data where context in time is important. Unlike traditional static neural networks that process data in a fixed order, dynamic neural networks use information about past inputs, which is important if the dynamic of a certain process is emphasized. They are commonly used in natural language processing, speech recognition, and time series prediction. In industrial processes, their use is interesting for the prediction of difficult-to-measure process variables. In an industrial isomerization process, it is crucial to measure the quality attributes that affect the octane number of gasoline. Process analyzers commonly used for this purpose are expensive and subject to failure. Therefore, to achieve continuous production in the event of a malfunction, mathematical models for estimating product quality attributes are imposed as a solution. In this paper, mathematical models were developed using dynamic recurrent neural networks (RNNs), i.e., their subtype of a long short-term memory (LSTM) architecture. The results of the developed models were compared with the results of several types of other data-driven models developed for an isomerization process, such as multilayer perceptron (MLP) artificial neural networks, support vector machines (SVM), and dynamic polynomial models. The obtained results are satisfactory, suggesting a good possibility of application. [ABSTRACT FROM AUTHOR]

Nonlinear functions are widely used as activation functions in artificial neural networks, which have a great impact on the fitting ability of artificial neural networks. Due to the complexity of the activation function, the computation of the activation function and its derivative requires a lot of computing resources and time during training. In order to improve the computational efficiency of the derivatives of the activation function in the back-propagation of artificial neural networks, this paper proposes a method based on piecewise linear approximation method to calculate the derivative of the activation function. This method is hardware-friendly and universal, it can efficiently compute various nonlinear activation functions in the field of neural network hardware accelerators. In this paper, we use least squares to improve a piecewise linear approximation calculation method that can control the absolute error and get less number of segments or smaller average error, which means fewer hardware resources are required. We use this method to perform a segmented linear approximation to the original or derivative function of the activation function. Both types of activation functions are substituted into a multilayer perceptron for binary classification experiments to verify the effectiveness of the proposed method. Experimental results show that the same or even slightly higher classification accuracy can be achieved by using this method, and the computation time of the back-propagation is reduced by 4–6% compared to the direct calculation of the derivative directly from the function expression using the operator encapsulated in PyTorch. This shows that the proposed method provides an efficient solution of nonlinear activation functions for hardware acceleration of neural networks. [ABSTRACT FROM AUTHOR]

Recent years have seen an increase in events of drone incursion into airport terminal areas, leading to safety concerns and disruptions to airline operations. It is of great importance to identify the potential conflict, especially for those non-cooperative drones, as their intentions are always unknown. For the safe operation of air traffic, this paper proposes a conflict risk assessment method between non-cooperative drones and manned aircraft in the terminal area. First, the trajectory data of manned aircraft and drones are obtained. Real-time cylindrical protection zones are established around manned aircraft according to the separation interval for safe operation between the drone and the manned aircraft at different altitudes. Secondly, trajectory predictions for the manned aircraft and the drone are conducted, respectively. A quartile regression bidirectional gate recurrent unit neural network is proposed in this research for the trajectory prediction of the drones. The model integrates the bidirectional gated recurrent unit structure and the quartile regression structure. The performance indicators confirm the superiority of the proposed model. Based on the trajectory prediction results, it is then determined whether there is a conflict risk between the drone and manned aircraft by comparing the position distribution of the drone as well as the real-time cylindrical protection zone of the manned aircraft. The conflict probability between the drone and the manned aircraft is then calculated. The prediction accuracy of conflict probability is estimated by Monte Carlo simulation methods. The collision probability prediction accuracy of manned aircraft and drones at different flight stages and altitudes ranges from 73% to 97%, which shows the reliability of the proposed method. Finally, the collision probability between the drone and the manned aircraft at the closest encountering point and the estimated time to reach the closest encountering point are calculated. This paper predicts the conflict risk between the drone and manned aircraft, thus providing theoretical support for the safe operation of air transport in low-altitude environments. [ABSTRACT FROM AUTHOR]

Despite advancements in smart grid (SG) technology, effective load forecasting utilizing big data or large-scale datasets remains a complex task for energy management, planning, and control. The Saudi SGs, in alignment with the Saudi Vision 2030, have been envisioned as future electrical grids with a bidirectional flow of power and data. To that end, data analysis and predictive models can enhance Saudi SG planning and control via artificial intelligence (AI). Recently, many AI methods including deep learning (DL) algorithms for SG applications have been published in the literature and have shown superior time series predictions compared with conventional prediction models. Current load-prediction research for the Saudi grid focuses on identifying anticipated loads and consumptions, on utilizing limited historical data and the behavior of the load's consumption, and on conducting shallow forecasting models. However, little scientific proof on complex DL models or real-life application has been conducted by researchers; few articles have studied sophisticated large-scale prediction models for Saudi grids. This paper proposes hybrid DL methods to enhance the outcomes in Saudi SG load forecasting, to improve problem-relevant features, and to accurately predict complicated power consumption, with the goal of developing reliable forecasting models and of obtaining knowledge of the relationships between the various features and attributes in the Saudi SGs. The model in this paper utilizes a real dataset from the Jeddah and Medinah grids in Saudi Arabia for a full year, 2021, with a one-hour time resolution. A benchmark strategy using different conventional DL methods including artificial neural network, recurrent neural network (RNN), conventional neural networks (CNN), long short-term memory (LSTM), gated recurrent unit (GRU), and different real datasets is used to verify the proposed models. The prediction results demonstrate the effectiveness of the proposed hybrid DL models, with CNN–GRU and CNN–RNN with NRMSE obtaining 1.4673% and 1.222% improvements, respectively, in load forecasting accuracy. [ABSTRACT FROM AUTHOR]

Catastrophic forgetting, which means a rapid forgetting of learned representations while learning new data/samples, is one of the main problems of deep neural networks. In this paper, we propose a novel incremental learning framework that can address the forgetting problem by learning new incoming data in an online manner. We develop a new incremental learning framework that can learn extra data or new classes with less catastrophic forgetting. We adopt the hippocampal memory process to the deep neural networks by defining the effective maximum of neural activation and its boundary to represent a feature distribution. In addition, we incorporate incremental QR factorization into the deep neural networks to learn new data with both existing labels and new labels with less forgetting. The QR factorization can provide the accurate subspace prior, and incremental QR factorization can reasonably express the collaboration between new data with both existing classes and new class with less forgetting. In our framework, a set of appropriate features (i.e., nodes) provides improved representation for each class. We apply our method to the convolutional neural network (CNN) for learning Cifar-100 and Cifar-10 datasets. The experimental results show that the proposed method efficiently alleviates the stability and plasticity dilemma in the deep neural networks by providing the performance stability of a trained network while effectively learning unseen data and additional new classes. [ABSTRACT FROM AUTHOR]

The rapid momentum of deep neural networks (DNNs) in recent years has yielded state-of-the-art performance in various machine-learning tasks using speaker identification systems. Speaker identification is based on the speech signals and the features that can be extracted from them. In this article, we proposed a speaker identification system using the developed DNNs models. The system is based on the acoustic and prosodic features of the speech signal, such as pitch frequency (vocal cords vibration rate), energy (loudness of speech), their derivations, and any additional acoustic and prosodic features. Additionally, the article investigates the existing recurrent neural networks (RNNs) models and adapts them to design a speaker identification system using the public YOHO LDC dataset. The average accuracy of the system was 91.93% in the best experiment for speaker identification. Furthermore, this paper helps uncover reasons for analyzing speakers and tokens yielding major errors to increase the system's robustness regarding feature selection and system tune-up. [ABSTRACT FROM AUTHOR]

Unlimited access to information and data sharing wherever and at any time for anyone and anything is a fundamental component of fifth-generation (5G) wireless communication and beyond. Therefore, it has become inevitable to exploit the super-high frequency (SHF) and millimeter-wave (mmWave) frequency bands for future wireless networks due to their attractive ability to provide extremely high data rates because of the availability of vast amounts of bandwidth. However, due to the characteristics and sensitivity of wireless signals to the propagation effects in these frequency bands, more accurate path loss prediction models are vital for the planning, evaluating, and optimizing future wireless communication networks. This paper presents and evaluates the performance of several well-known machine learning methods, including multiple linear regression (MLR), polynomial regression (PR), support vector regression (SVR), as well as the methods using decision trees (DT), random forests (RF), K-nearest neighbors (KNN), artificial neural networks (ANN), and artificial recurrent neural networks (RNN). RNNs are mainly based on long short-term memory (LSTM). The models are compared based on measurement data to provide the best fitting machine-learning-based path loss prediction models. The main results obtained from this study show that the best root-mean-square error (RMSE) performance is given by the ANN and RNN-LSTM methods, while the worst is for the MLR method. All the RMSE values for the given learning techniques are in the range of 0.0216 to 2.9008 dB. Furthermore, this work shows that the models (except for the MLR model) perform excellently in fitting actual measurement data for wireless communications in enclosed indoor environments since they provide R-squared and correlation values higher than 0.91 and 0.96 , respectively. The paper shows that these learning methods could be used as accurate and stable models for predicting path loss in the mmWave frequency regime. [ABSTRACT FROM AUTHOR]

The paper presents a neuro-evolutionary algorithm called Hill Climb Assembler Encoding (HCAE) which is a light variant of Hill Climb Modular Assembler Encoding (HCMAE). While HCMAE, as the name implies, is dedicated to modular neural networks, the target application of HCAE is to evolve small/mid-scale monolithic neural networks which, in spite of the great success of deep architectures, are still in use, for example, in robotic systems. The paper analyses the influence of different mechanisms incorporated into HCAE on the effectiveness of evolved neural networks and compares it with a number of rival algorithms. In order to verify the ability of HCAE to evolve effective small/mid-scale neural networks, both feed forward and recurrent, it was tested on fourteen identification problems including the two-spiral problem, which is a well-known binary classification benchmark, and on two control problems, i.e., the inverted-pendulum problem, which is a classical control benchmark, and the trajectory-following problem, which is a real problem in underwater robotics. Four other neuro-evolutionary algorithms, four particle swarm optimization methods, differential evolution, and a well-known back-propagation algorithm, were applied as a point of reference for HCAE. The experiments reported in the paper revealed that the evolutionary approach applied in the proposed algorithm makes it a more effective tool for solving the test problems than all the rivals. [ABSTRACT FROM AUTHOR]

Every data and kind of data need a physical drive to store it. There has been an explosion in the volume of images, video, and other similar data types circulated over the internet. Users using the internet expect intelligible data, even under the pressure of multiple resource constraints such as bandwidth bottleneck and noisy channels. Therefore, data compression is becoming a fundamental problem in wider engineering communities. There has been some related work on data compression using neural networks. Various machine learning approaches are currently applied in data compression techniques and tested to obtain better lossy and lossless compression results. A very efficient and variety of research is already available for image compression. However, this is not the case for video compression. Because of the explosion of big data and the excess use of cameras in various places globally, around 82% of the data generated involve videos. Proposed approaches have used Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs), and various variants of Autoencoders (AEs) are used in their approaches. All newly proposed methods aim to increase performance (reducing bitrate up to 50% at the same data quality and complexity). This paper presents a bibliometric analysis and literature survey of all Deep Learning (DL) methods used in video compression in recent years. Scopus and Web of Science are well-known research databases. The results retrieved from them are used for this analytical study. Two types of analysis are performed on the extracted documents. They include quantitative and qualitative results. In quantitative analysis, records are analyzed based on their citations, keywords, source of publication, and country of publication. The qualitative analysis provides information on DL-based approaches for video compression, as well as the advantages, disadvantages, and challenges of using them. [ABSTRACT FROM AUTHOR]

Recurrent neural networks (RNNs) have been widely used to solve sequence problems due to their capability of modeling temporal dependency. Despite the rich varieties of RNN models proposed in the literature, the problem of different sampling rates or performing speeds in sequence tasks has not been explicitly considered in the network and the corresponding training and testing processes. This paper addresses the problem of different sampling rates or performing speeds in the skeleton-based action recognition with RNNs. Specifically, the recently proposed independently recurrent neural network (IndRNN) is used as the RNN network due to its well-behaved and easily regulated gradient backpropagation through time. Samples are extracted with variable sampling rates and thus of different lengths, then processed by IndRNN with different time steps. In order to accommodate the differences in terms of gradients introduced by the backpropagation through time under variable time steps, a learning rate adjustment method is further proposed in the paper. Different learning rate adjustment factors are obtained for different layers by analyzing the gradient behavior under IndRNN. Experiments on skeleton-based action recognition are conducted to verify its effectiveness, and the results show that the proposed variable rate IndRNN network can significantly improve the performance over the RNN models under the conventional training strategies. [ABSTRACT FROM AUTHOR]

In the energy-planning sector, the precise prediction of electrical load is a critical matter for the functional operation of power systems and the efficient management of markets. Numerous forecasting platforms have been proposed in the literature to tackle this issue. This paper introduces an effective framework, coded in Python, that can forecast future electrical load based on hourly or daily load inputs. The framework utilizes a recurrent neural network model, consisting of two simpleRNN layers and a dense layer, and adopts the Adam optimizer and tanh loss function during the training process. Depending on the size of the input dataset, the proposed system can handle both short-term and medium-term load-forecasting categories. The network was extensively tested using multiple datasets, and the results were found to be highly promising. All variations of the network were able to capture the underlying patterns and achieved a small test error in terms of root mean square error and mean absolute error. Notably, the proposed framework outperformed more complex neural networks, with a root mean square error of 0.033, indicating a high degree of accuracy in predicting future load, due to its ability to capture data patterns and trends. [ABSTRACT FROM AUTHOR]

The increase in the concentration of geological gas emissions in the atmosphere and particularly the increase of methane is considered by the majority of the scientific community as the main cause of global climate change. The main reasons that place methane at the center of interest, lie in its high global warming potential (GWP) and its lifetime in the atmosphere. Anthropogenic processes, like engineering geology ones, highly affect the daily profile of gasses in the atmosphere. Should direct measures be taken to reduce emissions of methane, immediate global warming mitigation could be achieved. Due to its significance, methane has been monitored by many space missions over the years and as of 2017 by the Sentinel-5P mission. Considering the above, we conclude that monitoring and predicting future methane concentration based on past data is of vital importance for the course of climate change over the next decades. To that end, we introduce a method exploiting state-of-the-art recurrent neural networks (RNNs), which have been proven particularly effective in regression problems, such as time-series forecasting. Aligned with the green artificial intelligence (AI) initiative, the paper at hand investigates the ability of different RNN architectures to predict future methane concentration in the most active regions of Texas, Pennsylvania and West Virginia, by using Sentinel-5P methane data and focusing on computational and complexity efficiency. We conduct several empirical studies and utilize the obtained results to conclude the most effective architecture for the specific use case, establishing a competitive prediction performance that reaches up to a 0.7578 mean squared error on the evaluation set. Yet, taking into consideration the overall efficiency of the investigated models, we conclude that the exploitation of RNN architectures with less number of layers and a restricted number of units, i.e., one recurrent layer with 8 neurons, is able to better compensate for competitive prediction performance, meanwhile sustaining lower computational complexity and execution time. Finally, we compare RNN models against deep neural networks along with the well-established support vector regression, clearly highlighting the supremacy of the recurrent ones, as well as discuss future extensions of the introduced work. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) is an important research problem in computer vision. This problem is widely applied to building applications in human–machine interactions, monitoring, etc. Especially, HAR based on the human skeleton creates intuitive applications. Therefore, determining the current results of these studies is very important in selecting solutions and developing commercial products. In this paper, we perform a full survey on using deep learning to recognize human activity based on three-dimensional (3D) human skeleton data as input. Our research is based on four types of deep learning networks for activity recognition based on extracted feature vectors: Recurrent Neural Network (RNN) using extracted activity sequence features; Convolutional Neural Network (CNN) uses feature vectors extracted based on the projection of the skeleton into the image space; Graph Convolution Network (GCN) uses features extracted from the skeleton graph and the temporal–spatial function of the skeleton; Hybrid Deep Neural Network (Hybrid–DNN) uses many other types of features in combination. Our survey research is fully implemented from models, databases, metrics, and results from 2019 to March 2023, and they are presented in ascending order of time. In particular, we also carried out a comparative study on HAR based on a 3D human skeleton on the KLHA3D 102 and KLYOGA3D datasets. At the same time, we performed analysis and discussed the obtained results when applying CNN-based, GCN-based, and Hybrid–DNN-based deep learning networks. [ABSTRACT FROM AUTHOR]

To enhance the precision of fault diagnosis for high-voltage direct-current (HVDC) systems by effectively extracting various types of fault characteristics, a fault diagnosis method based on the long short-term memory network (LSTM) is proposed in this paper. The method relies on a knowledge graph platform and is developed using measured data from four fault types in an HVDC substation located in southwest China. Firstly, a knowledge graph for the HVDC systems is constructed, then the fault waveform data is preprocessed and divided into a training set and a test set. Various optimizers are employed to train and test the LSTM. The proposed strategy's accuracy is calculated and compared with recurrent neural network (RNN), eXtreme Gradient Boosting (XGBoost), support vector machine (SVM), Naive Bayes classifier, probabilistic neural networks (PNN), and classification learner (CL), which are commonly used in fault diagnosis. Results indicate that the proposed method achieves an accuracy of over 95%, which is 30% higher than RNN, 8% higher than XGBoost, 4% higher than SVM, 7% higher than Naive Bayes, 40% higher than PNN, and 42% higher than classification learner (CL), respectively; the method also has the minimum time cost, fully demonstrating its superiority and effectiveness compared to other methods. [ABSTRACT FROM AUTHOR]

No study has been reported to use machine learning methods to predict the full-range test curves of circular CFST columns. In this paper, the long short-term memory (LSTM) network was introduced to calculate the axially compressive load–strain curves of the circular CFST columns according to an experiment database of limited scale. To improve the feasibility of input data for the recurrent neural network algorithm, data preprocessing methods and data configurations were discussed. The prediction results indicate that the LSTM network provides more accurate estimations compared with the artificial neural networks, random forest and support vector regression. Meanwhile, this method can be used to calculate the mechanical properties including the elastic modulus, ultimate bearing capacity, and the ductility of the columns with acceptable accuracy for engineering practice (the prediction error within 20%). For future research, it is expected that the machine learning method will be applied to predict the structural response of different members under various loading conditions. [ABSTRACT FROM AUTHOR]

Maintenance processes are of high importance for industrial plants. They have to be performed regularly and uninterruptedly. To assist maintenance personnel, industrial sensors monitored by distributed control systems observe and collect several machinery parameters in the cloud. Then, machine learning algorithms try to match patterns and classify abnormal behaviors. This paper presents a new deep learning model called stranded-NN. This model uses a set of NN models of variable layer depths depending on the input. This way, the proposed model can classify different types of emergencies occurring in different time intervals; real-time, close-to-real-time, or periodic. The proposed stranded-NN model has been compared against existing fixed-depth MLPs and LSTM networks used by the industry. Experimentation has shown that the stranded-NN model can outperform fixed depth MLPs 15–21% more in terms of accuracy for real-time events and at least 10–14% more for close-to-real-time events. Regarding LSTMs of the same memory depth as the NN strand input, the stranded NN presents similar results in terms of accuracy for a specific number of strands. Nevertheless, the stranded-NN model's ability to maintain multiple trained strands makes it a superior and more flexible classification and prediction solution than its LSTM counterpart, as well as being faster at training and classification. [ABSTRACT FROM AUTHOR]

The large-scale integration into electrical systems of intermittent power-generation sources, such as wind power plants, requires greater efforts and knowledge from operators to keep these systems operating efficiently. These sources require reliable output power forecasts to set up the optimal operating point of the electrical system. In previous research, the authors developed an evolutionary approach algorithm called RCDESIGN to optimize the hyperparameters and topology of Echo State Networks (ESN), and applied the model in different time series forecasting, including wind speed. In this paper, RCDESIGN was modified in some aspects of the genetic algorithm, and now it optimizes an ESN with augmented states (ESN-AS) and has been called RCDESIGN-AS. The evolutionary algorithm allows the search for the best parameters and topology of the recurrent neural network to be performed simultaneously. In addition, RCDESIGN-AS has the important characteristic of requiring little computational effort and processing time since it is not necessary for the eigenvalues of the reservoir weight matrix to be reduced and also due to the fact that the augmented states make it possible to reduce the number of neurons in the reservoir. The method was applied for wind speed forecasting with a 24-h ahead horizon using real data of wind speed from five cities in the Northeast Region of Brazil. All results obtained with the proposed method overcame forecasting performed by the persistence method, obtaining prediction gains ranging from 60% to 80% in relation to this reference method. In some datasets, the proposed method also yielded better results than the traditional ESN, showing that RCDESIGN-AS can be a powerful tool for wind-speed forecasting and possibly for other types of time series. [ABSTRACT FROM AUTHOR]

Recently, a layer-stacked ESN model named deep echo state network (DeepESN) has been established. As an interactional model of a recurrent neural network and deep neural network, investigations of DeepESN are of significant importance in both areas. Optimizing the structure of neural networks remains a common task in artificial neural networks, and the question of how many neurons should be used in each layer of DeepESN must be stressed. In this paper, our aim is to solve the problem of choosing the optimized size of DeepESN. Inspired by the sensitive iterative pruning algorithm, a neuronal similarity-based iterative pruning merging algorithm (NS-IPMA) is proposed to iteratively prune or merge the most similar neurons in DeepESN. Two chaotic time series prediction tasks are applied to demonstrate the effectiveness of NS-IPMA. The results show that the DeepESN pruned by NS-IPMA outperforms the unpruned DeepESN with the same network size, and NS-IPMA is a feasible and superior approach to improving the generalization performance of DeepESN. The newly proposed method has broad application prospects in real-time systems. [ABSTRACT FROM AUTHOR]

Microsystems are widely used in 5G, the Internet of Things, smart electronic devices and other fields, and signal integrity (SI) determines their performance. Establishing accurate and fast predictive models and intelligent optimization models for SI in microsystems is extremely essential. Recently, neural networks (NNs) and heuristic optimization algorithms have been widely used to predict the SI performance of microsystems. This paper systematically summarizes the neural network methods applied in the prediction of microsystem SI performance, including artificial neural network (ANN), deep neural network (DNN), recurrent neural network (RNN), convolutional neural network (CNN), etc., as well as intelligent algorithms applied in the optimization of microsystem SI, including genetic algorithm (GA), differential evolution (DE), deep partition tree Bayesian optimization (DPTBO), two stage Bayesian optimization (TSBO), etc., and compares and discusses the characteristics and application fields of the current applied methods. The future development prospects are also predicted. Finally, the article is summarized. [ABSTRACT FROM AUTHOR]

Smart grids are able to forecast customers' consumption patterns, i.e., their energy demand, and consequently electricity can be transmitted after taking into account the expected demand. To face today's demand forecasting challenges, where the data generated by smart grids is huge, modern data-driven techniques need to be used. In this scenario, Deep Learning models are a good alternative to learn patterns from customer data and then forecast demand for different forecasting horizons. Among the commonly used Artificial Neural Networks, Long Short-Term Memory networks—based on Recurrent Neural Networks—are playing a prominent role. This paper provides an insight into the importance of the demand forecasting issue, and other related factors, in the context of smart grids, and collects some experiences of the use of Deep Learning techniques, for demand forecasting purposes. To have an efficient power system, a balance between supply and demand is necessary. Therefore, industry stakeholders and researchers should make a special effort in load forecasting, especially in the short term, which is critical for demand response. [ABSTRACT FROM AUTHOR]

In this paper, the useability of feedforward and recurrent neural networks for fusion of data from impulse-radar sensors and depth sensors, in the context of healthcare-oriented monitoring of elderly persons, is investigated. Two methods of data fusion are considered, viz., one based on a multilayer perceptron and one based on a nonlinear autoregressive network with exogenous inputs. These two methods are compared with a reference method with respect to their capacity for decreasing the uncertainty of estimation of a monitored person's position and uncertainty of estimation of several parameters enabling medical personnel to make useful inferences on the health condition of that person, viz., the number of turns made during walking, the travelled distance, and the mean walking speed. Both artificial neural networks were trained on the synthetic data. The numerical experiments show the superiority of the method based on a nonlinear autoregressive network with exogenous inputs. This may be explained by the fact that for this type of network, the prediction of the person's position at each time instant is based on the position of that person at the previous time instants. [ABSTRACT FROM AUTHOR]

In the field of natural language processing, machine translation is a colossally developing research area that helps humans communicate more effectively by bridging the linguistic gap. In machine translation, normalization and morphological analyses are the first and perhaps the most important modules for information retrieval (IR). To build a morphological analyzer, or to complete the normalization process, it is important to extract the correct root out of different words. Stemming and lemmatization are techniques commonly used to find the correct root words in a language. However, a few studies on IR systems for the Urdu language have shown that lemmatization is more effective than stemming due to infixes found in Urdu words. This paper presents a lemmatization algorithm based on recurrent neural network models for the Urdu language. However, lemmatization techniques for resource-scarce languages such as Urdu are not very common. The proposed model is trained and tested on two datasets, namely, the Urdu Monolingual Corpus (UMC) and the Universal Dependencies Corpus of Urdu (UDU). The datasets are lemmatized with the help of recurrent neural network models. The Word2Vec model and edit trees are used to generate semantic and syntactic embedding. Bidirectional long short-term memory (BiLSTM), bidirectional gated recurrent unit (BiGRU), bidirectional gated recurrent neural network (BiGRNN), and attention-free encoder–decoder (AFED) models are trained under defined hyperparameters. Experimental results show that the attention-free encoder-decoder model achieves an accuracy, precision, recall, and F-score of 0.96, 0.95, 0.95, and 0.95, respectively, and outperforms existing models. [ABSTRACT FROM AUTHOR]

Forecasting algorithms have been used to support decision making in companies, and it is necessary to apply approaches that facilitate a good forecasting result. The present paper describes assessments based on a combination of different neural network models, tested to forecast steel production in the world. The main goal is to find the best machine learning model that fits the steel production data in the world to make a forecast for a nine-year period. The study is important for understanding the behavior of the models and sensitivity to hyperparameters of convolutional LSTM and GRU recurrent neural networks. The results show that for long-term prediction, the GRU model is easier to train and provides better results. The article contributes to the validation of the use of other variables that are correlated with the steel production variable, thus increasing forecast accuracy. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel technique for the inspection of high-density polyethylene (HDPE) pipes using ultrasonic sensors, signal processing, and deep neural networks (DNNs). Specifically, we propose a technique that detects whether there is a diversion on a pipe or not. The proposed model transmits ultrasound signals through a pipe using a custom-designed array of piezoelectric transmitters and receivers. We propose to use the Zadoff–Chu sequence to modulate the input signals, then utilize its correlation properties to estimate the pipe channel response. The processed signal is then fed to a DNN that extracts the features and decides whether there is a diversion or not. The proposed technique demonstrates an average classification accuracy of 90.3 % (when one sensor is used) and 99.6 % (when two sensors are used) on 3 4 inch pipes. The technique can be readily generalized for pipes of different diameters and materials. [ABSTRACT FROM AUTHOR]

This paper focuses on the classification of seven locomotion modes (sitting, standing, level ground walking, ramp ascent and descent, stair ascent and descent), the transitions among these modes, and the gait phases within each mode, by only using data in the frequency domain from one or two inertial measurement units. Different deep neural network configurations are investigated and compared by combining convolutional and recurrent layers. The results show that a system composed of a convolutional neural network followed by a long short-term memory network is able to classify with a mean F 1 -score of 0.89 and 0.91 for ten healthy subjects, and of 0.92 and 0.95 for one osseointegrated transfemoral amputee subject (excluding the gait phases because they are not labeled in the data-set), using one and two inertial measurement units, respectively, with a 5-fold cross-validation. The promising results obtained in this study pave the way for using deep learning for the control of transfemoral prostheses with a minimum number of inertial measurement units. [ABSTRACT FROM AUTHOR]

The tremendous rise of electrical energy demand worldwide has led to many problems related to efficient use of electrical energy, consequently posing difficult challenges to electricity consumers of all levels—from households to large companies' facilities. Most of these challenges could be overcome by the accurate prediction of electricity demand. Additionally, balance responsibility includes the penalty-based financial mechanism causing extra expense for badly estimated consumption, above the allowed imbalance limits. In this paper, a method for electricity consumption prediction based on artificial neural networks is proposed. The electricity consumption dataset is obtained from a cold storage facility, which generates data in hourly intervals. The data obtained are measured for a period of over 2 years and then separated to four seasons, so different models are developed for each season. Five different network structures (ordinary RNN, LSTM, GRU, bidirectional LSTM, bidirectional GRU) for five different values of horizon, i.e., input data (one day, two days, four days, one week, two weeks) are examined. Performance indices, such as mean absolute percentage error (MAPE), root mean square error (RMSE), mean absolute error (MAE) and mean square error (MSE), are used in order to obtain qualitative and quantitative comparisons among the obtained models. The results show that the modifications of recurrent neural networks perform much better than ordinary recurrent neural networks. GRU and LSTMB structures with horizons of 168h and 336h are found to have the best performances. [ABSTRACT FROM AUTHOR]