Search

In this paper, we explore a process called neural teleportation, a mathematical consequence of applying quiver representation theory to neural networks. Neural teleportation teleports a network to a new position in the weight space and preserves its function. This phenomenon comes directly from the definitions of representation theory applied to neural networks and it turns out to be a very simple operation that has remarkable properties. We shed light on the surprising and counter-intuitive consequences neural teleportation has on the loss landscape. In particular, we show that teleportation can be used to explore loss level curves, that it changes the local loss landscape, sharpens global minima and boosts back-propagated gradients at any moment during the learning process. [ABSTRACT FROM AUTHOR]

In the field of natural language processing, machine translation is a colossally developing research area that helps humans communicate more effectively by bridging the linguistic gap. In machine translation, normalization and morphological analyses are the first and perhaps the most important modules for information retrieval (IR). To build a morphological analyzer, or to complete the normalization process, it is important to extract the correct root out of different words. Stemming and lemmatization are techniques commonly used to find the correct root words in a language. However, a few studies on IR systems for the Urdu language have shown that lemmatization is more effective than stemming due to infixes found in Urdu words. This paper presents a lemmatization algorithm based on recurrent neural network models for the Urdu language. However, lemmatization techniques for resource-scarce languages such as Urdu are not very common. The proposed model is trained and tested on two datasets, namely, the Urdu Monolingual Corpus (UMC) and the Universal Dependencies Corpus of Urdu (UDU). The datasets are lemmatized with the help of recurrent neural network models. The Word2Vec model and edit trees are used to generate semantic and syntactic embedding. Bidirectional long short-term memory (BiLSTM), bidirectional gated recurrent unit (BiGRU), bidirectional gated recurrent neural network (BiGRNN), and attention-free encoder–decoder (AFED) models are trained under defined hyperparameters. Experimental results show that the attention-free encoder-decoder model achieves an accuracy, precision, recall, and F-score of 0.96, 0.95, 0.95, and 0.95, respectively, and outperforms existing models. [ABSTRACT FROM AUTHOR]

Complex network science is an interdisciplinary field of study based on graph theory, statistical mechanics, and data science. With the powerful tools now available in complex network theory for the study of network topology, it is obvious that complex network topology models can be applied to enhance artificial neural network models. In this paper, we provide an overview of the most important works published within the past 10 years on the topic of complex network theory-based optimization methods. This review of the most up-to-date optimized neural network systems reveals that the fusion of complex and neural networks improves both accuracy and robustness. By setting out our review findings here, we seek to promote a better understanding of basic concepts and offer a deeper insight into the various research efforts that have led to the use of complex network theory in the optimized neural networks of today. [ABSTRACT FROM AUTHOR]

The seasonal autoregressive integrated moving average with exogenous factors (SARIMAX) has shown promising results in modeling small and sparse observed time-series data by capturing linear features using independent and dependent variables. Long short-term memory (LSTM) is a promising neural network for learning nonlinear dependence features from data. With the increase in wildlife roadkill patterns, the SARIMAX-only and LSTM-only models would likely fail to learn the precise endogenous and/or exogenous variables driven by this wildlife roadkill data. In this paper, we design and implement an error correction mathematical framework based on LSTM-only. The framework extracts features from the residual error generated by a SARIMAX-only model. The learned residual features correct the output time-series prediction of the SARIMAX-only model. The process combines SARIMAX-only predictions and LSTM-only residual predictions to obtain a hybrid SARIMAX-LSTM. The models are evaluated using South African wildlife–vehicle collision datasets, and the experiments show that compared to single models, SARIMAX-LSTM increases the accuracy of a taxon whose linear components outweigh the nonlinear ones. In addition, the hybrid model fails to outperform LSTM-only when a taxon contains more nonlinear components rather than linear components. Our assumption of the results is that the collected exogenous and endogenous data are insufficient, which limits the hybrid model's performance since it cannot accurately detect seasonality on residuals from SARIMAX-only and minimize the SARIMAX-LSTM error. We conclude that the error correction framework should be preferred over single models in wildlife time-series modeling and predictions when a dataset contains more linear components. Adding more related data may improve the prediction performance of SARIMAX-LSTM. [ABSTRACT FROM AUTHOR]

Adversarial examples easily mislead vision systems based on deep neural networks (DNNs) trained with softmax cross entropy (SCE) loss. The vulnerability of DNN comes from the fact that SCE drives DNNs to fit on the training examples, whereas the resultant feature distributions between the training and adversarial examples are unfortunately misaligned. Several state-of-the-art methods start from improving the inter-class separability of training examples by modifying loss functions, where we argue that the adversarial examples are ignored, thus resulting in a limited robustness to adversarial attacks. In this paper, we exploited the inference region, which inspired us to apply margin-like inference information to SCE, resulting in a novel inference-softmax cross entropy (I-SCE) loss, which is intuitively appealing and interpretable. The inference information guarantees that it is difficult for neural networks to cross the decision boundary under an adversarial attack, and guarantees both the inter-class separability and the improved generalization to adversarial examples, which was further demonstrated and proved under the min-max framework. Extensive experiments show that the DNN models trained with the proposed I-SCE loss achieve a superior performance and robustness over the state-of-the-arts under different prevalent adversarial attacks; for example, the accuracy of I-SCE is 63% higher than SCE under the P G D 50 u n attack on the MNIST dataset. These experiments also show that the inference region can effectively solve the misaligned distribution. [ABSTRACT FROM AUTHOR]

Credit risk refers to the possibility of borrower default, and its assessment is crucial for maintaining financial stability. However, the journey of credit risk data generation is often gradual, and machine learning techniques may not be readily applicable for crafting evaluations at the initial stage of the data accumulation process. This article proposes a credit risk modeling methodology, TED-NN, that first constructs an indicator system based on expert experience, assigns initial weights to the indicator system using the Analytic Hierarchy Process, and then constructs a neural network model based on the indicator system to achieve a smooth transition from an empirical model to a data-driven model. TED-NN can automatically adapt to the gradual accumulation of data, which effectively solves the problem of risk modeling and the smooth transition from no to sufficient data. The effectiveness of this methodology is validated through a specific case of credit risk assessment. Experimental results on a real-world dataset demonstrate that, in the absence of data, the performance of TED-NN is equivalent to the AHP and better than untrained neural networks. As the amount of data increases, TED-NN gradually improves and then surpasses the AHP. When there are sufficient data, its performance approaches that of a fully data-driven neural network model. [ABSTRACT FROM AUTHOR]

The actual behavior of welded T-junctions in tubular structures depends strongly on the topology of the junction at the joint level. In finite element analysis, beam-type elements are usually employed due to their simplicity and low computational cost, even though they cannot reproduce the joints topologies and characteristics. To adjust their behavior to a more realistic situation, elastic elements can be introduced at the joint level, whose characteristics must be determined through costly validations. This paper studies the optimization and implementation of the validation data, through the creation of an optimal surrogate model based on neural networks, leading to a model that predicts the stiffness of elastic elements, introduced at the joint level based on available data. The paper focuses on how the neural network should be chosen, when training data is very limited and, more importantly, which of the available data should be used for training and which for verification. The methodology used is based on a Monte Carlo analysis that allows an exhaustive study of both the network parameters and the distribution and choice of the limited data in the training set to optimize its performance. The results obtained indicate that the use of neural networks without a careful methodology in this type of problems could lead to inaccurate results. It is also shown that a conscientious choice of training data, among the data available in the problem of choice of elastic parameters for T-junctions in finite elements, is fundamental to achieve functional surrogate models. [ABSTRACT FROM AUTHOR]

Currently, direct surveys are used less and less to assess satisfaction with the quality of user services. One of the most effective methods to solve this problem is to extract user attitudes from social media texts using natural language text mining. This approach helps to obtain more objective results by increasing the representativeness and independence of the sample of service consumers being studied. The purpose of this article is to improve existing methods and test a method for classifying Russian-language text reviews of patients about the work of medical institutions and doctors, extracted from social media resources. The authors developed a hybrid method for classifying text reviews about the work of medical institutions and tested machine learning methods using various neural network architectures (GRU, LSTM, CNN) to achieve this goal. More than 60,000 reviews posted by patients on the two most popular doctor review sites in Russia were analysed. Main results: (1) the developed classification algorithm is highly efficient—the best result was shown by the GRU-based architecture (val_accuracy = 0.9271); (2) the application of the method of searching for named entities to text messages after their division made it possible to increase the classification efficiency for each of the classifiers based on the use of artificial neural networks. This study has scientific novelty and practical significance in the field of social and demographic research. To improve the quality of classification, in the future, it is planned to expand the semantic division of the review by object of appeal and sentiment and take into account the resulting fragments separately from each other. [ABSTRACT FROM AUTHOR]

In this paper, we present an impulsive pinning control algorithm for discrete-time complex networks with different node dynamics, using a linear algebra approach and a neural network as an identifier, to synthesize a learning control law. The model of the complex network used in the analysis has unknown node self-dynamics, linear connections between nodes, where the impulsive dynamics add feedback control input only to the pinned nodes. The proposed controller consists of the linearization for the node dynamics and a reorder of the resulting quadratic Lyapunov function using the Rayleigh quotient. The learning part of the control is done with a discrete-time recurrent high order neural network used for identification of the pinned nodes, which is trained using an extended Kalman filter algorithm. A numerical simulation is included in order to illustrate the behavior of the system under the developed controller. For this simulation, a 20-node complex network with 5 different node dynamics is used. The node dynamics consists of discretized versions of well-known continuous chaotic attractors. [ABSTRACT FROM AUTHOR]

Screening and evaluation of developmental disorders include complex and challenging procedures, exhibit uncertainties in the diagnostic fit, and require high clinical expertise. Although typically, clinicians' evaluations rely on diagnostic instrumentation, child observations, and parents' reports, these may occasionally result in subjective evaluation outcomes. Current advances in artificial intelligence offer new opportunities for decision making, classification, and clinical assessment. This study explores the performance of different neural network optimizers in biometric datasets for screening typically and non-typically developed children for speech and language communication deficiencies. The primary motivation was to give clinicians a robust tool to help them identify speech disorders automatically using artificial intelligence methodologies. For this reason, in this study, we use a new dataset from an innovative, recently developed serious game collecting various data on children's speech and language responses. Specifically, we employed different neural network approaches such as Artificial Neural Networks (ANNs), K-Nearest Neighbor (KNN), Support Vector Machines (SVM), along with state-of-the-art Optimizers, namely the Adam, the Broyden–Fletcher–Goldfarb–Shanno (BFGS), Genetic algorithm (GAs), and Particle Swarm Optimization algorithm (PSO). The results were promising, while Integer-bounded Neural Network proved to be the best competitor, opening new inquiries for future work towards automated classification supporting clinicians' decisions on neurodevelopmental disorders. [ABSTRACT FROM AUTHOR]

The industrial production of microalgae is a process as sustainable as it is interesting in terms of its diverse applications, especially for wastewater treatment. Its optimization requires an exhaustive knowledge of the system, which is commonly achieved through models that describe its dynamics. Although not widely used in this field, artificial neural networks are presented as an appropriate technique to develop this type of model, having the ability to adapt to complex and nonlinear problems solely from the process data. In this work, neural network models have been developed to characterize the pH dynamics in two different raceway reactors, one with freshwater and the other with wastewater. The models are able to predict pH profiles with a prediction horizon of up to eleven hours and only using available measurable process data, such as medimum level, CO 2 injection, and solar radiation. The results demonstrate the potential of artificial neural networks in the modeling of continuous dynamic systems in the field of industry, obtaining accurate, fast-running models that can adapt to different circumstances. Moreover, these models open the field to the design of data-driven model-based control algorithms to account for the nonlinear dynamics of this biological system. [ABSTRACT FROM AUTHOR]

Sperm morphological analysis (SMA) is an essential step in diagnosing male infertility. Using images of human sperm cells, this research proposes a unique sequential deep-learning method to detect abnormalities in semen samples. The proposed technique identifies and examines several components of human sperm. In order to conduct this study, we used the online Modified Human Sperm Morphology Analysis (MHSMA) dataset containing 1540 sperm images collected from 235 infertile individuals. For research purposes, this dataset is freely available online. To identify morphological abnormalities in different parts of human sperm, such as the head, vacuole, and acrosome, we proposed sequential deep neural network (SDNN) architecture. This technique is also particularly effective with low-resolution, unstained images. Sequential deep neural networks (SDNNs) demonstrate high accuracy in diagnosing morphological abnormalities based on the given dataset in our tests on the benchmark. Our proposed algorithm successfully detected abnormalities in the acrosome, head, and vacuole with an accuracy of 89%, 90%, and 92%, respectively. It is noteworthy that our system detects abnormalities of the acrosome and head with greater accuracy than current state-of-the-art approaches on the suggested benchmark. On a low-specification computer/laptop, our algorithm also requires less execution time. Additionally, it can classify photos in real time. Based on the results of our study, an embryologist can quickly decide whether to use the given sperm. [ABSTRACT FROM AUTHOR]

Forecasting exchange rates is a complex problem that has benefitted from recent advances and research in machine learning. The main goal of this study is to design and implement a method to improve the learning performance of artificial neural networks with large volumes of data using population-based metaheuristics. The micro-genetic training algorithm is thoroughly analyzed using profiling tools to find bottlenecks. We compare the use of a micro-genetic algorithm to predict changes in currency exchange rates on a data set containing more than 500,000 values. To find the best parameters of neural networks, we propose an improved micro-genetic training algorithm by dividing the training data into mini batches. In this case, the improved micro-genetic algorithm proved to be much faster compared to the standard genetic algorithm, while achieving the same prediction accuracy. This allows for the use of this algorithm for just-in-time predictions of high frequency data. Here, neural network models are first created and validated on an existing data set. Then, the new data values can be added to neural network models and retrained in a short time. [ABSTRACT FROM AUTHOR]

Software requirement changes, code changes, software reuse, and testing are important activities in software engineering that involve the traceability links between software requirements and code. Software requirement documents, design documents, code documents, and test case documents are the intermediate products of software development. The lack of interrelationship between these documents can make it extremely difficult to change and maintain the software. Frequent requirements and code changes are inevitable in software development. Software reuse, change impact analysis, and testing also require the relationship between software requirements and code. Using these traceability links can improve the efficiency and quality of related software activities. Existing methods for constructing these links need to be better automated and accurate. To address these problems, we propose to embed software requirements and source code into feature vectors containing their semantic information based on four neural networks (NBOW, RNN, CNN, and self-attention). Accurate traceability links from requirements to code are established by comparing the similarity between these vectors. We develop a prototype tool RCT based on this method. These four networks' performances in constructing links are explored on 18 open-source projects. The experimental results show that the self-attention network performs best, with an average R e c a l l @ 50 value of 0.687 on the 18 projects, which is higher than the other three neural network models and much higher than previous approaches using information retrieval and machine learning. [ABSTRACT FROM AUTHOR]

The significance of back-propagated intelligent neural networks (BINs) to investigate the transmission of heat in spinning nanofluid over a rotating system is analyzed in this study. The buoyancy effect is incorporated along with the constant thermophysical properties of nanofluids. Levenberg–Marquardt intelligent networks (ANNLMBs) are employed to study heat transmission by using a trained artificial neural network. The system of highly non-linear flow governing partial differential equations (PDEs) is transformed into ordinary differential equations (ODEs) which is taken as a system model. This achieved system model is utilized to generate data set using the "Adams" method for distinct scenarios of heat transmission investigation in a spinning nanofluid over a rotating system for the implementation of the proposed ANNLMB. Additionally, with the help of training, testing, and validation, the approximate solution of heat transmission in a spinning nanofluid in a rotating system is obtained using a BNN-based solver. The generated reference data achieved employing the proposed artificial neural network based on a Levenberg–Marquardt intelligent network is distributed in the following manner: training at 82%, testing at 9%, and validation at 9%. Furthermore, MSE, histograms, and regression analyses are performed to depict and discuss the impact of the varying influence of key parameters, such as unsteadiness "s" in spinning flow, Prandtl number effect "pr", the rotational ratio of nanofluid and cone α 1 and buoyancy effect γ 1 on velocities F ′ G and temperature Θ profiles. The mean square error confirms the accuracy of the achieved results. Prandtl number and unsteadiness decrease the temperature profile and thermal boundary layer of the rotating nanofluid. [ABSTRACT FROM AUTHOR]

Deep transfer learning (DTL), which incorporates new ideas from deep neural networks into transfer learning (TL), has achieved excellent success in computer vision, text classification, behavior recognition, and natural language processing. As a branch of machine learning, DTL applies end-to-end learning to overcome the drawback of traditional machine learning that regards each dataset individually. Although some valuable and impressive general surveys exist on TL, special attention and recent advances in DTL are lacking. In this survey, we first review more than 50 representative approaches of DTL in the last decade and systematically summarize them into four categories. In particular, we further divide each category into subcategories according to models, functions, and operation objects. In addition, we discuss recent advances in TL in other fields and unsupervised TL. Finally, we provide some possible and exciting future research directions. [ABSTRACT FROM AUTHOR]

In this paper, the global asymptotical stability of Riemann-Liouville fractional-order neural networks with time-varying delays is studied. By combining the Lyapunov functional function and LMI approach, some sufficient criteria that guarantee the global asymptotical stability of such fractional-order neural networks with both discrete time-varying delay and distributed time-varying delay are derived. The stability criteria is suitable for application and easy to be verified by software. Lastly, some numerical examples are presented to check the validity of the obtained results. [ABSTRACT FROM AUTHOR]

With the evolution of modern digital pathology, examining cancer cell tissues has paved the way to quantify subtle symptoms, for example, by means of image staining procedures using Eosin and Hematoxylin. Cancer tissues in the case of breast and lung cancer are quite challenging to examine by manual expert analysis of patients suffering from cancer. Merely relying on the observable characteristics by histopathologists for cell profiling may under-constrain the scale and diagnostic quality due to tedious repetition with constant concentration. Thus, automatic analysis of cancer cells has been proposed with algorithmic and soft-computing techniques to leverage speed and reliability. The paper's novelty lies in the utility of Zernike image moments to extract complex features from cancer cell images and using simple neural networks for classification, followed by explainability on the test results using the Local Interpretable Model-Agnostic Explanations (LIME) technique and Explainable Artificial Intelligence (XAI). The general workflow of the proposed high throughput strategy involves acquiring the BreakHis public dataset, which consists of microscopic images, followed by the application of image processing and machine learning techniques. The recommended technique has been mathematically substantiated and compared with the state-of-the-art to justify the empirical basis in the pursuit of our algorithmic discovery. The proposed system is able to classify malignant and benign cancer cell images of 40× resolution with 100% recognition rate. XAI interprets and reasons the test results obtained from the machine learning model, making it reliable and transparent for analysis and parameter tuning. [ABSTRACT FROM AUTHOR]

As the amount of historical data available in the legal arena has grown over time, industry specialists are driven to gather, compile, and analyze this data in order to forecast court case rulings. However, predicting and justifying court rulings while using judicial facts is no easy task. Currently, previous research on forecasting court outcomes using small experimental datasets yielded a number of unanticipated predictions utilizing machine learning (ML) models and conventional methodologies for categorical feature encoding. The current work proposes forecasting court judgments using a hybrid neural network model, namely a long short-term memory (LSTM) network with a CNN, in order to effectively forecast court rulings using historic judicial datasets. By prioritizing and choosing features that scored the highest in the provided legal data set, only the most pertinent features were picked. After that, the LSTM+CNN model was utilized to forecast lawsuit verdicts. In contrast to previous related experiments, this composite model's testing results were promising, showing 92.05 percent accuracy, 93 percent precision, 94 percent recall, and a 93 percent F1-score. [ABSTRACT FROM AUTHOR]

In this presented communication, a novel design of intelligent Bayesian regularization backpropagation networks (IBRBNs) based on stochastic numerical computing is presented. The dynamics of peristaltic motion of a third-grade fluid in a planar channel is examined by IBRBNs using multilayer structure modeling competency of neural networks trained with efficient optimization ability of Bayesian regularization method. The reference dataset used as inputs and targets parameters of IBRBN has been obtained via the state-of-the-art Adams numerical method. The data of solution dynamics is created for multiple scenarios of the peristaltic transport model by varying the volume flow rate, material parametric of a third-grade fluid model, wave amplitude, and inclination angles. The designed integrated IBRBNs are constructed by exploiting training, testing, and validation operations at each epoch via optimization of a figure of merit on mean square error sense. Exhaustive simulation of IBRBNs with comparison on mean square error, histograms, and regression index substantiated the precision, stability, and reliability to solve the peristaltic transport model. [ABSTRACT FROM AUTHOR]

The function and performance of neural networks are largely determined by the evolution of their weights and biases in the process of training, starting from the initial configuration of these parameters to one of the local minima of the loss function. We perform the quantitative statistical characterization of the deviation of the weights of two-hidden-layer feedforward ReLU networks of various sizes trained via Stochastic Gradient Descent (SGD) from their initial random configuration. We compare the evolution of the distribution function of this deviation with the evolution of the loss during training. We observed that successful training via SGD leaves the network in the close neighborhood of the initial configuration of its weights. For each initial weight of a link we measured the distribution function of the deviation from this value after training and found how the moments of this distribution and its peak depend on the initial weight. We explored the evolution of these deviations during training and observed an abrupt increase within the overfitting region. This jump occurs simultaneously with a similarly abrupt increase recorded in the evolution of the loss function. Our results suggest that SGD's ability to efficiently find local minima is restricted to the vicinity of the random initial configuration of weights. [ABSTRACT FROM AUTHOR]

Complex networks constitute a new field of scientific research that is derived from the observation and analysis of real-world networks, for example, biological, computer and social ones. An important subset of complex networks is the biological, which deals with the numerical examination of connections/associations among different nodes, namely interfaces. These interfaces are evolutionary and physiological, where network epidemic models or even neural networks can be considered as representative examples. The investigation of the corresponding biological networks along with the study of human diseases has resulted in an examination of networks regarding medical supplies. This examination aims at a more profound understanding of concrete networks. Fuzzy logic is considered one of the most powerful mathematical tools for dealing with imprecision, uncertainties and partial truth. It was developed to consider partial truth values, between completely true and completely false, and aims to provide robust and low-cost solutions to real-world problems. In this manuscript, we introduce a fuzzy implementation of epidemic models regarding the Human Immunodeficiency Virus (HIV) spreading in a sample of needle drug individuals. Various fuzzy scenarios for a different number of users and different number of HIV test samples per year are analyzed in order for the samples used in the experiments to vary from case to case. To the best of our knowledge, analyzing HIV spreading with fuzzy-based simulation scenarios is a research topic that has not been particularly investigated in the literature. The simulation results of the considered scenarios demonstrate that the existence of fuzziness plays an important role in the model setup process as well as in analyzing the effects of the disease spread. [ABSTRACT FROM AUTHOR]

Piezoelectric actuators (PEA) are frequently employed in applications where nano-Micr-odisplacement is required because of their high-precision performance. However, the positioning is affected substantially by the hysteresis which resembles in an nonlinear effect. In addition, hysteresis mathematical models own deficiencies that can influence on the reference following performance. The objective of this study was to enhance the tracking accuracy of a commercial PEA stack actuator with the implementation of a novel approach which consists in the use of a Super-Twisting Algorithm (STA) combined with artificial neural networks (ANN). A Lyapunov stability proof is bestowed to explain the theoretical solution. Experimental results of the proposed method were compared with a proportional-integral-derivative (PID) controller. The outcomes in a real PEA reported that the novel structure is stable as it was proved theoretically, and the experiments provided a significant error reduction in contrast with the PID. [ABSTRACT FROM AUTHOR]