Search

The COVID-19 pandemic has put massive strains on hospitals, and tools to guide hospital planners in resource allocation during the ebbs and flows of the pandemic are urgently needed. We investigate whether machine learning (ML) can be used for predictions of intensive care requirements a fixed number of days into the future. Retrospective design where health Records from 42,526 SARS-CoV-2 positive patients in Denmark was extracted. Random Forest (RF) models were trained to predict risk of ICU admission and use of mechanical ventilation after n days (n = 1, 2, …, 15). An extended analysis was provided for n = 5 and n = 10. Models predicted n-day risk of ICU admission with an area under the receiver operator characteristic curve (ROC-AUC) between 0.981 and 0.995, and n-day risk of use of ventilation with an ROC-AUC between 0.982 and 0.997. The corresponding n-day forecasting models predicted the needed ICU capacity with a coefficient of determination (R2) between 0.334 and 0.989 and use of ventilation with an R2 between 0.446 and 0.973. The forecasting models performed worst, when forecasting many days into the future (for large n). For n = 5, ICU capacity was predicted with ROC-AUC 0.990 and R2 0.928, and use of ventilator was predicted with ROC-AUC 0.994 and R2 0.854. Random Forest-based modelling can be used for accurate n-day forecasting predictions of ICU resource requirements, when n is not too large.

A grand challenge of analytical chemistry is the identification of unknown molecules based on tandem mass spectrometry (MS/MS) spectra. Current metabolite annotation approaches are often manual or partially automated, and commonly rely on a spectral database to search from or train machine learning classifiers on. Unfortunately, spectral databases are often instrument specific and incomplete due to the limited availability of compound standards or a molecular database, which limits the ability of methods utilizing them to predict novel molecule structures. We describe a generative modeling approach that can leverage the vast amount of unpaired and/or unlabeled molecule structures and MS/MS spectra to learn general rules for synthesizing molecule structures and MS/MS spectra. The approach is based on recent work using semi-supervised deep variational autoencoders to learn joint latent representations of multiple and complex modalities. We show that adding molecule structures with no spectra to the training set improves the prediction quality on spectra from a structure disjoint dataset of new molecules, which is not possible using bi-modal supervised approaches. The described methodology provides a demonstration and framework for how recent advances in semi-supervised machine learning can be applied to overcome bottlenecks in missing annotations and noisy data to tackle unaddressed problems in the life sciences where large volumes of data are available.

ImportanceThe COVID-19 pandemic has put massive strains on hospitals, and tools to guide hospital planners in resource allocation during the ebbs and flows of the pandemic are urgently needed.ObjectiveWe investigate whether Machine Learning (ML) can be used for predictions of intensive care requirements 5 and 10 days into the future.DesignRetrospective design where health Records from 34,012 SARS-CoV-2 positive patients was extracted. Random Forest (RF) models were trained to predict risk of ICU admission and use of mechanical ventilation after n days (n = 5, 10).SettingTwo Danish regions, encompassing approx. 2.5 million citizens.ParticipantsAll patients from the bi-regional area with a registered positive SARS-CoV-2 test from March 2020 to January 2021.Main outcomesPrediction of future 5- and 10-day requirements of ICU admission and ventilator use. Mortality was also predicted.ResultsModels predicted 5-day risk of ICU admission with an area under the receiver operator characteristic curve (ROC-AUC) of 0.986 and 5-day risk of use of ventilation with an ROC-AUC of 0.995. The corresponding 5-day forecasting models predicted the needed ICU capacity with a coefficient of determination (R2) of 0.930 and use of ventilation with an R2 of 0.934. Performance was comparable but slightly reduced for 10-day forecasting models.ConclusionsRandom Forest-based modelling can be used for accurate 5- and 10-day forecasting predictions of ICU resource requirements.FundingThe study was funded by grants from the Novo Nordisk Foundation to MS (#NNF20SA0062879 and #NNF19OC0055183) and MN (#NNF20SA0062879).The foundation took no part in project design, data handling and manuscript preparation.KEY POINTSQuestionCan machine learning models (ML) be used for predicting hospital and intensive care unit (ICU) resource requirements, and thus assist in logistics crisis management during the COVID-19 pandemic?FindingsRetrospective study of the resource use of 34.012 COVID-19 patients during the first and second COVID-19 wave in Denmark. ML models were trained for the purpose of predicting the number of patients needing ICU admission and ventilators 5 and 10 day after their first positive SARS-CoV-2 test. The study demonstrates that ML models can accurately predict intensive care admission requirements with 5-day area under the receiver operator characteristic curve (ROC-AUC) of 0.986 and need for ventilator support with a ROC-AUC of 0.995. 10-day predictions were comparable.MeaningThe study demonstrates that ML modelled could be a useful tool for hospital planners during crisis management, including the current COVID-19 pandemic.

The two most common paradigms for end-to-end speech recognition are connectionist temporal classification (CTC) and attention-based encoder-decoder (AED) models. It has been argued that the latter is better suited for learning an implicit language model. We test this hypothesis by measuring temporal context sensitivity and evaluate how the models perform when we constrain the amount of contextual information in the audio input. We find that the AED model is indeed more context sensitive, but that the gap can be closed by adding self-attention to the CTC model. Furthermore, the two models perform similarly when contextual information is constrained. Finally, in contrast to previous research, our results show that the CTC model is highly competitive on WSJ and LibriSpeech without the help of an external language model., Published in the proceedings of INTERSPEECH 2020, pp. 4352-4356

Recent advances in self-supervised learning through contrastive training have shown that it is possible to learn a competitive speech recognition system with as little as 10 minutes of labeled data. However, these systems are computationally expensive since they require pre-training followed by fine-tuning in a large parameter space. We explore the performance of such systems without fine-tuning by training a state-of-the-art speech recognizer on the fixed representations from the computationally demanding wav2vec 2.0 framework. We find performance to decrease without fine-tuning and, in the extreme low-resource setting, wav2vec 2.0 is inferior to its predecessor. In addition, we find that wav2vec 2.0 representations live in a low dimensional subspace and that decorrelating the features of the representations can stabilize training of the automatic speech recognizer. Finally, we propose a bidirectional extension to the original wav2vec framework that consistently improves performance., {\copyright} 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

Sleep disorders affect a large portion of the global population and are strong predictors of morbidity and all-cause mortality. Sleep staging segments a period of sleep into a sequence of phases providing the basis for most clinical decisions in sleep medicine. Manual sleep staging is difficult and time-consuming as experts must evaluate hours of polysomnography (PSG) recordings with electroencephalography (EEG) and electrooculography (EOG) data for each patient. Here, we present U-Sleep, a publicly available, ready-to-use deep-learning-based system for automated sleep staging (sleep.ai.ku.dk). U-Sleep is a fully convolutional neural network, which was trained and evaluated on PSG recordings from 15,660 participants of 16 clinical studies. It provides accurate segmentations across a wide range of patient cohorts and PSG protocols not considered when building the system. U-Sleep works for arbitrary combinations of typical EEG and EOG channels, and its special deep learning architecture can label sleep stages at shorter intervals than the typical 30 s periods used during training. We show that these labels can provide additional diagnostic information and lead to new ways of analyzing sleep. U-Sleep performs on par with state-of-the-art automatic sleep staging systems on multiple clinical datasets, even if the other systems were built specifically for the particular data. A comparison with consensus-scores from a previously unseen clinic shows that U-Sleep performs as accurately as the best of the human experts. U-Sleep can support the sleep staging workflow of medical experts, which decreases healthcare costs, and can provide highly accurate segmentations when human expertize is lacking.

Curriculum learning can improve neural network training by guiding the optimization to desirable optima. We propose a novel curriculum learning approach for image classification that adapts the loss function by changing the label representation.

A texture descriptor based on the shape index and the accompanying curvedness measure is proposed, and it is evaluated for the automated analysis of astronomical image data. A representative sample of images of low-red shift galaxies from the Sloan Digital Sky Survey (SDSS) serves as a test bed. The goal of applying texture descriptors to these data is to extract novel information about galaxies, information which is often lost in more traditional analysis. In this study, we build a regression model for predicting a spectroscopic quantity, the specific star-formation rate (sSFR). As texture features we consider multi-scale gradient orientation histograms as well as multi-scale shape index histograms, which lead to a new descriptor. Our results show that we can successfully predict spectroscopic quantities from the texture in optical multi-band images. We successfully recover the observed bi-modal distribution of galaxies into quiescent and star-forming. The state-of-the-art for predicting the sSFR is a color-based physical model. We significantly improve its accuracy by augmenting the model with texture information. This study is the first step towards enabling the quantification of physical galaxy properties from imaging data alone.

Purpose: To organize a multi-institute knee MRI segmentation challenge for characterizing the semantic and clinical efficacy of automatic segmentation methods relevant for monitoring osteoarthritis progression.Materials and Methods: A dataset partition consisting of three-dimensional knee MRI from 88 retrospective patients at two time points (baseline and 1-year follow-up) with ground truth articular (femoral, tibial, and patellar) cartilage and meniscus segmentations was standardized. Challenge submissions and a majority-vote ensemble were evaluated against ground truth segmentations using Dice score, average symmetric surface distance, volumetric overlap error, and coefficient of variation on a holdout test set. Similarities in automated segmentations were measured using pairwise Dice coefficient correlations. Articular cartilage thickness was computed longitudinally and with scans. Correlation between thickness error and segmentation metrics was measured using the Pearson correlation coefficient. Two empirical upper bounds for ensemble performance were computed using combinations of model outputs that consolidated true positives and true negatives.Results: Six teams (T 1-T 6) submitted entries for the challenge. No differences were observed across any segmentation metrics for any tissues (P = .99) among the four top-performing networks (T 2, T 3, T 4, T 6). Dice coefficient correlations between network pairs were high (> 0.85). Per-scan thickness errors were negligible among networks T 1-T 4 (P = .99), and longitudinal changes showed minimal bias (< 0.03 mm). Low correlations (ρ < 0.41) were observed between segmentation metrics and thickness error. The majority-vote ensemble was comparable to top-performing networks (P = .99). Empirical upper-bound performances were similar for both combinations (P = .99).Conclusion: Diverse networks learned to segment the knee similarly, where high segmentation accuracy did not correlate with cartilage thickness accuracy and voting ensembles did not exceed individual network performance.See also the commentary by Elhalawani and Mak in this issue.Keywords: Cartilage, Knee, MR-Imaging, Segmentation © RSNA, 2020Supplemental material is available for this article.

Fully convolutional neural networks (FCNs) have proven to be powerful tools for medical image segmentation. We apply an FCN based on the U-Net architecture for the challenging task of semantic segmentation of dental panoramic radiographs and discuss general tricks for improving segmentation performance. Among those are network ensembling, test-time augmentation, data symmetry exploitation and bootstrapping of low quality annotations. The performance of our approach was tested on a highly variable dataset of 1500 dental panoramic radiographs. A single network reached the Dice score of 0.934 where 1201 images were used for training, forming an ensemble increased the score to 0.936.

A buffer \(k\)-d tree is a \(k\)-d tree variant for massively-parallel nearest neighbor search. While providing valuable speed-ups on modern many-core devices in case both a large number of reference and query points are given, buffer \(k\)-d trees are limited by the amount of points that can fit on a single device. In this work, we show how to modify the original data structure and the associated workflow to make the overall approach capable of dealing with massive data sets. We further provide a simple yet efficient way of using multiple devices given in a single workstation. The applicability of the modified framework is demonstrated in the context of astronomy, a field that is faced with huge amounts of data.

In the absence of sufficient data variation (e.g., scanner and protocol variability) in annotated data, deep neural networks (DNNs) tend to overfit during training. As a result, their performance is significantly lower on data from unseen sources compared to the performance on data from the same source as the training data. Semi-supervised domain adaptation methods can alleviate this problem by tuning networks to new target domains without the need for annotated data from these domains. Adversarial domain adaptation (ADA) methods are a popular choice that aim to train networks in such a way that the features generated are domain agnostic. However, these methods require careful dataset-specific selection of hyperparameters such as the complexity of the discriminator in order to achieve a reasonable performance. We propose to use knowledge distillation (KD) -- an efficient way of transferring knowledge between different DNNs -- for semi-supervised domain adaption of DNNs. It does not require dataset-specific hyperparameter tuning, making it generally applicable. The proposed method is compared to ADA for segmentation of white matter hyperintensities (WMH) in magnetic resonance imaging (MRI) scans generated by scanners that are not a part of the training set. Compared with both the baseline DNN (trained on source domain only and without any adaption to target domain) and with using ADA for semi-supervised domain adaptation, the proposed method achieves significantly higher WMH dice scores., Comment: MLCN MICCAI workshop

Many recent medical segmentation systems rely on powerful deep learning models to solve highly specific tasks. To maximize performance, it is standard practice to evaluate numerous pipelines with varying model topologies, optimization parameters, pre- & postprocessing steps, and even model cascades. It is often not clear how the resulting pipeline transfers to different tasks. We propose a simple and thoroughly evaluated deep learning framework for segmentation of arbitrary medical image volumes. The system requires no task-specific information, no human interaction and is based on a fixed model topology and a fixed hyperparameter set, eliminating the process of model selection and its inherent tendency to cause method-level over-fitting. The system is available in open source and does not require deep learning expertise to use. Without task-specific modifications, the system performed better than or similar to highly specialized deep learning methods across 3 separate segmentation tasks. In addition, it ranked 5-th and 6-th in the first and second round of the 2018 Medical Segmentation Decathlon comprising another 10 tasks. The system relies on multi-planar data augmentation which facilitates the application of a single 2D architecture based on the familiar U-Net. Multi-planar training combines the parameter efficiency of a 2D fully convolutional neural network with a systematic train- and test-time augmentation scheme, which allows the 2D model to learn a representation of the 3D image volume that fosters generalization.

We designed and jointly optimized an integrated signal processing chain for detection and classification of long-range passive-acoustic underwater signals recorded by the global geophysical monitoring network of the Comprehensive Nuclear-Test-Ban Treaty Organization. Starting at the level of raw waveform data, a processing chain of signal detection, feature extraction, and signal classification was designed and jointly optimized to the task. Relevant waveform segments were in a first step identified by a generic, flexibly parameterized detection algorithm on a long-to short-term averages' ratio of the spectral energy. For representation, general-purpose sound processing features, with an added focus on spectral and cepstral features, were extracted from the detected segments. As classifiers, support vector machines with different kernel functions were employed alongside other baseline learning algorithms. The free parameters of the overall toolchain (i.e., trigger algorithm parameters and classifier hyperparameters) were jointly optimized in a cross-validation setting, either according to the cross-validation classification error or the cross-validation area under the receiver operating characteristic curve. Experiments demonstrate that our method outperforms machine learning algorithms task-tailored to a previous, human-expert-designed preprocessing chain. The presented approach can be adapted to a wide range of problems that can benefit from jointly optimizing parameters of preprocessing and classification algorithm. © 2016 IEEE.

Existing guarantees in terms of rigorous upper bounds on the generalization error for the original random forest algorithm, one of the most frequently used machine learning methods, are unsatisfying. We discuss and evaluate various PAC-Bayesian approaches to derive such bounds. The bounds do not require additional hold-out data, because the out-of-bag samples from the bagging in the training process can be exploited. A random forest predicts by taking a majority vote of an ensemble of decision trees. The first approach is to bound the error of the vote by twice the error of the corresponding Gibbs classifier (classifying with a single member of the ensemble selected at random). However, this approach does not take into account the effect of averaging out of errors of individual classifiers when taking the majority vote. This effect provides a significant boost in performance when the errors are independent or negatively correlated, but when the correlations are strong the advantage from taking the majority vote is small. The second approach based on PAC-Bayesian $$C$$ -bounds takes dependencies between ensemble members into account, but it requires estimating correlations between the errors of the individual classifiers. When the correlations are high or the estimation is poor, the bounds degrade. In our experiments, we compute generalization bounds for random forests on various benchmark data sets. Because the individual decision trees already perform well, their predictions are highly correlated and the $$C$$ -bounds do not lead to satisfactory results. For the same reason, the bounds based on the analysis of Gibbs classifiers are typically superior and often reasonably tight. Bounds based on a validation set coming at the cost of a smaller training set gave better performance guarantees, but worse performance in most experiments.

For proper generalization performance of convolutional neural networks (CNNs) in medical image segmentation, the learnt features should be invariant under particular non-linear shape variations of the input. To induce invariance in CNNs to such transformations, we propose Probabilistic Augmentation of Data using Diffeomorphic Image Transformation (PADDIT) – a systematic framework for generating realistic transformations that can be used to augment data for training CNNs. The main advantage of PADDIT is the ability to produce transformations that capture the morphological variability in the training data. To this end, PADDIT constructs a mean template which represents the main shape tendency of the training data. A Hamiltonian Monte Carlo(HMC) scheme is used to sample transformations which warp the training images to the generated mean template. Augmented images are created by warping the training images using the sampled transformations. We show that CNNs trained with PADDIT outperforms CNNs trained without augmentation and with generic augmentation (0.2 and 0.15 higher dice accuracy respectively) in segmenting white matter hyperintensities from T1 and FLAIR brain MRI scans.

Without access to large compute clusters, building random forests on large datasets is still a challenging problem. This is, in particular, the case if fully-grown trees are desired. We propose a simple yet effective framework that allows to efficiently construct ensembles of huge trees for hundreds of millions or even billions of training instances using a cheap desktop computer with commodity hardware. The basic idea is to consider a multi-level construction scheme, which builds top trees for small random subsets of the available data and which subsequently distributes all training instances to the top trees' leaves for further processing. While being conceptually simple, the overall efficiency crucially depends on the particular implementation of the different phases. The practical merits of our approach are demonstrated using dense datasets with hundreds of millions of training instances., Comment: 9 pages, 9 Figures

Astrophysics and cosmology are rich with data. The advent of wide-area digital cameras on large aperture telescopes has led to ever more ambitious surveys of the sky. Data volumes of entire surveys a decade ago can now be acquired in a single night and real-time analysis is often desired. Thus, modern astronomy requires big data know-how, in particular it demands highly efficient machine learning and image analysis algorithms. But scalability is not the only challenge: Astronomy applications touch several current machine learning research questions, such as learning from biased data and dealing with label and measurement noise. We argue that this makes astronomy a great domain for computer science research, as it pushes the boundaries of data analysis. In the following, we will present this exciting application area for data scientists. We will focus on exemplary results, discuss main challenges, and highlight some recent methodological advancements in machine learning and image analysis triggered by astronomical applications.

With ever-increasing data in the field of medical imaging, the availability of robust methods for quantitative analysis in large-scale studies is the need of the hour. In recent times, there has been a significant increase in the use of deep learning, in particular of convolutional neural networks (CNNs), in the field of computer vision and image analysis. In contrast to traditional shallow classifiers, deep learning methods need less domain-specific feature engineering. The architecture can automatically learn hierarchies of relevant features from raw data. Despite the many success stories from computer vision, so far there are only rather few studies on deep learning in the field of medical imaging. In this chapter, we will look more closely at a specific application of CNNs, namely segmentation of normal brains from magnetic resonance images (MRI). We will characterize the types of errors from CNN-based segmentation and compare them with the errors from a model-based registration approach. The emphasis of this chapter is on comparing errors made by model-driven and data-driven approaches. In conclusion, we notice that the two methods make complementary errors. The CNN errors can be reduced by including more training data and by finding ways to incorporate the geometric information that registration-based algorithms rely on.

Selecting an optimal subset of k out of d features for linear regression models given n training instances is often considered intractable for feature spaces with hundreds or thousands of dimensions. We propose an efficient massively-parallel implementation for selecting such optimal feature subsets in a brute-force fashion for small k. By exploiting the enormous compute power provided by modern parallel devices such as graphics processing units, it can deal with thousands of input dimensions even using standard commodity hardware only. We evaluate the practical runtime using artificial datasets and sketch the applicability of our framework in the context of astronomy.

The bufferkdtree package is an open-source software that provides an efficient implementation for processing huge amounts of nearest neighbor queries in Euclidean spaces of moderate dimensionality. Its underlying implementation resorts to a variant of the classical k-d tree data structure, called buffer k-d tree, which can be used to efficiently perform bulk nearest neighbor searches on modern many-core devices. The package, which is based on Python , C , and OpenCL , is made publicly available online at https://github.com/gieseke/bufferkdtree under the GPLv2 license.

In machine learning, active learning refers to algorithms that autonomously select the data points from which they will learn. There are many data mining applications in which large amounts of unlabeled data are readily available, but labels e.g., human annotations or results coming from complex experiments are costly to obtain. In such scenarios, an active learning algorithm aims at identifying data points that, if labeled and used for training, would most improve the learned model. Labels are then obtained only for the most promising data points. This speeds up learning and reduces labeling costs. Support vector machine SVM classifiers are particularly well-suited for active learning due to their convenient mathematical properties. They perform linear classification, typically in a kernel-induced feature space, which makes expressing the distance of a data point from the decision boundary straightforward. Furthermore, heuristics can efficiently help estimate how strongly learning from a data point influences the current model. This information can be used to actively select training samples. After a brief introduction to the active learning problem, we discuss different query strategies for selecting informative data points and review how these strategies give rise to different variants of active learning with SVMs.

Computer-aided sports analysis is demanded by coaches and the media. Image processing and machine learning techniques that allow for “live” recognition and tracking of players exist. But these methods are far from collecting and analyzing event data fully autonomously. To generate accurate results, human interaction is required at different stages including system setup, calibration, supervision of classifier training, and resolution of tracking conflicts. Furthermore, the real-time constraints are challenging: in contrast to other object recognition and tracking applications, we cannot treat data collection, annotation, and learning as an offline task. A semi-automatic labeling of training data and robust learning given few examples from unbalanced classes are required. We present a real-time system acquiring and analyzing video sequences from soccer matches. It estimates each player’s position throughout the whole match in real-time. Performance measures derived from these raw data allow for an objective evaluation of physical and tactical profiles of teams and individuals. The need for precise object recognition, the restricted working environment, and the technical limitations of a mobile setup are taken into account. Our contribution is twofold: (1) the deliberate use of machine learning and pattern recognition techniques allows us to achieve high classification accuracy in varying environments. We systematically evaluate combinations of image features and learning machines in the given online scenario. Switching between classifiers depending on the amount of training data and available training time improves robustness and efficiency. (2) A proper human–machine interface decreases the number of required operators who are incorporated into the system’s learning process. Their main task reduces to the identification of players in uncertain situations. Our experiments showed high performance in the classification task achieving an average error rate of 3 % on three real-world datasets. The system was proved to collect accurate tracking statistics throughout different soccer matches in real-time by incorporating two human operators only. We finally show how the resulting data can be used instantly for consumer applications and discuss further development in the context of behavior analysis.

Restricted Boltzmann machines (RBMs) are probabilistic graphical models that can be interpreted as stochastic neural networks. They have attracted much attention as building blocks for the multi-layer learning systems called deep belief networks, and variants and extensions of RBMs have found application in a wide range of pattern recognition tasks. This tutorial introduces RBMs from the viewpoint of Markov random fields, starting with the required concepts of undirected graphical models. Different learning algorithms for RBMs, including contrastive divergence learning and parallel tempering, are discussed. As sampling from RBMs, and therefore also most of their learning algorithms, are based on Markov chain Monte Carlo (MCMC) methods, an introduction to Markov chains and MCMC techniques is provided. Experiments demonstrate relevant aspects of RBM training. HighlightsWe review the state-of-the-art in training restricted Boltzmann machines (RBMs) from the perspective of graphical models.Variants and extensions of RBMs are used in a wide range of pattern recognition tasks.The required background on graphical models and Markov chain Monte Carlo methods is provided.Theoretical and experimental results are presented.

Evolutionary computing provides powerful methods for designing pattern recognition systems. This design process is typically based on finite sample data and therefore bears the risk of overfitting. This paper aims at raising the awareness of various types of overfitting and at providing guidelines for how to deal with them. We restrict our considerations to the predominant scenario in which fitness computations are based on point estimates. Three different sources of losing generalization performance when evolving learning machines, namely overfitting to training, test, and final selection data, are identified, discussed, and experimentally demonstrated. The importance of a pristine hold-out data set for the selection of the final result from the evolved candidates is highlighted. It is shown that it may be beneficial to restrict this last selection process to a subset of the evolved candidates.

A central task in the analysis of human movement behavior is to determine systematic patterns and differences across experimental conditions, participants and repetitions. This is possible because human movement is highly regular, being constrained by invariance principles. Movement timing and movement path, in particular, are linked through scaling laws. Separating variations of movement timing from the spatial variations of movements is a well-known challenge that is addressed in current approaches only through forms of preprocessing that bias analysis. Here we propose a novel nonlinear mixed-effects model for analyzing temporally continuous signals that contain systematic effects in both timing and path. Identifiability issues of path relative to timing are overcome by using maximum likelihood estimation in which the most likely separation of space and time is chosen given the variation found in data. The model is applied to analyze experimental data of human arm movements in which participants move a hand-held object to a target location while avoiding an obstacle. The model is used to classify movement data according to participant. Comparison to alternative approaches establishes nonlinear mixed-effects models as viable alternatives to conventional analysis frameworks. The model is then combined with a novel factor-analysis model that estimates the low-dimensional subspace within which movements vary when the task demands vary. Our framework enables us to visualize different dimensions of movement variation and to test hypotheses about the effect of obstacle placement and height on the movement path. We demonstrate that the approach can be used to uncover new properties of human movement., Author Summary When you move a cup to a new location on a table, the movement of lifting, transporting, and setting down the cup appears to be completely automatic. Although the hand could take continuously many different paths and move on any temporal trajectory, real movements are highly regular and reproducible. From repetition to repetition movements vary, and the pattern of variance reflects movement conditions and movement timing. If another person performs the same task, the movement will be similar. When we look more closely, however, there are systematic individual differences. Some people will overcompensate when avoiding an obstacle and some people will systematically move slower than others. When we want to understand human movement, all these aspects are important. We want to know which parts of a movement are common across people and we want to quantify the different types of variability. Thus, the models we use to analyze movement data should contain all the mentioned effects. In this work, we developed a framework for statistical analysis of movement data that respects these structures of movements. We showed how this framework modeled the individual characteristics of participants better than other state-of-the-art modeling approaches. We combined the timing-and-path-separating model with a novel factor analysis model for analyzing the effect of obstacles on spatial movement paths. This combination allowed for an unprecedented ability to quantify and display different sources of variation in the data. We analyzed data from a designed experiment of arm movements under various obstacle avoidance conditions. Using the proposed statistical models, we documented three findings: a linearly amplified deviation in mean path related to increase in obstacle height; a consistent asymmetric pattern of variation along the movement path related to obstacle placement; and the existence of obstacle-distance invariant focal points where mean trajectories intersect in the frontal and vertical planes.

Reservoir computing originally relies on a random, xed population of recurrently connected neurons, but in practice there is a need for algorithms that tailor this reservoir to problem classes. We propose an evolutionary algorithm to adapt size and topology of the network as well as synaptic strengths of connections within the reservoir. Experiments show that the evolved networks signicantly outperform standard architectures in terms of prediction performance and compare well with the results of alternative neuroevolution approaches. For applications in which both space and execution time complexity are important, we present an evolutionary multi-objective algorithm addressing the trade-o between identifying dynamical systems with maximum accuracy and minimizing reservoir complexity. This vector optimization algorithm utilizes the contributed hypervolume for selection and for online adaptation of operator probabilities, which considerably improves the search process. Analysis of the evolved networks reveals insights about task-dependent macroscopic properties of the adapted reservoirs.

A system for the registration of computed tomography and 3-D intraoperative ultrasound images is presented. Three gradient-based methods and one evolutionary algorithm are compared with regard to their suitability to solve this image registration problem. The system has been developed for pedicle screw insertion during spinal surgery. With clinical preoperative and intraoperative data, it is demonstrated that precise registration is possible within a realistic range of initial misalignment. Significant differences can be observed between the optimization methods. The covariance matrix adaptation evolution strategy shows the best overall performance, only four of 12 000 registration trials with patient data failed to register correctly.

Oligo kernels for biological sequence classification have a high discriminative power. A new parameterization for the K-mer oligo kernel is presented, where all oligomers of length K are weighted individually. The task specific choice of these parameters increases the classification performance and reveals information about discriminative features. For adapting the multiple kernel parameters based on cross-validation the covariance matrix adaptation evolution strategy is proposed. It is applied to optimize the trimer oligo kernels for the detection of bacterial gene starts. The resulting kernels lead to higher classification rates, and the adapted parameters reveal the importance of particular triplets for classification, for example of those occurring in the Shine-Dalgarno Sequence.

In astronomical applications of machine learning, the distribution of objects used for building a model is often different from the distribution of the objects the model is later applied to. This is known as sample selection bias, which is a major challenge for statistical inference as one can no longer assume that the labeled training data are representative. To address this issue, one can re-weight the labeled training patterns to match the distribution of unlabeled data that are available already in the training phase. There are many examples in practice where this strategy yielded good results, but estimating the weights reliably from a finite sample is challenging. We consider an efficient nearest neighbor density ratio estimator that can exploit large samples to increase the accuracy of the weight estimates. To solve the problem of choosing the right neighborhood size, we propose to use cross-validation on a model selection criterion that is unbiased under covariate shift. The resulting algorithm is our method of choice for density ratio estimation when the feature space dimensionality is small and sample sizes are large. The approach is simple and, because of the model selection, robust. We empirically find that it is on a par with established kernel-based methods on relatively small regression benchmark datasets. However, when applied to large-scale photometric redshift estimation, our approach outperforms the state-of-the-art.

We developed a fast and robust algorithm to register intraoperative three-dimensional ultrasound data of the spine with preoperative CT data. We compared different gradient-based and evolutionary optimization strategies for solving the registration problem. The iRprop, a fast gradient-based optimization algorithm, quickly and reliably led to higher registration rates than the two established methods BFGS and conjugate gradient descent (CG). The Covariance Matrix Adaptation evolution strategy (CMA) yielded the best results concerning registration rate and accuracy but at the cost of a slightly higher number of evaluations of the optimization criterion compared to CG and iRprop. The CMA was able to register patient data starting from a realistic misalignment in 98% of the trials in about 15 s per registration.

Automobile industry and academic researchers expend considerable effort to develop driver assistance systems. DASs are aware of certain driving situations and support drivers through information, warnings, or even intervention. Many DAS applications are safety oriented, such as lane departure warning systems. Some are comfort oriented, such as automated parking assistants. Moreover, DAS human-machine interfaces must support careful communication with potentially taxed drivers. Driver models support DAS design in several ways. Our work focuses on modeling the tactical level of driving decisions, such as when to brake and whether to accelerate and pass another vehicle. Such decisions are based on local, instantaneously available environmental information about the road and other cars in the same or an adjacent lane.

For face recognition from video streams speed and accuracy are vital aspects. The first decision whether a preprocessed image region represents a human face or not is often made by a feed-forward neural network (NN), e.g. in the Viisage-FaceFINDER® video surveillance system. We describe the optimisation of such a NN by a hybrid algorithm combining evolutionary multi-objective optimisation (EMO) and gradient-based learning. The evolved solutions perform considerably faster than an expert-designed architecture without loss of accuracy. We compare an EMO and a single objective approach, both with online search strategy adaptation. It turns out that EMO is preferable to the single objective approach in several respects.

We propose evolutionary "analysis by synthesis" as a powerful tool in computational neuroscience. We present applications of evolution strategies to the adaptation of dynamical systems for brain modeling. First, we compare evolutionary and gradient-based optimization of dynamic neural fields on an artificial benchmark problem. Then we adjust a few-neuron model developed for explaining our recent findings in a neurobiological experiment, in which we studied the processing of temporal sequences of stimuli in the cortex.

This Special Issue, comprised of four articles, is dedicated to recent developments in the application of machine learning algorithms to traffic sign recognition. The recognition of traffic signs is a challenging real-world problem, which is relevant for intelligent transportation systems participating in traffic environments. It poses a multicategory classification problem with unbalanced class frequencies. Traffic signs show a wide range of variations between classes in terms of color, shape, and the presence of pictograms or text. However, some subsets of classes are very similar to each other (e.g., speed limit signs). In addition to these interclass differences and similarities, the classifier has to cope with large variations in visual appearances due to illumination changes, partial occlusions, rotations, weather conditions, etc.

The problem of finding a suitable neural network topology for a given task is often solved by evolutionary computation. In this paper, we show empirically that online adaptation of the search strategy can increase the performance of evolutionary structure optimization. After a brief overview of strategy adaptation in evolutionary computation, we present a general method for adjusting the probabilities of applying variation operators. In example problems the adaptation method leads to faster optimization and better solutions when used in structure optimization of neural networks. We observe that during the evolutionary process the operator probabilities change in an intuitive way depending on the task.

The Rprop algorithm proposed by Riedmiller and Braun is one of the best performing first-order learning methods for neural networks. We discuss modifications of this algorithm that improve its learning speed. The new optimization methods are empirically compared to the existing Rprop variants, the conjugate gradient method, Quickprop, and the BFGS algorithm on a set of neural network benchmark problems. The improved Rprop outperforms the other methods; only the BFGS performs better in the later stages of learning on some of the test problems. For the analysis of the local search behavior, we compare the Rprop algorithms on general hyperparabolic error landscapes, where the new variants confirm their improvement.

There exist many ideas and assumptions about the development and meaning of modularity in biological and technical neural systems. We empirically study the evolution of connectionist models in the context of modular problems. For this purpose, we define quantitative measures for the degree of modularity and monitor them during evolutionary processes under different constraints. It turns out that the modularity of the problem is reflected by the architecture of adapted systems, although learning can counterbalance some imperfection of the architecture. The demand for fast learning systems increases the selective pressure towards modularity.

We present a nonlinear field model based on linear couplings for inhibition effects in early vision. The model is fitted to data from single unit recordings in the primary visual cortex of the cat. We focus on the prominent effect that responses to second stimuli are amplified, reduced, or unaffected depending on the temporal and spatial distance of the stimuli. The model is adjusted using an elaborated self-adaptive evolution strategy resulting in an accurate, easy to interpret, and well generalizing model.

There is a growing interest in using dynamic neural fields for modeling biological and technical systems, but constructive ways to set up such models are still missing. We discuss gradient-based, evolutionary and hybrid algorithms for data-driven adaptation of neural field parameters. The proposed methods are evaluated using artificial and neuro-physiological data.

Organic Computing has emerged recently as a challenging vision for future information processing systems, based on the insight that we will soon be surrounded by systems with massive numbers of processing elements, sensors and actuators. Because of the scale and complexity of these systems it is infeasible to monitor and control them entirely from external observations; instead they will need to widely monitor, control and adapt themselves. To do so, these components will need to be aware of their environment, to communicate freely, and to organize themselves in order to perform the required actions and services. The presence of such networks of intelligent systems in our environment opens up fascinating application areas but, at the same time, bears the problem of their controllability. Hence, these systems must be as robust, safe, and trustworthy as possible. In order to achieve all these goals, computing systems will have to act more independently, flexibly, and autonomously. Currently, only living beings have these properties to a satisfactory degree. Therefore, Organic Computing puts special emphasis on systems that exhibit some lifelike properties, such as being self-organizing, self-configuring, self-healing, selfprotecting, and context-aware. Many local processes of life are becoming better and better understood, but the organization of all these processes into a coherent and functional organic system requires further research both on experimental and theoretical aspects of organization. This is the view from which the name of Organic Computing originates. Its computing metaphors and guiding principles are directly informed by what is known about living systems. Basic strategies and first applications of Organic Computing can be found in [Wurtz 2008]. The vision of Organic Computing and its fundamental concepts arose independently in different research areas like Neuroscience, Molecular Biology, and Computer Science and Engineering. Its engineering aspect overlaps considerably with Autonomic Computing [Murch 2006] and other self-organizing and self-adaptive computing paradigms (see, e.g. the special issue in volume 4 of this journal). Organic Computing is broader in scope than Autonomic Computing, which restricts its natural metaphor to the autonomic nervous system and its applications to computing services. The dissociation from self-organizing and self-adaptive computing is naturally less sharp, as is their mutual distinction. In system construction, Organic Computing focuses on the integration of bottom-up emerging properties (self-organizing) and top-down design, which self-adapts its parameters. The final goal is the same — systems which reliably do not bother the developer with unnecessary details and yet produce an overall behavior within the desired confinements. For the special issue on Organic Computing, eight manuscripts were submitted. Three of them have been selected in a thorough two-tier review process and now form this issue. The first paper [Schmeck et al. 2010] directly approaches the conflicting objectives of creating self-organizing systems and being able to control the result. To that

The No Free Lunch (NFL) theorems for search and optimization are reviewed and their implications for the design of metaheuristics are discussed. The theorems state that any two search or optimization algorithms are equivalent when their performance is averaged across all possible problems and even over subsets of problems fulfilling certain constraints. The NFL results show that if there is no assumption regarding the relation between visited and unseen search points, efficient search and optimization is impossible. There is no well-performing universal metaheuristic, but the heuristics must be tailored to the problem class at hand using prior knowledge. In practice, it is not likely that the preconditions of the NFL theorems are fulfilled for a problem class and thus differences between algorithms exist. Therefore, tailored algorithms can exploit structure underlying the optimization problem. Given full knowledge about the problem class, it is, in theory, possible to construct an optimal algorithm.

Data in astronomy is rapidly growing with upcoming surveys producing 30 TB of images per night. Highly informative spectra are too expensive to measure for each detected object, hence ways of reliably estimating physical properties from images alone are paramount. The objective of this work is to test whether a “big data ready” k-nearest neighbour regression can successfully estimate the specific star formation rate (sSFR) from colours of low-redshift galaxies. The nearest neighbour algorithm achieves a root mean square error (RMSE) of 0.30, outperforming the state-of-the-art astronomical model achieving a RMSE of 0.36.

Bottom-k sketches are an alternative to k×minwise sketches when using hashing to estimate the similarity of documents represented by shingles (or set similarity in general) in large-scale machine learning. They are faster to compute and have nicer theoretical properties. In the case of k×minwise hashing, the bias introduced by not truly random hash function is independent of the number k of hashes, while this bias decreases with increasing k when employing bottom-k. In practice, bottom-k sketches can expedite classification systems if the trained classifiers are applied to many data points with a lot of features (i.e., to many documents encoded by a large number of shingles on average). An advantage of b-bit k×minwise hashing is that it can be efficiently incorporated into machine learning methods relying on scalar products, such as support vector machines (SVMs). Still, experimental results indicate that a nearest neighbors classifier with bottom-k sketches can be preferable to using a linear SVM and b-bit k×minwise hashing if the amount of training data is low or the number of features is high.

Real-time detection of traffic signs, the task of pinpointing a traffic sign's location in natural images, is a challenging computer vision task of high industrial relevance. Various algorithms have been proposed, and advanced driver assistance systems supporting detection and recognition of traffic signs have reached the market. Despite the many competing approaches, there is no clear consensus on what the state-of-the-art in this field is. This can be accounted to the lack of comprehensive, unbiased comparisons of those methods. We aim at closing this gap by the “German Traffic Sign Detection Benchmark” presented as a competition at IJCNN 2013 (International Joint Conference on Neural Networks). We introduce a real-world benchmark data set for traffic sign detection together with carefully chosen evaluation metrics, baseline results, and a web-interface for comparing approaches. In our evaluation, we separate sign detection from classification, but still measure the performance on relevant categories of signs to allow for benchmarking specialized solutions. The considered baseline algorithms represent some of the most popular detection approaches such as the Viola-Jones detector based on Haar features and a linear classifier relying on HOG descriptors. Further, a recently proposed problem-specific algorithm exploiting shape and color in a model-based Houghlike voting scheme is evaluated. Finally, we present the best-performing algorithms of the IJCNN competition.