Search

Computational approaches such as genome and metabolome mining are becoming essential to natural products (NPs) research. Consequently, a need exists for an automated structure-type classification system to handle the massive amounts of data appearing for NP structures. An ideal semantic ontology for the classification of NPs should go beyond the simple presence/absence of chemical substructures, but also include the taxonomy of the producing organism, the nature of the biosynthetic pathway, and/or their biological properties. Thus, a holistic and automatic NP classification framework could have considerable value to comprehensively navigate the relatedness of NPs, and especially so when analyzing large numbers of NPs. Here, we introduce NPClassifier, a deep-learning tool for the automated structural classification of NPs from their counted Morgan fingerprints. NPClassifier is expected to accelerate and enhance NP discovery by linking NP structures to their underlying properties.

A thiazole-containing cyclic depsipeptide with 11 amino acid residues, named pagoamide A (1), was isolated from laboratory cultures of a marine Chlorophyte, Derbesia sp. This green algal sample was collected from America Samoa, and pagoamide A was isolated using guidance by MS/MS-based molecular networking. Cultures were grown in a light- and temperature-controlled environment and harvested after several months of growth. The planar structure of pagoamide A (1) was characterized by detailed 1D and 2D NMR experiments along with MS and UV analysis. The absolute configurations of its amino acid residues were determined by advanced Marfey's analysis following chemical hydrolysis and hydrazinolysis reactions. Two of the residues in pagoamide A (1), phenylalanine and serine, each occurred twice in the molecule, once in the d- and once in the l-configuration. The biosynthetic origin of pagoamide A (1) was considered in light of other natural products investigations with coenocytic green algae.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

This report describes the first application of the novel NMR-based machine learning tool "Small Molecule Accurate Recognition Technology" (SMART 2.0) for mixture analysis and subsequent accelerated discovery and characterization of new natural products. The concept was applied to the extract of a filamentous marine cyanobacterium known to be a prolific producer of cytotoxic natural products. This environmental Symploca extract was roughly fractionated, and then prioritized and guided by cancer cell cytotoxicity, NMR-based SMART 2.0, and MS2-based molecular networking. This led to the isolation and rapid identification of a new chimeric swinholide-like macrolide, symplocolide A, as well as the annotation of swinholide A, samholides A-I, and several new derivatives. The planar structure of symplocolide A was confirmed to be a structural hybrid between swinholide A and luminaolide B by 1D/2D NMR and LC-MS2 analysis. A second example applies SMART 2.0 to the characterization of structurally novel cyclic peptides, and compares this approach to the recently appearing "atomic sort" method. This study exemplifies the revolutionary potential of combined traditional and deep learning-assisted analytical approaches to overcome longstanding challenges in natural products drug discovery.

Echo state networks (ESNs) are randomly connected recurrent neural networks (RNNs) that can be used as a temporal kernel for modeling time series data, and have been successfully applied on time series prediction tasks. Recently, ESNs have been applied to time series classification (TSC) tasks. However, previous ESN-based classifiers involve either training the model by predicting the next item of a sequence, or predicting the class label at each time step. The former is essentially a predictive model adapted from time series prediction work, rather than a model designed specifically for the classification task. The latter approach only considers local patterns at each time step and then averages over the classifications. Hence, rather than selecting the most discriminating sections of the time series, this approach will incorporate non-discriminative information into the classification, reducing accuracy. In this paper, we propose a novel end-to-end framework called the Echo Memory Network (EMN) in which the time series dynamics and multi-scale discriminative features are efficiently learned from an unrolled echo memory using multi-scale convolution and max-over-time pooling. First, the time series data are projected into the high dimensional nonlinear space of the reservoir and the echo states are collected into the echo memory matrix, followed by a single multi-scale convolutional layer to extract multi-scale features from the echo memory matrix. Max-over-time pooling is used to maintain temporal invariance and select the most important local patterns. Finally, a fully-connected hidden layer feeds into a softmax layer for classification. This architecture is applied to both time series classification and human action recognition datasets. For the human action recognition datasets, we divide the action data into five different components of the human body, and propose two spatial information fusion strategies to integrate the spatial information over them. With one training-free recurrent layer and only one layer of convolution, the EMN is a very efficient end-to-end model, and ranks first in overall classification ability on 55 TSC benchmark datasets and four 3D skeleton-based human action recognition tasks.

Various algorithms comparing 2D NMR spectra have been explored for their ability to dereplicate natural products as well as determine molecular structures. However, spectroscopic artefacts, solvent effects, and the interactive effect of functional group(s) on chemical shifts combine to hinder their effectiveness. Here, we leveraged Non-Uniform Sampling (NUS) 2D NMR techniques and deep Convolutional Neural Networks (CNNs) to create a tool, SMART, that can assist in natural products discovery efforts. First, an NUS heteronuclear single quantum coherence (HSQC) NMR pulse sequence was adapted to a state-of-the-art nuclear magnetic resonance (NMR) instrument, and data reconstruction methods were optimized, and second, a deep CNN with contrastive loss was trained on a database containing over 2,054 HSQC spectra as the training set. To demonstrate the utility of SMART, several newly isolated compounds were automatically located with their known analogues in the embedded clustering space, thereby streamlining the discovery pipeline for new natural products.

The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.

What are the roles of central and peripheral vision in human scene recognition? Larson and Loschky (2009) showed that peripheral vision contributes more than central vision in obtaining maximum scene recognition accuracy. However, central vision is more efficient for scene recognition than peripheral, based on the amount of visual area needed for accurate recognition. In this study, we model and explain the results of Larson and Loschky (2009) using a neurocomputational modeling approach. We show that the advantage of peripheral vision in scene recognition, as well as the efficiency advantage for central vision, can be replicated using state-of-the-art deep neural network models. In addition, we propose and provide support for the hypothesis that the peripheral advantage comes from the inherent usefulness of peripheral features. This result is consistent with data presented by Thibaut, Tran, Szaffarczyk, and Boucart (2014), who showed that patients with central vision loss can still categorize natural scenes efficiently. Furthermore, by using a deep mixture-of-experts model ("The Deep Model," or TDM) that receives central and peripheral visual information on separate channels simultaneously, we show that the peripheral advantage emerges naturally in the learning process: When trained to categorize scenes, the model weights the peripheral pathway more than the central pathway. As we have seen in our previous modeling work, learning creates a transform that spreads different scene categories into different regions in representational space. Finally, we visualize the features for the two pathways, and find that different preferences for scene categories emerge for the two pathways during the training process.

An active object recognition system has the advantage of acting in the environment to capture images that are more suited for training and lead to better performance at test time. In this paper, we utilize deep convolutional neural networks for active object recognition by simultaneously predicting the object label and the next action to be performed on the object with the aim of improving recognition performance. We treat active object recognition as a reinforcement learning problem and derive the cost function to train the network for joint prediction of the object label and the action. A generative model of object similarities based on the Dirichlet distribution is proposed and embedded in the network for encoding the state of the system. The training is carried out by simultaneously minimizing the label and action prediction errors using gradient descent. We empirically show that the proposed network is able to predict both the object label and the actions on GERMS, a dataset for active object recognition. We compare the test label prediction accuracy of the proposed model with Dirichlet and Naive Bayes state encoding. The results of experiments suggest that the proposed model equipped with Dirichlet state encoding is superior in performance, and selects images that lead to better training and higher accuracy of label prediction at test time.

Categorical and coordinate stimulus processing were hypoth-esized by Kosslyn (1987) to be lateralized visual tasks, dif-ferentiated by task-relevant spatial frequencies. Slotnick et al.(2001) directly tested Kosslyn’s hypothesis and concluded thatthe lateralization presents only when tasks are sufficiently dif-ficult. Our differential encoding model is a three layer neuralnetwork that accounts for lateralization in visual processingvia the biologically plausible mechanism of differences in con-nection spread of long-range lateral neural connections (Hsiao,Cipollini, & Cottrell, 2013). We show that our model accountsfor Slotnick’s data and that Slotnick’s analysis does not con-vincingly explain their results. Instead, we propose that Koss-lyn’s initial hypothesis was based on an incorrect assumption:categorical and coordinate stimuli are not solely differentiatedby spatial frequencies. The results that our model capturescannot be reproduced by Ivry and Robertson’s (1998) Dou-ble Filtering by Frequency theory, which is driven solely bylateralized spatial frequency processing.

In previous work, we showed that a simple neurocomputa-tional model The Model, or TM) trained on the Ekman &Friesen Pictures of Facial Affect (POFA) dataset to catego-rize the images into the six basic expressions can account forwide array of data (albeit from a single study) on facial ex-pression processing. The model demonstrated categorical per-ception of facial expressions, as well as the so-called facialexpression circumplex, a circular configuration based on MDSresults that places the categories in the order happy, surprise,fear, sadness, anger and disgust. Somewhat ironically, the cir-cumplex in TM was generated from the similarity between thecategorical outputs of the network, i.e., the six numbers rep-resenting the probability of the category given the face. Here,we extend this work by 1) using a new dataset, NimsStims,that is much larger than POFA, and is not as tightly controlledfor the correct Facial Action Units; 2) using a completely dif-ferent neural network architecture, a Siamese Neural Network(SNN) that maps two faces through twin networks into a 2Dsimilarity space; and 3) training the network only implicitly,based on a teaching signal that pairs of faces are in either inthe same or different categories. Our results show that in thissetting, the network learns a representation that is very similarto the original circumplex. Fear and surprise overlap, whichis consistent with the inherent confusability between these twofacial expressions. Our results suggest that humans evolvedin such a way that nearby emotions are represented by similarappearances.

It is commonly believed that the central visual field (fovea andparafovea) is important for recognizing objects and faces, andthe peripheral region is useful for scene recognition. However,the relative importance of central versus peripheral informa-tion for object, scene, and face recognition is unclear. Larsonand Loschky (2009) investigated this question in the context ofscene processing using experimental conditions where a cir-cular region only reveals the central visual field and blocksperipheral information (”Window”), and in a ”Scotoma” con-dition, where only the peripheral region is available. Theymeasured the scene recognition accuracy as a function of vi-sual angle, and demonstrated that peripheral vision was indeedmore useful in recognizing scenes than central vision in termsof achieving maximum recognition accuracy. In this work,we modeled and replicated the result of Larson and Loschky(2009), using deep convolutional neural networks (CNNs).Having fit the data for scenes, we used the model to predictfuture data for large-scale scene recognition as well as for ob-jects and faces. Our results suggest that the relative order ofimportance of using central visual field information is facerecognition>object recognition>scene recognition, and vice-versa for peripheral information. Furthermore, our results pre-dict that central information is more efficient than peripheralinformation on a per-pixel basis across all categories, which isconsistent with Larson and Loschky’s data.

The visual word form area (VWFA) is a region of the cortex lo-cated in the left fusiform gyrus, that appears to be a waystationin the reading pathway. The discovery of the VWFA occurredin the late twentieth century with the advancement of func-tional magnetic resonance imaging (fMRI). Since then, therehas been an increasing number of neuroimaging studies to un-derstand the VWFA, and there are disagreements as to its prop-erties. One such disagreement is regarding whether or not theVWFA is more selective for real words over pseudowords1. Arecent study using fMRI adaptation (Glezer, et al., 2009) pro-vided evidence that neurons in the VWFA are selectively tunedto real words. This contradicts the hypothesis that the VWFAis tuned to the sublexical structure of visual words, and there-fore has no preference for real words over pseudowords. Inthis paper, we develop a realistic model of the VWFA by train-ing a deep convolutional neural network to map printed wordsto their labels. The network is able to achieve an accuracy of98.5% on the test set. We then analyze this network to see ifit can account for the data Glezer et al. found for words andpseudowords, and find that it does.

Previous functional magnetic resonance imaging (fMRI) research on action observation has emphasized the role of putative mirror neuron areas such as Broca's area, ventral premotor cortex, and the inferior parietal lobule. However, recent evidence suggests action observation involves many distributed cortical regions, including dorsal premotor and superior parietal cortex. How these different regions relate to traditional mirror neuron areas, and whether traditional mirror neuron areas play a special role in action representation, is unclear. Here we use multi-voxel pattern analysis (MVPA) to show that action representations, including observation, imagery, and execution of reaching movements: (1) are distributed across both dorsal (superior) and ventral (inferior) premotor and parietal areas; (2) can be decoded from areas that are jointly activated by observation, execution, and imagery of reaching movements, even in cases of equal-amplitude blood oxygen level-dependent (BOLD) responses; and (3) can be equally accurately classified from either posterior parietal or frontal (premotor and inferior frontal) regions. These results challenge the presumed dominance of traditional mirror neuron areas such as Broca's area in action observation and action representation more generally. Unlike traditional univariate fMRI analyses, MVPA was able to discriminate between imagined and observed movements from previously indistinguishable BOLD activations in commonly activated regions, suggesting finer-grained distributed patterns of activation.

Since Yarbus's seminal work, vision scientists have argued that our eye movement patterns differ depending upon our task. This has recently motivated the creation of multi-fixation pattern analysis algorithms that try to infer a person's task (or mental state) from their eye movements alone. Here, we introduce new algorithms for multi-fixation pattern analysis, and we use them to argue that people have scanpath routines for judging faces. We tested our methods on the eye movements of subjects as they made six distinct judgments about faces. We found that our algorithms could detect whether a participant is trying to distinguish angriness, happiness, trustworthiness, tiredness, attractiveness, or age. However, our algorithms were more accurate at inferring a subject's task when only trained on data from that subject than when trained on data gathered from other subjects, and we were able to infer the identity of our subjects using the same algorithms. These results suggest that (1) individuals have scanpath routines for judging faces, and that (2) these are diagnostic of that subject, but that (3) at least for the tasks we used, subjects do not converge on the same "ideal" scanpath pattern. Whether universal scanpath patterns exist for a task, we suggest, depends on the task's constraints and the level of expertise of the subject.

The processing of Navon figures (Navon, 1977), i.e.,hierarchical letter stimuli, has been studied in experimentalsettings for many years. In particular, they have been studiedin the context of visual hemifield studies and yielded aninteraction between hemifield and whether a target is at thelocal or global level, with a right hemisphere advantage forthe global level, and a left hemisphere advantage for thetargets at the local level (Sergent, 1982). This is a ventralstream process, however, and we were interested in whetherthere might be a similar interaction for hierarchical motionstimuli, presumably a dorsal stream process. Hence wedeveloped a series of dynamic geometric Navon figures inorder to study global/local rotation processing. These figuresconsist of a global figure (a triangle or a square) made up oflocal figures (also triangles or squares). Both global and localfigures can rotate in either clockwise or counterclockwisedirections independently. We found that there is no right orleft visual field perceptual advantage for either the global orlocal levels of these figures. However, curiously enough, wefound that there is a significant processing advantage forclockwise motion compared to counterclockwise motion. Wealso found a highly significant interaction between thedetection of a particular rotational motion and the presence orabsence of that motion in the figure being examined. Finally,our data strongly support the Global Precedence Hypothesiswhich says that people generally tend to focus on the globalproperties of an object before local properties and thatprocessing proceeds in a global-to-local direction.

Recognition of social styles of people is an interesting but relatively unexplored task. Recognizing "style" appears to be a quite different problem than categorization, it is like recognizing a letter's font as opposed to recognizing the letter itself. Similar-looking things must be mapped to different categories. Hence a priori it would appear that features that are good for categorization should not be good for style recognition. Here we show this is not the case by starting with a convolutional deep network pre-trained on Image Net (Caffe), a categorization problem, and using the features as input to a classifier for urban tribes. Combining the results from individuals in group pictures and the group itself, with some fine-tuning of the network, we reduce the previous state of the art error by almost half, going from 46% recognition rate to 71%. To explore how the networks perform this task, we compute the mutual information between the Image Net output category activations and the urban tribe categories, and find, for example, that bikers are well categorized as whiptail lizards by Caffe, and that better recognized social groups have more highly-correlated Image Net categories. This gives us insight into the features useful for categorizing urban tribes.