Search

Editor’s notes: This article discusses current directions in computation-aware deep learning and describes two new challenges in the IoT era: 1) data-independent deployment of learning and 2) communication-aware distributed inference. It presents new directions to alleviate these challenges. — Deming Chen, University of Illinois at Urbana-Champaign

Invasive lobular carcinoma (ILC) accounts for 10-15% of primary breast cancer and is typically ER+ and ERBB2 non-amplified. There is preclinical evidence that somatic ERBB2 mutation may provide an alternative and tractable mechanism for upregulation of HER2 activity in tumors that do not express HER2 by current clinical criteria. Using large public datasets, we previously demonstrated that targetable ERBB2 mutations are enriched in ILC versus invasive ductal carcinoma (IDC) and are an independent prognostic factor in ILC (HR=3.7, 95% CI 1.2-11.0; p=0.021)*. We next hypothesized that a gene expression signature incorporating HER2 activity due to ERBB2 mutation and / or amplification would validate the prognostic signal we found in ILC. To derive a novel gene expression signature of HER2 activity that accounted for the effect of potentially targetable ERBB2 mutations in ERBB2 non-amplified tumors, we applied a weighted average difference method to gene expression data in cases from the METABRIC 2012 (N=1,980) and TCGA 2015 (N=817) cohorts. To compare our novel gene expression signature with an established signature of HER2 activity, we performed multivariate regression modeling of response to neratinib, a small molecule tyrosine kinase inhibitor of HER1, 2 and 4, using pharmacogenomic data accessed via the CellMinerCDB online portal. We show that our novel HER2 pathway signature score uniquely enriches for ERBB2 mutated tumors. Using a Cox regression model and stratifying gene expression scores into upper versus lower quartiles, we were able to validate the prognostic signal of ERBB2 mutations in ILC tumors (HR for 10-year OS in ILC=2.3, 95% CI 1.04-5.05; p=0.040). In contrast, no relationship was found between ERBB2 mutation status or novel HER2 pathway enrichment score and patient outcome in cases of IDC. We conclude that ERBB2 mutations that are enriched in ILC provide a robust biomarker of HER2 pathway activation and can be detected via gene expression signature. Novel clinical trials of HER2-targeted therapy in ERBB2 non-amplified primary ILC are warranted. *Reference: Kurozumi S et al, Cancer Research 2019, 80(4) suppl: SABCS 2019 Abstract P1-18-06 and medRxiv 2020.01.24.20018622 Citation Format: Mansour Alsaleem, Sasagu Kurozumi, Kartikeya Bhardwaj, Cintia Monteiro, Stacey EP Joosten, Andrew R Green, Takaaki Fujii, Ken Shirabe, Ian O Ellis, Emad A Rakha, Nigel P Mongan, David M Heery, Wilbert Zwart, Steffi Oesterreich, Simon J Johnston. Targetable ERBB2 mutation status is an independent marker of adverse prognosis in estrogen receptor positive, ERBB2 non-amplified primary lobular breast carcinoma: Validation using a novel gene signature of HER2 activation [abstract]. In: Proceedings of the 2020 San Antonio Breast Cancer Virtual Symposium; 2020 Dec 8-11; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2021;81(4 Suppl):Abstract nr PS6-11.

In this paper, we identify a new phenomenon called activation-divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets (The code is available at https://github.com/weichennone/FedMAX).

BackgroundInvasive lobular carcinoma (ILC) accounts for 10-15% of primary breast cancers and is typically estrogen receptor alpha positive (ER+) and ERBB2 non-amplified. Somatic mutations in ERBB2/3 are emerging as a tractable mechanism underlying enhanced human epidermal growth factor 2 (HER2) activity. We tested the hypothesis that therapeutically targetable ERBB2/3 mutations in primary ILC of the breast associate with poor survival outcome in large public datasets.MethodsWe performed in silico comparison of ERBB2 non-amplified cases of ER+ stage I-III primary ILC (N=279) and invasive ductal carcinoma (IDC, N=1,301) using METABRIC, TCGA and MSK-IMPACT information. Activating mutations amenable to HER2-directed therapy with neratinib were identified using existing functional data from in vitro cell line and xenograft experiments. Multivariate analysis of 10-year overall survival (OS) with tumor size, grade and lymph node status was performed using a Cox regression model. Differential gene expression analyses by ERBB2 mutation and amplification status was performed using weighted average differences and an in-silico model of response to neratinib derived from breast cancer cell lines.ResultsILC tumors comprised 17.7% of all cases in the dataset but accounted for 47.1% of ERBB2-mutated cases. Mutations in ERBB2 were enriched in ILC versus IDC cases (5.7%, N=16 vs. 1.4%, N=18, pERBB3 mutations were not enriched in ILC (1.1%, N=3 vs. 1.8%, N=23; p=0.604). Median OS for patients with ERBB2-mutant ILC tumors was 66 months versus 211 months for ERBB2 wild-type (p=0.0001), and 159 vs. 166 months (p=0.733) for IDC tumors. Targetable ERBB2 mutational status was an independent prognostic marker of 10-year OS – but only in ILC (hazard ratio, HR=3.7, 95% CI 1.2–11.0; p=0.021). Findings were validated using a novel ERBB2 mutation gene enrichment score (HR for 10-year OS in ILC=2.3, 95% CI 1.04–5.05; p=0.040).ConclusionsTargetable ERBB2 mutations are enriched in primary ILC and their detection represents an actionable strategy with the potential to improve patient outcomes. Biomarker-led clinical trials of adjuvant HER-targeted therapy are warranted for patients with ERBB2-mutated primary ILC.

In this paper, we first highlight three major challenges to large-scale adoption of deep learning at the edge: (i) Hardware-constrained IoT devices, (ii) Data security and privacy in the IoT era, and (iii) Lack of network-aware deep learning algorithms for distributed inference across multiple IoT devices. We then provide a unified view targeting three research directions that naturally emerge from the above challenges: (1) Federated learning for training deep networks, (2) Data-independent deployment of learning algorithms, and (3) Communication-aware distributed inference. We believe that the above research directions need a network-centric approach to enable the edge intelligence and, therefore, fully exploit the true potential of IoT.

This paper presents a hardware prototype and a framework for a new communication-aware model compression for distributed on-device inference. Our approach relies on Knowledge Distillation (KD) and achieves orders of magnitude compression ratios on a large pre-trained teacher model. The distributed hardware prototype consists of multiple student models deployed on Raspberry-Pi 3 nodes that run Wide ResNet and VGG models on the CIFAR10 dataset for real-time image classification. We observe significant reductions in memory footprint (50×), energy consumption (14×), latency (33×) and an increase in performance (12×) without any significant accuracy loss compared to the initial teacher model. This is an important step towards deploying deep learning models for IoT applications.

Additional file 2: Supplemental Tables. (S1) list of genes in the novel (N = 20) and established (Desmedt et al [31], N = 24) gene signatures of HER2 activity; (S2 and S3) top 500 DEGs by WAD score stratified into genes that are up-regulated (S2) and down-regulated (S3) for METABRIC ERBB2 amplified vs. non-amplified (ERBB2amp), METABRIC ERBB2mut vs. wild-type (ERBB2mut) and oncERBB2mut vs. wild-type (oncERBB2mut), and TCGA HER2+ vs. HER2- (TCGA HER2+).

Additional file 1: Supplemental Figure S1. (A) 10-year OS KM plot stratified by individual study cohort: blue – METABRIC, solid gray – TCGA, dotted gray – MSK-IMPACT; (B) 10-year OS KM plot stratified by study category: blue – long follow-up (METABRIC), gray – short follow-up (TCGA and MSK-IMPACT combined); Logrank p = 0.225.

Traditional drug delivery systems are often inefficient and imprecise as they mainly rely on diffusion processes. For example, the blood circulation system is often used to transport and deliver drugs throughout the body; due to the randomness involved in molecular diffusion, this approach can cause inefficient and imprecise delivery of drugs. To address this problem, we propose an autonomous and adaptive bacteria-based drug delivery system that integrates bacterial chemotaxis and quorum sensing in order to deliver drugs efficiently and precisely at various location in the human body. More specifically, we design a synthetic AND gate that enables bacteria to detect molecules produced by tumors and release the appropriate drugs in a coordinated manner; the system can also dynamically adjust the amount of drugs released based on tumor size and activity level. Our experimental results show that the proposed system can be effectively used for cell-based therapeutics while preventing drug overuse and multi-drug resistance.

DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs' test performance. To this end, we introduce a new metric called NN-Mass to quantify how effectively information flows through DNNs. Moreover, we empirically show that NN-Mass also works for other types of skip connections, e.g., for ResNets, Wide-ResNets (WRNs), and MobileNets, which contain addition-type skip connections (i.e., residuals or inverted residuals). As such, for both DenseNet-like CNNs and ResNets/WRNs/MobileNets, our theoretically grounded NN-Mass can identify models with similar accuracy, despite having significantly different size/compute requirements. Detailed experiments on both synthetic and real datasets (e.g., MNIST, CIFAR-10, CIFAR-100, ImageNet) provide extensive evidence for our insights. Finally, the closed-form equation of our NN-Mass enables us to design significantly compressed DenseNets (for CIFAR-10) and MobileNets (for ImageNet) directly at initialization without time-consuming training and/or searching., Accepted at CVPR 2021

Networks are ubiquitous for many real-world problems such as modeling information diffusion over social networks, transportation systems, understanding protein-proteininteractions, human mobility, computational sustainability, among many others. Recently, due to the ongoing Big Data revolution, the fields of machine learning and Artificial Intelligence (AI) have also become extremely important, with AI mostly being dominated by representation learning techniques such as deep learning. However, research at the intersection of network science, machine learning and AI has been mostly unexplored. Specifically, most of the prior research focuses on how machine learning techniques can be used to solve “network” problems such as predicting information diffusion on social networks or classifying blogger interests in a blog network, etc. On the contrary, in this thesis, we answer the following key question: How canwe exploit network science to improve machine learning and representation learning models when addressing general problems? To answer the above question, we address several problems at the intersection of network science, machine learning, and AI. Specifically, we address four fundamental research challenges: (i) Network Science for Traditional Machine Learning, (ii) Representation Learning for Small-Sample Datasets, (iii) Network Science-BasedDeep Learning Model Compression, and (iv) Network Science for Neural Architecture Space Exploration. In other words, we show that many problems are governed by latent network dynamics which must be incorporated into the machine learning or representation learning models.To this end, we first demonstrate how network science can be used for traditional machine learning problems such as spatiotemporal timeseries prediction and application-specific feature extraction. More precisely, we propose a new framework called Network-of-Dynamic Bayesian Networks (NDBN) to address a complex probabilistic learning problem over networks with known but rapidly changing structure. We also propose a new domain-specific network inference approach when the network structure is unknown and only the high-dimensional data is available. We further introducea new network science-based, application-specific feature extraction method called K-Hop Learning. As concrete case studies, we show that both NDBN framework and K-Hop Learning significantly outperform traditional machine learning techniques for computational sustainability problems such as short-term solar energy and river flow prediction, respectively. We then discuss how network science can be used to address general representationlearning problems with high-dimensional and small-sample datasets. Here, we propose a new network community-based dimensionality reduction framework calledFeatureNet. Our approach is based on a new correlations-based network construction technique that explicitly discovers hidden communities in high-dimensional raw data.We show the effectiveness of FeatureNet on many diverse small-sample problems as deep learning typically overfits for such problems. We demonstrate that our techniqueachieves significantly higher accuracy than ten state-of-the-art dimensionality reduction methods (up to 40% improvement) for the small-sample problems. Since a simple correlations-based network alone cannot capture meaningful features for problems like image classification, we focus on deep learning models like Convolutional Neural Networks (CNN). Indeed, in the era of Internet-of-Things (IoT),computational costs of deep networks have become a critical challenge for deploying such models on resource-constrained edge devices. Towards this, model compressionhas emerged as an important area of research. However, when a computationally expensive CNN (or even a compressed model) cannot fit within the memory-budgetof a single IoT-device, it must be distributed across multiple devices which leads to significant inter-device communication. To alleviate the above problem, we propose a new model compression framework called the Network-of-Neural Networks (NoNN) which first exploits network science to partition a large “teacher” model’s knowledge into disjoint groups and then trains individual “student” models for each group. This results in a set of student moduleswhich satisfy the strict resource-constraints of individual IoT-devices. Extensive experiments on five well-known image classification tasks show that NoNN achieves similar accuracy as the teacher model and significantly outperforms the prior art. We also deploy our proposed framework on real hardware such as Raspberry Pi’s and Odroids to demonstrate that NoNN results in up to 12 reduction in latency, and up to 14 reduction in energy per device with negligible loss of accuracy. Finally, since deep networks are essentially a network of (artificial) neurons, networkscience is a perfect candidate to study their architectural characteristics. Hence, we model deep networks from a network science perspective to identify which architecture level characteristics enable models with different number of parameters and layers to achieve comparable accuracy. To this end, we propose new metrics called NN-Massand NN-Density to study the architecture design space of deep networks. We further theoretically demonstrate that (i) For a given depth and width, CNN architectures withhigher NN-Mass achieve lower generalization error, and (ii) Irrespective of number of parameters and layers (but same width), models with similar NN-Mass yield similar test accuracy. We then present extensive empirical evidence towards the above two theoretical insights by conducting experiments on real image classification tasks suchas CIFAR-10 and CIFAR-100. Lastly, we exploit the latter insight to directly design efficient architectures which achieve comparable accuracy to large models ( 97%on CIFAR-10 dataset) with up to 3 reduction in total parameters. This ultimately reveals how model sizes can be reduced directly from the architecture perspective.In summary, in this thesis, we address several problems at the intersection of network science, machine learning, and representation learning. Our research comprehensivelydemonstrates that network science can not only play a significant role but also lead to excellent results in both machine learning and representation learning.

Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained 'teacher' deep network into several disjoint and highly-compressed 'student' modules, without loss of accuracy. Moreover, we propose a network science-based knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for user-defined memory/performance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR-10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24x), performance (up to 12x), and energy per node (up to 14x) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33x reduction in total latency w.r.t. a state-of-the-art model compression baseline., Comment: This preprint is for personal use only. The official article will appear as part of the ESWEEK-TECS special issue and will be presented in the International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS), 2019

To improve precipitation predictions and enable accurate rainfall models for smart water networks, it is imperative to account for multivariate and multiscale precipitation dynamics with long-memory temporal relationships, long-range spatial dependencies, and low-frequency variability. While prior art has motivated the use of complex networks to capture these trends, existing work is limited to specific climate phenomena (such as El Nino) and regions. In this paper we employ a comprehensive assessment of complex networks with respect to multivariate dynamics across multiple temporal and spatial scales (multiscale). Our work is the first to incorporate both carbon emissions and precipitation anomalies data into a multivariate analysis. By effectively substantiating the ability of complex networks to capture multivariate and multiscale climate dynamics, we postulate their potential as reanalysis and assessment tools to enhance regional water models of arbitrary range and granularity, and to eventually enable reliable smart water systems.

Forecasting short-term river flowrate is an important problem for flood planning/preparation and hydropower management. In this paper, we propose K-Hop Learning, a novel feature extraction method, which utilizes network information to learn upcoming river flow peaks 24-hours in advance. We then use these learned features to forecast flowrate 24-hours ahead at various locations in the Ohio River Basin, USA. Our results clearly demonstrate that the predictions using K-Hop Learning are far superior to several traditional timeseries-based, machine learning models such as SVM Regression. Further, in contrast to these existing timeseries-based models, the upcoming river flow peaks are predicted very accurately by our approach. Overall, using K-Hop Learning, we significantly reduce the prediction RMSE by 57--82% depending on the river size.

Traditional drug delivery systems are often inefficient and imprecise as they mainly rely on diffusion processes. For example, the blood circulation system is often used to transport and deliver drugs throughout the body; due to the randomness involved in molecular diffusion, this approach may cause inefficient delivery of drugs. To deal with this problem, we propose an autonomous and adaptive bacteria-based drug delivery system that integrates bacterial chemotaxis and quorum sensing to deliver drugs efficiently and precisely. More specifically, we design a synthetic AND gate that enables bacteria to detect the molecules produced by a particular type of tumor and release the appropriate drugs in a coordinated manner; the system can also dynamically adjust the amount of drugs released based on tumor size and activity. We envision that such a system can be used for cell-based therapeutics while preventing drug overuse and multidrug resistance.

River networks and river basins are considered as one of the natural components in the Water Networks (WN). In this paper, we propose, for the very first time, a network-based model of cloud movement; this can also become part of natural sources of freshwater in the Water Networks. More precisely, we use the cloud network model over the Ohio river basin to estimate the value of cloud fraction for the next day using the current day cloud fraction. The experimental results indicate that our model produces good results and is computationally less intensive. This is in contrast with numerical weather predicting techniques which are highly accurate, but make use of powerful servers. Further, accurate cloud fraction estimation shows that our network model is a correct representation of cloud movement dynamics. We also analyze the network properties and find that the cloud network generated by our algorithm is close to a random graph and not to a small-world network. Finally, we present a thorough analysis of cloud fraction and precipitation in the Ohio river basin. These results are a first step towards modeling the complete hydrological cycle.