Search

There are three main research conducted in this paper, including using deep learning methods with vision-based technologies on Structural Damage Detection (SDD), Structural Health Monitoring (SHM) and progressive collapse study. During the learning and improvement process, many goals of automation in SDD and SHM have been achieved, although there will be a large room for further improvement and development on these studies. In progressive collapse study, remote sensing technologies and data fusion are applied on a field experiment of a real building at the Central Campus of the Ohio State University. The major contributions of this paper are shown as follows:A few comprehensive experimental studies for automated SDD in extreme events using deep learning methods for processing 2D images. In the first study, a 152-layer Residual Network (ResNet) is utilized to identify multiple classes in eight SDD tasks, which include identification of scene levels, damage levels, material types, etc. The proposed ResNet achieved high accuracy for each task while the positions of the damage are not identifiable. In the second study, the existing ResNet and a segmentation network (U-Net) are combined into a new pipeline, cascaded networks, for categorizing and locating structural damage. The results show that the accuracy of damage detection is significantly improved compared to only using a segmentation network. In the third and fourth studies, end-to-end networks are developed and tested as a new solution to directly detect cracks and spalling in the image collections of recent large earthquakes. One of the proposed networks can achieve an accuracy above $67.6\%$ for all tested images at various scales and resolutions, and shows its robustness for these human-free detection tasks.Studies are conducted with a pipeline to automatically track and measure displacements and vibrations of structures or structural components in laboratory and field experiments. This novel framework that uses computer vision and deep learning methods to mimic human vision system for the dynamic performance assessment of in-service infrastructure with various camera placements. On one hand, the static deformations of cylinders and small-scale reinforced concrete beams in the laboratory tests are captured and measured by the proposed framework at first. Then two shaking table tests in the lab are utilized to assess the dynamic performance of the simulated structures. On the other hand, several bridges, including pedestrian, railway, and traffic bridges, are tested for their dynamic performance in field experiments with different camera placements: remote, structure-mounted, and drone-mounted cameras. To remove systematic motions of cameras and to capture the fundamental frequency of these tested structures, two techniques, displacement subtraction and frequency subtraction, are applied. To better understanding of practical applications, critical parameters for camera settings and data processing techniques, such as video frame-rates, window size and locations, and sampling rates on visual data are studied. It shows that not only the vibrations and the frequencies of the simulated structures (i.e., in lab tests) or the in-service structures (i.e., in filed experiments), but also the static deformation of structures or structural components, can be tracked and measured accurately by our proposed framework.The performance of a building in progressive collapse study is monitored and analyzed with the methods developed in previous studies. On one hand, we applied the proposed methods to detect structural damage on this reinforced concrete (RC) structure that was tested, including visual data from cell phones, inside and outside cameras, and drones. The experiments indicate that these solutions for automatic detection of structural damage using deep learning methods are feasible and promising. On the other hand, the proposed methods to measure the deformations and vibrations are utilized to process visual data from outside and inside cameras, and drones. Data from different sources are fused for capturing the performance of the structural elements under the scenarios of the sudden loss of the columns and slabs. The data fusion technique is useful to investigate the characteristics of this framed structure, especially when the traditional sensors such as strain gauges and Linear Variable Displacement Transducers (LVDTs) didn't work in the experiments. Since visual data acquisition and preparation, and techniques of data processing have been inclusively researched for real applications with deep learning, many experimental studies on SDD and SHM are carried out and promising results are obtained in this paper. In addition, the vibration-based technologies or traditional sensors are used as the reference. Our goal is to meet the needs for automatic detection of structural damage and accurate measurements for the performance of the structures or structural elements, thus, the efficiency and effectiveness of these frameworks are tested and analyzed. In summary, these research in this paper indicate that vision-based technologies with deep learning can be applied well on structural engineering domain, and facilitate structural engineers' job by providing reliable data and credential results. The methodologies developed in this paper can fill the gap of research and engineering applications in the future.

At-home exercising strongly predicts physical therapy patient outcomes, underscoring the need for analyzing patient behaviors at-home via remote patient monitoring. Contemporary methods for remote patient monitoring rely on specialized sensors, i.e., Inertial Measurement Units, RGB-Depth cameras, motion capture systems, or stereo vision which are costly and not scalable to all physical therapy patients. Here, we observe a lack of literature using only a monocular RGB camera. In this paper, we demonstrate a skeletal feedback model for at-home exercises using only video acquired from a smartphone camera. We propose models for (i) Patient Performance Evaluation - which classifies the correctness of exercises, and (ii) Guidance - which identifies why the exercise went wrong so the patient can correct themselves. We use these models on our dataset of four common physical therapy exercises labeled by a physical therapist. Our results demonstrate the feasibility of using skeletal data from state-of-the-art 3D human pose estimation models for physical rehabilitation exercise evaluation and guidance. Thus, we enable remote patient monitoring and guidance from a single camera - making it highly cost-effective and scalable.

The continued growth and application of deep learning has resulted in a vast increase in energy and computational requirements. Biologically inspired spiking neural networks (SNNs) and neuromorphic hardware pose one possible solution to this issue. Optimization of these methods, however, remains difficult and less effective compared with that of traditional artificial neural networks (ANNs). A number of methods have been recently proposed to optimize SNNs through the conversion of architecturally equivalent ANNs. However, most benchmarking of these methods has only been done separately through experiments in the respective papers. Therefore, the performance of the solutions is inevitably biased due to the differences in levels and goals of optimization. Moreover, certain papers also relied heavily on architectural improvements to the base ANN which can be separated from the actual method of conversion [1] [2]. In this thesis, we thoroughly evaluate and compare the performance of the major ANN-to SNN conversion solutions based on a new set of performance metrics we proposed. Additionally, we implement expansions to certain methods, allowing for more comprehensive and fair comparisons. Furthermore, the hyperparameters of each method are optimized uniformly to reduce biases towards specific methods. Our implementations and comparisons of SNN solutions are carried out on one-dimensional radar data. To the best of our knowledge, this is the first such effort in the domain of radar applications.

Many techniques for analyzing cardiovascular health rely on cardiac magnetic resonance images that have been segmented to identify various components of the heart. Manually segmenting these images is cumbersome and prone to variability, which calls for the development of accessible automation tools for cardiac researchers. In order to benefit the developing symbiosis between machine learning and medicine, such tools must be accurate, efficient and inferentially transparent. This paper introduces a U-Net-based pipeline for left ventricular segmentation of short-axis CMRs. The U-Net, a Fully Convolutional Network known for its success in biomedical image segmentation, is a natural candidate for our task. This work constitutes the core of a larger-scale project focused on improving human disease models through the acceleration of animal cardiac research. Accordingly, experiments discussed here leverage both human and animal data to explore the efficacy of image processing and model training strategies.This paper focuses on optimizing our U-Nets for resource constrained environments, and demonstrates that these models require only a fraction of their typical convolutional filters. This reduction affords efficiency with the added benefit of explainability by improving the practicality of visualizing learned features. Inference can also be further optimized by pruning trained models without any loss of accuracy or the need for retraining. Specifically, we show that U-Nets with less than 2% of their original parameters train in minutes on a single GPU and achieve Dice scores above 0.95 on multiple CMR datasets. Furthermore, inference on hundreds of images can be performed in seconds on a laptop.

The transfer of facial expressions from people to 3D face models is a classic computer graphics problem. In this paper, we present a novel, learning-based approach to transferring facial expressions and head movements from images and videos to a biomechanical model of the face-head-neck musculoskeletal complex. Specifically, leveraging the Facial Action Coding System (FACS) as an intermediate representation of the expression space, we train a deep neural network to take in FACS Action Units (AUs) and output suitable facial muscle and jaw activations for the biomechanical model. Through biomechanical simulation, the activations deform the face, thereby transferring the expression to the model. Our approach has advantages over previous approaches. First, the facial expressions are anatomically consistent as our biomechanical model emulates the relevant anatomy of the head, neck, and face. Second, by training the neural network using data generated from the biomechanical model itself, we eliminate the manual effort of data collection for expression transfer. The success of our approach is demonstrated through experiments involving the transfer of a range of expressive facial images and videos onto our biomechanical face-head-neck model.

Recent advances in Machine Learning (ML) and Deep Learning (DL) techniques have triggered key success stories in many application domains like Computer Vision, Speech Comprehension and Recognition, and Natural Language Processing. Large-scale Deep Neural Networks (DNNs) are primary drivers of these success stories. However, training complex DNN architectures that consist of millions of trainable parameters is compute-intensive. Training is done using a large number of examples (training data set) and can take from weeks to months to achieve state-of-the-art prediction capabilities (accuracy). To achieve higher accuracy, making the DNN deeper and larger has become a common strategy but it also leads to a significantly bigger memory footprint. Thus, DNN training is not only compute-intensive but also a memory-hungry process requiring gigabytes of memory.To accelerate the process of large-scale DNN training, this dissertation is focused on designing high-performance systems that can exploit thousands of CPUs and GPUs for faster training. The novel approach presented in this work is called the co-design of high-perfo-rmance communication middleware and DL frameworks. Co-design is necessary because of the complexity of the overall execution stack for modern DL frameworks. Broadly, this stack consists of many layers, which start from the application layer followed by the DL framework layer (e.g. TensorFlow). The next layer in the stack is the distributed training middleware layer (e.g. Horovod) that connects a DL framework to an underlying communication middleware (e.g. a Message Passing Interface (MPI) library). Finally, the communication middleware layer sits directly on top of the parallel hardware that consists of multiple CPU/GPU nodes connected with a high-performance network. \textit{The complexity of this stack coupled with inefficient existing approaches to utilize it has led to several problems}: First, there is a lack of credible and systematic performance studies of Machine Learning (ML) and Deep Learning (DL) frameworks. This is partly because frameworks are being designed by scientists who are experts in AI and ML but not in high-performance systems. Second, state-of-the-art ML/DL frameworks are built for productivity but not necessarily performance and scalability. Frameworks either support training models on a single CPU/GPU only, which is not enough for training in a reasonable time, or they only offer rudimentary support for distributed training, which is not sufficiently scalable on modern High Performance Computing (HPC) systems. Third, larger and deeper models being proposed do not scale well even though they offer better prediction capabilities (or accuracy). This is because these models have a much higher number of parameters (order of millions) that lead to large communication buffers and eventually to excessive synchronization overheads for distributed training. Fourth, the established communication middleware (e.g. MPI) that we wish to use for distributed training of DNNs has only been optimized for scientific applications. Thus, fundamental primitives in the communication middleware like Broadcast, Reduce, and Allreduce need to be redesigned to better support the emerging DL workloads. Fifth and final, because of the fundamental memory limit, emerging DNNs that are larger than the CPU's or the GPU's memory cannot be trained without explicit memory management schemes and novel model partitioning strategies.The proposed co-design approach alleviates these performance inefficiencies, leads to a white-box strategy for performance optimization, ensures scalability of DL frameworks to thousands of CPUs and GPUs, and acts as a path forward for investigation of next-generation DNNs. The key idea is that one should neither develop DL frameworks in isolation nor without matching the requirements of the framework to the capabilities of the underlying software and hardware stack. Instead, a comprehensive co-design across multiple layers should be employed so that the entire stack is optimized in a connected and coherent fashion. To address the problems in existing approaches, we investigate how HPC techniques and optimizations can be brought to DL systems. Broadly, an iterative approach is performed: we start with performance evaluation followed by the design and development of an actual system. Once a particular system is functional, we develop new techniques and/or apply HPC optimizations like faster collective communication and overlap of computation and communication to further improve its performance. The proposed systems that have been developed by following this approach are: 1) S-Caffe, 2) OC-DNN, and 3) HyPar-Flow. All these systems have been fully developed, scaled to hundreds of CPUs and GPUs on various HPC systems like ORNL Summit and TACC Frontera, and have been made publicly available through software releases and published papers.

Smart speakers like the Amazon Echo and Google Home offer users the convenience of hands-free information retrieval by utilizing the power of virtual assistant technology and the extensive cloud resources provided by their platform's service provider. These devices have saturated the market and millions of users worldwide have welcomed them to their homes, placing them on their kitchen counters, bedside tables and living room stands. Characteristics about a smart speaker’s encrypted network traffic can be used by an attacker to infer what a user is saying to their smart speaker. In this paper, we propose an encrypted network traffic analysis attack on privacy. This attack uses an ensemble of deep learning models to predict what voice command a user issues to their smart speaker. We introduce a data collection platform which we designed specifically for smart speakers. This smart speaker automated crawler allows an attacker to collect the data needed to train the attack models. An evaluation of the voice command fingerprinting attack presented in this work shows that it can be used to correctly predict over 91% of voice commands issued by a user.

Speculation about artificial intelligence and big data has become commonplace. Foremost among these discussions is the potential for these technologies to displace the value of human labor. However, these discussions have become untethered from the history and development of the technologies in question. This paper analyzes paradigms of artificial intelligence to reveal that they are closely tied to different types of logical inference. The properties of computing machinery provide them with an advantage at deductive and inductive tasks when compared with human cognition. But, the understudied method of Peircean abduction poses several problems for computing machinery. Moreover, Peircean abduction, also called “creative abduction,” bears resemblance to important themes in the history of humanism. Human cognition handles the issues of Peircean abduction with remarkable ease, suggesting that humans will maintain a comparative advantage at this type of logical inference for the foreseeable future.

Binary diffing analysis quantitatively measures the differences between two given binaries and produces fine-grained basic block matching. It has been widely used to enable different kinds of critical security analysis. However, all existing program analysis and learning based techniques suffer from low accuracy, poor scalability, coarse granularity especially on COTS binaries which did not contains complete debug information. On the other hands, some learning based approaches require extensive labeled training data to function, so that precise labelled and representative dataset is needed to obtain great results. To surmount such limitations, in this paper, we come up with a novel learning based code representation generation approach to solve the binary diffing problem. We rely only on the code semantic information as well as the program-wide control flow structural information to generate block embeddings without supporting of any debug information. Furthermore, we propose a K-hop greedy matching algorithm to find the optimal diffing results using the generated block representations. We implement a prototype called DeepBinDiff and evaluate its effectiveness and efficiency with large number of binaries and real-world vulnerabilities. The results show that our tool could outperform the state-of-the-art binary diffing tools by large margin for both cross-version and cross-optimization level diffing. A case study for OpenSSL using real-world vulnerabilities further demonstrates the usefulness of our system.

P300 is an event-related potential evoked as a response to external stimuli. The P300-speller is a widely used BCI that has proven to be a reliable method in enabling people who cannot communicate via normal methods. Improving single-trial P300 classification helps increase communication bandwidth as it reduces the averaging process necessary to reduce noise. Deep learning approaches in this area have become more popular recently [2][3]. Certain architectural choices result in a better classifier. In this paper, we performed a study of EEGNet [2] as it performed well in our experiments. We report some improvements to the model. We also confirm that these suggestions improve other CNN based models by showing improved results with CNN-1 proposed in [4] on BCI Competition III Dataset II. We perform knowledge distillation of the resultant linear model into a logistic regression model and show that the model learns information similar but superior to LDA. We show that the logistic regression model obtained this way outperforms more complex models and is easy to interpret.

In sentence modeling, neural network approaches that leverage the tree-structural features of sentences have recently achieved state-of-the-art results. However, such approaches require complex architectures and are not easily extensible to document modeling. In this paper, we propose a very simple convolutional neural network model that incorporates Part-Of-Speech tag information (PCNN). While our model can be easily extensible to document modeling, it shows great performance on both sentence and document modeling tasks. As a result of sentiment analysis and question classification tasks, PCNN achieves the performance comparable to that of other more complex state-of-the-art models on sentence modeling and outperforms them on document modeling. We also make efforts to explore the effect of POS tag embeddings more thoroughly by conducting various experiments.

Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision. But despite their popularity, the ``correctness'' of the implicitly-learned attention maps has only been assessed qualitatively by visualization of several examples. In this paper we focus on evaluating and improving the correctness of attention in neural image captioning models. Specifically, we propose a quantitative evaluation metric for how well the attention maps align with human judgment, using recently released datasets with alignment between regions in images and entities in captions. We then propose novel models with different levels of explicit supervision for learning attention maps during training. The supervision can be strong when alignment between regions and caption entities are available, or weak when only object segments and categories are provided. We show on the popular Flickr30k and COCO datasets that introducing supervision of attention maps during training solidly improves both attention correctness and caption quality.

In this paper, we propose an approach to classify action sequences. We observe that in action sequences the critical features for discriminating between actions occur only within sub-regions of the image. Hence deep network approaches will address the entire image are at a disadvantage. This motivates our strategy which uses static and spatio-temporal visual cues to isolate static and spatio-temporal regions of interest (ROIs). We then use weakly supervised learning to train deep network classifiers using the ROIs as input. More specifically, we combine multiple instance learning (MIL) with convolutional neural networks (CNNs) to select discriminative action cues. This yields classifiers for static images, using the static ROIs, as well as classifiers for short image sequences (16 frames), using spatio-temporal ROIs. Extensive experiments performed on the UCF101 and HMDB51 benchmarks show that both these types of classifiers perform well individually and achieve state of the art performance when combined together. We also show qualitatively that our ROIs (selected by the algorithms) capture the most relevant parts of the image sequences.