Search

Bayesian inference in its simplest forms is the act of moving from sample data to generalisations with a measurable degree of certainty through the utilisation of probability theory generally and Bayes' theorem specifically. Probabilistic Graphical Models (PGMs) on the other hand provide means to describe complex multivariate systems as joint probability distributions over multidimensional space yet pertaining a graph's unique ability to describe relations between system variables naturally and intuitively. Bayesian Networks (BNs) are a special type of PGMs that focuson modelling conditional joint probability distributions over multivariate domains, Dynamic Bayesian Networks (DBNs) are an extension of BNs to model conditionaljoint probability distributions over sequences of random variable spanning the time domain. The work in this dissertation investigates the potential and power of using Dynamic Bayesian Networks and Bayesian inference techniques for the sole purpose of reasoning over uncertainty in three challenging domains. These are the domains of human activity recognition and detection based multi-object tracking, where recognising human activities and tracking humans plays a fundamental role in the fields of machine learning and computer vision due to its potential in applications such human computer interaction and video surveillance systems, yet they are considered as very challenging time series classification problems, that involves the use of large streams of sensor data to enable action recognition and tracking. Another domain is anomaly detection which involves finding patterns of data that do not conform to what is normal, which can be very advantageous in critical system applications in fields such as network security and bank fraud detection. The first part of this dissertation will go through the theoretical background of Bayesian Networks and Dynamic Bayesian Networks, different aspects will be explored, from representation to parameter estimation and inference techniques. Next a probabilistic modelling and inferencing framework using DBNs and machine learning techniques is proposed, for classifying and predicting human activities from a temporal perspective. This model is first implemented on an aeroplane pilot flying activity domain, through the use of a software simulator. Next, a framework for anomaly detection through means of classification using DBNs is proposed. Afterwards this framework is extended for anomaly detection in the temporal domain using DBNs. These models are then implemented for detecting pilot errors, utilising the previous DBN pilot activity model along with different probabilistic inference techniques. The results are then evaluated and analysed. At the end, a novel Multiple Object Tracking (MOT) algorithm based on the Tracking-by-detection paradigm using DBNs is proposed. The tracker is tested using the MOT15, MOT16, MOT17 challenge benchmarks of video sequences. This algorithm is further enhanced by the introduction of additional affinity measures, in the form of a state of the art feature extractor built using a residual neural network and trained on the MARS person re-identification database. Finally, we improve the tracker performance by the use of a state of the art object detector (YOLOV5), custom trained on Google's Open Images dataset. Results are analysed and evaluated using the MOTChallenge Evaluation Kit, and then compared to other methods.

This thesis intends to contribute towards Computer Vision (CV)-based functional assessment of physically impaired persons involving Activities of Daily Living (ADL). Patients rehabilitating from conditions like stroke, spinal cord injury, Parkinson's disease, among other symptoms experience difficulty in physical movements which limit their functional ability and independence. Such patients usually undergo physical rehabilitation programs which require constant monitoring of their (ADL) to record their progress. This monitoring is currently done by Healthcare professionals which is not only labour intensive and expensive but also error-prone. The study aims to address this problem by proposing CV-based methods for detecting ADL which can be used for functional assessment of patients automatically from recorded videos. This has the potential of lessening the labour-intensive manual annotation and continuous human observation, resulting in reduction of the overall cost of rehabilitation for such patients. While the current CV literature is replete with methods for human activity or ADL recognition, there are very few that aim to detect impairment-specific versions of ADL executed or exhibited by physically impaired persons. A part of the problem lies in the unavailability of labelled datasets for such activities making it difficult for researchers to develop the necessary methods to detect and recognise them. In recent years, the field of CV has seen increasing use of Deep Learning (DL) methods for ADL recognition. However, DL-based models are almost exclusively data-driven and require very large datasets often containing thousands of human activity videos to successfully train and validate. The current study attempts to address this issue by developing and contributing a novel multi-label dataset that includes labelled videos of several categories of normal and impairment-specific executions of ADL similar to what is exhibited by normal persons and physically impaired persons, respectively. This dataset has been developed under the guidance of an Occupational Therapist providing the necessary credibility to the entire exercise. This is an inter-disciplinary research involving CV, Artificial Intelligence and Health and Social Care. One of the key focus of this thesis is to contribute towards the advancement of research in DL. To this end, the thesis presents three novel human activity recognition models based on DL. The first model uses an intelligent or learn-able pooling method based on Fisher Vector (FV) to propose a better alternative to the standard statistical pooling method known as Global Average Pooling (GAP). In this model, FV with activity-aware pooling method is integrated within the DL model to semantically cluster the structural information contained in Attention-focused hidden LSTM states in a novel manner. It leads the network to pool more relevant information in contrast to normally used statistical pooling methods. The model achieves better performance than the state-of-the-art video-based models. The second activity recognition model introduces a novel 3D human body-pose encoding method. The body-pose encoding algorithm learns the spatial arrangement between various body joints to present an enriched pose information to the network for improved performance. The algorithm also encodes the frame-wise positions of body joints and presents a temporally enriched representation for each joint, individually. The pose encoding algorithm coupled with an Attention mechanism is presented as a part of combined video and pose-based activity recognition model that achieves state-of-the-art results on three challenging benchmark datasets. The third is a pure human body pose-based lightweight DL model based on Temporal Convolution Networks. The 5 spatial-temporal two-stream model takes advantage of the pose encoding algorithm and the learnable pooling method introduced earlier to impact the model performance, positively. The model is not only able to recognise an ADL, but also discriminate between the normal and different physical impairment-specific variations of the same ADL when evaluated on the multi-label dataset. Thus, it fulfills the main research aim. To the best of my knowledge, this is an unique inter-disciplinary research that attempts to recognise physical impairment-specific ADL through multi-label video analysis and recognition. In addition to the three activity recognition models, the thesis presents a mobile-based DL approach for human pose estimation. The model introduces a novel Split-Stream architecture as an alternative to the standard GAP method present towards the end of many DL models. The thesis also presents a critical review of existing research on CV-based rehabilitation and assessment. The review proposes its own taxonomy and analyses articles from a CV perspective compared to other reviews that mainly focus on the clinical perspective. The literature review, the mobile-based human body-pose estimation model, the multi-label dataset and the three human activity recognition models are the major contributions of this inter-disciplinary research.

This thesis describes the classification human activity for the purpose of detecting falls, later extended to multiple activities. This was done with the final intention of implementing a robotic companion for older persons that can provide a certain level of automated care in case of some sort of emergency. The complexity of this work, combined with restrictions of the robot, motivated a creation of an infrastructure abstraction to allow deferred (decentralized) processing. The initial work was done by implementing classifiers that work use pre-processed skeleton data extracted from RGBD sensors and implements some steps in order to make classification robust to changes. RGB-D classification first focused on falling detection and then extended into general activities which could be classified from skeleton data. A later attempt used CNNs for classification of video footage of activities. All of those algorithms were modified to output classifications in real time. Results achieved were around 90% in accuracy for a simple fall vs. not fall activity in the TST fall v2 dataset, 70% global combined accuracy for the 12 actions of the CAD60 using skeleton data and 75% accuracy for the 51 actions on the HMDB51, all of those showing close to state-of-the-art performance on datasets. On new activity data based on skeletons and video, however, results were less encouraging with 33.5% accuracy on skeleton data and 37.9% accuracy based on video. While these results do not allow for a robotic platform to perform action detection currently, the overarching structure of systems necessary to execute it was demonstrated and used successfully, opening up doors for future research using more complex systems.

Emerging mobile and IoT technologies, in conjunction with the progress which has been made in the field of Artificial Intelligence, are paving the way to novel smart technology based applications capable of delivering innovative e-health solutions. Despite the level of maturity reached by these solutions to date both in simulated and controlled environments, several studies have highlighted their inability to adapt to the new conditions they are faced with when they are deployed in a free-living uncontrolled environment, and in particular when dealing with new subjects. Within this context, the main contributions of this Thesis can be summarised as follows. Firstly, a flexible software architecture supporting the in situ adaptation of models has been proposed. Consequently, personalisation methods have been investigated in an attempt to identify a viable approach to implementing a model's adaptation based on the characteristics of a final target subject. As a result of this investigation, a personalisation method has been proposed and evaluated on a large publicly available real-world dataset. Results obtained on the dataset including data from 57 subjects confirmed that the personalised model outperforms its generic equivalent improving the recognition balanced accuracy value by almost 20%, from 55.60% to 74.81%. Finally, the proposed elements within this body of research have been assembled with the aim of building a flexible software architecture capable of supporting the application of smart technologies within a behaviour change intervention framework for the monitoring of physical activities in older adults.

Scientific problems are solved by finding the optimal solution for a specific task. Some problems can be solved analytically while other problems are solved using data driven methods. The use of digital technologies to improve the transportation of people and goods, which is referred to as intelligent mobility, is one of the principal beneficiaries of data driven solutions. Autonomous vehicles are at the heart of the developments that propel Intelligent Mobility. Due to the high dimensionality and complexities involved in real-world environments, it needs to become commonplace for intelligent mobility to use data-driven solutions. As it is near impossible to program decision making logic for every eventuality manually. While recent developments of data-driven solutions such as deep learning facilitate machines to learn effectively from large datasets, the application of techniques within safety-critical systems such as driverless cars remain scarce. Autonomous vehicles need to be able to make context-driven decisions autonomously in different environments in which they operate. The recent literature on driverless vehicle research is heavily focused only on road or highway environments but have discounted pedestrianized areas and indoor environments. These unstructured environments tend to have more clutter and change rapidly over time. Therefore, for intelligent mobility to make a significant impact on human life, it is vital to extend the application beyond the structured environments. To further advance intelligent mobility, researchers need to take cues from multiple sensor streams, and multiple machine learning algorithms so that decisions can be robust and reliable. Only then will machines indeed be able to operate in unstructured and dynamic environments safely. Towards addressing these limitations, this thesis investigates data driven solutions towards crucial building blocks in intelligent mobility. Specifically, the thesis investigates multimodal sensor data fusion, machine learning, multimodal deep representation learning and its application of intelligent mobility. This work demonstrates that mobile robots can use multimodal machine learning to derive driver policy and therefore make autonomous decisions. To facilitate autonomous decisions necessary to derive safe driving algorithms, we present an algorithm for free space detection and human activity recognition. Driving these decision-making algorithms are specific datasets collected throughout this study. They include the Loughborough London Autonomous Vehicle dataset, and the Loughborough London Human Activity Recognition dataset. The datasets were collected using an autonomous platform design and developed in house as part of this research activity. The proposed framework for Free-Space Detection is based on an active learning paradigm that leverages the relative uncertainty of multimodal sensor data streams (ultrasound and camera). It utilizes an online learning methodology to continuously update the learnt model whenever the vehicle experiences new environments. The proposed Free Space Detection algorithm enables an autonomous vehicle to self-learn, evolve and adapt to new environments never encountered before. The results illustrate that online learning mechanism is superior to one-off training of deep neural networks that require large datasets to generalize to unfamiliar surroundings. The thesis takes the view that human should be at the centre of any technological development related to artificial intelligence. It is imperative within the spectrum of intelligent mobility where an autonomous vehicle should be aware of what humans are doing in its vicinity. Towards improving the robustness of human activity recognition, this thesis proposes a novel algorithm that classifies point-cloud data originated from Light Detection and Ranging sensors. The proposed algorithm leverages multimodality by using the camera data to identify humans and segment the region of interest in point cloud data. The corresponding 3-dimensional data was converted to a Fisher Vector Representation before being classified by a deep Convolutional Neural Network. The proposed algorithm classifies the indoor activities performed by a human subject with an average precision of 90.3%. When compared to an alternative point cloud classifier, PointNet[1], [2], the proposed framework out preformed on all classes. The developed autonomous testbed for data collection and algorithm validation, as well as the multimodal data-driven solutions for driverless cars, is the major contributions of this thesis. It is anticipated that these results and the testbed will have significant implications on the future of intelligent mobility by amplifying the developments of intelligent driverless vehicles.

Securing smartphones has increasingly become inevitable due to their massive popularity and significant storage and access to sensitive information. The gatekeeper of securing the device is authenticating the user. Amongst the many solutions proposed, gait recognition has been suggested to provide a reliable yet non-intrusive authentication approach - enabling both security and usability. While several studies exploring mobile-based gait recognition have taken place, studies have been mainly preliminary, with various methodological restrictions that have limited the number of participants, samples, and type of features; in addition, prior studies have depended on limited datasets, actual controlled experimental environments, and many activities. They suffered from the absence of real-world datasets, which lead to verify individuals incorrectly. This thesis has sought to overcome these weaknesses and provide, a comprehensive evaluation, including an analysis of smartphone-based motion sensors (accelerometer and gyroscope), understanding the variability of feature vectors during differing activities across a multi-day collection involving 60 participants. This framed into two experiments involving five types of activities: standard, fast, with a bag, downstairs, and upstairs walking. The first experiment explores the classification performance in order to understand whether a single classifier or multi-algorithmic approach would provide a better level of performance. The second experiment investigated the feature vector (comprising of a possible 304 unique features) to understand how its composition affects performance and for a comparison a more particular set of the minimal features are involved. The controlled dataset achieved performance exceeded the prior work using same and cross day methodologies (e.g., for the regular walk activity, the best results EER of 0.70% and EER of 6.30% for the same and cross day scenarios respectively). Moreover, multi-algorithmic approach achieved significant improvement over the single classifier approach and thus a more practical approach to managing the problem of feature vector variability. An Activity recognition model was applied to the real-life gait dataset containing a more significant number of gait samples employed from 44 users (7-10 days for each user). A human physical motion activity identification modelling was built to classify a given individual's activity signal into a predefined class belongs to. As such, the thesis implemented a novel real-world gait recognition system that recognises the subject utilising smartphone-based real-world dataset. It also investigates whether these authentication technologies can recognise the genuine user and rejecting an imposter. Real dataset experiment results are offered a promising level of security particularly when the majority voting techniques were applied. As well as, the proposed multi-algorithmic approach seems to be more reliable and tends to perform relatively well in practice on real live user data, an improved model employing multi-activity regarding the security and transparency of the system within a smartphone. Overall, results from the experimentation have shown an EER of 7.45% for a single classifier (All activities dataset). The multi-algorithmic approach achieved EERs of 5.31%, 6.43% and 5.87% for normal, fast and normal and fast walk respectively using both accelerometer and gyroscope-based features - showing a significant improvement over the single classifier approach. Ultimately, the evaluation of the smartphone-based, gait authentication system over a long period of time under realistic scenarios has revealed that it could provide a secured and appropriate activities identification and user authentication system.

The field of Human Activity Recognition (HAR) has witnessed remarkable advancements in the past decade, fueled by advancements in mobile devices, sensors, and computational methods. HAR has found widespread applications in areas such as mobile health monitoring, disorder diagnosis, personal assistance, and smart environments. However, despite significant progress, HAR faces critical learning challenges that hinder its deployment in real-world scenarios. The prevailing approach in HAR relies on offline data collected under controlled laboratory settings, which fails to capture the dynamic nature of real-world environments and individuals’ behaviors. Moreover, the heterogeneity of sensor devices, specifications, and placements introduces further challenges for long-term deployment. To address these shortcomings, lifelong adaptive learning or continual learning, in HAR, becomes crucial to enable large-scale, long-term, and sustainable activity recognition systems. This dissertation aims to tackle the challenges of lifelong adaptive learning in HAR, focusing on three key components: ground truth annotation with minimal user burden, distribution shifts in HAR, and continual model adaptation. The first component tackles the time-consuming task of acquiring ground truth information by exploring active learning in an effort to minimize data annotation effort and in turn user burden. The second component delves into the problem of distribution shifts introduced by device heterogeneity, sensor placements, and contextual environments, and their impact on model generalizability. The third component explores incremental model adaptation to accommodate changes in behavior and environment as well as introducing new activity classes while mitigating catastrophic forgetting of previously learnt information. A proposed continual learning framework, LAPNet, addresses such challenges. To drive new research in this area, we also introduce a large-scale, stronglylabelled, multi-device, and multi-location human activity dataset collected from inertial sensors, MotionPrint. This dataset was collected from 50 people, time-aligned across six locations, with sampling rates up to 800 Hz, resulting in 200 cumulative hours and 408 million samples of data. Our comprehensive analysis reveals a significant challenge in constructing lifelong deployable motion models that can effectively generalize to diverse sensor locations. This observation presents a valuable opportunity for the machine learning community to engage in new research endeavors focused on developing adaptable and generalizable learning algorithms. Addressing the issue of sensor heterogeneities is a critical aspect in realizing our vision of lifelong learning for HAR. Furthermore, we introduce a pioneering line of research, cross-location data synthesis. This research explores the problem of generating synthetic motion data in one location using available data from another location. The work presented in this dissertation contributes to pushing the field of HAR towards long-term sustainable deployment in real-world settings. By addressing the challenges of achieving practical deployable recognition systems, the research provides insights and methodologies for developing personalized and adaptable activity recognition systems that can continuously learn and adapt to evolving circumstances, enhancing their usability and effectiveness in practical applications. In addition, the release of MotionPrint lowers the entry barrier and facilitates progress in the field of lifelong adaptive learning for HAR, fostering the development of innovative techniques and impactful applications in ubiquitous computing and interactive systems.

With advances in wearable and mobile computing technology, a significant body of work has been dedicated to the field of automated dietary monitoring (ADM). The ability to detect eating activities in naturalistic environments steadily improved over the past decade. Nonetheless, mainstream adoption of the technologies has been hampered by obtrusive form factors (e.g., neck band, headphones, chest band) and high false positive rates in naturalistic settings. This dissertation addresses such problems through small-scale (

Stroke is a leading cause of disability, with over 80% of patients experiencing chronic upper extremity (UE) impairments that impede daily activities and degrade quality of life. While wearable sensors in the form of step counters have foundapplication in stroke rehabilitation, effective home rehabilitation of the UE using wearable sensors remains an open question. This dissertation focuses on three research questions in UE rehabilitation using a wrist-worn wearable inertial measurement unit (IMU): (1) identifying hand movements, (2) assessing the quality of movement experience (QOME) in daily life, and (3) identifying ways to improve UE QOME with wearable feedback.We begin by reviewing existing wearable sensor technologies for at-home rehabilitation, setting the stage for our exploration of hand movement identification and arm movement quality assessment. We then introduce a spectrogram-basedconvolutional neural network (CNN) algorithm for hand movement recognition using a single wrist-worn inertial measurement unit (IMU). Our working hypothesis was that we could use machine learning to identify active flexion and extension of the fingers/wrist based on the vibrational patterns produced at the distal end of the forearm. Using wristworn IMU recordings from 22 individuals with a stroke, we found we could identify the occurrence of finger/wrist movements with approximately 75% accuracy. Thus, ringless sensing of finger/wrist movement occurrence is feasible using wrist-worn IMUs, opening up new avenues for hand-related healthcare applications.Subsequently, we posited that it may be more effective to encourage an increase in beneficial patterns of movement (i.e. QOME), rather than simply the overall amount of movement. As a first step toward this goal, we sought to identify statisticalcharacteristics of daily arm movements that become more prominent as arm impairment decreases. Using the same data set as for the hand movement study, we identified several measures that increased as UE Fugl-Meyer (UEFM) score increased: forearm speed, forearm postural diversity (quantified by kurtosis of the tilt-angle), and forearm postural complexity (quantified by sample entropy of tilt angle). Dividing participants into severe, moderate, and mild impairment groups, we found that forearm postural diversity and complexity were best able to distinguish the groups (Cohen’s D = 1.1, and 0.99, respectively) and were also the best subset of predictors for UEFM score. Based on these findings and a large body of research in motor learning that indicates the importance of challenging and variable movement practice, we posit that encouraging people to achieve more forearm postural diversity and complexity will improve QOME and therefore will be therapeutically beneficial.Finally, we sought to identify practical ways to improve QOME with wearable feedback. In an experiment with unimpaired individuals, we examined specific exercises, chosen from a candidate list of common exercise activities, to find whichexercises create a high level of movement diversity and complexity. Engaging in conventional rehabilitation therapy exercises created high values for forearm postural diversity but not complexity, as measured with a wrist-worn IMU. Playing the card game Speed and exercising with a commercial, gamified, home exercise sensor system produced the highest values for both postural diversity and complexity. Then, we designed an implementation for providing real-time, wearable feedback based on diversity and complexity, working toward a randomized controlled trial to assess the efficacy of QOME feedback compared to quantity feedback alone. We introduced the concept of "Quality Time" to measure the amount of time users perform complex, diverse movements. We propose an adaptive goal-setting strategy based on clinical scores of UE impairment and the statistics of complexity and diversity measured across a wide range of persons with stroke.

This thesis considers the problem of detecting periods of eating in free-living conditions by analyzing wrist motion data collected using sensors embedded within a typical smartwatch. Previous work by our research group included the collection of a dataset containing 354 days of recorded wrist motion data from 351 different people (approximately one day of data per person) [42]. A machine learning model was then trained to classify this wrist motion data as either eating or non-eating [40]. We refer to this model as the group model. Subsequent work in our research group collected approximately ten days of data each for eight new individuals and trained a model for each person solely using their own data [51]. We refer to these models as individual models. It was observed that, in most cases, the individual models outperformed the group model when evaluating the data of their corresponding individual, but at the cost of requiring each individual to collect two weeks of additional data. The novelty of this work is using transfer learning to leverage features learned within the group model and apply them to new individual models to further increase performance and possibly reduce the amount of individual data needed. Two datasets were used in this work. The first was the Clemson All Day (CAD) dataset, which contains 354 days of recorded wrist motion data from 351 different participants (approximately one day of data per participant). The CAD dataset includes a total of 4,680 hours of data, including 1,063 meals. The second dataset used was the Multiday dataset, which is comprised of at least ten days of free-living wrist motion data each for eight individuals. Both datasets were pre-processed using smoothing and normalization techniques. Training samples were then generated using a sliding window approach with a window size of six minutes. All group, individual, and transfer learning models evaluated in this work utilized an identical convolutional neural network (CNN) architecture. For a given window, the classifier generated a value that represented the probability of eating (P(E)) in the window. Entire days of wrist motion data were passed to the network to produce a continuous P(E) sequence for an entire day. This sequence was processed using a dual thresholding technique to locate predicted segments of eating within the recording. In our results, the transfer learning model achieved an eating episode true positive rate (TPR) of 81% with a false positive per true positive ratio (FP/TP) of 1.40. Compared to the individual model, this was a 6% decrease in episode TPR but a 43% improvement in FP/TP. The transfer learning model showed a time weighted accuracy (AccW) of 80%, which was only a 1% decrease relative to the individual model. After removing an outlier from the Multiday dataset and rerunning our experiments, the transfer learning model showed an episode TPR of 86% with an FP/TP of 1.34. Compared to the individual model, this was only a 3% decrease in TPR and a 46% improvement in FP/TP. By excluding the outlier, the transfer learning model also showed an 83% AccW, which was a 1% increase relative to the individual model. Furthermore, the transfer learning model was able to reduce training times by 12% compared to the individual model. In conclusion, we were able to find evidence that transfer learning could be utilized in order to improve individualized eating detection models by increasing weighted accuracy and decreasing false detections.

In recent years there has been an explosion in the availability and use of mobile, wearable, and Internet-of-Things (IoT) devices which generate vast volumes of sensory data. Moreover, deep learning techniques have shown excellent performance on a wide range of tasks such as visual understanding, natural language processing, and speech recognition. Inspired by the success of deep neural networks and the abundance of sensory data, researchers got increasing opportunities to adopt the successful deep learning techniques to various sensing applications. However, real-life sensing tasks usually involve complex spatiotemporal dependencies, which can hardly be captured using only a data-driven approach. The dynamic nature of the sensing environments exacerbates this problem, especially when labeled data are scarce. Unlike visual and natural language data, raw sensor data are opaque to humans, and cannot be labeled retrospectively in a crowdsourced manner. Therefore, training and deploying deep learning models for complex sensing tasks in dynamic scenarios remains a challenge.In this dissertation, we first focus on the problem of complex event detection over heterogeneous sensory data. To address this problem, the designed system requires not only the perception ability for extracting informative and useful features from raw data, but also the reasoning ability for mining and analyzing the dependencies between the higher-level concepts. We propose DeepCEP, a neural-symbolic framework that combines the power of deep learning models and Complex Event Processing (CEP) engines. DeepCEP encodes prior knowledge provided by the users to help make effective inferences over the long-term, state-based complex events. To enable the training of this neural-symbolic system, we introduce Neuroplex, which learns from scratch efficiently in complex event settings, under the guidance of high-level prior knowledge. Compared with mainstream deep learning models, Neuroplex reduces the data annotation requirement by 100 times and speeds up the learning process by four times. Furthermore, we propose the generalized framework DeepSQA for flexible inferencing over heterogeneous sensory data. Given a sensory data context and a task defined on the runtime as natural language questions about the data, DeepSQA can provide an accurate natural language answer. In addition to the DeepSQA, we create SQAGen, a software framework for generating SQA datasets using labeled source sensory data, and also generate OppQA with SQA-Gen for benchmarking different SQA models. Lastly, we investigate the transferability and adaptability of deep learning models under dynamic environments. We propose RecycleML, a unified framework that transfers knowledge among deep learning models of IoT devices with different input modalities, deployments, or tasks. Using human activity recognition as a case study, over our collected CMActivity dataset, we observe that RecycleML reduces the amount of required labeled data by at least 90% and speeds up the training process by up to 50 times.

Classification of specialized human activity datasets utilizing methods not requiring manual feature extraction is an underserved area of research in the field of human activity recognition (HAR). In this thesis, we present a convolutional neural network (CNN)-based method to classify a dataset consisting of subjects lifting an object from various positions relative to their bodies, labeled by the level of back pain risk attributed to the action. Specific improvements over other CNN-based models for both general and activity-based purposes include the use of average pooling and dropout layers. Methods to reshape accelerometer and gyroscope sensor data are also presented to encourage the model’s use with other datasets. When developing the model, a dataset previously developed by the National Institute for Occupational Safety and Health (NIOSH) was used. It consists of 720 total trials of accelerometer and gyroscope data from subjects lifting an object at various relative distances from the body. In testing, 90.6% accuracy was achieved on the NIOSH lifting dataset, a significant improvement over other models tested. Saliency results are also presented to investigate underlying feature extraction and justify the results collected.

This thesis presents applications of deep learning to three domains of sequential data; music, human activity recognition and natural language understanding. In part one, we present work on automatic music generation and musical genre transfer. We show how a model based on two hierarchical LSTMs learns an important concept from music theory. We introduce a VAE model that leverages a novel representation of symbolic music to generate music, create interpolations and mixtures of songs, and even perform genre transfer. We then focus exclusively on genre transfer. To that end we employ a model based on CycleGAN to perform genre transfer solely based on note pitches. We further discuss several extensions to this model as well as a simple content retention metric backed by a user study. In part two we move away from music and instead investigate how inertial sensor data from smartwatches can be leveraged by neural networks to perform activity recognition for strength exercises and swimming. In addition to activity recognition we also tackle repetition counting as well as swimming lane counting. For both tasks we collect large datasets using off-the-shelf smartwatches; in total our datasets features almost 100 participants. We find that sports activity recognition can be done with high accuracy by neural networks. Remarkably, the neural network approaches require little fine-tuning and generalize very well across athletes. We also saw how wearing a second smartwatch on the ankle can help with certain exercises, but is in general not absolutely necessary. We introduce a constrained data collection approach where participants are prompted to perform the next repetition by the smartwatch. This allows to collect more and cleaner data, as well as automate some of the labelling. We verify the ability of models trained on constrained data to generalize to unconstrained settings; such a constrained data collection setting could help with crowd-sourced (unsupervised) data collection. In terms of swimming style recognition, we achieve strong results that are good enough for the use in semi-professional swimming training. In part three shift gears for the last time and arrive at the third sequential data domain; natural language. In particular, we investigate the inner workings of neural networks trained on natural language tasks. We find that the properties of the latent space of character-based sequence-to-sequence models can be manipulated by adding decoders that solve auxiliary language tasks. For instance, adding a POS tagging decoder causes sentences with different syntactic structures to be well separated in the latent space. Interestingly, models that have such a disentangled latent space also perform better in interpolation and latent vector algebra tasks. We then transition from our RNN-based models to the state-of-the-art transformer architecture, and in particular, decoder-only transformers like BERT. In transformers, input tokens aggregate context as they pass through the model, hence the name \emph{contextualized word embeddings}. We challenge and verify common assumptions underlying work trying to understand transformers by interpreting the learned self-attention distributions. We find that the tokens remain largely identifiable as they pass through the model, which gives some support to work that relates hidden tokens to input words. However, at the same time, the information is heavily mixed as tokens aggregate context, and the original input token is often not the largest contributor as measured by gradient attribution. This indicates that BERT somehow keeps track of token identity. Overall, we caution against the direct interpretation of self-attention distributions and suggest more principled methods such as gradient attribution.

The vast proliferation of sensor devices and Internet of Things enables the applications of sensor-based activity recognition. However, there exist substantial challenges that could influence the performance of the recognition system in practical scenarios. Recently, as deep learning has demonstrated its effectiveness in many areas, plenty of deep methods have been investigated to address the challenges in activity recognition. In this thesis, we propose a set of deep learning methods aiming to mitigate some of the challenges. First, we review the latest state-of-the-art deep learning methods for sensor-based human activity recognition and specific challenges so as to form an overview of the current research progress. Challenges and challenge-related deep methods are summarized and analyzed. In order to alleviate the feature extraction and interpretability challenges that the human activity recognition community is faced with, we propose an interpretable parallel recurrent model with convolutional attentions. We consider that multimodal features play a crucial role in wearable sensor-based human activity recognition and assume that selecting the most salient features adaptively is a promising way to maximize the effectiveness of sensory data. Furthermore, we consider two inherent characteristics of human activities, that is, the spatially-temporally varying salience of features and the relations between activities and corresponding body part motions. Based on these, we propose a multiagent spatial-temporal attention model to further address the challenges of feature extraction and interpretability. Then we propose a semi-supervised deep model for imbalanced activity recognition from multimodal wearable sensory data. We aim to address not only the challenges of annotation scarcity but also the class imbalance issue. Last but not least, we propose a generic distributionally robust model for semisupervised learning on distributionally shifted data to address the heterogeneity issue caused by users. To be specific, we learn the latent features that reduce person-specific discrepancy and preserve task-specific consistency. In the discussion part, we discuss a few open issues and provide some insights for future directions.

This Ph.D. dissertation presents a development of calibration methods of microelectromechanical (MEMS) gyroscopes for a vestibular prosthesis. The vestibular prosthesis senses head orientation and provides electrical stimulation to the ampullary nerves. The prosthesis utilizes a micro sensing element, which is susceptible to environmental changes resulting in drift. Consequently, encoding the input rotation to stimulate pulses using a MEMS sensing architecture would mislead the nervous system and inadvertently cause additional damage to other sensing organs. Not mitigating the issues of short-term and long-term drift would take away all benefits of the prosthesis. The focus of this dissertation is toward algorithm design and hardware development for the identification of sources of errors in bias and scale-factor of a MEMS gyroscope and to introduce sensor calibration methods for providing accurate electrical stimulation. A Silicon micromachined Quad Mass Gyroscope (QMG) was used, as a test structure device, and was utilized to identify sensor-level calibration algorithms needed for the prosthesis. For this realization, a standalone electronics platform for MEMS gyroscope characterization was designed and developed to identify the needed control algorithms to satisfy the required bandwidth and linearity of the prosthesis. As a calibration technique for the prosthesis system, a flexible solution platform for integration and data processing from multiple foot-mounted inertial sensors was designed and developed for the purpose of providing an aiding solution to calibrate the prosthesis sensitivity and drift. The developed calibration methods demonstrated the potential to identify sources of errors and compensate for the drift, a functionality that would not be possible by simply utilizing off-the-shelf MEMS gyroscopes for the vestibular prosthesis.

Human Activity Recognition is an actively researched domain for the past fewdecades, and is one of the most eminent applications of today. It is already part of our life,but due to high level of uncertainty and challenges of human detection, we have onlyapplication specific solutions. Thus, the problem being very demanding and still remainsunsolved.Within this PhD we delve into the problem, and approach it from a variety of viewpoints.At start, we present and evaluate different architectures and frameworks for activityrecognition.Henceforward, the focal point of our attention is automatic human activityrecognition. We conducted and present a survey that compares, categorizes, and evaluatesresearch surveys and reviews into four categories.Then a novel fully automatic view-independent multi-formal languagescollaborative scheme is presented for complex activity and emotion recognition, which isthe main contribution of this dissertation.We propose a collaborative three formal-languages, that is responsible for parsingmanipulating, and understanding all the data needed. Artificial Neural Networks are usedto classify an action primitive (simple activity), as well as to define change of activity.Finally, we capitalize the advantages of Fuzzy Cognitive Maps, and Rule-Based ColoredPetri-Nets to be able to classify a sequence of activities as normal or ab-normal.

The ability to automatically recognize a person's behavioral context (including where they are, what they are doing, who they are with, etc.) is greatly beneficial in health monitoring, aging care, personal assistants, smart homes, customized entertainment, and many other domains. For all of these different applications to succeed on a larger scale, the context-recognition component must be unobtrusive and work smoothly, without making people adjust their behavior. It is important for research to validate context-recognition systems in the real world - under the same conditions in which such applications will eventually be deployed. In this thesis, I promote context recognition in-the-wild, capturing people's authentic behavior in their natural environments using natural, everyday devices.In Chapter 1, I introduce the field of behavioral context recognition, and describe three parts of research in the field: defining the problem (what are the inputs and what are the outputs), collecting data, and artificial intelligence (AI) / machine learning (ML) methods.In Chapter 2, I present the problem of behavioral context recognition and the challenges of addressing behavior in-the-wild. I introduce the ExtraSensory Dataset, which was collected from 60 participants in-the-wild, and is publicly available at http://extrasensory.ucsd.edu. I describe simple machine learning methods and demonstrate that smartphones and smartwatches can be used to successfully recognize diverse contexts in regular life (e.g., walking, sleeping, at school, on a bus, cooking, shower, phone in pocket).In Chapter 3, I specifically address machine learning solutions that facilitate training classifiers with irregular data from the wild - highly unbalanced, with occasions of missing labels or sensors, and potentially collected in phases addressing different sets of labels.In Chapter 4, I address the challenge of collecting labeled data in-the-wild and describe our self-reporting solution - the ExtraSensory App. I analyze the collected data and subjective feedback from the participants to gain insights about user-interface design to engage users to contribute labels about their own behavior. A revised version of the app, with improvements based on this dissertation, is publicly available at http://extrasensory.ucsd.edu/ExtraSensoryApp.In Chapter 5, I discuss the progress this work makes in the field of behavioral context recognition, and suggest directions for future improvements.

This thesis is motivated by the rapid increase in global life expectancy without the respective improvements in quality of life. I propose several novel machine learning and data mining methodologies for approaching a paramount component of quality of life, the translational science field of sleep research. Inadequate sleep negatively affects both mental and physical well-being, and exacerbates many non-communicable health problems such as diabetes, depression, cancer and obesity. Taking advantage of the ubiquitous adoption of wearable devices, I create algorithmic solutions to analyse sensor data. The goal is to improve the quality of life of wearable device users, as well as provide clinical insights and tools for sleep researchers and care-providers. Chapter 1 is the introduction. This section substantiates the timely relevance of sleep research for today's society, and its contribution towards improved global health. It covers the history of sleep science technology and identifies core computing challenges in the field. The scope of the thesis is established and an approach is articulated. Useful definitions, sleep domain terminology, and some pre-processing steps are defined. Lastly, an outline for the remainder of the thesis is included. Chapter 2 dives into my proposed methodology for widespread screening of sleep disorders. It surveys results from the application of several statistical and data mining methods. It also introduces my novel deep learning architecture optimized for the unique dimensionality and nature of wearable device data. Chapter 3 focuses on the diagnosis stage of the sleep science process. I introduce a human activity recognition algorithm called RAHAR, Robust Automated Human Activity Recognition. This algorithm is unique in a number of ways, including its objective of annotating a behavioural time series with exertion levels rather than activity type. Chapter 4 focuses on the last step of the sleep science process, therapy. I define a pipeline to identify \textit{behavioural recipes}. These \textit{recipes} are the target behaviour that a user should complete in order to have good quality sleep. This work provides the foundation for building out a dynamic real-time recommender system for wearable device users, or a clinically administered cognitive behavioural therapy program. Chapter 5 summarizes the impact of this body of work, and takes a look into next steps. This chapter concludes my thesis.

This thesis proposes an open-source, maintainable system for detecting human activity in large video datasets using scalable hardware architectures. The system is validated by detecting writing and typing activities that were collected as part of the Advancing Out of School Learning in Mathematics and Engineering (AOLME) project. The implementation of the system using Amazon Web Services (AWS) is shown to be both horizontally and vertically scalable. The software associated with the system was designed to be robust so as to facilitate reproducibility and extensibility for future research.

Advances in energy harvesting hardware have created an opportunity to realise self-powered wearables for continuous and pervasive Human Activity Recognition (HAR). Unfortunately, the power requirements of continuous activity sensing using accelerometer sensors and burdensome on-node classification are relatively high compared to the amount of power that can be practically harvested, which limits the usefulness of energy harvesting. This thesis makes three fundamental contributions. First, we propose HARKE, Human Activity Recognition from Kinetic Energy, a novel approach to HAR that does not use an accelerometer. Instead, HARKE employs and infers human physical activities directly from the Kinetic Energy Harvesting (KEH) patterns generated from a device that harvests kinetic energy to power the wearable device. We also show the ability of HARKE to detect related details such as the steps taken by the user in a walking activity. By not using an accelerometer, a significant percentage of the limited harvested energy can be saved. Second, we introduce a novel framework that reduces the on-node classification overhead and guarantees energy neutrality. The proposed framework transmits an unmodulated signal, called an activity pulse, and uses only the received signal strength of the activity pulse to classify human activities. Neither accelerometer nor classifier is required on the wearable device, which therefore, guarantees energy neutrality. Finally, we validate the feasibility of using KEH patterns generated from human speech as a potential new source of information for detecting hotwords, such as ``OK Google", which are used by voice control applications to differentiate user commands from background conversations. Unlike methods that use existing sensors like microphones or accelerometers, our proposal enables pervasive voice control and HAR at a minimum energy cost. We believe that the findings in this thesis will open the door for a new direction of research and development to realise the vision of pervasive self-powered HAR, moving us closer towards self-powered autonomous wearables.

Understanding human activities in unconstrained natural videos is a widely studied problem, yet it remains as one of the most challenging problems in computer vision. It has many practical applications, including, but not limited to security, surveillance, and robotic vision. The primary requirement of these applications is to learn a recognition model that can differentiate among different types of activities. However, learning a recognition model requires lots of data and these data need to be labeled. Enormous amount of unlabeled videos are being generated from various sources. Labeling these video data is the biggest challenge that an activity understanding method encounters. State-of-the-art approaches generally learn fixed or static recognition model based on the assumptions that all the training data are labeled and available beforehand. But these assumptions are unrealistic for many applications where we have to deal with streaming videos or surveillance cameras. In such scenarios, new unlabeled video data may come over time and can be leveraged upon to incrementally improve the current model. In this thesis, we aim to develop frameworks for online activity understanding by taking advantage of unlabeled video data with a reduced labeling cost and without compromising performance.We present four distinct frameworks for continuous learning of activity recognition models. We leverage upon tools and techniques from various branches of machine learning and deep learning and effectively combine them together with computer vision in order to develop novel and efficient solutions. Our proposed methods reduce the labeling cost by a significant margin, yet their performances are highly competitive with the state-of-the-art methods. An integral part of activity understanding is to temporally segment the activities from a long video sequence. In this thesis, we also present an approach for activity segmentation that requires limited human supervision.The first approach we propose is based on an ensemble of SVM classifiers. Given the set of unlabeled data, we select the most informative queries to be labeled by a human annotator. Then, we train new SVMs and include them into the ensemble with updated weights. In the second approach, we take the advantage of contextual relationship among the activities and the objects in the video sequence. We encode contextual information using a conditional random field and then, present a novel active leaning algorithm that utilizes both of the entropy and the mutual information of the activity nodes. In the third approach, we further reduce human effort by making early prediction of the activity labels and providing dynamic suggestions to the annotator. State-of-the-art approaches do not scale with the growing number of video categories and they do not consider the cost of long viewing time for video labeling. Our proposed framework uses label propagation and LSTM based recurrent neural network that effectively selects informative queries and provides early suggestions to the annotator. In the fourth approach, we propose a continuous activity learning framework by intricately tying together deep hybrid feature models and active learning. This allows us to automatically select the most suitable features and to take the advantage of incoming unlabeled instances to improve the existing model incrementally. Finally, we propose a method for temporally segmenting meaningful activities that require very limited supervision. Perceiving these activities in a long video sequence is a challenging problem due to ambiguous definition of meaningfulness as well as clutters in the scene. We approach this problem by learning a generative model for regular motion patterns. We propose two methods that are built upon the autoencoders for their ability to work with unlabeled data.

Monitoring and surveillance of humans is one of the most prominent applications of today and it is expected to be part of many future aspects of our life, for safety reasons, assisted living and many others. Many efforts have been made towards automatic and robust solutions, but the general problem is very challenging and remains still open. In this PhD dissertation we examine the problem from many perspectives. First, we study the performance of a hardware architecture designed for large-scale surveillance systems. Then, we focus on the general problem of human activityrecognition, present an extensive survey of methodologies that deal with this subject and propose a maturity metric to evaluate them.One of the numerous and most popular algorithms for image processing found in the field is image segmentation and we propose a blind metric to evaluate their results regarding the activity at local regions. Finally, we propose a fully automatic system for segmenting and extracting human bodies from challenging single images, which is the main contribution of the dissertation. Our methodology is a novel bottom-up approach relying mostly on anthropometric constraints and is facilitated by our research in the fields of face, skin and hands detection. Experimental results and comparison with state-of-the-art methodologies demonstrate the success of our approach.

This dissertation describes two distinctive methods for human-vehicle interaction recognition: one for ground level videos and the other for aerial videos. For ground level videos, this dissertation presents a novel methodology which is able to estimate a detailed status of a scene involving multiple humans and vehicles. The system tracks their configuration even when they are performing complex interactions with severe occlusion such as when four persons are exiting a car together. The motivation is to identify the 3-D states of vehicles (e.g. status of doors), their relations with persons, which is necessary to analyze complex human-vehicle interactions (e.g. breaking into or stealing a vehicle), and the motion of humans and car doors to detect atomic human-vehicle interactions. A probabilistic algorithm has been designed to track humans and analyze their dynamic relationships with vehicles using a dynamic context. We have focused on two ideas. One is that many simple events can be detected based on a low-level analysis, and these detected events must contextually meet with human/vehicle status tracking results. The other is that the motion clue interferes with states in the current and future frames, and analyzing the motion is critical to detect such simple events. Our approach updates the probability of a person (or a vehicle) having a particular state based on these basic observed events. The probabilistic inference is made for the tracking process to match event-based evidence and motion-based evidence. For aerial videos, the object resolution is low, the visual cues are vague, and the detection and tracking of objects is less reliable as a consequence. Any method that requires accurate tracking of objects or the exact matching of event definition are better avoided. To address these issues, we present a temporal logic based approach which does not require training from event examples. At the low-level, we employ dynamic programming to perform fast model fitting between the tracked vehicle and the rendered 3-D vehicle models. At the semantic-level, given the localized event region of interest (ROI), we verify the time series of human-vehicle relationships with the pre-specified event definitions in a piecewise fashion. With special interest in recognizing a person getting into and out of a vehicle, we have tested our method on a subset of the VIRAT Aerial Video dataset and achieved superior results.

Human activity recognition is one of the intensively studied areas in computer vision. Most existing works do not assume video resolution to be a problem due to general applications of interests. However, with continuous concerns about global security and emerging needs for intelligent video analysis tools, activity recognition from low-resolution and low-quality videos has become a crucial topic for further research. In this dissertation, We present a series of approaches which are developed specifically to address the related issues regarding low-level image preprocessing, single person activity recognition, and human-vehicle interaction reasoning from low-resolution surveillance videos. Human cast shadows are one of the major issues which adversely effect the performance of an activity recognition system. This is because human shadow direction varies depending on the time of the day and the date of the year. To better resolve this problem, we propose a shadow removal technique which effectively eliminates a human shadow cast from a light source of unknown direction. A multi-cue shadow descriptor is employed to characterize the distinctive properties of shadows. Our approach detects, segments, and then removes shadows. We propose two different methods to recognize single person actions and activities from low-resolution surveillance videos. The first approach adopts a joint feature histogram based representation, which is the concatenation of subspace projected gradient and optical flow features in time. However, in this problem, the use of low-resolution, coarse, pixel-level features alone limits the recognition accuracy. Therefore, in the second work, we contributed a novel mid-level descriptor, which converts an activity sequence into simultaneous temporal signals at body parts. With our representation, activities are recognized through both the local video content and the short-time spectral properties of body parts' movements. We draw the analogies between activity and speech recognition and show that our speech-like representation and recognition scheme improves recognition performance in several low-resolution datasets. To complete the research on this subject, we also tackle the challenging problem of recognizing human-vehicle interactions from low-resolution aerial videos. We present a temporal logic based approach which does not require training from event examples. At the low-level, we employ dynamic programming to perform fast model fitting between the tracked vehicle and the rendered 3-D vehicle models. At the semantic-level, given the localized event region of interest (ROI), we verify the time series of human-vehicle spatial relationships with the pre-specified event definitions in a piecewise fashion. Our framework can be generalized to recognize any type of human-vehicle interaction from aerial videos.

A significant number of action recognition research efforts use spatio-temporal interest point detectors for feature extraction. Although the extracted features provide useful information for recognizing actions, a significant number of them contain irrelevant motion and background clutter. In many cases, the extracted features are included as is in the classification pipeline, and sophisticated noise removal techniques are subsequently used to alleviate their effect on classification. We introduce a new action database, created from the Weizmann database, that reveals a significant weakness in systems based on popular cuboid descriptors. Experiments show that introducing complex backgrounds, stationary or dynamic, into the video causes a significant degradation in recognition performance. Moreover, this degradation cannot be fixed by fine-tuning the system or selecting better interest points. Instead, we show that the problem lies at the descriptor level and must be addressed by modifying descriptors.

in the field of human activity recognition has existed for quite sometime, but has gained popularity in recent years for use in many areas of application. In the security industry, suspicious activities could be detected in high-profile areas. In the medical industry, systems could be trained to detect patterns of motion indicating distress or to detect a lack of motion if a person had fallen and was unable to move. However, algorithms with reliable accuracy are difficult to implement in a real-time environment due to computational complexity. This thesis developed a new way of extracting and using data from a human figure in a video frame to determine what type of activity the subject is performing. Following background subtraction, a thinning algorithm operating on the silhouette offered a more robust limb extraction method, while a six-segment representation of the human figure offered more accuracy in deriving limb parameters, or components, such as distance from torso, and angle of displacement from the vertical axis. Neural networks or nearest neighbor classifiers used the limb components to identify a number of activities, such as walking, running, waving and jumping. This entire human activity recognition system was tested with both a MATLAB implementation (non real-time) and a C++ implementation in OpenCV (real-time). The algorithm achieved 96% classification accuracy in video feeds, which is only slightly lower than that of intensive, non real-time systems.