Search

The Sussex-Huawei Locomotion-Transportation Recognition Challenge presented a unique opportunity to the activity-recognition community to test their approaches on a large, real-life benchmark dataset with activities different from those typically recognized. The goal of the challenge was to recognize, as accurately as possible, eight locomotion activities (Still, Walk, Run, Bike, Car, Bus, Train, Subway) using smartphone sensor data. This paper describes the method we developed to win this challenge, and provides an analysis of the effectiveness of its components. We used complex feature extraction and selection methods to train classical machine learning models. In addition, we trained deep learning models using a novel end-to-end architecture for deep multimodal spectro-temporal fusion. All the models were fused into an ensemble with the final predictions smoothed by a hidden Markov model to account for temporal dependencies of the activities. The presented method achieved an F1 score of 94.9% on the challenge test data. We tested different sampling frequencies, window sizes, feature types, classification models and the importance of stand-alone sensors and their fusion for the task. Finally, we present an energy-efficient smartphone implementation of the method.

In the past decade, public institutions and private entities have launched large campaigns of digitization of cultural artefacts leading to the creation of massive digital collections like Europeana, Europe's digital cultural library, museum, and archive. It offers public access to millions of digital objects from thousands of contributing heritage collections all across the European Union, using a multilingual interface, built upon the principles of the Semantic Web. However, such large digital libraries suffer due to low accessibility for the general public and difficult search and navigation through their items. Intelligent conversational agents have the potential of facilitating user-friendly access to the vast amount of information in the Semantic Web by natural language interaction and by providing structured answers to the user's queries. More recently, new technologies additionally fostered the development of intelligent conversational agents for a wide range of applications. Following this approach, we propose a solution to improve the accessibility and search accuracy of digital cultural heritage resources from Europeana through a system which integrates artificial intelligence, natural language technologies, web services, and APIs.

This paper presents relations between information society (IS), electronics and artificial intelligence (AI) mainly through twenty-four IS laws. The laws not only make up a novel collection, currently non-existing in the literature, but they also highlight the core boosting mechanism for the progress of what is called the information society and AI. The laws mainly describe the exponential growth in a particular field, be it the processing, storage or transmission capabilities of electronic devices. Other rules describe the relations to production prices and human interaction. Overall, the IS laws illustrate the most recent and most vibrant part of human history based on the unprecedented growth of device capabilities spurred by human innovation and ingenuity. Although there are signs of stalling, at the same time there are still many ways to prolong the fascinating progress of electronics that stimulates the field of artificial intelligence. There are constant leaps in new areas, such as the perception of real-world signals, where AI is already occasionally exceeding human capabilities and will do so even more in the future. In some areas where AI is presumed to be incapable of performing even at a modest level, such as the production of art or programming software, AI is making progress that can sometimes reflect true human skills. Maybe it is time for AI to boost the progress of electronics in return.

Human Activity Recognition (HAR) from wearable sensors has gained significant attention in the last few decades, largely because of the potential healthcare benefits. For many years, HAR was done using classical machine learning approaches that require extraction of features. With the resurgence of deep learning, a major shift happened and at the moment, HAR researchers are mainly investigating different kinds of deep neural networks. However, deep learning comes with the challenge of having access to large amounts of labeled examples, which in the field of HAR is considered an expensive task, both in terms of time and effort. Another challenge is the fact that the training and testing data in HAR can be different due to the personal preferences of different people when performing the same activity. In order to try and mitigate these problems, in this paper we explore transfer learning, a paradigm for transferring knowledge from a source domain, to another related target domain. More specifically, we explore the effects of transferring knowledge between two open-source datasets, the Opportunity and JSI-FOS datasets, using weight-transfer for the DeepConvLSTM architecture. We also explore the performance of this transfer at different amounts of labeled data from the target domain. The experiments showed that it is beneficial to transfer the weights of fewer layers, and that deep transfer learning can perform better than a domain-specific deep end-to-end model in specific circumstances. Finally, we show that deep transfer learning is a viable alternative to classical machine learning approaches as it produces comparable results and does not require feature extraction.

Falls are a significant threat to the health and independence of elderly people and represent an enormous burden on the healthcare system. Successfully predicting falls could be of great help, yet this requires a timely and accurate fall risk assessment. Gait abnormalities are one of the best predictive signs of underlying locomotion conditions and precursors of falls. The advent of wearable sensors and wrist-worn devices provides new opportunities for continuous and unobtrusive monitoring of gait during daily activities, including the identification of unexpected changes in gait. To this end, we present in this paper a novel method for determining gait abnormalities based on a wrist-worn device and a deep neural network. It integrates convolutional and bidirectional long short-term memory layers for successful learning of spatiotemporal features from multiple sensor signals. The proposed method was evaluated using data from 18 subjects, who recorded their normal gait and simulated abnormal gait while wearing impairment glasses. The data consist of inertial measurement unit (IMU) sensor signals obtained from smartwatches that the subjects wore on both wrists. Numerous experiments showed that the proposed method provides better results than the compared methods, achieving 88.9% accuracy, 90.6% sensitivity, and 86.2% specificity in the detection of abnormal walking patterns using data from an accelerometer, gyroscope, and rotation vector sensor. These results indicate that reliable fall risk assessment is possible based on the detection of walking abnormalities with the use of wearable sensors on a wrist.

This study introduces two datasets for multimodal research on cognitive load inference and personality traits. Different to other datasets in Affective Computing, which disregard participants&rsquo, personality traits or focus only on emotions, stress, or cognitive load from one specific task, the participants in our experiments performed seven different tasks in total. In the first dataset, 23 participants played a varying difficulty (easy, medium, and hard) game on a smartphone. In the second dataset, 23 participants performed six psychological tasks on a PC, again with varying difficulty. In both experiments, the participants filled personality trait questionnaires and marked their perceived cognitive load using NASA-TLX after each task. Additionally, the participants&rsquo, physiological response was recorded using a wrist device measuring heart rate, beat-to-beat intervals, galvanic skin response, skin temperature, and three-axis acceleration. The datasets allow multimodal study of physiological responses of individuals in relation to their personality and cognitive load. Various analyses of relationships between personality traits, subjective cognitive load (i.e., NASA-TLX), and objective cognitive load (i.e., task difficulty) are presented. Additionally, baseline machine learning models for recognizing task difficulty are presented, including a multitask learning (MTL) neural network that outperforms single-task neural network by simultaneously learning from the two datasets. The datasets are publicly available to advance the field of cognitive load inference using commercially available devices.

Human falls are common source of injury among the elderly, because often the elderly person is injured and cannot call for help. In the literature this is addressed by various fall-detection systems, of which most common are the ones that use wearable sensors. This paper describes the winning method developed for the Challenge Up: Multimodal Fall Detection competition. It is a multi-sensor data-fusion machine-learning method that recognizes human activities and falls using 5 wearable inertial sensors: accelerometers and gyroscopes. The method was evaluated on a dataset collected by 12 subjects of which 3 were used as a test-data for the challenge. In order to optimally adapt the method to the 3 test subjects, we performed an unsupervised similarity search—that finds the three most similar users to the three users in the test data. This helped us to tune the method and its parameters to the 3 most similar users as the ones used for the test. The internal evaluation on the 9 users showed that with this optimized configuration the method achieves 98% accuracy. During the final evaluation for the challenge, our method achieved the highest results (82.5% F1-score, and 98% accuracy) and won the competition.

We present the concept of an Intelligent Assistant Carer system for the elderly, designed to help with active aging and to facilitate the interactions with carers. The system is modular, allowing the users to choose the appropriate functions according to their needs, and is built on an open platform in order to make it compatible with third-party products and services. Currently, the system consists of a wearable device (a smartwatch) and an internet portal that manages the data and takes care of the interactions between the user, the carers, and the support services. We present in detail one of the modules, i.e., fall detection, and the results of a pilot study for the system on 150 users over the course of 3 months.

Being able to detect stress as it occurs can greatly contribute to dealing with its negative health and economic consequences. However, detecting stress in real life with an unobtrusive wrist device is a challenging task. The objective of this study is to develop a method for stress detection that can accurately, continuously and unobtrusively monitor psychological stress in real life. First, we explore the problem of stress detection using machine learning and signal processing techniques in laboratory conditions, and then we apply the extracted laboratory knowledge to real-life data. We propose a novel context-based stress-detection method. The method consists of three machine-learning components: a laboratory stress detector that is trained on laboratory data and detects short-term stress every 2 min; an activity recognizer that continuously recognizes the user’s activity and thus provides context information; and a context-based stress detector that uses the outputs of the laboratory stress detector, activity recognizer and other contexts, in order to provide the final decision on 20-min intervals. Experiments on 55 days of real-life data showed that the method detects (recalls) 70% of the stress events with a precision of 95%.

The Sussex-Huawei Locomotion Challenge 2019 was an open competition in activity recognition where the participants were tasked with recognizing eight different modes of locomotion and transportation. The main difficulty of the challenge is that the training data was recorded with a smartphone that was placed in a different body location than the test data. Only a small validation set with all locations was provided to enable transfer learning. This paper describes our (team JSI First) approach, in which we first derived additional sensor streams from the existing ones and on them calculated a large body of features. We then used cross-location transfer learning via specialized feature selection, and performed two-step classification. Finally, we used Hidden Markov Models to alter the predictions in order to take their temporal dependencies into account. Internal tests using this methodology yielded an accuracy of 83%.

Vehicle driving consists of selecting and applying the best control actions in real time to optimize several objectives such as the traveling time and the fuel consumption. Because more than one ob...

The Sussex-Huawei Locomotion-Transportation Recognition Challenge presented a unique opportunity to the activity-recognition community to test their approaches on a large, real-life benchmark dataset with activities different from those typically being recognized. The goal of the challenge was to recognize eight locomotion activities (Still, Walk, Run, Bike, Car, Bus, Train, Subway). This chapter describes the submissions winning the first and second place. They both start with data preprocessing, including a normalization of the phone orientation. Then, a wide set of hand-crafted domain features in both frequency and time domain are computed and their quality evaluated. The second-place submission feeds the best features into an XGBoost machine-learning model with optimized hyper-parameters, achieving the accuracy of 90.2%. The first-place submission builds an ensemble of models, including deep learning models, and finally refines the ensemble’s predictions by smoothing with a Hidden Markov model. Its accuracy on an internal test set was 96.0%.

Convolution Neural Network (CNN) filters learned on one domain can be used as feature extractors on another similar domain. Transferring filters allow reusing datasets across domains and reducing labelling costs. In this paper, four activity recognition datasets were analyzed to study the effects of transferring filters across the datasets. A spectro-temporal ResNet was implemented as a deep, end-to-end learning architecture. We analyzed the number of transferred CNN residual blocks with respect to the size of the target-adaptation data. The analysis showed that transfer learning using small adaptation subsets is more useful when the target domain contains a small number of different activities. Furthermore, the similarity between the domains, participating in the transfer learning scenario, seems to play a role in its success. The most successful transfer achieved an F1-score of 93%, which is an increase of 9 percentage points compared to a domain-specific baseline model.

This open access book constitutes the refereed proceedings of the First International Conference on VR Technologies in Cultural Heritage, VRTCH 2018, held in Brasov, Romania in May 2018. The 13 revised full papers along with the 5 short papers presented were carefully reviewed and selected from 21 submissions. The papers of this volume are organized in topical sections on data acquisition and modelling, visualization methods / audio, sensors and actuators, data management, restoration and digitization, cultural tourism.

Virtual assistants and similar software tools are gaining importance among phone and computer users. The most well-known assistants (Siri, Cortana, Google, etc.) provide general information to users and cannot be adapted to specific needs. Custom implementations usually cover specific domains and are specialized to provide a comprehensive set of information or functionalities from that area. Currently, there is a lack of general applications for cultural heritage, since most implementations are specialized virtual guides for museums or exhibitions. We present an overview of virtual assistants that we developed and are applied to the area of cultural and natural heritage, where we try to improve the current shortage of tools. The presented prototypes provide various functionalities to users and can help them to discover, learn and plan visits to cultural sights.

Intelligent cognitive assistant (ICA) technology is used in various domains to emulate human behavior expressed through synchronous communication, especially written conversation. Due to their ability to use individually tailored natural language, they present a powerful vessel to support attitude and behavior change. Behavior change support systems are emerging as a crucial tool in digital mental health services, and ICAs exceed in effective support, especially for stress, anxiety and depression (SAD), where ICAs guide people’s thought processes and actions by analyzing their affective and cognitive phenomena. Currently, there is no comprehensive review of such ICAs from a technical standpoint, and existing work is conducted exclusively from a psychological or medical perspective. This technical state-of-the-art review tried to discern and systematize current technological approaches and trends as well as detail the highly interdisciplinary landscape of intersections between ICAs, attitude and behavior change, and mental health, focusing on text-based ICAs for SAD. Ten papers with systems, fitting our criteria, were selected. The systems varied significantly in their approaches, with the most successful opting for comprehensive user models, classification-based assessment, personalized intervention, and dialogue tree conversational models.

Finding the best classifiers according to different criteria is often performed by a multi-objective machine learning algorithm. This study considers two criteria that are usually treated as the most important when deciding which classifier to apply in practice: comprehensibility and accuracy. A model that offers a broad range of trade-offs between the two criteria is introduced because they conflict; i.e., increasing one decreases the other. The choice of the model is motivated by the fact that domain experts often formalize decisions based on knowledge that can be represented by comprehensible rules and some tacit knowledge. This approach is mimicked by a hybrid tree that consists of comprehensible parts that originate from a regular classification tree and incomprehensible parts that originate from an accurate black-box classifier. An empirical evaluation on 23 UCI datasets shows that the hybrid trees provide trade-offs between the accuracy and comprehensibility that are not possible using traditional machine learning models. A corresponding hybrid-tree comprehensibility metric is also proposed. Furthermore, the paper presents a novel algorithm for learning MAchine LeArning Classifiers with HybrId TrEes (MALACHITE), and it proves that the algorithm finds a complete set of nondominated hybrid trees with regard to their accuracy and comprehensibility. The algorithm is shown to be faster than the well-known multi-objective evolutionary optimization algorithm NSGA-II for trees with moderate size, which is a prerequisite for comprehensibility. On the other hand, the MALACHITE algorithm can generate considerably larger hybrid-trees than a naive exhaustive search algorithm in a reasonable amount of time. In addition, an interactive iterative data mining process based on the algorithm is proposed that enables inspection of the Pareto set of hybrid trees. In each iteration, the domain expert analyzes the current set of nondominated hybrid trees, infers domain relations, and sets the parameters for the next machine learning step accordingly.

Purpose In this paper, the authors aim to propose a method for learning robotic assembly sequences, where precedence constraints and object relative size and location constraints can be learned by demonstration and autonomous robot exploration. Design/methodology/approach To successfully plan the operations involved in assembly tasks, the planner needs to know the constraints of the desired task. In this paper, the authors propose a methodology for learning such constraints by demonstration and autonomous exploration. The learning of precedence constraints and object relative size and location constraints, which are needed to construct a planner for automated assembly, were investigated. In the developed system, the learning of symbolic constraints is integrated with low-level control algorithms, which is essential to enable active robot learning. Findings The authors demonstrated that the proposed reasoning algorithms can be used to learn previously unknown assembly constraints that are needed to implement a planner for automated assembly. Cranfield benchmark, which is a standardized benchmark for testing algorithms for robot assembly, was used to evaluate the proposed approaches. The authors evaluated the learning performance both in simulation and on a real robot. Practical implications The authors' approach reduces the amount of programming that is needed to set up new assembly cells and consequently the overall set up time when new products are introduced into the workcell. Originality/value In this paper, the authors propose a new approach for learning assembly constraints based on programming by demonstration and active robot exploration to reduce the computational complexity of the underlying search problems. The authors developed algorithms for success/failure detection of assembly operations based on the comparison of expected signals (forces and torques, positions and orientations of the assembly parts) with the actual signals sensed by a robot. In this manner, all precedence and object size and location constraints can be learned, thereby providing the necessary input for the optimal planning of the entire assembly process.

The thermal comfort experience in conditioned environments is closely related to the indoor temperature andvaries mainly due to the dynamics of occupant state and the environmental state. The heating or cooling required to achievethe desired temperature and comfort influences the energy consumption. This article presents a multi-agent control systemthat primarily regulates thermal comfort rather than the indoor temperature. We developed this comfort regulator that basedon (i) the difference between the desired level of comfort and the current level of comfort and (ii) the difference between thecurrent temperature and the set point temperature adapts the set point temperature in order to achieve the desired comfort.An occupancy prediction algorithm and expert rules were designed to efficiently reduce unnecessary energy consumptionduring periods when the home is not occupied and the comfort experience is therefore not important. The results of experimentsare presented in a comfort/energy-consumption space. The comfort/energy-consumption space shows how the finalresult is influenced by (i) different versions of learning algorithms and (ii) different comfort threshold values. Comparingthe comfort/energy-consumption spaces for different occupancy patterns shows that the rule settings have similar impact oncontrol performance, which indicates that the rules are general. In nearly all experiments, the proposed multi-agent controlsystem assured better comfort experience with small increase of energy consumption compared to reactive control system.DOI: http://dx.doi.org/10.5755/j01.itc.44.4.10139

To further extend the applicability of wearable sensors in various domains such as mobile health systems and the automotive industry, new methods for accurately extracting subtle physiological information from these wearable sensors are required. However, the extraction of valuable information from physiological signals is still challenging&mdash, smartphones can count steps and compute heart rate, but they cannot recognize emotions and related affective states. This study analyzes the possibility of using end-to-end multimodal deep learning (DL) methods for affect recognition. Ten end-to-end DL architectures are compared on four different datasets with diverse raw physiological signals used for affect recognition, including emotional and stress states. The DL architectures specialized for time-series classification were enhanced to simultaneously facilitate learning from multiple sensors, each having their own sampling frequency. To enable fair comparison among the different DL architectures, Bayesian optimization was used for hyperparameter tuning. The experimental results showed that the performance of the models depends on the intensity of the physiological response induced by the affective stimuli, i.e., the DL models recognize stress induced by the Trier Social Stress Test more successfully than they recognize emotional changes induced by watching affective content, e.g., funny videos. Additionally, the results showed that the CNN-based architectures might be more suitable than LSTM-based architectures for affect recognition from physiological sensors.

Home energy-management systems can optimize performance either by computing the next step dynamically – online, or rely on a precomputed strategy used to introduce the next decision – offline. Further, such systems can optimize based on only one or several objectives. In this paper, the multi-objective optimization of offline strategies for home energy-management systems is addressed. Two approaches are compared: the common timetable-based versus our approach based on decision trees. The timetable-based strategy is optimized using a multi-objective genetic algorithm, while the tree-based strategy is optimized using multi-objective genetic programming. As a result, a set of rules that comprise the trees for efficient management of an energy system is generated automatically. First, the approaches are addressed theoretically, with the finding that the tree-based approach is more powerful than the timetable-based approach. Second, the performance of the tree-based approach is compared with the performance of the timetable-based approach and manually defined strategies in an experiment involving real-world data. A performance increase of up to 17% in terms of the cost objective was confirmed for the tree-based approach. This is achieved without changing the user habits, i.e., there is no need of having to adapt the appliance usage to the energy-management system.

In recent years, activity recognition (AR) has become prominent in ubiquitous systems. Following this trend, the Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge provides a unique opportunity for researchers to test their AR methods against a common, real-life and large-scale benchmark. The goal of the challenge is to recognize eight everyday activities including transit. Our team, JSI-Deep, utilized an AR approach based on combining multiple machine-learning methods following the principle of multiple knowledge. We first created several base learners using classical and deep learning approaches, then integrated them into an ensemble, and finally refined the ensemble's predictions by smoothing. On the internal test data, the approach achieved 96% accuracy, which is a significant leap over the baseline 60%.

The Sussex-Huawei Locomotion-Transportation recognition challenge presents a unique opportunity to the activity-recognition community - providing a large, real-life dataset with activities different from those typically being recognized. This paper describes our submission (team JSI Classic) to the competition that was organized by the dataset authors. We used a carefully executed machine learning approach, achieving 90% accuracy classifying eight different activities (Still, Walk, Run, Bike, Car, Bus, Train, Subway). The first step was data preprocessing, including a normalization of the phone orientation. Then, a wide set of hand-crafted domain features in both frequency and time domain were computed and their quality was evaluated. Finally, the appropriate machine learning model was chosen (XGBoost) and its hyper-parameters were optimized. The recognition result for the testing dataset will be presented in the summary paper of the challenge [13].

A rapidly ageing population in the developed world brings a necessity and opportunity for the AI-based ICT solutions. We present a system, developed within the scope of an EU H2020 project IN LIFE, which aims to prolong the age at which individual can still live at home independently while at the same time increasing comfort and safety. In this demo we present the final result of the project, a virtual AI carer monitoring user 24/7 in a form of a smartwatch. The system provides a range of useful services among which fall detection and activity monitoring are highlighted here. The device and accompanying services were tested during the project on 150 elderly users for several months, their feedback was taken into account for potential future improvements.

Background: Blood pressure (BP) measurements have been used widely in clinical and private environments. Recently, the use of ECG monitors has proliferated; however, they are not enabled with BP estimation. We have developed a method for BP estimation using only electrocardiogram (ECG) signals. Methods: Raw ECG data are filtered and segmented, and, following this, a complexity analysis is performed for feature extraction. Then, a machine-learning method is applied, combining a stacking-based classification module and a regression module for building systolic BP (SBP), diastolic BP (DBP), and mean arterial pressure (MAP) predictive models. In addition, the method allows a probability distribution-based calibration to adapt the models to a particular user. Results: Using ECG recordings from 51 different subjects, 3129 30-s ECG segments are constructed, and seven features are extracted. Using a train-validation-test evaluation, the method achieves a mean absolute error (MAE) of 8.64 mmHg for SBP, 18.20 mmHg for DBP, and 13.52 mmHg for the MAP prediction. When models are calibrated, the MAE decreases to 7.72 mmHg for SBP, 9.45 mmHg for DBP and 8.13 mmHg for MAP. Conclusion: The experimental results indicate that, when a probability distribution-based calibration is used, the proposed method can achieve results close to those of a certified medical device for BP estimation.

This paper presents a novel approach to the adaptation of multidimensional data models to user-specific needs. The multidimensional data models used in contemporary business-intelligence systems are inherently complex. In order to reduce the complexity of these models, we propose using a qualitative multiple-criteria decision modelling method that is based on using a hierarchical tree of the criteria to decompose the larger problem into a group of smaller problems. The final value is derived by aggregating the criteria values using simple "if-then" rules, which form the knowledge-based expert rules in the hierarchical criteria tree that reflect users' preferences. The multiple-criteria analysis of the multidimensional model structure results in a multidimensional model that exhibits a reduced complexity and is adapted to users' needs. The model was validated using sales data from a medium-size enterprise. The qualitative (through questionnaires) and the quantitative (through usage mining) evaluation of the proposed methodology both showed that the proposed approach increases the ease-of-use of business intelligence systems and also contributes to a higher user satisfaction.

Activity-recognition classifiers, which label an activity based on sensor data, have decreased classification accuracy when used in the real world with a particular person. To improve the classifier, a Multi-Classifier Adaptive-Training algorithm MCAT is proposed. The MCAT adapts activity recognition classifier to a particular person by using four classifiers to utilise unlabelled data. The general classifier is trained on the labelled data available before deployment and retrieved in the controlled environment. The specific classifier is trained on a limited amount of labelled data belonging to the new person in the new environment. A domain-independent meta-classifier decides whether to classify a new instance with the general or specific classifier. The final, second meta-classifier decides whether to include the new instance into the training set of the general classifier. The general classifier is periodically retrained, gradually adapting to the new person in the new environment. The adaptation results were evaluated for statistical significance. Results showed that the MCAT outperforms competing approaches and significantly increases the initial activity-recognition classifier classification accuracy.

Can a model constructed using data mining (DM) programs be trusted? It is known that a decision- tree model can contain relations that are statistically significant, but, in reality, meaningless to a human. When the task is domain analysis, meaningless relations are problematic, since they can lead to wrong conclusions and can consequently undermine a human’s trust in DM programs. To eliminate problematic relations from the conclusions of analysis, we propose an interactive method called Human–Machine Data Mining (HMDM). The method constructs multiple models in a specific way so that a human can reexamine the relations in different contexts and, based on observed evidence, conclude which relations and models are credible—that is, both meaningful and of high quality. Based on the extracted credible relations and models, the human can construct correct overall conclusions about the domain. The method is demonstrated in two complex domains, extracting credible relations and models that indicate the segments of the higher education sector and the research and development sector that influence the economic welfare of a country. An experimental evaluation shows that the method is capable of finding important relations and models that are better in both meaning and quality than those constructed solely by the DM programs.