Search

Mobile phones and other connected wearable systems transformed the way we interact with information, offering access to vast amounts of knowledge at our fingertips. However, the challenge remains on how to make this information more accessible and intuitive. The field of cognitive augmentation aims to enhance our cognitive abilities through technology, allowing us to interact with digital data more naturally and efficiently. This Dagstuhl Seminar brought together experts in neuroscience, psychology, physiology, wearable computing, human-computer interaction, machine perception, and pattern recognition to discuss the possibility of augmenting our cognitive skills and creating new digital senses. The seminar explored the latest findings in these fields and their potential for improving human performance, productivity, and creativity. Ultimately, the goal is to bridge the gap between humans and machines, enabling a more seamless and intuitive interaction between the two. The main discussion topic centered around the possibilities and challenges of digitally augmenting our cognition., DagRep, Volume 12, Issue 12, pages 1-26

The ability to discern between true and false information is essential to making sound decisions. However, with the recent increase in AI-based disinformation campaigns, it has become critical to understand the influence of deceptive systems on human information processing. In experiment (N=128), we investigated how susceptible people are to deceptive AI systems by examining how their ability to discern true news from fake news varies when AI systems are perceived as either human fact-checkers or AI fact-checking systems, and when explanations provided by those fact-checkers are either deceptive or honest. We find that deceitful explanations significantly reduce accuracy, indicating that people are just as likely to believe deceptive AI explanations as honest AI explanations. Although before getting assistance from an AI-system, people have significantly higher weighted discernment accuracy on false headlines than true headlines, we found that with assistance from an AI system, discernment accuracy increased significantly when given honest explanations on both true headlines and false headlines, and decreased significantly when given deceitful explanations on true headlines and false headlines. Further, we did not observe any significant differences in discernment between explanations perceived as coming from a human fact checker compared to an AI-fact checker. Similarly, we found no significant differences in trust. These findings exemplify the dangers of deceptive AI systems and the need for finding novel ways to limit their influence human information processing.

Our surrounding environment impacts our cognitive-emotional processes on a daily basis and shapes our physical, psychological and social wellbeing. Although the effects of the built environment on our psycho-physiological processes are well studied, virtual environment design with a potentially similar impact on the user, has received limited attention. Based on the influence of space design on a user and combining that with the dynamic affordances of virtual spaces, we present the idea of adaptive virtual neuroarchitecture (AVN), where virtual environments respond to the user and the user's real world context while simultaneously influencing them both in realtime. To show how AVN has been explored in current research, we present a sampling of recent work that demonstrates reciprocal relationships using physical affordances (space, objects), the user's state (physiological, cognitive, emotional), and the virtual world used in the design of novel virtual reality experiences. We believe AVN has the potential to help us learn how to design spaces and environments that can enhance the wellbeing of their inhabitants.

Self-supervised or weakly supervised models trained on large-scale datasets have shown sample-efficient transfer to diverse datasets in few-shot settings. We consider how upstream pretrained models can be leveraged for downstream few-shot, multilabel, and continual learning tasks. Our model CLIPPER (CLIP PERsonalized) uses image representations from CLIP, a large-scale image representation learning model trained using weak natural language supervision. We developed a technique, called Multi-label Weight Imprinting (MWI), for multi-label, continual, and few-shot learning, and CLIPPER uses MWI with image representations from CLIP. We evaluated CLIPPER on 10 single-label and 5 multi-label datasets. Our model shows robust and competitive performance, and we set new benchmarks for few-shot, multi-label, and continual learning. Our lightweight technique is also compute-efficient and enables privacy-preserving applications as the data is not sent to the upstream model for fine-tuning.

Data-efficiency and generalization are key challenges in deep learning and deep reinforcement learning as many models are trained on large-scale, domain-specific, and expensive-to-label datasets. Self-supervised models trained on large-scale uncurated datasets have shown successful transfer to diverse settings. We investigate using pretrained image representations and spatio-temporal attention for state representation learning in Atari. We also explore fine-tuning pretrained representations with self-supervised techniques, i.e., contrastive predictive coding, spatio-temporal contrastive learning, and augmentations. Our results show that pretrained representations are at par with state-of-the-art self-supervised methods trained on domain-specific data. Pretrained representations, thus, yield data and compute-efficient state representations. https://github.com/PAL-ML/PEARL_v1

Egocentric visual context detection can support intelligence augmentation applications. We created a wearable system, called PAL, for wearable, personalized, and privacy-preserving egocentric visual context detection. PAL has a wearable device with a camera, heart-rate sensor, on-device deep learning, and audio input/output. PAL also has a mobile/web application for personalized context labeling. We used on-device deep learning models for generic object and face detection, low-shot custom face and context recognition (e.g., activities like brushing teeth), and custom context clustering (e.g., indoor locations). The models had over 80\% accuracy in in-the-wild contexts (~1000 images) and we tested PAL for intelligence augmentation applications like behavior change. We have made PAL is open-source to further support intelligence augmentation using personalized and privacy-preserving egocentric visual contexts.

The study of nonverbal vocalizations, such as sighs, grunts, and monosyllabic sounds, has largely revolved around the social and affective implications of these sounds within typical speech. However, for individuals who do not use any traditional speech, including those with non- or minimally verbal (nv/mv) autism, these vocalizations contain important, individual-specific affective and communicative information. This paper outlines the methodology, analysis, and technology to investigate the production, perception, and meaning of nonverbal vocalizations from nv/mv individuals in natural environments. We are developing novel signal processing and machine learning methods that will help enable augmentative communication technology, and we are producing a nonverbal vocalization dataset for public release. We hope this work will expand the scientific understanding of these exceptional individuals’ language development and the field of communication more generally., Laughter and Other Non-Verbal Vocalisations Workshop: Proceedings (2020), Laughter and Other Non-​Verbal Vocalisations Workshop 2020

Reliability in Neural Networks (NNs) is crucial in safety-critical applications like healthcare, and uncertainty estimation is a widely researched method to highlight the confidence of NNs in deployment. In this work, we propose an uncertainty-aware boosting technique for multi-modal ensembling to predict Alzheimer's Dementia Severity. The propagation of uncertainty across acoustic, cognitive, and linguistic features produces an ensemble system robust to heteroscedasticity in the data. Weighing the different modalities based on the uncertainty estimates, we experiment on the benchmark ADReSS dataset, a subject-independent and balanced dataset, to show that our method outperforms the state-of-the-art methods while also reducing the overall entropy of the system. This work aims to encourage fair and aware models. The source code is available at https://github.com/wazeerzulfikar/alzheimers-dementia, Comment: To appear at NeurIPS Machine Learning for Health (ML4H) 2020

Understanding and quantifying uncertainty in black box Neural Networks (NNs) is critical when deployed in real-world settings such as healthcare. Recent works using Bayesian and non-Bayesian methods have shown how a unified predictive uncertainty can be modelled for NNs. Decomposing this uncertainty to disentangle the granular sources of heteroscedasticity in data provides rich information about its underlying causes. We propose a conceptually simple non-Bayesian approach, deep split ensemble, to disentangle the predictive uncertainties using a multivariate Gaussian mixture model. The NNs are trained with clusters of input features, for uncertainty estimates per cluster. We evaluate our approach on a series of benchmark regression datasets, while also comparing with unified uncertainty methods. Extensive analyses using dataset shits and empirical rule highlight our inherently well-calibrated models. Our work further demonstrates its applicability in a multi-modal setting using a benchmark Alzheimer's dataset and also shows how deep split ensembles can highlight hidden modality-specific biases. The minimal changes required to NNs and the training procedure, and the high flexibility to group features into clusters makes it readily deployable and useful. The source code is available at https://github.com/wazeerzulfikar/deep-split-ensembles, Comment: 9 pages including references, + 10 pages appendix

This paper introduces studies conducted with musicians that aim to understand modes of human-robot interaction, situated between automation and human augmentation. Our robotic guitar system used for the study consists of various sound generating mechanisms, either driven by software or by a musician directly. The control mechanism allows the musician to have a varying degree of agency over the overall musical direction. We present interviews and discussions on open-ended experiments conducted with music students and musicians. The outcome of this research includes new modes of playing the guitar given the robotic capabilities, and an understanding of how automation can be integrated into instrument-playing processes. The results present insights into how a human-machine hybrid system can increase the efficacy of training or exploration, without compromising human engagement with a task.

Personalized Active Learner (PAL) is a wearable system for real-time, personalized, and context-aware health and cognition support. PAL's system consists of a wearable device, mobile app, cloud database, data visualization web app, and machine learning server. PAL's wearable device uses multi-modal sensors (camera, microphone, heart-rate) with on-device machine learning and open-ear audio output to provide real-time and context-aware cognitive, behavioral and psychological interventions. PAL also allows users to track the long-term correlations between their activities and physiological states to make well-informed lifestyle decisions. In this paper, we present and open-source PAL's system so that people can use it for health and cognition support applications. We also open-source three fully-developed example applications using PAL for face-based memory augmentation, contextual language learning, and heart-rate-based psychological support. PAL's flexible, modular and extensible platform combines trends in data-driven medicine, mobile psychology, and cognitive enhancement to support data-driven and empowering health and cognition applications.

The rise of technology that supports a partnership between user and computer highlights an opportunity for a new era of "human-computer integration", contrasting the previously dominant paradigm of computers functioning as tools. However, most work around these technologies only focused on the instrumental perspective to achieve extrinsic performance objectives. However, phenomenology emphasizes that it is also important to support the experiential perspective, which indicates that technology should also help people pay attention to their lived experiences and personal growth in order to deepen their understanding of their own bodies. This seminar focuses on embodied integration, where a computer tightly integrates with the person's body. Although an increasing number of systems are emerging, a thorough understanding of how to design such systems is notably absent. The reason for this is the limited knowledge about how such embodied partnerships unfold, and what underlying theory could guide such developments. This seminar brought together leading experts from industry and academia, including those who are central to the development of products and ideas such as wearables, on-body robotics, and exertion systems. The goal was to address key questions around the design of embodied integration and to jump-start collaborations to pioneer new approaches for a human-computer integrated future.

© 2018 Copyright held by the owner/author(s). Virtual reality can help realize mediated social experiences where distance disappears and we interact as richly with those around the world as we do with those in the same room. The design of social virtual experiences presents a challenge for remotely located users with room-scale setups like those afforded by recent commodity virtual reality devices. Since users inhabit different physical spaces that may not be the same size, a mapping to a shared virtual space is needed for creating experiences that allow everyone to use real walking for locomotion. We designed three mapping techniques that enable users from diverse room-scale setups to interact together in virtual reality. Results from our user study (N = 26) show that our mapping techniques positively influence the perceived degree of togetherness and copresence while the size of each user's tracked space influences individual presence.

Playing musical instruments involves producing gradually more challenging body movements and transitions, where the kinematic constraints of the body play a crucial role in structuring the resulting music. We seek to make a bridge between currently accessible motor patterns, and musical possibilities beyond those - afforded through the use of a robotic augmentation. Guitar Machine is a robotic device that presses on guitar strings and assists a musician by fretting alongside her on the same guitar. This paper discusses the design of the system, strategies for using the system to create novel musical patterns, and a user study that looks at the effects of the temporary acquisition of enhanced physical ability. Our results indicate that the proposed human-robot interaction would equip users to explore new musical avenues on the guitar, as well as provide an enhanced understanding of the task at hand on the basis of the robotically acquired ability.

We present the first real-time sleep staging system that uses deep learning without the need for servers in a smartphone application for a wearable EEG. We employ real-time adaptation of a single channel Electroencephalography (EEG) to infer from a Time-Distributed 1-D Deep Convolutional Neural Network. Polysomnography (PSG)-the gold standard for sleep staging, requires a human scorer and is both complex and resource-intensive. Our work demonstrates an end-to-end on-smartphone pipeline that can infer sleep stages in just single 30-second epochs, with an overall accuracy of 83.5% on 20-fold cross validation for five-class classification of sleep stages using the open Sleep-EDF dataset., Comment: Machine Learning for Health (ML4H) Workshop at NeurIPS 2018 arXiv:1811.07216

iLRN 2018 Montana

Virtual Reality (VR) has the potential to overcome natural constraints and present things that would not be visible in the physical world. This makes the medium of VR a powerful tool for learning that allows users to become highly immersed in complex topics. In this paper, we compare a VR learning environment with a traditional 2D learning environment. To investigate the differences between VR and 2D learning environments, we designed two activities that help learners gain an intuitive understanding of concepts from electricity and magnetism.We performed an experiment comparing the learning that took place using these two variant. Although our quantitative learning measures did not reveal a significant difference between 2D and VR, VR was perceived by learners to have advantages.We did find significant quantitative differences in learners' completion times. We share findings, based on the quantitative and qualitative feedback received, about what makes VR environments beneficial for learning about complex spatial topics, and propose corresponding design guidelines.

© Springer International Publishing AG 2017. Room-scale virtual reality (VR) holds great potential as a medium for communication and collaboration in remote and same-time, same-place settings. Related work has established that movement realism can create a strong sense of social presence, even in the absence of photorealism. Here, we explore the noteworthy attributes of communicative interaction using embodied minimal avatars in room-scale VR in the same-time, same-place setting. Our system is the first in the research community to enable this kind of interaction, as far as we are aware. We carried out an experiment in which pairs of users performed two activities in contrasting variants: VR vs. face-to-face (F2F), and 2D vs. 3D. Objective and subjective measures were used to compare these, including motion analysis, electrodermal activity, questionnaires, retrospective think-aloud protocol, and interviews. On the whole, participants communicated effectively in VR to complete their tasks, and reported a strong sense of social presence. The system’s high fidelity capture and display of movement seems to have been a key factor in supporting this. Our results confirm some expected shortcomings of VR compared to F2F, but also some non-obvious advantages. The limited anthropomorphic properties of the avatars presented some difficulties, but the impact of these varied widely between the activities. In the 2D vs. 3D comparison, the basic affordance of freehand drawing in 3D was new to most participants, resulting in novel observations and open questions. We also present methodological observations across all conditions concerning the measures that did and did not reveal differences between conditions, including unanticipated properties of the think-aloud protocol applied to VR.

With the advances in Virtual Reality (VR) and physiological sensing technology, even more immersive computer-mediated communication through life-like characteristics is now possible. In response to the current lack of culture, expression and emotions in VR avatars, we propose a two-fold solution. First, integration of bio-signal sensors into the HMD and techniques to detect aspects of the emotional state of the user. Second, the use of this data to generate expressive avatars which we refer to as Emotional Beasts. The creation of Emotional Beasts will allow us to experiment with the manipulation of a user's selfexpression in VR space and as well as the perception of others in it, providing some valuable tools to evoke a desired emotional reaction. As this medium moves forward, this and other tools are what will help the field of virtual reality expand from a medium of surface-level experience to one of deep, emotionally compelling human-to-human connection.

The 8th Augmented Human International Conference (AH2017) has gathered scientific papers from many different disciplines. Many papers concentrated on building the human augmentation technologies, which is necessary for them to emerge in the real world. However, few papers investigated the ethical or safety issues of augmented human technologies. The next edition may bring more papers on this essential aspect that must be taken into account for a long-term success of these technologies.

© Springer International Publishing AG 2017. This paper explores potential use cases for using augmented reality (AR) as a tool to operate industrial machines. As a baseline we use an additive manufacturing system, more commonly known as a 3D printer. We implement novel augmented interfaces and controls using readily available open source frameworks and low cost hardware. Our results show that the technology enables richer and more intuitive printer control and performance monitoring than currently available on the market. Therefore, there is a great deal of potential for these types of technologies in future digital factories.

This paper explores potential use cases for using augmented reality (AR) as a tool to operate industrial machines. As a baseline we use an additive manufacturing system, more commonly known as a 3D printer. We implement novel augmented interfaces and controls using readily available open source frameworks and low cost hardware. Our results show that the technology enables richer and more intuitive printer control and performance monitoring than currently available on the market. Therefore, there is a great deal of potential for these types of technologies in future digital factories.

© 2016 ACM. The Experience Sampling Method is widely used for collecting self-report responses from people in natural settings. While most traditional approaches rely on using a phone to trigger prompts and record information, wearable devices now offer new opportunities that may improve this method. This research quantitatively and qualitatively studies the experience sampling process on head-worn and wrist-worn wearable devices, and compares them to the traditional "smartphone in the pocket." To enable this work, we designed and implemented a custom application to provide similar prompts across the three types of devices and evaluated it with 15 individuals for five days (75 days total), in the context of real-life stress measurement. We found significant differences in response times across devices, and captured tradeoffs in interaction types, screen size, and device familiarity that can affect both users' experience and the reports made by users.

Description

Description

The opportunities for wearable technologies go well beyond always-available information displays or health sensing devices. The concept of the cyborg introduced by Clynes and Kline, along with works in various fields of research and the arts, offers a vision of what technology integrated with the body can offer. This paper identifies different categories of research aimed at augmenting humans. The paper specifically focuses on three areas of augmentation of the human body and its sensorimotor capabilities: physical morphology, skin display, and somatosensory extension. We discuss how such digital extensions relate to the malleable nature of our self-image. We argue that body-borne devices are no longer simply functional apparatus, but offer a direct interplay with the mind. Finally, we also showcase some of our own projects in this area and shed light on future challenges.

The Action Selection problem is the problem of run-time choice between conflicting and heterogenous goals, a central problem in the simulation of whole creatures (as opposed to the solution of isolated uninterrupted tasks). This thesis argues that Reinforcement Learning has been overlooked in the solution of the Action Selection problem. Considering a decentralised model of mind, with internal tension and competition between selfish behaviors, this thesis introduces an algorithm called “W-learning”, whereby different parts of the mind modify their behavior based on whether or not they are succeeding in getting the body to execute their actions. This thesis sets W-learning in context among the different ways of exploiting Reinforcement Learning numbers for the purposes of Action Selection. It is a ‘Minimize the Worst Unhappiness’ strategy. The different methods are tested and their strengths and weaknesses analysed in an artificial world.