Search

User interfaces often utilise imagery of pristine natural environments, even if the system’s purpose and context are unrelated to nature. In this paper, we build on evolutionary psychology to devel...

Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications., Comment: To be published in IEEE Transactions on Affective Computing. 20 pages, 7 figures, 6 tables

Complementors accumulate reputation on an ever-increasing number of online platforms. While the effects of reputation within individual platforms are well-understood, its potential effectiveness across platform boundaries has received much less attention. This research note considers complementors’ ability to increase their trustworthiness in the eyes of prospective consumers by importing reputational data from another platform. The study evaluates this potential lever by means of an online experiment, during which specific combinations of on-site and imported rating scores are tested. Results reveal that importing reputation can be advantageous – but also detrimental, depending on ratings’ values. Implications for complementors, platform operators, and regulatory bodies concerned with online reputation are considered.

Wrist-worn inertial measurement units have emerged as a promising technology to passively capture dietary intake data. State-of-the-art approaches use deep neural networks to process the collected inertial data and detect characteristic hand movements associated with intake gestures. In order to clarify the effects of data preprocessing, sensor modalities, and sensor positions, we collected and labeled inertial data from wrist-worn accelerometers and gyroscopes on both hands of 100 participants in a semi-controlled setting. The method included data preprocessing and data segmentation, followed by a two-stage approach. In Stage 1, we estimated the probability of each inertial data frame being intake or non-intake, benchmarking different deep learning models and architectures. Based on the probabilities estimated in Stage 1, we detected the intake gestures in Stage 2 and calculated the F1 score for each model. Results indicate that top model performance was achieved by a CNN-LSTM with earliest sensor data fusion through a dedicated CNN layer and a target matching technique (F1 = .778). As for data preprocessing, results show that applying a consecutive combination of mirroring, removing gravity effect, and standardization was beneficial for model performance, while smoothing had adverse effects. We further investigate the effectiveness of using different combinations of sensor modalities (i.e., accelerometer and/or gyroscope) and sensor positions (i.e., dominant intake hand and/or non-dominant intake hand).

With the ever-growing popularity of sharing economy platforms, complementors increasingly face the challenge to manage their reputation on different platforms. The paper reports the results from an experimental online survey to investigate how and under which conditions online reputation is effective to engender trust across platform boundaries. It shows that (1) cross-platform signaling is in fact a viable strategy to engender trust and that (2) its effectiveness crucially depends on source–target fit. Implications for three stakeholders are discussed. First, platform complementors may benefit from importing reputation, especially when they have just started on a new platform and have not earned on-site reputation yet. The results also show, however, that importing reputation (even if it is excellent) may be detrimental if there occurs a mismatch between source and target and that, hence, fit is of utmost importance. Second, regulatory authorities may consider reputation portability as a means to make platform boundaries more permeable and hence to tackle lock-in effects. Third, platform operators may employ cross-platform signaling as a competitive lever.

In today's fast-paced society, users of information technology increasingly operate under high time pressure. Loaded with multiple tasks and racing against deadlines, users experience considerable ...

Online user representation (UR) is a cornerstone of platform-mediated interactions within the sharing economy. While the general usefulness of UR artifacts for facilitating online and offline interactions is widely acknowledged and understood, the underlying mechanisms and operating principles often require a more detailed analysis. In this chapter, we thus introduce a systematic framework grounded in signaling and social presence theory for analyzing UR artifacts for online platforms in general—and the sharing economy in particular. We apply our framework as a structural lens in a case study on user profiles on Airbnb, unveiling structural similarities and differences between the opposing market sides. We discuss our findings against the backdrop of emerging information systems research directions and suggest paths for future work on the sharing economy.

Automatic detection of intake gestures is a key element of automatic dietary monitoring. Several types of sensors, including inertial measurement units (IMU) and video cameras, have been used for this purpose. The common machine learning approaches make use of the labeled sensor data to automatically learn how to make detections. One characteristic, especially for deep learning models, is the need for large datasets. To meet this need, we collected the Objectively Recognizing Eating Behavior and Associated Intake (OREBA) dataset. The OREBA dataset aims to provide comprehensive multi-sensor data recorded during the course of communal meals for researchers interested in intake gesture detection. Two scenarios are included, with 100 participants for a discrete dish and 102 participants for a shared dish, totalling 9069 intake gestures. Available sensor data consists of synchronized frontal video and IMU with accelerometer and gyroscope for both hands. We report the details of data collection and annotation, as well as details of sensor processing. The results of studies on IMU and video data involving deep learning models are reported to provide a baseline for future research. Specifically, the best baseline models achieve performances of $F_1$ = 0.853 for the discrete dish using video and $F_1$ = 0.852 for the shared dish using inertial data., To be published in IEEE Access

Remote photoplethysmography (rPPG) allows remote measurement of the heart rate using low-cost RGB imaging equipment. In this study, we review the development of the field of rPPG since its emergence in 2008. We also classify existing rPPG approaches and derive a framework that provides an overview of modular steps. Based on this framework, practitioners can use our classification to design algorithms for an rPPG approach that suits their specific needs. Researchers can use the reviewed and classified algorithms as a starting point to improve particular features of an rPPG algorithm.

Background The proliferation of mobile devices has enabled new ways of delivering health services through mobile health systems. Researchers and practitioners emphasize that the design of such systems is a complex endeavor with various pitfalls, including limited stakeholder involvement in design processes and the lack of integration into existing system landscapes. Co-design is an approach used to address these pitfalls. By recognizing users as experts of their own experience, co-design directly involves users in the design process and provides them an active role in knowledge development, idea generation, and concept development. Objective Despite the existence of a rich body of literature on co-design methodologies, limited research exists to guide the co-design of mobile health (mHealth) systems. This study aims to contextualize an existing co-design framework for mHealth applications and construct guidelines to address common challenges of co-designing mHealth systems. Methods Tapping into the knowledge and experience of experts in co-design and mHealth systems development, we conducted an exploratory qualitative study consisting of 16 semistructured interviews. Thereby, a constructivist ontological position was adopted while acknowledging the socially constructed nature of reality in mHealth system development. Purposive sampling across web-based platforms (eg, Google Scholar and ResearchGate) and publications by authors with co-design experience in mHealth were used to recruit co-design method experts (n=8) and mHealth system developers (n=8). Data were analyzed using thematic analysis along with our objectives of contextualizing the co-design framework and constructing guidelines for applying co-design to mHealth systems development. Results The contextualized framework captures important considerations of the mHealth context, including dedicated prototyping and implementation phases, and an emphasis on immersion in real-world contexts. In addition, 7 guidelines were constructed that directly pertain to mHealth: understanding stakeholder vulnerabilities and diversity, health behavior change, co-design facilitators, immersion in the mHealth ecosystem, postdesign advocates, health-specific evaluation criteria, and usage data and contextual research to understand impact. Conclusions System designers encounter unique challenges when engaging in mHealth systems development. The contextualized co-design framework and constructed guidelines have the potential to serve as a shared frame of reference to guide the co-design of mHealth systems and facilitate interdisciplinary collaboration at the nexus of information technology and health research.

The information systems (IS) field contains a rich body of knowledge on approaches, methods, and frameworks that supports researchers in conducting design science research (DSR). It also contains some consensus about the key elements of DSR projects—such as problem identification, design, implementation, evaluation, and abstraction of design knowledge. Still, we lack any commonly accepted tools that address the needs of DSR scholars who seek to structure, manage, and present their projects. Indeed, DSR endeavors, which are often complex and multi-faceted in nature and involve various stakeholders (e.g., researchers, developers, practitioners, and others), require the support that such tools provide. Thus, to investigate the tools that DSR scholars actually need to effectively and efficiently perform their work, we conducted an open workshop with DSR scholars at the 2017 DESRIST conference in Karlsruhe, Germany, to debate 1) the general requirement categories of DSR tool support and 2) the more specific requirements. This paper reports on the results from this workshop. Specifically, we identify nine categories of requirements that fall into the three broad phases (pre-design, design, and post design) and that contribute to a software ecosystem for supporting DSR endeavors.

Accurate detection of individual intake gestures is a key step towards automatic dietary monitoring. Both inertial sensor data of wrist movements and video data depicting the upper body have been used for this purpose. The most advanced approaches to date use a two-stage approach, in which (i) frame-level intake probabilities are learned from the sensor data using a deep neural network, and then (ii) sparse intake events are detected by finding the maxima of the frame-level probabilities. In this study, we propose a single-stage approach which directly decodes the probabilities learned from sensor data into sparse intake detections. This is achieved by weakly supervised training using Connectionist Temporal Classification (CTC) loss, and decoding using a novel extended prefix beam search decoding algorithm. Benefits of this approach include (i) end-to-end training for detections, (ii) simplified timing requirements for intake gesture labels, and (iii) improved detection performance compared to existing approaches. Across two separate datasets, we achieve relative $F_1$ score improvements between 1.9% and 6.2% over the two-stage approach for intake detection and eating/drinking detection tasks, for both video and inertial sensors.

Automatic detection of individual intake gestures during eating occasions has the potential to improve dietary monitoring and support dietary recommendations. Existing studies typically make use of on-body solutions such as inertial and audio sensors, while video is used as ground truth. Intake gesture detection directly based on video has rarely been attempted. In this study, we address this gap and show that deep learning architectures can successfully be applied to the problem of video-based detection of intake gestures. For this purpose, we collect and label video data of eating occasions using 360-degree video of 102 participants. Applying state-of-the-art approaches from video action recognition, our results show that (1) the best model achieves an $F_1$ score of 0.858, (2) appearance features contribute more than motion features, and (3) temporal context in form of multiple video frames is essential for top model performance., To be published in IEEE Journal of Biomedical and Health Informatics

In the NeuroISfield, experimental software needs to simultaneously present experimental stimuli to participants while recording, analyzing, or displaying neurophysiological measures. For example, a researchermight record a user’s heart beat (neurophysiological measure) asthe userinteracts with an e-commerce website (stimulus) to track changes in user arousalorshowa user’schanging arousal levels during an exciting game. In this paper, we identify requirements for a NeuroISexperimental platform that we call Brownie and present its architecture and functionality. We then evaluate Brownie viaa literature review and a case study that demonstrates Brownie’s capability to meet the requirements in a complex research context. We also verify Brownie’susabilityviaa quantitative study with prospective experimenters who implemented a test experiment in Brownie and an alternative software. We summarize the salient features of Brownie as follows: 1) it integrates neurophysiological measurements, 2) it incorporatesreal-time processing of neurophysiological data, 3i) it facilitates research on individual and group behavior in the lab, 4) it offers a large variety of options for presenting experimental stimuli, and 5) it is opensource and easily extensible with opensource libraries. In summary, we conclude that Brownie is innovativein its potential to reduce barriers for IS researchers by fostering replicability and research collaborationand to supportNeuroISand interdisciplinary research in cognate areas, such as management, economics, or human-computer interaction.

Purpose Human lab experiments have become an established method in information systems research for investigating user behavior, perception and even neurophysiology. The purpose of this paper is to facilitate experimental research by providing a practical guide on how to implement and conduct lab experiments in the freely available experimental platform Brownie. Design/methodology/approach Laying the groundwork of the tutorial, the paper first provides a brief overview of common design considerations for lab experiments and a generic session framework. Building on the use case of the widely used trust game, the paper then covers the different stages involved in running an experimental session and maps the conceptual elements of the study design to the implementation of the experimental software. Findings The paper generates findings on how computerized lab experiments can be designed and implemented. Furthermore, it maps out the design considerations an experimenter may take into account when implementing an experiment and organizing it along a session structure (e.g. participant instructions, individual and group interaction, state and trait questionnaires). Originality/value The paper reduces barriers for researchers to engage in experiment implementation and replication by providing a step-by-step tutorial for the design and implementation of human lab experiments.

While avatars are frequently employed as a user interface (UI) element for improving user experience in human-computer interaction, the current design of avatars is primarily dominated by non-Arabian cultures. To the best of our knowledge, no previous research or guidelines (based on empirical evidence) can be found for avatar design in the context of Arabian culture. Aiming to address this gap, we conducted an exploratory study to investigate how avatars can be designed for Arab users. Following a hybrid approach of deductive and inductive reasoning, we reviewed the literature on UI design for Arabian culture, (non-Arabian) avatar design in human-computer interaction research, and social response theory. Subsequently, we conducted 32 semi-structured interviews with Arabian culture experts, psychologists, and potential users. Based on thematic analysis of the interviews, we developed a set of six general guidelines for the design of avatars for users from Arabian culture. These six guidelines are expected to provide system or UI designers the ability to design and employ appropriate avatars that can promote Arab users’ experience and engagement.

Cybersecurity is a growing concern for private individuals and professional entities. Thereby, reports have shown that the majority of cybersecurity incidents occur because users fail to behave securely. Research on human cybersecurity (HCS) behavior suggests that time pressure is one of the important driving factors behind insecure HCS behavior. However, as our review reveals, studies on the role of time pressure in HCS are scant and there is no framework that can inform researchers and practitioners on this matter. In this paper, we present a conceptual framework consisting of contexts, psychological constructs, and boundary conditions pertaining to the role time pressure plays on HCS behavior. The framework is also validated and extended by findings from semi-structured interviews of different stakeholder groups comprising of cybersecurity experts, professionals, and general users. The framework will serve as a guideline for future studies exploring different aspects of time pressure in cybersecurity contexts and also to identify potential countermeasures for the detrimental impact of time pressure on HCS behavior.

Stock market prediction plays an important role in financial decision-making for investors. Many of them rely on news disclosures to make their decisions in buying or selling stocks. However, accurate modelling of stock market trends via news disclosures is a challenging task, considering the complexity and ambiguity of natural languages used. Unlike previous work along this line of research, which typically applies bag-of-words to extract tens of thousands of features to build a prediction model, we propose a sentiment analysis-based approach for financial market prediction using news disclosures. Specifically, sentiment analysis is carried out in the pre-processing phase to extract sentiment-related features from financial news. Historical stock market data from the perspective of time series analysis is also included as an input feature. With the extracted features, we use a support vector machine (SVM) to build the prediction model, with its parameters optimised through particle swarm optimisation (PSO). Experimental results show that our proposed SVM and PSO-based model is able to obtain better results than a deep learning model in terms of time and accuracy. The results presented here are to date the best in the literature based on the financial news dataset tested. This excellent performance is attributed to the sentiment analysis done during the pre-processing stage, as it reduces the feature dimensions significantly.

Emotion elicitation and classification have been performed on standardized stimuli sets, such as international affective picture systems and international affective digital sound. However, the literature which elicits and classifies emotions in a financial decision making context is scarce. In this paper, we present an evaluation to detect emotions of private investors through a controlled trading experiment. Subjects reported their level of rejoice and regret based on trading outcomes, and physiological measurements of skin conductance response and heart rate were obtained. To detect emotions, three labeling methods, namely binary, tri-, and tetrastate blended models were compared by means of C4.5, CART, and random forest algorithms, across different window lengths for heart rate. Taking moving window lengths of 2.5s prior to and 0.3s postevent (parasympathetic phase) led to the highest accuracies. Comparing labeling methods, accuracies were 67 for binary rejoice, 44 for a tristate, and 45 for a tetrastate blended emotion models. The CART yielded the highest accuracies.

Traders and investors are aware that emotional processes can have material consequences on their financial decision performance. However, typical learning approaches for debiasing fail to overcome emotionally driven financial dispositions, mostly because of subjects' limited capacity for self-monitoring. Our research aims at improving decision makers' performance by (1) boosting their awareness to their emotional state and (2) improving their skills for effective emotion regulation. To that end, we designed and implemented a serious game-based NeuroIS tool that continuously displays the player's individual emotional state, via biofeedback, and adapts the difficulty of the decision environment to this emotional state. The design artifact was then evaluated in two laboratory experiments. Taken together, our study demonstrates how information systems design science research can contribute to improving financial decision making by integrating physiological data into information technology artifacts. Moreover,...

Stress is a major problem in the human society, impairing the well-being, health, performance, and productivity of many people worldwide. Most notably, people increasingly experience stress during human-computer interactions because of the ubiquity of and permanent connection to information and communication technologies. This phenomenon is referred to as technostress. Enterprise systems, designed to improve the productivity of organizations, frequently contribute to this technostress and thereby counteract their objective. Based on theoretical foundations and input from exploratory interviews and focus group discussions, the paper presents a design blueprint for stress-sensitive adaptive enterprise systems (SSAESes). A major characteristic of SSAESes is that bio-signals (e.g., heart rate or skin conductance) are integrated as real-time stress measures, with the goal that systems automatically adapt to the users’ stress levels, thereby improving human-computer interactions. Various design interventions on the individual, technological, and organizational levels promise to directly affect stressors or moderate the impact of stressors on important negative effects (e.g., health or performance). However, designing and deploying SSAESes pose significant challenges with respect to technical feasibility, social and ethical acceptability, as well as adoption and use. Considering these challenges, the paper proposes a 4-stage step-by-step implementation approach. With this Research Note on technostress in organizations, the authors seek to stimulate the discussion about a timely and important phenomenon, particularly from a design science research perspective.

Heart rate measurements contain valuable information about a person’s affective state. There is a wide range of application domains for heart rate-based measures in information systems. To date, heart rate is typically measured using skin contact methods, where users must wear a measuring device. A non-contact and easy to use mobile approach, allowing heart rate measurements without interfering with the users’ natural environment, could prove to be a valuable NeuroIS tool. Hence, our two research objectives are (1) to develop an application for mobile devices that allows for non-contact, real-time heart rate measurement and (2) to evaluate this application in an IS context by benchmarking the results of our approach against established measurements. The proposed algorithm is based on non-contact photoplethysmography and hence takes advantage of slight skin color variations that occurs periodically with the user’s pulse.

Affective processes play an important role in determining human behavior in auctions. While previous research has shown that physiological measurements provide insights into these processes, it remains unclear which of the many features that can be computed from physiological data are particularly useful in predicting human behavior. Identifying these features is important for gaining a better understanding of affective processes in electronic auctions and for building biofeedback systems. In this study, we propose a new approach to identify physiological features for predicting auction behavior. We apply an Evolutionary Algorithm in combination with either the Multiple Linear Regression or Artificial Neural Network models to select physiological features and assess their predictive power. To test the approach, we use a unique dataset of participants' auction decisions and their synchronously recorded electrocardiography data. Our results show that the approach is able to identify subsets of physiological features that consistently outperform other physiological features.

This chapter describes cluster evaluation, description, and interpretation for evaluating player profiles based on log files available from a game server. Calculated variables were extracted from these logs in order to characterize players. Using circular statistics, we show how measures can be extracted that enable players to be characterized by the mean and standard deviation of the time that they interacted with the server. Feature selection was accomplished using a correlation study of variables extracted from the log data. This process favored a small number of the features, as judged by the results of clustering. The techniques are demonstrated based on a log file data set of the popular online game Minecraft. Automated clustering was able to suggest groups that Minecraft players fall into. Cluster evaluation, description, and interpretation techniques were applied to provide further insight into distinct behavioral characteristics, leading to a determination of the quality of clusters, using the Silhouette Width measure. We conclude by discussing how the techniques presented in this chapter can be applied in different areas of serious games analytics.