Search

Human activity recognition is a key to a lot of applications such as healthcare and smart home. In this study, we provide a comprehensive survey on recent advances and challenges in human activity recognition (HAR) with deep learning. Although there are many surveys on HAR, they focused mainly on the taxonomy of HAR and reviewed the state-of-the-art HAR systems implemented with conventional machine learning methods. Recently, several works have also been done on reviewing studies that use deep models for HAR, whereas these works cover few deep models and their variants. There is still a need for a comprehensive and in-depth survey on HAR with recently developed deep learning methods.

Cybersecurity is emerging as a major issue for many organizations and countries. Machine learning has been used to recognize threats, but it is difficult to predict future threats based on past events, since malicious attackers are constantly finding ways to circumvent defences and the algorithms that they rely on. Interactive Machine learning (iML) has been developed as a way to combine human and algorithmic expertise in a variety of domains and we are currently applying it to cybersecurity. In this application of iML, implicit knowledge about human behaviour, and about the changing nature of threats, can supplement the explicit knowledge encoded in algorithms to create more effective defences against cyber-attacks. In this paper we present the example problem of data exfiltration where insiders, or outsiders masquerading as insiders, who copy and transfer data maliciously, against the interests of an organization. We will review human factors issues associated with the development of iML solutions for data exfiltration. We also present a case study involving development of an iML solution for a large financial services company. In this case study we review work carried out on developing visualization dashboards and discussing prospects for further iML integration. Our goal in writing this paper is to motivate future researchers to consider the role of the human more fully in ML, not only in the data exfiltration and cybersecurity domain but also in a range of other applications where human expertise is important and needs to combine with ML prediction to solve challenging problems.

A novel quality assessment system design for closed captioning (CC) is proposed. CC was originally designed to serve hearing-impaired audiences. Traditional quality assessment models focus on empirical methods only, measuring quantitative accuracy by counting the number of word errors in the captions for a segment. However, hearing-impaired audiences have been outspoken about their dissatisfaction with the quality of current CC. One solution to this problem may involve inviting human evaluators, who represent different groups, to assess the quality of CC at the end of each segment, but in reality, this may be difficult to do and very labor-intensive. To solve this challenge, we propose the training and use of an artificial intelligence system to predict subjective CC quality based on human assessment data. In this paper, the design and development process, to be used in developing this system, is described .

There has been considerable research on design of menu hierarchies in general spanning several decades. However there is much less research on menus relating to specific types of product in online retail settings. Thus there is little guidance in the research literature on specific issues such as how to place luxury items within a beauty product hierarchy, which is the focus of this paper. We report on a study that addressed this problem for an ecommerce site associated with a large Canadian retailer. In a within subjects design, participants searched for four beauty-related products (two of which were classed as “luxury” items) either in a hierarchy where luxury items were intermingled with other products addressing the same need (the “Combined” condition), or in a hierarchy where there was a split between luxury and non-luxury products at the top level (the “Split” condition). Segregating luxury products in the product hierarchy was found to lead to significantly slower, and more lengthy (in terms of links traversed), searches. Searches were found to be more efficient in the “Combined” condition than the “Split” Condition both when searching for luxury items, and when searching for non-luxury items. This work has implications for existing brick-and-mortar retailers moving into or expanding e-commerce portals. Our results suggest that the separation of luxury from non-luxury items in bricks-and-mortar stores does not transfer well to online product hierarchies, where similar segregation leads to poorer digital navigation performance.

Closed Captioning (CC) is a service primarily designed for deaf and hard of hearing (D/HoH) viewers. The CC translates spoken speech into text for television or film screen display. The quality assessment methods for live captioning are limited to quantitative measures, while the viewers are still dissatisfied with the current quality. One method to improve the current quality assessment procedure is to include D/HoH viewers in the evaluation procedure for their subjective assessment input. However, it could be costly and impractical to perform evaluations for the entire broadcasted shows. Therefore, it would be helpful to model subjective assessments that could replicate and predict human decisions. In this article, we report on a model of probabilities of D/HoH viewer assessment decisions for CC quality factors based on actual user preferences. An online survey was designed and conducted to collect assessment data for 22 error variation samples from four quality factors: delay, speed, missing words, and paraphrasing of captions. The results are analyzed using the signal detection theory framework to create decision probability models for D/HoH viewers.

In this paper we explore how memories of experience with streaming video affect Quality of Experience (QoE) indicators that are of interest to service providers and marketers. Since observations of experience are time consuming, and the effects of technical quality (TQ) are difficult to entangle from content quality (CQ), we examined the impact of a visualization methodology for assessing experiences. A study was carried out to examine how well overall technical quality (TQ) judgments for a sequence of visualized video experience (a picture of a red video playbar with yellow portions indicating disrupted video in place of actually viewed video) would correspond to overall TQ judgments made after watching a sequence of actual videos. Sequencing effects found in overall TQ ratings, made after viewing visualizations (with their overlaid disruptions) were similar to sequencing effects found after viewing actual videos. However, the sequencing effects after viewing the visualizations were less pronounced than the corresponding sequencing effects that were found after viewing actual videos. Sequences of both visualized and actually viewed videos showed significant negative end effect and trend effects (both positive and negative). There was also evidence that sequencing effects respond to relative change in TQ rather than absolute TQ.

Curved displays promote three-dimensionality and may facilitate non-wearable virtual reality. Yet there is little design guidance on the optimal type of curvature that should be used. In this paper we have examined the perceived properties of convex and concave displays, at two different sizes. We conducted an experiment with 21 participants. Each participant was asked to make a series of binary choices after the participant viewed a display where one side was seen as convex and the opposite side was seen as concave. For each of 15 perceptual and aesthetic features, participants had to choose whether the convex or concave view/side of the display was stronger/more appropriate. Each participant made two binary choices (one for a small display viewed from its convex and concave sides, the other for a large display viewed from its convex and concave sides) for each perceptual and aesthetic feature, leading to a total of 30 judgments. Participants preferred the convex version of the small sized display and the concave version of the larger display. Individual differences were observed. Some participants generally preferred the convex displays, while others preferred the concave displays. The results are interpreted in terms of their implications for the future use of convex and concave displays in VR applications that do not use goggles.

Clustering prior to regression analysis improves the accuracy of prediction in clinical decision making. However, most previously described methods focused on numerical data only. This paper investigated how well textual features can improve the accuracy of regression predictions. Preliminary diagnosis, diagnosis summary, and drug names used in prescriptions as provided in the MIMIC II dataset were used to derive textual features. We proposed the bag-of-entities indexing method, which relies on named entity recognition, a machine learning technique used for locating and identifying words into predefined classes. The proposed technique captured meaningful phrases from texts in health records and represented them in numerical vector format. Dimensionality of the data space was reduced using principal component analysis. The additional well-tuned textual features were then combined with existing numerical features in using cluster-boosted regression to predict patient mortality in ICU. The experimental results showed prediction improvement obtained from textual features over the use of numerical features only. We found that using the proposed indexing method outperformed traditional word-vector representation approaches (bag-of-words and bag-of-bigrams) as well as a state-of-the-art approach (Doc2vec) in terms of resulting accuracy in predicting death status. Moreover, instead of directly interpreting, the identifiable individual features were grouped into types and summarized. The summarized de-identified data of textual features handled by the proposed framework can support predictive classification while also reducing privacy concerns. Grouping of similar patients based on their electronic health records also benefits physicians through the improved differential diagnosis and effective treatment planning.

This paper reports on the Patient Safety Research and Application Competition that was held in conjunction with the 2021 International Ergonomics Association Conference. The objectives of this competition were to: (1) Formulate research problem statements and innovative solutions to improve patient safety through the application of human factors/ergonomics (HF/E) to the healthcare system, (2) Showcase how the HF/E approach to this topic can lead to a useful, usable, and satisfying user experience while simultaneously improving outcomes relating to both functional and non-functional requirements, and (3) Provide an effective way of engaging students and early career researchers in IEA activities and initiatives. After reviewing the patient safety topics that motivated the design competition, we then report on the work carried out by the two finalists in the competition, and discuss lessons learned. We propose the continued use of design competitions in the future to motivate and showcase ergonomic problem-solving design by students and early career researchers and practitioners.

Serious games have grown significantly in popularity, but proving their scientific validity through research studies is a common hurdle for researchers and game developers. To scale up capacity to collaborate with different groups of researchers, Centivizer Inc. (a University of Toronto spinoff company) has employed a user-centered design process to design a BrainTagger Research Platform (BRP) that will largely automate the development process of its customizable serious games for cognitive assessment. This new development will increase the capacity to gather research data needed to improve game mechanisms and demonstrate game validity.

Data exfiltration is a serious threat to organizations. Such exfiltrations cause breach events that can lead to millions of dollars of loss. Perimeter defense is not enough by itself since successful exploits from insiders can also be very damaging. Internal network user activities need to be monitored to detect malicious actions. Automatic machine learning methods can be applied for network anomaly detection, but they create a lot of false alarms. Domain experts can identify malicious users, but they are unable to process large volumes of data. Interactive machine learning (iML) deals with this tradeoff by creating an efficient collaboration between domain experts and machine learning algorithms. Previous research in iML has focused mainly on collaboration with non-experts. The design and requirements for expertise-driven iML have yet to be delineated for cybersecurity applications. In this research, we proposed an Active Learning (AL) model trained with outputs from a liberal (outputting many false alarms as well as possible hits) anomaly detection (AD) criterion to study expert-iML collaboration in anomaly detection. The results showed that: iML in this context can prune false alarms and minimize misses; the performance/compatibility tradeoff that typically occurs in conventional machine learning updates may be less salient in iML. We suggest that compatibility between experts and algorithms can be improved by presenting information about feature relevance during the training process.

Accurately predicting the time to an event of interest is an important problem in a wide range of real-world applications. However, prediction is often difficult because many medical datasets have a large number of unlabeled (“censored”) instances because labeling is costly and time consuming. Survival analysis focuses on labeled data to predict the time to an event of interest, such as time of death, or conversion to a different stage in a progressive disease. Grouping structure, which naturally exists in medical datasets, can be exploited to improve generalization performance by learning multiple related survival prediction tasks for subgroups collaboratively. Thus a multi-task learning framework can connect multiple survival prediction tasks (for different subgroups) and learn them simultaneously. In order to take into account both censored information, as well as discover the grouping structure, we propose a novel cluster-boosted multitask learning framework for survival analysis that boosts survival prediction performance. We develop an efficient algorithm and demonstrate the performance of the proposed cluster-boosted multi-task survival analysis method on The Cancer Genome Atlas (TCGA) dataset. Our results show that the proposed approach can significantly improve prediction performance in survival analysis while also identifying different subgroups of cancer patients.

Quality of Experience (QoE) is a measure of the overall level of customer satisfaction with a vendor. In telecommunications, consumer satisfaction is of great interest in the adoption of novel multimedia products and services. A number of factors can greatly influence the customer experience during a video session. Factors such as user perception, experience, and expectations are expressed by QoE while factors such as application and network performance are expressed by Quality of Service (QoS) parameters. This paper studies the relationship between QoS and QoE in a session-based mobile video streaming. Specific QoS Application Performance Metrics (APMs) are examined based on a QoE assessment database which is built for experimentation and contains 108 subjects. It is shown that these APMs are highly related to two QoE factors, Technical Quality (TQ) and Acceptability. Furthermore, Viewing Ration (VR) parameter and the corresponding Kendall correlation between VR and QoE factors proves that VR is a valuable metric for mapping QoS to QoE. We further generated the compacted decision tree to predict QoE factors through Rebuffering Ratio (RR), Non-interruption Content Viewing Ratio (VRc), and Non-interruption Viewing Ratio (VRs). Through extensive experimentation, a general relationship between APMs and QoE factors has been examined and a primary QoE model is proposed based on this relationship.

Mobile computing provides new ways to interact with technology; applications such as navigation, social facilitation, and augmented reality are used while walking. We introduce Gait-Guided Adaptive Interfaces (GGAIs) as a way to manage cognitive load in dual-task conditions (walking while using a device). Gait markers that can be suitably assessed using smartphone sensors (decreased gait speed, increased variability) have been shown to be indicative of cognitive load in older adults. Motor-cognitive interference is a more significant issue as we age, gait becomes less automatic, and the risk of falls under distraction increases. Apps with GGAIs measure changes in gait to infer load and then adapt the way that the App interacts with the user accordingly. We validate this approach using a simple Go/No-Go task, and then show how gait responds to changes in task complexity. We conclude with a discussion of how GGAIs may be used by developers to improve the usability of apps for older users.

While previous research has shown that the sequencing of good and bad experience is an important predictor of overall evaluations of a set of experiences, the impact of sequencing effects on the experience of viewing online video has yet to be established. The aim of this study was to determine whether different sequences of good (G), mediocre (M), and bad (B) quality videos in different blocks would influence overall ratings after viewing those blocks. Thirty-two participants each watched 10 blocks of 4 videos and provided ratings of technical quality (TQ), satisfaction, and frustration for each video in the block, as well as overall ratings for each block (as a whole). Sequences of G, M, and B videos were designed to test whether block characteristics (features), like the peak-end effect and effect of linear trend, influenced summary evaluations of the block service. The results of the experiment show that overall block TQ, satisfaction, and frustration ratings differed significantly by sequencing feature. Difference scores were used to determine whether the features had an effect on overall evaluations beyond what could be explained by the total number of bad videos in the block or the average ratings of the videos in the block. Results showed a significant end effect for negative ends of a block, and an effect of linear trend (both increasing and decreasing). There was no evidence of a peak effect or an end effect for positive ends. The presence of a negative end effect and effect of linear trend indicate that where possible service providers should avoid service sessions with poor service quality at the end, or sessions that have decreasing quality as the session progresses.

This study measured Session Quality of Experience (SQE) by letting people look at visual proxies of a video play bar without having to watch the actual video, with its associated impairments and/or failures. We supplemented a laboratory study of 10 participants with two further online samples of 14, and 20, participants respectively, obtained using online surveys. Cluster analysis was carried out on the entire data set of 44 participants and clusters were obtained representing differences in sensitivity to pricing and to types of disruption. Crucially, there was no significant dependency between samples and cluster demonstrating that visualization can be used in a crowdsourcing, as well as a laboratory, context. This study demonstrates the combined use of crowdsourcing, visualizations of service experience, and cluster analysis to create more efficient, and nuanced, interpretations of SQE.

Technology questionnaires can assist in developing profiles that characterize the ability of people to use new technologies. Technology profiles may also be related to physical and cognitive abilities, and may possibly serve as proxies for constructs that may be more difficult to measure. The purpose of the research reported in this paper was to examine possible relationships between responses to questions about use of technologies and functional and frailty status as measured using a digital, tablet-based test battery. A battery of digitized cognitive and functional assessments was administered on a tablet, along with a technology questionnaire, to Canadian adults over 65 years who called 911 for paramedic services or who presented to an emergency department. 330 people between the ages of 65 and 97 years (mean = 75.8 years, standard deviation = 7.6) participated in the study. We observed significant relationships between elderly adults’ responses to questions about their technology use and their functional status and frailty scores with more technology use implying better functional status and less frailty. It is suggested that the present findings may lead to the use of more detailed technology profiles as efficient proxy estimates of overall functional ability and frailty status in elderly adults.

Walking and cognitive function are interrelated, and thus normal control of gait is perturbed by cognitive impairment. Changes in gait, as measured by quantitative analysis, can be a useful mechanism for detecting early indicators of cognitive decline. Observing changes in gait in the dual-task paradigm, i.e. when the subject walks while performing an attention-demanding task, can be a more sensitive measure. In the study reported in this paper, we developed a mobile application to perform quantitative gait assessment in the single and dual-task condition, using a smartphone’s built in sensor system. We show that the gait parameters, as measured by a mobile device, are significantly related to the corresponding parameters reported by a gold-standard motion capture system. We also demonstrate that this approach may be comparable to previously validated assessments of cognition and executive functioning. Mobile technologies offer a platform for sensitive assessment of changes in cognitive function through gait analysis in dual-task conditions. This methodology opens the way for more frequent assessments of cognition, as part of physiotherapy and rehabilitation practice.

This thesis introduces a new method to measure subjective ratings of technical quality (TQ) in disrupted video (DV). Current measurement methods are subject to extraneous factors such as delays introduced by the streaming software/hardware and the impact of video content. The proposed method is designed to supplement current methods by avoiding the confounding effects of content on TQ ratings and by reducing the time and effort necessary to run experiments. The new method replaces actual video with images representative of videos that contain various disruptions. In the research reported in this thesis, I ran an experiment with 37 participants to explore the viability of assessing TQ using visualizations instead of actual videos. The experiment compared ratings made after actually watching videos that varied in TQ versus ratings made after viewing the TQ visualizations, without actually seeing corresponding videos with associated impairments or failures. The results of the experiment showed that, with appropriate training, ratings of TQ made, after viewing the visualizations only, were similar to TQ ratings made after actually watching videos with corresponding impairments or failures.%%%%M.A.S.

This paper introduces research that addresses the subjective assessment of Quality of Experience (QoE) during the entire life cycle of a video session. We define a video session life cycle as the time from when a user attempts to initiate playback, until such time that the video ends either from normal video conclusion or through a network-induced failure. We provide a detailed description of our assessment methodology designed to discern whether a user's QoE would be impacted by the presence of failures. To accomphsh this, we carefully select various test conditions to take into consideration the rating scale used, the types of impairments and failures seen by the user, and whether impaired videos are seen together with failed videos in multi-video sessions. The selection and creation of source video sequences are also discussed, as well as the use of between-subjects and within-subjects approaches for running our experiments in a controlled laboratory setting. Statistical analysis was carried out to interpret our experimental results. We compared the results of the between-subjects measures and the results of the within-subjects measures, and concluded that the introduction of a scale with an extended lower bound enabled subjects to more clearly express their dissatisfaction of videos with failures when compared to the traditional ITU 5-point rating scale. In addition, we observed that videos that were simply impaired but concluded normally did not have a statistically significant difference when an extended scale was used.

In this paper we describe the development of a user interface for a liquid handling robot. We describe the user interface design process that we carried out, beginning with requirements analysis and rapid prototyping, followed by heuristic evaluation. We then demonstrate the resulting cloud interface where the robot is controlled remotely in a teleoperation mode to program common protocols in chemistry, and biology, laboratories. We describe the main characteristics of the web interface, focusing on solution strategies used to expose features for programming experiments, remotely. We also report on the System Usability Scale (SUS) scores obtained by testing the interface with 15 experts in the fields of chemistry, and biology.

With the current phenomenon of aging populations in most parts of the world, there are corresponding increases in age-related conditions associated with impaired cognitive status, such as dementia and delirium. Cognitive status is a key component in carrying out activities of daily living such as walking and bathing, and departures from normal cognitive status may be indicators of acute (e.g., delirium) or chronic (e.g., dementia) conditions. Individuals with cognitive impairments may benefit from playing serious games, which are games designed for a primary purpose other than entertainment. Serious games can potentially assess a variety of factors associated with cognitive decline in dementia, while keeping individuals active and stimulated, thereby potentially slowing down or furthering cognitive decline. In this workshop paper, we discuss the development and use of serious games focusing on cognitive functions that are affected by the progression of dementia.

Many countries are aging societies. Since abilities generally deteriorate with age, technologies can assist older adults in their daily life. Loss of cognitive status is particularly severe in cases of dementia, with around 70% (according to Alzheimers.net) of dementia cases involving Alzheimer's Dementia (AD), a progressive and currently incurable disease. There is considerable research on AD with thousands of relevant publications being added to the PubMed online database every year. The knowledge incorporated in this large body of work is spread across hundreds of thousands of pages of text, making it difficult to distill and mobilize that knowledge in terms of treatments and guidelines. Text mining technology may assist in distilling knowledge from the vast corpus of research literature on Alzheimer's dementia. In this paper, we apply the Named Entity Recognition (NER) system, a text mining (TM) method used to group words into classes, in order to extract useful information from free texts. We present findings concerning how well NER can extract information from a corpus of AD research publications.

Response time and accuracy are two of the most frequently collected dependent measures. Tradeoffs between speed and accuracy are often observed, both between people, and between experimental conditions. In this paper we consider how speed, and accuracy, can be combined into a single, overall measure of performance. We consider two different approaches that adjust accuracy scores based on observed speed of responding and we examine how well those measures work with different data sets. We then present a third approach that combines standardized speed and accuracy scores. We show how this latter approach can represent the data fairly well regardless of which (if any) speed-accuracy tradeoff occurs in the data. We also show how this measure can be further generalized by applying differential weightings to the standardized scores of speed, and accuracy, respectively. We conclude by discussing the value of the measure for use in analyzing human performance data where continuous indicators of accuracy or error can be collected or constructed relatively easily. Our goal in developing the global measure of performance is not to accurately model the speed-accuracy relationship, but rather to create a measure that is more sensitive to experimental differences and causal relationships than either speed or accuracy alone.

Results of an empirical study on the privacy concerns, attitudes, and behaviour of personal health record (PHR) users are presented. The study addressed the following questions: (1) What are the factors influencing privacy concerns of PHR users? (2) To what extent do PHR users read privacy agreements and which factors are in play when they fail to read them. (3) Are the behaviour and attitudes of PHR users consistent with respect to reading privacy agreements and using privacy settings? We infer from the study results that the factors influencing privacy concerns of general online users (as reported in literature) also apply to PHR users. In spite of privacy concerns, 60% of the respondents in our study reported not reading privacy agreements. PHR users who were highly concerned about privacy did not report changing their default privacy settings (as offered in a typical social networking website) in a manner consistent with their stated attitudes towards privacy. Based on the issues identified in the study we conclude with a number of design recommendations for privacy-friendly PHR systems.

Simple tools are needed to support differential diagnosis in a cognitively complex setting. Tools that show similar patients' diagnoses and treatment trajectories might provide useful clinical decision support for emergency physicians who use a case-based reasoning approach. However, privacy concerns that arise with indirect use of electronic health records (EHRs) must be addressed. The authors present a method to abstract a collection of EHRs into a set of summarized patient types and demonstrate its use on a database of medical records.

We conducted an experiment with 22 participants to investigate the effect of presentation style of a secondary task on a 1-D tracking task that simulated gap control in driving. Participants operated the tracking task with a foot pedal while performing a secondary task (counting vowels in a list of multiple letters) under conditions involving different modalities (audio/ visual), presentation styles (simultaneous/ sequential), task complexity (the number of distractors), and time dependency (list length). Our results showed that audio conditions with a longer and/or more complex secondary task did not improve primary (tracking) task performance, even though eye gaze dwelling time on the primary monitor in these cases tended to be substantially longer than the corresponding times in visual conditions. For a more complex version of the secondary task (longer list lengths) visual presentation of the task all at once (simultaneously) led to better performance then sequential presentation (whether visual or auditory). When given a choice people also tended to prefer simultaneous visual presentation of the secondary task. We discuss the effect of presentation modality of the secondary task in terms of its implications for user interface design in vehicles.

Audio and visual modalities are two common output channels in the user interfaces embedded in today’s mobile devices. However, these user interfaces typically center on the visual modality as the primary output channel, with audio output serving a secondary role. This paper argues for an increased need for shared input multimodal user interfaces for mobile devices. A shared input multimodal interface can be operated independently using a specific output modality, leaving users to choose the preferred method of interaction in different scenarios. We evaluate the value of a shared input multimodal menu system in both a single-task desktop setting and in a dynamic dual-task setting, in which the user was required to interact with the shared input multimodal menu system while driving a simulated vehicle. Results indicate that users were faster at locating a target item in the menu when visual feedback was provided in the single-task desktop setting, but in the dual-task driving setting, visual output presented a significant source of visual distraction that interfered with driving performance. In contrast, auditory output mitigated some of the risk associated with menu selection while driving. A shared input multimodal interface allows users to take advantage of multiple feedback modalities properly, providing a better overall experience.

Customer satisfaction is an important factor governing adoption and retention of multimedia products and services, such as Over-The-Top(OTT) video transmission. Quality of Experience involves user-centric evaluation of various services. However, users differ in terms of their ratings of service quality. Some rating differences are due to unreliability (outlier users who are not motivated, or are not sensitive to differences in quality), but others are systematic differences in rating that may reflect different perspectives on quality. In this paper, we explore the use of outlier analysis and clustering as tools for interpreting QoE data. We report on experimental results demonstrating the use of outlier analysis and clustering. In interpreting the clusters, we examine users' opinions on different types of video disruption, and their ability to distinguish the different levels of impairments/failures.

In this paper, we describe an Over-The-Top (OTT) video Quality of Experience (QoE) subjective evaluation experiment that was carried out to examine variations in the way subjects assess viewing experiences. The experiment focuses on different level of impairment and failure types, using 5-point measurement scales. Clustering is used to differentiate between unreliable and reliable participants, where reliability is defined in terms of criteria such as consistency of rating and ability to distinguish between qualitative differences in level of impairments. The results show that clustering a data set that is augmented with unreliable pseudo-participants can provide a new and improved perspective on individual differences in video QoE assessment.

The study reported in this paper demonstrated, for the first time, that the peak-end effect, commonly found with respect to memories of experience, also applies to overall Quality of Experience (QoE) measures obtained after participants view a sequence of videos. Sequences of videos shown in an experiment varied according to the sequencing and grouping of videos with better or worse Technical Quality (TQ). An end effect was found for both better TQ and worse TQ videos. However the peak effect was found for bad, but not good, videos. These results provide an important first step towards the development of models of Likelihood to Recommend (L2R) based on accumulated experience with a service.

In this paper, we study the relationships between Quality of Service (QoS) and Quality of Experience (QoE) in a session-based Over-The-Top (OTT) video service. A number of Performance Metrics (PMs) with and without the existence of failures during a video are examined. As QoE factors, Technical Quality (TQ) and Acceptability are used. We analyze the correlation between QoS performance metrics and QoE factors, and find new PMs should be employed because failures are included in QoE evaluation. We also summarize the relationships between QoS metrics and QoE factors through machine learning approaches. Using decision tree, we have a general idea about the relationships between PMs and QoE factors. We also understand the impact caused by failures and the value of rating scales.

Quality of Experience (QoE) is a crucial guiding factor for network management of an end-to-end service session. The network provider can control the resources allocated to sessions and in doing so, influence the Technical Quality (TQ), which covers the technical aspects of signal quality during the session. On the other hand, the network provider has no control over the Content Quality (CQ), which pertains to the user's level of interest in a particular video. Together TQ and CQ influence the Overall eXperience (OX) in a session. In this paper, we present results from a user subjective study in which the impact of TQ and CQ on OX was investigated for Over-The-Top (OTT) video sessions from the perspective of a network provider. This perspective places a focus on those elements of QoE that can be controlled by the provider. Various studies have shown that very high interest in a content can strongly influence QoE independent of other factors, so our study uses videos that are neutral with respect to content. We assess the TQ, CQ and OX for video sessions that contain Integrity impairments (in the form of image freezing) and failures in terms of session Accessibility and Retainability. Our findings indicate that TQ and CQ have a strong impact on OX in the presence of impairments, but no failures. On the other hand, TQ is the main determinant of OX when failures are present.

Requirements analysis defines the goals and evaluation criteria of system design. We introduce a methodology for requirements analysis for customization based on large sample interactive testing, with the premise that analysis of user behaviour with prototypes leads to requirements discoveries. The methodology uses a relatively large sample (1) to identify relevant user subgroups, (2) to observe significant empirically determined group differences in the context of task and tool use and (3) to estimate the groups’ different requirements and derive design implications. Between 20 and 50 participants are used per test, rather than the three to five often recommended for user testing. Statistical relationships are investigated between subgroups in terms of background variables, questionnaire items, performance data, and coded verbal statements. Customization requirements are inferred from the significant differences observed between empirically determined groups. The methodological framework is illustrated in a case study involving the use of clinical resources on handheld devices by three groups of physicians. The groups were found to have different needs and preferences for evidence-based resources and device form factor, implying opportunities and necessities for group customization requirements. Relevance to industry In safety-critical domains such as health care, it is essential to assess user needs and preferences regarding devices and systems to inform appropriate customizations. We present a methodological framework and case study that demonstrates how large sample user testing can supplement typical methods of requirements analysis to provide contextualized, quantitative accounts of group differences and customization requirements.

Web 2.0 tools in general, and Web video in particular, provide new ways for activists to express their viewpoints to a broad audience. In this paper we deployed tools that have been used to find subgroups automatically in social networks and applied them to the problem of distinguishing between two sides of a controversial issue based on patterns of online interaction. We explored the problem of distinguishing between anti- and pro-vaccination activists based on a social network of videos and associated comments posted on YouTube. Videos for the analysis were selected by submitting the term “vaccination” to a search on YouTube. A content analysis of the selected videos was then performed (Keelan et al, 2007) to classify videos as pro- or anti-vaccination. Then, a modified version of the SCAN method (Chin and Chignell, 2008) for identifying cohesive subgroups in social networks was applied to the social network inferred from the discussions about the videos. Results showed that a cohesive subgroup of anti-vaccination people existed in discussions around anti-vaccination videos, whereas discussions around pro-vaccination videos included both anti-vaccination and pro-vaccination people. Implications of the method and results for more general delineation of types of medical activism and the opposing camps within those camps are discussed.

As a first step in the development of community trackers for large-scale online interaction, this paper shows how cohesive subgroup analysis using the Social Cohesion Analysis of Networks (SCAN; Chin and Chignell 2008) and Data-Intensive Socially Similar Evolving Community Tracker (DISSECT; Chin and Chignell 2010) methods can be applied to the problem of identifying cohesive subgroups and tracking them over time. Three case studies are reported, and the findings are used to evaluate how well the SCAN and DISSECT methods work for different types of data. In the largest of the case studies, variations in temporal cohesiveness are identified across a set of subgroups extracted from the inferred social network. Further modifications to the DISSECT methodology are suggested based on the results obtained. The paper concludes with recommendations concerning further research that would be beneficial in addressing the community tracking problem for online data.

The problem of identifying cohesive subgroups in social hypertext is reviewed. A computationally efficient three-step framework for identifying cohesive subgroups is proposed, referred to as the Social Cohesion Analysis of Networks (SCAN) method. In the first step of this method (Select), people within a social network are screened using a level of network centrality to select possible subgroup members. In the second step (Collect), the people selected in the first step are collected into subgroups identified at each point in time using hierarchical cluster analysis. In the third step (Choose), similarity modeling is used to choose cohesive subgroups based on the similarity of subgroups when compared across different points in time. The application of this SCAN method is then demonstrated in a case study where a subgroup is automatically extracted from a social network formed based on the online interactions of a group of about 150 people that occurred over a two-year period. In addition, this paper also demonstrates that similarity-based cohesion can provide a different, and in this case more compelling, subgroup representation than a method based on splitting a hierarchical clustering dendrogram using an optimality criterion.

Consumer acceptance is of great interest in the adoption of novel multimedia products and services. A number of factors can greatly influence the customer experience during a video session, impacting the acceptability of the product or service. Factors such as the Technical Quality (TQ) which covers the technical aspects of the signal quality can be controlled from the network provider. On the other hand, the network provider has no control on the subject's level of interest in a video, the Content Quality (CQ). Together TQ and CQ influence the Overall eXperience (OX) of the subject and, in the case of Over The Top (OTT) video, in a video session. In this paper, we present results from a user subjective study in which the impact of TQ and OX on the acceptability was investigated for OTT video sessions. To minimize the CQ impact on OX, the videos were carefully selected to have relatively neutral content. We assess the impact of TQ and OX on acceptability for videos containing impairment and failure events during lifecycle, thus affecting Accessibility, Retainability and Integrity. Our experimental results indicate that TQ and OX have a strong impact on acceptability.

This study investigated the effectiveness of spatial auditory cues for tasks using different reference frames, and examined how congruency between auditory and visual displays affected performance. Performance with three types of auditory cues (egocentric, exocentric and non-spatial) was compared on three spatial tasks: target search, target localization and target recall. There was a clear effect of reference frame congruency between auditory and visual displays on target search and target localization tasks. Interestingly, even incongruent auditory cues improved performance relative to non-spatial control conditions. In addition, egocentric auditory cues facilitated performance more than exocentric cues on incongruent trials. The findings have important implications for display design in work environments that involve diverse spatial tasks and displays that use different reference frames.

This paper addresses issues raised by the ever-expanding role of email as a multi-faceted application that combines communication, collaboration, and task management. Individual differences analysis was used to contrast two email user interfaces in terms of their demands on users. The results of this analysis were then interpreted in terms of their implications for designing more inclusive interfaces that meet the needs of users with widely ranging abilities. The specific target of this research is the development of a new type of email message representation that makes pending tasks more visible. We describe a study that compared a new way of representing tasks in an email inbox, with a more standard representation (the Microsoft Outlook inbox). The study consisted of an experiment that examined how people with different levels of three specific cognitive capabilities (flexibility of closure, visual memory, and working memory) perform when using these representations. We then identified combinations of representation and task that are disadvantageous for people with low levels of the measured capabilities.

With advances in computer graphics, a number of innovative approaches to information visualization have been developed (e.g., Card et al, 1991). Some of these approaches create a mapping between information and corresponding structure in a virtual world. The resulting virtual worlds can be fully three dimensional (3D) or they can be implemented as a series of 2D birds-eye "snapshots" that are traversed as if they were in 3D, using operations such as panning and zooming interactively (2.5D). This paper reports a study that contrasted 3D and 2.5D performance for people with differing levels of spatial and structure learning ability. Four data collection methods were employed: search task scoring; subjective questionnaires; navigational activity logging and analysis; and administration of tests for spatial and structure-learning abilities. Analysis of the results revealed statistically significant effects of user abilities, and information environment designs. Overall, this research did not find a performance advantage for using a 3D rather than a 2.5D virtual world. In addition, users in the lowest quartile of spatial ability had significantly lower search performance in the 3D environment. The findings suggest that individual differences in traits such as spatial ability may be important in determining the usability and acceptability of 3D environments.

A study is reported where the rankings assigned by a search engine (Google) to the top 100 hits for seven queries are compared with the ratings of human judges (six per query). As assessed by correlations, agreement between judges was relatively good across all the queries, with a mean (Pearson) correlation of .475. In contrast, the correlations between Google rankings and human judges in this study were significantly lower, with a mean of .153. This study suggests that a high priority should be placed on finding improved methods for ranking the initial set of search engine results, so that they are more in line with human judgments of relevance.

In spite of the powerful functionality and message integration offered, unified communication (UC) has yet to be widely adopted as a solution that meets the demands and complexities of modern communicators. By understanding individual differences, requirements, and preferences, it should be possible to design better communication interfaces and services. This study examined the characteristics of communication-related preferences among users of a three-month UC trial. Survey data from 123 respondents were analyzed. A series of factor analyses identified 19 communication traits, 7 of which were particularly interesting. A cluster analysis suggested three types of communication styles: Low-tech, Power, and Strategic. Future research will attempt to relate the current findings to actual communication behaviour. The creation of a communication preference inventory (CPI) is also discussed.

Measurement of video Quality of Experience is needed to enable telecommunication service operators to provide acceptable video viewing experience to their customers. Without accurate Quality of Experience measurement, it is hard to predict customer satisfaction. Accessibility and Retain ability failures are important factors that can affect the experience of viewing video on the Internet. Most of the published work in the assessment of Video Quality of Experience has focused on the quality of video viewing sessions without considering the impact of Accessibility and Retain ability failures. In this paper, we focus on the Quality of Experience for video sessions in the presence of Retain ability failures. We primarily evaluate video Quality of Experience for videos with temporary interruptions, called Integrity impairments, as well as with permanent interruptions called Retain ability failures. A subjective study was conducted to observe the effect of Retain ability failure on video Quality of Experience. The results of the subjective study suggest that videos with Retain ability failures are rated much lower than videos without any failure. An initial linear model for QoE of videos subject to Retain ability failures is introduced.

Accurate Quality of Experience measurement for streaming video has become more crucial with the increase in demand for online video viewing. Quantifying video Quality of Experience is a challenging task. Significant efforts to quantify video Quality of Experience have primarily focused on the measurement of Quality of Experience for videos with network and compression related impairments. These impairments, however, may not always be the only main factors affecting Quality of Experience in an entire video viewing session. In this paper, we evaluate Quality of Experience for entire video viewing sessions, from the beginning to the end. In doing so, we evaluate videos with temporary interruptions as well as those with permanent interruptions or failures. We consider two types of failures, namely Accessibility and Retainability failures, and present the results of two subjective studies. These results indicate: (a) Accessibility and Retainability failures are rated lower compared to temporary interruption impairments; (b) Accessibility failures are rated close to the lowest value on the rating scale; and (c) the traditionally used 5-point scale to measure video Quality of Experience is not sufficient in the presence of Accessibility and Retainability failures.

Gamification and serious games are becoming increasingly important for training, wellness, and other applications. How can games be developed for non-traditional gaming populations such as the elderly, and how can gaming be applied in non-traditional areas such as cognitive assessment? The application that we were interested in is detection of cognitive impairment in the elderly. Example use cases where gamified cognitive assessment might be useful are: prediction of delirium onset risk in emergency departments and postoperative hospital wards; evaluation of recovery from stroke in neuro-rehabilitation; monitoring of transitions from mild cognitive impairment to dementia in long-term care. With the rapid increase in cognitive disorders in many countries, inexpensive methods of measuring cognitive status on an ongoing basis, and to large numbers of people, are needed. In order to address this challenge we have developed a novel game-based method of cognitive assessment. In this paper, we present findings from a usability study conducted on the game that we developed for measuring changes in cognitive status. We report on the game's ability to predict cognitive status under varying game parameters, and we introduce a method to calibrate the game that takes into account differences in speed and accuracy, and in motor coordination. Recommendations concerning the development of serious games for cognitive assessment are made, and detailed recommendations concerning future development of the whack-a-mole game are also provided.

The results from a series of three experiments that used Text Retrieval Conference (TREC) data and TREC search topics are compared. These experiments each involved three novel user interfaces (one per experiment). User interfaces that made it easier for users to view text were found to improve recall in all three experiments. A distinction was found between a cluster of subjects (a majority of whom were search experts) who tended to read fewer documents more carefully (readers, or exclusives) and subjects who skimmed through more documents without reading them as carefully (skimmers, or inclusives). Skimmers were found to have significantly better recall overall. A major outcome from our experiments at TREC and with the TREC data, is that hypertext interfaces to information retrieval (IR) tasks tend to increase recall. Our interpretation of this pattern of results across the three experiments is that increased interaction with the text (more pages viewed) generally improves recall. Findings from one of the experiments indicated that viewing a greater diversity of text on a single screen (i.e., not just more text per se, but more articles available at once) may also improve recall. In an experiment where a traditional (type-in) query interface was contrasted with a condition where queries were marked up on the text, the improvement in recall due to viewing more text was more pronounced with search novices. Our results demonstrate that markup and hypertext interfaces to text retrieval systems can benefit recall and can also benefit novices. The challenge now will be to find modified versions of hypertext interfaces that can improve precision, as well as recall and that can work with users who prefer to use different types of search strategy or have different types of training and experience.

There was a proliferation of electronic information sources and search engines in the 1990s. Many of these information sources became available through the ubiquitous interface of the Web browser. Diverse information sources became accessible to information professionals and casual end users alike. Much of the information was also hyperlinked, so that information could be explored by browsing as well as searching. While vast amounts of information were now just a few keystrokes and mouseclicks away, as the choices multiplied, so did the complexity of choosing where and how to look for the electronic information. Much of the complexity in information exploration at the turn of the twenty-first century arose because there was no common cataloguing and control system across the various electronic information sources. In addition, the many search engines available diAered widely in terms of their domain coverage, query methods and eAciency. Meta-search engines were developed to improve search performance by querying multiple search engines at once. In principle, meta-search engines could greatly simplify the search for electronic information by selecting a subset of first-level search engines and digital libraries to submit a query to based on the characteristics of the user, the query/topic, and the search strategy. This selection would be guided by diagnostic knowledge about which of the first-level search engines works best under what circumstances. Programmatic research is required to develop this diagnostic knowledge about first-level search engine performance. This paper introduces an evaluative framework for this type of research and illustrates its use in two experiments. The experimental results obtained are used to characterize some properties of leading search engines (as of 1998). Significant interactions were observed between search engine and two other factors (time of day and Web domain). These findings supplement those of earlier studies, providing preliminary information about the complex relationship between search engine functionality and performance in diAerent contexts. While the specific results obtained represent a time-dependent Information Processing and Management 35 (1999) 337‐362

This paper reports on an experimental comparison of static and dynamic hypertext. The static hypertext consisted of an online Bible with links defined by Thompson's Bible Chain References. The dynamic hypertext used the same content, but links were created dynamically by a search algorithm. In terms of task time and accuracy, overall performance was better in the dynamic hypertext condition. In addition, novices seemed to benefit more from the dynamic hypertext than did Bible experts. These results demonstrate that dynamic hypertext, using a querying approach, may sometimes be superior to static hypertext for some tasks. Thus dynamic hypertext, which requires no authoring effort, is an attractive alternative to the labor-intensive static hypertext that has been prevalent in the past.