Search

This article devises a photograph-based monitoring model to estimate the real-time PM2.5 concentrations, overcoming currently popular electrochemical sensor-based PM2.5 monitoring methods’ shortcomings such as low-density spatial distribution and time delay. Combining the proposed monitoring model, the photographs taken by various camera devices (e.g., surveillance camera, automobile data recorder, and mobile phone) can widely monitor PM2.5 concentration in megacities. This is beneficial to offering helpful decision-making information for atmospheric forecast and control, thus reducing the epidemic of COVID-19. To specify, the proposed model fuses Information Abundance measurement and Wide and Deep learning, dubbed as IAWD, for PM2.5 monitoring. First, our model extracts two categories of features in a newly proposed DS transform space to measure the information abundance (IA) of a given photograph since the growth of PM2.5 concentration decreases its IA. Second, to simultaneously possess the advantages of memorization and generalization, a new wide and deep neural network is devised to learn a nonlinear mapping between the above-mentioned extracted features and the groundtruth PM2.5 concentration. Experiments on two recently established datasets totally including more than 100 000 photographs demonstrate the effectiveness of our extracted features and the superiority of our proposed IAWD model as compared to state-of-the-art relevant computing techniques.

In this work, we resolve a big challenge that most current image quality metrics (IQMs) are unavailable across different image contents, especially simultaneously coping with natural scene (NS) images or screen content (SC) images. By comparison with existing works, this paper deploys on-line and off-line data for proposing a unified no-reference (NR) IQM, not only applied to different distortion types and intensities but also to various image contents including classical NS images and prevailing SC images. Our proposed NR IQM is developed with two data-driven learning processes following feature extraction, which is based on scene statistic models, free-energy brain principle, and human visual system (HVS) characteristics. In the first process, the scene statistic models and an image retrieve technique are combined, based on on-line and off-line training instances, to derive a novel loose classifier for retrieving clean images and helping to infer the image content. In the second process, the features extracted by incorporating the inferred image content, free-energy and low-level perceptual characteristics of the HVS are learned by utilizing off-line training samples to analyze the distortion types and intensities and thereby to predict the image quality. The two processes mentioned above depend on a gigantic quantity of training data, much exceeding the number of images applied to performance validation, and thus make our model's performance more reliable. Through extensive experiments, it has been validated that the proposed blind IQM is capable of simultaneously inferring the quality of NS and SC images, and it has attained superior performance as compared with popular and state-of-the-art IQMs on the subjective NS and SC image quality databases. The source code of our model will be released with the publication of the paper at https://kegu.netlify.com .

In this work we have enhanced the perception of a humanoid robot by integrating it with a social state estimation system. We present a user study of the humanoid Nao robot as a social mediator, comprising two sets of experiments. In the first sets of experiments, the participants rate their understanding of feedback messages delivered via the Nao robot. They also assess two modalities to deliver the feedback: audio only and audio combined with gestures. In almost all cases there is an improvement of 10% or more when audio and gesture modalities are combined to deliver feedback messages. For the second sets of experiments the sociofeedback system was integrated with the Nao robot. The participants engage in two-person scenario-based conversations while the Nao robot acts as a mediator. The sociofeedback system analyzes the conversations and provides feedback via Nao. Subsequently, the participants assess the received sociofeedback with respect to various aspects, including its content, appropriateness, and timing. Participants also evaluate their overall perception of Nao as social mediator via the Godspeed questionnaire. The results indicate that the social feedback system is able to detect the social scenario with 93.8% accuracy and that Nao can be effectively used to provide sociofeedback in discussions. The results of this paper pave the way to natural human-robot interactions for social mediators in multi-party dialog systems. NRF (Natl Research Foundation, S’pore) Accepted version

New challenges have been brought out along with the emerging of 3D-related technologies, such as virtual reality, augmented reality (AR), and mixed reality. Free viewpoint video (FVV), due to its applications in remote surveillance, remote education, and so on, based on the flexible selection of direction and viewpoint, has been perceived as the development direction of next-generation video technologies and has drawn a wide range of researchers' attention. Since FVV images are synthesized via a depth image-based rendering (DIBR) procedure in the "blind" environment (without reference images), a reliable real-time blind quality evaluation and monitoring system is urgently required. But existing assessment metrics do not render human judgments faithfully mainly because geometric distortions are generated by DIBR. To this end, this paper proposes a novel referenceless quality metric of DIBR-synthesized images using the autoregression (AR)-based local image description. It was found that, after the AR prediction, the reconstructed error between a DIBR-synthesized image and its AR-predicted image can accurately capture the geometry distortion. The visual saliency is then leveraged to modify the proposed blind quality metric to a sizable margin. Experiments validate the superiority of our no-reference quality method as compared with prevailing full-, reduced-, and no-reference models. MOE (Min. of Education, S’pore)

Human hand is the best sample for humanoid robotic hand and a nature grasping is the final target that most robotic hands are pursuing. Many prior researches had been done in virtual and real for simulation the human grasping. Unfortunately, there is no perfect solution to duplicate the nature grasping of human. The main difficulty comes from three points. 1. How to 3D modelling and fabricate the real hand. 2. How actuated the robotic hand as real hand. 3. How to grasp objects in different shapes like human hand. To deal with these three problems and further to provide a partial solution for duplicate human grasping, this paper introduces our method to solve these problems from robotic hand design, fabrication, actuation and grasping plan. Our modelling progress takes only around 12 minutes that include 10 minutes of 3D scanning of a real human hand and two minutes for changing the scanned model to an articulated model by running our algorithm. Our grasping plan is based on the sampled trajectory and easy to implement for grasping different objects. Followed these steps, a seven DOF robotic hand is created and tested in the experiments. Published version

It is often desirable for humanoid robots to have dexterous hands and perform human-like grasps. This requires a deliberate design of hands, in addition to a good actuation system and an efficient 3D vision system. In this paper, we present a simple method to produce a customized articulated robotic hand for a humanoid robot (eg., Nadine social robot). Our method acquires the 3D shape by 3D scanning, which can cost-effectively generate customized hands. Guided by the human hand anatomy and inspired by the progress of 3D printing technology, we model the structure of the robotic hand using standard shapes such as cuboids, spheres and cylinders, which are adapted to match the scanned shape and fabricated using 3D printing. Moreover, we choose flexible resin materials11https://formlabs.com/3d-printers/form-2/, which are supported by 3D printers, to make the fabricated hand flexible enough for natural deformation. As a result, our designed robotic hand has six degrees of freedom and together with a cable-driven actuation system can achieve the range of motion as a human hand. Experimental results demonstrated that our robotic hand is capable of grasping a variety of objects with different shapes. NRF (Natl Research Foundation, S’pore) Accepted version

This paper presents a system that aims to achieve autonomous grasping for micro-controller based humanoid robots such as the Inmoov robot [1]. The system consists of a visual sensor, a central controller and a manipulator. We modify the open sourced objection detection software YOLO (You Only Look Once) v2 [2] and associate it with the visual sensor to make the sensor be able to detect not only the category of the target object but also the location with the help of a depth camera. We also estimate the dimensions (i.e., the height and width) of the target based on the bounding box technique (Fig. 1). After that, we send the information to the central controller (a humanoid robot), which controls the manipulator (customised robotic hand) to grasp the object with the help of inverse kinematics theory. We conduct experiments to test our method with the Inmoov robot. The experiments show that our method is capable of detecting the object and driving the robotic hands to grasp the target object. Open image in new window Fig. 1. Autonomous grasping with real-time object detection

Reference EPFL-ARTICLE-221843doi:10.1007/s00371-016-1271-7View record in Web of Science Record created on 2016-10-18, modified on 2016-10-18

This paper presents a semi-automatic method of parameterizing an existing social context cognition model. It discusses benefits of the social context cognition models for example in personality modeling and their key issue that is parametrization. It briefly introduces social context cognition model and describes a new method of its crowd-sourcing-based parametrization. Later, validation is provided, and ability to recreate social context cognition in the provided samples is presented with good generalization for the unknown cases. Finally, model's stability for the continuous stream of dynamic social context input data is shown. Presented system contributes to the believable agent modeling and social simulations by making much needed applications of social context cognition models easier by addressing the so far unsolved troublesome parametrization issues. Copyright © 2016 John Wiley & Sons, Ltd.

In this paper, we present a novel method for real-time 3D hand pose estimation from single depth images using 3D Convolutional Neural Networks (CNNs). Image-based features extracted by 2D CNNs are not directly suitable for 3D hand pose estimation due to the lack of 3D spatial information. Our proposed 3D CNN-based method, taking a 3D volumetric representation of the hand depth image as input and extracting 3D features from the volumetric input, can capture the 3D spatial structure of the hand and accurately regress full 3D hand pose in a single pass. In order to make the 3D CNN robust to variations in hand sizes and global orientations, we perform 3D data augmentation on the training data. To further improve the estimation accuracy, we propose applying the 3D deep network architectures and leveraging the complete hand surface as intermediate supervision for learning 3D hand pose from depth images. Extensive experiments on three challenging datasets demonstrate that our proposed approach outperforms baselines and state-of-the-art methods. A cross-dataset experiment also shows that our method has good generalization ability. Furthermore, our method is fast as our implementation runs at over 91 frames per second on a standard computer with a single GPU. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore) Accepted version

Today, to create and to simulate a virtual anatomical version of a subject is useful in the decision process of surgical treatments. The muscular activity is one of the factors which can contribute to abnormal movements such as in spasticity or static contracture. In this paper, we propose a numerical solution, based on the Finite Element (FE) method, able to estimate muscles deformations during contraction. Organized around a finite element solver and a volumetric environment, this solution is made of all the modeling and simulation processes from the discretization of the studied domain to the visualization of the results. The choices of materials and properties of the FE model are also presented such as the hyperelasticity, the contention model based on inter-meshes neighboring nodes pairing, and the estimation of nodal forces based on the subject-specific muscular forces and action lines.

Nonverbal communication is an important aspect of real-life face-to-face interaction and one of the most efficient ways to convey emotions, therefore users should be provided the means to replicate it in the virtual world. Because articulated embodiments are well suited to provide body communication in virtual environments, this paper first reviews some of the advantages and disadvantages of complex embodiments. After a brief introduction to nonverbal communication theories, we present our solution, taking into account the practical limitations of input devices and social science aspects. We introduce our sample of actions and implementation using our VLNET (Virtual Life Network) networked virtual environment and discuss the results of an informal evaluation experiment.

Object-driven interactions have various applications, especially in team activities and sports like volleyball, football, tennis, etc. We define a multi-party virtual world as an environment which includes multiple avatars and computer agents. Latency is one of the primary issues while building a multi-party setup. Delays due to latency cause inconsistencies and unnecessary halts resulting in unrealistic experience in VR. For interactions using objects, it becomes vital to handle the latencies. We present a time scaling algorithm for latency management and synchronization between multiple avatars, while they interact using objects in the VR environment. We propose two time scaling schemes—source and target based. Both these methodologies are dependent on which user’s virtual view (source or target) needs to be uniform. Incorporating the presented time scaling schemes, we develop a comprehensive multi-party virtual reality platform to simulate applications which involves interactions between agents and avatars using objects. We adopt a client–server architecture for building our multiuser VR platform which induces flexibility of adding multiple avatars and/or agents to the application. The modular nature of our proposed system enables extension to different VR applications that encompass communication using objects. We exemplify our framework by developing a multi-player VR volleyball game which employs the features of our proposed scheme. We demonstrate the significance of time scaling in the object-driven interactive multi-party VR framework through a comparative user study. We also evaluate the immersion experience in our multi-player volleyball game by a user survey. National Research Foundation (NRF) This research is supported by the program titled Realistic Immersion with Virtual Humans, a collaboration between Nanyang Technological University (NTU) Singapore and Institute of Software, Chinese Academy of Sciences (ISCAS) in Beijing, China. The program is supported by the National Research Foundation, Prime Minister’s Office, Singapore, under its NRF-NSFC Joint Research Grant Call (Data Science).

Articulated hand pose estimation is one of core technologies in human-computer interaction. Despite the recent progress, most existing methods still cannot achieve satisfactory performance, partly due to the difficulty of the embedded high-dimensional nonlinear regression problem. Most existing data-driven methods directly regress 3D hand pose from 2D depth image, which cannot fully utilize the depth information. In this paper, we propose a novel multi-view convolutional neural network (CNN)-based approach for 3D hand pose estimation. To better exploit 3D information in the depth image, we project the point cloud generated from the query depth image onto multiple views of two projection settings and integrate them for more robust estimation. Multi-view CNNs are trained to learn the mapping from projected images to heat-maps, which reflect probability distributions of joints on each view. These multi-view heat-maps are then fused to estimate the optimal 3D hand pose with learned pose priors, and the unreliable information in multi-view heat-maps is suppressed using a view selection method. Experimental results show that the proposed method is superior to the state-of-the-art methods on two challenging data sets. Furthermore, a cross-data set experiment also validates that our proposed approach has good generalization ability. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore) Accepted version

When building robotic hands, researchers are always face with two main issues of how to make robotic hands look human-like and how to make robotic hands function like real hands. Most existing solutions solve these issues by manually modelling the robotic hand [10-18]. However, the design processes are long, and it is difficult to duplicate the geometry shape of a human hand. To solve these two issues, this paper presents a simple and effective method that combines 3D printing and digitization techniques to create a 3D printable cable-driven robotic hand from scanning a physical hand. The method involves segmenting the 3D scanned hand model, adding joints, and converting it into a 3D printable model. Comparing to other robotic solutions, our solution retains more than 90% geometry information of a human hand1, which is attained from 3D scanning. Our modelling progress takes around 15 minutes that include 10 minutes of 3D scanning and five minutes for changing the scanned model to an articulated model by running our algorithm. Compared to other articulated modelling solutions [19, 20], our solution is compatible with an actuation system which provides our robotic hand with the ability to mimic different gestures. We have also developed a way of representing hand skeletons based on the hand anthropometric. As a proof of concept, we demonstrate our robotic hand's performance in the grasping experiments. NRF (Natl Research Foundation, S’pore) Accepted version

The positions of the hand joints are important high-level features for hand-based human-computer interaction. We present a novel method to predict the 3-D joint positions from the depth images and the parsed hand parts obtained with a pretrained classifier. The hand parts are utilized as the additional cue to resolve the multimodal predictions produced by the previous regression-based method without increasing the computational cost significantly. In addition, we further enforce the hand motion constraints to fuse the per-pixel prediction results. The posterior distribution of the joints is formulated as a weighted product of experts model based on the individual pixel predictions, which is maximized via the expectation–maximization algorithm on a learned low-dimensional space of the hand joint parameters. The experimental results show the proposed method improves the prediction accuracy considerably compared with the rivals that also regress for the joint locations from the depth images. Especially, we show that the regressor learned on synthesized dataset also gives accurate prediction on real-world depth images by enforcing the hand part correlations despite their discrepancies.

Expanding the spectrum of agent social capabilities is an important challenge in agent-based simulation and other domains. While human-like emotionality has been vastly explored in the last 20years, little research addresses explicit, psychologically believable social situation modeling. Recently, some important elements have been underlined: hybrid connectionist models outside formal ontologies; complex subjective representations linking culture, personality and norms, and so on, but proposed solutions do not provide a formalized structure of a social experience, expressive and well-grounded in psychology. In this paper, we develop a new approach to social situation modeling based on the dramaturgical and dissonance theories. A new component Dramaturgical Module is described with implementation used to generate example behavior depicting new social modeling capabilities and a believable representation of the relevant psychological theories. We present a case scenario with a dramaturgical interpretation of dynamic social situations and related cognitive dissonances resulting in a simple and flexible classification. Easily usable in reasoning, planning or affect generation, dramaturgical interpretation is additionally presented here as basis of social affect generation. Copyright © 2015John Wiley & Sons, Ltd.

Vision-based hand pose estimation is important in human-computer interaction. While many recent works focus on full degree-of-freedom hand pose estimation, robust estimation of global hand pose remains a challenging problem. This paper presents a novel algorithm to optimize the leaf weights in a Hough forest to assist global hand pose estimation with a single depth camera. Different from traditional Hough forest, we propose to learn the vote weights stored at the leaf nodes of a forest in a principled way to minimize average pose prediction error, so that ambiguous votes are largely suppressed during prediction fusion. Experiments show that the proposed method largely improves pose estimation accuracy with optimized leaf weights on both synthesis and real datasets and performs favorably compared to state-of-the-art convolutional neural network-based methods. On real-world depth videos, the proposed method demonstrates improved robustness compared to several other recent hand tracking systems from both industry and academy. Moreover, we utilize the proposed method to build virtual/augmented reality applications to allow users to manipulate and examine virtual objects with bare hands. MOE (Min. of Education, S’pore)

This paper discusses the role of sensors and actuators in Human–Computer Interaction and Virtual Reality. We first introduce the need of sensory input and output and the user senses stimulation. We then present two experiences of immersive games with a comparison of the techniques 20 years ago and today and also discuss the importance of vision-based hand tracking. The main part of the paper is dedicated to haptic and tactile feedbacks with 4 different case studies: haptic feedback using a haptic workstation, a wearable system for mobility improvement of visually impaired people, a system for enhancing pilot performance, and actuators to feel the wind. Finally, we survey a few smart jackets with sensors and actuators. NRF (Natl Research Foundation, S’pore)

With the quick development and popularity of computers, computer-generated signals have drastically invaded into our daily lives. Screen content image is a typical example, since it also includes graphic and textual images as components as compared with natural scene images which have been deeply explored, and thus screen content image has posed novel challenges to current researches, such as compression, transmission, display, quality assessment, and more. In this paper, we focus our attention on evaluating the quality of screen content images based on the analysis of structural variation, which is caused by compression, transmission, and more. We classify structures into global and local structures, which correspond to basic and detailed perceptions of humans, respectively. The characteristics of graphic and textual images, e.g., limited color variations, and the human visual system are taken into consideration. Based on these concerns, we systematically combine the measurements of variations in the above-stated two types of structures to yield the final quality estimation of screen content images. Thorough experiments are conducted on three screen content image quality databases, in which the images are corrupted during capturing, compression, transmission, etc. Results demonstrate the superiority of our proposed quality model as compared with state-of-the-art relevant methods. MOE (Min. of Education, S’pore)

Random forest based Hough-voting techniques have been widely used in a variety of computer vision problems. As an ensemble learning method, the voting weights of leaf nodes in random forest play critical role to generate reliable estimation result. We propose to improve Hough-voting with random forest via simultaneously optimizing the weights of leaf votes and pruning unreliable leaf nodes in the forest. After constructing the random forest, the weight assignment problem at each tree is formulated as a L0-regularized optimization problem, where unreliable leaf nodes with zero voting weights are suppressed and trees are pruned to ignore sub-trees that contain only suppressed leaves. We apply our proposed techniques to several regression and classification problems such as hand gesture recognition, head pose estimation and articulated pose estimation. The experimental results demonstrate that by suppressing unreliable leaf nodes, it not only improves prediction accuracy, but also reduces both prediction time cost and model complexity of the random forest.

User ratings are the essence of recommender systems in e-commerce. Lack of motivation to provide ratings and eligibility to rate generally only after purchase restrain the effectiveness of such systems and contribute to the well-known data sparsity and cold start problems. This article proposes a new information source for recommender systems, called prior ratings. Prior ratings are based on users’ experiences of virtual products in a mediated environment, and they can be submitted prior to purchase. A conceptual model of prior ratings is proposed, integrating the environmental factor presence whose effects on product evaluation have not been studied previously. A user study conducted in website and virtual store modalities demonstrates the validity of the conceptual model, in that users are more willing and confident to provide prior ratings in virtual environments. A method is proposed to show how to leverage prior ratings in collaborative filtering. Experimental results indicate the effectiveness of prior ratings in improving predictive performance.

Hand pose tracking and gesture recognition are useful for human-computer interaction, while a major problem is the lack of discriminative features for compact hand representation. We present a robust hand parsing scheme to extract a high-level description of the hand from the depth image. A novel distance-adaptive selection method is proposed to get more discriminative depth-context features. Besides, we propose a Superpixel-Markov Random Field (SMRF) parsing scheme to enforce the spatial smoothness and the label co-occurrence prior to remove the misclassified regions. Compared to pixel-level filtering, the SMRF scheme is more suitable to model the misclassified regions. By fusing the temporal constraints, its performance can be further improved. Overall, the proposed hand parsing scheme is accurate and efficient. The tests on synthesized dataset show it gives much higher accuracy for single-frame parsing and enhanced robustness for continuous sequence parsing compared to benchmarks. The tests on real-world depth images of the hand and human body show the robustness to complex hand configurations of our method and its generalization power to different kinds of articulated objects.

Simulating a crowded scene like a busy shopping street requires tight packing of virtual characters. In such cases, collisions are likely to occur, and the choice in collision detection shape will influence how characters are allowed to intermingle. Full collision detection is too expensive for crowds, so simplifications are needed. The most common simplification, the fixed-width, pose-independent cylinder, does not allow intermingling of characters, as it will either cause too much empty space between characters or undetected penetrations. As a possible solution to this problem, we introduce the bounding cylinder hierarchy BCH, a bounding volume hierarchy that uses vertical cylinders as bounding shapes. Because the BCH is a generalization of the single cylinder, we expect that this representation can be easily integrated with existing crowd simulation systems. We compare our BCH with commonly used collision shapes, namely the single cylinder and oriented bounding box tree, in terms of query time, construction time, and represented volume. To get an indication of possible crowd densities, we investigate how close characters can be before collision is detected and finally propose a critical maximum depth for the BCH. Copyright © 2014 John Wiley & Sons, Ltd.

Providing high quality recommendations is important for e-commerce systems to assist users in making effective selection decisions from a plethora of choices. Collaborative filtering is a widely accepted technique to generate recommendations based on the ratings of like-minded users. However, it suffers from several inherent issues such as data sparsity and cold start. To address these problems, we propose a novel method called ''Merge'' to incorporate social trust information (i.e., trusted neighbors explicitly specified by users) in providing recommendations. Specifically, ratings of a user's trusted neighbors are merged to complement and represent the preferences of the user and to find other users with similar preferences (i.e., similar users). In addition, the quality of merged ratings is measured by the confidence considering the number of ratings and the ratio of conflicts between positive and negative opinions. Further, the rating confidence is incorporated into the computation of user similarity. The prediction for a given item is generated by aggregating the ratings of similar users. Experimental results based on three real-world data sets demonstrate that our method outperforms other counterparts both in terms of accuracy and coverage.

In this paper, we propose a method to achieve effective facial emotional expressivity for embodied conversational agents by considering two types of asymmetry when exploiting the valence-arousal-dominance representation of emotions. Indeed, the asymmetry of facial expressions helps to convey complex emotional feelings such as conflicting and/or hidden emotions due to social conventions. To achieve such a higher degree of facial expression in a generic way, we propose a new model for mapping the valence-arousal-dominance emotion model onto a set of 12 scalar facial part actions built mostly by combining pairs of antagonist action units from the Facial Action Coding System. The proposed linear model can automatically drive a large number of autonomous virtual humans or support the interactive design of complex facial expressions over time. By design, our approach produces symmetric facial expressions, as expected for most of the emotional spectrum. However, more complex ambivalent feelings can be produced when differing emotions are applied on the left and right sides of the face. We conducted an experiment on static images produced by our approach to compare the expressive power of symmetric and asymmetric facial expressions for a set of eight basic and complex emotions. Results confirm both the pertinence of our general mapping for expressing basic emotions and the significant improvement brought by asymmetry for expressing ambivalent feelings. Copyright © 2013 John Wiley & Sons, Ltd.

In this paper we present a novel vision-based markerless hand pose estimation scheme with the input of depth image sequences. The proposed scheme exploits both temporal constraints and spatial features of the input sequence, and focuses on hand parsing and 3D fingertip localization for hand pose estimation. The hand parsing algorithm incorporates a novel spatial-temporal feature into a Bayesian inference framework to assign the correct label to each image pixel. The 3D fingertip localization algorithm adapts a recently developed geodesic extrema extraction method to fingertip detection with the hand parsing algorithm, a novel path-reweighting method and K-means clustering in metric space. The detected 3D fingertip locations are finally used for hand pose estimation with an inverse kinematics solver. Quantitative experiments on synthetic data show the proposed hand pose estimation scheme can accurately capture the natural hand motion. A simulated water-oscillator application is also built to demonstrate the effectiveness of the proposed method in human-computer interaction scenarios.

Many on-line communities still rely on text messages as their privileged medium of communication and we believe it will remain so even if it can be complemented by images and video. Previous chapters have described how to infer emergent emotion dynamics from text corpus whereas we focus here on the instantaneous display of a detected emotion with a human avatar. The avatar acts as a new modality to provide an emotional visual stream of the messages through facial and body Non-Verbal Communication (NVC). In this chapter we concentrate on chat applications for which the time constraint is critical for preserving the natural flow of the discussion. We describe an efficient mapping of the Valence-Arousal-Dominance emotion model to real-time facial expressions. Within that framework, we stress the need for reconsidering the expressive power of asymmetric facial expression for better conveying a whole range of complex and ambivalent emotions. In the last section we briefly outline how the emotional avatar modality can be offered with recent software standards across systems and display environments.

Articulated hand pose estimation plays an important role in human-computer interaction. Despite the recent progress, the accuracy of existing methods is still not satisfactory, partially due to the difficulty of embedded high-dimensional and non-linear regression problem. Different from the existing discriminative methods that regress for the hand pose with a single depth image, we propose to first project the query depth image onto three orthogonal planes and utilize these multi-view projections to regress for 2D heat-maps which estimate the joint positions on each plane. These multi-view heat-maps are then fused to produce final 3D hand pose estimation with learned pose priors. Experiments show that the proposed method largely outperforms state-of-the-art on a challenging dataset. Moreover, a cross-dataset experiment also demonstrates the good generalization ability of the proposed method., Comment: 9 pages, 9 figures, published at Computer Vision and Pattern Recognition (CVPR) 2016

This paper presents an immersive volleyball game, where a player plays not only against virtual opponents but also with support on his/her side of virtual teammates. This volleyball game has been implemented for several 3D displays such as a stereoscopic display, an autostereoscopic display, Oculus Rift glasses, and a 320o Immersive Room. In this paper, we also propose a user study of the relations between virtual humans and the sense of presence in the different 3D displays. We particularly study how surrounding virtual humans affect the sense of presence. Results show that users more significantly perceived spatial presence of virtual environment and social presence of virtual humans with the Oculus Rift and the Immersive Room.

We present an Augmented Reality solution to allow users to manipulate and inspect 3D virtual objects freely with their bare hands on wearable devices. To this end, we use a head-mounted depth camera to capture the RGB-D hand images from egocentric view, and propose a unified framework to jointly recover the 6D palm pose and recognize the hand gesture from the depth images. The random forest is utilized to regress for the palm pose and classify the hand gesture simultaneously via a spatial-voting framework. With a real-world annotated training dataset, the proposed method shows to predict the palm pose and gesture accurately. The output of the forest is used to render the 3D virtual objects, which are overlaid onto the hand region in input RGB images with camera calibration parameters to provide seamless virtual and real scene synthesis.

Research on no-reference image quality assessment IQA aims to develop a computational model simulating the human perception of image quality accurately and automatically without any prior information about the reference clean image signals. In this paper, we introduce a novel no-reference IQA metric, based on the analysis of structural degradation and luminance changes. Since the human visual system HVS is highly sensitive to structural distortion, we encode the image structural information as the local binary pattern LBP distribution. Besides, image quality is also affected by luminance changes, which cannot be captured properly by LBP threshold mechanism. Hence, the distribution of normalized luminance magnitudes is also included in the proposed IQA metric. Extensive experiments conducted on two large public image databases have demonstrated the effectiveness and robustness of the proposed metric in comparison with the relevant state-of-the-art metrics.

Traversing dense crowds can be challenging, but it is especially difficult in a virtual environment where the user is limited to simple input devices. Predicting near-future user movements with a hidden Markov model allows nearby autonomous agents to react politely, that is, to specifically avoid impeding the movement of the user. This predictive model and simple avoidance scheme are tested by six subjects in 360 short interactive games and by a 10-participant two-alternative forced choice questionnaire. Polite agents are found to significantly improve the player's ability to navigate quickly and without collision, and the resulting character movements showed increased naturalness for two of the three game scenarios. Copyright © 2012 John Wiley & Sons, Ltd.

This paper presents an interactive multi-agent system based on a fully immersive virtual environment. A user can interact with the virtual characters in real time via an avatar by changing their moving behavior. Moreover, the user is allowed to select any character as the avatar to be controlled. A path planning algorithm is proposed to address the problem of dynamic navigation of individual and groups of characters in the multi-agent system. A natural interface is designed for the interaction between the user and the virtual characters, as well as the virtual environment, based on gesture recognition. To evaluate the efficiency of the dynamic navigation method, performance results are provided. The presented system has the potential to be used in the training and evaluation of emergency evacuation and other real-time applications of crowd simulation with interaction.

Computer-based simulations of human hip joints generally include investigating contacts happening among soft or hard tissues during hip movement. In many cases, hip movement is approximated as rotation about an estimated hip center. In this paper, we investigate the effect of different methods used for estimating hip joint center of rotation on the results acquired from hip simulation. For this reason, we use three dimensional models of hip tissues reconstructed from MRI datasets of 10 subjects, and estimate their center of rotation by applying five different methods (including both predictive and functional approaches). Then, we calculate the amount of angular and radial penetrations that happen among three dimensional meshes of cartilages, labrum, and femur bone, when hip is rotating about different estimated centers of rotation. The results indicate that hip simulation can be highly affected by the method used for estimating hip center of rotation. However, under some conditions (e.g. when Adduction or External Rotation are considered) we can expect to have a more robust simulation. In addition, it was observed that applying some methods (e.g. the predictive approach based on acetabulum) may result in less robust simulation, comparing to the other methods.

We present a visualization framework for exploring and analyzing data sets from biomechanical and neuromuscular simulations. These data sets describe versatile information related to the different stages of a motion analysis. In studying these data using a 3D visualization approach, interactive exploring is important, especially for supporting spatial analysis. Moreover, as these data contain many various but related elements, numerical analysis of neuromuscular simulations is complicated. Visualization techniques enhance the analysis process, thus improving the effectiveness of the experiments. Our approach allows convenient definitions of relationships between numerical data sets and 3D objects. Scientific simulation data sets appropriate for this style of analysis are present everywhere motion analysis is performed and are predominant in many clinical works. In this paper, we outline the functionalities of the framework as well as applications embedded within the OpenSim simulation platform. These functionalities form an effective approach specifically designed for the investigation of neuromuscular simulations. This claim is supported by evaluation experiments where the framework was used to analyze gaits and crouch motions.

In the recent years, usage of the third-person perspective (3PP) in virtual training methods has become increasingly viable and despite the growing interest in virtual reality and graphics underlying third-person perspective usage, not many studies have systematically looked at the dynamics and differences between the third and first-person perspectives (1PPs). The current study was designed to quantify the differences between the effects induced by training participants to the third-person and first-person perspectives in a ball catching task. Our results show that for a certain trajectory of the stimulus, the performance of the participants post3PP training is similar to their performance postnormal perspective training. Performance post1PP training varies significantly from both 3PP and the normal perspective.

This paper presents a novel concept: a graphical representation of human emotion extracted from text sentences. The major contributions of this paper are the following. First, we present a pipeline that extracts, processes, and renders emotion of 3D virtual human (VH). The extraction of emotion is based on data mining statistic of large cyberspace databases. Second, we propose methods to optimize this computational pipeline so that real-time virtual reality rendering can be achieved on common PCs. Third, we use the Poisson distribution to transfer database extracted lexical and language parameters into coherent intensities of valence and arousal—parameters of Russell’s circumplex model of emotion. The last contribution is a practical color interpretation of emotion that influences the emotional aspect of rendered VHs. To test our method’s efficiency, computational statistics related to classical or untypical cases of emotion are provided. In order to evaluate our approach, we applied our method to diverse areas such as cyberspace forums, comics, and theater dialogs.