Search

The perceived emotional intelligence of a conversational agent (CA) can significantly impact people’s interaction with the CA. Prior research applies text-based sentiment analysis and emotional response generation to improve CAs’ emotional intelligence. However, acoustic features in speech containing rich contexts are underexploited. In this work, we designed and implemented an emotionally aware CA, called HUE (Heard yoUr Emotion) that stylized responses with emotion regulation strategies and empathetic interjections. We conducted a user study with 75 participants to evaluate their perceived emotional intelligence (PEI) of HUE by having them observe conversations between people and HUE in different emotional scenarios. Our results show that participants’ PEI was significantly higher with the acoustic features than without.

This paper presents WalkingBot, a modular robot system that allows non-expert users to build a multi-legged robot in various morphologies using a set of building blocks with sensors and actuators embedded. The kinematic model of the built robot is interpreted automatically and revealed in a customized GUI through an integrated hardware and software design, so that users can understand, control, and program the robot easily. A Model Predictive Control (MPC) scheme is introduced to generate a control policy for various motions (e.g. moving forward, turning left) corresponding to the sensed robot structure, affording rich robot motions right after assembling. Targeting different levels of programming skill, two programming methods, visual block programming and events programming, are also presented to enable users to create their own interactive legged robot.

Visual-textual presentation layout (e.g., digital magazine cover, poster, Power Point slides, and any other rich media), which combines beautiful image and overlaid readable texts, can result in an eye candy touch to attract users’ attention. The designing of visual-textual presentation layout is therefore becoming ubiquitous in both commercially printed publications and online digital magazines. However, handcrafting aesthetically compelling layouts still remains challenging for many small businesses and amateur users. This article presents a system to automatically generate visual-textual presentation layouts by investigating a set of aesthetic design principles, through which an average user can easily create visually appealing layouts. The system is attributed with a set of topic-dependent layout templates and a computational framework integrating high-level aesthetic principles (in a top-down manner) and low-level image features (in a bottom-up manner). The layout templates, designed with prior knowledge from domain experts, define spatial layouts, semantic colors, harmonic color models, and font emotion and size constraints. We formulate the typography as an energy optimization problem by minimizing the cost of text intrusion, the utility of visual space, and the mismatch of information importance in perception and semantics, constrained by the automatically selected template and further preserving color harmonization. We demonstrate that our designs achieve the best reading experience compared with the reimplementation of parts of existing state-of-the-art designs through a series of user studies.

With the proliferation of mobile devices, people are taking videos of the same events anytime and anywhere. Even though these crowdsourced videos are uploaded to the cloud and shared, the viewing experience is very limited due to monotonous viewing, visual redundancy, and bad audio–video quality. In this paper, we present a fully automatic mobile video mashup system that works in the cloud to combine recordings captured by multiple devices from different view angles and at different time slots into a single yet enriched and professional looking video–audio stream. We summarize a set of computational filming principles for multicamera settings from a formal focus study. Based on these principles, given a set of recordings of the same event, our system is able to synchronize these recordings with audio fingerprints, assess audio and video quality, detect video cut points, and generate video and audio mashups. The audio mashup is the maximization of audio quality under the less switching principle , while the video mashup is formalized as maximizing video quality and content diversity, constrained by the summarized filming principles. Our system is different from any existing work in this field in three ways: 1) our system is fully automatic; 2) the system incorporates a set of computational domain-specific filming principles summarized from a formal focus study; and 3) in addition to video, we also consider audio mashup that is a key factor of user experience (UX) yet often overlooked in existing research. Evaluations show that our system achieves performance results that are superior to state-of-the-art video mashup techniques, thus providing a better UX.

We present style-aware image cloning , a novel image editing approach for artworks, which allows users to seamlessly insert any photorealistic or artificial objects into an artwork to create a new image that shares the same artistic style with the original artwork. To this end, a real-time image transfer algorithm is developed to stylize the cloned object according to a distance metric based on the artistic styles and semantic information. Several interactive functions, such as layering, shadowing, semantic labeling, and direction field editing, are provided to enhance the harmonization of the composite image. Extensive experimental results demonstrate the effectiveness of our method.

Mobile devices are becoming ubiquitous. People take pictures via their phone cameras to explore the world on the go. In many cases, they are concerned with the picture-related information. Understanding user intent conveyed by those pictures therefore becomes important. Existing mobile applications employ visual search to connect the captured picture with the physical world. However, they only achieve limited success due to the ambiguity nature of user intent in the picture-one picture usually contains multiple objects. By taking advantage of multitouch interactions on mobile devices, this paper presents a prototype of interactive mobile visual search, named TapTell, to help users formulate their visual intent more conveniently. This kind of search leverages limited yet natural user interactions on the phone to achieve more effective visual search while maintaining a satisfying user experience. We make three contributions in this work. First, we conduct a focus study on the usage patterns and concerned factors for mobile visual search, which in turn leads to the interactive design of expressing visual intent by gesture. Second, we introduce four modes of gesture-based interactions (crop, line, lasso, and tap) and develop a mobile prototype. Third, we perform an in-depth usability evaluation on these different modes, which demonstrates the advantage of interactions and shows that lasso is the most natural and effective interaction mode. We show that TapTell provides a natural user experience to use phone camera and gesture to explore the world. Based on the observation and conclusion, we also suggest some design principles for interactive mobile visual search in the future.

In this paper we present an approach that synthesizes a dynamic sequence from another related sequence, and apply it to a virtual conductor: to synthesize linked figure animation from an input music track. We propose that the mapping between two dynamic sequences can be modeled with a Kernel-based Hidden Markov model, or KHMM. A KHMM is an HMM for which the kernel-based functions are used to model the state observation density of the joint input and output distribution. Specifically, the state observation density is estimated by employing a likelihood-weighted sampling scheme. Our KHMM model is ideal for dynamic sequence synthesis because the global dynamics are learned by the HMM, and subtle details in the dynamic mapping are kept in the kernel-based state density. We demonstrate our virtual conductor by synthesizing extensive animation sequences from input music sequences with different styles and beat patterns.

Traditional manga (comic) and anime (cartoon) creation are painstaking processes. Even computers are utilized for the content creation, computers are mainly utilized as a naive digital canvas. With the increasing computing power and decreasing cost of CPU & GPU, more computing resource can be exploited cost-effectively for intelligent and semi-automatic creation of aesthetics content.In this course, we gather speeches from both artist and scientists. We shall discuss what kind of computer technologies an artist would prefer to use in their creation (i.e. user point of view), and recent technologies developed in graphics that facilitates or even inspires the content creation.

We introduce our interdisciplinary studies on information art & design, including 1) the interdisciplinary master program of information design at Tsinghua University that is a new platform to integrate art, new media and technology, to educate students through both theoretical and practical training on cutting-edge and social impact research projects; 2) e-Heritage, which is to apply the latest information technologies to aid the preservation, interpretation, and dissemination of cultural and natural heritages, in which we are working together with museums and cultural heritage sites to develop immersive multimedia exhibition systems, to recover old paintings, and to animate some art contents; 3) new media design and its applications of digital museum and gallery; 4) discussion on the future of information art & design.

Due to the rapid increase in video capture technology, more and more tourist videos are captured every day, creating a challenge for organization and association with metadata. In this paper, we present a novel system for annotating and navigating tourist videos. Placing annotations in a video is difficult because of the need to track the movement of the camera. Navigation of a regular video is also challenging due to the sequential nature of the media. To overcome these challenges, we introduce a system for registering videos to geo-referenced 3D models and analyzing the video contents. We also introduce a novel scheduling algorithm for showing annotations in video. We show results in automatically annotated videos and in a map-based application for browsing videos. Our user study indicates the system is very useful.

We present an interactive system that stylizes an input video into a painterly animation. The system consists of two phases. The first is an Video Parsing phase that extracts and labels semantic objects with different material properties (skin, hair, cloth, and so on) in the video, and then establishes robust correspondence between frames for discriminative image features inside each object. The second Painterly Rendering phase performs the stylization based on the video semantics and feature correspondence. Compared to the previous work, the proposed method advances painterly animation in three aspects: Firstly, we render artistic painterly styles using a rich set of example-based brush strokes. These strokes, placed in multiple layers and passes, are automatically selected according to the video semantics. Secondly, we warp brush strokes according to global object deformations, so that the strokes appear to be tightly attached to the object surfaces. Thirdly, we propose a series of novel teniques to reduce the scintillation effects. Results applying our system to several video clips show that it produces expressive oil painting animations.

We present an accelerated depth peeling algorithm for order-independent transparency rendering on graphics hardware. Unlike traditional depth peeling which only peels one layer of transparent pixels per rendering pass, our algorithm peels multiple layers simultaneously per rendering pass. Our acceleration is achieved via our fragment program which sorts and writes multiple fragment colors and depths via MRT. A notable feature of our algorithm is that it is robust against the unreliable parallel read-after-write behavior in current graphics hardware, guaranteeing correct transparency ordering. For ordinary scenes rendered under RGBA8 color precision, we achieve up to 8x speed-up over conventional depth peeling with current generation graphics hardware. Our algorithm is simple to implement on current GPU without any hardware modification. In addition, it does not require applications to perform any pre-sorting of transparent geometry.

This paper presents a sketch-based password authentication system called Scribble-a-Secret as a graphical password scheme in which free-form drawings are used as a means to authenticate users. Unlike existing schemes, this approach requires no input of graphical passwords in particular sequences of strokes. Moreover, the system allows for a modicum of variation when users recreate their passwords. Our technique uses edge orientations extracted from sketch images to discern one user from another. Our experiments show that our recognition technique is robust for recognizing sketches while differentiating from others with both a false acceptance rate and false rejection rate of less than 1%.

A family photo album based cartoon personalization system, EasyToon, is proposed in this paper. Using state of the art computer vision and graphics technologies and effective UI design, the interactive tool can quickly generate a personalized cartoon storyboard, which naturally blends a real face chosen from the family photo album into a cartoon picture. The personalized cartoon image is easily and quickly obtained in two main steps. First, the best face candidate is selected from the album interactively. Then a personalized cartoon image is automatically synthesized by blending the selected face into the interesting cartoon image. Experiments show that most users express great interest in our system. Without any art background, they can make a personalized cartoon of high quality using the EasyToon within minutes.

Cosmetic makeup is used worldwide as a means to enhance beauty and express moods. An art form in its own right, cosmetic styles continuously change and evolve to reflect cultural and societal trends. While countless magazines and books are dedicated to demonstrating cosmetic art, the actual application of makeup still remains a physical endeavor. In this paper, we describe a procedure to apply cosmetic makeup to the image of a person's face with the click of a mouse. Our approach works from before- and-after example images created by professional makeup artists. Using our "cosmetic-transfer" procedure, we can realistically transfer the cosmetic style captured in the example-pair to another person's face. This greatly reduces the time and effort needed to demonstrate a cosmetic style on a new person's face. In addition, our approach can be used to mix-and- match, and even fine-tune, example styles, all virtually, without the need for any physical makeup.

In this paper, we introduce an interactive tool for local color transfer. The new technique is based on the observation that color transfer operations are local in nature while at the same time should adhere global consistency. We introduce a brush by which the user specifies the source and destination image regions for color transfer. Color statistics in the source region are transferred to the destination region. A global optimization is then applied to eliminate vi- sual discontinuities that may result by the local operations. We demonstrate that our tool is easy to use yet effective in quickly generating diverse artistic effects.

We present a method that preserves the recognizability of key object interactions in a vector animation. The method allows an artist to author an animation once, and then output it to any display device. We specifically target mobile devices with small screen sizes. In order to adapt an animation, the author specifies an importance value for objects in the animation. The algorithm then identifies and categorizes the vector graphics objects that comprise the animation, leveraging the implicit relationship between extensible Markup Language (XML) and scalable vector graphics (SVG). Based on importance, the animation can then be automatically retargeted for any display using artistically motivated resizing and grouping algorithms that budget size and spatial detail for each object.

Lighting condition is an important factor in face analysis and synthesis, which has received extensive study in both computer vision and computer graphics. Motivated by the work on multilinear model, we propose a learning-based algorithm for relighting based on tensor framework, which explicitly accounts for the interaction of the identity factor and the lighting factor. The major contribution of our work is that we develop a novel algorithm based on a two-stage decomposition scheme to simultaneously and robustly solve for the identity parameter and the lighting parameter which are both unknown. Equipped with the decomposition algorithm, the capability of the tensor model is significantly extended. Experiment results illustrate the effectiveness of our algorithm.

Creating a portrait in the style of a particular artistic tradition or a particular artist is a difficult problem. Elusive to codify algorithmically, the nebulous qualities which combine to form artwork are often well captured using example-based approaches. These methods place the artist in the process, often during system training, in the hope that their talents may be tapped.Example based methods do not make this problem easy, however. Examples are precious, so training sets are small, reducing the number of techniques which may be employed. We propose a system which combines two separate but similar subsystems, one for the face and another for the hair, each of which employs a global and a local model. Facial exaggeration to achieve the desired stylistic look is handled during the global face phase. Each subsystem uses a divide-and-conquer approach, but while the face subsystem decomposes into separable subproblems for the eyes, mouth, nose, etc., the hair needs to be subdivided in a relatively arbitrary way, making the hair subproblem decomposition an important step which must be handled carefully with a structured model and a detailed model.

This paper describes an experimental study on the detection of emotion from speech. As computer-based characters such as avatars and virtual chat faces become more common, the use of emotion to drive the expression of the virtual characters becomes more important. This study utilizes a corpus containing emotional speech with 721 short utterances expressing four emotions: anger, happiness, sadness, and the neutral (unemotional) state, which were captured manually from movies and teleplays. We introduce a new concept to evaluate emotions in speech. Emotions are so complex that most speech sentences cannot be precisely assigned to a particular emotion category; however, most emotional states nevertheless can be described as a mixture of multiple emotions. Based on this concept we have trained SVMs (support vector machines) to recognize utterances within these four categories and developed an agent that can recognize and express emotions.