Search

Visual dialogue systems need to understand dynamic visual scenes and comprehend semantics in order to converse with users. Constructing video dialogue systems is more challenging than traditional image dialogue systems because the large feature space of videos makes it difficult to capture semantic information. Furthermore, the dialogue system also needs to precisely answer users’ question based on comprehensive understanding of the videos and the previous dialogue. In order to improve the performance of video dialogue system, we proposed an end-to-end recurrent cross-modality attention (ReCMA) model to answer a series of questions about a video from both visual and textual modality. The answer representation of the question is updated based on both visual representation and textual representation in each step of the reasoning process to have a better understanding of both modalities’ information. We evaluate our method on the challenging DSTC7 video scene-aware dialog dataset and the proposed ReCMA achieves a relative 20.8% improvement over the baseline on CIDEr.

The driver's physiological status has enormous value to public traffic safety and cannot be ignored nowadays. In addition, heart rate (HR) is one of the most important indicators of human's health status. When detecting the driver's HR, using traditional contact-type devices might bring about the driver's distraction or discomfort. On the contrary, the remote photoplethysmography (rPPG) technique is a better way to monitor a driver's HR in vehicle applications simply by using a web-camera without interfering the driver. Most of the rPPG studies intended to reduce the interference caused by facial motion or luminance changes in the indoor or controlled scenario, but there are relatively fewer discussions on outdoor scenarios. Consequently, the purpose of this paper is to enhance the rPPG technique to make it suitable for the outdoor driving scenarios and for monitoring the driver's HR in different weather conditions, including daytime and nighttime. We first utilize artificial neural network (ANN) and train multiple personalized ANN models for each driver. For predicting the drivers' HR beat more precisely, we propose the approach, adaptive neural network model selection (ANNMS), which adaptively selects a personalized ANN model based on different noise conditions. Our algorithm eliminates the effect of noises caused by the variations of facial luminance in eight outdoor driving scenarios. The proposed driver's HR beat monitoring system has been evaluated against the state-of-the-art rPPG techniques that are Chrominance signal-based (CHRO) and k-nearest neighbors-based (kNN) algorithms. Compared with the CHRO and kNN algorithms, the ANNMS reduces the mean absolute error from 14.71 bpm (CHRO) and 9.91 (kNN) to 4.51 bpm (ANNMS) and enhances the success-rate-10, the probability in which the absolute error is below 10 bpm, from 44.1% (CHRO) and 56.3% (kNN) to 91.5% (ANNMS).

Writing a coherent and engaging story is not easy. Creative writers use their knowledge and worldview to put disjointed elements together to form a coherent storyline, and work and rework iteratively toward perfection. Automated visual storytelling (VIST) models, however, make poor use of external knowledge and iterative generation when attempting to create stories. This paper introduces PR-VIST, a framework that represents the input image sequence as a story graph in which it finds the best path to form a storyline. PR-VIST then takes this path and learns to generate the final story via an iterative training process. This framework produces stories that are superior in terms of diversity, coherence, and humanness, per both automatic and human evaluations. An ablation study shows that both plotting and reworking contribute to the model's superiority., 9 pages, ACL-IJCNLP 2021 Findings

In visual storytelling, a short story is generated based on a given image sequence. Despite years of work, most visual storytelling models remain limited in terms of the generated stories’ fixed length: most models produce stories with exactly five sentences because five-sentence stories dominate the training data. The fix-length stories carry limited details and provide ambiguous textual information to the readers. Therefore, we propose to “stretch” the stories, which create the potential to present in-depth visual details. This paper presents Stretch-VST, a visual storytelling framework that enables the generation of prolonged stories by adding appropriate knowledge, which is selected by the proposed scoring function. We propose a length-controlled Transformer to generate long stories. This model introduces novel positional encoding methods to maintain story quality with lengthy inputs. Experiments confirm that long stories are generated without deteriorating the quality. The human evaluation further shows that Stretch-VST can provide better focus and detail when stories are prolonged compared to state of the art. We create a webpage to demonstrate our prolonged capability.

Researchers have begun to utilize heterogeneous knowledge graphs (KGs) as auxiliary information in recommendation systems to mitigate the cold start and sparsity issues. However, utilizing a graph neural network (GNN) to capture information in KG and further apply in RS is still problematic as it is unable to see each item's properties from multiple perspectives. To address these issues, we propose the multi-view item network (MVIN), a GNN-based recommendation model which provides superior recommendations by describing items from a unique mixed view from user and entity angles. MVIN learns item representations from both the user view and the entity view. From the user view, user-oriented modules score and aggregate features to make recommendations from a personalized perspective constructed according to KG entities which incorporates user click information. From the entity view, the mixing layer contrasts layer-wise GCN information to further obtain comprehensive features from internal entity-entity interactions in the KG. We evaluate MVIN on three real-world datasets: MovieLens-1M (ML-1M), LFM-1b 2015 (LFM-1b), and Amazon-Book (AZ-book). Results show that MVIN significantly outperforms state-of-the-art methods on these three datasets. In addition, from user-view cases, we find that MVIN indeed captures entities that attract users. Figures further illustrate that mixing layers in a heterogeneous KG plays a vital role in neighborhood information aggregation.

With the surging significance of personal health care, driver’s physiological state is no longer negligible nowadays. Among all the indicators of health state in human, heart rate (HR) is one of the most cardinal indicators. The commonly used HR measurement is contact-type, might result in driver’s distraction and discomfort in the vehicle applications. To cope with this problem, remote photoplethysmography (rPPG) is utilized to monitor HR at a distance via a web camera. Nevertheless, the rPPG is not without its flaw. The main concern of the rPPG technique is the potential not-robustness result from the arbitrary motion. Consequently, the contribution of this paper is to conquer the motion noise when the car is driving and the driver’s health state is well monitored to enhance the public safety. The proposed algorithm is investigated in not only the indoor environment but as well the outdoor driving, which contains much more unpredictable motion. With k-nearest neighbor (kNN) classifier on chrominance-based features, the mean square error can be reduced from 30.6 to 2.79 bpm, approaching the medical instrument level. The proposed method can be applied to human improving driving safety for Advanced Driver Assistance Systems.