Search

In the past few years, the online video streaming market has witnessed rapid growth and has become the most important form of entertainment. Motivated by the huge business opportunities, the advertisement insertion mechanisms have become a hot topic of research and represent the most important component of an online delivery ecosystem. In this paper, we introduce SemanticAd, a multimodal ad insertion framework designed from the viewers’ perspective in terms of the quality of experience and degree of intrusiveness. The core of the proposed approach involves a novel temporal segmentation algorithm that extracts story units with a frame level precision. To the best of our knowledge, the proposed solution is the most robust and accurate solution dedicated to TV news videos. In addition, by taking into consideration ad temporal distribution and semantic information, the framework proposes commercials that are contextually relevant with respect to video content. The quantitative and qualitative experimental results conducted on a challenging set of 50 multimedia documents validate the SemanticAd methodology, returning a F1-score superior to 92%. Moreover, when compared to other state-of-the-art methods, our system demonstrates its superiority with gains in performance ranging in the [4.19%, 10.22%] interval.

Background: Palliative radiotherapy plays a crucial role in managing symptomatic gynecological cancers (GCs). This article aims to systematically review literature studies on palliative pelvic radiotherapy in cervical, endometrial, ovarian, vaginal, and vulvar cancers. The primary focus is centered around evaluating symptom relief, quality of life (QOL), and toxicity in order to ascertain optimal radiotherapy regimens. Methodology: For this thorough review, we mainly relied on Medline to gather papers published until November 2023. Selected studies specifically detailed symptomatology and QOL responses in palliative pelvic radiotherapy used for GCs. Results: Thirty-one studies, mostly retrospective studies and those lacking standardized outcome measures, showed varied responses. Encouraging outcomes were noted in managing hemorrhage (55%) and pain control (70%). However, comprehensively assessing overall symptom response rates and toxicity remained challenging. Investigations into 10 Gy fractionation revealed benefits in addressing tumor-related bleeding and pain in female genital tract cancers. Conclusions: Palliative pelvic radiotherapy effectively manages symptomatic GCs. Nonetheless, unresolved dosing and fractionation considerations warrant further investigation. Embracing modern therapies alongside radiotherapy offers improved symptom control, emphasizing the importance of selecting suitable patients for successful GC palliation interventions.

In this paper, we introduce a subtitle synchronization and positioning system designed to increase the accessibility of deaf and hearing impaired people to multimedia documents. The main contributions of the paper concern: a novel synchronization algorithm able to robustly align, without any human intervention, the closed caption with the audio transcript and a timestamp refinement technique that adjusts the subtitle segments duration with respect to the audiovisual recommendations. Finally, we introduce a novel method that performs a high level understanding of the multimedia content, in order to determine the subtitle optimal positions, within the video frame, such that they do not overlap with other relevant textual information. The experimental evaluation performed on a large dataset of 30 videos taken from the French national television validates the approach with average accuracy scores superior to 90% regardless on the video genre. The subjective evaluation of the proposed subtitle synchronization and positioning system, performed with actual hearing impaired people, demonstrates the effectiveness of our approach.

In this paper, we introduce the DEEP-AD framework, a multimodal advertisement insertion system dedicated to online video platforms. The framework is designed from the viewer's perspective, in terms of commercial contextual relevance and degree of intrusiveness. The main contribution of the paper concerns a novel multimodal temporal video segmentation algorithm into scenes/stories, which makes it possible to determine automatically the temporal instants that are the most appropriate for inserting advertisement clips. The proposed algorithm exploits various deep convolutional neural networks, involved at several stages. The video stream is first divided into shots based on a graph partition method. The video shots are then clustered into scenes/story units with the help of an agglomerative clustering methodology taking as input visual, audio and semantic features. Furthermore, in order to facilitate the user's access to multimedia documents a novel thumbnail extraction method is proposed based on both semantic representativeness and visual quality information. Finally, the optimal advertisement insertion points are determined based on the ads temporal distribution, commercial diversity and degree of intrusiveness. The experimental results, carried out on a large dataset of more than 30 videos, taken from the French National Television and US TV series validate the proposed methodology with average accuracy and recognition rates superior to 88%. Moreover, when compared with other state of the art methods, the proposed temporal video segmentation yields gains of more than 6% in precision and recall rates.

In this paper, we introduce the DEEP-HEAR framework, a multimodal dynamic subtitle positioning system designed to increase the accessibility of deaf and hearing impaired people (HIP) to multimedia documents. The proposed system exploits both computer vision algorithms and deep convolutional neural networks specifically designed and tuned in order to detect and recognize the identity of the active speaker. The main contributions of the paper concern: a novel method dedicated to recognizing various characters existent in the video stream. A video temporal segmentation algorithm that divides the video sequence into semantic units, based on face tracks and visual consistency. Finally, the core of our approach concerns a novel active speaker recognition method relying on the multimodal information fusion from the text, audio, and video streams. The experimental results carried out on a large scale dataset of more than 30 videos, validate the proposed methodology with average accuracy and recognition rates superior to 90%. Moreover, the method shows robustness to important object/camera motion and face pose variation, yielding gains of more than 8% in precision and recall rates when compared with state-of-the-art techniques. The subjective evaluation of the proposed dynamic subtitle positioning system demonstrates the effectiveness of our approach.

In this paper, we introduce the DEEP-SEE FACE framework, an assistive device designed to improve cognition, interaction, and communication of visually impaired (VI) people in social encounters. The proposed approach jointly exploits computer vision algorithms (region proposal networks, ATLAS tracking and global, and low-level image descriptors) and deep convolutional neural networks in order to detect, track, and recognize, in real-time, various persons existent in the video streams. The major contribution of the paper concerns a global, fixed-size face representation that takes into the account of various video frames while remaining independent of the length of the image sequence. To this purpose, we introduce an effective weight adaptation scheme that is able to determine the relevance assigned to each face instance, depending on the frame degree of motion/camera blur, scale variation, and compression artifacts. Another relevant contribution involves a hard negative mining stage that helps us differentiating between known and unknown face identities. The experimental results, carried out on a large-scale data set, validate the proposed methodology with an average accuracy and recognition rates superior to 92%. When tested in real life, indoor/outdoor scenarios, the DEEP-SEE FACE prototype proves to be effective and easy to use, allowing the VI people to access visual information during social events.

Emotion is a form of high-level paralinguistic information that is intrinsically conveyed by human speech. Automatic speech emotion recognition is an essential challenge for various applications; including mental disease diagnosis; audio surveillance; human behavior understanding; e-learning and human–machine/robot interaction. In this paper, we introduce a novel speech emotion recognition method, based on the Squeeze and Excitation ResNet (SE-ResNet) model and fed with spectrogram inputs. In order to overcome the limitations of the state-of-the-art techniques, which fail in providing a robust feature representation at the utterance level, the CNN architecture is extended with a trainable discriminative GhostVLAD clustering layer that aggregates the audio features into compact, single-utterance vector representation. In addition, an end-to-end neural embedding approach is introduced, based on an emotionally constrained triplet loss function. The loss function integrates the relations between the various emotional patterns and thus improves the latent space data representation. The proposed methodology achieves 83.35% and 64.92% global accuracy rates on the RAVDESS and CREMA-D publicly available datasets, respectively. When compared with the results provided by human observers, the gains in global accuracy scores are superior to 24%. Finally, the objective comparative evaluation with state-of-the-art techniques demonstrates accuracy gains of more than 3%.

In this paper, we introduce the so-called DEEP-SEE framework that jointly exploits computer vision algorithms and deep convolutional neural networks (CNNs) to detect, track and recognize in real time objects encountered during navigation in the outdoor environment. A first feature concerns an object detection technique designed to localize both static and dynamic objects without any a priori knowledge about their position, type or shape. The methodological core of the proposed approach relies on a novel object tracking method based on two convolutional neural networks trained offline. The key principle consists of alternating between tracking using motion information and predicting the object location in time based on visual similarity. The validation of the tracking technique is performed on standard benchmark VOT datasets, and shows that the proposed approach returns state-of-the-art results while minimizing the computational complexity. Then, the DEEP-SEE framework is integrated into a novel assistive device, designed to improve cognition of VI people and to increase their safety when navigating in crowded urban scenes. The validation of our assistive device is performed on a video dataset with 30 elements acquired with the help of VI users. The proposed system shows high accuracy (>90%) and robustness (>90%) scores regardless on the scene dynamics.

In the most recent report published by the World Health Organization concerning people with visual disabilities it is highlighted that by the year 2020, worldwide, the number of completely blind people will reach 75 million, while the number of visually impaired (VI) people will rise to 250 million. Within this context, the development of dedicated electronic travel aid (ETA) systems, able to increase the safe displacement of VI people in indoor/outdoor spaces, while providing additional cognition of the environment becomes of outmost importance. This paper introduces a novel wearable assistive device designed to facilitate the autonomous navigation of blind and VI people in highly dynamic urban scenes. The system exploits two independent sources of information: ultrasonic sensors and the video camera embedded in a regular smartphone. The underlying methodology exploits computer vision and machine learning techniques and makes it possible to identify accurately both static and highly dynamic objects existent in a scene, regardless on their location, size or shape. In addition, the proposed system is able to acquire information about the environment, semantically interpret it and alert users about possible dangerous situations through acoustic feedback. To determine the performance of the proposed methodology we have performed an extensive objective and subjective experimental evaluation with the help of 21 VI subjects from two blind associations. The users pointed out that our prototype is highly helpful in increasing the mobility, while being friendly and easy to learn.