Search

In this paper, we introduce a subtitle synchronization and positioning system designed to increase the accessibility of deaf and hearing impaired people to multimedia documents. The main contributions of the paper concern: a novel synchronization algorithm able to robustly align, without any human intervention, the closed caption with the audio transcript and a timestamp refinement technique that adjusts the subtitle segments duration with respect to the audiovisual recommendations. Finally, we introduce a novel method that performs a high level understanding of the multimedia content, in order to determine the subtitle optimal positions, within the video frame, such that they do not overlap with other relevant textual information. The experimental evaluation performed on a large dataset of 30 videos taken from the French national television validates the approach with average accuracy scores superior to 90% regardless on the video genre. The subjective evaluation of the proposed subtitle synchronization and positioning system, performed with actual hearing impaired people, demonstrates the effectiveness of our approach.

Video Subtitles are not only an essential tool for the hearing impaired, but also enhance the user's viewing experience, as they allow users to better understand and interpret different accents, even if they are of the same familiar language. Automatic Speech Recognition Systems would also eradicate the strenuous mechanical process involved in creating subtitle files for movie videos. Searching and indexing of different scenes in a video is still far behind when compared to that available for other forms like text data. With the help of Video Captioning models, the accessibility and indexing requirements of video files can be significantly improved by allowing the users to search for a particular scene/event in a video. This paper discusses about the solution offered to these requirements with the help of sequence-to-sequence recurrent neural networks. The paper also includes the different techniques involved in preprocessing the audio data and extracting features from them, the network architectures, CTC algorithm for backpropagation of error through time, suitable evaluation metrics for Sequence-to-Sequence models and the challenges involved during the designing and training phase of such models.

Language learning with Netflix is not only a phenomenon observed in both informal and formal language learning but also the name of a Chrome extension that provides language features to enhance comprehension and language learning through watching second language (L2) content on Netflix. Equipped with access to Netflix and the extension, 39 intermediate-level tertiary language learners of German and Spanish explored the features of the tool to assist the viewing of self-selected TV series in their L2. Results indicate that formal language learners watch Netflix series in their second language rather for entertainment than for learning yet feel supported by the dual subtitle feature of Language Learning with Netflix which allowed them to transition from LI subtitle to L2 subtitle use and to watch L2 Netflix with confidence.

Sign language is a common language for deaf and mute people to communicate information. In order to make deaf people watch video without barrier, video plays and sign language translation solves the problem of deaf people obtaining video information. Subtitle processing and Chinese word segmentation were adopted to find corresponding words in sign language dictionary, transcoding was sent to unity through socket mechanism, realizing the process of sign language translation driven by video subtitle. Deaf people can watch video with Chinese subtitles and see clear, smooth and natural sign language translation animation when using it. The sign language translation animation library is based on the unified, standardized and common grammar sign language. Users can easily get the information in video and gradually standardize the use of sign language. Movie sign language translation system is of special significance to the hearing impaired people to improve their quality of life and has produced practical value for social progress.

Multi-speaker speech recognition of unsegmented recordings has diverse applications such as meeting transcription and automatic subtitle generation. With technical advances in systems dealing with speech separation, speaker diarization, and automatic speech recognition (ASR) in the last decade, it has become possible to build pipelines that achieve reasonable error rates on this task. In this paper, we propose an end-to-end modular system for the LibriCSS meeting data, which combines independently trained separation, diarization, and recognition components, in that order. We study the effect of different state-of-the-art methods at each stage of the pipeline, and report results using task-specific metrics like SDR and DER, as well as downstream WER. Experiments indicate that the problem of overlapping speech for diarization and ASR can be effectively mitigated with the presence of a well-trained separation module. Our best system achieves a speaker-attributed WER of 12.7%, which is close to that of a non-overlapping ASR., Accepted to IEEE SLT 2021

This paper considers recognizing texts shown in a source language and translating into a target language, without generating the intermediate source language text image recognition results. We call this problem Cross-Lingual Text Image Recognition (CLTIR). To solve this problem, we propose a multi-task system containing a main task of CLTIR and an auxiliary task of Mono-Lingual Text Image Recognition (MLTIR) simultaneously. Two different sequence to sequence learning methods, a convolution based attention model and a Bidirectional Long Short-Term Memory (BLSTM) model with Connectionist Temporal Classification (CTC), are adopted for these tasks respectively. We evaluate the system on a newly collected Chinese-English bilingual movie subtitle image dataset. Experimental results demonstrate the multi-task learning framework performs superiorly in both languages.

Humor is a very important communication tool; yet, it is an open problem for machines to understand humor. In this paper, we build a new multimodal dataset for humor prediction that includes subtitles and video frames, as well as humor labels associated with video’s timestamps. On top of it, we present a model to predict whether a subtitle causes laughter. Our model uses the visual modality through facial expression and character name recognition, together with the verbal modality, to explore how the visual modality helps. In addition, we use an attention mechanism to adjust the weight for each modality to facilitate humor prediction. Interestingly, our experimental results show that the performance boost by combinations of different modalities, and the attention mechanism and the model mostly relies on the verbal modality.

Subtitles are precious dialogue text data because of similarity to human conversation, but the lack of turn structures limits their applications in many NLP tasks. The previous work takes turn-taking detection (TTD) in subtitles as a sentence-pair classification problem to predict if there is a turn-taking happened between the two adjacent utterances. The results are not good enough. For considering the dialogue context information, we innovatively take TTD as a sentence-level sequence labelling problem, predicting the turn-takings through the whole subtitle stream. First, we present a novel fine-tuning method that enables BERT to encode utterances into embedding set with effective turntaking features. Then, we propose the Semantic Recall Transformer (SRT) model to detect the turn-takings among the utterance embedding set, by taking the turn-taking features and context information into account at the same time. Compared with the baselines, it achieves state-of-the-art on both English and Chinese subtitle corpus. Moreover, we explored the impacts of the length of the subtitle stream and the number of conversation participants on our model performance, which show the performance can be further improved in the future.

An obvious problem with automatic speech recognition (ASR) for Taigi is that the amount of training data is far from enough to build a practical ASR system. Collecting speech data with reliable transcripts for training the acoustic model (AM) is feasible but expensive. Moreover, text data used for language model (LM) training is extremely scarce and difficult to collect because Taigi is a spoken language, not a commonly used written language. Interestingly, the subtitles of Taigi drama in Taiwan have long been in Chinese characters for Mandarin. Since a large amount of Taigi drama episodes with Mandarin Chinese subtitles are available on YouTube, we propose a method to augment the training data for AM and LM of Taigi ASR. The idea is to use an initial Taigi ASR system to convert a Mandarin Chinese subtitle into the most likely Taigi word sequence by referring to the speech. Experimental results show that our ASR system can be remarkably improved by such training data augmentation.

This study selected a video lecture from a public MOOC platform and carried out an eye tracking experiment to investigate the effects of different representation styles on user attention. The results indicated that the subtitle positions and the background sounds in MOOC videos could significantly affect users’ attention, while same repeated scenes and different animated scenes produced using different cinematography techniques could not significantly affect users’ attention. The results of this work could benefit the designs of MOOC.

The project aims for a machine translation system for augmenting Kannada educational video by providing a translation to Indian Sign Language (ISL) using virtual animation approach along with the subtitles. The system accepts Kannada educational video clips as input and output is delivered as 3D character animation. The first phase approach of the proposed system makes use of SpeechRecognition like kaldi an open source for converting the input video files to their respective text which are used to generate the subtitles and these texts are further pre-processed in order to meet the sign language norms. The right SiGML file for these pre-processed words is picked from the database. These SiGML files consist of XML tags which correspond to the respective HamNoSys Notations. HamNoSys is an alphabetic system that mostly phonetically describes signs. A synthetic animation of a human in the tool in the URL APP, JA SiGML takes input as these SiGML files and generates signs of the respective pre-processed words. Synchronization of inputted video with the sign language and the subtitle generation is vital for the clear and clean flow. This system is Tested by professional sign language trainer/interpreter for the sign language generated by the avatar and is found effective. The accuracy of sign language shown was found out to be 63%.

Visual adversarial attacks inspired by Carlini-Wagner targeted audiovisual attacks can fool the state-of-the-art Google DeepMind LipNet model to subtitle anything with over 99% similarity. We explore several methods of visual adversarial attacks, including the vanilla fast gradient sign method (FGSM), the $L_{\infty}$ iterative fast gradient sign method, and the $L_{2}$ modified Carlini-Wagner attacks. The feasibility of these attacks raise privacy and false information threats, as video transcriptions are used to recommend and inform people worldwide and on social media.

Subtitles are content gotten from either a transcript or screenplay of the discourse or critique in movies, TV programs, computer games. In a lion's share of cases inside a video, the sound holds a critical spot. At present, to incorporate subtitles into the videos, we have to download them from a third-party source and manually add it to our media player of choice. This poses a few problems: The video content we are trying to transcribe may not always have subtitles readily available for it and an active internet connection would be required to even be able to search for such a resource before we can consume it. We aim to try to solve these problems by performing a comparative analysis of various speech recognition engines in a real-time use case, i.e. generating subtitles for a video being played on a media player in an offline environment. Among the speech recognition engines that were compared, DeepSpeech obtained the lower Word Error Rate (WER) of twenty-six percentage, but when we consider system resource usage as well, CMU Sphinx proves to be a better overall engine for the given use case.

Subtitle location and recognition technology is a special case of natural scene text recognition, but there are sorne differences between them, The subtitles in video frame have fixed position and lots of edge features. This papertakes that as a starting point, extracting edge features with Sobel operatorto locate the subtitle, adjusting color scale and taking the method of binarization to get the text image with pure background, finally obtains the text inthe frame via OCR technology.

With the development of information technology, more and more videos appear on the Internet. As relationship of characters is an important part of a video content, audiences can understand storyline of the video easily through the relationship. Meanwhile, the relationship can facilitate audiences to watch videos that they are interested in efficiently. In the last years, researchers have analyzed video characters in researches of video content detection. However, they treated characters as a number of independent objects and ignore their relationships. In this paper, we present a new approach to analyze the relationship of characters in a video by extracting the semantics of the video. The semantics of the video are a description of the things that made up with subtitle, action, character expression, Video semantic features are extracted and fused to identify the relationship of characters in the video. In our experiment, images are selected from the video that can easily show the relationship of the characters as the dataset. The accuracy of in the experiment is 89%. Its performance is better than the accuracy of using only one feature. At the same time, we predicted the relationship of characters using our model in Friends. The relationship of characters is almost accurately predicted under the comparison of the facts of Friends.

Complex word identification (CWI) is a fundamental educational natural language processing (NLP) task that identifies complex words in a text for use in many further applications, including text simplification. Word frequency features from a general corpus were known to be effective in the identification of complex words. Recent studies have independently discovered that, when used in combination with word frequency features from a general corpus, word frequency features from some types of non-general corpora, namely, a movie subtitle corpus (Peatzold and Specia, 2016) and a learner corpus (Kajiwara and Komachi, 2018), improved the CWI accuracy. As long as the reason why this combination of general and non-general corpora improved accuracy remained unclear, the search for effective corpora combinations and other types of effective corpora had to be exhaustive. To contribute to a non-exhaustive search, this study analyzes similarities among the word frequency distributions of various corpora, which can be conducted in an unsupervised manner. In experiments using a CWI shared task dataset, our method successfully explains why the previously reported combinations of corpora were effective, while finding effective corpora combinations.

International audience; In this paper we introduce an automatic subtitle positioning approach designed to enhance the video accessibility of deaf and hearing impaired people to multimedia documents. By using a dynamic subtitle and captioning approach, which exploits various computer vision techniques, including face detection, tracking and recognition, video temporal segmentation into shots and scenes and active speaker recognition, we are able to position each video subtitle segment in the near vicinity of the active speaker. The experimental evaluation performed on 30 video elements validates our approach with average F1-scores superior to 92%.

In recent years, the scarcity of dialogue corpus is becoming the bottleneck of Chinese dialogue generation systems. Although subtitles provide favorable material to construct dialogue corpus because of their abundance and diversity, lacking speaker information makes it hard to extract dialogues from subtitles directly. To utilize these resources, we proposed an improved method to automatically annotate bilingual TV subtitles with speaker and scene tags using their corresponding scripts. First, tags of speakers and scene boundaries in the scripts are mapped to the subtitles through an information retrieval method. Then, the mapping errors are detected with a convolutional network and corrected by heuristic strategies to improve the annotation quality. We applied this method on 779 bilingual subtitle files of 4 TV series and obtained a Chinese dialogue corpus Tv4Dialog11It is publicly available at https://github.com/zll17/TV4Dialog containing 260674 utterances. Experiment result shows that our method can achieve an accuracy of 94.62% on speaker tag annotation, improving nearly 12% on the previous state-of-the-art result.

In this paper, we introduce a novel dynamic subtitle positioning system designed to increase the accessibility of the deaf and hearing impaired people to video documents. Our framework places the subtitle in the near vicinity of the active speaker in order to allow the viewer to follow the visual content while regarding the textual information. The proposed system is based on a multimodal fusion of text, audio and visual information in order to detect and recognize the identity of the active speaker. The experimental evaluation, performed on a large dataset of more than 30 videos, validates the methodology with average accuracy and recognition rates superior to 92%. The subjective evaluation demonstrates the effectiveness of our approach outperforming both conventional (static) subtitling and other state of the art techniques in terms of enhancement of the overall viewing experience and eyestrain reduction.

Joint vision and language tasks like visual question answering are fascinating because they explore high-level understanding, but at the same time, can be more prone to language biases. In this paper, we explore the biases in the MovieQA dataset and propose a strikingly simple model which can exploit them. We find that using the right word embedding is of utmost importance. By using an appropriately trained word embedding, about half the Question-Answers (QAs) can be answered by looking at the questions and answers alone, completely ignoring narrative context from video clips, subtitles, and movie scripts. Compared to the best published papers on the leaderboard, our simple question + answer only model improves accuracy by 5% for video + subtitle category, 5% for subtitle, 15% for DVS and 6% higher for scripts., Spotlight presentation at CLVL workshop, ICCV 2019. Project page: https://bhavanj.github.io/MovieQAWithoutMovies/

In order to solve the problem of three-dimensional (3D) pose and shape detection, a 3D shape detection method based on multi-core fiber distortion compensation was designed. This method is a 3D shape detection method based on three-core fiber grating array, which adopts interpolation, cyclic iteration and distortion compensation method. The simulation results show that this method can achieve high precision 3D shape detection.

This paper deals with the social magazine creation service using chatbot and video creation based on the contents of the social magazine. The social magazine service is to provide a story-formatted magazine from particular topic with a variety of articles, through a series of processes such as collecting and analyzing user conversations, extracting keywords appropriate to the situation, eliminating duplicated contents, and summarizing text. Reprocessing the social magazine in the form of video is also provided using speech synthesis, subtitle generation, and some effects with images and text information. The proposed system is tested in the real environment and found that it works well.

Today, there are quite a few ways of promoting coming-soon movies to various target audiences. Putting up clips of movie trailers on Youtube channel is one of a kind promotion publicly. A good movie is always equipped with its movie trailer to gain as many movie-goers' interest as possible. However, there is one shortcoming of a movie trailer clip found on Youtube channel. The shortcoming is that almost every single movie trailer has no subtitles of the target audience's. Only closed-captions dialogues in the source language are currently available instead of the written dialogues in the target audience's language. This study discusses three matters on movie trailers. Firstly, it aims at knowing the working process of operating the AEGISUB software by the English department students of Universitas Dian Nuswantoro. Secondly, it discusses the work efficiency of the software dealing with subtitling matters. Thirdly, it describes meaning accuracy in translating idiomatic expressions of the SL with the software. From the data analysis, it is found that all the students made a good understanding by following all the subtitling procedures with the software. The rendering process was found more efficient due to the subtitling rules. Each text line in the TL was made efficient according to the rules in terms of the number of characters, time duration for one line to appear on the screen, and the number of text lines in one subtitle. Each idiomatic expression was accurately rendered in the target audience's language. It was made possible because the software allows the user to identify any idiomatic expression found in one text line.

In recent years, multimedia service providers have been focusing on improving the users quality of experience (QoE) of audiovisual services. To examine the factors that affect QoE efficiently, we explore models to reflect the quantitative relationship between influential factors and QoE. We carry out a series of subjective quality tests to investigate the effects of audiovisual and subtitle synchronization, and playback buffering under a limited transmission bandwidth, on perceived quality. We analyze statistics and propose an analytical model to determine the quantitative relationship between the aforementioned factors and perceived audiovisual quality. The proposed model is content-dependent and can monitor QoE when delivering multimedia data through a dynamic network. Therefore, it is capable of providing more accurate guidance for improving audiovisual transmission services.

The aim of the present experiments is to study the effect of relative depth and horizontal position of two lines of subtitles in 3D films on stereoscopic visual comfort. The study included two experiments. In the experiment 1, it mainly studied the influence of the relative disparity of two-line 3D subtitles on visual comfort. The results showed the relative disparity of two-line subtitles was +42.64 arc min that was more comfortable for the viewers. And In experiment 2, the study was on the effect of the relative horizontal displacement of the two lines of subtitles on visual comfort. The results showed the horizontal position of two-line subtitles had no effect on visual comfort.

Dialogue generation has been gaining ever-increasing attention, and various models have been proposed and adopted in many fields in recent years. How to evaluate their performance is critical. However, current evaluation metrics tend to be insufficient because of their simplicity and crudeness, resulting in weak correlation with human judgements. To solve this issue, we propose an automatic and comprehensive evaluation metric, which consists of three assessment criteria: Semantic Coherence, Syntactic Validity and Ability of Expression (SSA). The first two criteria are used to evaluate the generations from semantic and syntactic aspects respectively at the sentence level and the last one is to evaluate the overall performance at the model level. With two generative models, we conduct experiments on three datasets, including Twitter, Subtitle and Lenovo. Comparing with the previous metrics such as BLEU, METEOR and ROUGE, the correlation coefficient between SSA and human judgements is increased by 0.23-0.35, i.e. 324%-864% relative improvements. The experimental results demonstrate that SSA correlates more strongly with human judgements on the evaluation for open dialogue generation. Additionally, SSA is able to evaluate the semantic coherence and syntactic validity of generations exactly. More importantly, the evaluation models can be trained without human annotations. Thus, SSA is flexible and extensible to different datasets.

Video lectures improve the learning experiences considering individual's needs and learning styles. However, the large amount of educational content and their availability in a fragmented way turns difficult the tasks of accessing these resources and understanding the concepts under study. Extracting relevant information from video lectures can be useful for recommendation purposes and for helping the interpretation of a concept in an exact moment of a lecture. The extraction of semantic metadata from a video natural language subtitle involves challenges in dealing with informal aspects of language and the detection of semantic classes from the text. In this paper, we conduct an empirical analysis of semantic annotation approaches supported by ontologies in the extraction of relevant metadata from textual transcriptions of video lectures in Computer Science. The obtained results indicate that existing tools can be useful for the studied task and the video lecture semantic metadata extraction process is highly influenced by the underlying ontologies.

Association Rule Hiding is the Data Mining method which is used for extracting hidden information from huge dataset. In our paper, Two approaches are introduced. In first approach FP Growth Algorithm is being presented that generate association rules efficiently and it reduces time of forming frequent item sets every time. In second approach we have tried to hide sensitive association rules by Genetic Algorithm. Generally from large databases frequent items are extracted by applying different algorithms. In this paper, we compare all the algorithms for extracting frequent items which are Association Rule Mining, Apriori Algorithm and FP growth Algorithm. We also compare Fuzzy Logic Algorithms and Genetic Algorithm for hiding sensitive association rules.

The media conveyed may contain questionable substance, for example, a smoking scene, and related substance, for example, a heavy drinker scene, and so on. These days the movie producers furthermore, editors need to physically check the whole video feed to recognize the frightful substance. After scanning all the scenes, they put enough time and exertion into putting statutory alerts and signs in video substance. Statutory admonitions are shown in public interest declaring that the motion picture/TV indicates does not encourage smoking, drinking liquor and so on. In this paper, we propose a programmed method of statutory admonitions utilizing profound learning framework dependent on one of the profound learning strategy, a Darknet-53 based Convolutional Neural Network which is called YOLOv3. Utilizing this, it is possible to computerize this procedure of including statutory alerts in movies or TV arrangement or videos. For example, if there is a smoking scene in a film, at that point the undertaking will identify it and will demonstrate a related statutory cautioning as a subtitle.

With rapid development of the internet, the tremendous growth has been observed in video sharing platform. Millions of users have been using this platform as source of knowledge, entertainment, self-development, but language barrier has confined the reach of this platform to certain amount of population. Also the hearing-impaired people could not make satisfactory use of this platform. Subtitles play important role in such scenarios, with the help of which people can understand the content of the video. Various applications have been developed for subtitle generation to resolve issue of language barrier and hearing impairment. In this paper we have surveyed different methods for speech recognition and bilingual subtitle generation and proposed a system for instant bi-lingual subtitle generation based on MFCC and HMM.

We demonstrate the potential for using aligned bilingual word embeddings to create an unsupervised method to evaluate machine translations without a need for parallel translation corpus or reference corpus. We explain why movie subtitles differ from other text and share our experimental results conducted on them for four target languages (French, German, Portuguese and Spanish) with English source subtitles. We propose a novel automated evaluation method of calculating edits (insertion, deletion, substitution and shifts) to indicate translation quality and human aided post edit requirements to perfect machine translation.

This research focused on the subtitles of 360-degree video journalism to explore the impact of subtitles presentation on video viewing behavior. 27 participants were asked to watch the 360-degree video journalism by using the Head Mount Display (HMD). The content which the participants watched was recorded, and the participants were required to complete a memory test related to the video journalism content after watching. By analyzing the content of the video and combining with the results of the memory test, the results showed that participants' subtitle viewing behavior had certain commonality, that was, participants paid more attention to the position with subtitles, and the subtitle viewing behavior was affected by the screen content; moreover, the duration time of the subtitles was positively related to video journalism content memory.

The Hybrid Broadcast Broadband Television (HbbTV) has become very popular in the last few years because it provides richer and more interactive viewing experience to the end user. To obtain a HbbTV certificate, a device must satisfy some rules and norms. One of these rules is the subtitle and video synchronization of a given video content. Therefore, in this paper, a new application for checking the subtitle and video content synchronization is proposed. The algorithm is written in the programming language C and it consists of three modules. These modules are modules for reading XML files, a module for the video frame analysis, and report generation module.

Science is an important part of human life. To communicate environmental information adequately and effectively, the mass media need to provide suitable information and channels so the community can acquire new information, and thus generate innovations. In addition, new media have become quite important and convenient to a new generation. The purpose of this study is to investigate ways to shoot and produce an interactive short film for new media regarding popular environmentally-important issues that will increase audience learning and involvement. The researchers chose to use as a sample topic an environmental news feature issue: ways microplastics spell big problems for future generations. This application paper indicates methods of producing an interactive popular science video step-by-step, and especially focus on shooting methods, one of the most important parts for producing short films. Having produced an interactive short film, the researchers launched two focus group interviews of 12 audience members to discuss their perceptions and understanding of this short film. This study aims to compare their interests, perception and understanding regarding this popular science short film against the skills involved in video shooting and production to gain insight into the factors which affected audience understanding of the film.

Videos are good for learning vocabulary, since they present the vocabulary in the context of natural dialogs. However, general videos do not provide good supports for language learners. This paper presents a subtitle summarization system, which is designed for vocabulary learning. The system can help a learner watch only meaningful scenes that contain the suitable vocabulary. Using the system, a learner can acquire vocabulary more efficiently since the suitable vocabulary is extracted based on the learning level of the learner. To investigate whether the system enhances learning efficiency or not, an evaluation experiment was conducted. The learning efficiency means the score of vocabulary test compared to the length of the learning time. Experimental participants were intermediate-level learners studying Japanese. They were divided into two groups: the experimental group with our system and control group with a system by a previous work. However, though our system seemed to be more advantageous, the result of the experiment showed no significant difference between two groups. The considerable reason for this result is that the summarization feature of the system is not effective enough since the contexts of the video contents are not considered when the system extracts subtitles containing the suitable vocabulary.

In this paper, we describe a relatively convenient way to extract and recognize text from a background in various circumstances of complicity. One general purpose in many applications is to extract hardcode subtitles from videos. Popular hardcode subtitle-rip tools (applications) nowadays follow a similar procedure from text location settings, customized image post-editing to OCR process. The quality of results usually may not be satisfied, and the supporting video formats are limited. Thanks to "GrabCut" image segmentation algorithm and "Tesseract-OCR" technology, we have developed a more augmented version in python for hardcode subtitle extraction on a fixed location, based on many attempts of experimental practice and research. The method we proposed outperforms most of the competitive tools on the quality of results.

With the enormous growth rate in user-generated videos, it is becoming increasingly important to be able to navigate them efficiently. Video summarization is considered a promising approach for efficacious realization of video content through Identifying and picking out descriptive frames of the video. In this paper, we propose an adaptive framework called Smart-Trailer (S-Trailer) to automate the process of creating an online trailer for any movie based only on its subtitle. The language used in the subtitle is English. The framework analyzes the movie subtitle file to extract relevant textual features that are used to classify the movie into its corresponding genre(s). Initial experimentation resulted in generating genre-classification corpus. The generated corpus is tested against real movies dataset and showed high classification accuracy rate (0.89) in classifying movies into their corresponding genre(s). The proposed system returned automated trailers that contain on average 47% accuracy in terms of recalling scenes appeared on the original movie trailer for different movie genres. Currently, we employ deep learning techniques to captures user behaviors and opinions in order to adapt our system to provide users with relevant video scenes recommendations that match their preferences.

This paper presents a novel method to detect the static subtitle using the saliency map. In the proposed method, subtitle candidates are first extracted by using an unconstrained scene text detector algorithm, called FASText. Then, the saliency map is employed to select the final subtitle among the subtitle candidates. Experimental results show that the proposed method achieves more accurate subtitle detection performance as compared with the conventional methods.

Existing implementation of HLS protocol was able to download only one video stream during one protocol session. In this paper is described extension and integration of existing HLS software support. This extension provides support for downloading audio and subtitle streams in parallel with already implemented video stream downloading with option to change current audio or subtitle stream. ExoPlayer library is used for verification of this extension.

In this paper, the design and implementation of European Broadcasting Union Timed Text part 'D' (EBU-TT-D) subtitles interpreter, based on eXtensible Markup Language (XML), is presented. The design is realized using web technologies HyperText Markup Language (HTML), Cascading Style Sheets (CSS), while implementation is performed with JavaScript programming language, which allows its easy integration with different HTML5 video players. In summary, the proposed solution parses and displays XML subtitles in the video player. Targeted platforms, for the proposed solution, are all platforms that have support for different web browsers.

In this paper we propose an integrated framework of automatic bilingual subtitle generation for lecture videos, especially for MOOCs. The framework consists of Automatic Speech Recognition (ASR), Sentence Boundary Detection (SBD), and Machine Translation (MT). Then we quantitatively evaluate the auto-generated subtitles, the manually produced subtitles from scratch, and the auto-generated subtitles with manual modification in term of accuracy and time expenditure, in both original and target languages. The result shows that the auto-generated subtitles in the original language (English) are fairly accurate already. By using them as the draft, human subtitle producers can save 54% of the working time and simultaneously reduce the error rate by 54.3%, which is a significant improvement. However, the effectiveness of machine translated subtitles (English to Chinese) is limited. In the end, if the proposed framework is applied, the total working time in preparing bilingual subtitles can be shortened by approximately 1/3, with no decline in quality.

The paper describes the formal model of the information video blocks submitting and synchronization. Also, the article describes the technology of multi-language video content support implemented in polysubtitles video player format based on that model. The essence of this technology consists of displaying multiple subtitle streams in different languages. This approach allows us to provide detailed technical information to the staff that is servicing the digital substation (and other complex objects) and speaking different languages. The technology development of the player is based on the UML diagrams using. The functionality of this software allows playing video files of various formats with parallel display of multiple subtitle streams in a single window. There is a possibility of video playback in multiple windows with different subtitles and centralized management and fast search and switching of video blocks. During the development of this software there were reviewed multiple formats video files and subtitle files. The .mkv fail was parsed during the coding in C#, on the Windows Presentation Foundation platform. The information inside this container became available for playback and setting up. The result of the application of this technology is the simultaneous display of multiple subtitle streams in different languages. During the development of this desktop application the authors also designed an easy intuitive user-friendly interface. The design elements are standard Windows Presentation Foundation applications.

In this paper we propose a solution which highlights key sentences in lecture video transcripts based on acoustic analysis. The basic idea is that a good lecturer knows when and where to emphasize when giving a lecture and these emphases can be detected by acoustic features of the speech, such as energy, pitch, speaking rate, etc. The selected key sentences are then specially marked in the lecture subtitle and presented to MOOC learners. An evaluation of both example result analysis and user feedback shows that the proposed lecture highlights are relatively logical and accurate. This form of highlighted lecture transcript is also welcomed by our learners.

This paper proposes an interactive annotation technique for 360° videos that allows the use of traditional video editing techniques to add content to immersive videos. Using the case study of immersive journalism the main objective is to diminish the entry barrier for annotating 360° video pieces, by providing a different annotation paradigm and a set of tools for annotation. The spread of virtual reality systems and immersive content has been growing substantially due to technological progress and cost reductions in equipment and software. From all the technologies employed in virtual reality systems, 360° video is one that currently presents unique conditions to be widely used by various industries — especially for communication purposes. From the various areas that can benefit from the usage of virtual reality systems, the communication field is one that requires innovation in the way that narratives are built, especially in virtual reality systems. In the case of immersive journalism, 360° video technology is currently one of the most used mediums by several media outlets. This kind of news content, whose innovative role should be highlighted, is still being studied in the field of journalism, needing a clearly defined set of rules and good practises. In order to improve the introduction of virtual elements in the 360° videos this paper proposes a set of annotation paradigms for 1) Media information display and 2) Narrative and attention focusing. In this paper we present a list of possible techniques that solve the problem of immersive annotation, as well as a description of a prototype that was developed to test these concepts. The prototype implements an annotation technique based on masked videos and the extension of standard subtitle file formats. Finally a fast-track user study was developed to evaluate the acceptance of the visualisation techniques and to refine the set of tools.

Language modeling plays a critical role in many natural language processing (NLP) tasks such as text prediction, machine translation and speech recognition. Traditional statistical language models (e.g. n-gram models) can only offer words that have been seen before and can not capture long word context. Neural language model provides a promising solution to surpass this shortcoming of statistical language model. This paper investigates Recurrent Neural Networks (RNNs) language model for Vietnamese, at character and syllable-levels. Experiments were conducted on a large dataset of 24M syllables, constructed from 1,500 movie subtitles. The experimental results show that our RNN-based language models yield reasonable performance on the movie subtitle dataset. Concretely, our models outperform n-gram language models in term of perplexity score.

In this paper we propose a novel video retrieval approach based on relevant topics extraction from video subtitles using natural language processing strategies and movie temporal segmentation into scenes. The proposed method is able to identify various subjects existent in a subtitle document and deals with the polysemantic character of words. The approach has been tested on various movie genres, including documentaries, TV series, news, sports. The objective evaluation carried out on a video dataset selected from one week video archive of France Television proves the performance of our technique that returns a mean average precision score superior to 0.5.

In order to study the impact of 3D subtitle characteristics on visual comfort, this paper included two experiments. In experiment 1, the study was on the impact of 3D subtitle fonts on visual comfort. The results showed the 3D subtitle fonts could not reduce the visual comfort. The subtitles with Official Script font, NSimSum font and bold font had no significant differences on visual comfort. And In experiment 2, the study was on the effect of font size on visual comfort. The results showed the font size had a little impact on visual comfort. The font size 12 of the subtitles was relatively comfortable.

With the ever-growing availability of multimedia content produced, broadcast and consumed worldwide, subtitling is becoming an essential service to quickly share understandable content. Simultaneously, the increased resolution of the ultra high definition (UHD) standard comes with wider screens and new viewing conditions. Services as the display of subtitles thus require adaptation to better fit the new induced viewing visual angle. This paper aims at evaluating quality of experience of subtitled movies in UHD to propose guidelines for the appearance of subtitles. From an eye-tracking experiment conducted on 68 observers and 30 video sequences, viewing behavior and visual saliency are analyzed with and without subtitles and for different subtitle styles. Various metrics based on eye-tracking data, such as the Reading Index for Dynamic Texts (RIDT), are computed to objectively measure the ease of reading and subtitle disturbance. The results mainly show that doubling the visual angle of subtitles from HD to UHD guarantees subtitle readability without compromising the enjoyment of the video content.

Selective attention is the idea that visual attention is not granted equally to all regions of the visual field, but rather focuses on high-attention regions. In this paper, we propose that selectively improving the performance of video in regions that are given high attention by users can significantly improve the quality of experience (QoE) of video services. We investigate users' attention regions, and choose the subtitle region as a representative high-attention region to present a subtitle-based measurement study. This study concentrates on how and how much the subtitle region affects the QoE of video. First of all, we verify that changes in performance of the subtitle region can cause an illusion about users' judgment on video bitrate. Secondly, we present a logarithmic relationship between video bitrate and the QoE of video. Thirdly, we measure the improved QoE by improving the performance of the subtitle region. Finally, we demonstrate that selectively improving the performance of the subtitle region can significantly improve the QoE of video at little cost.