Search

We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini's MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models., Comment: Project Page: https://mardini-vidgen.github.io

As an intangible cultural heritage, Chinese shadow puppetry is facing challenges in terms of its appeal and comprehension, especially among audiences from different cultural backgrounds. Additionally, the fragile materials of the puppets and obstacles to preservation pose further challenges. This study creates a digital archive of the Qinhuai River Lantern Festival shadow puppetry, utilizing digital technology to recreate scenes depicted in traditional Chinese poetry and painting. Moreover, this study employs a mixed-method approach, combining qualitative and quantitative methods, to evaluate the acceptance and audience experience of immersive shadow puppetry. An in-depth exploration was conducted from sensory, emotional, cultural dimensions and research hypotheses were tested using structural equation modeling and other methods. The results indicate that enhancing ease of use and cultural experience can improve audience appeal and comprehension, while enhancing emotional experience can increase audience participation intention. Our research holds profound significance for the preservation and transmission of shadow puppetry.

It is proposed a model that makes use of the Entropy concept and the Jensen-Shannon information divergence function to simulate computationally the dissemination of opinions on a Barab\'asi-Albert (BA) scale-free network. The simulation considers individual memory, ideological proximity between individuals, and distortion influenced by polarization., Comment: 21 pages, 15 figures

This paper studies the convergence properties of a family of Relaxed $\ell$-Minimal Gradient Descent methods for quadratic optimization; the family includes the omnipresent Steepest Descent method, as well as the Minimal Gradient method. Simple proofs are provided that show, in an appropriately chosen norm, the gradient and the distance of the iterates from optimality converge linearly, for all members of the family. Moreover, the function values decrease linearly, and iteration complexity results are provided. All theoretical results hold when (fixed) relaxation is employed. It is also shown that, given a fixed overhead and storage budget, every Relaxed $\ell$-Minimal Gradient Descent method can be implemented using exactly one matrix vector product. Numerical experiments are presented that illustrate the benefits of relaxation across the family., Comment: 23 pages, 6 figures, 2 tables

We explore the role of attention mechanism during inference in text-conditional diffusion models. Empirical observations suggest that cross-attention outputs converge to a fixed point after several inference steps. The convergence time naturally divides the entire inference process into two phases: an initial phase for planning text-oriented visual semantics, which are then translated into images in a subsequent fidelity-improving phase. Cross-attention is essential in the initial phase but almost irrelevant thereafter. However, self-attention initially plays a minor role but becomes crucial in the second phase. These findings yield a simple and training-free method known as temporally gating the attention (TGATE), which efficiently generates images by caching and reusing attention outputs at scheduled time steps. Experimental results show when widely applied to various existing text-conditional diffusion models, TGATE accelerates these models by 10%-50%. The code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.

Solving image-to-3D from a single view is an ill-posed problem, and current neural reconstruction methods addressing it through diffusion models still rely on scene-specific optimization, constraining their generalization capability. To overcome the limitations of existing approaches regarding generalization and consistency, we introduce a novel neural rendering technique. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Specifically, our method builds neural encoding volumes from generated multi-view inputs. We adjust the weights of the SDF network conditioned on an input image at test-time to allow model adaptation to novel scenes in a feed-forward manner via HyperNetworks. To mitigate artifacts derived from the synthesized views, we propose the use of a volume transformer module to improve the aggregation of image features instead of processing each viewpoint separately. Through our proposed method, dubbed as Hyper-VolTran, we avoid the bottleneck of scene-specific optimization and maintain consistency across the images generated from multiple viewpoints. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.

In this study, we explore Transformer-based diffusion models for image and video generation. Despite the dominance of Transformer architectures in various fields due to their flexibility and scalability, the visual generative domain primarily utilizes CNN-based U-Net architectures, particularly in diffusion-based models. We introduce GenTron, a family of Generative models employing Transformer-based diffusion, to address this gap. Our initial step was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a process involving thorough empirical exploration of the conditioning mechanism. We then scale GenTron from approximately 900M to over 3B parameters, observing significant improvements in visual quality. Furthermore, we extend GenTron to text-to-video generation, incorporating novel motion-free guidance to enhance video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench, underscoring its strengths in compositional generation. We believe this work will provide meaningful insights and serve as a valuable reference for future research., Comment: CVPR2024 Camera Ready. Website: https://www.shoufachen.com/gentron_website/

Large Language Models (LLMs), as the case of OpenAI ChatGPT-4 Turbo, are revolutionizing several industries, including higher education. In this context, LLMs can be personalized through a fine-tuning process to meet the student demands on every particular subject, like statistics. Recently, OpenAI has launched the possibility to fine-tune their model with a natural language web interface, enabling the possibility to create customized GPT version deliberately conditioned to meet the demands of a specific task. The objective of this research is to assess the potential of the customized GPTs that have recently been launched by OpenAI. After developing a Business Statistics Virtual Professor (BSVP), tailored for students at the Universidad Pontificia Comillas, its behavior was evaluated and compared with that of ChatGPT-4 Turbo. The results lead to several conclusions. Firstly, a substantial modification in the style of communication was observed. Following the instructions it was trained with, BSVP provided responses in a more relatable and friendly tone, even incorporating a few minor jokes. Secondly, and this is a matter of relevance, when explicitly asked for something like, "I would like to practice a programming exercise similar to those in R practice 4," BSVP was capable of providing a far superior response: having access to contextual documentation, it could fulfill the request, something beyond ChatGPT-4 Turbo's capabilities. On the downside, the response times were generally higher. Lastly, regarding overall performance, quality, depth, and alignment with the specific content of the course, no statistically significant differences were observed in the responses between BSVP and ChatGPT-4 Turbo. It appears that customized assistants trained with prompts present advantages as virtual aids for students, yet they do not constitute a substantial improvement over ChatGPT-4 Turbo., Comment: 9 pages, 1 figure, 1 table

While measures, such as traffic calming and advance driver assistance systems, can improve safety for Vulnerable Road Users (VRUs), their effectiveness ultimately relies on the responsible behavior of drivers and pedestrians who must adhere to traffic rules or take appropriate actions. However, these measures offer no solution in scenarios where a collision becomes imminent, leaving no time for warning or corrective actions. Recently, connected vehicle technology has introduced warning services that can alert drivers and VRUs about potential collisions. Nevertheless, there is still a significant gap in the system's ability to predict collisions in advance. The objective of this study is to utilize Digital Twin (DT) technology to enable a proactive safety alert system for VRUs. A pedestrian-vehicle trajectory prediction model has been developed using the Encoder-Decoder Long Short-Term Memory (LSTM) architecture to predict future trajectories of pedestrians and vehicles. Subsequently, parallel evaluation of all potential future safety-critical scenarios is carried out. Three Encoder-Decoder LSTM models, namely pedestrian-LSTM, vehicle-through-LSTM, and vehicle-left-turn-LSTM, are trained and validated using field-collected data, achieving corresponding root mean square errors (RMSE) of 0.049, 1.175, and 0.355 meters, respectively. A real-world case study has been conducted where a pedestrian crosses a road, and vehicles have the option to proceed through or left-turn, to evaluate the efficacy of DT-enabled proactive safety alert systems. Experimental results confirm that DT-enabled safety alert systems were succesfully able to detect potential crashes and proactively generate safety alerts to reduce potential crash risk., Comment: 19 pages, 9 figures, submitted to the Transportation Research Board 2024 TRB Annual Meeting

Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts. A major challenge in this task is to ensure that all frames in the edited video are visually consistent. Most recent works apply advanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into spatio-temporal attention. Although temporal context can be added through spatio-temporal attention, it may introduce some irrelevant information for each patch and therefore cause inconsistency in the edited video. In this paper, for the first time, we introduce optical flow into the attention module in the diffusion model's U-Net to address the inconsistency issue for text-to-video editing. Our method, FLATTEN, enforces the patches on the same flow path across different frames to attend to each other in the attention module, thus improving the visual consistency in the edited videos. Additionally, our method is training-free and can be seamlessly integrated into any diffusion-based text-to-video editing methods and improve their visual consistency. Experiment results on existing text-to-video editing benchmarks show that our proposed method achieves the new state-of-the-art performance. In particular, our method excels in maintaining the visual consistency in the edited videos., Comment: Accepted by ICLR2024. Project page: https://flatten-video-editing.github.io/

Models of invasive species spread often assume that landscapes are spatially homogeneous; thus simplifying analysis but potentially reducing accuracy. We extend a recently developed partial differential equation model for invasive conifer spread to account for spatial heterogeneity in parameter values and introduce a method to obtain key outputs (e.g. spread rates) from computational simulations. Simulations produce patterns of spatial spread remarkably similar to observed patterns in grassland ecosystems invaded by exotic conifers, validating our spatially explicit strategy. We find that incorporating spatial variation in different parameters does not significantly affect the evolution of invasions (which are characterised by a long quiescent period followed by rapid evolution towards to a constant rate of invasion) but that distributional assumptions can have a significant impact on the spread rate of invasions. Our work demonstrates that spatial variation in site-suitability or other parameters can have a significant impact on invasions, Comment: 13 pages, 18 figures

Invasive pine trees pose a threat to biodiversity in a variety of Southern Hemisphere countries, but understanding of the dynamics of invasions and the factors that retard or accelerate spread is limited. Here, we consider the past models of wilding pine spread and develop a new model of pine invasion. We show that many prior models feature parameter estimates which are not biologically supported and rely on a conjecture to obtain an asymptotic spread speed of invasive pine populations, the main output of these models. In contrast to prior approaches, we use partial differential equations to model an invasion. We show that invasions are almost static for a significant period of time before rapidly accelerating to spread at a constant rate, matching observed behaviour in at least some field sites. Our work suggests that prior methods for estimating invasion speeds may not accurately predict spread and are sensitive to assumptions about the distribution of parameters. However, we present alternative estimation methods and suggest directions for further research., Comment: 36 pages, 9 figures

Studies on artificial neural networks rarely address both vanishing gradients and overfitting issues. In this study, we follow the pupil learning procedure, which has the features of interpreting, picking, understanding, cramming, and organizing, to derive the pupil learning mechanism (PLM) by which to modify the network structure and weights of 2-layer neural networks (2LNNs). The PLM consists of modules for sequential learning, adaptive learning, perfect learning, and less-overfitted learning. Based upon a copper price forecasting dataset, we conduct an experiment to validate the PLM module design modules, and an experiment to evaluate the performance of PLM. The empirical results indeed approve the PLM module design and show the superiority of the proposed PLM model over the linear regression model and the conventional backpropagation-based 2LNN model.

Video activity localization aims at understanding the semantic content in long untrimmed videos and retrieving actions of interest. The retrieved action with its start and end locations can be used for highlight generation, temporal action detection, etc. Unfortunately, learning the exact boundary location of activities is highly challenging because temporal activities are continuous in time, and there are often no clear-cut transitions between actions. Moreover, the definition of the start and end of events is subjective, which may confuse the model. To alleviate the boundary ambiguity, we propose to study the video activity localization problem from a denoising perspective. Specifically, we propose an encoder-decoder model named DenoiseLoc. During training, a set of action spans is randomly generated from the ground truth with a controlled noise scale. Then we attempt to reverse this process by boundary denoising, allowing the localizer to predict activities with precise boundaries and resulting in faster convergence speed. Experiments show that DenoiseLoc advances %in several video activity understanding tasks. For example, we observe a gain of +12.36% average mAP on QV-Highlights dataset and +1.64% mAP@0.5 on THUMOS'14 dataset over the baseline. Moreover, DenoiseLoc achieves state-of-the-art performance on TACoS and MAD datasets, but with much fewer predictions compared to other current methods.

Blockchain technology and, more generally, distributed ledger technology (DLT) systems, face public scrutiny for their energy consumption levels. However, many point out that high energy consumption is a feature of (small block size) proof-of-work (PoW) DLTs, but not of proof-of-stake (PoS) DLTs. With the energy consumption of PoS systems being an under-researched area, we replicate, expand and update embryonary work modelling it and comparing different PoS-based DLTs with each other and with other non-PoS systems. In doing so, we suggest and implement a number of improvements to an existing PoS energy consumption model. We find that there may be significant differences in the energy consumption of PoS systems analysed and confirm that, regardless of these differences, their energy consumption is several orders of magnitude below that of Bitcoin Core., Comment: 27 pages, 3 figures, working paper

Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection (TAD) to new classes. The former adapts a pretrained vision model to a new task represented by as few as a single video per class, whilst the latter requires no training examples by exploiting a semantic description of the new class. In this work, we introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD by leveraging few-shot support videos and new class names jointly. To tackle this problem, we further introduce a novel MUlti-modality PromPt mETa-learning (MUPPET) method. This is enabled by efficiently bridging pretrained vision and language models whilst maximally reusing already learned capacity. Concretely, we construct multi-modal prompts by mapping support videos into the textual token space of a vision-language model using a meta-learned adapter-equipped visual semantics tokenizer. To tackle large intra-class variation, we further design a query feature regulation scheme. Extensive experiments on ActivityNetv1.3 and THUMOS14 demonstrate that our MUPPET outperforms state-of-the-art alternative methods, often by a large margin. We also show that our MUPPET can be easily extended to tackle the few-shot object detection problem and again achieves the state-of-the-art performance on MS-COCO dataset. The code will be available in https://github.com/sauradip/MUPPET, Comment: Technical Report

This paper deals with the problem of localizing objects in image and video datasets from visual exemplars. In particular, we focus on the challenging problem of egocentric visual query localization. We first identify grave implicit biases in current query-conditioned model design and visual query datasets. Then, we directly tackle such biases at both frame and object set levels. Concretely, our method solves these issues by expanding limited annotations and dynamically dropping object proposals during training. Additionally, we propose a novel transformer-based module that allows for object-proposal set context to be considered while incorporating query information. We name our module Conditioned Contextual Transformer or CocoFormer. Our experiments show the proposed adaptations improve egocentric query detection, leading to a better visual query localization system in both 2D and 3D configurations. Thus, we are able to improve frame-level detection performance from 26.28% to 31.26 in AP, which correspondingly improves the VQ2D and VQ3D localization scores by significant margins. Our improved context-aware query object detector ranked first and second in the VQ2D and VQ3D tasks in the 2nd Ego4D challenge. In addition to this, we showcase the relevance of our proposed model in the Few-Shot Detection (FSD) task, where we also achieve SOTA results. Our code is available at https://github.com/facebookresearch/vq2d_cvpr., Comment: We ranked first and second in the VQ2D and VQ3D tasks in the 2nd Ego4D challenge

The objective of this paper is to develop simple techniques to enhance the performance of multi-user RIS aided wireless systems. Specifically, we develop a novel technique called \textit{channel separation} which provides a better understanding of how the RIS phases affect the uplink sum rate and sum rates for ZF and MMSE linear receivers. Leveraging channel separation, we propose a simple iterative algorithm to improve the uplink sum rate and the sum rates of ZF and MMSE linear receivers when discrete RIS phases are considered. For continuous RIS phases, we derive simple closed form solutions to enhance the uplink sum rate and reduce the total mean square error of the MMSE combiner. The latter metric is a tractable alternative to maximizing sum rates for ZF and MMSE. Numerical simulations are performed for all optimization techniques and the effectiveness of each technique is compared to a full numerical optimization procedure, namely an interior point (IP) algorithm., Comment: arXiv admin note: text overlap with arXiv:2110.02493

The recently released Ego4D dataset and benchmark significantly scales and diversifies the first-person visual perception data. In Ego4D, the Visual Queries 2D Localization task aims to retrieve objects appeared in the past from the recording in the first-person view. This task requires a system to spatially and temporally localize the most recent appearance of a given object query, where query is registered by a single tight visual crop of the object in a different scene. Our study is based on the three-stage baseline introduced in the Episodic Memory benchmark. The baseline solves the problem by detection and tracking: detect the similar objects in all the frames, then run a tracker from the most confident detection result. In the VQ2D challenge, we identified two limitations of the current baseline. (1) The training configuration has redundant computation. Although the training set has millions of instances, most of them are repetitive and the number of unique object is only around 14.6k. The repeated gradient computation of the same object lead to an inefficient training; (2) The false positive rate is high on background frames. This is due to the distribution gap between training and evaluation. During training, the model is only able to see the clean, stable, and labeled frames, but the egocentric videos also have noisy, blurry, or unlabeled background frames. To this end, we developed a more efficient and effective solution. Concretely, we bring the training loop from ~15 days to less than 24 hours, and we achieve 0.17% spatial-temporal AP, which is 31% higher than the baseline. Our solution got the first ranking on the public leaderboard. Our code is publicly available at https://github.com/facebookresearch/vq2d_cvpr., Comment: First place winning solution for VQ2D task in CVPR-2022 Ego4D Challenge. Our code is publicly available at https://github.com/facebookresearch/vq2d_cvpr