Search

To guide a learner to master the action skills, it is crucial for a coach to 1) reason through the learner's action execution and technical keypoints, and 2) provide detailed, understandable feedback on what is done well and what can be improved. However, existing score-based action assessment methods are still far from this practical scenario. To bridge this gap, we investigate a new task termed Descriptive Action Coaching (DAC) which requires a model to provide detailed commentary on what is done well and what can be improved beyond a quality score from an action execution. To this end, we construct a new dataset named EE4D-DAC. With an LLM-based annotation pipeline, our dataset goes beyond the existing action assessment datasets by providing the hierarchical coaching commentary at both keypoint and instance levels. Furthermore, we propose TechCoach, a new framework that explicitly incorporates keypoint-level reasoning into the DAC process. The central to our method lies in the Context-aware Keypoint Reasoner, which enables TechCoach to learn keypoint-related quality representations by querying visual context under the supervision of keypoint-level coaching commentary. Prompted by the visual context and the keypoint-related quality representations, a unified Keypoint-aware Action Assessor is then employed to provide the overall coaching commentary together with the quality score. Combining all of these, we build a new benchmark for DAC and evaluate the effectiveness of our method through extensive experiments. Data and code will be publicly available., Comment: 19 pages, 12 figures

We introduce ajdmom, a Python package designed for automatically deriving moment formulas for the well-established affine jump diffusion (AJD) processes. ajdmom can produce explicit closed-form expressions for moments or conditional moments of any order, significantly enhancing the usability of AJD models. Additionally, ajdmom can compute partial derivatives of these moments with respect to the model parameters, offering a valuable tool for sensitivity analysis. The package's modular architecture makes it easy for adaptation and extension by researchers. ajdmom is open-source and readily available for installation from GitHub or the Python package index (PyPI).

Visual-textual correlations in the attention maps derived from text-to-image diffusion models are proven beneficial to dense visual prediction tasks, e.g., semantic segmentation. However, a significant challenge arises due to the input distributional discrepancy between the context-rich sentences used for image generation and the isolated class names typically employed in semantic segmentation, hindering the diffusion models from capturing accurate visual-textual correlations. To solve this, we propose InvSeg, a test-time prompt inversion method that tackles open-vocabulary semantic segmentation by inverting image-specific visual context into text prompt embedding space, leveraging structure information derived from the diffusion model's reconstruction process to enrich text prompts so as to associate each class with a structure-consistent mask. Specifically, we introduce Contrastive Soft Clustering (CSC) to align derived masks with the image's structure information, softly selecting anchors for each class and calculating weighted distances to push inner-class pixels closer while separating inter-class pixels, thereby ensuring mask distinction and internal consistency. By incorporating sample-specific context, InvSeg learns context-rich text prompts in embedding space and achieves accurate semantic alignment across modalities. Experiments show that InvSeg achieves state-of-the-art performance on the PASCAL VOC and Context datasets. Project page: https://jylin8100.github.io/InvSegProject/.

Promptable segmentation typically requires instance-specific manual prompts to guide the segmentation of each desired object. To minimize such a need, task-generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to reason detailed instance-specific prompts from a task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucinations during reasoning, resulting in inaccurate prompting. While existing methods focus on eliminating hallucinations to improve a model, we argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we utilize hallucinations to mine task-related information from images and verify its accuracy for enhancing precision of the generated prompts. Specifically, we introduce an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator.The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image.These hallucinations are then reduced to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code given in https://lwpyh.github.io/ProMaC/., Comment: NeurIPS 2024

We develop moment estimators for the parameters of affine stochastic volatility models. We first address the challenge of calculating moments for the models by introducing a recursive equation for deriving closed-form expressions for moments of any order. Consequently, we propose our moment estimators. We then establish a central limit theorem for our estimators and derive the explicit formulas for the asymptotic covariance matrix. Finally, we provide numerical results to validate our method.

Video grounding is a fundamental problem in multimodal content understanding, aiming to localize specific natural language queries in an untrimmed video. However, current video grounding datasets merely focus on simple events and are either limited to shorter videos or brief sentences, which hinders the model from evolving toward stronger multimodal understanding capabilities. To address these limitations, we present a large-scale video grounding dataset named SynopGround, in which more than 2800 hours of videos are sourced from popular TV dramas and are paired with accurately localized human-written synopses. Each paragraph in the synopsis serves as a language query and is manually annotated with precise temporal boundaries in the long video. These paragraph queries are tightly correlated to each other and contain a wealth of abstract expressions summarizing video storylines and specific descriptions portraying event details, which enables the model to learn multimodal perception on more intricate concepts over longer context dependencies. Based on the dataset, we further introduce a more complex setting of video grounding dubbed Multi-Paragraph Video Grounding (MPVG), which takes as input multiple paragraphs and a long video for grounding each paragraph query to its temporal interval. In addition, we propose a novel Local-Global Multimodal Reasoner (LGMR) to explicitly model the local-global structures of long-term multimodal inputs for MPVG. Our method provides an effective baseline solution to the multi-paragraph video grounding problem. Extensive experiments verify the proposed model's effectiveness as well as its superiority in long-term multi-paragraph video grounding over prior state-of-the-arts. Dataset and code are publicly available. Project page: https://synopground.github.io/., Comment: Accepted to ACM MM 2024. Project page: https://synopground.github.io/

Human trajectory prediction is a practical task of predicting the future positions of pedestrians on the road, which typically covers all temporal ranges from short-term to long-term within a trajectory. However, existing works attempt to address the entire trajectory prediction with a singular, uniform training paradigm, neglecting the distinction between short-term and long-term dynamics in human trajectories. To overcome this limitation, we introduce a novel Progressive Pretext Task learning (PPT) framework, which progressively enhances the model's capacity of capturing short-term dynamics and long-term dependencies for the final entire trajectory prediction. Specifically, we elaborately design three stages of training tasks in the PPT framework. In the first stage, the model learns to comprehend the short-term dynamics through a stepwise next-position prediction task. In the second stage, the model is further enhanced to understand long-term dependencies through a destination prediction task. In the final stage, the model aims to address the entire future trajectory task by taking full advantage of the knowledge from previous stages. To alleviate the knowledge forgetting, we further apply a cross-task knowledge distillation. Additionally, we design a Transformer-based trajectory predictor, which is able to achieve highly efficient two-step reasoning by integrating a destination-driven prediction strategy and a group of learnable prompt embeddings. Extensive experiments on popular benchmarks have demonstrated that our proposed approach achieves state-of-the-art performance with high efficiency. Code is available at https://github.com/iSEE-Laboratory/PPT., Comment: Accepted to ECCV 2024

Superluminous supernovae (SLSNe) are a class of intense celestial events that can be standardized for measuring cosmological parameters, bridging the gap between type Ia supernovae and the cosmic microwave background. In this work, we discuss the cosmological applications of SLSNe from the Chinese Space Station Telescope (CSST). Our estimation suggests that SLSNe rate is biased tracing the cosmic star formation rate, exhibiting a factor of $(1+z)^{1.2}$. We futher predict that CSST is poised to observe $\sim 360$ SLSNe in the 10 square degrees ultra-deep field survey within a span of 2.5 years. A stringent constraint on cosmological parameters can be derived from their peak-color relationship. CSST is anticipated to uncover a substantial number of SLSNe, contributing to a deeper understanding of their central engines and shedding light on the nature of dark energy at high redshifts., Comment: 12 pages, 7 figures, accepted for publication in Science China Physics, Mechanics & Astronomy

Cosmography can be used to constrain the kinematics of universe in a model-independent way. In this work, we attempted to combine the Pad$\rm \acute{e}$ approximations with the latest Pantheon+ sample for testing cosmological principle. Based on the Pad$\rm \acute{e}$ approximations, we first gave the cosmographic constraints on the different order polynomials including third-order (Pad$\rm \acute{e}$$_{(2,1)}$), fourth-order (Pad$\rm \acute{e}$$_{(2,2)}$) and fifth-order (Pad$\rm \acute{e}$$_{(3,2)}$). Based on the Pad$\rm \acute{e}$$_{(2,1)}$ ($j_{0}$ = 1) polynomial and hemisphere comparison (HC) method, we tested the cosmological principle and found the preferred directions of cosmic anisotropy, such as (l, b) = (304.6$^{\circ}$$_{-37.4}^{+51.4}$, $-$18.7$^{\circ}$$_{-20.3}^{+14.7}$) and (311.1$^{\circ}$$_{-8.4}^{+17.4}$, $-$17.53$^{\circ}$$_{-7.7}^{+7.8}$) for $q_{0}$ and $H_{0}$, respectively. These two directions are consistent with each other in $1\sigma$ confidence level, but the corresponding results of statistical isotropy analyses including Isotropy and Isotropy with real positions (RP) are quite different. The statistical significance of $H_{0}$ are stronger than that of $q_{0}$, i.e., 4.75$\sigma$ and 4.39$\sigma$ for the Isotropy and Isotropy with RP respectively. Reanalysis with fixed $q_{0} = -0.55$ (corresponds to $\Omega_{m}$ = 0.30) gives similar results. Overall, our model-independent results provide clear indications for a possible cosmic anisotropy, which must be taken seriously. Further test is needed to better understand this signal., Comment: 11 pages, 12 figures, 3 tables. Accepted for publication in A&A, the abstract appearing here is slightly shorter than that in the PDF file

As one of the most energetic and brightest events, gamma-ray bursts (GRBs) have been used as a standard candle for cosmological probe. Based on the relevant features of GRBs light curves, a plateau phase followed a decay phase, we obtain X-ray samples of 31 GRBs and optical samples of 50 GRBs, which are thought to be caused by the same physical mechanism. We standardize GRBs using the two-dimension fundamental plane relation of the rest-frame luminosity of the plateau emission ($L_{b,z}$) and the end time of plateau ($T_{b,z}$) $L_{b,z}-T_{b,z}$, as well as the three-dimension fundamental plane correlation including the peak energy ($E_{p,i}$) $L_{b,z}-T_{b,z}-E_{p,i}$. For the cosmological probes, we consider the $\omega$CDM model in which the dark energy consists of one component, and mainly focus on the $X_1X_2$CDM model in which the dark energy is made up of two independent components. We obtain the constraints on the related parameters of the cosmological models using the type Ia supernovae (SNe Ia) data and selected X-ray and optical samples. For the $X_1X_2$CDM model, we find that the values of the equations of state parameters of two dark energies, $\omega_1$ and $\omega_2$, are very close. We also conduct the comparison between the models using the Bayesian information criterion, and find that the $\omega$CDM model is favoured., Comment: 13 pages, 8 figures and 3 tables, accepted for publication in Astronomy & Astrophysics

As large language models (LLMs) continue to grow by scaling laws, reinforcement learning from human feedback (RLHF) has gained significant attention due to its outstanding performance. However, unlike pretraining or fine-tuning a single model, scaling reinforcement learning from human feedback (RLHF) for training large language models poses coordination challenges across four models. We present OpenRLHF, an open-source framework enabling efficient RLHF scaling. Unlike existing RLHF frameworks that co-locate four models on the same GPUs, OpenRLHF re-designs scheduling for the models beyond 70B parameters using Ray, vLLM, and DeepSpeed, leveraging improved resource utilization and diverse training approaches. Integrating seamlessly with Hugging Face, OpenRLHF provides an out-of-the-box solution with optimized algorithms and launch scripts, which ensures user-friendliness. OpenRLHF implements RLHF, DPO, rejection sampling, and other alignment techniques. Empowering state-of-the-art LLM development, OpenRLHF's code is available at \url{https://github.com/OpenRLHF/OpenRLHF}.

This paper focuses on open-ended video question answering, which aims to find the correct answers from a large answer set in response to a video-related question. This is essentially a multi-label classification task, since a question may have multiple answers. However, due to annotation costs, the labels in existing benchmarks are always extremely insufficient, typically one answer per question. As a result, existing works tend to directly treat all the unlabeled answers as negative labels, leading to limited ability for generalization. In this work, we introduce a simple yet effective ranking distillation framework (RADI) to mitigate this problem without additional manual annotation. RADI employs a teacher model trained with incomplete labels to generate rankings for potential answers, which contain rich knowledge about label priority as well as label-associated visual cues, thereby enriching the insufficient labeling information. To avoid overconfidence in the imperfect teacher model, we further present two robust and parameter-free ranking distillation approaches: a pairwise approach which introduces adaptive soft margins to dynamically refine the optimization constraints on various pairwise rankings, and a listwise approach which adopts sampling-based partial listwise learning to resist the bias in teacher ranking. Extensive experiments on five popular benchmarks consistently show that both our pairwise and listwise RADIs outperform state-of-the-art methods. Further analysis demonstrates the effectiveness of our methods on the insufficient labeling problem., Comment: Accepted to CVPR 2024

Video Paragraph Grounding (VPG) is an emerging task in video-language understanding, which aims at localizing multiple sentences with semantic relations and temporal order from an untrimmed video. However, existing VPG approaches are heavily reliant on a considerable number of temporal labels that are laborious and time-consuming to acquire. In this work, we introduce and explore Weakly-Supervised Video Paragraph Grounding (WSVPG) to eliminate the need of temporal annotations. Different from previous weakly-supervised grounding frameworks based on multiple instance learning or reconstruction learning for two-stage candidate ranking, we propose a novel siamese learning framework that jointly learns the cross-modal feature alignment and temporal coordinate regression without timestamp labels to achieve concise one-stage localization for WSVPG. Specifically, we devise a Siamese Grounding TRansformer (SiamGTR) consisting of two weight-sharing branches for learning complementary supervision. An Augmentation Branch is utilized for directly regressing the temporal boundaries of a complete paragraph within a pseudo video, and an Inference Branch is designed to capture the order-guided feature correspondence for localizing multiple sentences in a normal video. We demonstrate by extensive experiments that our paradigm has superior practicability and flexibility to achieve efficient weakly-supervised or semi-supervised learning, outperforming state-of-the-art methods trained with the same or stronger supervision., Comment: Accepted to CVPR 2024. v2: fix a typo in figure 1

Universal image restoration is a practical and potential computer vision task for real-world applications. The main challenge of this task is handling the different degradation distributions at once. Existing methods mainly utilize task-specific conditions (e.g., prompt) to guide the model to learn different distributions separately, named multi-partite mapping. However, it is not suitable for universal model learning as it ignores the shared information between different tasks. In this work, we propose an advanced selective hourglass mapping strategy based on diffusion model, termed DiffUIR. Two novel considerations make our DiffUIR non-trivial. Firstly, we equip the model with strong condition guidance to obtain accurate generation direction of diffusion model (selective). More importantly, DiffUIR integrates a flexible shared distribution term (SDT) into the diffusion algorithm elegantly and naturally, which gradually maps different distributions into a shared one. In the reverse process, combined with SDT and strong condition guidance, DiffUIR iteratively guides the shared distribution to the task-specific distribution with high image quality (hourglass). Without bells and whistles, by only modifying the mapping strategy, we achieve state-of-the-art performance on five image restoration tasks, 22 benchmarks in the universal setting and zero-shot generalization setting. Surprisingly, by only using a lightweight model (only 0.89M), we could achieve outstanding performance. The source code and pre-trained models are available at https://github.com/iSEE-Laboratory/DiffUIR, Comment: Accepted to CVPR2024

Stochastic optimization is often hampered by distributional ambiguity, where critical probability distributions are poorly characterized or unknown. Addressing this challenge, we introduce a new framework that targets the minimization of a statistical upper bound for the expected value of uncertain objectives, facilitating more statistically robust decision-making. Central to our approach is the Average Percentile Upper Bound (APUB), a novel construct that simultaneously delivers a statistically rigorous upper bound for the population mean and a meaningful risk metric for the sample mean. The integration of APUB into stochastic optimization not only fortifies the process against distributional ambiguity but also reinforces key data-driven decision-making attributes, such as reliability, consistency, and comprehensibility. Notably, APUB-enriched optimization problems feature tractability, with particular advantages in two-stage stochastic optimization with random recourse. Empirical demonstrations on two-stage product mix and multi-product newsvendor benchmark problems reveal the benefit of the APUB optimization framework, in comparison with conventional techniques such as sample average approximation and distributionally robust optimization.

Traditional rigid endoscopes have challenges in flexibly treating tumors located deep in the brain, and low operability and fixed viewing angles limit its development. This study introduces a novel dual-segment flexible robotic endoscope MicroNeuro, designed to perform biopsies with dexterous surgical manipulation deep in the brain. Taking into account the uncertainty of the control model, an image-based visual servoing with online robot Jacobian estimation has been implemented to enhance motion accuracy. Furthermore, the application of model predictive control with constraints significantly bolsters the flexible robot's ability to adaptively track mobile objects and resist external interference. Experimental results underscore that the proposed control system enhances motion stability and precision. Phantom testing substantiates its considerable potential for deployment in neurosurgery., Comment: Accepted by IEEE International Conference on Robotics and Automation (ICRA) 2024, 7 pages, 9 figures