Search

In a recent paper, Phys. Rev. Lett. {\bf 126}, 132001 (2021), the LHCb data on the di-$J/\psi$ production in proton-proton collisions were analysed in a coupled-channel framework based on double-vector-charmonium channels. This investigation identified a robust pole near the $J/\psi J/\psi$ threshold, tagged $X(6200)$, suggesting it as a new state. The present work extends that investigation by incorporating recent di-$J/\psi$ production data from the CMS and ATLAS collaborations and performing a combined analysis of all three data sets. This study confirms the existence of the $X(6200)$, and its pole position is now determined with a higher precision than in the previous study, where only a single data set was employed. The pole corresponding to the $X(6900)$ is also extracted, though its actual position and residue depend on a particular coupled-channel model employed and deviate from values reported in the experimental investigations. The dielectron width of the $X(6200)$ is estimated under different conjectures on its quantum numbers, and the number of such states that can be annually produced at the future Super $\tau$-Charm Facility is estimated. The line shape in the complimentary $J/\psi\psi(2S)$ channel is discussed, and a good agreement with the ATLAS data is found., Comment: 9 pages, 5 figures, 7 tables

We study Defant and Kravitz's generalization of Sch\"utzenberger's promotion operator to arbitrary labelings of finite posets in two directions. Defant and Kravitz showed that applying the promotion operator $n-1$ times to a labeling of a poset on $n$ elements always gives a natural labeling of the poset and called a labeling tangled if it requires the full $n-1$ promotions to reach a natural labeling. They also conjectured that there are at most $(n-1)!$ tangled labelings for any poset on $n$ elements. In the first direction, we propose a further strengthening of their conjecture by partitioning tangled labelings according to the element labeled $n-1$ and prove that this stronger conjecture holds for inflated rooted forest posets and a new class of posets called shoelace posets. In the second direction, we introduce sorting generating functions and cumulative generating functions for the number of labelings that require $k$ applications of the promotion operator to give a natural labeling. We prove that the coefficients of the cumulative generating function of the ordinal sum of antichains are log-concave and obtain a refinement of the weak order on the symmetric group., Comment: 24 pages, 11 figures

The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to particular domains. This has enabled the creation of powerful and adaptive routing-based "Model MoErging" methods with the goal of using expert modules to create an aggregate system with improved performance or generalization. However, existing MoErging methods often prioritize generalization to unseen tasks at the expense of performance on held-in tasks, which limits its practical applicability in real-world deployment scenarios. We observe that current token-level routing mechanisms neglect the global semantic context of the input task. This token-wise independence hinders effective expert selection for held-in tasks, as routing decisions fail to incorporate the semantic properties of the task. To address this, we propose, Global and Local Instruction Driven Expert Router (GLIDER) that integrates a multi-scale routing mechanism, encompassing a semantic global router and a learned local router. The global router leverages LLM's advanced reasoning capabilities for semantic-related contexts to enhance expert selection. Given the input query and LLM, the router generates semantic task instructions that guide the retrieval of the most relevant experts across all layers. This global guidance is complemented by a local router that facilitates token-level routing decisions within each module, enabling finer control and enhanced performance on unseen tasks. Our experiments using T5-based models for T0 and FLAN tasks demonstrate that GLIDER achieves substantially improved held-in performance while maintaining strong generalization on held-out tasks. We also perform ablations experiments to dive deeper into the components of GLIDER. Our experiments highlight the importance of our multi-scale routing that leverages LLM-driven semantic reasoning for MoErging methods., Comment: Our code is available at https://github.com/UNITES-Lab/glider

For 6-DoF grasp detection, simulated data is expandable to train more powerful model, but it faces the challenge of the large gap between simulation and real world. Previous works bridge this gap with a sim-to-real way. However, this way explicitly or implicitly forces the simulated data to adapt to the noisy real data when training grasp detectors, where the positional drift and structural distortion within the camera noise will harm the grasp learning. In this work, we propose a Real-to-Sim framework for 6-DoF Grasp detection, named R2SGrasp, with the key insight of bridging this gap in a real-to-sim way, which directly bypasses the camera noise in grasp detector training through an inference-time real-to-sim adaption. To achieve this real-to-sim adaptation, our R2SGrasp designs the Real-to-Sim Data Repairer (R2SRepairer) to mitigate the camera noise of real depth maps in data-level, and the Real-to-Sim Feature Enhancer (R2SEnhancer) to enhance real features with precise simulated geometric primitives in feature-level. To endow our framework with the generalization ability, we construct a large-scale simulated dataset cost-efficiently to train our grasp detector, which includes 64,000 RGB-D images with 14.4 million grasp annotations. Sufficient experiments show that R2SGrasp is powerful and our real-to-sim perspective is effective. The real-world experiments further show great generalization ability of R2SGrasp. Project page is available on https://isee-laboratory.github.io/R2SGrasp.

Over the past three decades, multiple reverberation mapping (RM) campaigns conducted for the quasar PG 2130+099 have exhibited inconsistent findings with time delays ranging from $\sim$10 to $\sim$200 days. To achieve a comprehensive understanding of the geometry and dynamics of the broad-line region (BLR) in PG 2130+099, we continued an ongoing high-cadence RM monitoring campaign using the Calar Alto Observatory 2.2m optical telescope for an extra four years from 2019 to 2022. We measured the time lags of several broad emission lines (including He II, He I, H$\beta$, and Fe II) with respect to the 5100 {\AA} continuum, and their time lags continuously vary through the years. Especially, the H$\beta$ time lags exhibited approximately a factor of two increase in the last two years. Additionally, the velocity-resolved time delays of the broad H$\beta$ emission line reveal a back-and-forth change between signs of virial motion and inflow in the BLR. The combination of negligible ($\sim$10%) continuum change and substantial time-lag variation (over two times) results in significant scatter in the intrinsic $R_{\rm H\beta}-L_{\rm 5100}$ relationship for PG 2130+099. Taking into account the consistent changes in the continuum variability time scale and the size of the BLR, we tentatively propose that the changes in the measurement of the BLR size may be affected by 'geometric dilution'., Comment: 21 pages, 13 figures, 7 tables; accepted for publication in ApJ

Beyond maximum likelihood estimation (MLE), the standard objective of a language model (LM) that optimizes good examples probabilities, many studies have explored ways that also penalize bad examples for enhancing the quality of output distribution, including unlikelihood training, exponential maximizing average treatment effect (ExMATE), and direct preference optimization (DPO). To systematically compare these methods and further provide a unified recipe for LM optimization, in this paper, we present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. Through both mathematical results and experiments on CausalDialogue and Anthropic HH-RLHF datasets, we identify distinct functional characteristics among these methods. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.

Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks., Comment: 18 pages

Robotic grasping in clutters is a fundamental task in robotic manipulation. In this work, we propose an economic framework for 6-DoF grasp detection, aiming to economize the resource cost in training and meanwhile maintain effective grasp performance. To begin with, we discover that the dense supervision is the bottleneck of current SOTA methods that severely encumbers the entire training overload, meanwhile making the training difficult to converge. To solve the above problem, we first propose an economic supervision paradigm for efficient and effective grasping. This paradigm includes a well-designed supervision selection strategy, selecting key labels basically without ambiguity, and an economic pipeline to enable the training after selection. Furthermore, benefit from the economic supervision, we can focus on a specific grasp, and thus we devise a focal representation module, which comprises an interactive grasp head and a composite score estimation to generate the specific grasp more accurately. Combining all together, the EconomicGrasp framework is proposed. Our extensive experiments show that EconomicGrasp surpasses the SOTA grasp method by about 3AP on average, and with extremely low resource cost, for about 1/4 training time cost, 1/8 memory cost and 1/30 storage cost. Our code is available at https://github.com/iSEE-Laboratory/EconomicGrasp., Comment: 19 pages, 7 figures. Accepted in ECCV 2024!

We conduct a reverberation mapping (RM) campaign to spectroscopically monitor a sample of selected bright active galactic nuclei with large anticipated broad-line region (BLR) sizes adequate for spectroastrometric observations by the GRAVITY instrument on the Very Large Telescope Interferometer. We report the first results for five objects, IC 4329A, Mrk 335, Mrk 509, Mrk 1239, and PDS 456, among which Mrk 1239 and PDS 456 are for the first time spectroscopically monitored. We obtain multi-year monitoring data and perform multi-component spectral decomposition to extract the broad H$\beta$ profiles. We detect significant time lags between the H$\beta$ and continuum variations, generally obeying the previously established BLR size-luminosity relation. Velocity-resolved H$\beta$ time lags illustrate diverse, possibly evolving BLR kinematics. We further measure the H$\beta$ line widths from mean and rms spectra and the resulting virial products show good consistency among different seasons. Adopting a unity virial factor and the full width at half maximum of the broad H$\beta$ line from the mean spectrum as the measure of velocity, the obtained black hole mass averaged over seasons is $\log M_\bullet/M_\odot=8.02_{-0.14}^{+0.09}$, $6.92_{-0.12}^{+0.12}$, $8.01_{-0.25}^{+0.16}$, $7.44_{-0.14}^{+0.13}$, and $8.59_{-0.11}^{+0.07}$ for the five objects, respectively. The black hole mass estimations using other line width measures are also reported (up to the virial factors). For objects with previous RM campaigns, our mass estimates are in agreement with earlier results. In a companion paper, we will employ BLR dynamical modeling to directly infer the black hole mass and thereby determine the virial factors., Comment: 32 pages, 6 tables, 20 figures. To appear in ApJ

In this paper, we present a new version of the second author's factorization theorem for perfect matchings of symmetric graphs. We then use our result to solve four open problems of Propp on the enumeration of trimer tilings on the hexagonal lattice. As another application, we obtain a semi-factorization result for the number of lozenge tilings of a large class of hexagonal regions with holes (obtained by starting with an arbitrary symmetric hexagon with holes, and translating all the holes one unit lattice segment in the same direction). This in turn leads to the solution of two open problems posed by Lai, to an extension of a result due to Fulmek and Krattenthaler, and to exact enumeration formulas for some new families of hexagonal regions with holes. Our result also allows us to find new, simpler proofs (and in one case, a new, simpler form) of some formulas due to Krattenthaler for the number of perfect matchings of Aztec rectangles with unit holes along a lattice diagonal., Comment: 46 pages, 38 figures. The main changes compared to version 1 are: (a) we added Theorem 4.5, and (b) we rewrote Section 5 and made it explicit. We also corrected several minor typos

This paper explores a novel task "Dexterous Grasp as You Say" (DexGYS), enabling robots to perform dexterous grasping based on human commands expressed in natural language. However, the development of this field is hindered by the lack of datasets with natural human guidance; thus, we propose a language-guided dexterous grasp dataset, named DexGYSNet, offering high-quality dexterous grasp annotations along with flexible and fine-grained human language guidance. Our dataset construction is cost-efficient, with the carefully-design hand-object interaction retargeting strategy, and the LLM-assisted language guidance annotation system. Equipped with this dataset, we introduce the DexGYSGrasp framework for generating dexterous grasps based on human language instructions, with the capability of producing grasps that are intent-aligned, high quality and diversity. To achieve this capability, our framework decomposes the complex learning process into two manageable progressive objectives and introduce two components to realize them. The first component learns the grasp distribution focusing on intention alignment and generation diversity. And the second component refines the grasp quality while maintaining intention consistency. Extensive experiments are conducted on DexGYSNet and real world environments for validation., Comment: Accepted by NeurIPS2024

We introduce a block encoding method for mapping discrete subgroups to qubits on a quantum computer. This method is applicable to general discrete groups, including crystal-like subgroups such as $\mathbb{BI}$ of $SU(2)$ and $\mathbb{V}$ of $SU(3)$. We detail the construction of primitive gates -- the inversion gate, the group multiplication gate, the trace gate, and the group Fourier gate -- utilizing this encoding method for $\mathbb{BT}$ and for the first time $\mathbb{BI}$ group. We also provide resource estimations to extract the gluon viscosity. The inversion gates for $\mathbb{BT}$ and $\mathbb{BI}$ are benchmarked on the $\texttt{Baiwang}$ quantum computer with estimated fidelities of $40^{+5}_{-4}\%$ and $4^{+5}_{-3}\%$ respectively., Comment: 12 pages, 10 figures

Stars and compact objects embedded in accretion disks of active galactic nuclei (AGNs), dubbed as ``accretion-modified star" (AMS), often experience hyper-Eddington accretion in the dense gas environment, resulting in powerful outflows as the Bondi explosion and formation of cavities. The varying gas properties across different regions of the AGN disk can give rise to diverse and intriguing phenomena. In this paper, we conduct a study on the characteristics of AMSs situated in the outer, middle, and inner regions of the AGN disk, where growth of the AMSs during the shift inwards is considered. We calculate their multiwavelength spectral energy distributions (SEDs) and thermal light curves. Our results reveal that the thermal luminosity of Bondi explosion occurring in the middle region leads to UV flares with a luminosity of $\sim 10^{44}\rm \,erg\,s^{-1}$. The synchrotron radiation of Bondi explosion in the middle and inner regions peaks at the X-ray band with luminosities of $\sim 10^{43}$ and $\sim 10^{42}\rm \,erg\,s^{-1}$, respectively. The $\gamma$-ray luminosity of inverse Compton radiation spans from $10^{42}$ to $10^{43}\rm \,erg\,s^{-1}$ peaked at $\sim 10\,$MeV (outer region) and $\sim$ GeV (middle and inner regions) bands. The observable flares of AMS in the middle region exhibits a slow rise and rapid Gaussian decay with a duration of months, while in the inner region, it exhibits a fast rise and slow Gaussian decay with a duration of several hours. These various SED and light curve features provides valuable insights into the various astronomical transient timescales associated with AGNs., Comment: 20 pages, 5 figures, 2 tables. Accepted for publication in ApJ

Let $G\leqslant\mathrm{Sym}(\Omega)$ be a finite transitive permutation group with point stabiliser $H$. We say that a subgroup $K$ of $G$ is a fixer if every element of $K$ has fixed points, and we say that $K$ is large if $|K| \geqslant |H|$. There is a special interest in studying large fixers due to connections with Erd\H{o}s-Ko-Rado type problems. In this paper, we classify up to conjugacy the large fixers of the almost simple primitive groups with socle $\mathrm{PSL}_2(q)$, and we use this result to verify a special case of a conjecture of Spiga on permutation characters. We also present some results on large fixers of almost simple primitive groups with socle an alternating or sporadic group., Comment: 33 pages

In this work, we propose a novel discriminative framework for dexterous grasp generation, named Dexterous Grasp TRansformer (DGTR), capable of predicting a diverse set of feasible grasp poses by processing the object point cloud with only one forward pass. We formulate dexterous grasp generation as a set prediction task and design a transformer-based grasping model for it. However, we identify that this set prediction paradigm encounters several optimization challenges in the field of dexterous grasping and results in restricted performance. To address these issues, we propose progressive strategies for both the training and testing phases. First, the dynamic-static matching training (DSMT) strategy is presented to enhance the optimization stability during the training phase. Second, we introduce the adversarial-balanced test-time adaptation (AB-TTA) with a pair of adversarial losses to improve grasping quality during the testing phase. Experimental results on the DexGraspNet dataset demonstrate the capability of DGTR to predict dexterous grasp poses with both high quality and diversity. Notably, while keeping high quality, the diversity of grasp poses predicted by DGTR significantly outperforms previous works in multiple metrics without any data pre-processing. Codes are available at https://github.com/iSEE-Laboratory/DGTR ., Comment: Accepted to CVPR 2024

In this work, we explore a novel task of generating human grasps based on single-view scene point clouds, which more accurately mirrors the typical real-world situation of observing objects from a single viewpoint. Due to the incompleteness of object point clouds and the presence of numerous scene points, the generated hand is prone to penetrating into the invisible parts of the object and the model is easily affected by scene points. Thus, we introduce S2HGrasp, a framework composed of two key modules: the Global Perception module that globally perceives partial object point clouds, and the DiffuGrasp module designed to generate high-quality human grasps based on complex inputs that include scene points. Additionally, we introduce S2HGD dataset, which comprises approximately 99,000 single-object single-view scene point clouds of 1,668 unique objects, each annotated with one human grasp. Our extensive experiments demonstrate that S2HGrasp can not only generate natural human grasps regardless of scene points, but also effectively prevent penetration between the hand and invisible parts of the object. Moreover, our model showcases strong generalization capability when applied to unseen objects. Our code and dataset are available at https://github.com/iSEE-Laboratory/S2HGrasp.

We study the enumeration of off-diagonally symmetric domino tilings of odd-order Aztec diamonds in two directions: (1) with one boundary defect, and (2) with maximally-many zeroes on the diagonal. In the first direction, we prove a symmetry property which states that the numbers of off-diagonally symmetric domino tilings of the Aztec diamond of order $2n-1$ are equal when the boundary defect is at the $k$th position and the $(2n-k)$th position on the boundary, respectively. This symmetry property proves a special case of a recent conjecture by Behrend, Fischer, and Koutschan. In the second direction, a Pfaffian formula is obtained for the number of "nearly" off-diagonally symmetric domino tilings of odd-order Aztec diamonds, where the entries of the Pfaffian satisfy a simple recurrence relation. The numbers of domino tilings mentioned in the above two directions do not seem to have a simple product formula, but we show that these numbers satisfy simple matrix equations in which the entries of the matrix are given by Delannoy numbers. The proof of these results involves the method of non-intersecting lattice paths and a modification of Stembridge's Pfaffian formula for families of non-intersecting lattice paths. Finally, we propose conjectures concerning the log-concavity and asymptotic behavior of the number of off-diagonally symmetric domino tilings of odd-order Aztec diamonds., Comment: 24 pages, 6 figures. Comments are very welcome

As large language models (LLMs) become easily accessible nowadays, the trade-off between safety and helpfulness can significantly impact user experience. A model that prioritizes safety will cause users to feel less engaged and assisted while prioritizing helpfulness will potentially cause harm. Possible harms include teaching people how to build a bomb, exposing youth to inappropriate content, and hurting users' mental health. In this work, we propose to balance safety and helpfulness in diverse use cases by controlling both attributes in LLM. We explore training-free and fine-tuning methods that do not require extra human annotations and analyze the challenges of controlling safety and helpfulness in LLMs. Our experiments demonstrate that our method can rewind a learned model and unlock its controllability.

We present a parameter-efficient method for continual video question-answering (VidQA) learning. Our method, named DAM, uses the proposed Dynamic Adapter Merging to (i) mitigate catastrophic forgetting, (ii) enable efficient adaptation to continually arriving datasets, (iii) handle inputs from unknown datasets during inference, and (iv) enable knowledge sharing across similar dataset domains. Given a set of continually streaming VidQA datasets, we sequentially train dataset-specific adapters for each dataset while freezing the parameters of a large pretrained video-language backbone. During inference, given a video-question sample from an unknown domain, our method first uses the proposed non-parametric router function to compute a probability for each adapter, reflecting how relevant that adapter is to the current video-question input instance. Subsequently, the proposed dynamic adapter merging scheme aggregates all the adapter weights into a new adapter instance tailored for that particular test sample to compute the final VidQA prediction, mitigating the impact of inaccurate router predictions and facilitating knowledge sharing across domains. Our DAM model outperforms prior state-of-the-art continual learning approaches by 9.1% while exhibiting 1.9% less forgetting on 6 VidQA datasets spanning various domains. We further extend DAM to continual image classification and image QA and outperform prior methods by a large margin. The code is publicly available at: https://github.com/klauscc/DAM, Comment: The first two authors contribute equally

Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fall short of generating images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM's in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models., Comment: First two authors contributed equally; Project website: https://selma-t2i.github.io/

A simplex-based network is referred to as a higher-order network, in which describe that the interactions can include more than two nodes. Many multicomponent interactions can be grasped through simplicial complexes, which have recently found applications in social, technological, and biological contexts. The paper first proposes a competitive evolving model of higher-order networks. We introduce the difference equation analysis approach in the high-order network to make the analysis network more rigorous. It avoids the assumption that the degrees of nodes are continuous in the traditional analysis network. We obtain an analytical expression for the distribution of higher-order degrees by employing the theory of Poisson processes. The established results indicate that in a d-order network the scale-free behavior for the (d-1)-dim simplex with respect to the d-order degree is controlled by the competitiveness factor. As the competitiveness increases, the d-order degree of the (d-1)-dim simplex is bent under the logarithmic coordinates. While the e(

The charge density wave (CDW), as a hallmark of vanadium-based kagome superconductor AV3Sb5 (A = K, Rb, Cs), has attracted intensive attention. However, the fundamental controversy regarding the underlying mechanism of CDW therein persists. Recently, the vanadium-based bi-layered kagome metal ScV6Sn6, reported to exhibit a long-range charge order below 94 K, has emerged as a promising candidate to further clarify this core issue. Here, employing micro-focusing angle-resolved photoemission spectroscopy ({\mu}-ARPES) and first-principles calculations, we systematically studied the unique CDW order in vanadium-based bi-layered kagome metals by comparing ScV6Sn6 with its isostructural counterpart YV6Sn6, which lacks a CDW ground state. Combining ARPES data and the corresponding joint density of states (DOS), we suggest that the VHS nesting mechanism might be invalid in these materials. Besides, in ScV6Sn6, we identified multiple hybridization energy gaps resulting from CDW-induced band folding, along with an anomalous band dispersion, implying a potential electron-phonon coupling driven mechanism underlying the formation of the CDW order. Our finding not only comprehensively maps the electronic structure of V-based bi-layer kagome metals but also provide constructive experimental evidence for the unique origin of CDW in this system., Comment: 14 pages, 5 figures

A nanophotonic waveguide coupled with an atomic array forms one of the strongly-coupled quantum interfaces to showcase many fascinating collective features of quantum dynamics. In particular for a dissimilar array of two different interparticle spacings with competing photon-mediated dipole-dipole interactions and directionality of couplings, we study the steady-state phases of atomic excitations under a weakly-driven condition of laser field. We identify a partial set of steady-state phases of the driven system composed of combinations of steady-state solutions in a homogeneous array. We also reveal an intricate role of the atom at the interface of the dissimilar array in determining the steady-state phases and find an alteration in the dichotomy of the phases strongly associated with steady-state distributions with crystalline orders. We further investigate in detail the interaction-induced depletion in half of the dissimilar array. This blockaded region results from two contrasting interparticle spacings near the reciprocal coupling regime, which is evidenced from the analytical solutions under the reciprocal coupling. Our results can provide insights in the driven-dissipative quantum phases of atomic excitations with nonreciprocal couplings and pave the avenues toward quantum simulations of exotic many-body states essential for quantum information applications., Comment: 6 figures

Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities, achieving remarkable advancements on various multimodal downstream tasks. However, deploying LVLMs is often problematic due to their massive computational/energy costs and carbon consumption. Such issues make it infeasible to adopt conventional iterative global pruning, which is costly due to computing the Hessian matrix of the entire large model for sparsification. Alternatively, several studies have recently proposed layer-wise pruning approaches to avoid the expensive computation of global pruning and efficiently compress model weights according to their importance within a layer. However, they often suffer from suboptimal model compression due to their lack of a global perspective. To address this limitation in recent efficient pruning methods for large models, we propose Efficient Coarse-to-Fine LayerWise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs. We first determine the sparsity ratios of different layers or blocks by leveraging the global importance score, which is efficiently computed based on the zeroth-order approximation of the global model gradients. Then, the model performs local layer-wise unstructured weight pruning based on globally-informed sparsity ratios. We validate our proposed method across various multimodal and unimodal models and datasets, demonstrating significant performance improvements over prevalent pruning techniques in the high-sparsity regime., Comment: ICLR 2024 (project page: https://ecoflap.github.io/)

Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse. Therefore, vanilla SMoE models are memory inefficient and non-scalable, especially for resource-constrained downstream scenarios. In this paper, we ask: Can we craft a compact SMoE model by consolidating expert information? What is the best recipe to merge multiple experts into fewer but more knowledgeable experts? Our pilot investigation reveals that conventional model merging methods fail to be effective in such expert merging for SMoE. The potential reasons are: (1) redundant information overshadows critical experts; (2) appropriate neuron permutation for each expert is missing to bring all of them in alignment. To address this, we propose M-SMoE, which leverages routing statistics to guide expert merging. Specifically, it starts with neuron permutation alignment for experts; then, dominant experts and their "group members" are formed; lastly, every expert group is merged into a single expert by utilizing each expert's activation frequency as their weight for merging, thus diminishing the impact of insignificant experts. Moreover, we observed that our proposed merging promotes a low dimensionality in the merged expert's weight space, naturally paving the way for additional compression. Hence, our final method, MC-SMoE (i.e., Merge, then Compress SMoE), further decomposes the merged experts into low-rank and structural sparse alternatives. Extensive experiments across 8 benchmarks validate the effectiveness of MC-SMoE. For instance, our MC-SMoE achieves up to 80% memory and a 20% FLOPs reduction, with virtually no loss in performance., Comment: This paper is accepted in ICLR 2024

The canonical approach to video-text retrieval leverages a coarse-grained or fine-grained alignment between visual and textual information. However, retrieving the correct video according to the text query is often challenging as it requires the ability to reason about both high-level (scene) and low-level (object) visual clues and how they relate to the text query. To this end, we propose a Unified Coarse-to-fine Alignment model, dubbed UCoFiA. Specifically, our model captures the cross-modal similarity information at different granularity levels. To alleviate the effect of irrelevant visual clues, we also apply an Interactive Similarity Aggregation module (ISA) to consider the importance of different visual features while aggregating the cross-modal similarity to obtain a similarity score for each granularity. Finally, we apply the Sinkhorn-Knopp algorithm to normalize the similarities of each level before summing them, alleviating over- and under-representation issues at different levels. By jointly considering the crossmodal similarity of different granularity, UCoFiA allows the effective unification of multi-grained alignments. Empirically, UCoFiA outperforms previous state-of-the-art CLIP-based methods on multiple video-text retrieval benchmarks, achieving 2.4%, 1.4% and 1.3% improvements in text-to-video retrieval R@1 on MSR-VTT, Activity-Net, and DiDeMo, respectively. Our code is publicly available at https://github.com/Ziyang412/UCoFiA., Comment: ICCV 2023