Search

Multimodal Large Language Models (MLLMs) are widely regarded as crucial in the exploration of Artificial General Intelligence (AGI). The core of MLLMs lies in their capability to achieve cross-modal alignment. To attain this goal, current MLLMs typically follow a two-phase training paradigm: the pre-training phase and the instruction-tuning phase. Despite their success, there are shortcomings in the modeling of alignment capabilities within these models. Firstly, during the pre-training phase, the model usually assumes that all image-text pairs are uniformly aligned, but in fact the degree of alignment between different image-text pairs is inconsistent. Secondly, the instructions currently used for finetuning incorporate a variety of tasks, different tasks's instructions usually require different levels of alignment capabilities, but previous MLLMs overlook these differentiated alignment needs. To tackle these issues, we propose a new multimodal large language model AlignGPT. In the pre-training stage, instead of treating all image-text pairs equally, we assign different levels of alignment capabilities to different image-text pairs. Then, in the instruction-tuning phase, we adaptively combine these different levels of alignment capabilities to meet the dynamic alignment needs of different instructions. Extensive experimental results show that our model achieves competitive performance on 12 benchmarks., Comment: Code and models are available at $\href{https://aligngpt-vl.github.io/}{\textit{this https URL}}$

\textit{Knowledge-aware} recommendation methods (KGR) based on \textit{graph neural networks} (GNNs) and \textit{contrastive learning} (CL) have achieved promising performance. However, they fall short in modeling fine-grained user preferences and further fail to leverage the \textit{preference-attribute connection} to make predictions, leading to sub-optimal performance. To address the issue, we propose a method named \textit{\textbf{K}nowledge-aware \textbf{D}ual-side \textbf{A}ttribute-enhanced \textbf{R}ecommendation} (KDAR). Specifically, we build \textit{user preference representations} and \textit{attribute fusion representations} upon the attribute information in knowledge graphs, which are utilized to enhance \textit{collaborative filtering} (CF) based user and item representations, respectively. To discriminate the contribution of each attribute in these two types of attribute-based representations, a \textit{multi-level collaborative alignment contrasting} mechanism is proposed to align the importance of attributes with CF signals. Experimental results on four benchmark datasets demonstrate the superiority of KDAR over several state-of-the-art baselines. Further analyses verify the effectiveness of our method. The code of KDAR is released at: \href{https://github.com/TJTP/KDAR}{https://github.com/TJTP/KDAR}.

Evaluating the compatibility between textual descriptions and corresponding images represents a core endeavor within multi-modal research. In recent years, a proliferation of reference-free methods, leveraging visual-language pre-trained models (VLMs), has emerged. Empirical evidence has substantiated that these innovative approaches exhibit a higher correlation with human judgment, marking a significant advancement in the field. However, does a higher correlation with human evaluations alone sufficiently denote the complete of a metric? In response to this question, in this paper, we study if there are any deficiencies in reference-free metrics. Specifically, inspired by the Cobra Effect, we utilize metric scores as rewards to direct the captioning model toward generating descriptions that closely align with the metric's criteria. If a certain metric has flaws, it will be exploited by the model and reflected in the generated sentences. Our findings reveal that descriptions guided by these metrics contain significant flaws, e.g. incoherent statements and excessive repetition. Subsequently, we propose a novel method termed Self-Improving to rectify the identified shortcomings within these metrics. We employ GPT-4V as an evaluative tool to assess generated sentences and the result reveals that our approach achieves state-of-the-art (SOTA) performance. In addition, we also introduce a challenging evaluation benchmark called Flaws Caption to evaluate reference-free image captioning metrics comprehensively. Our code is available at https://github.com/aaronma2020/robust_captioning_metric, Comment: pre-print version

Multimodal large language models (MLLMs) have attracted increasing attention in the past few years, but they may still generate descriptions that include objects not present in the corresponding images, a phenomenon known as object hallucination. To eliminate hallucinations, existing methods manually annotate paired responses with and without hallucinations, and then employ various alignment algorithms to improve the alignment capability between images and text. However, they not only demand considerable computation resources during the finetuning stage but also require expensive human annotation to construct paired data needed by the alignment algorithms. To address these issues, we borrow the idea of unlearning and propose an efficient fine-grained unlearning framework (EFUF), which can eliminate hallucinations without the need for paired data. Extensive experiments show that our method consistently reduces hallucinations while preserving the generation quality with modest computational overhead. Our code and datasets will be publicly available.

We present the discoveries of two giant planets orbiting the red giant branch (RGB) star HD 112570 and the red clump (RC) star HD 154391, based on the radial velocity (RV) measurements from Xinglong station and Okayama Astrophysical Observatory (OAO). Spectroscopic and asteroseismic analyses suggest that HD 112570 has a mass of $1.15\pm0.12\,M_{\odot}$, a radius of $9.85\pm0.23\,R_{\odot}$, a metallicity [Fe/H] of $-0.46\pm0.1$ and a ${\rm log}\,g$ of $2.47\pm0.1$. With the joint analysis of RV and Hipparcos-Gaia astrometry, we obtain a dynamical mass of $M_{\rm p}={3.42}_{-0.84}^{+1.4}\ M_{\rm Jup}$, a period of $P={2615}_{-77}^{+85}$ days and a moderate eccentricity of $e={0.20}_{-0.14}^{+0.16}$ for the Jovian planet HD 112570 b. For HD 154391, it has a mass of $2.07\pm0.03\,M_{\odot}$, a radius of $8.56\pm0.05\,R_{\odot}$, a metallicity [Fe/H] of $0.07\pm0.1$ and a ${\rm log}\,g$ of $2.86\pm0.1$. The super-Jupiter HD 154391 b has a mass of $M_{\rm p}={9.1}_{-1.9}^{+2.8}\ M_{\rm Jup}$, a period of $P={5163}_{-57}^{+60}$ days and an eccentricity of $e={0.20}_{-0.04}^{+0.04}$. We found HD 154391 b has one of the longest orbital period among those ever discovered orbiting evolved stars, which may provide a valuable case in our understanding of planetary formation at wider orbits. Moreover, while a mass gap at $4\,M_{\rm Jup}$ seems to be present in the population of giant stars, there appears to be no significant differences in the distribution of metallicity among giant planets with masses above or below this threshold. Finally, The origin of the abnormal accumulation near 2 au for planets around large evolved stars ($R_{\star}>21\,R_{\odot}$), remains unclear., Comment: 26 pages, 18 figures, 8 tables, Accepted for publication in The Astronomical Journal

Multimodal Aspect-based Sentiment Analysis (MABSA) is a fine-grained Sentiment Analysis task, which has attracted growing research interests recently. Existing work mainly utilizes image information to improve the performance of MABSA task. However, most of the studies overestimate the importance of images since there are many noise images unrelated to the text in the dataset, which will have a negative impact on model learning. Although some work attempts to filter low-quality noise images by setting thresholds, relying on thresholds will inevitably filter out a lot of useful image information. Therefore, in this work, we focus on whether the negative impact of noisy images can be reduced without modifying the data. To achieve this goal, we borrow the idea of Curriculum Learning and propose a Multi-grained Multi-curriculum Denoising Framework (M2DF), which can achieve denoising by adjusting the order of training data. Extensive experimental results show that our framework consistently outperforms state-of-the-art work on three sub-tasks of MABSA., Comment: Accepted by EMNLP 2023

Multimodal Entity Linking (MEL) is a task that aims to link ambiguous mentions within multimodal contexts to referential entities in a multimodal knowledge base. Recent methods for MEL adopt a common framework: they first interact and fuse the text and image to obtain representations of the mention and entity respectively, and then compute the similarity between them to predict the correct entity. However, these methods still suffer from two limitations: first, as they fuse the features of text and image before matching, they cannot fully exploit the fine-grained alignment relations between the mention and entity. Second, their alignment is static, leading to low performance when dealing with complex and diverse data. To address these issues, we propose a novel framework called Dynamic Relation Interactive Network (DRIN) for MEL tasks. DRIN explicitly models four different types of alignment between a mention and entity and builds a dynamic Graph Convolutional Network (GCN) to dynamically select the corresponding alignment relations for different input samples. Experiments on two datasets show that DRIN outperforms state-of-the-art methods by a large margin, demonstrating the effectiveness of our approach., Comment: Accepted by ACM MM 2023

In-Context Learning (ICL) is a new paradigm for natural language processing (NLP), where a large language model (LLM) observes a small number of demonstrations and a test instance as its input, and directly makes predictions without updating model parameters. Previous studies have revealed that ICL is sensitive to the selection and the ordering of demonstrations. However, there are few studies regarding the impact of the demonstration number on the ICL performance within a limited input length of LLM, because it is commonly believed that the number of demonstrations is positively correlated with model performance. In this paper, we found this conclusion does not always hold true. Through pilot experiments, we discover that increasing the number of demonstrations does not necessarily lead to improved performance. Building upon this insight, we propose a Dynamic Demonstrations Controller (D$^2$Controller), which can improve the ICL performance by adjusting the number of demonstrations dynamically. The experimental results show that D$^2$Controller yields a 5.4% relative improvement on eight different sizes of LLMs across ten datasets. Moreover, we also extend our method to previous ICL models and achieve competitive results., Comment: Under review

Transit spectroscopy is the most frequently used technique to reveal the atmospheric properties of exoplanets, while that at high resolution has the advantage to resolve the small Doppler shift of spectral lines, and the trace signal of the exoplanet atmosphere can be separately extracted. We obtain the transmission spectra of the extrasolar planet WASP-85Ab, a hot Jupiter in a 2.655-day orbit around a G5, V=11.2 mag host star, observed by high-resolution spectrograph ESPRESSO at the Very Large Telescope array for three transits. We present an analysis of the Rossiter-McLaughlin effect on WASP-85A, and determine a spin-orbit angle ${\lambda = -16.155^{\circ}}^{+2.916}_{-2.879}$, suggesting that the planet is in an almost aligned orbit. Combining the transmission spectra of three nights, we tentatively detected H$\alpha$ and Ca II absorption with $\gtrapprox 3\sigma$ via direct visual inspection of the transmission spectra with the Center-to-Limb variation and the Rossiter-McLaughlin effects removed, which still remain visible after excluding the cores of these strong lines with a 0.1 A mask. These spectral signals seems likely to origin from the planetary atmosphere, but we can not fully exclude their stellar origins. Via the cross-correlation analysis of a set of atoms and molecules, Li I is marginally detected at $\sim4\sigma$ level, suggesting that Li might be present in the atmosphere of WASP-85Ab., Comment: Accepted for publication in A&A. 18 pages, 17 figures

Safe robot motion generation is critical for practical applications from manufacturing to homes. In this work, we proposed a stochastic optimization-based motion generation method to generate collision-free and time-optimal motion for the articulated robot represented by composite signed distance field (SDF) networks. First, we propose composite SDF networks to learn the SDF for articulated robots. The learned composite SDF networks combined with the kinematics of the robot allow for quick and accurate estimates of the minimum distance between the robot and obstacles in a batch fashion. Then, a stochastic optimization-based trajectory planning algorithm generates a spatial-optimized and collision-free trajectory offline with the learned composite SDF networks. This stochastic trajectory planner is formulated as a Bayesian Inference problem with a time-normalized Gaussian process prior and exponential likelihood function. The Gaussian process prior can enforce initial and goal position constraints in Configuration Space. Besides, it can encode the correlation of waypoints in time series. The likelihood function aims at encoding task-related cost terms, such as collision avoidance, trajectory length penalty, boundary avoidance, etc. The kernel updating strategies combined with model-predictive path integral (MPPI) is proposed to solve the maximum a posteriori inference problems. Lastly, we integrate the learned composite SDF networks into the trajectory planning algorithm and apply it to a Franka Emika Panda robot. The simulation and experiment results validate the effectiveness of the proposed method., Comment: 8 pages, 9 figures, under review of IEEE Robotics and Automation Letters

Aspect Sentiment Triplet Extraction (ASTE) is widely used in various applications. However, existing ASTE datasets are limited in their ability to represent real-world scenarios, hindering the advancement of research in this area. In this paper, we introduce a new dataset, named DMASTE, which is manually annotated to better fit real-world scenarios by providing more diverse and realistic reviews for the task. The dataset includes various lengths, diverse expressions, more aspect types, and more domains than existing datasets. We conduct extensive experiments on DMASTE in multiple settings to evaluate previous ASTE approaches. Empirical results demonstrate that DMASTE is a more challenging ASTE dataset. Further analyses of in-domain and cross-domain settings provide promising directions for future research. Our code and dataset are available at https://github.com/NJUNLP/DMASTE., Comment: 15pages, 5 figures, ACL2023

We report the transit observations of the ultra hot Jupiter WASP-121b using the Goodman High Throughput Spectrograph (GHTS) at the 4-meter ground-based telescope Southern Astrophysical Research Telescope (SOAR), covering the wavelength range $502-900$ nm. By dividing the target and reference star into 19 spectroscopic passbands and applying differential spectrophotometry, we derive spectroscopic transit light curves and fit them using Gaussian process framework to determine transit depths for every passbands. The obtained optical transmission spectrum shows a steep increased slope toward the blue wavelength, which seems to be too steep to be accounted for by the Rayleigh scattering alone. We note that the transmission spectrum from this work and other works differ obviously from each other, which was pointed out previously by \citet{Wilson2021} as evidence for temporal atmospheric variation. We perform a free chemistry retrieval analysis on the optical transmission spectra from this work and the literature HST/WFC3 NIR spectrum. We determine TiO, VO and H$_{2}$O with abundances of $-5.95_{-0.42}^{+0.47}$ dex, $-6.72_{-1.79}^{+0.51}$ dex, and $-4.13_{-0.46}^{+0.63}$ dex, respectively. We compare the abundances of all these three molecules derived from this work and previous works, and find that they are not consistent with each other, indicating the chemical compositions of the terminator region may change over long timescales. Future multi-epoch and high-precision transit observations are required to further confirm this phenomena. We note that when combining the transmission spectra in the optical and in NIR in retrieval analysis, the abundances of V and VO, the NIR-to-optical offset and the cloud deck pressure may be coupled with each other., Comment: 23 pages, 13 figures, 6 tables. Accepted for publication in Research in Astronomy and Astrophysics

Transiting planets provide unique opportunities for the atmospheric characterization of exoplanets as they can reveal composition and the temperature structures at the day-night terminator regions in planetary atmospheres, and help understand the atmospheric process and formation environments of exoplanets. Here, we present the optical transmission spectroscopic study of an inflated Saturn-mass planet WASP-69 b, obtained by the 4-meter ground-based telescope Southern Astrophysical Research Telescope (SOAR). We obtain spectroscopic transit light curves in 20 passbands from 502 to 890 nm, and fit them using Gaussian Processes and an analytical transit model to obtain independent transit depths for each. The derived transmission spectrum of WASP-69 b shows a slope with absorption depth increasing towards blue wavelengths, indicating a Rayleigh scattering in the atmosphere consistent with previous works. The retrieval analysis yields a tentative detection of TiO absorption feature in the transmission spectrum. We present the first results from the SOAR telescope to characterize exoplanetary atmospheres proving its capability and precision for hot Jupiters around bright stars in an area dominated by results from large ground-based telescopes or space telescopes., Comment: 21 pages, 17 figures, 9 tables, Accepted for publication in MNRAS

Being one of the most fundamental physical parameter of astronomical objects, mass plays a vital role in the study of exoplanets, including their temperature structure, chemical composition, formation, and evolution. However, nearly a quarter of the known confirmed exoplanets lack measurements of their masses. This is particularly severe for those discovered via the radial-velocity (RV) technique, which alone could only yield the minimum mass of planets. In this study, we use published RV data combined with astrometric data from a cross-calibrated Hipparcos-Gaia Catalog of Accelerations (HGCA) to jointly constrain the masses of 115 RV-detected substellar companions, by conducting full orbital fits using the public tool \texttt{orvara}. Among them, 9 exoplanets with $M_{\rm p}\,{\rm sin}\,i<13.5\ M_{\rm Jup}$ are reclassified to the brown dwarf (BD) regime, and 16 BD candidates ($13.5\leqslant M_{\rm p}\,{\rm sin}\,i<80\,M_{\rm Jup}$) turn out to be low-mass M dwarfs. We point out the presence of a transition in the BD regime as seen in the distributions of host star metallicity and orbital eccentricity with respect to planet masses. We confirm the previous findings that companions with masses below $42.5\ M_{\rm Jup}$ might primarily form in the protoplanetary disc through core accretion or disc gravitational instability, while those with masses above $42.5\ M_{\rm Jup}$ formed through the gravitational instability of molecular cloud like stars. Selection effects and detection biases which may affect our analysis to some extent, are discussed., Comment: 37 pages, 14 figures, accepted by Research in Astronomy and Astrophysics