Search

Multi-modal Large Language Models have recently experienced rapid developments and excel in various multi-modal tasks. However, they still struggle with mathematical geometric problem solving, which requires exceptional visual perception proficiency. Existing MLLMs mostly optimize the LLM backbone to acquire geometric reasoning capabilities, while rarely emphasizing improvements in visual comprehension. In this paper, we first investigate the visual perception performance of MLLMs when facing geometric diagrams. Our findings reveal that current MLLMs severely suffer from inaccurate geometric perception and hallucinations. To address these limitations, we propose EAGLE, a novel two-stage end-to-end visual enhancement MLLM framework designed to ElevAte Geometric reasoning through LLM-Empowered visual instruction tuning. Specifically, in the preliminary stage, we feed geometric image-caption pairs into our MLLM that contains a fully fine-tuning CLIP ViT and a frozen LLM, aiming to endow our model with basic geometric knowledge. In the subsequent advanced stage, we incorporate LoRA modules into the vision encoder and unfreeze the LLM backbone. This enables the model to leverage the inherent CoT rationales within question-answer pairs, guiding the MLLM to focus on nuanced visual cues and enhancing its overall perceptual capacity. Moreover, we optimize the cross-modal projector in both stages to foster adaptive visual-linguistic alignments. After the two-stage visual enhancement, we develop the geometry expert model EAGLE-7B. Extensive experiments on popular benchmarks demonstrate the effectiveness of our model. For example, on the GeoQA benchmark, EAGLE-7B not only surpasses the exemplary G-LLaVA 7B model by 2.9%, but also marginally outperforms the larger G-LLaVA 13B model. On the MathVista benchmark, EAGLE-7B achieves remarkable 3.8% improvements compared with the proprietary model GPT-4V.

Machine unlearning aims to revoke some training data after learning in response to requests from users, model developers, and administrators. Most previous methods are based on direct fine-tuning, which may neither remove data completely nor retain full performances on the remain data. In this work, we find that, by first masking some important parameters before fine-tuning, the performances of unlearning could be significantly improved. We propose a new masking strategy tailored to unlearning based on Fisher information. Experiments on various datasets and network structures show the effectiveness of the method: without any fine-tuning, the proposed Fisher masking could unlearn almost completely while maintaining most of the performance on the remain data. It also exhibits stronger stability compared to other unlearning baselines

Existing distantly supervised relation extractors usually rely on noisy data for both model training and evaluation, which may lead to garbage-in-garbage-out systems. To alleviate the problem, we study whether a small clean dataset could help improve the quality of distantly supervised models. We show that besides getting a more convincing evaluation of models, a small clean dataset also helps us to build more robust denoising models. Specifically, we propose a new criterion for clean instance selection based on influence functions. It collects sample-level evidence for recognizing good instances (which is more informative than loss-level evidence). We also propose a teacher-student mechanism for controlling purity of intermediate results when bootstrapping the clean set. The whole approach is model-agnostic and demonstrates strong performances on both denoising real (NYT) and synthetic noisy datasets., Comment: Accepted by COLING 2022

Higher-order exceptional points (EPs), which appear as multifold degeneracies in the spectra of non-Hermitian systems, are garnering extensive attention in various multidisciplinary fields. However, constructing higher-order EPs still remains as a challenge due to the strict requirement of the system symmetries. Here we demonstrate that higher-order EPs can be judiciously fabricated in PT -symmetric staggered rhombic lattices by introducing not only on-site gain/loss but also nonHermitian couplings. Zero-energy flatbands persist and symmetry-protected third-order EPs (EP3) arise in these systems owing to the non-Hermitian chiral/sublattice symmetry, but distinct phase transitions and propagation dynamics occur. Specifically, the EP3 arises at the Brillouin zone (BZ) boundary in the presence of on-site gain/loss. The single-site excitations display an exponential power increase in the PT -broken phase. Meanwhile, a nearly flatband sustains when a small lattice perturbation is applied. For the lattices with non-Hermitian couplings, however, the EP3 appears at the BZ center. Quite remarkably, our analysis unveils a dynamical delocalization-localization transition for the excitation of the dispersive bands and a quartic power increase beyond the EP3. Our scheme provides a new platform towards the investigation of the higher-order EPs, and can be further extended to the study of topological phase transitions or nonlinear processes associated with higher-order EPs., Comment: 10 pages, 10 figures