Search

Recent advances in Transformer have led to the revolution of table-based text generation. However, most existing Transformer-based architectures ignore the rich contexts among input tokens distributed in multi-level units (e.g., cell, row, or column), leading to sometimes unfaithful text generation that fails to establish accurate association relationships and misses vital information. In this paper, we propose M ulti- G rained Co ntextual T ransformer (MGCoT), a novel architecture that fully capitalizes on the multi-grained contexts among input tokens and thus strengthens the capacity of table-based text generation. The key primitive, M ulti- G rained Co ntexts (MGCo) module, involves two components: a local context sub-module that adaptively gathers neighboring tokens to form the token-wise local context features, and a global context sub-module that consistently aggregates tokens from a broader range to form the shared global context feature. The former aims at modeling the short-range dependencies that reflect the salience of tokens within similar fine-grained units (e.g., cell and row) attending to the query token, while the latter aims at capturing the long-range dependencies that reflect the significance of each token within similar coarse-grained units (e.g., multiple rows or columns). Based on the fused multi-grained contexts, MGCoT can flexibly and holistically model the content of a table across multi-level structures. On three benchmark datasets, ToTTo, FeTaQA, and Tablesum, MGCoT outperforms strong baselines by a large margin on the quality of the generated texts, demonstrating the effectiveness of multi-grained context modeling. Our source codes are available at https://github.com/Cedric-Mo/MGCoT. • The contexts for each token in a table are various from the structural perspective. • Forming the local contexts allows models to capture contexts in a dynamic range. • Forming the shared global context allows models to capture the consensus. • Models can flexibly and holistically comprehend a table via multi-grained contexts. [ABSTRACT FROM AUTHOR]

The excellent generalization, contextual learning, and emergence abilities in the pre-trained large models (PLMs) handle specific tasks without direct training data, making them the better foundation models in the adversarial domain adaptation (ADA) methods to transfer knowledge learned from the source domain to target domains. However, existing ADA methods fail to account for the confounder properly, which is the root cause of the source data distribution that differs from the target domains. This study proposes a confounder balancing method in adversarial domain adaptation for PLMs fine-tuning (CadaFT), which includes a PLM as the foundation model for a feature extractor, a domain classifier and a confounder classifier, and they are jointly trained with an adversarial loss. This loss is designed to improve the domain-invariant representation learning by diluting the discrimination in the domain classifier. At the same time, the adversarial loss also balances the confounder distribution among source and unmeasured domains in training. Compared to newest ADA methods, CadaFT can correctly identify confounders in domain-invariant features, thereby eliminating the confounder biases in the extracted features from PLMs. The confounder classifier in CadaFT is designed as a plug-and-play and can be applied in the confounder measurable, unmeasurable, or partially measurable environments. Empirical results on natural language processing and computer vision downstream tasks show that CadaFT outperforms the newest GPT-4, LLaMA2, ViT and ADA methods. [ABSTRACT FROM AUTHOR]