Search

Decision making via sequence modeling aims to mimic the success of language models, where actions taken by an embodied agent are modeled as tokens to predict. Despite their promising performance, it remains unclear if embodied sequence modeling leads to the emergence of internal representations that represent the environmental state information. A model that lacks abstract state representations would be liable to make decisions based on surface statistics which fail to generalize. We take the BabyAI environment, a grid world in which language-conditioned navigation tasks are performed, and build a sequence modeling Transformer, which takes a language instruction, a sequence of actions, and environmental observations as its inputs. In order to investigate the emergence of abstract state representations, we design a "blindfolded" navigation task, where only the initial environmental layout, the language instruction, and the action sequence to complete the task are available for training. Our probing results show that intermediate environmental layouts can be reasonably reconstructed from the internal activations of a trained model, and that language instructions play a role in the reconstruction accuracy. Our results suggest that many key features of state representations can emerge via embodied sequence modeling, supporting an optimistic outlook for applications of sequence modeling objectives to more complex embodied decision-making domains., Comment: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023). Project webpage: https://abstract-state-seqmodel.github.io/

Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multilingual visual question answering (mVQA), on both data and modeling fronts. We first propose a translation-based framework to mVQA data generation that requires much less human annotation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilingual captions in the Crossmodal-3600 dataset and develop an efficient annotation protocol to create MaXM, a test-only VQA benchmark in 7 diverse languages. Finally, we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on mVQA., Comment: EMNLP 2023 (Findings). https://github.com/google-research-datasets/maxm

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show superior correlation results with human evaluations when using XM3600 as golden references for automatic metrics., Comment: EMNLP 2022

Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event. Previous approaches usually follow a two-stage generative process, which first proposes a segment for each event, then renders a caption for each identified segment. Recent advances in large-scale sequence generation pretraining have seen great success in unifying task formulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this powerful paradigm. In this work, we show how to model the two subtasks of dense video captioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks such as end-to-end dense video captioning integrated into large-scale pretrained models., Comment: COLING 2022

Cross-modal language generation tasks such as image captioning are directly hurt in their ability to support non-English languages by the trend of data-hungry models combined with the lack of non-English annotations. We investigate potential solutions for combining existing language-generation annotations in English with translation capabilities in order to create solutions at web-scale in both domain and language coverage. We describe an approach called Pivot-Language Generation Stabilization (PLuGS), which leverages directly at training time both existing English annotations (gold data) as well as their machine-translated versions (silver data); at run-time, it generates first an English caption and then a corresponding target-language caption. We show that PLuGS models outperform other candidate solutions in evaluations performed over 5 different target languages, under a large-domain testset using images from the Open Images dataset. Furthermore, we find an interesting effect where the English captions generated by the PLuGS models are better than the captions generated by the original, monolingual English model., Comment: ACL 2020

Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and without access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems., Comment: 10 pages, 6 figures, 3 tables. Accepted to NAACL2021. https://www.aclweb.org/anthology/2021.naacl-main.253/

Reversible state transformations under entanglement non-increasing operations give rise to entanglement measures. It is well known that asymptotic local operations and classical communication (LOCC) are required to get a simple operational measure of bipartite pure state entanglement. For bipartite mixed states and multipartite pure states it is likely that a more powerful class of operations will be needed. To this end \cite{BPRST01} have defined more powerful versions of state transformations (or reducibilities), namely LOCCq (asymptotic LOCC with a sublinear amount of quantum communication) and CLOCC (asymptotic LOCC with catalysis). In this paper we show that {\em LOCCq state transformations are only as powerful as asymptotic LOCC state transformations} for multipartite pure states. We first generalize the concept of entanglement gambling from two parties to multiple parties: any pure multipartite entangled state can be transformed to an EPR pair shared by some pair of parties and that any irreducible $m$ $(m\ge 2)$ party pure state can be used to create any other state (pure or mixed), using only local operations and classical communication (LOCC). We then use this tool to prove the result. We mention some applications of multipartite entanglement gambling to multipartite distillability and to characterizations of multipartite minimal entanglement generating sets. Finally we discuss generalizations of this result to mixed states by defining the class of {\em cat distillable states}.

The entanglement-assisted classical capacity of a noisy quantum channel is the amount of information per channel use that can be sent over the channel in the limit of many uses of the channel, assuming that the sender and receiver have access to the resource of shared quantum entanglement, which may be used up by the communication protocol. We show that this capacity is given by an expression parallel to that for the capacity of a purely classical channel: i.e., the maximum, over channel inputs $\rho$, of the entropy of the channel input plus the entropy of the channel output minus their joint entropy, the latter being defined as the entropy of an entangled purification of $\rho$ after half of it has passed through the channel. We calculate entanglement-assisted capacities for two interesting quantum channels, the qubit amplitude damping channel and the bosonic channel with amplification/attenuation and Gaussian noise. We discuss how many independent parameters are required to completely characterize the asymptotic behavior of a general quantum channel, alone or in the presence of ancillary resources such as prior entanglement. In the classical analog of entanglement assisted communication--communication over a discrete memoryless channel (DMC) between parties who share prior random information--we show that one parameter is sufficient, i.e., that in the presence of prior shared random information, all DMC's of equal capacity can simulate one another with unit asymptotic efficiency., Comment: 42 pages, substantial revision with several corrections and clarifications

We show that, in a multi-party setting, two non-distillable (bound-entangled) states tensored together can make a distillable state. This is an example of true superadditivity of distillable entanglement. We also show that unlockable bound-entangled states cannot be asymptotically unentangled, providing the first proof that some states are truly bound-entangled in the sense of being both non-distillable and non-separable asymptotically., Comment: 4 pages revtex, resubmitted with improvements after referee

We explore the relation between the rank of a bipartite density matrix and the existence of bound entanglement. We show a relation between the rank, marginal ranks, and distillability of a mixed state and use this to prove that any rank n bound entangled state must have support on no more than an n \times n Hilbert space. A direct consequence of this result is that there are no bipartite bound entangled states of rank two. We also show that a separability condition in terms of a quantum entropy inequality is associated with the above results. We explore the idea of how many pure states are needed in a mixture to cancel the distillable entanglement of a Schmidt rank n pure state and provide a lower bound of n-1. We also prove that a mixture of a non-zero amount of any pure entangled state with a pure product state is distillable., Comment: 5 pages LaTeX, many corrections and updated references

We exhibit a two-parameter family of bipartite mixed states $\rho_{bc}$, in a $d\otimes d$ Hilbert space, which are negative under partial transposition (NPT), but for which we conjecture that no maximally entangled pure states in $2\otimes 2$ can be distilled by local quantum operations and classical communication (LQ+CC). Evidence for this undistillability is provided by the result that, for certain states in this family, we cannot extract entanglement from any arbitrarily large number of copies of $\rho_{bc}$ using a projection on $2\otimes 2$. These states are canonical NPT states in the sense that any bipartite mixed state in any dimension with NPT can be reduced by LQ+CC operations to an NPT state of the $\rho_{bc}$ form. We show that the main question about the distillability of mixed states can be formulated as an open mathematical question about the properties of composed positive linear maps., Comment: Revtex, 19 pages, 2 eps figures. v2,3: very minor changes, submitted to Phys. Rev. A. v4: minor typos corrected

In an effort to simplify the classification of pure entangled states of multi (m) -partite quantum systems, we study exactly and asymptotically (in n) reversible transformations among n'th tensor powers of such states (ie n copies of the state shared among the same m parties) under local quantum operations and classical communication (LOCC). With regard to exact transformations, we show that two states whose 1-party entropies agree are either locally-unitarily (LU) equivalent or else LOCC-incomparable. In particular we show that two tripartite Greenberger-Horne-Zeilinger (GHZ) states are LOCC-incomparable to three bipartite Einstein-Podolsky-Rosen (EPR) states symmetrically shared among the three parties. Asymptotic transformations result in a simpler classification than exact transformations. We show that m-partite pure states having an m-way Schmidt decomposition are simply parameterizable, with the partial entropy across any nontrivial partition representing the number of standard ``Cat'' states (|0^m>+|1^m>) asymptotically interconvertible to the state in question. For general m-partite states, partial entropies across different partitions need not be equal, and since partial entropies are conserved by asymptotically reversible LOCC operations, a multicomponent entanglement measure is needed, with each scalar component representing a different kind of entanglement, not asymptotically interconvertible to the other kinds. In particular the m=4 Cat state is not isentropic to, and therefore not asymptotically interconvertible to, any combination of bipartite and tripartite states shared among the four parties. Thus, although the m=4 cat state can be prepared from bipartite EPR states, the preparation process is necessarily irreversible, and remains so even asymptotically., Comment: 13 pages including 3 PostScript figures. v3 has updated references and discussion, to appear Phys. Rev. A

Prior entanglement between sender and receiver, which exactly doubles the classical capacity of a noiseless quantum channel, can increase the classical capacity of some noisy quantum channels by an arbitrarily large constant factor depending on the channel, relative to the best known classical capacity achievable without entanglement. The enhancement factor is greatest for very noisy channels, with positive classical capacity but zero quantum capacity. Although such quantum channels can be simulated classically, no violation of causality is implied, because the simulation requires at least as much forward classical communication as the entanglement-assisted classical capacity of the channel being simulated. We obtain exact expressions for the entanglement-assisted capacity of depolarizing and erasure channels in d dimensions., Comment: 4 pages including 3 figures. Eq 4 restored to be an equality, not merely the lower bound claimed in v4; reference added; one figure removed to save space

Two families of bipartite mixed quantum states are studied for which it is proved that the number of members in the optimal-decomposition ensemble --- the ensemble realizing the entanglement of formation --- is greater than the rank of the mixed state. We find examples for which the number of states in this optimal ensemble can be larger than the rank by an arbitrarily large factor. In one case the proof relies on the fact that the partial transpose of the mixed state has zero eigenvalues; in the other case the result arises from the properties of product bases that are completable only by embedding in a larger Hilbert space., Comment: 14 Pages (RevTeX), 1 figure (eps). Submitted to the special issue of the J. Mod. Opt. V2: Change in terminology from "ensemble length" to "ensemble cardinality"

We show that pure states of multipartite quantum systems are multiseparable (i.e. give separable density matrices on tracing any party) if and only if they have a generalized Schmidt decomposition. Implications of this result for the quantification of multipartite pure-state entanglement are discussed. Further, as an application of the techniques used here, we show that any purification of a bipartite PPT bound entangled state is tri-inseparable, i.e. has none of its three bipartite partial traces separable., Comment: 8 Pages ReVTeX, 4 figures (eps); v2: Revised terminology, added two references and other minor changes; v3: Minor changes, added two references, added author's middle initial; v4: One footnote removed