Search

The comic domain is rapidly advancing with the development of single- and multi-page analysis and synthesis models. Recent benchmarks and datasets have been introduced to support and assess models' capabilities in tasks such as detection (panels, characters, text), linking (character re-identification and speaker identification), and analysis of comic elements (e.g., dialog transcription). However, to provide a comprehensive understanding of the storyline, a model must not only extract elements but also understand their relationships and generate highly informative captions. In this work, we propose a pipeline that leverages Vision-Language Models (VLMs) to obtain dense, grounded captions. To construct our pipeline, we introduce an attribute-retaining metric that assesses whether all important attributes are identified in the caption. Additionally, we created a densely annotated test set to fairly evaluate open-source VLMs and select the best captioning model according to our metric. Our pipeline generates dense captions with bounding boxes that are quantitatively and qualitatively superior to those produced by specifically trained models, without requiring any additional training. Using this pipeline, we annotated over 2 million panels across 13,000 books, which will be available on the project page https://github.com/emanuelevivoli/ComiCap., Comment: Accepted at ECCV 2024 Workshop (AI for Visual Art), repo: https://github.com/emanuelevivoli/ComiCap

Vision-language models have recently evolved into versatile systems capable of high performance across a range of tasks, such as document understanding, visual question answering, and grounding, often in zero-shot settings. Comics Understanding, a complex and multifaceted field, stands to greatly benefit from these advances. Comics, as a medium, combine rich visual and textual narratives, challenging AI models with tasks that span image classification, object detection, instance segmentation, and deeper narrative comprehension through sequential panels. However, the unique structure of comics -- characterized by creative variations in style, reading order, and non-linear storytelling -- presents a set of challenges distinct from those in other visual-language domains. In this survey, we present a comprehensive review of Comics Understanding from both dataset and task perspectives. Our contributions are fivefold: (1) We analyze the structure of the comics medium, detailing its distinctive compositional elements; (2) We survey the widely used datasets and tasks in comics research, emphasizing their role in advancing the field; (3) We introduce the Layer of Comics Understanding (LoCU) framework, a novel taxonomy that redefines vision-language tasks within comics and lays the foundation for future work; (4) We provide a detailed review and categorization of existing methods following the LoCU framework; (5) Finally, we highlight current research challenges and propose directions for future exploration, particularly in the context of vision-language models applied to comics. This survey is the first to propose a task-oriented framework for comics intelligence and aims to guide future research by addressing critical gaps in data availability and task definition. A project associated with this survey is available at https://github.com/emanuelevivoli/awesome-comics-understanding., Comment: under review. project website: https://github.com/emanuelevivoli/awesome-comics-understanding

Although foundational vision-language models (VLMs) have proven to be very successful for various semantic discrimination tasks, they still struggle to perform faithfully for fine-grained categorization. Moreover, foundational models trained on one domain do not generalize well on a different domain without fine-tuning. We attribute these to the limitations of the VLM's semantic representations and attempt to improve their fine-grained visual awareness using generative modeling. Specifically, we propose two novel methods: Generative Class Prompt Learning (GCPL) and Contrastive Multi-class Prompt Learning (CoMPLe). Utilizing text-to-image diffusion models, GCPL significantly improves the visio-linguistic synergy in class embeddings by conditioning on few-shot exemplars with learnable class prompts. CoMPLe builds on this foundation by introducing a contrastive learning component that encourages inter-class separation during the generative optimization process. Our empirical results demonstrate that such a generative class prompt learning approach substantially outperform existing methods, offering a better alternative to few shot image recognition challenges. The source code will be made available at: https://github.com/soumitri2001/GCPL., Comment: Accepted in BMVC 2024

The comic domain is rapidly advancing with the development of single-page analysis and synthesis models. However, evaluation metrics and datasets lag behind, often limited to small-scale or single-style test sets. We introduce a novel benchmark, CoMix, designed to evaluate the multi-task capabilities of models in comic analysis. Unlike existing benchmarks that focus on isolated tasks such as object detection or text recognition, CoMix addresses a broader range of tasks including object detection, speaker identification, character re-identification, reading order, and multi-modal reasoning tasks like character naming and dialogue generation. Our benchmark comprises three existing datasets with expanded annotations to support multi-task evaluation. To mitigate the over-representation of manga-style data, we have incorporated a new dataset of carefully selected American comic-style books, thereby enriching the diversity of comic styles. CoMix is designed to assess pre-trained models in zero-shot and limited fine-tuning settings, probing their transfer capabilities across different comic styles and tasks. The validation split of the benchmark is publicly available for research purposes, and an evaluation server for the held-out test split is also provided. Comparative results between human performance and state-of-the-art models reveal a significant performance gap, highlighting substantial opportunities for advancements in comic understanding. The dataset, baseline models, and code are accessible at https://github.com/emanuelevivoli/CoMix-dataset. This initiative sets a new standard for comprehensive comic analysis, providing the community with a common benchmark for evaluation on a large and varied set., Comment: Accepted at NeurIPS 2024 (D&B)

Comics, as a medium, uniquely combine text and images in styles often distinct from real-world visuals. For the past three decades, computational research on comics has evolved from basic object detection to more sophisticated tasks. However, the field faces persistent challenges such as small datasets, inconsistent annotations, inaccessible model weights, and results that cannot be directly compared due to varying train/test splits and metrics. To address these issues, we aim to standardize annotations across datasets, introduce a variety of comic styles into the datasets, and establish benchmark results with clear, replicable settings. Our proposed Comics Datasets Framework standardizes dataset annotations into a common format and addresses the overrepresentation of manga by introducing Comics100, a curated collection of 100 books from the Digital Comics Museum, annotated for detection in our uniform format. We have benchmarked a variety of detection architectures using the Comics Datasets Framework. All related code, model weights, and detailed evaluation processes are available at https://github.com/emanuelevivoli/cdf, ensuring transparency and facilitating replication. This initiative is a significant advancement towards improving object detection in comics, laying the groundwork for more complex computational tasks dependent on precise object recognition., Comment: Accepted at MANPU - COMICS workshop at ICDAR

This work explores a closure task in comics, a medium where visual and textual elements are intricately intertwined. Specifically, Text-cloze refers to the task of selecting the correct text to use in a comic panel, given its neighboring panels. Traditional methods based on recurrent neural networks have struggled with this task due to limited OCR accuracy and inherent model limitations. We introduce a novel Multimodal Large Language Model (Multimodal-LLM) architecture, specifically designed for Text-cloze, achieving a 10% improvement over existing state-of-the-art models in both its easy and hard variants. Central to our approach is a Domain-Adapted ResNet-50 based visual encoder, fine-tuned to the comics domain in a self-supervised manner using SimCLR. This encoder delivers comparable results to more complex models with just one-fifth of the parameters. Additionally, we release new OCR annotations for this dataset, enhancing model input quality and resulting in another 1% improvement. Finally, we extend the task to a generative format, establishing new baselines and expanding the research possibilities in the field of comics analysis.

Holographic imaging is a technique that uses microwave energy to create a three-dimensional image of an object or scene. This technology has potential applications in land mine detection, as the long-wavelength microwave energy can penetrate the ground and create an image of hidden objects without the need for direct physical contact. However, the inversion algorithms commonly used to digitally reconstruct 3D images from holographic images, such as Convolution, Angular Spectrum, and Fresnel, are known to have limitations and can introduce errors in the reconstructed image. Despite these challenges, the use of holographic radar at around 2 GHz in combination with holographic imaging techniques for land mine detection allows to recover size and shape of buried objects. In this paper, we estimate the reconstruction error for the convolution algorithm based on hologram imaging simulation and assess these errors recommending an increase in the scanner area, considering the limitations that the system has and the expected error reduction., Comment: accepted at IWA-GPR

Relevant information in documents is often summarized in tables, helping the reader to identify useful facts. Most benchmark datasets support either document layout analysis or table understanding, but lack in providing data to apply both tasks in a unified way. We define the task of Contextualized Table Extraction (CTE), which aims to extract and define the structure of tables considering the textual context of the document. The dataset comprises 75k fully annotated pages of scientific papers, including more than 35k tables. Data are gathered from PubMed Central, merging the information provided by annotations in the PubTables-1M and PubLayNet datasets. The dataset can support CTE and adds new classes to the original ones. The generated annotations can be used to develop end-to-end pipelines for various tasks, including document layout analysis, table detection, structure recognition, and functional analysis. We formally define CTE and evaluation metrics, showing which subtasks can be tackled, describing advantages, limitations, and future works of this collection of data. Annotations and code will be accessible a https://github.com/AILab-UniFI/cte-dataset.

In this paper, we present a framework for Multilingual Scene Text Visual Question Answering that deals with new languages in a zero-shot fashion. Specifically, we consider the task of Scene Text Visual Question Answering (STVQA) in which the question can be asked in different languages and it is not necessarily aligned to the scene text language. Thus, we first introduce a natural step towards a more generalized version of STVQA: MUST-VQA. Accounting for this, we discuss two evaluation scenarios in the constrained setting, namely IID and zero-shot and we demonstrate that the models can perform on a par on a zero-shot setting. We further provide extensive experimentation and show the effectiveness of adapting multilingual language models into STVQA tasks., Comment: To be appeared in Text In Everything Workshop in ECCV 2022

Tables are widely used in several types of documents since they can bring important information in a structured way. In scientific papers, tables can sum up novel discoveries and summarize experimental results, making the research comparable and easily understandable by scholars. Several methods perform table analysis working on document images, losing useful information during the conversion from the PDF files since OCR tools can be prone to recognition errors, in particular for text inside tables. The main contribution of this work is to tackle the problem of table extraction, exploiting Graph Neural Networks. Node features are enriched with suitably designed representation embeddings. These representations help to better distinguish not only tables from the other parts of the paper, but also table cells from table headers. We experimentally evaluated the proposed approach on a new dataset obtained by merging the information provided in the PubLayNet and PubTables-1M datasets., Comment: ICPR 2022

Despite their importance as subfields of mathematics and engineering, signal and image processing have not received much attention in the field of quantum computation. Cross-correlations are instrumental to all the aforementioned fields. In this article we help fill this void by providing two quantum algorithms, one for computing and storing cross-correlations, and one for implementing the expectation maximization maximum likelihood algorithm. In addition we show that the quantum expectation maximization maximum likelihood algorithm has a quadratic speed-up compared to the classical analog., Comment: 8 pages,2 figures, comments are welcomed

Generating entanglement in a distributed scenario is a fundamental task for implementing the quantum network of the future. We here report a protocol that uses only linear optics for generating GHZ states with high fidelities in a nearby node configuration. Moreover, we analytically show that the scheme provides the highest success probability, and, then, the highest generation rate for sequential protocols. We furthermore show how to retrieve the same results using a numerical approach. Both the analytical proof and the numerical optimization represent a novelty in the field of entanglement generation in quantum networks and are tools for further investigation. Finally, we give some estimates for the generation rate in a real scenario., Comment: 10 pages, 10 figures, 1 figure changed

The generation of GHZ states in quantum networks is a key element for the realization of several quantum information tasks. Given the complexity of the implementation of such generation, it is not easy to find an unambigous proof for an optimal protocol. Motivated by recent improvements in NV center manipulation, we present and compare an extensive list of protocols for generating GHZ states using realistic parameters. Furthermore, in order to establish the goodness of the various protocols, we test them on a specific application, i.e. conference key agreement. We show that for an high number of nodes the best protocol is one presented here for the first time., Comment: 10 pages, 9 figures. Comments are welcome

Quantum key distribution allows for the generation of a secret key between distant parties connected by a quantum channel such as optical fibre or free space. Unfortunately, the rate of generation of a secret key by direct transmission is fundamentally limited by the distance. This limit can be overcome by the implementation of so-called quantum repeaters. Here, we assess the performance of a specific but very natural setup called a single sequential repeater for quantum key distribution. We offer a fine-grained assessment of the repeater by introducing a series of benchmarks. The benchmarks, which should be surpassed to claim a working repeater, are based on finite-energy considerations, thermal noise and the losses in the setup. In order to boost the performance of the studied repeaters we introduce two methods. The first one corresponds to the concept of a cut-off, which reduces the effect of decoherence during storage of a quantum state by introducing a maximum storage time. Secondly, we supplement the standard classical post-processing with an advantage distillation procedure. Using these methods, we find realistic parameters for which it is possible to achieve rates greater than each of the benchmarks, guiding the way towards implementing quantum repeaters., Comment: 29 pages (16 main, 13 supplementary material), 13 figures. Added a figure showing the rate as a function of the distance in kilometres including frequency conversion, as well as a discussion on this issue. Added a figure and section in the main text detailing the experimental setup. Added a figure and discussion to the appendix on how the setup relates to memory-assisted MDI-QKD

The U.S. Particle Physics Project Prioritization Panel (P5) report encouraged the realization of Fermilab's Proton Improvement Plan II (PIP-II) to support future neutrino programs in the United States. PIP-II aims at enhancing the capabilities of the Fermilab existing accelerator complex while simultaneously providing a flexible platform for its future upgrades. The central part of PIP-II project is the construction of a new 800 MeV H- Superconducting (SC) Linac together with upgrades of the Booster and Main Injector synchrotrons. New transfer lines will also be needed to deliver beam to the downstream accelerators and facilities. In this paper we present the recent development of the design of the transfer lines discussing the principles that guided their design, the constraints and requirements imposed by the existing accelerator complex and the following modifications implemented to comply with a better understanding of the limitations and further requirements that emerged during the development of the project., Comment: 3 pp

Device-independent quantum key distribution (DI-QKD) represents one of the most fascinating challenges in quantum communication, exploiting concepts of fundamental physics, namely Bell tests of nonlocality, to ensure the security of a communication link. This requires the loophole-free violation of a Bell inequality, which is intrinsically difficult due to losses in fibre optic transmission channels. Heralded photon amplification is a teleportation-based protocol that has been proposed as a means to overcome transmission loss for DI-QKD. Here we demonstrate heralded photon amplification for path entangled states and characterise the entanglement before and after loss by exploiting a recently developed displacement-based detection scheme. We demonstrate that by exploiting heralded photon amplification we are able to reliably maintain high fidelity entangled states over loss-equivalent distances of more than 50~km., Comment: 7 pages, 6 figures

The Compact Linear Collider (CLIC) is a multi-TeV high-luminosity linear e+e- collider under development. For an optimal exploitation of its physics potential, CLIC is foreseen to be built and operated in a staged approach with three centre-of-mass energy stages ranging from a few hundred GeV up to 3 TeV. The first stage will focus on precision Standard Model physics, in particular Higgs and top-quark measurements. Subsequent stages will focus on measurements of rare Higgs processes, as well as searches for new physics processes and precision measurements of new states, e.g. states previously discovered at LHC or at CLIC itself. In the 2012 CLIC Conceptual Design Report, a fully optimised 3 TeV collider was presented, while the proposed lower energy stages were not studied to the same level of detail. This report presents an updated baseline staging scenario for CLIC. The scenario is the result of a comprehensive study addressing the performance, cost and power of the CLIC accelerator complex as a function of centre-of-mass energy and it targets optimal physics output based on the current physics landscape. The optimised staging scenario foresees three main centre-of-mass energy stages at 380 GeV, 1.5 TeV and 3 TeV for a full CLIC programme spanning 22 years. For the first stage, an alternative to the CLIC drive beam scheme is presented in which the main linac power is produced using X-band klystrons., Comment: 57 pages, 27 figures, 12 tables, published as CERN Yellow Report. Updated version: Minor layout changes for print version

Tremendous progress has been realized in quantum optics for engineering and detecting the quantum properties of light. Today, photon pairs are routinely created in entangled states. Entanglement is revealed using single-photon detectors in which a single photon triggers an avalanche current. The resulting signal is then processed and stored in a computer. Here, we propose an approach to get rid of all the electronic devices between the photons and the experimentalist i.e. to use the experimentalist's eye to detect entanglement. We show in particular, that the micro entanglement that is produced by sending a single photon into a beam-splitter can be detected with the eye using the magnifying glass of a displacement in phase space. The feasibility study convincingly demonstrates the possibility to realize the first experiment where entanglement is observed with the eye., Comment: 7 pages, 4 figures

Photons of a laser beam driving the upper motional sideband of an optomechanical cavity can decay into photon-phonon pairs by means of an optomechanical parametric process. The phononic state can subsequently be mapped to a photonic state by exciting the lower sideband, hence creating photon-photon pairs out of an optomechanical system. We here demonstrate that these pairs can violate a Bell inequality when they are measured with photon counting techniques preceded by small displacement operations in the phase space. Our results show how to detect non-local light-mechanical correlations and open the way for device-independent quantum optomechanics where non-classical light-mechanical correlations can be certified without assumptions on the state structure or on the measurement devices., Comment: 5 pages, 2 figures, the submission has been coordinated with Hofer et al. who reported on similar results

How can one detect entanglement between multiple optical paths sharing a single photon? We address this question by proposing a scalable protocol, which only uses local measurements where single photon detection is combined with small displacement operations. The resulting entanglement witness does not require post-selection, nor assumptions about the photon number in each path. Furthermore, it guarantees that entanglement lies in a subspace with at most one photon per optical path and reveals genuinely multipartite entanglement. We demonstrate its scalability and resistance to loss by performing various experiments with two and three optical paths. We anticipate applications of our results for quantum network certification., Comment: 4 pages + Appendix, 7 figures

What is the most efficient way to generate random numbers device-independently using a photon pair source based on spontaneous parametric down conversion (SPDC)? We consider this question by comparing two implementations of a detection-loophole-free Bell test. In particular, we study in detail a scenario where a heralded single photon source (HSPS) is used to herald path-entangled states, i.e. entanglement between two spatial modes sharing a single photon and where non-locality is revealed using photon counting preceded by small displacement operations. We start by giving a theoretical description of such a measurement. We then show how to optimize the Bell-CHSH violation through a non-perturbative calculation, taking the main experimental imperfections into account. We finally bound the amount of randomness that can be extracted and compare it to the one obtained with the conventional scenario using photon pairs entangled e.g. in polarization and analyzed through photon counting. While the former requires higher overall detection efficiencies, it is far more efficient in terms of both the entropy per experimental run and the rate of random bit generation., Comment: 12 pages, 5 figures

Motivated by very recent experiments, we consider a scenario "\`a la Bell" in which two protagonists test the Clauser-Horne-Shimony-Holt (CHSH) inequality using a photon-pair source based on spontaneous parametric down conversion and imperfect photon detectors. The conventional wisdom says that (i) if the detectors have unit efficiency, the CHSH violation can reach its maximum quantum value $2\sqrt{2}$. To obtain the maximal possible violation, it suffices that the source emits (ii) maximally entangled photon pairs (iii) in two well defined single modes. Through a non-perturabive calculation of non-local correlations, we show that none of these statements are true. By providing the optimal pump parameters, measurement settings and state structure for any detection efficiency and dark count probability, our results give the recipe to close all the loopholes in a Bell test using photon pairs., Comment: 5 pages, 2 figures,updated figure 1 and appendix

We present a detailed analysis of a high-bandwidth quantum memory protocol for storing single photons in a rare-earth-ion doped crystal. The basic idea is to benefit from a coherent free-induced decay type re-emission which occurs naturally when a photon with a broadband spectrum is absorbed by a narrow atomic transition in an optically dense ensemble. This allows for a high-bandwidth memory for realistic material parameters. Long storage time and on-demand readout are obtained by means of spins states in a lambda-type configuration, through the transfer of the optical coherence to a spin coherence (so-called spin-wave storage). We give explicit formulas and show numerical results which make it possible to gain insight into the dependence of the memory efficiency on the optical depth and on the width and the shape of stored photons. We present a feasibility study in rare-earth doped crystals and show that high efficiencies and high bandwidth can be obtained with realistic parameters. High-bandwidth memories using spin-wave storage offers the possibility of very high time-bandwidth products, which is important for experiment where high repetition rates are needed., Comment: 16 pages, 6 figures

The present note relies on the recently published conceptual design report of the LHeC and extends the first contribution to the European strategy debate in emphasising the role of the LHeC to complement and complete the high luminosity LHC programme. The brief discussion therefore focuses on the importance of high precision PDF and $\alpha_s$ determinations for the physics beyond the Standard Model (GUTs, SUSY, Higgs). Emphasis is also given to the importance of high parton density phenomena in nuclei and their relevance to the heavy ion physics programme at the LHC.