Search

Mining and conveying actionable insights from complex data is a key challenge of exploratory data analysis (EDA) and storytelling. To address this challenge, we present a design space for actionable EDA and storytelling. Synthesizing theory and expert interviews, we highlight how semantic precision, rhetorical persuasion, and pragmatic relevance underpin effective EDA and storytelling. We also show how this design space subsumes common challenges in actionable EDA and storytelling, such as identifying appropriate analytical strategies and leveraging relevant domain knowledge. Building on the potential of LLMs to generate coherent narratives with commonsense reasoning, we contribute Jupybara, an AI-enabled assistant for actionable EDA and storytelling implemented as a Jupyter Notebook extension. Jupybara employs two strategies -- design-space-aware prompting and multi-agent architectures -- to operationalize our design space. An expert evaluation confirms Jupybara's usability, steerability, explainability, and reparability, as well as the effectiveness of our strategies in operationalizing the design space framework with LLMs., Comment: Conditionally Accepted at CHI '25

The treatment of intracranial aneurysms (IA) relies on angiography guidance using biplane views. However, accurate flow estimation and device sizing for treatment are often compromised by vessel overlap and foreshortening, which can obscure critical details. This study introduces an epipolar reconstruction approach to enhance 3D rendering of the internal carotid artery (ICA) and aneurysm dome using routinely acquired biplane angiographic data. Our method aims to improve procedural guidance by overcoming the limitations of traditional two-dimensional imaging techniques. This study employed three 3D geometries of ICA aneurysms to simulate virtual angiograms. These angiograms were generated using a computational fluid dynamics (CFD) solver, followed by the simulation of biplane angiography using a cone-beam geometry. Self-calibration was accomplished by matching contrast media position as a function of time between biplane views. Feature-matching was used to triangulate and reconstruct vascular structures in 3D. The projection data was utilized to refine the 3D estimation, including elimination of erroneous structures and ellipse-fitting. The accuracy of the reconstructions was evaluated using the Dice-Sorensen coefficient, comparing the 3D reconstructions to the original models. The proposed epipolar reconstruction method generalized well across the three tested aneurysm models, with respective Dice-Sorensen coefficients of 0.745, 0.759, and 0.654. Errors were primarily due to partial vessel overlap. The average reconstruction time for all three volumes was approximately 10 seconds. The proposed epipolar reconstruction method enhanced 3D visualization, addressing challenges such as projection-induced vessel foreshortening. This method provides a solution to the complexity of IA visualization, with the potential to provide more accurate analysis and device sizing for treatment., Comment: 15 pages, 10 figures, under review by Medical Physics

Compositional image retrieval (CIR) is a multimodal learning task where a model combines a query image with a user-provided text modification to retrieve a target image. CIR finds applications in a variety of domains including product retrieval (e-commerce) and web search. Existing methods primarily focus on fully-supervised learning, wherein models are trained on datasets of labeled triplets such as FashionIQ and CIRR. This poses two significant challenges: (i) curating such triplet datasets is labor intensive; and (ii) models lack generalization to unseen objects and domains. In this work, we propose SCOT (Self-supervised COmpositional Training), a novel zero-shot compositional pretraining strategy that combines existing large image-text pair datasets with the generative capabilities of large language models to contrastively train an embedding composition network. Specifically, we show that the text embedding from a large-scale contrastively-pretrained vision-language model can be utilized as proxy target supervision during compositional pretraining, replacing the target image embedding. In zero-shot settings, this strategy surpasses SOTA zero-shot compositional retrieval methods as well as many fully-supervised methods on standard benchmarks such as FashionIQ and CIRR., Comment: Paper accepted at WACV 2025 in round 1

We show that the fermion, in the context of a system that comprises many such entities - which, by virtue of the Pauli exclusion principle, possesses a Fermi surface at zero temperature - may itself be thought of as a collection of non-local particle-hole excitations across this Fermi surface. This result is purely kinematical and completely general - not being restricted to any specific dimension, applicable to both continuum and lattice systems. There is also no implication that it is applicable only to low-energy phenomena close to the Fermi surface. We are able to derive the full single-particle dynamical Green function of this fermion at finite temperature by viewing it as a collection of these non-local particle-hole excitations. The Green function of the fermion then manifests itself as a solution to a first-order differential equation in a parameter that controls the number of particle-hole pairs across the Fermi surface, and this equation itself reveals variable coefficients that may be identified with a Bose-Einstein distribution - implying that there is a sense in which the non-local particle-hole excitations have bosonic qualities while not being exact bosons at the level of operators. We also recall the definition of the non-local particle-hole operator that may be used to diagonalize the kinetic energy of free fermions of the sort mentioned above. Number-conserving products of creation and annihilation operators of fermions are expressible as a (rather complicated) combination of these non-local particle-hole operators.

Subarachnoid hemorrhage (SAH), typically due to intracranial aneurysms, demands precise imaging for effective treatment. Digital Subtraction Angiography (DSA), despite being the gold standard, broadly visualizes cerebral blood flow, potentially masking key details in areas. This study introduces an approach integrating a 3D vascular atlas with 2D DSA images to allow targeted quantitative analysis in these crucial regions, thus enhancing diagnostic accuracy during interventions. Initially, DSA data was examined to ascertain the injection site. Following this, the appropriate viewing angle was determined to align accurately with the 3D vascular atlas. Utilizing this atlas, regions corresponding to the areas indicated as perfused were selected. Concurrently, a mask representing the perfused areas was created from the DSA sequence. This mask facilitated the initial coarse alignment of the projected 3D atlas to the DSA perfused territory deformable registration techniques, ensuring a precise overlay with the DSAs perfused territories. The performance of each overlay was measured using the Structural Similarity Index Measure (SSIM). The coregistration process revealed that deformable registrations was essential to achieve precise overlays of the 3D atlas projections with the 2D DSA perfused areas. This approach enabled the extraction of targeted quantitative angiography parameters, essential for detailed vascular assessment in subarachnoid hemorrhage cases. The integration of 3D atlas registration with 2D DSA projections facilitates a more precise and targeted diagnostic process for SAH during critical interventions. This image processing strategy enhances the visualization of affected arterial territories, potentially improving the accuracy of diagnostics and supporting better informed clinical decisions at the time of intervention, Comment: 5 pages, 2 figures, to be published in SPIE Medical Imaging 2025 conference proceeds

This study evaluates a multimodal machine learning framework for predicting treatment outcomes in intracranial aneurysms (IAs). Combining angiographic parametric imaging (API), patient biomarkers, and disease morphology, the framework aims to enhance prognostic accuracy. Data from 340 patients were analyzed, with separate deep neural networks processing quantitative and categorical data. These networks' pre decision layers were concatenated and inputted into a final predictive network. Various data augmentation strategies, including Synthetic Minority Oversampling Technique for Nominal and Continuous data (SMOTE NC), addressed dataset imbalances. Performance metrics, evaluated through Monte Carlo cross validation, showed significant improvements with augmentation, particularly in intermediate fusion models. This study validates the framework's efficacy in accurately predicting IA treatment outcomes, demonstrating that data augmentation techniques can substantially enhance model performance., Comment: 6 pages

In intracranial aneurysm (IA) treatment, digital subtraction angiography (DSA) monitors device-induced hemodynamic changes. Quantitative angiography (QA) provides more precise assessments but is limited by hand-injection variability. This study evaluates correction methods using in vitro phantoms that mimic diverse aneurysm morphologies and locations, addressing the 2D and temporal limitations of DSA. We used a patient-specific phantom to replicate three distinct IA morphologies at various Circle of Willis points: the middle cerebral artery (MCA), anterior communicating artery (ACA), and the internal carotid artery (ICA), each varying in size and shape. The diameters of the IA at MCA, ACA and ICA are 10.1, 10 and 7 millimeters, respectively. QA parameters for both non-stenosed and stenosed conditions were measured with 5ml and 10ml boluses over various injection durations to generate time density curves (TDCs). To address the variability in injection, several singular value decomposition (SVD) variants, standard SVD (sSVD) with Tikhonov regularization, block-circulant SVD (bSVD), and oscillation index SVD (oSVD) were applied. These methods enabled the extraction of IA impulse response function (IRF), peak height (PHIRF), area under the curve (AUCIRF), and mean transit time (MTT). We evaluated the robustness of bias-reducing methods by observing the invariance of these parameters with respect to the injection conditions, and the location and size of the aneurysm. The application of SVD variants, sSVD, bSVD, and oSVD, significantly reduced QA parameter variability due to injection techniques., Comment: arXiv admin note: text overlap with arXiv:2411.03655

This study leverages convolutional neural networks to enhance the temporal resolution of 3D angiography in intracranial aneurysms focusing on the reconstruction of volumetric contrast data from sparse and limited projections. Three patient-specific IA geometries were segmented and converted into stereolithography files to facilitate computational fluid dynamics simulations. These simulations first modeled blood flow under steady conditions with varying inlet velocities: 0.25 m/s, 0.35 m/s, and 0.45 m/s. Subsequently, 3D angiograms were simulated by labeling inlet particles to represent contrast bolus injections over durations of 0.5s, 1.0s, 1.5s, and 2.0s. The angiographic simulations were then used within a simulated cone beam C arm CT system to generate in-silico rotational DSAs, capturing projections every 10 ms over a 220-degree arc at 27 frames per second. From these simulations, both fully sampled (108 projections) and truncated projection datasets were generated the latter using a maximum of 49 projections. High fidelity volumetric images were reconstructed using a Parker weighted Feldkamp Davis Kress algorithm. A modified U Net CNN was subsequently trained on these datasets to reconstruct 3D angiographic volumes from the truncated projections. The network incorporated multiple convolutional layers with ReLU activations and Max pooling, complemented by upsampling and concatenation to preserve spatial detail. Model performance was evaluated using mean squared error (MSE). Evaluating our U net model across the test set yielded a MSE of 0.0001, indicating good agreement with ground truth reconstructions and demonstrating acceptable capabilities in capturing relevant transient angiographic features. This study confirms the feasibility of using CNNs for reconstructing 3D angiographic images from truncated projections.

Despite the remarkable capabilities of modern large language models (LLMs), the mechanisms behind their problem-solving abilities remain elusive. In this work, we aim to better understand how the learning dynamics of LLM finetuning shapes downstream generalization. Our analysis focuses on reasoning tasks, whose problem structure allows us to distinguish between memorization (the exact replication of reasoning steps from the training data) and performance (the correctness of the final solution). We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy: the accuracy of model samples on training queries before they begin to copy the exact reasoning steps from the training set. On the dataset level, this metric is able to reliably predict test accuracy, achieving $R^2$ of around or exceeding 0.9 across various models (Llama3 8, Gemma2 9B), datasets (GSM8k, MATH), and training configurations. On a per-example level, this metric is also indicative of whether individual model predictions are robust to perturbations in the training query. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies. We focus on data curation as an example, and show that prioritizing examples with low pre-memorization accuracy leads to 1.5-2x improvements in data efficiency compared to i.i.d. data scaling, and outperforms other standard data curation techniques.

This study aims to mitigate these biases and enhance QA analysis by applying a path-length correction (PLC) correction, followed by singular value decomposition (SVD)-based deconvolution, to angiograms obtained through both in-silico and in-vitro methods. We utilized DSA data from in-silico and in-vitro patient-specific intracranial aneurysm models. To remove projection bias, PLC for various views were developed by co-registering the pre-existing 3D vascular geometry mask with the DSA projections, followed by ray tracing to determine paths across 3D vessel structures. These maps were used to normalize the logarithmic angiographic images, correcting for projection-induced foreshortening across different angles. Subsequently, we focused on eliminating injection bias by analyzing the corrected angiograms under varied projection views, injection rates, and flow conditions. Regions of interest at the aneurysm dome and inlet were placed to extract Time Density Curves for the lesion and the arterial input function, respectively. Using three standard SVD methodologies, we extracted the aneurysm Impulse Response function (IRF) and its associated parameters Peak Height (PHIRF), Area Under the Curve (AUCIRF), and Mean Transit Time (MTT). Our methodology employing PLC and SVD-based deconvolution ensures reliable quantitative angiographic measurements across varying conditions, supporting consistent assessments of disease severity and treatment efficacy. This approach significantly enhances intrapatient and intraprocedural reliability in neurovascular diagnostics., Comment: 11 pages, 6figures

Intraoperative 2D quantitative angiography (QA) for intracranial aneurysms (IAs) has accuracy challenges due to the variability of hand injections. Despite the success of singular value decomposition (SVD) algorithms in reducing biases in computed tomography perfusion (CTP), their application in 2D QA has not been extensively explored. This study seeks to bridge this gap by investigating the potential of SVD-based deconvolution methods in 2D QA, particularly in addressing the variability of injection durations. The study included three internal carotid aneurysm (ICA) cases. Virtual angiograms were generated using Computational Fluid Dynamics (CFD) for three physiologically relevant inlet velocities to simulate contrast media injection durations. Time-density curves (TDCs) were produced for both the inlet and aneurysm dome. Various SVD variants, including standard SVD (sSVD) with and without classical Tikhonov regularization, block-circulant SVD (bSVD), and oscillation index SVD (oSVD), were applied to virtual angiograms. The method was applied on virtual angiograms to recover the aneurysmal dome impulse response function (IRF) and extract flow related parameters such as Peak Height PHIRF, Area Under the Curve AUCIRF, and Mean transit time MTT. Furthermore, we found that SVD can effectively reduce QA parameter variability across various injection durations, enhancing the potential of QA analysis parameters in neurovascular disease diagnosis and treatment. Implementing SVD-based deconvolution techniques in QA analysis can enhance the precision and reliability of neurovascular diagnostics by effectively reducing the impact of injection duration on hemodynamic parameters.

We introduce Visual Premise Proving (VPP), a novel task tailored to refine the process of chart question answering by deconstructing it into a series of logical premises. Each of these premises represents an essential step in comprehending a chart's content and deriving logical conclusions, thereby providing a granular look at a model's reasoning abilities. This approach represents a departure from conventional accuracy-based evaluation methods, emphasizing the model's ability to sequentially validate each premise and ideally mimic human analytical processes. A model adept at reasoning is expected to demonstrate proficiency in both data retrieval and the structural understanding of charts, suggesting a synergy between these competencies. However, in our zero-shot study using the sophisticated MATCHA model on a scientific chart question answering dataset, an intriguing pattern emerged. The model showcased superior performance in chart reasoning (27\%) over chart structure (19\%) and data retrieval (14\%). This performance gap suggests that models might more readily generalize reasoning capabilities across datasets, benefiting from consistent mathematical and linguistic semantics, even when challenged by changes in the visual domain that complicate structure comprehension and data retrieval. Furthermore, the efficacy of using accuracy of binary QA for evaluating chart reasoning comes into question if models can deduce correct answers without parsing chart data or structure. VPP highlights the importance of integrating reasoning with visual comprehension to enhance model performance in chart analysis, pushing for a balanced approach in evaluating visual data interpretation capabilities., Comment: Under Review : Code and Data will be made public soon

A promising approach for improving reasoning in large language models is to use process reward models (PRMs). PRMs provide feedback at each step of a multi-step reasoning trace, potentially improving credit assignment over outcome reward models (ORMs) that only provide feedback at the final step. However, collecting dense, per-step human labels is not scalable, and training PRMs from automatically-labeled data has thus far led to limited gains. To improve a base policy by running search against a PRM or using it as dense rewards for reinforcement learning (RL), we ask: "How should we design process rewards?". Our key insight is that, to be effective, the process reward for a step should measure progress: a change in the likelihood of producing a correct response in the future, before and after taking the step, corresponding to the notion of step-level advantages in RL. Crucially, this progress should be measured under a prover policy distinct from the base policy. We theoretically characterize the set of good provers and our results show that optimizing process rewards from such provers improves exploration during test-time search and online RL. In fact, our characterization shows that weak prover policies can substantially improve a stronger base policy, which we also observe empirically. We validate our claims by training process advantage verifiers (PAVs) to predict progress under such provers, and show that compared to ORMs, test-time search against PAVs is $>8\%$ more accurate, and $1.5-5\times$ more compute-efficient. Online RL with dense rewards from PAVs enables one of the first results with $5-6\times$ gain in sample efficiency, and $>6\%$ gain in accuracy, over ORMs.

Reconstructing 3D faces with facial geometry from single images has allowed for major advances in animation, generative models, and virtual reality. However, this ability to represent faces with their 3D features is not as fully explored by the facial expression inference (FEI) community. This study therefore aims to investigate the impacts of integrating such 3D representations into the FEI task, specifically for facial expression classification and face-based valence-arousal (VA) estimation. To accomplish this, we first assess the performance of two 3D face representations (both based on the 3D morphable model, FLAME) for the FEI tasks. We further explore two fusion architectures, intermediate fusion and late fusion, for integrating the 3D face representations with existing 2D inference frameworks. To evaluate our proposed architecture, we extract the corresponding 3D representations and perform extensive tests on the AffectNet and RAF-DB datasets. Our experimental results demonstrate that our proposed method outperforms the state-of-the-art AffectNet VA estimation and RAF-DB classification tasks. Moreover, our method can act as a complement to other existing methods to boost performance in many emotion inference tasks., Comment: Accepted by ECCVW 2024

Communicating data insights in an accessible and engaging manner to a broader audience remains a significant challenge. To address this problem, we introduce the Emoji Encoder, a tool that generates a set of emoji recommendations for the field and category names appearing in a tabular dataset. The selected set of emoji encodings can be used to generate configurable unit charts that combine plain text and emojis as word-scale graphics. These charts can serve to contrast values across multiple quantitative fields for each row in the data or to communicate trends over time. Any resulting chart is simply a block of text characters, meaning that it can be directly copied into a text message or posted on a communication platform such as Slack or Teams. This work represents a step toward our larger goal of developing novel, fun, and succinct data storytelling experiences that engage those who do not identify as data analysts. Emoji-based unit charts can offer contextual cues related to the data at the center of a conversation on platforms where emoji-rich communication is typical., Comment: To appear IEEE VIS 2024 Workshop on Data Storytelling in an Era of Generative AI (Gen4DS): https://gen4ds.github.io/gen4ds/

A "match cut" is a common video editing technique where a pair of shots that have a similar composition transition fluidly from one to another. Although match cuts are often visual, certain match cuts involve the fluid transition of audio, where sounds from different sources merge into one indistinguishable transition between two shots. In this paper, we explore the ability to automatically find and create "audio match cuts" within videos and movies. We create a self-supervised audio representation for audio match cutting and develop a coarse-to-fine audio match pipeline that recommends matching shots and creates the blended audio. We further annotate a dataset for the proposed audio match cut task and compare the ability of multiple audio representations to find audio match cut candidates. Finally, we evaluate multiple methods to blend two matching audio candidates with the goal of creating a smooth transition. Project page and examples are available at: https://denfed.github.io/audiomatchcut/, Comment: Accepted to ICASSP 2024

Understanding and communicating data uncertainty is crucial for informed decision-making across various domains, including finance, healthcare, and public policy. This study investigates the impact of gender and acoustic variables on decision-making, confidence, and trust through a crowdsourced experiment. We compared visualization-only representations of uncertainty to text-forward and speech-forward bimodal representations, including multiple synthetic voices across gender. Speech-forward representations led to an increase in risky decisions, and text-forward representations led to lower confidence. Contrary to prior work, speech-forward forecasts did not receive higher ratings of trust. Higher normalized pitch led to a slight increase in decision confidence, but other voice characteristics had minimal impact on decisions and trust. An exploratory analysis of accented speech showed consistent results with the main experiment and additionally indicated lower trust ratings for information presented in Indian and Kenyan accents. The results underscore the importance of considering acoustic and contextual factors in presentation of data uncertainty., Comment: 11 pages, 5 figures, Uncertainty Workshop at IEEE VIS 2024 Conference

Integrating textual content, such as titles, annotations, and captions, with visualizations facilitates comprehension and takeaways during data exploration. Yet current tools often lack mechanisms for integrating meaningful long-form prose with visual data. This paper introduces DASH, a bimodal data exploration tool that supports integrating semantic levels into the interactive process of visualization and text-based analysis. DASH operationalizes a modified version of Lundgard et al.'s semantic hierarchy model that categorizes data descriptions into four levels ranging from basic encodings to high-level insights. By leveraging this structured semantic level framework and a large language model's text generation capabilities, DASH enables the creation of data-driven narratives via drag-and-drop user interaction. Through a preliminary user evaluation, we discuss the utility of DASH's text and chart integration capabilities when participants perform data exploration with the tool., Comment: 5 pages, 2 figures, 1 table

There is increased interest in the interplay between text and visuals in the field of data visualization. However, this attention has predominantly been on the use of text in standalone visualizations or augmenting text stories supported by a series of independent views. In this paper, we shift from the traditional focus on single-chart annotations to characterize the nuanced but crucial communication role of text in the complex environment of interactive dashboards. Through a survey and analysis of 190 dashboards in the wild, plus 13 expert interview sessions with experienced dashboard authors, we highlight the distinctive nature of text as an integral component of the dashboard experience, while delving into the categories, semantic levels, and functional roles of text, and exploring how these text elements are coalesced by dashboard authors to guide and inform dashboard users. Our contributions are: 1) we distill qualitative and quantitative findings from our studies to characterize current practices of text use in dashboards, including a categorization of text-based components and design patterns; 2) we leverage current practices and existing literature to propose, discuss, and validate recommended practices for text in dashboards, embodied as 12 heuristics that underscore the semantic and functional role of text in offering navigational cues, contextualizing data insights, supporting reading order, etc; 3) we reflect on our findings to identify gaps and propose opportunities for data visualization researchers to push the boundaries on text usage for dashboards, from authoring support and interactivity to text generation and content personalization. Our research underscores the significance of elevating text as a first-class citizen in data visualization, and the need to support the inclusion of textual components and their interactive affordances in dashboard design., Comment: VIS 2024 Full Paper, 11 pages

Training on model-generated synthetic data is a promising approach for finetuning LLMs, but it remains unclear when it helps or hurts. In this paper, we investigate this question for math reasoning via an empirical study, followed by building a conceptual understanding of our observations. First, we find that while the typical approach of finetuning a model on synthetic correct or positive problem-solution pairs generated by capable models offers modest performance gains, sampling more correct solutions from the finetuned learner itself followed by subsequent fine-tuning on this self-generated data $\textbf{doubles}$ the efficiency of the same synthetic problems. At the same time, training on model-generated positives can amplify various spurious correlations, resulting in flat or even inverse scaling trends as the amount of data increases. Surprisingly, we find that several of these issues can be addressed if we also utilize negative responses, i.e., model-generated responses that are deemed incorrect by a final answer verifier. Crucially, these negatives must be constructed such that the training can appropriately recover the utility or advantage of each intermediate step in the negative response. With this per-step scheme, we are able to attain consistent gains over only positive data, attaining performance similar to amplifying the amount of synthetic data by $\mathbf{8 \times}$. We show that training on per-step negatives can help to unlearn spurious correlations in the positive data, and is equivalent to advantage-weighted reinforcement learning (RL), implying that it inherits robustness benefits of RL over imitating positive data alone.

Real-valued functions on geometric data -- such as node attributes on a graph -- can be optimized using descriptors from persistent homology, allowing the user to incorporate topological terms in the loss function. When optimizing a single real-valued function (the one-parameter setting), there is a canonical choice of descriptor for persistent homology: the barcode. The operation mapping a real-valued function to its barcode is differentiable almost everywhere, and the convergence of gradient descent for losses using barcodes is relatively well understood. When optimizing a vector-valued function (the multiparameter setting), there is no unique choice of descriptor for multiparameter persistent homology, and many distinct descriptors have been proposed. This calls for the development of a general framework for differentiability and optimization that applies to a wide range of multiparameter homological descriptors. In this article, we develop such a framework and show that it encompasses well-known descriptors of different flavors, such as signed barcodes and the multiparameter persistence landscape. We complement the theory with numerical experiments supporting the idea that optimizing multiparameter homological descriptors can lead to improved performances compared to optimizing one-parameter descriptors, even when using the simplest and most efficiently computable multiparameter descriptors., Comment: 13 pages + 13 page appendix, 8 figures. v2: update references and fix typos

Relevant language describing trends in data can be useful for generating summaries to help with readers' takeaways. However, the language employed in these often template-generated summaries tends to be simple, ranging from describing simple statistical information (e.g., extrema and trends) without additional context and richer language to provide actionable insights. Recent advances in Large Language Models (LLMs) have shown promising capabilities in capturing subtle nuances in language when describing information. This workshop paper specifically explores how LLMs can provide more actionable insights when describing trends by focusing on three dimensions of analytical narrative structure: semantic, rhetorical, and pragmatic. Building on prior research that examines visual and linguistic signatures for univariate line charts, we examine how LLMs can further leverage the semantic dimension of analytical narratives using quantified semantics to describe shapes in trends as people intuitively view them. These semantic descriptions help convey insights in a way that leads to a pragmatic outcome, i.e., a call to action, persuasion, warning vs. alert, and situational awareness. Finally, we identify rhetorical implications for how well these generated narratives align with the perceived shape of the data, thereby empowering users to make informed decisions and take meaningful actions based on these data insights., Comment: 5 pages, 1 figure

Interpreting uncertain data can be difficult, particularly if the data presentation is complex. We investigate the efficacy of different modalities for representing data and how to combine the strengths of each modality to facilitate the communication of data uncertainty. We implemented two multimodal prototypes to explore the design space of integrating speech, text, and visualization elements. A preliminary evaluation with 20 participants from academic and industry communities demonstrates that there exists no one-size-fits-all approach for uncertainty communication strategies; rather, the effectiveness of conveying uncertain data is intertwined with user preferences and situational context, necessitating a more refined, multimodal strategy for future interface design., Comment: 5 pages, 2 figures, accepted at Eurographics Conference on Visualization (EuroVis) 2024

Understanding and communicating data uncertainty is crucial for making informed decisions in sectors like finance and healthcare. Previous work has explored how to express uncertainty in various modes. For example, uncertainty can be expressed visually with quantile dot plots or linguistically with hedge words and prosody. Our research aims to systematically explore how variations within each mode contribute to communicating uncertainty to the user; this allows us to better understand each mode's affordances and limitations. We completed an exploration of the uncertainty design space based on pilot studies and ran two crowdsourced experiments examining how speech, text, and visualization modes and variants within them impact decision-making with uncertain data. Visualization and text were most effective for rational decision-making, though text resulted in lower confidence. Speech garnered the highest trust despite sometimes leading to risky decisions. Results from these studies indicate meaningful trade-offs among modes of information and encourage exploration of multimodal data representations., Comment: 14 pages, 3 figures, accepted at Eurographics Conference on Visualization (EuroVis) 2024

Sound event detection (SED) is an active area of audio research that aims to detect the temporal occurrence of sounds. In this paper, we apply SED to engine fault detection by introducing a multimodal SED framework that detects fine-grained engine faults of automobile engines using audio and accelerometer-recorded vibration. We first introduce the problem of engine fault SED on a dataset collected from a large variety of vehicles with expertly-labeled engine fault sound events. Next, we propose a SED model to temporally detect ten fine-grained engine faults that occur within vehicle engines and further explore a pretraining strategy using a large-scale weakly-labeled engine fault dataset. Through multiple evaluations, we show our proposed framework is able to effectively detect engine fault sound events. Finally, we investigate the interaction and characteristics of each modality and show that fusing features from audio and vibration improves overall engine fault SED capabilities., Comment: Accepted to ICASSP 2024

Natural language and search interfaces intuitively facilitate data exploration and provide visualization responses to diverse analytical queries based on the underlying datasets. However, these interfaces often fail to interpret more complex analytical intents, such as discerning subtleties and quantifiable differences between terms like "bump" and "spike" in the context of COVID cases, for example. We address this gap by extending the capabilities of a data exploration search interface for interpreting semantic concepts in time series trends. We first create a comprehensive dataset of semantic concepts by mapping quantifiable univariate data trends such as slope and angle to crowdsourced, semantically meaningful trend labels. The dataset contains quantifiable properties that capture the slope-scalar effect of semantic modifiers like "sharply" and "gradually," as well as multi-line trends (e.g., "peak," "valley"). We demonstrate the utility of this dataset in SlopeSeeker, a tool that supports natural language querying of quantifiable trends, such as "show me stocks that tanked in 2010." The tool incorporates novel scoring and ranking techniques based on semantic relevance and visual prominence to present relevant trend chart responses containing these semantic trend concepts. In addition, SlopeSeeker provides a faceted search interface for users to navigate a semantic hierarchy of concepts from general trends (e.g., "increase") to more specific ones (e.g., "sharp increase"). A preliminary user evaluation of the tool demonstrates that the search interface supports greater expressivity of queries containing concepts that describe data trends. We identify potential future directions for leveraging our publicly available quantitative semantics dataset in other data domains and for novel visual analytics interfaces., Comment: 21 pages, 19 figures. Intelligent User Interfaces (IUI) 2024

Public pretraining is a promising approach to improve differentially private model training. However, recent work has noted that many positive research results studying this paradigm only consider in-distribution tasks, and may not apply to settings where there is distribution shift between the pretraining and finetuning data -- a scenario that is likely when finetuning private tasks due to the sensitive nature of the data. In this work, we show empirically across three tasks that even in settings with large distribution shift, where both zero-shot performance from public data and training from scratch with private data give unusably weak results, public features can in fact improve private training accuracy by up to 67\% over private training from scratch. We provide a theoretical explanation for this phenomenon, showing that if the public and private data share a low-dimensional representation, public representations can improve the sample complexity of private training even if it is impossible to learn the private task from the public data alone. Altogether, our results provide evidence that public data can indeed make private training practical in realistic settings of extreme distribution shift.

The density-density correlation functions (most singular parts) of chiral Luttinger liquids forming the fractional quantum Hall effect (FQHE) edge are systematically derived in presence of a point-contact junction acting as a localised scalar impurity and are shown to be expressible as compact analytical functions with second order poles and involving the bare scale-independent reflection and transmission coefficients. The results are validated on comparison with standard fermionic perturbation theory. The linear response Hall conductance in the absence of a point-contact is recovered from the obtained density-density correlation functions (DDCF). The system under consideration is inhomogeneous with broken translational invariance and for such systems in one dimension, the connected moments of the density fluctuation operator beyond second order need not be included in order to retrieve the most singular parts of the correlations. The reason being that all odd moments of the density are zero and all higher order even moments are less singular than the quadratic moment. The implications of these results when used in conjunction with bosonization methods in presence of impurity backscattering is briefly discussed., Comment: 22 pages, 3 figures

Self-training and contrastive learning have emerged as leading techniques for incorporating unlabeled data, both under distribution shift (unsupervised domain adaptation) and when it is absent (semi-supervised learning). However, despite the popularity and compatibility of these techniques, their efficacy in combination remains unexplored. In this paper, we undertake a systematic empirical investigation of this combination, finding that (i) in domain adaptation settings, self-training and contrastive learning offer significant complementary gains; and (ii) in semi-supervised learning settings, surprisingly, the benefits are not synergistic. Across eight distribution shift datasets (e.g., BREEDs, WILDS), we demonstrate that the combined method obtains 3--8% higher accuracy than either approach independently. We then theoretically analyze these techniques in a simplified model of distribution shift, demonstrating scenarios under which the features produced by contrastive learning can yield a good initialization for self-training to further amplify gains and achieve optimal performance, even when either method alone would fail., Comment: NeurIPS 2023

In order to create machine learning systems that serve a variety of users well, it is vital to not only achieve high average performance but also ensure equitable outcomes across diverse groups. However, most machine learning methods are designed to improve a model's average performance on a chosen end task without consideration for their impact on worst group error. Multitask learning (MTL) is one such widely used technique. In this paper, we seek not only to understand the impact of MTL on worst-group accuracy but also to explore its potential as a tool to address the challenge of group-wise fairness. We primarily consider the standard setting of fine-tuning a pre-trained model, where, following recent work \citep{gururangan2020don, dery2023aang}, we multitask the end task with the pre-training objective constructed from the end task data itself. In settings with few or no group annotations, we find that multitasking often, but not consistently, achieves better worst-group accuracy than Just-Train-Twice (JTT; \citet{pmlr-v139-liu21f}) -- a representative distributionally robust optimization (DRO) method. Leveraging insights from synthetic data experiments, we propose to modify standard MTL by regularizing the joint multitask representation space. We run a large number of fine-tuning experiments across computer vision and natural language processing datasets and find that our regularized MTL approach \emph{consistently} outperforms JTT on both average and worst-group outcomes. Our official code can be found here: \href{https://github.com/atharvajk98/MTL-group-robustness.git}{\url{https://github.com/atharvajk98/MTL-group-robustness}}., Comment: 20 pages, 7 tables, 6 Figures

Authors often add text annotations to charts to provide additional context for visually prominent features such as peaks, valleys, and trends. However, writing annotations that provide contextual information, such as descriptions of temporal events, often requires considerable manual effort. To address this problem, we introduce Almanac, a JavaScript API that recommends annotations sourced from the New York Times Archive of news headlines. Almanac consists of two independent parts: (1) a prominence feature detector and (2) a contextual annotation recommender. We demonstrate the utility of the API using D3.js and Vega-Lite to annotate a variety of time-series charts covering many different data domains. Preliminary user feedback shows that Almanac is useful to support the authoring of charts with more descriptive annotations.

Objective functions that optimize deep neural networks play a vital role in creating an enhanced feature representation of the input data. Although cross-entropy-based loss formulations have been extensively used in a variety of supervised deep-learning applications, these methods tend to be less adequate when there is large intra-class variance and low inter-class variance in input data distribution. Deep Metric Learning seeks to develop methods that aim to measure the similarity between data samples by learning a representation function that maps these data samples into a representative embedding space. It leverages carefully designed sampling strategies and loss functions that aid in optimizing the generation of a discriminative embedding space even for distributions having low inter-class and high intra-class variances. In this chapter, we will provide an overview of recent progress in this area and discuss state-of-the-art Deep Metric Learning approaches., Comment: Book Chapter Published In Handbook of Statistics, Special Issue - Deep Learning 48, 59

An unconventional bosonization approach that employs a modified Fermi-Bose correspondence is used to obtain the tunneling density of states (TDOS) of fractional quantum Hall (FQHE) edges in the vicinity of a point contact. The chiral Luttinger liquid model is generally used to describe FQHE edge excitations. We introduce a bosonization procedure to study edge state transport in Laughlin states at filling $\nu = 1/m$ with $m$ odd (single edge mode) in the presence of a point contact constriction that brings the top and bottom edges of the sample into close proximity. The unconventional bosonization involves modifying the Fermi-Bose correspondence to incorporate backscattering at the point contact, leaving the action of the theory purely quadratic even in presence of the inhomogeneity. We have shown convincingly in earlier works that this procedure correctly reproduces the most singular parts of the Green functions of the system even when mutual forward scattering between fermions are included. The most singular part of the density-density correlation function (DDCF) relevant to TDOS calculation is computed using a generating functional approach. The TDOS for both the electron tunneling as well as the Laughlin quasiparticle tunneling cases is obtained and is found to agree with previous results in the literature. For electron tunneling the well-known universal power laws for TDOS viz. $ \sim \mbox{ }\omega^{ m-1 }$ and for quasi-particle tunneling the power law $ \sim \mbox{ } \omega^{ \frac{1}{m}-1 } $ are both correctly recovered using our unconventional bosonization scheme. This demonstrates convincingly the utility of the present method which unlike conventional approaches, does not treat the point-contact as an afterthought and yet remains solvable so long as only the most singular parts of the correlation functions are desired., Comment: 14 pages, 2 figures

Conventional wisdom suggests that neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. Our work reassesses this assumption for neural networks with high-dimensional inputs. Rather than extrapolating in arbitrary ways, we observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. Moreover, we find that this value often closely approximates the optimal constant solution (OCS), i.e., the prediction that minimizes the average loss over the training data without observing the input. We present results showing this phenomenon across 8 datasets with different distributional shifts (including CIFAR10-C and ImageNet-R, S), different loss functions (cross entropy, MSE, and Gaussian NLL), and different architectures (CNNs and transformers). Furthermore, we present an explanation for this behavior, which we first validate empirically and then study theoretically in a simplified setting involving deep homogeneous networks with ReLU activations. Finally, we show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.

Liveness Detection (LivDet) is an international competition series open to academia and industry with the objec-tive to assess and report state-of-the-art in Presentation Attack Detection (PAD). LivDet-2023 Noncontact Fingerprint is the first edition of the noncontact fingerprint-based PAD competition for algorithms and systems. The competition serves as an important benchmark in noncontact-based fingerprint PAD, offering (a) independent assessment of the state-of-the-art in noncontact-based fingerprint PAD for algorithms and systems, and (b) common evaluation protocol, which includes finger photos of a variety of Presentation Attack Instruments (PAIs) and live fingers to the biometric research community (c) provides standard algorithm and system evaluation protocols, along with the comparative analysis of state-of-the-art algorithms from academia and industry with both old and new android smartphones. The winning algorithm achieved an APCER of 11.35% averaged overall PAIs and a BPCER of 0.62%. The winning system achieved an APCER of 13.0.4%, averaged over all PAIs tested over all the smartphones, and a BPCER of 1.68% over all smartphones tested. Four-finger systems that make individual finger-based PAD decisions were also tested. The dataset used for competition will be available 1 to all researchers as per data share protocol

In recent times, there is an increased interest in the identification and re-identification of people at long distances, such as from rooftop cameras, UAV cameras, street cams, and others. Such recognition needs to go beyond face and use whole-body markers such as gait. However, datasets to train and test such recognition algorithms are not widely prevalent, and fewer are labeled. This paper introduces DIOR -- a framework for data collection, semi-automated annotation, and also provides a dataset with 14 subjects and 1.649 million RGB frames with 3D/2D skeleton gait labels, including 200 thousands frames from a long range camera. Our approach leverages advanced 3D computer vision techniques to attain pixel-level accuracy in indoor settings with motion capture systems. Additionally, for outdoor long-range settings, we remove the dependency on motion capture systems and adopt a low-cost, hybrid 3D computer vision and learning pipeline with only 4 low-cost RGB cameras, successfully achieving precise skeleton labeling on far-away subjects, even when their height is limited to a mere 20-25 pixels within an RGB frame. On publication, we will make our pipeline open for others to use.

Dashboards are no longer mere static displays of metrics; through functionality such as interaction and storytelling, they have evolved to support analytic and communicative goals like monitoring and reporting. Existing dashboard design guidelines, however, are often unable to account for this expanded scope as they largely focus on best practices for visual design. In contrast, we frame dashboard design as facilitating an analytical conversation: a cooperative, interactive experience where a user may interact with, reason about, or freely query the underlying data. By drawing on established principles of conversational flow and communication, we define the concept of a cooperative dashboard as one that enables a fruitful and productive analytical conversation, and derive a set of 39 dashboard design heuristics to support effective analytical conversations. To assess the utility of this framing, we asked 52 computer science and engineering graduate students to apply our heuristics to critique and design dashboards as part of an ungraded, opt-in homework assignment. Feedback from participants demonstrates that our heuristics surface new reasons dashboards may fail, and encourage a more fluid, supportive, and responsive style of dashboard design. Our approach suggests several compelling directions for future work, including dashboard authoring tools that better anticipate conversational turn-taking, repair, and refinement and extending cooperative principles to other analytical workflows., Comment: 11 pages, 3 figures, 3 tables

We present a comprehensive study of chart visual question-answering(QA) task, to address the challenges faced in comprehending and extracting data from chart visualizations within documents. Despite efforts to tackle this problem using synthetic charts, solutions are limited by the shortage of annotated real-world data. To fill this gap, we introduce a benchmark and dataset for chart visual QA on real-world charts, offering a systematic analysis of the task and a novel taxonomy for template-based chart question creation. Our contribution includes the introduction of a new answer type, 'list', with both ranked and unranked variations. Our study is conducted on a real-world chart dataset from scientific literature, showcasing higher visual complexity compared to other works. Our focus is on template-based QA and how it can serve as a standard for evaluating the first-order logic capabilities of models. The results of our experiments, conducted on a real-world out-of-distribution dataset, provide a robust evaluation of large-scale pre-trained models and advance the field of chart visual QA and formal logic verification for neural networks in general., Comment: This a pre-print version. Accepted at ICDAR '23

We introduce a novel bottom-up approach for the extraction of chart data. Our model utilizes images of charts as inputs and learns to detect keypoints (KP), which are used to reconstruct the components within the plot area. Our novelty lies in detecting a fusion of continuous and discrete KP as predicted heatmaps. A combination of sparse and dense per-pixel objectives coupled with a uni-modal self-attention-based feature-fusion layer is applied to learn KP embeddings. Further leveraging deep metric learning for unsupervised clustering, allows us to segment the chart plot area into various objects. By further matching the chart components to the legend, we are able to obtain the data series names. A post-processing threshold is applied to the KP embeddings to refine the object reconstructions and improve accuracy. Our extensive experiments include an evaluation of different modules for KP estimation and the combination of deep layer aggregation and corner pooling approaches. The results of our experiments provide extensive evaluation for the task of real-world chart data extraction., Comment: Accepted ORAL at ICDAR 23

Relevant language describing visual features in charts can be useful for authoring captions and summaries about the charts to help with readers' takeaways. To better understand the interplay between concepts that describe visual features and the semantic relationships among those concepts (e.g., 'sharp increase' vs. 'gradual rise'), we conducted a crowdsourced study to collect labels and visual feature pairs for univariate line charts. Using this crowdsourced dataset of labeled visual signatures, this paper proposes a novel method for labeling visual chart features based on combining feature-word distributions with the visual features and the data domain of the charts. These feature-word-topic models identify word associations with similar yet subtle differences in semantics, such as 'flat,' 'plateau,' and 'stagnant,' and descriptors of the visual features, such as 'sharp increase,' 'slow climb,' and 'peak.' Our feature-word-topic model is computed using both a quantified semantics approach and a signal processing-inspired least-errors shape-similarity approach. We finally demonstrate the application of this dataset for annotating charts and generating textual data summaries., Comment: 5 pages and 3 figures

Search and information retrieval systems are becoming more expressive in interpreting user queries beyond the traditional weighted bag-of-words model of document retrieval. For example, searching for a flight status or a game score returns a dynamically generated response along with supporting, pre-authored documents contextually relevant to the query. In this paper, we extend this hybrid search paradigm to data repositories that contain curated data sources and visualization content. We introduce a semantic search interface, OLIO, that provides a hybrid set of results comprising both auto-generated visualization responses and pre-authored charts to blend analytical question-answering with content discovery search goals. We specifically explore three search scenarios - question-and-answering, exploratory search, and design search over data repositories. The interface also provides faceted search support for users to refine and filter the conventional best-first search results based on parameters such as author name, time, and chart type. A preliminary user evaluation of the system demonstrates that OLIO's interface and the hybrid search paradigm collectively afford greater expressivity in how users discover insights and visualization content in data repositories., Comment: 14 pages, 9 figures

Recent work has shown that when both the chart and caption emphasize the same aspects of the data, readers tend to remember the doubly-emphasized features as takeaways; when there is a mismatch, readers rely on the chart to form takeaways and can miss information in the caption text. Through a survey of 280 chart-caption pairs in real-world sources (e.g., news media, poll reports, government reports, academic articles, and Tableau Public), we find that captions often do not emphasize the same information in practice, which could limit how effectively readers take away the authors' intended messages. Motivated by the survey findings, we present EmphasisChecker, an interactive tool that highlights visually prominent chart features as well as the features emphasized by the caption text along with any mismatches in the emphasis. The tool implements a time-series prominent feature detector based on the Ramer-Douglas-Peucker algorithm and a text reference extractor that identifies time references and data descriptions in the caption and matches them with chart data. This information enables authors to compare features emphasized by these two modalities, quickly see mismatches, and make necessary revisions. A user study confirms that our tool is both useful and easy to use when authoring charts and captions., Comment: IEEE VIS 2023