Search

Recent advancements in deep learning and computer vision have led to a surge of interest in generating realistic talking heads. This paper presents a comprehensive survey of state-of-the-art methods for talking head generation. We systematically categorises them into four main approaches: image-driven, audio-driven, video-driven and others (including neural radiance fields (NeRF), and 3D-based methods). We provide an in-depth analysis of each method, highlighting their unique contributions, strengths, and limitations. Furthermore, we thoroughly compare publicly available models, evaluating them on key aspects such as inference time and human-rated quality of the generated outputs. Our aim is to provide a clear and concise overview of the current landscape in talking head generation, elucidating the relationships between different approaches and identifying promising directions for future research. This survey will serve as a valuable reference for researchers and practitioners interested in this rapidly evolving field.

Low-cost sensors (LCS) are affordable, compact, and often portable devices designed to measure various environmental parameters, including air quality. These sensors are intended to provide accessible and cost-effective solutions for monitoring pollution levels in different settings, such as indoor, outdoor and moving vehicles. However, the data produced by LCS is prone to various sources of error that can affect accuracy. Calibration is a well-known procedure to improve the reliability of the data produced by LCS, and several developments and efforts have been made to calibrate the LCS. This work proposes a novel Estimated Error Augmented Two-phase Calibration (\textit{EEATC}) approach to calibrate the LCS in stationary and mobile deployments. In contrast to the existing approaches, the \textit{EEATC} calibrates the LCS in two phases, where the error estimated in the first phase calibration is augmented with the input to the second phase, which helps the second phase to learn the distributional features better to produce more accurate results. We show that the \textit{EEATC} outperforms well-known single-phase calibration models such as linear regression models (single variable linear regression (SLR) and multiple variable linear regression (MLR)) and Random forest (RF) in stationary and mobile deployments. To test the \textit{EEATC} in stationary deployments, we have used the Community Air Sensor Network (CAIRSENSE) data set approved by the United States Environmental Protection Agency (USEPA), and the mobile deployments are tested with the real-time data obtained from SensurAir, an LCS device developed and deployed on moving vehicle in Chennai, India., Comment: Accepted for IEEE sensors Journal

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning -- which distinguish between its many forms -- correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables., Comment: 143 pages, 79 tables, 4 figures

Thin films of HfO2 doped with 4% La were fabricated on LSMO/STO (100) substrates using pulsed laser deposition. The stability of the ferroelectric orthorhombic phase in the hafnia films was investigated with respect to varying oxygen pressure during deposition. X-ray diffraction and X-ray photoelectron spectroscopy measurements were carried out to analyze the structure and composition of the films and correlated with their ferroelectric properties. Surprisingly, the ferroelectricity of the hafnia films showed a dependence on oxygen pressure during deposition of LSMO bottom electrode as well. The reason for this dependence is discussed in terms of the active role of non-lattice oxygen in the ferroelectric switching of hafnia.

Current Machine Translation systems achieve very good results on a growing variety of language pairs and data sets. However, it is now well known that they produce fluent translation outputs that often can contain important meaning errors. Quality Estimation task deals with the estimation of quality of translations produced by a Machine Translation system without depending on Reference Translations. A number of approaches have been suggested over the years. In this paper we show that the parallel corpus used as training data for training the MT system holds direct clues for estimating the quality of translations produced by the MT system. Our experiments show that this simple and direct method holds promise for quality estimation of translations produced by any purely data driven machine translation system.

Modern computing requires efficient and dependable data transport. Current solutions like Bluetooth, SMS (Short Message Service), and Email have their restrictions on efficiency, file size, compatibility, and cost. In order to facilitate direct communication and resource sharing amongst linked devices, this research study offers a cross-platform peer-to-peer (P2P) data transmission solution that takes advantage of P2P networks' features. The system enables cost-effective and high-performance data transport by using the compute, storage, and network resources of the participating devices. Simple file sharing, adaptability, dependability, and high performance are some of the important benefits. The examination of the suggested solution is presented in this paper and includes discussion of the P2P architecture, data transfer mechanisms, performance assessment, implementation issues, security concerns, and the potential difficulties that needs to be addressed. The research intends to validate the efficacy and potential of the suggested cross-platform P2P data transfer solution, delivering better efficiency and dependability for users across various platforms, through practical investigations and comparisons with existing approaches.

Whilst a majority of affective computing research focuses on inferring emotions, examining mood or understanding the \textit{mood-emotion interplay} has received significantly less attention. Building on prior work, we (a) deduce and incorporate emotion-change ($\Delta$) information for inferring mood, without resorting to annotated labels, and (b) attempt mood prediction for long duration video clips, in alignment with the characterisation of mood. We generate the emotion-change ($\Delta$) labels via metric learning from a pre-trained Siamese Network, and use these in addition to mood labels for mood classification. Experiments evaluating \textit{unimodal} (training only using mood labels) vs \textit{multimodal} (training using mood plus $\Delta$ labels) models show that mood prediction benefits from the incorporation of emotion-change information, emphasising the importance of modelling the mood-emotion interplay for effective mood inference., Comment: 9 pages, 3 figures, 6 tables, published in IEEE International Conference on Affective Computing and Intelligent Interaction

The translation of AI-generated brain metastases (BM) segmentation into clinical practice relies heavily on diverse, high-quality annotated medical imaging datasets. The BraTS-METS 2023 challenge has gained momentum for testing and benchmarking algorithms using rigorously annotated internationally compiled real-world datasets. This study presents the results of the segmentation challenge and characterizes the challenging cases that impacted the performance of the winning algorithms. Untreated brain metastases on standard anatomic MRI sequences (T1, T2, FLAIR, T1PG) from eight contributed international datasets were annotated in stepwise method: published UNET algorithms, student, neuroradiologist, final approver neuroradiologist. Segmentations were ranked based on lesion-wise Dice and Hausdorff distance (HD95) scores. False positives (FP) and false negatives (FN) were rigorously penalized, receiving a score of 0 for Dice and a fixed penalty of 374 for HD95. Eight datasets comprising 1303 studies were annotated, with 402 studies (3076 lesions) released on Synapse as publicly available datasets to challenge competitors. Additionally, 31 studies (139 lesions) were held out for validation, and 59 studies (218 lesions) were used for testing. Segmentation accuracy was measured as rank across subjects, with the winning team achieving a LesionWise mean score of 7.9. Common errors among the leading teams included false negatives for small lesions and misregistration of masks in space.The BraTS-METS 2023 challenge successfully curated well-annotated, diverse datasets and identified common errors, facilitating the translation of BM segmentation across varied clinical environments and providing personalized volumetric reports to patients undergoing BM treatment.

The present study examines the linear instability characteristics of double-diffusive mixed convective flow in a vertical channel with viscosity stratification. The viscosity of the fluid is modelled as an exponential function of temperature and concentration, with an activation energy parameter determining its sensitivity to temperature variation. Three scenarios are considered: buoyancy force due to thermal diffusion only, buoyancy force due to temperature and solute acting in the same direction, and buoyancy force due to temperature and solute acting in opposite directions. A generalized eigenvalue problem is derived and solved numerically for linear stability analysis via the Chebyshev spectral collocation method. Results indicate that higher values of the activation energy parameter lead to increased flow stability. Additionally, when both buoyant forces act in opposite directions, the Schmidt number has both stabilizing and destabilizing effects across the range of activation energy parameters, similar to the case of pure thermal diffusion. Furthermore, the solutal-buoyancy-opposed base flow is found to be the most stable, while the solutal-buoyancy-assisted base flow is the least stable. As expected, an increase in Reynolds number is shown to decrease the critical Rayleigh number., Comment: 10 pages, 9 figures, Physics of Fluids

Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry., Comment: ACL 2023

Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation. Since text-to-image generation often requires models to generate visual concepts with fine-grained details and attributes specified in text prompts, can we leverage the powerful representations learned by pre-trained diffusion models for discriminative tasks such as image-text matching? To answer this question, we propose a novel approach, Discriminative Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach mainly uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information and fine-tune the model via efficient attention-based prompt learning to perform image-text matching. By comparing DSD with state-of-the-art methods on several benchmark datasets, we demonstrate the potential of using pre-trained diffusion models for discriminative tasks with superior results on few-shot image-text matching.

Recent developments in the Internet of Things (IoT) and real-time applications, have led to the unprecedented growth in the connected devices and their generated data. Traditionally, this sensor data is transferred and processed at the cloud, and the control signals are sent back to the relevant actuators, as part of the IoT applications. This cloud-centric IoT model, resulted in increased latencies and network load, and compromised privacy. To address these problems, Fog Computing was coined by Cisco in 2012, a decade ago, which utilizes proximal computational resources for processing the sensor data. Ever since its proposal, fog computing has attracted significant attention and the research fraternity focused at addressing different challenges such as fog frameworks, simulators, resource management, placement strategies, quality of service aspects, fog economics etc. However, after a decade of research, we still do not see large-scale deployments of public/private fog networks, which can be utilized in realizing interesting IoT applications. In the literature, we only see pilot case studies and small-scale testbeds, and utilization of simulators for demonstrating scale of the specified models addressing the respective technical challenges. There are several reasons for this, and most importantly, fog computing did not present a clear business case for the companies and participating individuals yet. This paper summarizes the technical, non-functional and economic challenges, which have been posing hurdles in adopting fog computing, by consolidating them across different clusters. The paper also summarizes the relevant academic and industrial contributions in addressing these challenges and provides future research directions in realizing real-time fog computing applications, also considering the emerging trends such as federated learning and quantum computing., Comment: Accepted for publication at Wiley Software: Practice and Experience journal

In this paper, we address the problem of wide-baseline camera pose estimation from a group of 360$^\circ$ panoramas under upright-camera assumption. Recent work has demonstrated the merit of deep-learning for end-to-end direct relative pose regression in 360$^\circ$ panorama pairs [11]. To exploit the benefits of multi-view logic in a learning-based framework, we introduce Graph-CoVis, which non-trivially extends CoVisPose [11] from relative two-view to global multi-view spherical camera pose estimation. Graph-CoVis is a novel Graph Neural Network based architecture that jointly learns the co-visible structure and global motion in an end-to-end and fully-supervised approach. Using the ZInD [4] dataset, which features real homes presenting wide-baselines, occlusion, and limited visual overlap, we show that our model performs competitively to state-of-the-art approaches.

Material deformation and failure under impact loading is a subject of active investigation in space science and often requires very specialized equipment for testing. In this work, we present the design, operational analysis and application of a low-velocity ($\sim 100$ m/s) projectile impact framework for evaluating the deformation and failure of space-grade materials. The system is designed to be modular and easily adaptable to various test geometries, while enabling accurate quantitative evaluation of plastic flow. Using coupled numerical methods and experimental techniques, we first establish an operating procedure for the system. Following this, its performance in two complementary impact configurations is demonstrated using numerical and experimental analysis. In the first, a Taylor impact test is performed for predicting the deformed shape of a cylindrical projectile impinging on a rigid substrate. In the second, deformation of a plate struck by a rigid projectile is evaluated. In both cases, physics-based models are used to interpret the resulting fields. We present a discussion of how the system may be used both for material property estimation (e.g., dynamic yield strength) as well as for failure evaluation (e.g., perforation and fracture) in the same projectile impact configuration.

In many clinical conditions, such as head trauma, stroke, and low cardiac output states, the brain is at risk for hypoxic-ischemic injury. The metabolic rate and oxygen consumption of the brain are reflected in internal jugular venous oxygen saturation (sijvO2). The current gold standard for monitoring brain oxygenation is invasive; it requires jugular vein catheterization under fluoroscopic guidance and therefore is rarely used. Photoacoustic (PA) measurement, on the other hand, can estimate the oxygen consumption of the brain non-invasively in real time. Such a convenient method can potentially aid earlier detection and prevention of impending hypoxic brain injury. A dual-wavelength photoacoustic tomography (PAT) and ultrasound imaging (US) system was used to image the internal jugular vein (IJV) and estimate the sijvO2 in seven healthy volunteers. The system captured simultaneous co-registered PAT and US images at a rate of five frames per second. For each volunteer, the internal jugular vein was identified under ultrasound guidance, then, additional PA images from two optical wavelengths were collected and used to estimate the oxygen saturation of the internal jugular vein. For each volunteer, the oxygen saturation was calculated from transverse and longitudinal views of the internal jugular vein. The average sijvO2 was 72 +/- 7 %. The preliminary results are encouraging and agree with those reported in the literature. Photoacoustic images were successfully used to calculate the blood hemoglobin oxygen saturation in the internal jugular vein. These results raise confidence that this emerging technology can be used clinically for accurate, noninvasive estimation of sijvO2. In addition, the fast co-registration with US images makes the technique suitable for studying the temporal variations of oxygen saturation in response to physiologic challenges in clinical settings.

By bringing computing capacity from a remote cloud environment closer to the user, fog computing is introduced. As a result, users can access the services from more nearby computing environments, resulting in better quality of service and lower latency on the network. From the service providers' point of view, this addresses the network latency and congestion issues. This is achieved by deploying the services in cloud and fog computing environments. The responsibility of service providers is to manage the heterogeneous resources available in both computing environments. In recent years, resource management strategies have made it possible to efficiently allocate resources from nearby fog and clouds to users' applications. Unfortunately, these existing resource management strategies fail to give the desired result when the service providers have the opportunity to allocate the resources to the users' application from fog nodes that are at a multi-hop distance from the nearby fog node. The complexity of this resource management problem drastically increases in a MultiFog-Cloud environment. This problem motivates us to revisit and present a novel Heuristic Resource Allocation and Optimization algorithm in a MultiFog-Cloud (HeRAFC) environment. Taking users' application priority, execution time, and communication latency into account, HeRAFC optimizes resource utilization and minimizes cloud load. The proposed algorithm is evaluated and compared with related algorithms. The simulation results show the efficiency of the proposed HeRAFC over other algorithms., Comment: Currently under review with Elsevier JPDC journal

While emotion and mood interchangeably used, they differ in terms of duration, intensity and attributes. Even as multiple psychology studies examine the mood-emotion relationship, mood prediction has barely been studied. Recent machine learning advances such as the attention mechanism to focus on salient parts of the input data, have only been applied to infer emotions rather than mood. We perform mood prediction by incorporating both mood and emotion change information. We additionally explore spatial and temporal attention, and parallel/sequential arrangements of the spatial and temporal attention modules to improve mood prediction performance. To examine generalizability of the proposed method, we evaluate models trained on the AFEW dataset with EMMA. Experiments reveal that (a) emotion change information is inherently beneficial to mood prediction, and (b) prediction performance improves with the integration of sequential and parallel spatial-temporal attention modules.

In two-dimensional (2D) van der Waals (vdW) layered materials the application of pressure often induces a giant lattice collapse, which can subsequently drive an associated Mott transition. Here, we investigate room-temperature layer-dependent insulator-metal transition (IMT) and probable spin-crossover (SCO) in vdW magnet, FePS$_3$, under high-pressure using micro-Raman scattering. Experimentally obtained spectra, in agreement with the computed Raman modes, indicate evidence of IMT of FePS$_3$ started with a thickness-dependent critical pressure ($P_c$) which reduces to 1.5 GPa in trilayer flakes compared to 10.8 GPa for the bulk counterpart. Using a phenomenological model, we argue that strong structural anisotropy in few-layer flakes enhances the in-plane strain under applied pressure and is, therefore, ultimately responsible for reducing the critical pressure for the IMT with decreasing layer numbers. Reduction of the critical pressure for phase transition in vdW magnets to 1-2 GPa marks the possibility of using intercalated few-layers in the field-effect transistor device architecture, and thereby, avoiding the conventional use of the diamond anvil cell (DAC)., Comment: Accepted in Physical Review B

Network-on-Chip (NoC) congestion builds up during heavy traffic load and cripples the system performance by stalling the cores. Moreover, congestion leads to wasted link bandwidth due to blocked buffers and bouncing packets. Existing approaches throttle the cores after congestion is detected, reducing efficiency and wasting line bandwidth unnecessarily. In contrast, we propose a lightweight machine learning-based technique that helps predict congestion in the network. Specifically, our proposed technique collects the features related to traffic at each destination. Then, it labels the features using a novel time reversal approach. The labeled data is used to design a low overhead and an explainable decision tree model used at runtime congestion control. Experimental evaluations with synthetic and real traffic on industrial 6$\times$6 NoC show that the proposed approach increases fairness and memory read bandwidth by up to 114\% with respect to existing congestion control technique while incurring less than 0.01\% of overhead., Comment: The short version of the paper has been accepted in DATE'23

Over the last several years, use cases for robotics based solutions have diversified from factory floors to domestic applications. In parallel, Deep Learning approaches are replacing traditional techniques in Computer Vision, Natural Language Processing, Speech processing, etc. and are delivering robust results. Our goal is to survey a number of research internship projects in the broad area of 'Deep Learning as applied to Robotics' and present a concise view for the benefit of aspiring student interns. In this paper, we survey the research work done by Robotic Institute Summer Scholars (RISS), CMU. We particularly focus on papers that use deep learning to solve core robotic problems and also robotic solutions. We trust this would be useful particularly for internship aspirants for the Robotics Institute, CMU, Comment: This document is a draft version at this stage and the final version will be created soon

Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor Classification, Localization, Understanding (retrieval, question answering, captioning) and gEneration (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities., Comment: Accepted in CVPR 2023. Project page: https://metaclue.github.io/ , Video summary: https://youtu.be/V3TmeNETL-o

Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. We observe that keys and values in cross-attention layers have strong semantic meanings associated with object layouts and content. Therefore, we can better preserve the compositional semantics in the generated image by manipulating the cross-attention representations based on linguistic insights. Built upon Stable Diffusion, a SOTA T2I model, our structured cross-attention design is efficient that requires no additional training samples. We achieve better compositional skills in qualitative and quantitative results, leading to a 5-8% advantage in head-to-head user comparison studies. Lastly, we conduct an in-depth analysis to reveal potential causes of incorrect image compositions and justify the properties of cross-attention layers in the generation process., Comment: ICLR 2023 Camera Ready version