Search

Regulatory frameworks for the use of AI are emerging. However, they trail behind the fast-evolving malicious AI technologies that can quickly cause lasting societal damage. In response, we introduce a pioneering Assistive AI framework designed to enhance human decision-making capabilities. This framework aims to establish a trust network across various fields, especially within legal contexts, serving as a proactive complement to ongoing regulatory efforts. Central to our framework are the principles of privacy, accountability, and credibility. In our methodology, the foundation of reliability of information and information sources is built upon the ability to uphold accountability, enhance security, and protect privacy. This approach supports, filters, and potentially guides communication, thereby empowering individuals and communities to make well-informed decisions based on cutting-edge advancements in AI. Our framework uses the concept of Boards as proxies to collectively ensure that AI-assisted decisions are reliable, accountable, and in alignment with societal values and legal standards. Through a detailed exploration of our framework, including its main components, operations, and sample use cases, the paper shows how AI can assist in the complex process of decision-making while maintaining human oversight. The proposed framework not only extends regulatory landscapes but also highlights the synergy between AI technology and human judgement, underscoring the potential of AI to serve as a vital instrument in discerning reality from fiction and thus enhancing the decision-making process. Furthermore, we provide domain-specific use cases to highlight the applicability of our framework., Comment: 37 pages, 6 figures; Changes: Revised references (k-anonimity)

Conversational memory is the process by which humans encode, retain and retrieve verbal, non-verbal and contextual information from a conversation. Since human memory is selective, differing recollections of the same events can lead to misunderstandings and misalignments within a group. Yet, conversational facilitation systems, aimed at advancing the quality of group interactions, usually focus on tracking users' states within an individual session, ignoring what remains in each participant's memory after the interaction. Understanding conversational memory can be used as a source of information on the long-term development of social connections within a group. This paper introduces the MeMo corpus, the first conversational dataset annotated with participants' memory retention reports, aimed at facilitating computational modelling of human conversational memory. The MeMo corpus includes 31 hours of small-group discussions on Covid-19, repeated 3 times over the term of 2 weeks. It integrates validated behavioural and perceptual measures, audio, video, and multimodal annotations, offering a valuable resource for studying and modelling conversational memory and group dynamics. By introducing the MeMo corpus, analysing its validity, and demonstrating its usefulness for future research, this paper aims to pave the way for future research in conversational memory modelling for intelligent system development.

Recognition of individual components and keypoint detection supported by instance segmentation is crucial to analyze the behavior of agents on the scene. Such systems could be used for surveillance, self-driving cars, and also for medical research, where behavior analysis of laboratory animals is used to confirm the aftereffects of a given medicine. A method capable of solving the aforementioned tasks usually requires a large amount of high-quality hand-annotated data, which takes time and money to produce. In this paper, we propose a method that alleviates the need for manual labeling of laboratory rats. To do so, first, we generate initial annotations with a computer vision-based approach, then through extensive augmentation, we train a deep neural network on the generated data. The final system is capable of instance segmentation, keypoint detection, and body part segmentation even when the objects are heavily occluded.

Automatic personality trait assessment is essential for high-quality human-machine interactions. Systems capable of human behavior analysis could be used for self-driving cars, medical research, and surveillance, among many others. We present a multimodal deep neural network with a Siamese extension for apparent personality trait prediction trained on short video recordings and exploiting modality invariant embeddings. Acoustic, visual, and textual information are utilized to reach high-performance solutions in this task. Due to the highly centralized target distribution of the analyzed dataset, the changes in the third digit are relevant. Our proposed method addresses the challenge of under-represented extreme values, achieves 0.0033 MAE average improvement, and shows a clear advantage over the baseline multimodal DNN without the introduced module., Comment: 14 pages, 4 figures

Let either $GL(E)\times SO(F)$ or $GL(E)\times Sp(F)$ act naturally on the space of matrices $E\otimes F$. There are only finitely many orbits, and the orbit closures are orthogonal and symplectic generalizations of determinantal varieties, which can be described similarly using rank conditions. In this paper, we study the singularities of these varieties and describe their defining equations. We prove that in the symplectic case, the orbit closures are normal with good filtrations, and in characteristic $0$ have rational singularities. In the orthogonal case we show that most orbit closures will have the same properties, and determine precisely the exceptions to this.

We study D-modules and related invariants on the space of 2 x 2 x n hypermatrices for n >= 3, which has finitely many orbits under the action of G = GL_2 x GL_2 x GL_n. We describe the category of coherent G-equivariant D-modules as the category of representations of a quiver with relations. We classify the simple equivariant D-modules, determine their characteristic cycles and find special representations that appear in their G-structures. We determine the explicit D-module structure of the local cohomology groups with supports given by orbit closures. As a consequence, we calculate the Lyubeznik numbers and intersection cohomology groups of the orbit closures. All but one of the orbit closures have rational singularities: we use local cohomology to prove that the one exception is neither normal nor Cohen--Macaulay. While our results display special behavior in the cases n=3 and n=4, they are completely uniform for n >= 5., Comment: 45 pages

Cloud-based deployment of content production and broadcast workflows has continued to disrupt the industry after the pandemic. The key tools required for unlocking cloud workflows, e.g., transcoding, metadata parsing, and streaming playback, are increasingly commoditized. However, as video traffic continues to increase there is a need to consider tools which offer opportunities for further bitrate/quality gains as well as those which facilitate cloud deployment. In this paper we consider preprocessing, rate/distortion optimisation and cloud cost prediction tools which are only just emerging from the research community. These tools are posed as part of the per-clip optimisation approach to transcoding which has been adopted by large streaming media processing entities but has yet to be made more widely available for the industry., Comment: Camera-ready version of BEIT Conference at NAB 2023

Mendelian randomization (MR) is an instrumental variable (IV) approach to infer causal relationships between exposures and outcomes with genome-wide association studies (GWAS) summary data. However, the multivariable inverse-variance weighting (IVW) approach, which serves as the foundation for most MR approaches, cannot yield unbiased causal effect estimates in the presence of many weak IVs. To address this problem, we proposed the MR using Bias-corrected Estimating Equation (MRBEE) that can infer unbiased causal relationships with many weak IVs and account for horizontal pleiotropy simultaneously. While the practical significance of MRBEE was demonstrated in our parallel work (Lorincz-Comi (2023)), this paper established the statistical theories of multivariable IVW and MRBEE with many weak IVs. First, we showed that the bias of the multivariable IVW estimate is caused by the error-in-variable bias, whose scale and direction are inflated and influenced by weak instrument bias and sample overlaps of exposures and outcome GWAS cohorts, respectively. Second, we investigated the asymptotic properties of multivariable IVW and MRBEE, showing that MRBEE outperforms multivariable IVW regarding unbiasedness of causal effect estimation and asymptotic validity of causal inference. Finally, we applied MRBEE to examine myopia and revealed that education and outdoor activity are causal to myopia whereas indoor activity is not., Comment: We have observed potential competitors, so we reverted to the version prior to the fourth update (v4). However, this paper and https://www.biorxiv.org/content/10.1101/2023.01.10.523480v3.abstract have been merged, with the main content summarized in Supplemental Material 2

Cities host diverse people and their mixing is the engine of prosperity. In turn, segregation and inequalities are common features of most cities and locations that enable the meeting of people with different socio-economic status are key for urban inclusion. In this study, we adopt the concept of economic complexity to quantify the sophistication of amenity supply at urban locations. We propose that neighborhood complexity and amenity complexity are connected to the ability of locations to attract diverse visitors from various socio-economic backgrounds across the city. We construct the measures of amenity complexity based on the local portfolio of diverse and non-ubiquitous amenities in Budapest, Hungary. Socio-economic mixing at visited third places is investigated by tracing the daily mobility of individuals and by characterizing their status by the real-estate price of their home locations. Results suggest that measures of ubiquity and diversity of amenities do not, but neighborhood complexity and amenity complexity are correlated with the urban centrality of locations. Urban centrality is a strong predictor of socio-economic mixing, but both neighborhood complexity and amenity complexity add further explanatory power to our models. Our work combines urban mobility data with economic complexity thinking to show that the diversity of non-ubiquitous amenities, central locations, and the potentials for socio-economic mixing are interrelated.

Dyadic and small group collaboration is an evolutionary advantageous behaviour and the need for such collaboration is a regular occurrence in day to day life. In this paper we estimate the perceived personality traits of individuals in dyadic and small groups over thin-slices of interaction on four multimodal datasets. We find that our transformer based predictive model performs similarly to human annotators tasked with predicting the perceived big-five personality traits of participants. Using this model we analyse the estimated perceived personality traits of individuals performing tasks in small groups and dyads. Permutation analysis shows that in the case of small groups undergoing collaborative tasks, the perceived personality of group members clusters, this is also observed for dyads in a collaborative problem solving task, but not in dyads under non-collaborative task settings. Additionally, we find that the group level average perceived personality traits provide a better predictor of group performance than the group level average self-reported personality traits., Comment: 11 pages, 4 figures

Imitation Learning uses the demonstrations of an expert to uncover the optimal policy and it is suitable for real-world robotics tasks as well. In this case, however, the training of the agent is carried out in a simulation environment due to safety, economic and time constraints. Later, the agent is applied in the real-life domain using sim-to-real methods. In this paper, we apply Imitation Learning methods that solve a robotics task in a simulated environment and use transfer learning to apply these solutions in the real-world environment. Our task is set in the Duckietown environment, where the robotic agent has to follow the right lane based on the input images of a single forward-facing camera. We present three Imitation Learning and two sim-to-real methods capable of achieving this task. A detailed comparison is provided on these techniques to highlight their advantages and disadvantages., Comment: Accepted by ICLR 2022 Workshop on Generalizable Policy Learning in Physical World. Source code is available at: https://github.com/lzoltan35/duckietown_imitation_learning

The task of converting text input into video content is becoming an important topic for synthetic media generation. Several methods have been proposed with some of them reaching close-to-natural performances in constrained tasks. In this paper, we tackle a subissue of the text-to-video generation problem, by converting the text into lip landmarks. However, we do this using a modular, controllable system architecture and evaluate each of its individual components. Our system, entitled FlexLip, is split into two separate modules: text-to-speech and speech-to-lip, both having underlying controllable deep neural network architectures. This modularity enables the easy replacement of each of its components, while also ensuring the fast adaptation to new speaker identities by disentangling or projecting the input features. We show that by using as little as 20 min of data for the audio generation component, and as little as 5 min for the speech-to-lip component, the objective measures of the generated lip landmarks are comparable with those obtained when using a larger set of training samples. We also introduce a series of objective evaluation measures over the complete flow of our system by taking into consideration several aspects of the data and system configuration. These aspects pertain to the quality and amount of training data, the use of pretrained models, and the data contained therein, as well as the identity of the target speaker; with regard to the latter, we show that we can perform zero-shot lip adaptation to an unseen identity by simply updating the shape of the lips in our model., Comment: 16 pages, 4 tables, 4 figures

While deep neural networks are sensitive to adversarial noise, sparse coding using the Basis Pursuit (BP) method is robust against such attacks, including its multi-layer extensions. We prove that the stability theorem of BP holds upon the following generalizations: (i) the regularization procedure can be separated into disjoint groups with different weights, (ii) neurons or full layers may form groups, and (iii) the regularizer takes various generalized forms of the $\ell_1$ norm. This result provides the proof for the architectural generalizations of Cazenavette et al. (2021), including (iv) an approximation of the complete architecture as a shallow sparse coding network. Due to this approximation, we settled to experimenting with shallow networks and studied their robustness against the Iterative Fast Gradient Sign Method on a synthetic dataset and MNIST. We introduce classification based on the $\ell_2$ norms of the groups and show numerically that it can be accurate and offers considerable speedups. In this family, linear transformer shows the best performance. Based on the theoretical results and the numerical simulations, we highlight numerical matters that may improve performance further., Comment: Supplementary material for DeLTA 2022 accepted short paper. Includes all theorems and proofs

This paper explores how the modelling of energy systems may lead to undue closure of alternatives by generating an excess of certainty around some of the possible policy options. We exemplify the problem with two cases: first, the International Institute for Applied Systems Analysis (IIASA) global modelling in the 1980s; and second, the modelling activity undertaken in support of the construction of a radioactive waste repository at Yucca Mountain (Nevada, USA). We discuss different methodologies for quality assessment that may help remedy this issue, which include NUSAP (Numeral Unit Spread Assessment Pedigree), diagnostic diagrams, and sensitivity auditing. We demonstrate the potential of these reflexive modelling practices in energy policy making with four additional cases: (i) stakeholders evaluation of the assessment of the external costs of a potential large-scale nuclear accident in Belgium in the context of the ExternE (External Costs of Energy) project; (ii) the case of the ESME (Energy System Modelling Environment) for the creation of UK energy policy; (iii) the NETs (Negative Emission Technologies) uptake in Integrated Assessment Models (IAMs); and (iv) the Ecological Footprint (EF) indicator. We encourage modellers to widely adopt these approaches to achieve more robust and inclusive modelling activities in the field of energy modelling., Comment: 39 pages, 2 tables, 3 figures

We compute the rational Borel-Moore homology groups for affine determinantal varieties in the spaces of general, symmetric, and skew-symmetric matrices, solving a problem suggested by the work of Pragacz and Ratajski. The main ingredient is the relation with Hartshorne's algebraic de Rham homology theory, and the calculation of the singular cohomology of matrix orbits, using the methods of Cartan and Borel. We also establish the degeneration of the \v{C}ech-de Rham spectral sequence for determinantal varieties, and compute explicitly the dimensions of de Rham cohomology groups of local cohomology with determinantal support, which are analogues of Lyubeznik numbers first introduced by Switala. Additionally, in the case of general matrices we further determine the Hodge numbers of the singular cohomology of matrix orbits and of the Borel-Moore homology of their closures, based on Saito's theory of mixed Hodge modules., Comment: 28 pages, v2: added results on mixed Hodge structures. New sections: 2.2, 4.3, 7

Building multispeaker neural network-based text-to-speech synthesis systems commonly relies on the availability of large amounts of high quality recordings from each speaker and conditioning the training process on the speaker's identity or on a learned representation of it. However, when little data is available from each speaker, or the number of speakers is limited, the multispeaker TTS can be hard to train and will result in poor speaker similarity and naturalness. In order to address this issue, we explore two directions: forcing the network to learn a better speaker identity representation by appending an additional loss term; and augmenting the input data pertaining to each speaker using waveform manipulation methods. We show that both methods are efficient when evaluated with both objective and subjective measures. The additional loss term aids the speaker similarity, while the data augmentation improves the intelligibility of the multispeaker TTS system., Comment: Accepted at EUSIPCO 2021

Multi-speaker spoken datasets enable the creation of text-to-speech synthesis (TTS) systems which can output several voice identities. The multi-speaker (MSPK) scenario also enables the use of fewer training samples per speaker. However, in the resulting acoustic model, not all speakers exhibit the same synthetic quality, and some of the voice identities cannot be used at all. In this paper we evaluate the influence of the recording conditions, speaker gender, and speaker particularities over the quality of the synthesised output of a deep neural TTS architecture, namely Tacotron2. The evaluation is possible due to the use of a large Romanian parallel spoken corpus containing over 81 hours of data. Within this setup, we also evaluate the influence of different types of text representations: orthographic, phonetic, and phonetic extended with syllable boundaries and lexical stress markings. We evaluate the results of the MSPK system using the objective measures of equal error rate (EER) and word error rate (WER), and also look into the distances between natural and synthesised t-SNE projections of the embeddings computed by an accurate speaker verification network. The results show that there is indeed a large correlation between the recording conditions and the speaker's synthetic voice quality. The speaker gender does not influence the output, and that extending the input text representation with syllable boundaries and lexical stress information does not equally enhance the generated audio across all speaker identities. The visualisation of the t-SNE projections of the natural and synthesised speaker embeddings show that the acoustic model shifts some of the speakers' neural representation, but not all of them. As a result, these speakers have lower performances of the output speech., Comment: Accepted at 25th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES 2021)

The goal of this note is to explain a derivation of the formulas for the local Euler obstructions of determinantal varieties of general, symmetric and skew-symmetric matrices, by studying the invariant de Rham complex and using character formulas for simple equivariant $D$-modules. These calculations are then combined with standard arguments involving Kashiwara's local index formula and the description of characteristic cycles of simple equivariant $D$-modules. The formulas are implicit in the work of Boe and Fu, and in the case of general matrices they have also been obtained recently by Gaffney--Grulha--Ruas, for skew-symmetric matrices by Promtapan and Rim\'anyi, and for all cases by Zhang., Comment: 19 pages. Final version

Pixelwise annotation of image sequences can be very tedious for humans. Interactive video object segmentation aims to utilize automatic methods to speed up the process and reduce the workload of the annotators. Most contemporary approaches rely on deep convolutional networks to collect and process information from human annotations throughout the video. However, such networks contain millions of parameters and need huge amounts of labeled training data to avoid overfitting. Beyond that, label propagation is usually executed as a series of frame-by-frame inference steps, which is difficult to be parallelized and is thus time consuming. In this paper we present a graph neural network based approach for tackling the problem of interactive video object segmentation. Our network operates on superpixel-graphs which allow us to reduce the dimensionality of the problem by several magnitudes. We show, that our network possessing only a few thousand parameters is able to achieve state-of-the-art performance, while inference remains fast and can be trained quickly with very little data.

Our goal is to bridge human and machine intelligence in melanoma detection. We develop a classification system exploiting a combination of visual pre-processing, deep learning, and ensembling for providing explanations to experts and to minimize false negative rate while maintaining high accuracy in melanoma detection. Source images are first automatically segmented using a U-net CNN. The result of the segmentation is then used to extract image sub-areas and specific parameters relevant in human evaluation, namely center, border, and asymmetry measures. These data are then processed by tailored neural networks which include structure searching algorithms. Partial results are then ensembled by a committee machine. Our evaluation on the largest skin lesion dataset which is publicly available today, ISIC-2019, shows improvement in all evaluated metrics over a baseline using the original images only. We also showed that indicative scores computed by the feature classifiers can provide useful insight into the various features on which the decision can be based., Comment: supplementary materials included

A function that is analytic on a domain of $\mathbb{C}^n$ is holonomic if it is the solution to a holonomic system of linear homogeneous differential equations with polynomial coefficients. We define and study the Bernstein-Sato polynomial of a holonomic function on a smooth algebraic variety. We analyze the structure of certain sheaves of holonomic functions, such as the algebraic functions along a hypersurface, determining their direct sum decompositions into indecomposables, that further respect decompositions of Bernstein-Sato polynomials. When the space is endowed with the action of a linear algebraic group $G$, we study the class of $G$-finite analytic functions, i.e. functions that under the action of the Lie algebra of $G$ generate a finite dimensional rational $G$-module. These are automatically algebraic functions on a variety with a dense orbit. When $G$ is reductive, we give several representation-theoretic techniques toward the determination of Bernstein-Sato polynomials of $G$-finite functions. We classify the $G$-finite functions on all but one of the irreducible reduced prehomogeneous vector spaces, and compute the Bernstein-Sato polynomials for distinguished $G$-finite functions. The results can be used to construct explicitly equivariant $\mathcal{D}$-modules., Comment: 35 pages

In multi-person pose estimation actors can be heavily occluded, even become fully invisible behind another person. While temporal methods can still predict a reasonable estimation for a temporarily disappeared pose using past and future frames, they exhibit large errors nevertheless. We present an energy minimization approach to generate smooth, valid trajectories in time, bridging gaps in visibility. We show that it is better than other interpolation based approaches and achieves state of the art results. In addition, we present the synthetic MuCo-Temp dataset, a temporal extension of the MuCo-3DHP dataset. Our code is made publicly available., Comment: ICONIP 2020