Search

Due to the fundamental transition to a power electronic dominated power system, the increasing diversity of dynamic elements underscores the need to assess their similarity to mature electrical engineering models. This article addresses the concept of the Dynamic Similarity Index (DSI) for its use in, power electronics-dominated networks. The DSI is a multipurpose tool developed to be used by different stakeholders (e.g., converter manufacturers and system operators). Such an index is calculated per frequency, which serves to anticipate potential differences in particular frequency ranges of interest between the model under study and the reference model. Within the scope of this study, the dynamic similarity of inverter-based generators to an ideal voltage source behind an impedance is assessed, due to the relevance of this fundamental circuit in the representation of generation units in power system studies. The article presents two potential applications based on this mathematical framework. First, for manufacturers to evaluate control performance compared to a reference model and second, it enables operators to diagnose buses with voltage vulnerability based on a user-defined reference Short-Circuit Ratio (SCR) value. The DSI results for these two case studies are validated using Matlab Simulink simulations.

In this paper, frequency dynamics in modern power systems with a high penetration of converter-based generation is analysed. A fundamental analysis of the frequency dynamics is performed to identify the limitations and challenges when the converter penetration is increased. The voltage-source behaviour is found as an essential characteristic of converters to improve the initial frequency derivative of Synchronous Generators (SGs). A detailed small-signal analysis, based on the system's eigenvalues, participation factors and mode shapes, is then performed in a reduced system for different converter penetrations, showing that the flexibility of grid-forming (GFOR) converters as well as the system's inertia reduction may lead to have a more controllable system frequency. First-order frequency responses can be programmed for high converter penetrations, when GFOR operation can impose their dominance over SGs. These results have been validated in the IEEE 118-bus system simulated in PSCAD., Comment: 11 pages, 17 figures

Various linearizations have been proposed to cast syntactic dependency parsing as sequence labeling. However, these approaches do not support more complex graph-based representations, such as semantic dependencies or enhanced universal dependencies, as they cannot handle reentrancy or cycles. By extending them, we define a range of unbounded and bounded linearizations that can be used to cast graph parsing as a tagging task, enlarging the toolbox of problems that can be solved under this paradigm. Experimental results on semantic dependency and enhanced UD parsing show that with a good choice of encoding, sequence-labeling dependency graph parsers combine high efficiency with accuracies close to the state of the art, in spite of their simplicity., Comment: Accepted at EMNLP-2024

Inter-area oscillations have been extensively studied in conventional power systems dominated by synchronous machines, as well as methods to mitigate them. Several publications have addressed Power Oscillation Damping (POD) controllers in grid-following voltage source converters (GFOL). However, the performance of POD controllers for Grid-Forming voltage source converters (GFOR) in modern power systems with increased penetration of power electronics requires further investigation. This paper investigates the performance of GFORs and supplementary POD controllers in the damping of electromechanical oscillations in modern power systems. This paper proposes POD controllers in GFORs by supplementary modulation of active- and reactive-power injections of the converter and both simultaneously (POD- P, POD-Q and POD-PQ, respectively). The proposed POD controllers use the frequency imposed by the GFOR as the input signal, which has a simple implementation and it eliminates the need for additional measurements. Eigenvalue-sensitivity methods using a synthetic test system are applied to the design of POD controllers in GFORs, which is useful when limited information of the power system is available. This paper demonstrates the effectiveness of POD controllers in GFOR converters to damp electromechanical oscillations, by small-signal stability analysis and non-linear time-domain simulations in a small test system and in a large-scale power system., Comment: 11 pages, 15 figures

This paper presents a comprehensive review of stochastic processes, with a particular focus on Markov chains and jump processes. The main results related to queuing systems are analyzed. Additionally, conditions that ensure the stability, or ergodicity, of such systems are presented. The paper also discusses stability results for queuing networks and their extension to visiting systems. Finally, key contributions concerning the Probability Generating Function, an essential tool in the analysis of the aforementioned processes, are introduced. The review is conducted from the perspective of queuing theory, grounded in the Kendall-Lee notation, emphasizing stability results and the computation of performance measures based on the specific characteristics of each process., Comment: in Spanish language

This document presents a compilation of results related to the theory of stochastic processes, with a specific focus on Markov processes, regenerative processes, renewal processes, and stationary processes. The relevance of these topics lies in the ability to identify regeneration points and the necessary conditions to ensure the stationarity of the process. The study begins with a review of Markov chains and continues with the analysis of processes that satisfy the strong Markov property. Subsequently, it delves into renewal processes, regenerative processes, and finally, stationary regenerative processes, highlighting the results presented by Thorisson [22]. This work is not intended to be exhaustive but aims to provide a solid foundation for further deepening the knowledge of these processes, given their broad range of applications in cryptography [16], queueing theory [15], and Monte Carlo methods [23]. Additionally, the importance of Poisson-type processes is emphasized due to their numerous applications (see [8])., Comment: 21 pages, in Spanish language

Automatic grammar coaching serves an important purpose of advising on standard grammar varieties while not imposing social pressures or reinforcing established social roles. Such systems already exist but most of them are for English and few of them offer meaningful feedback. Furthermore, they typically rely completely on neural methods and require huge computational resources which most of the world cannot afford. We propose a grammar coaching system for Spanish that relies on (i) a rich linguistic formalism capable of giving informative feedback; and (ii) a faster parsing algorithm which makes using this formalism practical in a real-world application. The approach is feasible for any language for which there is a computerized grammar and is less reliant on expensive and environmentally costly neural methods. We seek to contribute to Greener AI and to address global education challenges by raising the standards of inclusivity and engagement in grammar coaching., Comment: 5 pages, 4 figures, project summary for CEDI-SEPLN Seminar of the Spanish Society for Natural Language Processing at the 7th Spanish Conference on Informatics, June 19-20, 2024, A Coru\~na, Spain

Sentiment analysis is a key technology for companies and institutions to gauge public opinion on products, services or events. However, for large-scale sentiment analysis to be accessible to entities with modest computational resources, it needs to be performed in a resource-efficient way. While some efficient sentiment analysis systems exist, they tend to apply shallow heuristics, which do not take into account syntactic phenomena that can radically change sentiment. Conversely, alternatives that take syntax into account are computationally expensive. The SALSA project, funded by the European Research Council under a Proof-of-Concept Grant, aims to leverage recently-developed fast syntactic parsing techniques to build sentiment analysis systems that are lightweight and efficient, while still providing accuracy and explainability through the explicit use of syntax. We intend our approaches to be the backbone of a working product of interest for SMEs to use in production., Comment: Accepted for publication at SEPLN-CEDI2024: Seminar of the Spanish Society for Natural Language Processing at the 7th Spanish Conference on Informatics

This is a summary of the paper "A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing", which was published in Findings of EMNLP 2023. We evaluate a range of recent state-of-the-art, instruction-tuned large language models (LLMs) on an English creative writing task, and compare them to human writers. For this purpose, we use a specifically-tailored prompt (based on an epic combat between Ignatius J. Reilly, main character of John Kennedy Toole's "A Confederacy of Dunces", and a pterodactyl) to minimize the risk of training data leakage and force the models to be creative rather than reusing existing stories. The same prompt is presented to LLMs and human writers, and evaluation is performed by humans using a detailed rubric including various aspects like fluency, style, originality or humor. Results show that some state-of-the-art commercial LLMs match or slightly outperform our human writers in most of the evaluated dimensions. Open-source LLMs lag behind. Humans keep a close lead in originality, and only the top three LLMs can handle humor at human-like levels., Comment: Published in the XIX Conference of the Spanish Association for Artificial Intelligence (CAEPIA), 2024. Summary of our paper "A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing", published in Findings of EMNLP

Sentiment Analysis (SA) is a crucial aspect of Natural Language Processing (NLP), addressing subjective assessments in textual content. Syntactic parsing is useful in SA because explicit syntactic information can improve accuracy while providing explainability, but it tends to be a computational bottleneck in practice due to the slowness of parsing algorithms. This paper addresses said bottleneck by using a SEquence Labeling Syntactic Parser (SELSP) to inject syntax into SA. By treating dependency parsing as a sequence labeling problem, we greatly enhance the speed of syntax-based SA. SELSP is trained and evaluated on a ternary polarity classification task, demonstrating its faster performance and better accuracy in polarity prediction tasks compared to conventional parsers like Stanza and to heuristic approaches that use shallow syntactic rules for SA like VADER. This increased speed and improved accuracy make SELSP particularly appealing to SA practitioners in both research and industry. In addition, we test several sentiment dictionaries on our SELSP to see which one improves the performance in polarity prediction tasks. Moreover, we compare the SELSP with Transformer-based models trained on a 5-label classification task. The results show that dictionaries that capture polarity judgment variation provide better results than dictionaries that ignore polarity judgment variation. Moreover, we show that SELSP is considerably faster than Transformer-based models in polarity prediction tasks.

In this study, we investigate a novel "dimerized" dielectric metasurface featuring dual-mode resonances governed by symmetry-protected bound states in the continuum (BICs). The metasurface design offers advantages such as insensitivity to incident light polarization and exceptionally high quality factors exceeding 10$^5$ for low and moderate structural deviations from the monoatomic array. By introducing mastered perturbations in the metasurface via the Brillouin zone folding method, without reducing symmetry, we explore the behavior of symmetry-protected BIC states and their polarization-independent responses. Through numerical simulations and analysis, we demonstrate the superiority of Q factors for specific BIC-based resonances, leading to precise control over interaction behaviors and light engineering in both near and far fields. Our findings contribute to the understanding of BIC resonance interactions and offer insights into the design of high-performance sensing applications and meta-devices with enhanced functionalities.

TripAdvisor reviews and comparable data sources play an important role in many tasks in Natural Language Processing (NLP), providing a data basis for the identification and classification of subjective judgments, such as hotel or restaurant reviews, into positive or negative polarities. This study explores three important factors influencing variation in crowdsourced polarity judgments, focusing on TripAdvisor reviews in Spanish. Three hypotheses are tested: the role of Part Of Speech (POS), the impact of sentiment words such as "tasty", and the influence of neutral words like "ok" on judgment variation. The study's methodology employs one-word titles, demonstrating their efficacy in studying polarity variation of words. Statistical tests on mean equality are performed on word groups of our interest. The results of this study reveal that adjectives in one-word titles tend to result in lower judgment variation compared to other word types or POS. Sentiment words contribute to lower judgment variation as well, emphasizing the significance of sentiment words in research on polarity judgments, and neutral words are associated with higher judgment variation as expected. However, these effects cannot be always reproduced in longer titles, which suggests that longer titles do not represent the best data source for testing the ambiguity of single words due to the influence on word polarity by other words like negation in longer titles. This empirical investigation contributes valuable insights into the factors influencing polarity variation of words, providing a foundation for NLP practitioners that aim to capture and predict polarity judgments in Spanish and for researchers that aim to understand factors influencing judgment variation., Comment: Accepted manuscript to be published in PLoS One

This study presents a novel approach to activate a narrowband transparency line within a reflecting broadband window in all-dielectric metasurfaces, in analogy to the electromagnetically-induced transparency effect, by means of a quasi-bound state in the continuum (qBIC). We demonstrate that the resonance overlapping of a bright mode and a qBIC-based nearly-dark mode with distinct Q-factor can be fully governed by a silicon trimer-based unit cell with broken-inversion-symmetry cross shape, thus providing the required response under normal incidence of a linearly-polarized light. Our analysis that is derived from the far-field multipolar decomposition and near-field electromagnetic distributions uncovers the main contributions of different multipoles on the qBIC resonance, with governing magnetic dipole and electric quadrupole terms supplied by distinct parts of the dielectric ``molecule.'' The findings extracted from this research open up new avenues for the development of polarization-dependent technologies, with particular interest in its capabilities for sensing and biosensing.

We study incremental constituent parsers to assess their capacity to output trees based on prefix representations alone. Guided by strictly left-to-right generative language models and tree-decoding modules, we build parsers that adhere to a strong definition of incrementality across languages. This builds upon work that asserted incrementality, but that mostly only enforced it on either the encoder or the decoder. Finally, we conduct an analysis against non-incremental and partially incremental models., Comment: Accepted at EACL 2024

Input-output conformance simulation (iocos) has been proposed by Gregorio-Rodr\'iguez, Llana and Mart\'inez-Torres as a simulation-based behavioural preorder underlying model-based testing. This relation is inspired by Tretmans' classic ioco relation, but has better worst-case complexity than ioco and supports stepwise refinement. The goal of this paper is to develop the theory of iocos by studying logical characterisations of this relation, rule formats for it and its compositionality. More specifically, this article presents characterisations of iocos in terms of modal logics and compares them with an existing logical characterisation for ioco proposed by Beohar and Mousavi. It also offers a characteristic-formula construction for iocos over finite processes in an extension of the proposed modal logics with greatest fixed points. A precongruence rule format for iocos and a rule format ensuring that operations take quiescence properly into account are also given. Both rule formats are based on the GSOS format by Bloom, Istrail and Meyer. The general modal decomposition methodology of Fokkink and van Glabbeek is used to show how to check the satisfaction of properties expressed in the logic for iocos in a compositional way for operations specified by rules in the precongruence rule format for iocos ., Comment: Replaced conference version for extended version (this includes a change in the name of the paper plus additional sections and results)

We study the twisted (co)homology of a family of genus-one integrals -- the so called Riemann-Wirtinger integrals. These integrals are closely related to one-loop string amplitudes in chiral splitting where one leaves the loop-momentum, modulus and all but one puncture un-integrated. While not actual one-loop string integrals, they share many properties and are simple enough that the associated twisted (co)homologies have been completely characterized [Goto2022,arXiv:2206.03177]. Using intersection numbers -- an inner product on the vector space of allowed differential forms -- we derive the Gauss-Manin connection for two bases of the twisted cohomology providing an independent check of [Mano&Watanabe2012]. We also use the intersection index -- an inner product on the vector space of allowed contours -- to derive a double-copy formula for the closed-string analogues of Riemann-Wirtinger integrals (one-dimensional integrals over the torus). Similar to the celebrated KLT formula between open- and closed-string tree-level amplitudes, these intersection indices form a genus-one KLT-like kernel defining bilinears in meromorphic Riemann-Wirtinger integrals that are equal to their complex counterparts., Comment: 45+23 pages, 16 Figures. Updated to match version in JHEP. Text streamlined to focus on double copy result -- progress towards understanding Riemann-Wirtinger integrals in terms of eMPLs has been moved to appendix E

We introduce a new Lie-algebraic approach to explicitly construct the motivic coaction and single-valued map of multiple polylogarithms in any number of variables. In both cases, the appearance of multiple zeta values is controlled by conjugating generating series of polylogarithms with Lie-algebra generators associated with odd zeta values. Our reformulation of earlier constructions of coactions and single-valued polylogarithms preserves choices of fibration bases, exposes the correlation between multiple zeta values of different depths and paves the way for generalizations beyond genus zero., Comment: 13 pages; v2: minor corrections and clarifications; matches published version

We introduce an encoding for parsing as sequence labeling that can represent any projective dependency tree as a sequence of 4-bit labels, one per word. The bits in each word's label represent (1) whether it is a right or left dependent, (2) whether it is the outermost (left/right) dependent of its parent, (3) whether it has any left children and (4) whether it has any right children. We show that this provides an injective mapping from trees to labels that can be encoded and decoded in linear time. We then define a 7-bit extension that represents an extra plane of arcs, extending the coverage to almost full non-projectivity (over 99.9% empirical arc coverage). Results on a set of diverse treebanks show that our 7-bit encoding obtains substantial accuracy gains over the previously best-performing sequence labeling encodings., Comment: Accepted for publication at EMNLP 2023

We evaluate a range of recent LLMs on English creative writing, a challenging and complex task that requires imagination, coherence, and style. We use a difficult, open-ended scenario chosen to avoid training data reuse: an epic narration of a single combat between Ignatius J. Reilly, the protagonist of the Pulitzer Prize-winning novel A Confederacy of Dunces (1980), and a pterodactyl, a prehistoric flying reptile. We ask several LLMs and humans to write such a story and conduct a human evalution involving various criteria such as fluency, coherence, originality, humor, and style. Our results show that some state-of-the-art commercial LLMs match or slightly outperform our writers in most dimensions; whereas open-source LLMs lag behind. Humans retain an edge in creativity, while humor shows a binary divide between LLMs that can handle it comparably to humans and those that fail at it. We discuss the implications and limitations of our study and suggest directions for future research., Comment: Accepted for publication in Findings of EMNLP 2023

Since the popularization of BiLSTMs and Transformer-based bidirectional encoders, state-of-the-art syntactic parsers have lacked incrementality, requiring access to the whole sentence and deviating from human language processing. This paper explores whether fully incremental dependency parsing with modern architectures can be competitive. We build parsers combining strictly left-to-right neural encoders with fully incremental sequence-labeling and transition-based decoders. The results show that fully incremental parsing with modern architectures considerably lags behind bidirectional parsing, noting the challenges of psycholinguistically plausible parsing., Comment: Accepted at IJCNLP-AACL 2023

One-loop scattering amplitudes in string theories involve configuration-space integrals over genus-one surfaces with coefficients of Kronecker-Eisenstein series in the integrand. A conjectural genus-one basis of integrands under Fay identities and integration by parts was recently constructed out of chains of Kronecker-Eisenstein series. In this work, we decompose a variety of more general genus-one integrands into the conjectural chain basis. The explicit form of the expansion coefficients is worked out for infinite families of cases where the Kronecker-Eisenstein series form cycles. Our results can be used to simplify multiparticle amplitudes in supersymmetric, heterotic and bosonic string theories and to investigate loop-level echoes of the field-theory double-copy structures of string tree-level amplitudes. The multitude of basis reductions in this work strongly validate the recently proposed chain basis and stimulate mathematical follow-up studies of more general configuration-space integrals with additional marked points or at higher genus., Comment: 56+8 pages, 9 figures. The companion paper 2403.18078 provides a Mathematica implementation. v2: minimal cosmetic changes, matches the published version in JHEP

We present the latest version of the Spanish Resource Grammar (SRG), a grammar of Spanish implemented in the HPSG formalism. Such grammars encode a complex set of hypotheses about syntax making them a resource for empirical testing of linguistic theory. They also encode a strict notion of grammaticality which makes them a resource for natural language processing applications in computer-assisted language learning. This version of the SRG uses the recent version of the Freeling morphological analyzer and is released along with an automatically created, manually verified treebank of 2,291 sentences. We explain the treebanking process, emphasizing how it is different from treebanking with manual annotation and how it contributes to empirically-driven development of syntactic theory. The treebanks' high level of consistency and detail makes them a resource for training high-quality semantic parsers and generally systems that benefit from precise and detailed semantics. Finally, we present the grammar's coverage and overgeneration on 100 sentences from a learner corpus, a new research line related to developing methodologies for robust empirical evaluation of hypotheses in second language acquisition., Comment: 12 pages, 5 figures

We present an approach for assessing how multilingual large language models (LLMs) learn syntax in terms of multi-formalism syntactic structures. We aim to recover constituent and dependency structures by casting parsing as sequence labeling. To do so, we select a few LLMs and study them on 13 diverse UD treebanks for dependency parsing and 10 treebanks for constituent parsing. Our results show that: (i) the framework is consistent across encodings, (ii) pre-trained word vectors do not favor constituency representations of syntax over dependencies, (iii) sub-word tokenization is needed to represent syntax, in contrast to character-based models, and (iv) occurrence of a language in the pretraining data is more important than the amount of task data when recovering syntax from the word vectors., Comment: Accepted at IJCNLP-AACL 2023

We present new supertaggers trained on English grammar-based treebanks and test the effects of the best tagger on parsing speed and accuracy. The treebanks are produced automatically by large manually built grammars and feature high-quality annotation based on a well-developed linguistic theory (HPSG). The English Resource Grammar treebanks include diverse and challenging test datasets, beyond the usual WSJ section 23 and Wikipedia data. HPSG supertagging has previously relied on MaxEnt-based models. We use SVM and neural CRF- and BERT-based methods and show that both SVM and neural supertaggers achieve considerably higher accuracy compared to the baseline and lead to an increase not only in the parsing speed but also the parser accuracy with respect to gold dependency structures. Our fine-tuned BERT-based tagger achieves 97.26\% accuracy on 950 sentences from WSJ23 and 93.88% on the out-of-domain technical essay The Cathedral and the Bazaar (cb). We present experiments with integrating the best supertagger into an HPSG parser and observe a speedup of a factor of 3 with respect to the system which uses no tagging at all, as well as large recall gains and an overall precision gain. We also compare our system to an existing integrated tagger and show that although the well-integrated tagger remains the fastest, our experimental system can be more accurate. Finally, we hope that the diverse and difficult datasets we used for evaluation will gain more popularity in the field: we show that results can differ depending on the dataset, even if it is an in-domain one. We contribute the complete datasets reformatted for Huggingface token classification., Comment: 16 pages, 3 figures. To appear at EMNLP-2024

This paper summarizes the results of experimenting with Universal Dependencies (UD) adaptation of an Unsupervised, Compositional and Recursive (UCR) rule-based approach for Sentiment Analysis (SA) submitted to the Shared Task at Rest-Mex 2023 (Team Olga/LyS-SALSA) (within the IberLEF 2023 conference). By using basic syntactic rules such as rules of modification and negation applied on words from sentiment dictionaries, our approach exploits some advantages of an unsupervised method for SA: (1) interpretability and explainability of SA, (2) robustness across datasets, languages and domains and (3) usability by non-experts in NLP. We compare our approach with other unsupervised approaches of SA that in contrast to our UCR rule-based approach use simple heuristic rules to deal with negation and modification. Our results show a considerable improvement over these approaches. We discuss future improvements of our results by using modality features as another shifting rule of polarity and word disambiguation techniques to identify the right sentiment words., Comment: Proceedings of IberLEF 2023, Ja\'en, Spain, 2023

We conduct a quantitative analysis contrasting human-written English news text with comparable large language model (LLM) output from six different LLMs that cover three different families and four sizes in total. Our analysis spans several measurable linguistic dimensions, including morphological, syntactic, psychometric, and sociolinguistic aspects. The results reveal various measurable differences between human and AI-generated texts. Human texts exhibit more scattered sentence length distributions, more variety of vocabulary, a distinct use of dependency and constituent types, shorter constituents, and more optimized dependency distances. Humans tend to exhibit stronger negative emotions (such as fear and disgust) and less joy compared to text generated by LLMs, with the toxicity of these models increasing as their size grows. LLM outputs use more numbers, symbols and auxiliaries (suggesting objective language) than human texts, as well as more pronouns. The sexist bias prevalent in human text is also expressed by LLMs, and even magnified in all of them but one. Differences between LLMs and humans are larger than between LLMs., Comment: Published at Artificial Intelligence Review vol. 57, 265

An increasing number of small ventures are launching missions to space with small volume satellite platforms. These small spacecraft are now being seriously considered for deep-space missions, creating a need for ground stations capable of detecting the faint signals they will transmit to Earth. Here, recent developments in room-temperature solid-state masers are reviewed to determine their readiness for use as a cheap low-noise amplifier for deep-space communications. Masers based on Pentacene-doped Para-terphenyl (Pc:PTP), Pentacene-doped Picene, Diazapentacene-doped Para-Terphenyl (DAP:PTP), Phenazine/1,2,4,5-Tetracyanobenzene (PNZ/TCNB) co-crystal, NV Diamond, Cuprous Oxide, and Silicon Carbide are considered for comparison. Pc:PTP offers good spin polarisation density and output power but suffers from thermal dissipation problems, DAP:PTP may help to obtain a lower threshold power than that achieved with Pentacene, PNZ/TCNB stands out in spin polarization density but has not achieved room-temperature masing, and NV Diamond is the only medium to have sustained continuous operation but has very limited power output. The other gain media proposed offer theoretical advantages but have not been tested in a working maser device., Comment: 6 pages, 6 figures

PoS tags, once taken for granted as a useful resource for syntactic parsing, have become more situational with the popularization of deep learning. Recent work on the impact of PoS tags on graph- and transition-based parsers suggests that they are only useful when tagging accuracy is prohibitively high, or in low-resource scenarios. However, such an analysis is lacking for the emerging sequence labeling parsing paradigm, where it is especially relevant as some models explicitly use PoS tags for encoding and decoding. We undertake a study and uncover some trends. Among them, PoS tags are generally more useful for sequence labeling parsers than for other paradigms, but the impact of their accuracy is highly encoding-dependent, with the PoS-based head-selection encoding being best only when both tagging accuracy and resource availability are high., Comment: Accepted at AACL 2022

We contribute to the discussion on parsing performance in NLP by introducing a measurement that evaluates the differences between the distributions of edge displacement (the directed distance of edges) seen in training and test data. We hypothesize that this measurement will be related to differences observed in parsing performance across treebanks. We motivate this by building upon previous work and then attempt to falsify this hypothesis by using a number of statistical methods. We establish that there is a statistical correlation between this measurement and parsing performance even when controlling for potential covariants. We then use this to establish a sampling technique that gives us an adversarial and complementary split. This gives an idea of the lower and upper bounds of parsing systems for a given treebank in lieu of freshly sampled data. In a broader sense, the methodology presented here can act as a reference for future correlation-based exploratory work in NLP., Comment: This is the final peer-reviewed manuscript accepted for publication in Computational Linguistics. The journal version with the final editorial and typesetting changes is available open-access at https://doi.org/10.1162/coli_a_00440

Treebank selection for parsing evaluation and the spurious effects that might arise from a biased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclusions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this experiment, create vast amounts of random subsets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, it is possible to detect potentially harmful strategies., Comment: Accepted at COLING 2022