Search

This paper reports on the shared tasks organized by the 21st IWSLT Conference. The shared tasks address 7 scientific challenges in spoken language translation: simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, dialect and low-resource speech translation, and Indic languages. The shared tasks attracted 18 teams whose submissions are documented in 26 system papers. The growing interest towards spoken language translation is also witnessed by the constantly increasing number of shared task organizers and contributors to the overview paper, almost evenly distributed across industry and academia., Comment: IWSLT 2024; 59 pages

This research is focused on traffic congestion in the small island of Malta which is the most densely populated country in the EU with about 1,672 inhabitants per square kilometre (4,331 inhabitants/sq mi). Furthermore, Malta has a rapid vehicle growth. Based on our research, the number of vehicles increased by around 11,000 in a little more than 6 months, which shows how important it is to have an accurate and comprehensive means of collecting data to tackle the issue of fluctuating traffic in Malta. In this paper, we first present the newly built comprehensive traffic dataset, called MalTra. This dataset includes realistic trips made by members of the public across the island over a period of 200 days. We then describe the methodology we adopted to generate syntactic data to complete our data set as much as possible. In our research, we consider both MalTra and the Q-Traffic dataset, which has been used in several other research studies. The statistical ARIMA model and two graph neural networks, the spatial temporal graph convolutional network (STGCN) and the diffusion convolutional recurrent network (DCRNN) were used to analyse and compare the results with existing research. From the evaluation, we found that the DCRNN model outperforms the STGCN with the former resulting in MAE of 3.98 (6.65 in the case of the latter) and a RMSE of 7.78 (against 12.73 of the latter).

The ideal LLM content moderation system would be both structurally interpretable (so its decisions can be explained to users) and steerable (to reflect a community's values or align to safety standards). However, current systems fall short on both of these dimensions. To address this gap, we present SafetyAnalyst, a novel LLM safety moderation framework. Given a prompt, SafetyAnalyst creates a structured "harm-benefit tree," which identifies 1) the actions that could be taken if a compliant response were provided, 2) the harmful and beneficial effects of those actions (along with their likelihood, severity, and immediacy), and 3) the stakeholders that would be impacted by those effects. It then aggregates this structured representation into a harmfulness score based on a parameterized set of safety preferences, which can be transparently aligned to particular values. Using extensive harm-benefit features generated by SOTA LLMs on 19k prompts, we fine-tuned an open-weight LM to specialize in generating harm-benefit trees through symbolic knowledge distillation. On a comprehensive set of prompt safety benchmarks, we show that our system (average F1=0.75) outperforms existing LLM safety moderation systems (average F1$<$0.72) on prompt harmfulness classification, while offering the additional advantages of interpretability and steerability.

Bayesian optimization (BO) is an attractive machine learning framework for performing sample-efficient global optimization of black-box functions. The optimization process is guided by an acquisition function that selects points to acquire in each round of BO. In batched BO, when multiple points are acquired in parallel, commonly used acquisition functions are often high-dimensional and intractable, leading to the use of sampling-based alternatives. We propose a statistical physics inspired acquisition function for BO with Gaussian processes that can natively handle batches. Batched Energy-Entropy acquisition for BO (BEEBO) enables tight control of the explore-exploit trade-off of the optimization process and generalizes to heteroskedastic black-box problems. We demonstrate the applicability of BEEBO on a range of problems, showing competitive performance to existing methods., Comment: 14 pages (+31 appendix), 21 figures. Accepted at NeurIPS 2024

This paper presents uniGasFoam, an open-source particle-based solver for multiscale rarefied gas flow simulations, which has been developed within the well-established OpenFOAM framework, and is an extension of the direct simulation Monte Carlo (DSMC) solver dsmcFoam+. The developed solver addresses the coupling challenges inherent in hybrid continuum-particle methods, originating from the disparate nature of finite-volume (FV) solvers found in computational fluid dynamics (CFD) software and DSMC particle solvers. This is achieved by employing alternative stochastic particle methods, resembling DSMC, to tackle the continuum limit. The uniGasFoam particle-particle coupling produces a numerical implementation that is simpler and more robust, faster in many steady-state flows, and more scalable for transient flows compared to conventional continuum-particle coupling. The presented framework is unified and generic, and can couple DSMC with stochastic particle (SP) and unified stochastic particle (USP) methods, or be employed for pure DSMC, SP, and USP gas simulations. To enhance user experience, optimise computational resources and minimise user error, advanced adaptive algorithms such as transient adaptive sub-cells, non-uniform cell weighting, and adaptive global time stepping have been integrated into uniGasFoam. In this paper, the hybrid USP-DSMC module of uniGasFoam is rigorously validated through multiple benchmark cases, consistently showing excellent agreement with pure DSMC, hybrid CFD-DSMC, and literature results. Notably, uniGasFoam achieves significant computational gains compared to pure dsmcFoam+ simulations, rendering it a robust computational tool well-suited for addressing multiscale rarefied gas flows of engineering importance.

Interest is growing in artificial empathy, but so is confusion about what artificial empathy is or needs to be. This confusion makes it challenging to navigate the technical and ethical issues that accompany empathic AI development. Here, we outline a framework for thinking about empathic AI based on the premise that different constellations of capabilities associated with empathy are important for different empathic AI applications. We describe distinctions of capabilities that we argue belong under the empathy umbrella, and show how three medical empathic AI use cases require different sets of these capabilities. We conclude by discussing why appreciation of the diverse capabilities under the empathy umbrella is important for both AI creators and users., Comment: To appear at the 7th AAAI/ACM Conference on AI, Ethics, and Society, 2024

Let $\mathcal{F}$ be a set of graphs. A set $D$ of vertices of a graph $G$ is $\mathcal{F}$-isolating if the graph obtained by removing from $G$ all the vertices in $D$ and their neighbors does not have a copy of a graph in $\mathcal{F}$ as a subgraph. The $\mathcal{F}$-isolation number of $G$, denoted by $\iota(G, \mathcal{F})$, is the size of a smallest $\mathcal{F}$-isolating set of $G$. We establish sharp upper bounds on the $\mathcal{F}$-isolation number of a tree in terms of the order and the number of leaves for the case where $\mathcal{F}$ consists of the star $K_{1,k}$ for some $k\geq 1$. Moreover, we characterize all trees attaining the bounds., Comment: 27 pages, 14 figures

[Background/Context] AI assistants like GitHub Copilot are transforming software engineering; several studies have highlighted productivity improvements. However, their impact on code quality, particularly in terms of maintainability, requires further investigation. [Objective/Aim] This study aims to examine the influence of AI assistants on software maintainability, specifically assessing how these tools affect the ability of developers to evolve code. [Method] We will conduct a two-phased controlled experiment involving professional developers. In Phase 1, developers will add a new feature to a Java project, with or without the aid of an AI assistant. Phase 2, a randomized controlled trial, will involve a different set of developers evolving random Phase 1 projects - working without AI assistants. We will employ Bayesian analysis to evaluate differences in completion time, perceived productivity, code quality, and test coverage., Comment: Accepted as In-Principal Acceptance (IPA) for a Stage 1 registration of the Registered Report Track at the 40th IEEE International Conference on Software Maintenance and Evolution (ICSME)

As generative AI is expected to increase global code volumes, the importance of maintainability from a human perspective will become even greater. Various methods have been developed to identify the most important maintainability issues, including aggregated metrics and advanced Machine Learning (ML) models. This study benchmarks several maintainability prediction approaches, including State-of-the-Art (SotA) ML, SonarQube's Maintainability Rating, CodeScene's Code Health, and Microsoft's Maintainability Index. Our results indicate that CodeScene matches the accuracy of SotA ML and outperforms the average human expert. Importantly, unlike SotA ML, CodeScene also provides end users with actionable code smell details to remedy identified issues. Finally, caution is advised with SonarQube due to its tendency to generate many false positives. Unfortunately, our findings call into question the validity of previous studies that solely relied on SonarQube output for establishing ground truth labels. To improve reliability in future maintainability and technical debt studies, we recommend employing more accurate metrics. Moreover, reevaluating previous findings with Code Health would mitigate this revealed validity threat., Comment: Accepted for publication in the Proc. of the 40th International Conference on Software Maintenance and Evolution (ICSME)

Preference elicitation frameworks feature heavily in the research on participatory ethical AI tools and provide a viable mechanism to enquire and incorporate the moral values of various stakeholders. As part of the elicitation process, surveys about moral preferences, opinions, and judgments are typically administered only once to each participant. This methodological practice is reasonable if participants' responses are stable over time such that, all other relevant factors being held constant, their responses today will be the same as their responses to the same questions at a later time. However, we do not know how often that is the case. It is possible that participants' true moral preferences change, are subject to temporary moods or whims, or are influenced by environmental factors we don't track. If participants' moral responses are unstable in such ways, it would raise important methodological and theoretical issues for how participants' true moral preferences, opinions, and judgments can be ascertained. We address this possibility here by asking the same survey participants the same moral questions about which patient should receive a kidney when only one is available ten times in ten different sessions over two weeks, varying only presentation order across sessions. We measured how often participants gave different responses to simple (Study One) and more complicated (Study Two) repeated scenarios. On average, the fraction of times participants changed their responses to controversial scenarios was around 10-18% across studies, and this instability is observed to have positive associations with response time and decision-making difficulty. We discuss the implications of these results for the efficacy of moral preference elicitation, highlighting the role of response instability in causing value misalignment between stakeholders and AI tools trained on their moral judgments., Comment: To appear in AIES 2024

Computational preference elicitation methods are tools used to learn people's preferences quantitatively in a given context. Recent works on preference elicitation advocate for active learning as an efficient method to iteratively construct queries (framed as comparisons between context-specific cases) that are likely to be most informative about an agent's underlying preferences. In this work, we argue that the use of active learning for moral preference elicitation relies on certain assumptions about the underlying moral preferences, which can be violated in practice. Specifically, we highlight the following common assumptions (a) preferences are stable over time and not sensitive to the sequence of presented queries, (b) the appropriate hypothesis class is chosen to model moral preferences, and (c) noise in the agent's responses is limited. While these assumptions can be appropriate for preference elicitation in certain domains, prior research on moral psychology suggests they may not be valid for moral judgments. Through a synthetic simulation of preferences that violate the above assumptions, we observe that active learning can have similar or worse performance than a basic random query selection method in certain settings. Yet, simulation results also demonstrate that active learning can still be viable if the degree of instability or noise is relatively small and when the agent's preferences can be approximately represented with the hypothesis class used for learning. Our study highlights the nuances associated with effective moral preference elicitation in practice and advocates for the cautious use of active learning as a methodology to learn moral preferences., Comment: To appear in AIES 2024

A copy of a graph $F$ is called an $F$-copy. For any graph $G$, the $F$-isolation number of $G$, denoted by $\iota(G,F)$, is the size of a smallest subset $D$ of the vertex set of $G$ such that the closed neighbourhood $N[D]$ of $D$ in $G$ intersects the vertex sets of the $F$-copies contained by $G$ (equivalently, $G-N[D]$ contains no $F$-copy). Thus, $\iota(G,K_1)$ is the domination number $\gamma(G)$ of $G$, and $\iota(G,K_2)$ is the vertex-edge domination number of $G$. We prove that if $F$ is a $k$-edge graph, $\gamma(F) = 1$ (that is, $F$ has a vertex that is adjacent to all the other vertices of $F$), and $G$ is a connected $m$-edge graph, then $\iota(G,F) \leq \big\lfloor \frac{m+1}{k+2} \big\rfloor$ unless $G$ is an $F$-copy or $F$ is a $3$-path and $G$ is a $6$-cycle. This was recently posed as a conjecture by Zhang and Wu, who settled the case where $F$ is a star. The result for the case where $F$ is a clique had been obtained by Fenech, Kaemawichanurat and the present author. The bound is attainable for any $m \geq 0$ unless $1 \leq m = k \leq 2$. New ideas, including divisibility considerations, are introduced in the proof of the conjecture., Comment: 10 pages, minor corrections made. arXiv admin note: text overlap with arXiv:2304.10659, arXiv:2303.13709

Deep learning-based object detection algorithms enable the simultaneous classification and localization of any number of objects in image data. Many of these algorithms are capable of operating in real-time on high resolution images, attributing to their widespread usage across many fields. We present an end-to-end object detection pipeline designed for real-time rare event searches for the Migdal effect, using high-resolution image data from a state-of-the-art scientific CMOS camera in the MIGDAL experiment. The Migdal effect in nuclear scattering, crucial for sub-GeV dark matter searches, has yet to be experimentally confirmed, making its detection a primary goal of the MIGDAL experiment. Our pipeline employs the YOLOv8 object detection algorithm and is trained on real data to enhance the detection efficiency of nuclear and electronic recoils, particularly those exhibiting overlapping tracks that are indicative of the Migdal effect. When deployed online on the MIGDAL readout PC, we demonstrate our pipeline to process and perform the rare event search on 2D image data faster than the peak 120 frame per second acquisition rate of the CMOS camera. Applying these same steps offline, we demonstrate that we can reduce a sample of 20 million camera frames to around 1000 frames while maintaining nearly all signal that YOLOv8 is able to detect, thereby transforming a rare search into a much more manageable search. Our studies highlight the potential of pipelines similar to ours significantly improving the detection capabilities of experiments requiring rapid and precise object identification in high-throughput data environments.

Can the right requirements boost developer satisfaction and happiness? We believe they can. In keeping with this issue's theme, "Well-Being for Resilience: Developers Thrive," we discuss the connection between the three keywords, well-being, resilience, and thriving. How could requirements engineering foster these qualities? While there hasn't been much research on this topic, we see opportunities for future work. Let's initiate the discussion!, Comment: 5 pages

While recommender systems with multi-modal item representations (image, audio, and text), have been widely explored, learning recommendations from multi-modal user interactions (e.g., clicks and speech) remains an open problem. We study the case of multi-modal user interactions in a setting where users engage with a service provider through multiple channels (website and call center). In such cases, incomplete modalities naturally occur, since not all users interact through all the available channels. To address these challenges, we publish a real-world dataset that allows progress in this under-researched area. We further present and benchmark various methods for leveraging multi-modal user interactions for item recommendations, and propose a novel approach that specifically deals with missing modalities by mapping user interactions to a common feature space. Our analysis reveals important interactions between the different modalities and that a frequently occurring modality can enhance learning from a less frequent one.

Continuum fluid dynamic models based on the Navier-Stokes equations have previously been used to simulate granular media undergoing fluid-like shearing. These models, however, typically fail to predict the flow behaviour in confined environments as non-equilibrium particle effects dominate near walls. We adapt an extended hydrodynamic model for granular flows, which uses a density-gradient dependent ``volume diffusion'' term to correct the viscous stress tensor and heat flux, to simulate the shearing of a granular gas between two rough walls, and with corresponding boundary conditions. We use our volume diffusion model to predict channel flows for a range of mean volume fraction $\bar{\phi}=0.01$--$0.4$, and inter-particle coefficients of restitution $e=0.8$ and $0.9$, and compare with Discrete Element Method (DEM) simulations and classical Navier-Stokes equations. Our model is capable of predicting non-uniform pressure, volume fraction and granular temperature, which become more significant for cases with mean volume fraction $\bar{\phi}\sim0.1$, in which we typically observe non-uniform peak density variations, and large volume fraction gradients.

Providing personalized recommendations for insurance products is particularly challenging due to the intrinsic and distinctive features of the insurance domain. First, unlike more traditional domains like retail, movie etc., a large amount of user feedback is not available and the item catalog is smaller. Second, due to the higher complexity of products, the majority of users still prefer to complete their purchases over the phone instead of online. We present different recommender models to address such data scarcity in the insurance domain. We use recurrent neural networks with 3 different types of loss functions and architectures (cross-entropy, censored Weibull, attention). Our models cope with data scarcity by learning from multiple sessions and different types of user actions. Moreover, differently from previous session-based models, our models learn to predict a target action that does not happen within the session. Our models outperform state-of-the-art baselines on a real-world insurance dataset, with ca. 44K users, 16 items, 54K purchases and 117K sessions. Moreover, combining our models with demographic data boosts the performance. Analysis shows that considering multiple sessions and several types of actions are both beneficial for the models, and that our models are not unfair with respect to age, gender and income., Comment: arXiv admin note: substantial text overlap with arXiv:2211.15360

EU directives stipulate a systematic follow-up of train delays. In Sweden, the Swedish Transport Administration registers and assigns an appropriate delay attribution code. However, this delay attribution code is assigned manually, which is a complex task. In this paper, a machine learning-based decision support for assigning delay attribution codes based on event descriptions is investigated. The text is transformed using TF-IDF, and two models, Random Forest and Support Vector Machine, are evaluated against a random uniform classifier and the classification performance of the Swedish Transport Administration. Further, the problem is modeled as both a hierarchical and flat approach. The results indicate that a hierarchical approach performs better than a flat approach. Both approaches perform better than the random uniform classifier but perform worse than the manual classification., Comment: 22 pages, 7 figures

Safety measures need to be systemically investigated to what extent they evaluate the intended performance of Deep Neural Networks (DNNs) for critical applications. Due to a lack of verification methods for high-dimensional DNNs, a trade-off is needed between accepted performance and handling of out-of-distribution (OOD) samples. This work evaluates rejecting outputs from semantic segmentation DNNs by applying a Mahalanobis distance (MD) based on the most probable class-conditional Gaussian distribution for the predicted class as an OOD score. The evaluation follows three DNNs trained on the Cityscapes dataset and tested on four automotive datasets and finds that classification risk can drastically be reduced at the cost of pixel coverage, even when applied on unseen datasets. The applicability of our findings will support legitimizing safety measures and motivate their usage when arguing for safe usage of DNNs in automotive perception., Comment: Preprint to 2023 IEEE International Conference On Artificial Intelligence Testing

Although multilingual language models exhibit impressive cross-lingual transfer capabilities on unseen languages, the performance on downstream tasks is impacted when there is a script disparity with the languages used in the multilingual model's pre-training data. Using transliteration offers a straightforward yet effective means to align the script of a resource-rich language with a target language, thereby enhancing cross-lingual transfer capabilities. However, for mixed languages, this approach is suboptimal, since only a subset of the language benefits from the cross-lingual transfer while the remainder is impeded. In this work, we focus on Maltese, a Semitic language, with substantial influences from Arabic, Italian, and English, and notably written in Latin script. We present a novel dataset annotated with word-level etymology. We use this dataset to train a classifier that enables us to make informed decisions regarding the appropriate processing of each token in the Maltese language. We contrast indiscriminate transliteration or translation to mixing processing pipelines that only transliterate words of Arabic origin, thereby resulting in text with a mixture of scripts. We fine-tune the processed data on four downstream tasks and show that conditional transliteration based on word etymology yields the best results, surpassing fine-tuning with raw Maltese or Maltese processed with non-selective pipelines., Comment: EACL 2024 camera-ready version

Understanding and effectively managing Technical Debt (TD) remains a vital challenge in software engineering. While many studies on code-level TD have been published, few illustrate the business impact of low-quality source code. In this study, we combine two publicly available datasets to study the association between code quality on the one hand, and defect count and implementation time on the other hand. We introduce a value-creation model, derived from regression analyses, to explore relative changes from a baseline. Our results show that the associations vary across different intervals of code quality. Furthermore, the value model suggests strong non-linearities at the extremes of the code quality spectrum. Most importantly, the model suggests amplified returns on investment in the upper end. We discuss the findings within the context of the "broken windows" theory and recommend organizations to diligently prevent the introduction of code smells in files with high churn. Finally, we argue that the value-creation model can be used to initiate discussions regarding the return on investment in refactoring efforts., Comment: Paper accepted at the 7th International Conference on Technical Debt 2024, Lisbon, Portugal, May 14-15, 2024. The replication package is available here: https://zenodo.org/records/10560722

Quality requirements are critical for successful software engineering, with maintainability being a key internal quality. Despite significant attention in software metrics research, maintainability has attracted surprisingly little focus in the Requirements Engineering (RE) community. This position paper proposes a synergistic approach, combining code-oriented research with RE expertise, to create meaningful industrial impact. We introduce six illustrative use cases and propose three future research directions. Preliminary findings indicate that the established QUPER model, designed for setting quality targets, does not adequately address the unique aspects of maintainability., Comment: Accepted for the 1st Workshop on Multi-disciplinary, Open, and RElevant Requirements Engineering (MO2RE), 2024

The Washington Student Achievement Council (WSAC) has prepared this analysis in collaboration with the State Board for Community and Technical Colleges (SBCTC), the Workforce Training and Education Coordinating Board (Workforce Board), and the Association of Washington Business, as part of a broad educational needs assessment outlined in RCW 28B.77.080. The report has four primary purposes: (1) to assess how well the state's postsecondary education system is aligned with the demands of the labor market, (2) to identify key drivers of employer demand in the various occupational clusters, (3) to highlight industries and occupational fields in which students and adult learners may find expanding employment opportunities, and (4) to explore factors, such as the advance of workplace automation, which may be impacting the labor market. [This report was co-produced with Association of Washington Business (AWB).]

A new Micro-Macro-Surrogate (MMS) hybrid method is presented that couples the Direct Simulation Monte Carlo (DSMC) method with Computational Fluid Dynamics (CFD) to simulate low-speed rarefied gas flows. The proposed MMS method incorporates surrogate modelling instead of direct coupling of DSMC data with the CFD, addressing the limitations CFD has in accurately modelling rarefied gas flows, the computational cost of DSMC for low-speed and multiscale flows, as well as the pitfalls of noise in conventional direct coupling approaches. The surrogate models, trained on the DSMC data using Bayesian inference, provide noise-free and accurate corrections to the CFD simulation enabling it to capture the non-continuum physics. The MMS hybrid approach is validated by simulating low-speed, force-driven rarefied gas flows in a canonical parallel-plate system and shows excellent agreement with DSMC benchmark results. A comparison with the typical domain decomposition DSMC-CFD hybrid method is also presented, to demonstrate the advantages of noise-avoidance in the proposed approach. The method also inherently captures the uncertainty arising from micro-model fluctuations, allowing for the quantification of noise-related uncertainty in the predictions. The proposed MMS method demonstrates the potential to enable multiscale simulations where CFD is inaccurate and DSMC is prohibitively expensive.

Although the convergence of high-performance computing, automation, and machine learning has significantly altered the materials design timeline, transformative advances in functional materials and acceleration of their design will require addressing the deficiencies that currently exist in materials informatics, particularly a lack of standardized experimental data management. The challenges associated with experimental data management are especially true for combinatorial materials science, where advancements in automation of experimental workflows have produced datasets that are often too large and too complex for human reasoning. The data management challenge is further compounded by the multi-modal and multi-institutional nature of these datasets, as they tend to be distributed across multiple institutions and can vary substantially in format, size, and content. To adequately map a materials design space from such datasets, an ideal materials data infrastructure would contain data and metadata describing i) synthesis and processing conditions, ii) characterization results, and iii) property and performance measurements. Here, we present a case study for the low-barrier development of such a dashboard that enables standardized organization, analysis, and visualization of a large data lake consisting of combinatorial datasets of synthesis and processing conditions, X-ray diffraction patterns, and materials property measurements generated at several different institutions. While this dashboard was developed specifically for data-driven thermoelectric materials discovery, we envision the adaptation of this prototype to other materials applications, and, more ambitiously, future integration into an all-encompassing materials data management infrastructure.

We propose a human-centred explanation method for rule-based automated decision-making systems in the legal domain. Firstly, we establish a conceptual framework for developing explanation methods, representing its key internal components (content, communication and adaptation) and external dependencies (decision-making system, human recipient and domain). Secondly, we propose an explanation method that uses a graph database to enable question-driven explanations and multimedia display. This way, we can tailor the explanation to the user. Finally, we show how our conceptual framework is applicable to a real-world scenario at the Dutch Tax and Customs Administration and implement our explanation method for this scenario., Comment: This is the full version of a demo at the 36th International Conference on Legal Knowledge and Information Systems (JURIX'23)

Given a set $\mathcal{F}$ of graphs, we call a copy of a graph in $\mathcal{F}$ an $\mathcal{F}$-graph. The $\mathcal{F}$-isolation number of a graph $G$, denoted by $\iota(G,\mathcal{F})$, is the size of a smallest subset $D$ of the vertex set $V(G)$ such that the closed neighbourhood of $D$ intersects the vertex sets of the $\mathcal{F}$-graphs contained by $G$ (equivalently, $G - N[D]$ contains no $\mathcal{F}$-graph). Thus, $\iota(G,\{K_1\})$ is the domination number of $G$. The second author showed that if $\mathcal{F}$ is the set of cycles and $G$ is a connected $n$-vertex graph that is not a triangle, then $\iota(G,\mathcal{F}) \leq \left \lfloor \frac{n}{4} \right \rfloor$. This bound is attainable for every $n$ and solved a problem of Caro and Hansberg. A question that arises immediately is how much smaller an upper bound can be if $\mathcal{F} = \{C_k\}$ for some $k \geq 3$, where $C_k$ is a cycle of length $k$. The problem is to determine the smallest real number $c_k$ (if it exists) such that for some finite set $\mathcal{E}_k$ of graphs, $\iota(G, \{C_k\}) \leq c_k |V(G)|$ for every connected graph $G$ that is not an $\mathcal{E}_k$-graph. The above-mentioned result yields $c_3 = \frac{1}{4}$ and $\mathcal{E}_3 = \{C_3\}$. The second author also showed that if $k \geq 5$ and $c_k$ exists, then $c_k \geq \frac{2}{2k + 1}$. We prove that $c_4 = \frac{1}{5}$ and determine $\mathcal{E}_4$, which consists of three $4$-vertex graphs and six $9$-vertex graphs. The $9$-vertex graphs in $\mathcal{E}_4$ were fully determined by means of a computer program. A method that has the potential of yielding similar results is introduced., Comment: 16 pages, 1 figure, minor corrections made