Search

The ever-improving quality of LLMs has fueled the growth of a diverse range of downstream tasks, leading to an increased demand for AI automation and a burgeoning interest in developing foundation model (FM)-based autonomous agents. As AI agent systems tackle more complex tasks and evolve, they involve a wider range of stakeholders, including agent users, agentic system developers and deployers, and AI model developers. These systems also integrate multiple components such as AI agent workflows, RAG pipelines, prompt management, agent capabilities, and observability features. In this case, obtaining reliable outputs and answers from these agents remains challenging, necessitating a dependable execution process and end-to-end observability solutions. To build reliable AI agents and LLM applications, it is essential to shift towards designing AgentOps platforms that ensure observability and traceability across the entire development-to-production life-cycle. To this end, we conducted a rapid review and identified relevant AgentOps tools from the agentic ecosystem. Based on this review, we provide an overview of the essential features of AgentOps and propose a comprehensive overview of observability data/traceable artifacts across the agent production life-cycle. Our findings provide a systematic overview of the current AgentOps landscape, emphasizing the critical role of observability/traceability in enhancing the reliability of autonomous agent systems., Comment: 19 pages, 9 figures

The pull-based development model facilitates global collaboration within open-source software projects. Most research on the pull request decision-making process explored factors within projects, not the broader software ecosystem they comprise. We uncover ecosystem-wide factors that influence pull request acceptance decisions. We collected a dataset of approximately 1.8 million pull requests and 2.1 million issues from 20,052 GitHub projects within the NPM ecosystem. Of these, 98% depend on another project in the dataset, enabling studying collaboration across dependent projects. We employed social network analysis to create a collaboration network in the ecosystem, and mixed effects logistic regression and random forest techniques to measure the impact and predictive strength of the tested features. We find that gaining experience within the software ecosystem through active participation in issue-tracking systems, submitting pull requests, and collaborating with pull request integrators and experienced developers benefits all open-source contributors, especially project newcomers. The results show that combining ecosystem-wide factors with features studied in previous work to predict the outcome of pull requests reached an overall F1 score of 0.92., Comment: 34 pages, 2 figures, 4 tables

Simulation is a common approach to predict the effect of business process changes on quantitative performance. The starting point of Business Process Simulation (BPS) is a process model enriched with simulation parameters. To cope with the typically large parameter spaces of BPS models, several methods have been proposed to automatically discover BPS models from event logs. Virtually all these approaches neglect the data perspective of business processes. Yet, the data attributes manipulated by a business process often determine which activities are performed, how many times, and when. This paper addresses this gap by introducing a data-aware BPS modeling approach and a method to discover data-aware BPS models from event logs. The BPS modeling approach supports three types of data attributes (global, case-level, and event-level) as well as deterministic and stochastic attribute update rules and data-aware branching conditions. An empirical evaluation shows that the proposed method accurately discovers the type of each data attribute and its associated update rules, and that the resulting BPS models more closely replicate the process execution control flow relative to data-unaware BPS models.

In the evolving landscape of AI regulation, it is crucial for companies to conduct impact assessments and document their compliance through comprehensive reports. However, current reports lack grounding in regulations and often focus on specific aspects like privacy in relation to AI systems, without addressing the real-world uses of these systems. Moreover, there is no systematic effort to design and evaluate these reports with both AI practitioners and AI compliance experts. To address this gap, we conducted an iterative co-design process with 14 AI practitioners and 6 AI compliance experts and proposed a template for impact assessment reports grounded in the EU AI Act, NIST's AI Risk Management Framework, and ISO 42001 AI Management System. We evaluated the template by producing an impact assessment report for an AI-based meeting companion at a major tech company. A user study with 8 AI practitioners from the same company and 5 AI compliance experts from industry and academia revealed that our template effectively provides necessary information for impact assessments and documents the broad impacts of AI systems. Participants envisioned using the template not only at the pre-deployment stage for compliance but also as a tool to guide the design stage of AI uses., Comment: 16 pages, 6 figures

Smart-city services are typically developed as closed systems within each city's vertical, communicating and interacting with cloud services while remaining isolated within each provider's domain. With the emergence of 5G private domains and the introduction of new M2M services focusing on autonomous systems, there is a shift from the cloud-based approach to a distributed edge computing paradigm, in a \textit{continuum} orchestration. However, an essential component is missing. Current orchestration tools, designed for cloud-based deployments, lack robust workload isolation, fail to meet timing constraints, and are not tailored to the resource-constrained nature of edge devices. Therefore, new orchestration methods are needed to support MEC environments. The work presented in this paper addresses this gap. Based on the real needs of a smart-city testbed - the Aveiro Living Lab-, we developed a set of orchestration components to facilitate the seamless orchestration of both cloud and edge-based services, encompassing both critical and non-critical services. This work extends the current Kubernetes orchestration platform to include a novel location-specific resource definition, a custom scheduler to accommodate real-time and legacy services, continuous service monitoring to detect sub-optimal states, and a refined load balancing mechanism that prioritizes the fastest response times.

Today's visualization tools for conveying the risks and benefits of AI technologies are largely tailored for those with technical expertise. To bridge this gap, we have developed a visualization that employs narrative patterns and interactive elements, enabling the broader public to gradually grasp the diverse risks and benefits associated with AI. Using a dataset of 54 real-world incidents involving AI in mobile computing, we examined design choices that enhance public understanding and provoke reflection on how certain AI applications - even those deemed low-risk by law - can still lead to significant incidents. Visualization: https://social-dynamics.net/mobile-ai-risks, Comment: 8 pages, 3 figures

Software has the potential to be a key driver in fostering sustainability. Despite this potential, it is not clear if and how the software industry integrates consideration of sustainability into its common software development processes. This research starts by investigating the current state of sustainability consideration within the software engineering industry through a survey. The results highlight a lack of progress in practically integrating sustainability considerations into software development activities. To address this gap, a case study with an industry partner is conducted to demonstrate how sustainability concerns and effects can be integrated into agile software development. The findings of this case study demonstrate practical approaches to integrating sustainability into software development practices. Reflecting on the findings from the survey and the case study, we note some insights on scaling up the adoption of sustainability consideration into the daily practice of agile software development., Comment: 9 pages, FSE 2024 Workshop on Software Engineering 2030

This paper describes the iterative use of service learning to develop, review, and improve computing-based artifacts. It is well-known that computing students benefit from service-learning experiences as do the community partners. It is also well-known that computing artifacts rarely function well long-term without versioning and updates. Service-learning projects are often one-time engagements, completed by single teams of students over the course of a semester course. This limits the benefit for community partners that do not have the expertise or resources to review and update a project on their own. Over several years, teams of undergraduate students in a capstone course created tailored social media plans for numerous small rural organizations. The projects were required to meet client specific needs, with identified audiences, measurable goals, and strategies and tactics to reach the identified goals. This paper builds on previously results for 60 projects conducted over several years. Nine clients were selected to participate in the iterative follow-up process, where new student teams conducted client interviews, reviewed the initial plans, and analyzed metrics from the current strategies and tactics to provide updated, improved artifacts. Using ABET learning objectives as a basis, clients reviewed the student teams and artifacts. This longitudinal study discusses the impact of this intervention to increase implementation and sustained use rates of computing artifacts developed through service learning. Both students and clients reported high satisfaction levels, and clients were particularly satisfied with the iterative improvement process. This research demonstrates an innovative practice for creating and maintaining computing artifacts through iterative service learning, while addressing the resource constraints of small organizations., Comment: 9 pages, 0 figures

With the urgent need to secure supply chains among Open Source libraries, attention has focused on mitigating vulnerabilities detected in these libraries. Although awareness has improved recently, most studies still report delays in the mitigation process. This suggests that developers still have to deal with other contributions that occur during the period of fixing vulnerabilities, such as coinciding Pull Requests (PRs) and Issues, yet the impact of these contributions remains unclear. To characterize these contributions, we conducted a mixed-method empirical study to analyze NPM GitHub projects affected by 554 different vulnerability advisories, mining a total of 4,699 coinciding PRs and Issues. We believe that tool development and improved workload management for developers have the potential to create a more efficient and effective vulnerability mitigation process., Comment: 6 pages, 3 figures, 3 tables, 22nd IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2024)

Software security has been an important research topic over the years. The community has proposed processes and tools for secure software development and security analysis. However, a significant number of vulnerabilities remains in real-world software-driven systems and products. To alleviate this problem, legislation is being established to oblige manufacturers, for example, to comply with essential security requirements and to establish appropriate development practices. We argue that software engineering research needs to provide better tools and support that helps industry comply with the new standards while retaining effcient processes. We argue for a stronger cooperation between legal scholars and computer scientists, and for bridging the gap between higher-level regulation and code-level engineering., Comment: 5 pages, submitted to SE2023 workshop at FSE 2024

The rapid integration of Large Language Models (LLMs) into various industries presents both revolutionary opportunities and unique challenges. This research aims to establish a scalable and efficient framework for LLM customization, exploring how DevOps practices should be adapted to meet the specific demands of LLM customization. By integrating ontologies, knowledge maps, and prompt engineering into the DevOps pipeline, we propose a robust framework that enhances continuous learning, seamless deployment, and rigorous version control of LLMs. This methodology is demonstrated through the development of a domain-specific chatbot for the agricultural sector, utilizing heterogeneous data to deliver actionable insights. The proposed methodology, so called DOLLmC, not only addresses the immediate challenges of LLM customization but also promotes scalability and operational efficiency. However, the methodology's primary limitation lies in the need for extensive testing, validation, and broader adoption across different domains., Comment: 20 pages, 1 figure, 1 table

Context: Scholars in the software engineering (SE) research community have investigated career advancement in the software industry. Research topics have included how individual and external factors can impact career mobility of software professionals, and how gender affects career advancement. However, the community has yet to look at career mobility from the lens of accessibility. Specifically, there is a pressing need to illuminate the factors that hinder the career mobility of blind and low vision software professionals (BLVSPs). Objective: This study aims to understand aspects of the workplace that impact career mobility for BLVSPs. Methods: We interviewed 26 BLVSPs with different roles, years of experience, and industry sectors. Thematic analysis was used to identify common factors related to career mobility. Results: We found four factors that impacted the career mobility of BLVSPs: (1) technical challenges, (2) colleagues' perceptions of BLVSPs, (3) BLVSPs' own perceptions on managerial progression, and (4) BLVSPs' investment in accessibility at the workplace. Conclusion: We suggest implications for tool designers, organizations, and researchers towards fostering more accessible workplaces to support the career mobility of BLVSPs., Comment: 12 pages, 1 table, conference paper, 2024 ACM / IEEE 17th International Conference on Cooperative and Human Aspects of Software Engineering

The Scrum framework has gained widespread adoption in the industry for its emphasis on collaboration and continuous improvement. However, it has not reached a similar relevance in Software Engineering (SE) curricula. This work reports the experience of five editions of a SE course within an MSc. Degree in Computer Engineering. The course primary educational objective is to provide students with the skills to manage software development projects with Scrum. The course is based on the execution of a team project and on the definition of qualitative and quantitative means of assessment of the application of Scrum. The conduction of five editions of the course allowed us to identify several lessons learned about time budgeting and team compositions in agile student projects and its evidence of the applicability of the framework to software development courses., Comment: Submitted to LEARNER 2024 workshop

Hybrid work, a fusion of different work environments that allow employees to work in and outside their offices, represents a new frontier for agile researchers to explore. However, due to the nascent nature of the research phenomena, we are yet to achieve a good understanding of the research terrain formulated when hybrid work meets agile software development. This systematic mapping study, we aimed to provide a good understanding of this emerging research area. The systematic process we followed led to a collection of 12 primary studies, which is less than what we expected. All the papers are empirical studies, with most of them employing case studies as the research methodology. The people-centric nature of agile methods is yet to be adequately reflected in the studies in this area. Similarly, there is a lack of a richer understanding of hybrid work in terms of flexible work arrangements. Our mapping study identified various research opportunities that can be explored in future research., Comment: 11 pages

Cyber-Physical Production Systems (CPPSs), such as automated car manufacturing plants, execute a configurable sequence of production steps to manufacture products from a product portfolio. In CPPS engineering, domain experts start with manually determining feasible production step sequences and resources based on implicit knowledge. This process is hard to reproduce and highly inefficient. In this paper, we present the Extended Iterative Process Sequence Exploration (eIPSE) approach to derive variability models for products, processes, and resources from a domain-specific description. To automate the integrated exploration and configuration process for a CPPS, we provide a toolchain which automatically reduces the configuration space and allows to generate CPPS artifacts, such as control code for resources. We evaluate the approach with four real-world use cases, including the generation of control code artifacts, and an observational user study to collect feedback from engineers with different backgrounds. The results confirm the usefulness of the eIPSE approach and accompanying prototype to straightforwardly configure a desired CPPS., Comment: 26 pages, 10 figures

A number of methods for large-scale agile development have recently been suggested. Much of the advice in agile methods focuses on teamwork. Prior research has established that teamwork quality influences project success both for traditional software development teams and agile teams. Further, prior studies have also suggested that teamwork quality may play out differently in large projects compared to small. We investigated the relationship between teamwork quality and project success with a survey of 196 project participants across 34 teams in four projects, replicating a previous study on single teams. The new data do not fit the previously established theoretical model, which raises several concerns. The observed effect of teamwork quality on project success operates differently across projects. We discuss possible reasons, which include disagreements on what characterises success in large-scale agile development, "concept drift" of teamwork quality factors, the possibility that interteam factors might have more influence on project success than intrateam factors, and finally, that our study design does not capture all relevant levels and functions. We conclude with a call for more studies on the quality and frequency of interaction between teams in addition to internal team factors to further advance theory and practice within large-scale agile software development.

Published studies on agile effort estimation predominantly focus on comparisons of the accuracy of different estimation methods, while efficiency comparisons, i.e. how much time the estimation methods consume was not in the forefront. However, for practical use in software development, the time required can be a very important cost factor for enterprises, especially when the accuracy of different agile effort estimations is similar. In this study, we thus try to advance the current standard accuracy comparison between methods by introducing efficiency i.e. time it takes to use a method as an additional dimension of comparison. We conduct this comparison between three agile effort estimation methods that were not yet compared in the literature, namely Planning Poker, Bucket System and Affinity Estimation. For the comparison, we used eight student teams with 29 students that had to use all the effort estimation methods during the course where they had to finish a programming project in 3 weeks. The results indicate that after the students get used to using the different methods the accuracy between them is not statistically significantly different, however, the efficiency is. On average Bucket System and Affinity Estimation methods take half as much time as Planning Poker.

Cloud computing has emerged as a crucial solution for handling data- and compute-intensive workflows, offering scalability to address dynamic demands. However, ensuring the secure execution of workflows in the untrusted multi-cloud environment poses significant challenges, given the sensitive nature of the involved data and tasks. The lack of comprehensive approaches for detecting attacks during workflow execution, coupled with inadequate measures for reacting to security and privacy breaches has been identified in the literature. To close this gap, in this work, we propose an approach that focuses on monitoring cloud services and networks to detect security violations during workflow executions. Upon detection, our approach selects the optimal adaptation action to minimize the impact on the workflow. To mitigate the uncertain cost associated with such adaptations and their potential impact on other tasks in the workflow, we employ adaptive learning to determine the most suitable adaptation action. Our approach is evaluated based on the performance of the detection procedure and the impact of the selected adaptations on the workflows., Comment: 18 pages, 3 figures, 4 tables

Open Source Software (OSS) communities often resist regulation typical of traditional organizations. Yet formal governance systems are being increasingly adopted among communities, particularly through non-profit mentor foundations. Our study looks at the Apache Software Foundation Incubator program and 208 projects it supports. We assemble a scalable, semantic pipeline to discover and analyze the governance behavior of projects from their mailing lists. We then investigate the reception of formal policies among communities, through their own governance priorities and internalization of the policies. Our findings indicate that while communities observe formal requirements and policies as extensively as they are defined, their day-to-day governance focus does not dwell on topics that see most formal policy-making. Moreover formalization, be it dedicating governance focus or adopting policy, has limited association with project sustenance.

The software bill of materials (SBOM) concept aims to include more information about a software build such as copyrights, dependencies and security references. But SBOM lacks visibility into the process for building a package. Efforts such as Supply-chain Levels for Software Artifacts (SLSA) try to remedy this by focusing on the quality of the build process. But they lack quantitative assessment of that quality. They are purely qualitative. A new form of assurance case and new technique for structuring it, called process reduction, are presented. An assurance case for a toolchain is quantitative and when structured as a process reduction can measure the strength of the toolchain via the strength of the reduction. An example is given for a simple toolchain., Comment: 7 pages, 3 figures

During software development, balancing security and non security issues is challenging. We focus on security awareness and approaches taken by non-security experts using software development issue trackers when considering security. We first analyse interfaces from prominent issue trackers to see how they support security communication and how they integrate security scoring. Then, we investigate through a small scale user study what criteria developers take when prioritising issues, in particular observing their attitudes to security. We find projects make reference to CVSS summaries (Common Vulnerability Scoring System), often alongside CVE reports (Common Vulnerabilities and Exposures), but issue trackers do not often have interfaces designed for this. Users in our study were not comfortable with CVSS analysis, though were able to reason in a manner compatible with CVSS. Detailed explanations and advice were seen as helpful in making security decisions. This suggests that adding improvements to communication through CVSS-like questioning in issue tracking software can elicit better security interactions., Comment: 9th Workshop on Security Information Workers WSIW 2023 - Co-located Workshop at SOUPS 2023, Anaheim, United States

Virtual Reality (VR) technology has become increasingly popular in recent years as a key enabler of the Metaverse. VR applications have unique characteristics, including the revolutionized human-computer interaction mechanisms, that distinguish them from traditional software. Hence, user expectations for the software quality of VR applications diverge from those for traditional software. Investigating these quality expectations is crucial for the effective development and maintenance of VR applications, which remains an under-explored area in prior research. To bridge the gap, we conduct the first large-scale empirical study to model the software quality of VR applications from users' perspectives. To this end, we analyze 1,132,056 user reviews of 14,150 VR applications across seven app stores through a semiautomatic review mining approach. We construct a taxonomy of 12 software quality attributes that are of major concern to VR users. Our analysis reveals that the VR-specific quality attributes are of utmost importance to users, which are closely related to the most unique properties of VR applications like revolutionized interaction mechanisms and immersive experiences. Our examination of relevant user complaints reveals the major factors impacting user satisfaction with VR-specific quality attributes. We identify that poor design or implementation of the movement mechanisms, control mechanisms, multimedia systems, and physics, can significantly degrade the user experience. Moreover, we discuss the implications of VR quality assurance for both developers and researchers to shed light on future work. For instance, we suggest developers implement sufficient accessibility and comfort options for users with mobility limitations, sensory impairments, and other specific needs to customize the interaction mechanisms. Our datasets and results will be released to facilitate follow-up studies.

Vulnerabilities from third-party libraries (TPLs) have been unveiled to threaten the Maven ecosystem. Despite patches being released promptly after vulnerabilities are disclosed, the libraries and applications in the community still use the vulnerable versions, which makes the vulnerabilities persistent in the Maven ecosystem (e.g., the notorious Log4Shell still greatly influences the Maven ecosystem nowadays from 2021). Both academic and industrial researchers have proposed user-oriented standards and solutions to address vulnerabilities, while such solutions fail to tackle the ecosystem-wide persistent vulnerabilities because it requires a collective effort from the community to timely adopt patches without introducing breaking issues. To seek an ecosystem-wide solution, we first carried out an empirical study to examine the prevalence of persistent vulnerabilities in the Maven ecosystem. Then, we identified affected libraries for alerts by implementing an algorithm monitoring downstream dependents of vulnerabilities based on an up-to-date dependency graph. Based on them, we further quantitatively revealed that patches blocked by upstream libraries caused the persistence of vulnerabilities. After reviewing the drawbacks of existing countermeasures, to address them, we proposed a solution for range restoration (Ranger) to automatically restore the compatible and secure version ranges of dependencies for downstream dependents. The automatic restoration requires no manual effort from the community, and the code-centric compatibility assurance ensures smooth upgrades to patched versions. Moreover, Ranger along with the ecosystem monitoring can timely alert developers of blocking libraries and suggest flexible version ranges to rapidly unblock patch versions. By evaluation, Ranger could restore 75.64% of ranges which automatically remediated 90.32% of vulnerable downstream projects., Comment: 12 pages, 9 figures, the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE2023)

Machine learning (ML) models trained on subjective self-report scores struggle to objectively classify pain accurately due to the significant variance between real-time pain experiences and recorded scores afterwards. This study developed two devices for acquisition of real-time, continuous in-session pain scores and gathering of ANS-modulated endodermal activity (EDA).The experiment recruited N = 24 subjects who underwent a post-exercise circulatory occlusion (PECO) with stretch, inducing discomfort. Subject data were stored in a custom pain platform, facilitating extraction of time-domain EDA features and in-session ground truth scores. Moreover, post-experiment visual analog scale (VAS) scores were collected from each subject. Machine learning models, namely Multi-layer Perceptron (MLP) and Random Forest (RF), were trained using corresponding objective EDA features combined with in-session scores and post-session scores, respectively. Over a 10-fold cross-validation, the macro-averaged geometric mean score revealed MLP and RF models trained with objective EDA features and in-session scores achieved superior performance (75.9% and 78.3%) compared to models trained with post-session scores (70.3% and 74.6%) respectively. This pioneering study demonstrates that using continuous in-session ground truth scores significantly enhances ML performance in pain intensity characterization, overcoming ground truth sparsity-related issues, data imbalance, and high variance. This study informs future objective-based ML pain system training., Comment: 18 pages, 2-page Appendix, 7 figures

Background: Code review, the discussion around a code change among humans, forms a communication network that enables its participants to exchange and spread information. Although reported by qualitative studies, our understanding of the capability of code review as a communication network is still limited. Objective: In this article, we report on a first step towards evaluating the capability of code review as a communication network by quantifying how fast and how far information can spread through code review: the upper bound of information diffusion in code review. Method: In an in-silico experiment, we simulate an artificial information diffusion within large (Microsoft), mid-sized (Spotify), and small code review systems (Trivago) modelled as communication networks. We then measure the minimal topological and temporal distances between the participants to quantify how far and how fast information can spread in code review. Results: An average code review participants in the small and mid-sized code review systems can spread information to between 72% and 85% of all code review participants within four weeks independently of network size and tooling; for the large code review systems, we found an absolute boundary of about 11000 reachable participants. On average (median), information can spread between two participants in code review in less than five hops and less than five days. Conclusion: We found evidence that the communication network emerging from code review scales well and spreads information fast and broadly, corroborating the findings of prior qualitative work. The study lays the foundation for understanding and improving code review as a communication network., Comment: To appear in Empirical Software Engineering Journal

Modern software heavily relies on the use of components. Those components are usually published in central repositories, and managed by build systems via dependencies. Due to issues around vulnerabilities, licenses and the propagation of bugs, the study of those dependencies is of utmost importance, and numerous software composition analysis tools have emerged for this purpose. A particular challenge are hidden dependencies that are the result of cloning or shading where code from a component is "inlined", and, in the case of shading, moved to different namespaces. We present a novel approach to detect vulnerable clones in the Maven repository. Our approach is lightweight in that it does not require the creation and maintenance of a custom index. Starting with 29 vulnerabilities with assigned CVEs and proof-of-vulnerability projects, we retrieve over 53k potential vulnerable clones from Maven Central. After running our analysis on this set, we detect 727 confirmed vulnerable clones (86 if versions are aggregated) and synthesize a testable proof-of-vulnerability project for each of those. We demonstrate that existing SCA tools often miss those exposures. At the time of submission those results have led to changes to the entries for six CVEs in the GitHub Security Advisory Database (GHSA) via accepted pull requests, with more pending., Comment: 16 pages, 1 figure

The comprehension and adoption of Artificial Intelligence (AI) are beset with practical and ethical problems. This article presents a 5-level AI Capability Assessment Model (AI-CAM) and a related AI Capabilities Matrix (AI-CM) to assist practitioners in AI comprehension and adoption. These practical tools were developed with business executives, technologists, and other organisational stakeholders in mind. They are founded on a comprehensive conception of AI compared to those in other AI adoption models and are also open-source artefacts. Thus, the AI-CAM and AI-CM present an accessible resource to help inform organisational decision-makers on the capability requirements for (1) AI-based data analytics use cases based on machine learning technologies; (2) Knowledge representation to engineer and represent data, information and knowledge using semantic technologies; and (3) AI-based solutions that seek to emulate human reasoning and decision-making. The AI-CAM covers the core capability dimensions (business, data, technology, organisation, AI skills, risks, and ethical considerations) required at the five capability maturity levels to achieve optimal use of AI in organisations.

Context: The popularity of cloud computing as the primary platform for developing, deploying, and delivering software is largely driven by the promise of cost savings. Therefore, it is surprising that no empirical evidence has been collected to determine whether cost awareness permeates the development process and how it manifests in practice. Objective: This study aims to provide empirical evidence of cost awareness by mining open source repositories of cloud-based applications. The focus is on Infrastructure as Code artifacts that automate software (re)deployment on the cloud. Methods: A systematic search through 152,735 repositories resulted in the selection of 2,010 relevant ones. We then analyzed 538 relevant commits and 208 relevant issues using a combination of inductive and deductive coding. Results: The findings indicate that developers are not only concerned with the cost of their application deployments but also take actions to reduce these costs beyond selecting cheaper cloud services. We also identify research areas for future consideration. Conclusion: Although we focus on a particular Infrastructure as Code technology (Terraform), the findings can be applicable to cloud-based application development in general. The provided empirical grounding can serve developers seeking to reduce costs through service selection, resource allocation, deployment optimization, and other techniques., Comment: 16 pages, 11 figures, 1 table

Making sure that users understand privacy policies that impact them is a key challenge for a real GDPR deployment. Research studies are mostly carried in English, but in Europe and elsewhere, users speak a language that is not English. Replicating studies in different languages requires the availability of comparable cross-language privacy policies corpora. This work provides a methodology for building comparable cross-language in a national language and a reference study language. We provide an application example of our methodology comparing English and Italian extending the corpus of one of the first studies about users understanding of technical terms in privacy policies. We also investigate other open issues that can make replication harder.

Among all cybersecurity and privacy workers, the Data Protection Officer (DPO) stands between those auditing a company's compliance and those acting as management advisors. A person that must be somehow versed in legal, management, and cybersecurity technical skills. We describe how this role tackles socio-technical risks in everyday scenarios., Comment: Short version accepted for publication in IEEE Security & Privacy, Special Issue on Usable Security for Security Workers, 2023. IEEE Press. This report includes several supplemental materials not included in the journal publication

Today, the number of data-intensive and compute-intensive applications like business and scientific workflows has dramatically increased, which made cloud computing more popular in the matter of delivering a large amount of computing resources on demand. On the other hand, security is a critical issue affecting the wide adoption of cloud technologies, especially for workflows that are mostly dealing with sensitive data and tasks. In this paper, we carry out a review of the state-of-the-art on how security and privacy concerns in scientific and business workflows in cloud environments are being addressed and identify the limitations and gaps in the current body of knowledge in this area. In this extensive literature review, we first present a classification of the state-of-the-art security solutions organized according to the phases of the workflow life cycle they target. Based on our findings, we provide a detailed review and classification of the most relevant available literature focusing on the execution, monitoring, and adaptation phases of workflows. Finally, we present a list of open research issues related to the security of cloud-based workflows and discuss them., Comment: 16 pages, 8 figures, 5 tables

This article provides a rich discussion on how the sustainability of agile development processes can be enhanced. In particular, we focus on a recently developed framework, named GLUX, that integrates Lean UX into Scrum. GLUX's main goal is to facilitate a seamless integration between agile and user experience (UX) by using gamification to motivate agile teams to adopt a user-centered mindset and carry out UX activities collaboratively throughout the development process. Our role as software researchers is to contribute towards improving software sustainability and provide the software engineering community with the tools and techniques that will improve the human, economic, and environmental sustainability of software development. We found that GLUX addresses human sustainability by empowering self-sufficient, problem-focused teams, building a motivating and engaging environment, and developing team cooperation. Economic sustainability is addressed by minimizing UX debt and using gamification techniques to direct the focus of the behavior and mindset of agile teams towards value creation. Finally, environmental sustainability is promoted by encouraging agile teams to build a minimum viable product (MVP)., Comment: 8 pages, 2 figures, XP 2022 conference

The desire to build quality software systems has been the focus of most software developers and researchers for decades. This has culminated in the design of practices that promote quality in the designed software. Originating from the inception of the traditional software development life cycle (SDLC), through to the object-oriented methods, Iterative development, and now the agile methods, these practices have persisted through different periods. Such practices play the same quality role regardless of the perspective of the software development process they are part of. In this paper we review three software development methods representative of the software development history, with the aim of i) identifying key quality practices, ii) identifying the quality role played by the practice in the method, and iii) noting those quality practices that have persisted through the software development history. The identified quality practices that have persisted throughout the history of the software development processes include prototyping, iterative development, incremental development, risk-driven development, phase planning, and phase retrospection. These results would be useful to method engineers who seek to design high-quality software development methods as these practices serve as candidates for inclusion in their development processes. Software development practitioners seeking to design quality software would also benefit from adopting these practices in developing their software., Comment: 7 pages, 1 figure

Teamwork is critical in many industrial sectors. When creating complex software solutions, most companies and public institutions organize work in cross-functional teams and follow the principles of agile development. This approach to knowledge-intensive work seeks to empower team members, ensures that the most competent people make decisions, and manages uncertainty by allowing members to learn and adapt as the work progresses. Advice on teamwork is abundant. For example, the Google re:Work model offers advice to development teams in the form of five key factors for successful teams, including psychological safety, structure and clarity, and teamwork that the team members consider meaningful. There is also general advice from years of studies of teamwork and from empirical studies on agile development teams. However, there has yet to be a model that draws together the knowledge from all of these sources and specifically focuses on the effectiveness of agile teamwork. To fill this gap, we have developed an Agile Teamwork Effectiveness Model (ATEM). Our model is based on a review of empirical studies on agile development teams, general studies of effective teams and teamwork, and practitioner advice. We also incorporated findings from our own two case studies and 22 focus groups. Though primarily intended for collocated agile software development teams, the increasing adoption of agile methods outside IT departments may make the model valuable for other agile workplaces., Comment: 9 pages, 1 figure

Treasure Data is processing millions of distributed SQL queries every day on the cloud. Upgrading the query engine service at this scale is challenging because we need to migrate all of the production queries of the customers to a new version while preserving the correctness and performance of the data processing pipelines. To ensure the quality of the query engines, we utilize our query logs to build customer-specific benchmarks and replay these queries with real customer data in a secure pre-production environment. To simulate millions of queries, we need effective minimization of test query sets and better reporting of the simulation results to proactively find incompatible changes and performance regression of the new version. This paper describes the overall design of our system and shares various challenges in maintaining the quality of the query engine service on the cloud., Comment: This version is published in DBTest '22: Proceedings of the 2022 workshop on 9th International Workshop of Testing Database Systems

An assurance case is intended to provide justifiable confidence in the truth of its top claim, which typically concerns safety or security. A natural question is then "how much" confidence does the case provide? We argue that confidence cannot be reduced to a single attribute or measurement. Instead, we suggest it should be based on attributes that draw on three different perspectives: positive, negative, and residual doubts. Positive Perspectives consider the extent to which the evidence and overall argument of the case combine to make a positive statement justifying belief in its claims. We set a high bar for justification, requiring it to be indefeasible. The primary positive measure for this is soundness, which interprets the argument as a logical proof. Confidence in evidence can be expressed probabilistically and we use confirmation measures to ensure that the "weight" of evidence crosses some threshold. In addition, probabilities can be aggregated from evidence through the steps of the argument using probability logics to yield what we call probabilistic valuations for the claims. Negative Perspectives record doubts and challenges to the case, typically expressed as defeaters, and their exploration and resolution. Assurance developers must guard against confirmation bias and should vigorously explore potential defeaters as they develop the case, and should record them and their resolution to avoid rework and to aid reviewers. Residual Doubts: the world is uncertain so not all potential defeaters can be resolved. We explore risks and may deem them acceptable or unavoidable. It is crucial however that these judgments are conscious ones and that they are recorded in the assurance case. This report examines the perspectives in detail and indicates how Clarissa, our prototype toolset for Assurance 2.0, assists in their evaluation., Comment: Second Edition

The integration of Agile development and user experience (UX) has received significant attention over the past decade. The literature contains several process models and wide-ranging discussion about the benefits and challenges of this integration. However, academia has given this integration short shrift. In fact, there are very few publications covering educational courses dealing with Agile development and UX. In this paper, we report on our experience of designing and running a graduate software engineering course that covers the integration of Lean UX into Scrum and employs gamification to improve student engagement. We identified six lessons learned that new point to important aspects to be considered when integrating Agile development and UX in academia. For example, we discuss the complexity of managing Lean UX activities in short sprints or how the design of a testable and tactical hypothesis can be one of the most challenging aspects of the Lean UX process., Comment: 12 pages. 44th International Conference on Software Engineering Software Engineering Education and Training (ICSE-SEET 22), Pittsburgh, PA, USA. ACM, New York, NY, USA

This paper presents the types of teams that are currently in place at the Netherlands eScience Center. We categorize our teams into four different types: Project Teams, Collectives, Agile Teams and Scrum Teams. We provide a brief description of each team type and share stories from teams themselves to reflect on the strengths and shortcomings of each model. From our observation, we conclude that the type of team that is most suitable for each project depends on the specific needs of that project. More importantly, different types of teams are suitable for the different types of people working at the Netherlands eScience Center., Comment: 8 pages, 1 figure, 1 info panel, to appear in IEEE Computing in Science & Engineering

Software estimation is critical for a software project's success and a challenging activity. We argue that estimation problems are not restricted to the generation of estimates but also their use for commitment establishment: project stakeholders pressure estimators to change their estimates or to accept unrealistic commitments to attain business goals. In this study, we employed a Design Science Research (DSR) methodology to design an artifact based on negotiation methods, to empower software estimators in defending their estimates and searching for alternatives to unrealistic commitments when facing pressure. The artifact is a concrete step towards disseminating the soft skill of negotiation among practitioners. We present the preliminary results from a focus group that showed that practitioners from the software industry could use the artifact in a concrete scenario when estimating and establishing commitments about a software project. Our future steps include improving the artifact with the suggestions from focus group participants and evaluating it empirically in real software projects in the industry., Comment: 5 pages, 1 figure, 1 table

Over the last years, the software industry has adopted several DevOps technologies related to practices such as continuous integration and continuous delivery. The high demand for DevOps practitioners requires non-trivial adjustments in traditional software engineering courses and educational methodologies. This work presents an interview study with 14 DevOps educators from different universities and countries, aiming to identify the main challenges and recommendations for DevOps teaching. Our study identified 83 challenges, 185 recommendations, and several association links and conflicts between them. Our findings can help educators plan, execute and evaluate DevOps courses. They also highlight several opportunities for researchers to propose new methods and tools for teaching DevOps., Comment: 12 pages, 6 figures, 5 tables, ICSE 2022 SEET

Big data, i.e. collecting, storing and processing of data at scale, has recently been possible due to the arrival of clusters of commodity computers powered by application-level distributed parallel operating systems like HDFS/Hadoop/Spark, and such infrastructures have revolutionized data mining at scale. For data mining project to succeed more consistently, some methodologies were developed (e.g. CRISP-DM, SEMMA, KDD), but these do not account for (1) very large scales of processing, (2) dealing with textual (unstructured) data (i.e. Natural Language Processing (NLP, "text analytics"), and (3) non-technical considerations (e.g. legal, ethical, project managerial aspects). To address these shortcomings, a new methodology, called "Data to Value" (D2V), is introduced, which is guided by a detailed catalog of questions in order to avoid a disconnect of big data text analytics project team with the topic when facing rather abstract box-and-arrow diagrams commonly associated with methodologies., Comment: 9 pages, 6 figures, 4 tables

Business process (BP) analysis represents a first key phase of information system development. It consists in the gathering of domain knowledge and its organization to be later used in the software development, and beyond (e.g., for Business Process Reengineering). The quality of the developed information system largely depends on how the BP analysis has been carried out and the quality of the produced requirement specification documents. Despite the fact that the issue is on the table for decades, business process analysis is still a critical phase of information systems development. One promising strategy is an early and more important involvement of business experts in the BP analysis. This paper presents a methodology that aims at an early involvement of business experts while providing a formal grounding that guarantees the quality of the produced specifications. To this end, we propose the Business Process Analysis Canvas, a knowledge framework organized in eight knowledge sections aimed at supporting the business expert in carrying out the analysis, eventually yielding a BP analysis Ontology., Comment: 15 pages, 3 fugures, 4 tables

Within the rapidly diversifying field of computational science and engineering (CSE), research software engineers (RSEs) represent a shift towards the adoption of mainstream software engineering tools and practices into scientific software development. An unresolved challenge is the need to effectively integrate RSEs and their expertise into multi-disciplinary scientific software teams. There has been a long-standing "chasm" between the domains of CSE and software engineering, and the emergence of RSEs as a professional identity within CSE presents an opportunity to finally bridge that divide. For this reason, we argue there is an urgent need for systematic investigation into multi-disciplinary teaming strategies which could promote a more productive relationship between the two fields., Comment: Presented at the Workshop on the Science of Scientific-Software Development and Use, sponsored by U.S. Department of Energy, Office of Advanced Scientific Computing Research, Dec 13-15, 2021. 2 pages

There has been growing interest within the computational science and engineering (CSE) community in engaging with software engineering research -- the systematic study of software systems and their development, operation, and maintenance -- to solve challenges in scientific software development. Historically, there has been little interaction between scientific computing and the field, which has held back progress. With the ranks of scientific software teams expanding to include software engineering researchers and practitioners, we can work to build bridges to software science and reap the rewards of evidence-based practice in software development., Comment: Presented at the Workshop on the Science of Scientific-Software Development and Use, sponsored by U.S. Department of Energy, Office of Advanced Scientific Computing Research, Dec 13-15, 2021. 2 pages

In recent years, the role and the importance of software in the automotive domain have changed dramatically. Being able to systematically evaluate and manage software quality is becoming even more crucial. In practice, however, we still find a largely static approach for measuring software quality based on a predefined list of complexity metrics with static thresholds to fulfill. We propose using a more flexible framework instead, which systematically derives measures and evaluation rules based on the goals and context of a development project., Comment: Position Paper, 21 pages, 1 figure, and 3 tables

As a newly designated professional title, research software engineers (RSEs) link the two worlds of software engineering and research science. They lack clear development and training opportunities, particularly in the realm of mentoring. In this paper, we discuss mentorship as it pertains to the unique needs of RSEs and propose ways in which organizations and institutions can support mentor/mentee relationships for RSEs, Comment: 3 pages, Presented at Research Software Engineers in HPC (RSE-HPC-2021), co-located with Supercomputing'21 (SC'21)

Code comments are the primary means to document implementation and facilitate program comprehension. Thus, their quality should be a primary concern to improve program maintenance. While much effort has been dedicated to detecting bad smells, such as clones in code, little work has focused on comments. In this paper we present our solution to detect clones in comments that developers should fix. RepliComment can automatically analyze Java projects and report instances of copy-and-paste errors in comments, and can point developers to which comments should be fixed. Moreover, it can report when clones are signs of poorly written comments. Developers should fix these instances too in order to improve the quality of the code documentation. Our evaluation of 10 well-known open source Java projects identified over 11K instances of comment clones, and over 1,300 of them are potentially critical. We improve on our own previous work, which could only find 36 issues in the same dataset. Our manual inspection of 412 issues reported by RepliComment reveals that it achieves a precision of 79% in reporting critical comment clones. The manual inspection of 200 additional comment clones that RepliComment filters out as being legitimate, could not evince any false negative., Comment: 31 pages, 1 figure, 9 tables. To appear in the Journal of Systems and Software

Scientific modeling provides mathematical abstractions of real-world systems and builds software as implementations of these mathematical abstractions. Ocean science is a multidisciplinary discipline developing scientific models and simulations as ocean system models that are an essential research asset. In software engineering and information systems research, modeling is also an essential activity. In particular, business process modeling for business process management and systems engineering is the activity of representing processes of an enterprise, so that the current process may be analyzed, improved, and automated. In this paper, we employ process modeling for analyzing scientific software development in ocean science to advance the state in engineering of ocean system models and to better understand how ocean system models are developed and maintained in ocean science. We interviewed domain experts in semi-structured interviews, analyzed the results via thematic analysis, and modeled the results via the business process modeling notation BPMN. The processes modeled as a result describe an aspired state of software development in the domain, which are often not (yet) implemented. This enables existing processes in simulation-based system engineering to be improved with the help of these process models., Comment: 18 pages

The presence of source code that is no longer needed is a handicap to project maintainability. The larger and longer-lived the project, the higher the chances of accumulating dead code in its different forms. Manually detecting unused code is time-consuming, tedious, error-prone, and requires a great level of deep knowledge about the codebase. In this paper, we examine the kinds of dead code (specifically, oxbow code) that can appear in Erlang projects, and formulate rules to identify them with high accuracy. We also present an open-source static analyzer that implements these rules, allowing for the automatic detection and confident removal of oxbow code in Erlang codebases, actively contributing to increasing their quality and maintainability., Comment: 13 pages, 1 figure, 2 tables

An Infrastructure as a Service (IaaS) cloud provider is committed to each tenant by a service level agreement (SLA) which indicates the terms of commitment, e.g. the level of availability of the IaaS cloud service.The different resources providing this IaaS cloud service may need to be upgraded several times throughout their life-cycle; and these upgrades may affect the service delivered by the IaaS layer. This may violate the SLAs towards the tenants and result in penalty as they impact the tenant services relying on the IaaS.Therefore, it is important to handle upgrades properly with respect to the SLAs.The upgrade of IaaS cloud systems inherits all the challenges of clustered systems and faces other, cloud specific challenges, such as size and dynamicity due to elasticity.In this paper, we propose a novel approach to automatically upgrade an IaaS cloud system under SLA constraints such as availability and elasticity.In this approach, the upgrade methods and actions appropriate for each upgrade request are identified, scheduled, and applied automatically in an iterative manner based on the vendors descriptions of the infrastructure components, the tenant SLAs, and the status of the system. The proposed approach allows new upgrade requests during ongoing upgrades, which makes it suitable for continuous delivery.In addition, it also handles failures of upgrade actions through localized retry and undo operations automatically., Comment: 20 pages, 11 figures