Search

Sentiment analysis has been widely explored in many text domains, including product reviews, movie reviews, tweets, and so on. However, there are very few studies trying to perform sentiment analysis in the domain of peer reviews for scholarly papers, which are usually long and introducing both pros and cons of a paper submission. In this paper, we for the first time investigate the task of automatically predicting the overall recommendation/decision (accept, reject, or sometimes borderline) and further identifying the sentences with positive and negative sentiment polarities from a peer review text written by a reviewer for a paper submission. We propose a multiple instance learning network with a novel abstract-based memory mechanism (MILAM) to address this challenging task. Two evaluation datasets are constructed from the ICLR open reviews and evaluation results verified the efficacy of our proposed model. Our model much outperforms a few existing models in different experimental settings. We also find the generally good consistency between the review texts and the recommended decisions, except for the borderline reviews.

Many studies demonstrate that peer reviewing provides pedagogical benefits such as inspiration and developing expert vision, and changes classroom culture by encouraging reciprocity. However, much large-scale research in peer assessment has focused on MOOCs, where students have short tenures, and is unable to describe how reciprocity-oriented classroom cultures evolve over time. This short paper presents the first long-term analysis of peer reviewing with 304 students, conducted in three large physical classes in a year-long undergraduate series. Surprisingly, this analysis reveals that when students receive better reviews on their work, they write worse reviews in the future. This suggests that while students believe in the reciprocal nature of peer review, they act anti-reciprocally. Therefore, battling the emergent norm of anti-reciprocity is crucial both for system designers and practitioners who use peer assessment.

In the 10 years since the term was coined 'design fiction' has become an increasingly common approach in HCI research. The practice involves working with 'diegetic prototypes', that is prototypes that need not exist in reality, but instead exist from within a 'story world'. Although fictional aspects are not unusual in HCI prototyping methods (e.g. storyboards, personas, Wizard-of-Oz), the breadth and flexibility of design fiction poses new challenges. This paper originally featured quotes from peer reviews of design fiction orientated papers that have previously been submitted to ACM SIGCHI conferences in order to highlight inherent challenges when reviewing research that may be based upon or include elements of fiction. In response to the SIGCHI Executive Committee's request to not directly quote reviewers the quotes have now been redacted. This somewhat changes the paper's tone and also makes very clear that publishing discussions relating to peer reviews (or the reviews themselves) is extremely challenging.

Context. Empirical research consistently demonstrates that that scholarly peer review is ineffective, unreliable, and prejudiced. In principle, the solution is to move from contemporary, unstructured, essay-like reviewing to more structured, checklist-like reviewing. The Task Force created models—called “empirical standards”—of the software engineering community’s expectations for different popular methodologies. Objective. This paper presents a tool for facilitating more structured reviewing by generating review checklists from the empirical standards. Design. A tool that generates pre-submission and review forms using the empirical standards for software engineering research was designed and implemented. The pre-submission and review forms can be used by authors and reviewers, respectively, to determine whether a manuscript meets the software engineering community’s expectations for the particular kind of research conducted. Evaluation. The proposed tool can be empirically evaluated using lab or field randomized experiments as well as qualitative research. Huge, impractical studies involving splitting a conference program committee are not necessary to establish the effectiveness of the standards, checklists and structured review. Conclusions. The checklist generator enables more structured peer reviews, which in turn should improve review quality, reliability, thoroughness, and readability. Empirical research is needed to assess the effectiveness of the tool and the standards.

Transparency in process and its reporting is paramount for establishing the rigor of qualitative studies. However, the CHI conference receives submissions with varying levels of transparency and oftentimes, papers that are more transparent can be inadvertently subjected to more scrutiny in the review process, raising issues of fairness. In this panel, we bring together researchers with diverse qualitative work experiences to present examples of transparency-related initiatives and their corresponding review responses. We aim to work towards setting standards for transparent reporting in qualitative-work submissions and increasing fairness in the review process. We focus on the challenges in achieving transparency in qualitative research and current workarounds to overcome frictions in the reviewing process through engaging discussions involving panelists and the audience.

Modelling techniques have been part of the software engineering practice for decades. Despite Model-Driven Software Engineering (Mde) being spread in the industry, effective teaching through meaningful use cases remains a challenge. Some guidelines and teaching experiences have been reported that either focus on the content of an Mde course, or on tooling environments designed for students. Furthermore, little emphasis has been put on the usability in the context of Software Language Engineering (Sle), which may lead to technically valid languages, but which are difficult to use. In this paper, we report on our attempt to combine a practical use case for Sle and a structured peer review focusing on the language usability and its tooling environment. By putting students in both the seats of language designers and users, we believe to empower students with suitable technical Sle skills as well as making them aware that a good software language requires more than theoretical validity to be adopted.

The Scaled Agile Framework (SAFe) has been adopted by a large number of organizations to scale agile to large enterprises that develop software. At the moment, SAFe seems to be the most predominant agile scaling framework. Despite the current popularity of SAFe in the software intensive industry, there exists surprisingly little research on the adoption and usage of SAFe. To study what is known about SAFe with respect to benefits and challenges of its usage, we conducted a multivocal literature review. The preliminary version of the results are presented in this poster version of paper. We identified 48 unique adoption cases in 46 organizations from both peer reviewed and non peer reviewed (grey literature) sources.

Human-Computer Interaction (HCI) is dynamic and interdisciplinary. Diverse areas of knowledge contribute to form and advance the state of the art, but also bring different views about how to do research in HCI. In this paper we aim at promoting discussion and increased awareness on different kinds of research by comparing the three HCI waves with concepts of Philosophy of Science and general research paradigms. We believe this awareness can play an important role in improving the fairness of the peer review process and support researchers in positioning themselves and their investigations.

Design challenges refer to a difficulty of corresponding human and contextual complexity (i.e. needs, roles, and resources) in design practices. Such an issue calls for combining deep investigations with relevant design experiences. We propose a workshop for disentangling and discussing design practices by adopting the Theatre of the Oppressed techniques. These techniques allow enacting personal performances as well as the construction of a shared narrative about the participants' roles, needs, and resources. By this workshop, we aim to improve the participants' competence in understanding people's needs and developing a design solution accordingly. Finally, possible outcomes are: a special issue of an international peer reviewed journal, and/or a live performance that the conference attendees can enjoy as an experiential design occasion.

In online collaboration projects, conflicts often arise in the peer review process, due to the disagreement over whether one's contribution should be accepted. These conflicts generally have detrimental effects on contributors' continuing participation in the community. Few studies have investigated how to manage these conflicts effectively. This paper aims to examine the effectiveness of three strategies - rational explanation, constructive suggestion, and social encouragement - in managing conflicts. In an analysis of 170 online software development projects, we investigated how different conflict management strategies aimed at handling contributors' arguments during the peer review process influenced their subsequent participation in the projects. The results show that (i) conflicts significantly increase contributors' likelihood of leaving the communities; (ii) neither rational explanations nor social encouragement could reduce the negative consequences of conflicts; (iii) only constructive suggestions have a positive effect in retaining the contributors.