Search

Papers describing software are an important part of computational fields of scientific research. These “software papers” are unique in a number of ways, and they require special consideration to improve their impact on the scientific community and their efficacy at conveying important information. Here, we discuss 10 specific rules for writing software papers, covering some of the different scenarios and publication types that might be encountered, and important questions from which all computational researchers would benefit by asking along the way., Author summary Computational researchers have a responsibility to ensure that the software they write stands up to the same scientific scrutiny as traditional research studies. These 10 simple rules make doing so easier by enhancing usability, reproducibility, transparency, and other crucial characteristics that aren’t taught in most computer science or research methods curricula.

Sentiment analysis has been widely explored in many text domains, including product reviews, movie reviews, tweets, and so on. However, there are very few studies trying to perform sentiment analysis in the domain of peer reviews for scholarly papers, which are usually long and introducing both pros and cons of a paper submission. In this paper, we for the first time investigate the task of automatically predicting the overall recommendation/decision (accept, reject, or sometimes borderline) and further identifying the sentences with positive and negative sentiment polarities from a peer review text written by a reviewer for a paper submission. We propose a multiple instance learning network with a novel abstract-based memory mechanism (MILAM) to address this challenging task. Two evaluation datasets are constructed from the ICLR open reviews and evaluation results verified the efficacy of our proposed model. Our model much outperforms a few existing models in different experimental settings. We also find the generally good consistency between the review texts and the recommended decisions, except for the borderline reviews.

Part 1: The International Cross Domain Conference (CD-ARES 2016); International audience; Peer review is widely viewed as an essential step for ensuring scientific quality of a work and is a cornerstone of scholarly publishing. On the other hand, the actors involved in the publishing process are often driven by incentives which may, and increasingly do, undermine the quality of published work, especially in the presence of unethical conduits. In this work we investigate the feasibility of a tool capable of generating fake reviews for a given scientific paper automatically. While a tool of this kind cannot possibly deceive any rigorous editorial procedure, it could nevertheless find a role in several questionable scenarios and magnify the scale of scholarly frauds.A key feature of our tool is that it is built upon a small knowledge base, which is very important in our context due to the difficulty of finding large amounts of scientific reviews. We experimentally assessed our method 16 human subjects. We presented to these subjects a mix of genuine and machine generated reviews and we measured the ability of our proposal to actually deceive subjects judgment. The results highlight the ability of our method to produce reviews that often look credible and may subvert the decision.

الملخص: عادة ما تكون المقالات ما قبل الطباعة بمثابة مقدمة أولية لأوراق تمت مراجعتها من قِبل النظراء ويتم وضعها على الفور تقريبا، باستثناء بعض الفحص السطحي، في مستودع وصول مفتوح للجمهور للسماح للمعلومات بالوصول إلى القراء بسرعة، والتغلب على الخمول المرتبط عادة بالمعالجة بطريقة المجلات التي تعتمد على مراجعة الأقران. بالنسبة للباحثين في بداية حياتهم المهنية الذين قد يكونون متحمسين للحصول على بعض التقدير لجهودهم، أو الرغبة في الحصول على مدخلات مفتوحة وعلنية حول عملهم، فإن النشر قبل الطباعة هو بالتأكيد خيار نشر مفيد. ومع ذلك، إذا لم يتم فحص البيانات والمعلومات المتعلقة بالصحة بعناية، فقد تشكل خطرا، وقد تعمل أيضا كمصدر لمعلومات مضللة عن الصحة العامة. تشير زيادة النمو والمنافسة بين خوادم ما قبل الطباعة، إلى جانب حجم هائل من المطبوعات المسبقة المتعلقة بـ كوفيد-١٩، وخاصة على خادم ما قبل الطباعة لعلم الأحياء، وخادم ما قبل الطباعة للعلوم الطبية، بالإضافة إلى تحديد الفهرسة التي يتم اختبارها الآن على موقع بوب ميد، إلى أن المطبوعات المسبقة يتم استخدامها بشكل متزايد في العلوم الطبية الحيوية. وهناك حاجة إلى سياسات أخلاقية أقوى وأكثر قوة لفحص المطبوعات المسبقة قبل نشرها للجمهور، وحتى إذا كان هذا يعني تأخيرا طفيفا في النشر، فقد يؤدي ذلك إلى زيادة ثقة الأكاديميين في هذا الشكل من المعلومات العلمية والتواصل. يجب تقديم سياسات أخلاقية واضحة وصارمة بشكل عاجل من قبل مجموعات الأخلاقيات مثل لجنة أخلاقيات النشر، واللجنة الدولية لمحرري المجلات الطبية للمجلات الأعضاء التي تسمح بنشر مقالات ما قبل الطباعة قبل مراجعة النظراء التقليدية. ستفيد الإرشادات الأخلاقية الصارمة التي تتعامل مع سوء السلوك على قدم المساواة في مقالات ما قبل الطباعة والأبحاث التي راجعها النظراء، ستفيد نزاهة النشر الأكاديمي. Abstract: Preprints are typically crude precursors of peer-reviewed papers that are placed almost immediately, save for some superficial screening, on an open-access repository to allow the information to reach readers quickly, circumventing the long-drawn process typically associated with processing in peer-reviewed journals. For early-career researchers who might be enthusiastic about obtaining some recognition for their efforts, or wanting open and public input about their work, preprints are certainly a useful publication choice. However, if health-related data and information have not been carefully scrutinised, they may pose a risk and may even serve as a source of public health misinformation. Surging growth and competition among preprint servers, coupled with a massive volume of COVID-19-related preprints, mainly on bioRxiv and medRxiv, as well as select indexing now being tested on PubMed, suggests that preprints are being increasingly used in the biomedical sciences. Stronger and more robust ethical policies are needed to screen preprints before they are released to the public, and even if this implies a slight delay in publication, it may increase academics' trust in this form of scientific information and communication. Clear and stringent ethical policies need to be urgently introduced by ethics groups such as COPE and the ICMJE, whose many member journals allow preprints to be posted before traditional peer review. Stringent ethical guidelines that treat misconduct equally in preprints and peer-reviewed papers will boost the integrity of academic publishing.

Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1),1 providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as ‘originality’ and ‘impact’.

Peer reviews form an essential part of scientific communications. Research papers and proposals are reviewed by several peers before they are finally accepted or rejected for publication and funding, respectively. With the steady increase in the number of research domains, scholarly venues (journal and/or conference), researchers, and papers, managing the peer review process is becoming a daunting task. Application of recommender systems to assist peer reviewing is, therefore, being explored and becoming an emerging research area. In this paper, we present a deep learning network based Meta-Review Generation considering peer review prediction of the scholarly article (MRGen). MRGen is able to provide solutions for: (i) Peer review prediction (Task 1) and (ii) Meta-review generation (Task 2). First, the system takes the peer reviews as input and produces a draft meta-review. Then it employs an integrated framework of convolution layer, long short-term memory (LSTM) model, Bi-LSTM model, and attention mechanism to predict the final decision (accept/reject) of the scholarly article. Based on the final decision, the proposed model MRGen incorporates Pointer Generator Network-based abstractive summarization to generate the final meta-review. The focus of our approach is to give a concise meta-review that maximizes information coverage, coherence, readability and also reduces redundancy. Extensive experiments conducted on the PeerRead dataset demonstrate good consistency between the recommended decisions and original decisions. We also compare the performance of MRGen with some of the existing state-ofthe- art multi-document summarization methods. The system also outperforms a few existing models based on accuracy, Rouge scores, readability, non-redundancy, and cohesion.

Replicability tests of scientific papers show that the majority of papers fail replication. Moreover, failed papers circulate through the literature as quickly as replicating papers. This dynamic weakens the literature, raises research costs, and demonstrates the need for new approaches for estimating a study’s replicability. Here, we trained an artificial intelligence model to estimate a paper’s replicability using ground truth data on studies that had passed or failed manual replication tests, and then tested the model’s generalizability on an extensive set of out-of-sample studies. The model predicts replicability better than the base rate of reviewers and comparably as well as prediction markets, the best present-day method for predicting replicability. In out-of-sample tests on manually replicated papers from diverse disciplines and methods, the model had strong accuracy levels of 0.65 to 0.78. Exploring the reasons behind the model’s predictions, we found no evidence for bias based on topics, journals, disciplines, base rates of failure, persuasion words, or novelty words like “remarkable” or “unexpected.” We did find that the model’s accuracy is higher when trained on a paper’s text rather than its reported statistics and that n-grams, higher order word combinations that humans have difficulty processing, correlate with replication. We discuss how combining human and machine intelligence can raise confidence in research, provide research self-assessment techniques, and create methods that are scalable and efficient enough to review the ever-growing numbers of publications—a task that entails extensive human resources to accomplish with prediction markets and manual replication alone.

This paper develops a theoretically underpinned peer review approach (called Viable System Review) to assess and strengthen disaster risk reduction (DRR) practices. This topic is timely as countries and cities look to learn lessons from others to respond to global priorities on DRR. Unlike other peer review methodologies which rely on expert intuition to navigate peer review issues, our new methodology structures the peer review process through an organising framework of concepts. That framework is theoretically underpinned by systems thinking and concepts from the Viable System Model which provides a comprehensive, tested approach to assess organisational performance. The paper explores the development and application of our new peer review methodology (which we call Viable System Review) to the policy developed by Concepcion Province (Chile) on involving spontaneous volunteers in disaster situations. The peer review took place over six days, involved seven reviewers, and focused on identifying good practices, opportunities to enhance practice, and lessons they could take back to their home organisations. The paper presents our Viable System Review methodology, reflects on its application, and discusses its practical and theoretical contributions.

Scientific papers are complex and understanding the usefulness of these papers requires prior knowledge. Peer reviews are comments on a paper provided by designated experts on that field and hold a substantial amount of information, not only for the editors and chairs to make the final decision, but also to judge the potential impact of the paper. In this paper, we propose to use aspect-based sentiment analysis of scientific reviews to be able to extract useful information, which correlates well with the accept/reject decision. While working on a dataset of close to 8k reviews from ICLR, one of the top conferences in the field of machine learning, we use an active learning framework to build a training dataset for aspect prediction, which is further used to obtain the aspects and sentiments for the entire dataset. We show that the distribution of aspect-based sentiments obtained from a review is significantly different for accepted and rejected papers. We use the aspect sentiments from these reviews to make an intriguing observation, certain aspects present in a paper and discussed in the review strongly determine the final recommendation. As a second objective, we quantify the extent of disagreement among the reviewers refereeing a paper. We also investigate the extent of disagreement between the reviewers and the chair and find that the inter-reviewer disagreement may have a link to the disagreement with the chair. One of the most interesting observations from this study is that reviews, where the reviewer score and the aspect sentiments extracted from the review text written by the reviewer are consistent, are also more likely to be concurrent with the chair's decision., Comment: Accepted in JCDL'20

Advances in technologies (e.g. smartphones, wearables) have resulted in the concept of ‘self-tracking’, and the use of self-tracking technologies in physical activity (i.e. fitness tracking) is on the rise. For example, many people track and monitor their fitness-related metrics (e.g. steps walked, distance ran, and calories burned) to change their behaviours or keep themselves active. Despite the widespread application of self-tracking in fitness, relatively little is known about its drivers and outcomes. To address this gap, the current paper provides an overview of the literature (empirical papers) on self-tracking with a focus on the drivers and outcomes of fitness tracking behaviour and offers four important contributions. First, it identifies 19 drivers of fitness tracking technology usage. Second, it discusses four main outcomes of fitness tracking behaviour. Third, by drawing on the existing studies conducted across various fitness tracking technologies (e.g. fitness trackers, apps) and user groups (e.g. patients, seniors, and females), it provides valuable insights that can be generalisable to other settings (e.g. other types of users and fitness tracking products). Finally, the current paper provides important practical implications and addresses avenues for future research.

The development of navigation tools for people who are visually impaired had become an important concern in the research area of assistive technologies. This paper gives a comprehensive review of different articles published in the area of navigation solutions for people who are visually impaired. Unlike other review papers, this review considers major solutions that work in both the indoor or/and outdoor environments which are based on different technology. From the review, it became clear that the navigation systems proposed for the target users lack some core features that are quite important for independent navigation. Also, there can be instances in which humanitarian conditions also have to be considered in the navigation system design. Based on these findings, a set of recommendations are also given which can be considered in the future design of navigation systems for blind and visually impaired people.

With a tremendous growth in the number of scientific papers, researchers have to spend too much time and struggle to find the appropriate papers they are looking for. Local citation recommendation that provides a list of references based on a text segment could alleviate the problem. Most existing local citation recommendation approaches concentrate on how to narrow the semantic difference between the scientific papers’ and citation context’s text content, completely neglecting other information. Inspired by the successful use of the encoder-decoder framework in machine translation, we develop an attention-based encoder-decoder (AED) model for local citation recommendation. The proposed AED model integrates venue information and author information in attention mechanism and learns relations between variable-length texts of the two text objects, i.e., citation contexts and scientific papers. Specifically, we first construct an encoder to represent a citation context as a vector in a low-dimensional space; after that, we construct an attention mechanism integrating venue information and author information and use RNN to construct a decoder, then we map the decoder’s output into a softmax layer, and score the scientific papers. Finally, we select papers which have high scores and generate a recommended reference paper list. We conduct experiments on the DBLP and ACL Anthology Network (AAN) datasets, and the results illustrate that the performance of the proposed approach is better than the other three state-of-the-art approaches.

Publication and researcher bias is common in software engineering experiments.Our model shows how these biases lead to a high proportion of incorrect results.Increased statistical power is a key factor to improve the trustworthiness. ContextThe trustworthiness of research results is a growing concern in many empirical disciplines. AimThe goals of this paper are to assess how much the trustworthiness of results reported in software engineering experiments is affected by researcher and publication bias, given typical statistical power and significance levels, and to suggest improved research practices. MethodFirst, we conducted a small-scale survey to document the presence of researcher and publication biases in software engineering experiments. Then, we built a model that estimates the proportion of correct results for different levels of researcher and publication bias. A review of 150 randomly selected software engineering experiments published in the period 20022013 was conducted to provide input to the model. ResultsThe survey indicates that researcher and publication bias is quite common. This finding is supported by the observation that the actual proportion of statistically significant results reported in the reviewed papers was about twice as high as the one expected assuming no researcher and publication bias. Our models suggest a high proportion of incorrect results even with quite conservative assumptions. ConclusionResearch practices must improve to increase the trustworthiness of software engineering experiments. A key to this improvement is to avoid conducting studies with unsatisfactory low statistical power.

Project owners are important. They shall support the project manager and provide senior backing to a project. However, there are some different aspects of project ownership, which are discussed in this paper. The main research questions addressed in the paper are; how is the project owner role described in the literature, and how is the role in practice? The paper is mainly a theory review of the topic project ownership. We present a simple model for the relation between project owner, project manager and operation of the project delivery. The basis for the model and discussion is a theoretical review, but we also test the proposed model on a sample of projects. As we will see, this testing creates the need for a distinction between two types of project owners; one that is mainly concerned with supporting the project manager and enabling project delivery, the other being focused on the business case and having responsibility for both project delivery and benefit realization. There has been a development where the importance of the role as project owner has been highlighted. It is our impression that this development has had two ambitions. One is to strengthen project execution. The other ambition is to emphasize the business case perspective. These two perspectives call for different organizational positions of the persons called project owners. We urge future researchers and practitioners to clarify what type of project owner role that is referred to in different contexts. We also call for an awareness of the fact that different project owners will have different set of incentives and priorities. © 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license.

Alignment has been identified as an important criteria for successful project teams. However, we need more knowledge on how alignment is achieved in project teams and also about potential negative effects of focusing alignment. This paper responds to a call for more in-depth empirical accounts of what goes on in projects and project management. More specifically, it provides insight into practices of achieving alignment in an inter-organizational project collaboration. The paper addresses potential negative effects of alignment, as findings show how these practices reduce the potential for diverse discussions over ideas, risks and solutions leaving project teams as an untapped resource. The empirical material originates from a PhD project studying the appeal, use and effect of the risk matrix in an inter-organizational project within the petroleum sector in Norway using a qualitative approach.

Industry 4.0 is the first industrial revolution to be announced a priori, and there is thus a significant ambiguity surrounding the term and what it actually entails. This paper aims to clearly define digitalization, a key enabler of Industry 4.0, and illustrate how it can be used for improvement through proposing an improvement cycle and an associated digitalization typology. These tools can be used by organizations to guide improvement processes, focusing on the new possibilities introduced by the enormous amounts of data currently available. The usage of the tools is illustrated by presenting four scenarios from Kanban control, where each scenario is mapped according to their digitalization level. © 2018, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd.

Software startups have long been a significant driver in economic growth and innovation. The on-going failure of the major number of startups calls for a better understanding of state-of-the-practice of startup activities. Objective With a focus on engineering perspective, this study aims at identifying the change in focus of research area and thematic concepts operating startup research. A systematic mapping study on 74 primary papers (in which 27 papers are newly selected) from 1994 to 2017 was conducted with a comparison with findings from previous mapping studies. A classification schema was developed, and the primary studies were ranked according to their rigour. We discovered that most research has been conducted within the SWEBOK knowledge areas software engineering process, management, construction, design, and requirements, with the shift of focus towards process and management areas. We also provide an alternative classification for future startup research. We find that the rigour of the primary papers was assessed to be higher between 2013–2017 than that of 1994–2013. We also find an inconsistency of characterizing startups. Future work can focus on certain research themes, such as startup evolution models and human aspects, and consolidate the thematic concepts describing software startups. © 2018. This is the authors’ accepted and refereed manuscript to the article. Locked until 19.6.2020 due to copyright restrictions. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/

Knowledge management plays a significant role in organizations; supporting organizations to deal effectively with changes, increasing their productivity and paving the way to development and innovation. Several scientific studies have addressed the relevance of applying knowledge management initiatives to improve projects as well as organizations that conduct projects. This paper will look at the interaction between knowledge management and big data within the context of projects. In this regard, this paper will discuss, among other things, (1) how big data can contribute to enhance knowledge management in projects and project-based organizations (2) what kind of pitfalls, challenges and opportunities that are associated with the interplay between knowledge management and big data (3) how this interplay can improve projects so that the projects can be carried out effectively and efficiently. These three questions are addressed by taking into consideration some of the important, underlying issues that are essential for ensuring improved decision making and performance in projects and project-based organizations. © 2018 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)

The peer review process is central to the scientific method, the advancement and spread of research, as well as crucial for individual careers. However, the single-blind review mode currently used in most Software Engineering (SE) venues is susceptible to apparent and hidden biases, since reviewers know the identity of authors. We perform a study on the benefits and costs that are associated with introducing double- blind review in SE venues. We surveyed the SE community’s opinion and interviewed experts on double-blind reviewing. Our results indicate that the costs, mostly logistic challenges and side effects, outnumber its benefits and mostly regard difficulty for authors in blinding papers, for reviewers in understanding the increment with respect to previous work from the same authors, and for organizers to manage a complex transition. While the surveyed community largely consents on the costs of DBR, only less than one-third disagree with a switch to DBR for SE journals, all SE conferences, and, in particular, ICSE; the analysis of a survey with authors of submitted papers at ICSE 2016 run by the program chairs of that edition corroborates our result.

In recent years however, more re-identification attacks have been attempted by researchers elsewhere in the world.Further findings are that most research in this area has been conducted on Global datasets by American researchers. The publication of increasing amounts of anonymised open source data has resulted in a worryingly rising number of successful re-identification attacks. This has a number of privacy and security implications both on an individual and corporate level.This paper uses a systematic literature review to investigate the depth and extent of this problem as reported in peer reviewed literature. Using a detailed protocol,seven research portals were explored, 10,873 database entries were searched, from which a subset of 220 papers were selected for further review. From this total, 55 papers were selected as being within scope and to be included in the final review.The main review findings are that 72.7% of all successful re-identification attacks have taken place since 2009. Most attacks use multiple datasets. The majority of them have taken place on global datasets such as social networking data, and have been conducted by US based researchers. Furthermore, the number of datasets can be used as an attribute.Because privacy breaches have security, policy and legal implications (e.g. data protection, Safe Harbor etc.), the work highlights the need for new and improved anonymisation techniques or indeed, a fresh approach to open source publishing.

This paper explores the application of learning designs featuring formalised and structured technology enhanced peer learning. These include student produced learning elements, peer review discussions and peer assessment in the BSc/MSc level summer course Restoration of European Ecosystems and Freshwaters (REEF), the Master thesis preparation seminars for the Master of Public Health (MPH) and the MOOC course Global Environmental Management (GEM). The application of student produced learning elements and peer review discussions is investigated by analyzing quotes from course evaluations and performing focus group interviews. The application of peer assessment is investigated by analyzing the agreement of peer assessment between students assessing the same assignment. Our analyses confirm previous research on the value of peer learning and peer assessment and we argue that there could also be a huge benefit from developing learning design patterns that facilitate informal peer learning and reinforce knowledge sharing practices. This paper explores the application of learning designs featuring formalised and structured technology enhanced peer learning. These include student produced learning elements, peer review discussions and peer assessment in the BSc/MSc level summer course Restoration of European Ecosystems and Freshwaters (REEF), the Master thesis preparation seminars for the Master of Public Health (MPH) and the MOOC course Global Environmental Management (GEM). The application of student produced learning elements and peer review discussions is investigated by analyzing quotes from course evaluations and performing focus group interviews. The application of peer assessment is investigated by analyzing the agreement of peer assessment between students assessing the same assignment. Our analyses confirm previous research on the value of peer learning and peer assessment and we argue that there could also be a huge benefit from developing learning design patterns that facilitate informal peer learning and reinforce knowledge sharing practices.

Published version. Source at http://dx.doi.org/10.1016/j.procs.2016.09.039 The implementation of Electronic Health Records (EHRs) in developing countries is considered a means for improving data quality and high quality care. However, existing research, indicate that EHRs have recorded greater quantity of bad data instead of improving the quality of data. The reasons for this is not exactly clear. Accordingly, this paper takes a broader socio-technical approach to explore the issues in more detail. The paper explores the relationship between EHRs and quality at the Pentecost Hospital Madina-Ghana by using an interpretative research approach. The result of our study indicates that the EHRs introduced at the hospital have so far had limited effect on data quality and that context-related challenges are the major pitfalls identified in the study. Based on a socio- technical approach the paper discusses the need to; consider users and technology as intertwined, how technology changes practice and how it formalizes roles and responsibilities in the process, why it requires collaborative work and why realistic goals as opposed to exaggerated expectation

Source: doi: 10.1109/ACCESS.2016.2599851 Collecting data via a questionnaire and analyzing them while preserving respondents' privacy may increase the number of respondents and the truthfulness of their responses. It may also reduce the systematic differences between respondents and non-respondents. In this paper, we propose a privacy- preserving method for collecting and analyzing survey responses using secure multi-party computation. The method is secure under the semi-honest adversarial model. The proposed method computes a wide variety of statistics. Total and stratified statistical counts are computed using the secure protocols developed in this paper. Then, additional statistics, such as a contingency table, a chi-square test, an odds ratio, and logistic regression, are computed within the R statistical environment using the statistical counts as building blocks. The method was evaluated on a questionnaire data set of 3158 respondents sampled for a medical study and simulated questionnaire data sets of up to 50 000 respondents. The computation time for the statistical analyses linearly scales as the number of respondents increases. The results show that the method is efficient and scalable for practical use. It can also be used for other applications in which categorical data are collected.

In the 10 years since the term was coined 'design fiction' has become an increasingly common approach in HCI research. The practice involves working with 'diegetic prototypes', that is prototypes that need not exist in reality, but instead exist from within a 'story world'. Although fictional aspects are not unusual in HCI prototyping methods (e.g. storyboards, personas, Wizard-of-Oz), the breadth and flexibility of design fiction poses new challenges. This paper originally featured quotes from peer reviews of design fiction orientated papers that have previously been submitted to ACM SIGCHI conferences in order to highlight inherent challenges when reviewing research that may be based upon or include elements of fiction. In response to the SIGCHI Executive Committee's request to not directly quote reviewers the quotes have now been redacted. This somewhat changes the paper's tone and also makes very clear that publishing discussions relating to peer reviews (or the reviews themselves) is extremely challenging.

Context. Empirical research consistently demonstrates that that scholarly peer review is ineffective, unreliable, and prejudiced. In principle, the solution is to move from contemporary, unstructured, essay-like reviewing to more structured, checklist-like reviewing. The Task Force created models—called “empirical standards”—of the software engineering community’s expectations for different popular methodologies. Objective. This paper presents a tool for facilitating more structured reviewing by generating review checklists from the empirical standards. Design. A tool that generates pre-submission and review forms using the empirical standards for software engineering research was designed and implemented. The pre-submission and review forms can be used by authors and reviewers, respectively, to determine whether a manuscript meets the software engineering community’s expectations for the particular kind of research conducted. Evaluation. The proposed tool can be empirically evaluated using lab or field randomized experiments as well as qualitative research. Huge, impractical studies involving splitting a conference program committee are not necessary to establish the effectiveness of the standards, checklists and structured review. Conclusions. The checklist generator enables more structured peer reviews, which in turn should improve review quality, reliability, thoroughness, and readability. Empirical research is needed to assess the effectiveness of the tool and the standards.

A simulation model based on parallel systems is established, aiming to explore the relation between the number of submissions and the overall standard of academic journals within a similar discipline under peer review. The model can effectively simulate the submission, review, and acceptance behaviors of academic journals in a distributed manner. According to the simulation experiments, it could possibly happen that the overall standard of academic journals deteriorates due to excessive submissions., Comment: This paper is already published in the journal IEEE Access and its copyright is fully open. The title was adjusted according to the review comments

This study answers a call for more qualitative research on project management to unveil “what goes on in social construction of projects and project management”. More precisely, we focus on how trust and control are balanced in an inter-organizational project collaboration in the Norwegian petroleum industry. Our findings suggest that balancing trust and control is a significant part of the project manager role. We claim that this area is both neglected in education and training and under-researched. The aim of the paper is to provide more knowledge on the trust-control dilemma in project management. Our qualitative approach provides rich empirical examples of dilemma situations and how balancing trust and control take place in a real-life project management context. The data was collected between 2010 and 2013 and consists of observational data from project meetings, document analysis and interviews.

Scientific peer review is pivotal to maintain quality standards for academic publication. The effectiveness of the reviewing process is currently being challenged by the rapid increase of paper submissions in various conferences. Those venues need to recruit a large number of reviewers of different levels of expertise and background. The submitted reviews often do not meet the conformity standards of the conferences. Such a situation poses an ever-bigger burden on the meta-reviewers when trying to reach a final decision. In this work, we propose a human-AI approach that estimates the conformity of reviews to the conference standards. Specifically, we ask peers to grade each other’s reviews anonymously with respect to important criteria of review conformity such as sufficient justification and objectivity. We introduce a Bayesian framework that learns the conformity of reviews from both the peer grading process, historical reviews and decisions of a conference, while taking into account grading reliability. Our approach helps meta-reviewers easily identify reviews that require clarification and detect submissions requiring discussions while not inducing additional overhead from reviewers. Through a large-scale crowdsourced study where crowd workers are recruited as graders, we show that the proposed approach outperforms machine learning or review grades alone and that it can be easily integrated into existing peer review systems.

User-centred design involves end-users or user groups during all the parts of the development process. The research project Chronic Pain aims to develop a shared decision making application for patients and physicians, addressing individually adapted pain treatment. The project employs a user-centred design process, and in middle of it, Covid-19 pandemic social distancing restrictions were imposed. This paper presents how the user-centred design process together with a patient organisation was transformed to a digital approach and the experiences from performing a remote co-creation user workshop. The digital approximation had a satisfactory result and the main contribution lies in the sharing of recommendations for how to practically apply a remote user-centred design methodology.

In this paper, the passing ability of football players is determined by building three generalized additive mixed models that each explains a different aspect of a pass’ success: difficulty, risk and potential. The models are built on data from the 2014–2016 seasons of the Norwegian top division Eliteserien, and their predictive power is tested on the 2017 season. The results provide insight into the factors affecting the success of a pass in Eliteserien. These include the location of the pass, the relationship to previous passes and to situations such as throw-ins, corners, free kicks or tackles, as well as conditions specific to the Eliteserien, such as the time of season and the ground surface type. Finally, the key pass makers in the league are identified.

This paper introduces the application and the results of a fuzzy number based questionnaire used for evaluating lecturers’ performance in a newly launched peer review programme in order to enhance ...

Personalization of the e-learning systems according to the learner’s needs and knowledge level presents the key element in a learning process. E-learning systems with personalized recommendations should adapt the learning experience according to the goals of the individual learner. Aiming to facilitate personalization of a learning content, various kinds of techniques can be applied. Collaborative and social tagging techniques could be useful for enhancing recommendation of learning resources. In this paper, we analyze the suitability of different techniques for applying tag-based recommendations in e-learning environments. The most appropriate model ranking, based on tensor factorization technique, has been modified to gain the most efficient recommendation results. We propose reducing tag space with clustering technique based on learning style model, in order to improve execution time and decrease memory requirements, while preserving the quality of the recommendations. Such reduced model for providing tag-based recommendations has been used and evaluated in a programming tutoring system. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Peer assessment is widely used in educational settings as an alternative evaluation approach. It brings various benefits to the students, increasing engagement and interactivity and fostering critical thinking and reflection. Several platforms for managing the peer review process have been proposed in the literature, but most of them are confined to a particular domain or course and have various limitations related to reliability, reviewer allocation, reputation and training of reviewers or instructor support. In an attempt to address these challenges, we propose an innovative general-purpose peer assessment platform, called LearnEval. In this paper we focus on the student module part of the system, describing its functionalities, pedagogical rationale and implementation details.

Serious and fatal accidents happen on construction sites every year. However, earlier research indicate that the root of the accidents can often be found in the early phases of the project process. In this paper, we examine what occurs in the early phases of project management in construction that can be harmful to health and safety in the construction phase. The investigation focuses on two main research questions: 1) can health and safety in the construction phase be improved during the early phases of project management, and, if so, 2) what are the main challenges of health and safety in the early phases of project management that need to be addressed in order to do so? The data presented is collected by interviewing regional safety representatives in Norway. They point to three main challenges to health and safety in the early phases of construction projects: 1) lack of competence, 2) lack of prioritization, and 3) lack of consequence. In conclusion, we suggest that educational institutions, construction clients and public authorities all need to play their part to improve the situation.

Coordinating research objectives concurrently with product development and engineering is a challenge in student-run CubeSat projects. The literature review from several universities and the International Council of Systems Engineering (INCOSE) Space Systems Working Group (SSWG) all acknowledge the need for a better methodology. This paper describes findings from a university CubeSat exploratory case study related to team management, reviews and design. Introducing Systems Engineering (SE) tools such as formalized reviews, A3 reports, systemigrams and N2 dependency maps have aided the development process. This was evident through faster development iterations and fewer design confusions. Results show that relocation into a common space based on recommendations from Concurrent Engineering (CE) has improved overall communication and team work. © 2019 The Authors. Published by Elsevier B.V.This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)

It is estimated that over 1 billion people are active mobile phone users in 2018. When using a mobile phone, there are a variety of ways to authenticate and “secure” the device, and biometric authentication is becoming an increasingly common way to do this, however, biometric authentication is not always as usable as it could be. Both usability and security are important, yet many people believe that there is a trade-off between the two. The focus of this paper was to better understand usability, computer security, and within computer security more specifically biometric authentication, and how all three can work together to create systems that are both usable and secure. A survey and interviews were conducted based on previous research to understand perceptions from the general population, usability experts, and security/biometrics experts. The results do indicate that there is indeed a perceived trade-off between usability and computer security.

This is the peer reviewed version which has been published in final form at [DOI]. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions., Software testing is crucial in continuous integration (CI). Ideally, at every commit, all the test cases should be executed, and moreover, new test cases should be generated for the new source code. This is especially true in a Continuous Test Generation (CTG) environment, where the automatic generation of test cases is integrated into the continuous integration pipeline. In this context, developers want to achieve a certain minimum level of coverage for every software build. However, executing all the test cases and, moreover, generating new ones for all the classes at every commit is not feasible. As a consequence, developers have to select which subset of classes has to be tested and/or targeted by test‐case generation. We argue that knowing a priori the branch coverage that can be achieved with test‐data generation tools can help developers into taking informed decision about those issues. In this paper, we investigate the possibility to use source‐code metrics to predict the coverage achieved by test‐data generation tools. We use four different categories of source‐code features and assess the prediction on a large data set involving more than 3'000 Java classes. We compare different machine learning algorithms and conduct a fine‐grained feature analysis aimed at investigating the factors that most impact the prediction accuracy. Moreover, we extend our investigation to four different search budgets. Our evaluation shows that the best model achieves an average 0.15 and 0.21 MAE on nested cross‐validation over the different budgets, respectively, on EVOSUITE and RANDOOP. Finally, the discussion of the results demonstrate the relevance of coupling‐related features for the prediction accuracy.

Modelling techniques have been part of the software engineering practice for decades. Despite Model-Driven Software Engineering (Mde) being spread in the industry, effective teaching through meaningful use cases remains a challenge. Some guidelines and teaching experiences have been reported that either focus on the content of an Mde course, or on tooling environments designed for students. Furthermore, little emphasis has been put on the usability in the context of Software Language Engineering (Sle), which may lead to technically valid languages, but which are difficult to use. In this paper, we report on our attempt to combine a practical use case for Sle and a structured peer review focusing on the language usability and its tooling environment. By putting students in both the seats of language designers and users, we believe to empower students with suitable technical Sle skills as well as making them aware that a good software language requires more than theoretical validity to be adopted.

This paper describes Readersourcing 2.0, an ecosystem providing an implementation of the Readersourcing approach. Readersourcing is proposed as an alternative to the standard peer review activity that aims to exploit the otherwise lost opinions of readers. To achieve this, Readersourcing 2.0 implements two different models based on the so-called codetermination algorithms. We describe the requirements, present the overall architecture, and show how the end-user can interact with the system. Readersourcing 2.0 will be used in the future to study also other topics, like the idea of sheperding the users to achieve a better quality of the reviews and the differences between a review activity carried out with a single-blind or a double-blind approach., Preprint for the IRCDL 2019 conference

This paper analyzes and presents the solar energy research status in the GCC countries using Scopus-database. Its findings may be valuable for researchers, politicians, industry, or decision makers to see how much the participation of the GCC countries in the field of solar energy and how it compares to other countries. The data gathered from Scopus database aided in outlining and identifying the active institutions and researchers in the field of solar energy in the GCC. In terms of the largest contribution, Saudi Arabia has the highest research output (with more than 60% of the published articles in the GCC), followed by the UAE (about 17%) and the remaining percentage is the contribution of the other GCC countries.

ContextThe power of open source software peer review lies in the involvement of virtual communities, especially users who typically do not have a formal role in the development process. As communities grow to a certain extent, how to organize and support the peer review process becomes increasingly challenging. A universal solution is likely to fail for communities with varying characteristics. ObjectiveThis paper investigates differences of peer review practices across different open source software communities, especially the ones engage distinct types of users, in order to offer contextualized guidance for developing open source software projects. MethodComparative case studies were conducted in two well-established large open source communities, Mozilla and Python, which engage extremely different types of users. Bug reports from their bug tracking systems were examined primarily, complemented by secondary sources such as meeting notes, blog posts, messages from mailing lists, and online documentations. ResultsThe two communities differ in the key activities of peer review processes, including different characteristics with respect to bug reporting, design decision making, to patch development and review. Their variances also involve the designs of supporting technology. The results highlight the emerging role of triagers, who bridge the core and peripheral contributors and facilitate the peer review process. The two communities demonstrate alternative designs of open source software peer review and their tradeoffs were discussed. ConclusionIt is concluded that contextualized designs of social and technological solutions to open source software peer review practices are important. The two cases can serve as learning resources for open source software projects, or other types of large software projects in general, to cope with challenges of leveraging enormous contributions and coordinating core developers. It is also important to improve support for triagers, who have not received much research effort yet.

The Scaled Agile Framework (SAFe) has been adopted by a large number of organizations to scale agile to large enterprises that develop software. At the moment, SAFe seems to be the most predominant agile scaling framework. Despite the current popularity of SAFe in the software intensive industry, there exists surprisingly little research on the adoption and usage of SAFe. To study what is known about SAFe with respect to benefits and challenges of its usage, we conducted a multivocal literature review. The preliminary version of the results are presented in this poster version of paper. We identified 48 unique adoption cases in 46 organizations from both peer reviewed and non peer reviewed (grey literature) sources.

The construction industry uses project work as its main working method and is at the same time exposed to serious risks to health, safety and environment (HSE). Project managers play an important role to ensure high performance on HSE in construction projects, and the first position gained by many engineers after graduation is often the role of project manager. This paper examines how different aspects of HSE are included in the teaching of project management (PM) in the bachelor education of construction engineers in Norway. The examination includes analysing learning outcomes, curriculum, teaching and final exams in PM courses at the three largest educational institutions in respect to graduating bachelors in construction engineering in Norway. The findings show that the inclusion of HSE in PM courses varies and that students of construction engineering need to learn more about how to influence HSE in projects. Two elements are largely lacking in the education: 1) how to influence HSE performance during the early stages of construction projects, and 2) how to influence safety culture in projects. A stronger inclusion of these perspectives in teaching could improve the ability of future engineers to reinforce effective health and safety measures in the construction industry.

Enterprise software needs to be customizable, and the customization needs from a customer are often beyond what the software vendor can predict in advance. In the on-premises era, customers do deep customizations beyond vendor's prediction by directly modifying the vendor's source code and then build and operate it on their own premises. When enterprise software is moving to cloud-based multi-tenant SaaS (Software as a Service), it is no longer possible for customers to directly modify the vendor's source code, because the same instance of code is shared by multiple customers at runtime. Therefore, the question is whether it is still possible to do deep customization on multi-tenant SaaS. In this paper, we give an answer to this question with a novel architecture style to realize deep customization of SaaS using intrusive microservices. We evaluate the approach on an open source online commercial system, and discuss the further research questions to make deep customization applicable in practice. Deep customization of multi-tenant SaaS using intrusive microservices

The subject of this study was the product development project creating a new innovative proof-of-concept (POC) prototype device that could control a connected industrial overhead crane in order to perform automatic or semi-automatic high precision lifts within a limited time frame. The development work focused on innovating a new measuring concept, which was parallel to finding suitable sensors for the application. Furthermore, the project resulted in a closed loop control system with Industrial Internet connected sensors and a user interface for factory workers. The prototyping journey is depicted to illustrate the decisions made during the product development project to contribute to both the pragmatic and the process discussion in the field of Industrial Internet. The purpose of this research was to explore and generate hypotheses for how new applications should be developed for heavy industry connected devices. The research question is: what are the implications of applying agile product development methods, such as Wayfaring, to heavy industrial machinery and Industrial Internet -based problems? The methodologies used in this paper, in addition to developing the device, are case study research and hypotheses generated from case studies. The hypotheses generated include that it is also possible to prototype large size connected machinery with low-cost and in a short time, and investment decisions for heavy Industrial Internet products become easier with concrete data from proof-of-concept prototypes by creating knowledge about the investment risk and the value proposition.

This is a report on the ninth edition of the Conference and Labs of the Evaluation Forum (CLEF 2018), held in early September 2018, in Avignon, France. CLEF was a four day event combining a Conference and an Evaluation Forum. The Conference featured keynotes by Nicholas Belkin, Julio Gonzalo, and Gabriella Pasi, and presentation of 29 peer reviewed research papers covering a wide range of topics in addition to many posters. The Evaluation Forum consisted to ten Labs: CENTRE, CheckThat, DynSe, eRisk, eHealth, ImageCLEF, LifeCLEF, Cultural Microblog Contextualization, PAN, and PIR-CLEF, addressing a wide range of tasks, media, languages, and ways to go beyond standard test collections.

The Wireless Body Area Network (WBAN) has gained popularity as a new technology for e-Health, and is considered as one of the key research areas in computer science and healthcare applications. WBAN collects patients’ data, monitors constantly their physiological parameters, using small implantable or wearable sensors, and communicates these data using wireless communication techniques in short range. WBAN is playing a huge role in improving the quality of healthcare. Still, due to sensitive and concurrent nature of e-Heath systems, current research has showed that designers must take into considerations the security and privacy protection of the data collected by a WBAN to safeguard patients from different exploits or malicious attacks, since e-Health technologies are increasingly connected to the Internet via wireless communications. In this paper we outline the most important security requirements for WBANs. Furthermore, we discuss key security threats to avoid. Finally, we conclude with a summary of security mechanisms to follow that address security and privacy concerns of WBANs, and need to be explored in an increasingly connected healthcare world. This Article is brought to you for free and open access by the Publication and Journals at UBT Knowledge Center.

Resource Allocation (RAlloc) is one of the most important tasks in organizing humanitarian response to humanitarian crises. It is not only that adequate and efficient RAlloc save lives and reduce damages caused by humanitarian crises, but RAlloc must be fast and efficient to save time and resources. Given that RAlloc is a type of a decision making process, it is expected that decision on RAlloc are based on accurate and relevant information generated at various stages of humanitarian response. In this paper we promote Semantic Resource Allocation (SemRAlloc) tool which a) collects and interprets the semantics of an environment where RAlloc is required and b) the reasons upon the semantics of that environment in order to make appropriate RAlloc. The tool is built with computations based on SWRL enabled OWL ontologies. The prototype has been implemented as a desk-top application which can also run in mobile/wireless environments, including Android smart phones.

Peer review is a valuable educational activity, especially in social learning settings, group work and project-based learning scenarios. It has the potential to foster critical thinking and reflection, expand students’ perspectives and understanding and increase engagement and interactivity. Several software tools for supporting the student peer review process have been proposed so far, but few of them are integrated in an all-encompassing learning environment. Therefore, our proposal is to extend an existing fully-fledged social learning platform, called eMUSE, with a peer evaluation module; the tool offers a wide range of functionalities, both for the student and the instructor. The peer evaluation is closely integrated with the educational social media tools and the project-based learning scenario; rather than focusing on a single written assignment, the module supports a more in-depth monitoring and assessment of peers’ work. The platform has been successfully used in practice and preliminary evaluation results are reported in the paper.

Peer assessment is the most common approach to evaluating scientific work, and it is also gaining popularity for scaling evaluation of student work in large and distributed classes. The key idea is that each peer reviewer or grader rates a relatively small subset of the items, and that some method of manual, semi-automatic, or fully-automatic aggregation of all assessments defines the eventual rating of all items – the grade in peer grading, or whether to accept or reject a scientific manuscript. In this paper, we explore in how far a Bayesian Ordinal Peer Assessment (BOPA) method can provide additional decision support when making acceptance/rejection decisions for a scientific conference. Using data from the 2015 ACM Conference on Knowledge Discovery and Data Mining (KDD), where this system was deployed, we discuss the potential merit of the BOPA approach compared to conventional decision support offered by the Microsoft Conference Management System (CMT).