Search

For the field experimentcarried out in a public Andalusian software company to evaluate the effects of the practise of mindfulness, this laboratory package contains i) the experimental material used during its execution, ii) the data sets (both raw data and curatedfiles) derived from the experimental tasks (questionnaires and performance), and iii) a Jupyter notebook containing the statistical analysis scripts, and raw results of script execution, including plots. Additionally, the repository includes a full report of the sphericity and homoscedasticity tests in the results, which are also available for studying the applicability of parametric tests (https://github.com/isa-group/ESEM-2023-Mindfulness-Labpack).

Context. Mindfulness is a meditation technique whose main goal is keeping the mind calm and educating attention by focusing only on one thing at a time, usually breathing. The reported benefits of its continued practice can be of interest for Software Engineering students and practitioners, especially in tasks like conceptual modeling, in which concentration and clearness of mind are crucial. Goal. In order to evaluate whether Software Engineering students enhance their conceptual modeling performance after several weeks of mindfulness practice, a series of three controlled experiments were carried out at the University of Seville during three consecutive academic years (2013–2016) involving 130 students. Method. In all the experiments, the subjects were divided into two groups. While the experimental group practiced mindfulness, the control group was trained in public speaking as a placebo treatment. All the subjects developed two conceptual models based on a transcript of an interview, one before and another one after the treatment. The results were compared in terms of conceptual modeling quality (measured as effectiveness, i.e. the percentage of model elements correctly identified) and productivity (measured as efficiency, i.e. the number of model elements correctly identified per unit of time). Results. The statistically significant results of the series of experiments revealed that the subjects who practiced mindfulness developed slightly better conceptual models (their quality was 8.16% higher) and they did it faster (they were 46.67% more productive) than the control group, even if they did not have a previous interest in meditation. Conclusions. The practice of mindfulness improves the performance of Software Engineering students in conceptual modeling, especially their productivity. Nevertheless, more experimentation is needed in order to confirm the outcomes in other Software Engineering tasks and populations. Ministerio de Economía y Competitividad TIN2015–70560–R Junta de Andalucía US–1264651 Ministerio de Ciencia, Innovación y Universidades RTI2018–101204–B–C22 Ministerio de Ciencia, Innovación y Universidades RTI2018- 101204–B–C21 Junta de Andalucía P18–FR–2895

Expert judgement is a common method for software effort estimations in practice today. Estimators are often shown extra obsolete requirements together with the real ones to be implemented. Only one previous study has been conducted on if such practices bias the estimations. We conducted six experiments with both students and practitioners to study, and quantify, the effects of obsolete requirements on software estimation. By conducting a family of six experiments using both students and practitioners as research subjects ($$N=461$$ N = 461 ), and by using a Bayesian Data Analysis approach, we investigated different aspects of this effect. We also argue for, and show an example of, how we by using a Bayesian approach can be more confident in our results and enable further studies with small sample sizes. We found that the presence of obsolete requirements triggered an overestimation in effort across all experiments. The effect, however, was smaller in a field setting compared to using students as subjects. Still, the over-estimations triggered by the obsolete requirements were systematically around twice the percentage of the included obsolete ones, but with a large 95% credible interval. The results have implications for both research and practice in that the found systematic error should be accounted for in both studies on software estimation and, maybe more importantly, in estimation practices to avoid over-estimations due to this systematic error. We partly explain this error to be stemming from the cognitive bias of anchoring-and-adjustment, i.e. the obsolete requirements anchored a much larger software. However, further studies are needed in order to accurately predict this effect.

Doing the right kind of testing has always been one of main challenging and a decisive task for industry. To choose right software testing method(s), industry needs to have an exact objective knowledge of their effectiveness, efficiency, and applicability conditions. The most common way to evaluate testing methods, for such knowledge, is with empirical studies. Reliable and comprehensive evidence can be obtained by aggregating the results of different empirical studies (family of experiments) taking into account their findings and limitations. We conducted a study to investigate the current state of the art of empirical knowledge base of three testing methods. We found that although the empirical studies conducted so far to evaluate testing methods contain many important and interesting results; however, we still lack factual and generalizable knowledge about performance and applicability conditions of testing methods(s), making it unfeasible to be readily adopted by the industry. Moreover, we tried to identify the major factors responsible for limiting academia from producing significantly reliable results having an industrial impact. We believe that besides effective and long-term academia-industry collaboration, there is a need for more systematic, quantifiable and comprehensive empirical studies (which provides scope for aggregation using rigorous techniques), mainly replications so as to create an effective and applicable knowledge base about testing methods which potentially can fill the gap between academia and industry.

Context: Test-driven development (TDD) is an agile software development approach that has been widely claimed to improve software quality. However, the extent to which TDD improves quality appears to be largely dependent upon the characteristics of the study in which it is evaluated (e.g., the research method, participant type, programming environment, etc.). The particularities of each study make the aggregation of results untenable. Objectives: The goal of this paper is to: increase the accuracy and generalizability of the results achieved in isolated experiments on TDD, provide joint conclusions on the performance of TDD across different industrial and academic settings, and assess the extent to which the characteristics of the experiments affect the quality-related performance of TDD. Method: We conduct a family of 12 experiments on TDD in academia and industry. We aggregate their results by means of meta-analysis. We perform exploratory analyses to identify variables impacting the quality-related performance of TDD. Results: TDD novices achieve a slightly higher code quality with iterative test-last development (i.e., ITL, the reverse approach of TDD) than with TDD. The task being developed largely determines quality. The programming environment, the order in which TDD and ITL are applied, or the learning effects from one development approach to another do not appear to affect quality. The quality-related performance of professionals using TDD drops more than for students. We hypothesize that this may be due to their being more resistant to change and potentially less motivated than students. Conclusion: Previous studies seem to provide conflicting results on TDD performance (i.e., positive vs. negative, respectively). We hypothesize that these conflicting results may be due to different study durations, experiment participants being unfamiliar with the TDD process, or case studies comparing the performance achieved by TDD vs. the control approach (e.g., the waterfall model), each applied to develop a different system. Further experiments with TDD experts are needed to validate these hypotheses.

Context: Software Engineering students are often excellent developers although they may occasionally encounter difficulties with certain tasks such as conceptual modelling, in which mental clarity plays an essential role. With several years of experience in practising mindfulness (a meditation technique that calm the mind to see with clarity), we hypothesise that several weeks of continued mindfulness practise may increase the performance of students regarding conceptual modelling since the proven benefits of mindfulness include increased concentration and mental clarity. In order to ascertain this hypothesis, a family of controlled experiments, involving 130 students, has been carried out at the University of Seville over three consecutive academic years. Subsequent to the analysis of the individual experiments, a meta—analysis was conducted. Aim: This chapter helps understand how various datasets have been generated and traces their evolution across the family of experiments and the resulting meta—analysis. Not only does the process include the integration of datasets into a single target dataset with the complete set of data, but it also improves the structure of the dataset and adapts it to several statistical analysis tools. Method: Data collection was manually carried out during the individual experiments by means of gathering the scores of the students attained in conceptual modelling exercises. In order to obtain the complete but also simple dataset of the family of experiments, certain dataset columns were necessarily selected and renamed. New relevant information was also added to form a conclusive meta—analysis. The dataset was adapted to the type and format of SPSS for the execution of a further analysis, complementary to the main study in R. The MetaMind dataset is the set of datasets involved in the present research. The MetaMind dataset has become a helpful resource for the independent reproducibility of the results, to ascertain the evolution of the process across a family of experiments, and to enable external replications of the controlled experiment. Ministerio de Ciencia e Innovación HORATIO (RTI2018-101204–B–C21) Junta de Andalucía APOLO (US-1264651) Junta de Andalucía EKIPMENTPLUS (P18–FR–2895)

Domain-specific languages (DSLs) allow developers to write code at a higher level of abstraction compared with general-purpose languages (GPLs). Developers often use DSLs to reduce the complexity of GPLs. Our previous study found that developers performed program comprehension tasks more accurately and efficiently with DSLs than with corresponding APIs in GPLs. This study replicates our previous study to validate and extend the results when developers use IDEs to perform program comprehension tasks. We performed a dependent replication of a family of experiments. We made two specific changes to the original study: (1) participants used IDEs to perform the program comprehension tasks, to address a threat to validity in the original experiment and (2) each participant performed program comprehension tasks on either DSLs or GPLs, not both as in the original experiment. The results of the replication are consistent with and expanded the results of the original study. Developers are significantly more effective and efficient in tool-based program comprehension when using a DSL than when using a corresponding API in a GPL. The results indicate that, where a DSL is available, developers will perform program comprehension better using the DSL than when using the corresponding API in a GPL.

Automated web testing is an appealing option, especially when continuous testing practices are adopted. However, web test cases are known to be fragile and to break easily when a web application evolves. The Page Object (PO) design pattern addresses such problem by providing a layer of indirection that decouples test cases from the internals of the web page, where web page elements are located and triggered by the web tests. However, PO development could potentially introduce an additional burden to the already strictly constrained testing activities. This paper reports an empirical investigation of costs and benefits due to the introduction of the PO pattern in web test suite development. In particular, we conducted a family of controlled experiments in which test cases were developed with and without the PO pattern. While the benefits of POs did not compensate for the extra development effort they require in the limited experimental setting of our study, results indicate that when the test suite to be developed is at least 10× larger, test development becomes more efficient with than without POs.

Context There are several methods to elicit requirements through interviews between an end-user and a team of software developers. The choice of the best method in this context is usually on subjective developers’ preferences instead of objective reasons. There is a lack of empirical evaluations of methods to elicit requirements that help developers to choose the most suitable one. Objective This paper designs and conducts a family of experiments to compare three methods to elicit requirements: Unstructured Interviews, where there is no specific protocol or artifacts; Joint Application Design (JAD), where each member of the development team has a specific role; Paper Prototyping, where developers contrast the requirements with the end-user through prototypes. Method The experiment is a between-subjects design with next response variables: number of requirements, time, diversity, completeness, quality and performance. The experiment consists of a maximum of 4 rounds of interviews between students that play the role of developers and an instructor that plays the role of client. Subjects had to elaborate a requirements specification document as results of the interviews. We recruited 167 subjects in 4 replications in 3 years. Subjects were gathered in development teams of 6 developers at most, and each team was an experimental unit. Results We found some significant differences. Paper Prototyping yields the best results to elicit as many requirements as possible, JAD requires the highest time to report the requirements and the least overlapping, and Unstructured Interviews yields the highest overlapping and the lowest time to report the requirements. Conclusions Paper Prototyping is the most suitable for eliciting functional requirements, JAD is the most suitable for non-functional requirements and to avoid overlapping, Unstructured Interviews is the fastest but with poor quality in the results.

Context: Families of experiments (i.e., groups of experiments with the same goal) are on the rise in Software Engineering (SE). Selecting unsuitable aggregation techniques to analyze families may undermine their potential to provide in-depth insights from experiments’ results. Objectives: Identifying the techniques used to aggregate experiments’ results within families in SE. Raising awareness of the importance of applying suitable aggregation techniques to reach reliable conclusions within families. Method: We conduct a systematic mapping study (SMS) to identify the aggregation techniques used to analyze families of experiments in SE. We outline the advantages and disadvantages of each aggregation technique according to mature experimental disciplines such as medicine and pharmacology. We provide preliminary recommendations to analyze and report families of experiments in view of families’ common limitations with regard to joint data analysis. Results: Several aggregation techniques have been used to analyze SE families of experiments, including Narrative synthesis, Aggregated Data (AD), Individual Participant Data (IPD) mega-trial or stratified, and Aggregation of $p$ p -values. The rationale used to select aggregation techniques is rarely discussed within families. Families of experiments are commonly analyzed with unsuitable aggregation techniques according to the literature of mature experimental disciplines. Conclusion: Data analysis’ reporting practices should be improved to increase the reliability and transparency of joint results. AD and IPD stratified appear to be suitable to analyze SE families of experiments.

Context: Model-driven approaches are well-known in the academia but one possible showstopper to a wider adoption in the industry is the limited empirical evidence for their claimed advantages and benefits, that could convince the decision makers. Objective: The aim of this work is gauging one of the claimed benefits of model-driven approaches, namely improvement in maintainability, with respect to a code-centric approach. Method: We conducted a family of five experiments involving 100 students that possessed different levels of education (64 Bachelor, 25 Master, and 11 PhD students; in groups sized 11 to 26 per individual experiment). In these experiments, UniMod – a State-based tool for Model-Driven Development using the Unified Modeling Language – is compared with Java-based code-centric programming, in a software maintenance scenario, with the goal of analyzing the effect on the time to perform the maintenance tasks, the correctness of the modified artifacts, and the efficiency. Results: The results show a reduction in time to accomplish the tasks and no impact on correctness. The adoption of the UniMod-MDD approach almost doubles the developers’efficiency, and in presence of a higher software engineering experience the efficiency is even three times higher. Conclusions: We found that the usage of the UniMod-MDD approach in a software maintenance scenario provides benefits over a pure code-centric approach. The benefits deriving from the UniMod-MDD approach are appreciable for all the categories of students, although with differences.

It is increasingly common to use languages and notations, mainly of a graphical nature, to assist in the design and specification of learning systems. There are several proposals, although few of them support the modeling of collaborative tasks. In this paper, we identify the main features to be considered for modeling this kind of activities and we propose the use of the CIAN notation for this purpose. In this work, we also try to empirically analyze the quality (in particular the understandability) of that notation. To this end, three empirical studies have been conducted. In these experiments we used several sources of information: subjective perception of the designers, their profiles and their performance on a set of understandability exercises, as well as the physical evidence provided by an eye tracker device. The results obtained denote positive perceptions about the use of the CIAN notation for modeling collaborative learning activities.

We carried out a family of experiments to investigate whether the use of UML models produced in the requirements analysis process helps in the comprehensibility and modifiability of source code. The family consists of a controlled experiment and 3 external replications carried out with students and professionals from Italy and Spain. 86 participants with different abilities and levels of experience with UML took part. The results of the experiments were integrated through the use of meta-analysis. The results of both the individual experiments and meta-analysis indicate that UML models produced in the requirements analysis process influence neither the comprehensibility of source code nor its modifiability.

We carried out a family of controlled experiments to investigate whether the use of abbreviated identifier names, with respect to full-word identifier names, affects fault fixing in C and Java source code. This family consists of an original (or baseline) controlled experiment and three replications. We involved 100 participants with different backgrounds and experiences in total. Overall results suggested that there is no difference in terms of effort, effectiveness, and efficiency to fix faults, when source code contains either only abbreviated or only full-word identifier names. We also conducted a qualitative study to understand the values, beliefs, and assumptions that inform and shape fault fixing when identifier names are either abbreviated or full-word. We involved in this qualitative study six professional developers with 1--3 years of work experience. A number of insights emerged from this qualitative study and can be considered a useful complement to the quantitative results from our family of experiments. One of the most interesting insights is that developers, when working on source code with abbreviated identifier names, adopt a more methodical approach to identify and fix faults by extending their focus point and only in a few cases do they expand abbreviated identifiers.

Objective : The main objective of our study is to assess whether the use of UML (Unified Modeling Language) object diagrams improves comprehensibility of software design when this kind of diagrams is added to UML class diagrams. Method : We have conducted a family of four controlled experiments. We involved groups of bachelor and master students. Results : Results suggest that the use of object diagrams does not always introduce significant benefits in terms of design comprehensibility. We found that benefits strongly depend on the experience of participants and their familiarity with UML. More experienced participants achieved better design comprehensibility when provided with both class and object diagrams, while less experienced seemed to be damaged when using class and object diagrams together. Results also showed the absence of substantial variations in the time needed to comprehend UML models, with or without object diagrams. Implications : Our results suggest that it is important to be aware and take into account experience and UML familiarity before using object diagrams in software modeling.

Web applications should be usable in order to be accepted by users and to improve their success probability. Despite the fact that this requirement has promoted the emergence of several usability evaluation methods, there is a need for empirically validated methods that provide evidence about their effectiveness and that can be properly integrated into early stages of Web development processes. Model-driven Web development processes have grown in popularity over the last few years, and offer a suitable context in which to perform early usability evaluations due to their intrinsic traceability mechanisms. These issues have motivated us to propose a Web Usability Evaluation Process (WUEP) which can be integrated into model-driven Web development processes. This paper presents a family of experiments that we have carried out to empirically validate WUEP. The family of experiments was carried out by 64 participants, including PhD and Master¿s computer science students. The objective of the experiments was to evaluate the participants¿ effectiveness, efficiency, perceived ease of use and perceived satisfaction when using WUEP in comparison to an industrial widely used inspection method: Heuristic Evaluation (HE). The statistical analysis and meta-analysis of the data obtained separately from each experiment indicated that WUEP is more effective and efficient than HE in the detection of usability problems. The evaluators were also more satisfied when applying WUEP, and found it easier to use than HE. Although further experiments must be carried out to strengthen these results, WUEP has proved to be a promising usability inspection method for Web applications which have been developed by using model-driven development processes., The authors would like to thank all the participants in the experiments, along with the usability experts that supported certain tasks of the evaluation design stage, and of which the control group was composed. This research work is funded by the MULTIPLE project (TIN2009-13838) and the FPU program (AP2007-03731) from the Spanish Ministry of Science and Education.

I will talk about the family of experiments known as precision measurement and null experiments. These experiments, often associated with the determination of a fundamental constant or the testing of one of the accepted laws of Nature at some extreme of magnitude, have always provided a challenge to both a scientist's cunning and his or her experimental skills. In my talk I will use as one example, C.V. Boys' 1894 measurement of big G. [C.V. Boys, one of my physics heroes, wrote papers that evidenced his humanity and sense of humor and are, accordingly, very enjoyable to read. Unfortunately, journal editors go to great lengths to remove from today's publications any hint of humanity, humor, or enjoyment derived from doing science.]

The objective of this paper is to empirically evaluate OOmFPWeb, a functional size measurement procedure for Web applications. We analyzed four data sets from a family of experiments conducted in Spain, Argentina and Austria. Results showed that OOmFPWeb is efficient when compared to current industry practices. OOmFPWeb produced reproducible functional size measurements and was perceived as easy to use and useful by the study participants, who also expressed their intention to use OOmFPWeb in the future. The analysis further supports the validity and reliability of the technology acceptance model (TAM)-based evaluation instrument used in the study.

Magic angle spinning solid-state NMR (MAS ssNMR) spectroscopy is a powerful method for structure determination of biomacromolecules that are recalcitrant to crystallization (membrane proteins and fibrils). Conventional multidimensional ssNMR methods acquire one experiment at a time. This approach is time consuming and discards orphan (unused) spin operators. Relatively low sensitivity and poor resolution of protein samples require long acquisition times for multidimensional ssNMR experiments. Here, we describe our recent progress in the development of multiple acquisition solid-state NMR methods for protein structure determination. A family of experiments called Polarization Optimized Experiments (POE) were designed, in which we utilized the orphan spin operators that are discarded in classical multidimensional NMR experiments, recovering them to allow simultaneous acquisition of multiple 2D and 3D experiments, all while using conventional probes with spectrometers equipped with one receiver. Three strategies namely, DUMAS, MEIOSIS, and MAeSTOSO were used for the concatenation of various 2D and 3D experiments. These methods open up new avenues for reducing the acquisition times of multidimensional experiments for biomolecular ssNMR spectroscopy.

The Teleo-Reactive approach facilitates reactive system development without losing sight of the system goals. Objective: To introduce TRiStar as an extension of i* notation to specify Teleo-Reactive systems. To evaluate whether the notational extension is an improvement in terms of effectiveness and efficiency over the original language when it is used to specify Teleo-Reactive systems. Method: A family of experiments was carried out with final-year engineering students and experienced software development professionals in which the participants were asked to fill in a form designed to evaluate the efficiency and effectiveness of each of the languages. Results: Both the statistical results of the experiments, analyzed separately, and the meta-analysis of the experiments as a whole, allow us to conclude that TRiStar notation is more effective and efficient than i* as a requirements specification language for modeling Teleo-Reactive systems. Conclusion: The extensions made on i* have led to TRiStar definition, a more effective and efficient goal-oriented notation than the original i* language. This work was partially supported by the insPIre (ref. TIN2012-34003), cDrone (ref. TIN2013- 45920-R) and ViSelTR (ref. TIN2012-39279) projects of the Spanish Government. Also Spanish Ministerio de Educación, Cultura y Deporte, Subprograma Estatal de Movilidad, Plan Estatal de Investigación Científica y Técnica y de Innovación 2013-2016 for grant CAS14/00238

Context: Although various success stories of model-based approaches are reported in literature, there is still a significant resistance to model-based development in many software organizations because the UML is perceived to be expensive and not necessarily cost-effective. It is also important to gather empirical evidence in which context and under which conditions the UML makes or does not make a practical difference. Objective: Our objective is to provide empirical evidence as to which UML diagrams are more helpful during software maintenance: Forward Designed (FD) UML diagrams or Reverse Engineered (RE) UML diagrams. Method: We carried out a family of experiments which consisted of one experiment and two replications with a total of 169 Computer Science undergraduate students. Results: The individual data analysis and the meta-analysis conducted on the whole family, show a tendency in favor of FD diagrams and are significantly different as regards the effectiveness and efficiency of the subjects who participated and played the role of maintainers. The analysis of the qualitative data, collected using a post-experiment survey, reveals that the subjects did not consider RE diagrams helpful. Conclusions: Our findings show that there are some objective results (descriptive statistics or statistical tests) related to the maintenance effectiveness and efficiency in favor of the use of FD UML diagrams during software maintenance. Subjective opinions also lead us to recommend the use of UML diagrams (especially class diagrams) created during the design phase for software maintenance because they improve the understanding of the system in comparison with RE diagrams. Nevertheless, we can only assume that these results are valid in the context of Computer Science undergraduate students when working with small systems related to well-known domains, and other contexts should be explored in order to reaffirm the results in an industrial context by carrying out replications with professionals. Ministerio de Economía y Competitividad TIN2012-37493-C03-01

In this paper, we present the results from a family of experiments conducted to assess whether the level of formality/precision in workflow modeling, based on UML activity diagrams, influences two aspects of construct comprehensibility: correctness of understanding and task completion time. In particular, we have considered two styles for workflow modeling with different levels of formality: a precise style (with specific rules and imposed constraints) and an ultra-light style (no rules, no imposed constraints). Experiments were conducted with 111 participants (Bachelor and Master students). In each experiment, participants accomplished comprehension tasks on two workflows, modeled either with the precise style or with a lighter variant. The main results from our data analysis can be summarized as follows: (i) all participants achieved a significantly better comprehension of workflows written in the precise style, (ii) the style had no significant impact on task completion time, (iii) more experienced participants benefited more, with respect to less experienced ones, from the precise style, as for their correctness of understanding, and (iv) all participants found the precise style useful in comprehending workflows.

Context: Software architectures should be evaluated during the early stages of software development in order to verify whether the non-functional requirements (NFRs) of the product can be fulfilled. This activity is even more crucial in software product line (SPL) development, since it is also necessary to identify whether the NFRs of a particular product can be achieved by exercising the variation mechanisms provided by the product line architecture or whether additional transformations are required. These issues have motivated us to propose QuaDAI, a method for the derivation, evaluation and improvement of software architectures in model-driven SPL development. Objective: We present in this paper the results of a family of four experiments carried out to empirically validate the evaluation and improvement strategy of QuaDAI. Method: The family of experiments was carried out by 92 participants: Computer Science Master s and undergraduate students from Spain and Italy. The goal was to compare the effectiveness, efficiency, perceived ease of use, perceived usefulness and intention to use with regard to participants using the evaluation and improvement strategy of QuaDAI as opposed to the Architecture Tradeoff Analysis Method (ATAM). Results: The main result was that the participants produced their best results when applying QuaDAI, signifying that the participants obtained architectures with better values for the NFRs faster, and that they found the method easier to use, more useful and more likely to be used. The results of the meta-analysis carried out to aggregate the results obtained in the individual experiments also confirmed these results. Conclusions: The results support the hypothesis that QuaDAI would achieve better results than ATAM in the experiments and that QuaDAI can be considered as a promising approach with which to perform architectural evaluations that occur after the product architecture derivation in model-driven SPL development processes when carried out by novice software evaluators., The authors would like to thank all the participants in the experiments for their selfless involvement in this research. This research is supported by the MULTIPLE Project (MICINN TIN2009-13838) and the ValI+D Program (ACIF/2011/235).

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition, must take every care to prevent bias that can result in incorrect results.