Search

The quality of a research study application sends a distinct signal to the institutional review board (IRB) about the skills, capacities, preparation, communication, experience, and resources of its authors. However, efforts to research and define IRB application quality have been insufficient. Inattention to the quality of an IRB application is consequential because the application precedes IRB review, and perceptions of quality between the two may be interrelated and interdependent. Without a clear understanding of quality, IRBs do not know how to define quality and researchers do not know how to achieve quality. This position has not been systematically studied to date, and future research could provide much-needed empirical validation. This paper lays the conceptual groundwork for future investigation into what constitutes quality in an IRB application. It includes a landscape review of multidisciplinary research on quality, as well as a discussion of quality frameworks analogous to research with human participants that exist in the published literature. It also examines the background and significance of federal research regulations, regulatory burdens, researchers' regulatory literacy, and the roles and responsibilities of IRB professionals within this ecosystem.

This article describes rates of missing item responses in personal digital assistant (PDA) assessments as compared to paper assessments. Data come from the evaluation of a classroom-based leisure, life skills, and sexuality education program delivered to high school students in Cape Town, South Africa. Analyses show that the paper assessments had much higher rates of missingness than PDA assessments. This association is moderated by item order. Certain analyses also suggest that paper assessments have higher rates of missingness for items pertaining to participants' sexual behavior. Implications of these results for evaluation research will be discussed. (Contains 5 tables, 1 figure, and 2 notes.)

Web-based surveys have become more prevalent in areas such as evaluation, research, and marketing research to name a few. The proliferation of these online surveys raises the question, how do response rates compare with traditional surveys and at what cost? This research explored response rates and costs for Web-based surveys, paper surveys, and mixed-mode surveys. The participants included evaluators from the American Evaluation Association (AEA). Results included that mixed-mode, while more expensive, had higher response rates. (Contains 3 tables and 1 figure.)

This article presents the author's address at the 2007 "Journal of Applied Quantitative Methods" ("JAQM") prize awarding festivity. The festivity was included in the opening of the 4th International Conference on Applied Statistics, November 22, 2008, Bucharest, Romania. In the address, the author reflects on three theses that question the gnoseological and operational efficiency of quantitative methods in the social domain. The first refers to symbolical analysis, where the connections between different qualitative categories are transposed into equation systems, including different types of inequalities or existence conditions. The second refers to empirical research. The third thesis upon which the author reflects refers to the skepticism concerning the predictive capacity of quantitative analysis. Here, the author cites several causal factors contributing to the recent global crisis in the context of these theories. (Contains 1 note.)

Objective: To assess differences between traditional paper bag ordering and online ordering from primary school canteens in terms of menu, usage and lunch order characteristics.Design: A cross-sectional study.Setting: New South Wales (NSW) primary schools that offered both paper bag and online canteen ordering.Participants: Students (aged 5-12 years) with a lunch order on the day of the observation.Results: Across the six school canteens, 59-90 % of all available items were listed on both the online and paper menus, with no significant differences in the nutritional quality ('Everyday'/'Occasional') or nutritional content (kJ/saturated fat/sugar/sodium) of menu items. In total, 387 student lunch orders were placed, containing 776 menu items. Most orders (68 %) were placed online. There were no significant differences between order modality in the quantity of items ordered or the cost of orders, or the nutritional quality of orders based on the classification system of the NSW Healthy School Canteen Strategy ('Everyday'/'Occasional'). However, nutritional analysis revealed that paper bag orders contained 222 fewer kJ than online orders (P = 0·001), 0·65 g less saturated fat (P = 0·04) and 4·7 g less sugar (P < 0·001).Conclusions: Online canteens are commonly used to order canteen lunches for primary school children. This is the first study to investigate differences between traditional paper bag ordering and online ordering in this setting. Given the rapid increase in the use of online ordering systems in schools and other food settings and their potential to deliver public health nutrition interventions, additional research is warranted to further investigate differences in ordering modalities. [ABSTRACT FROM AUTHOR]

The article is concerned with the research approaches and nature of methodologies and epistemologies used by practitioner-researchers in a range of occupational areas. The authors use the experiences of candidates on a generic Doctorate in Professional Studies who undertake a module on research and development for professional practice. They also bring their experience as tutors and programme leaders and the results from focus groups with their peers in other universities to this current and topical area of the higher education curriculum. Part of the module asks candidates to write an assignment in which they investigate the methodologies and epistemologies used as a practitioner-researcher. A close analysis of ten of these assignments reveals that the methodologies selected for practice-based projects may be as a result of the candidates' personal, professional and disciplinary backgrounds as well as a time imperative and desire for brevity that arises from an immediacy in the situation. The authors also conclude that a range of factors were important to the practitioner-researchers and that these have important epistemological implications. The social-science-oriented approaches introduced on the programme offer tried and tested ways to research, but more practice-based approaches to practitioner-research could be developed further. (Contains 2 figures.)

With the push for evidence-informed policy and practice, schools and policy makers are now increasingly encouraged and supported to use and enagage with research evidence. This means that consumers of research will now need to be discerning in judging the quality of research evidence that will inform their decisions. This paper evaluates the quality of evidence behind some well-known education programmes using examples from previous reviews of over 5,000 studies on a range of topics. It shows that much of the evidence is weak, and fundamental flaws in research are not uncommon. This is a serious problem if teaching practices and important policy decisions are made based on such flawed evidence. Lives may be damaged and opportunities missed. The aim of this paper is to show how widespread this problem is and to suggest ways by which the quality of education research may be improved. For example, funders of research and research bodies need to insist on quality research and fund only those that meet the minimum quality criteria. Journal editors and reviewers need to be cognizant of fundamental flaws in research and reject such submissions. One way to do this is to encourage submission of the research design and research protocol prior to acceptance, so acceptance or rejection is based on the design and not on the outcomes. This helps prevent publication bias and biased reporting. Individual researchers can improve quality by making it their moral responsibility to be truthful and transparent.

Experimental methods have played a significant role in the growth of English teaching and learning studies. The paper presented here outlines basic features of experimental design, including the manipulation of independent variables, the role and practicality of randomised controlled trials (RCTs) in educational research, and alternative methods and techniques available in the absence of RCTs. It further reviews validity issues inherent in conducting experimental research, in particular sources of internal and external invalidity, and how to remedy them. Along the way, the author suggests that researchers remain mindful of these threats, and calls for the replication of studies across different research contexts with the purposes of the cross-validation and generalisation of findings. The remainder of the paper concludes with suggestions on how to develop a more embedded and sophisticated experimental design in light of the current literature of mixed methodology (Creswell & Plano Clark, 2011), in order to have more explanatory power and compensate for the weaknesses associated with the experimental method. Throughout the paper, the author illustrates the points with examples relevant to English teaching and learning research. (Contains 1 figure and 1 table.)

This paper was designed to explore the means in the field of observing classroom teaching/learning from both a general, and an "English as a Foreign Language" (EFL), viewpoint. The aim was to browse the relevant literature, and lead to consider observation tools which might serve to research in EFL. This paper summarizes the reading by surveying the field of classroom observation, and then proceeds to evaluate the likely usefulness of a number of selected observation instruments.

This paper reviews a series of formative assessment studies that were conducted to inform and evaluate a large-scale instructional software development project at the University of Missouri-Rolla (UMR). The three-year project, entitled "Taking the Next Step in Engineering Education: Integrating Educational Software and Active Learning," was funded by the U.S. Department of Education Fund for the Improvement of Post Secondary Education (FIPSE). The assessment was carried out under the auspices of UMR's Laboratory for Information Technology Evaluation (LITE) and guided by the LITE model for evaluation of learning technologies. The fundamental premise of the model is that evaluation should consist of the triangulation of multiple research methodologies and measurement tools. Five representative evaluation studies, consisting of eight experiments, are presented here. The studies range from basic experimentation and usability testing to applied research conducted within the classroom as well as a multi-national cross-cultural applied dissemination survey conducted during the last semester of the project. This paper demonstrates that the LITE model can be an effective tool for guiding a comprehensive evaluation program. In addition, the research findings provide evidence that the instructional multimedia developed in this project can have a substantial positive impact in enhancing fundamental engineering classes. (Contains 4 endnotes, 6 tables, and 8 figures.)

Many practitioner researchers strive to understand which assessment practices have the best impact on learning, but in authentic educational settings, it can be difficult to determine whether one intervention, for example the introduction of an online quiz to a course studied by diverse students, is responsible for the observed effect. This paper uses examples to highlight some of the difficulties inherent in assessment research and suggests some ways to overcome them. Problems observed in the literature include: assuming that if two effects are correlated then one must have caused the other; confounding variables obscuring the true relationships; experimental approaches that are too far removed from reality; and the danger that self-reported behaviour and opinion is sometimes different from student's actual behaviour. Practical solutions include: the use of an experimental or pseudo-experimental approach; the use of mixed methods; and the use of meta-analysis.

The author previously published a paper discussing how to conduct an analysis based on a cluster sample. In that paper, the author outlined several widely adopted alternative approaches, and pointed out that such approaches are anyway not needed for population figures, and not possible for non-probability samples. Thus, the author queried the prevalence and relevance of complex approaches such as multi-level modelling (MLM) for most real-life analyses. Hutchison and Schagen (2008) responded with a paper that misrepresents what the author said, in an attempt to defend their favoured technique of MLM. In this article, the author wants to establish their misunderstanding of this important topic. This paper deals first with areas in which they agree, and then areas in which Hutchison and Schagen (2008) simply restate their position, before turning to the more substantive consideration of why it is not necessary or appropriate to use MLM in most real analytical situations. (Contains 2 notes.)

Carol Weiss did much to enhance the role of evaluation in her writings. Her work shows evaluators what affects their roles as they evaluate programs. Furthermore, her theory of change spells out the complexities involved in program evaluation. There are various processes involved in the evaluation of programs. The paper looks at some of the aspects that inform this process. Effective evaluation of programs can lead to improvement of programs and good delivery of projects in future. It is Weiss who stated that evaluation should have an influence on policy and practice. Amongst other themes explored here are the politics of evaluation as well as the crucial role of the evaluator within the complexity of evaluation of programs.

Background: Previous research has found that a "web-push" approach to data collection, which involves contacting people by mail to request an Internet survey response while withholding a paper response option until later in the contact process, consistently achieves lower response rates than a "paper-only" approach, whereby all respondents are contacted and requested to respond by mail.Method: An experiment was designed, as part of the Millennium Cohort Family Study, to compare response rates, sample representativeness, and cost between a web-push and a paper-only approach; each approach comprised 3 stages of mail contacts. The invited sample (n = 4,935) consisted of spouses married to U.S. Service members, who had been serving in the military between 2 and 5 years as of October, 2011.Results: The web-push methodology produced a significantly higher response rate, 32.8% compared to 27.8%. Each of the 3 stages of postal contact significantly contributed to response for both treatments with 87.1% of the web-push responses received over the Internet. The per-respondent cost of the paper-only treatment was almost 40% higher than the web-push treatment group. Analyses revealed no meaningfully significant differences between treatment groups in representation.Conclusion: These results provide evidence that a web-push methodology is more effective and less expensive than a paper-only approach among young military spouses, perhaps due to their heavy reliance on the internet, and we suggest that this approach may be more effective with the general population as they become more uniformly internet savvy. [ABSTRACT FROM AUTHOR]

As is typically the case when this author sets fingers to keyboard and writes about evaluation, his intent is to offer his work and perspective as "one" African American evaluator. The impetus for the thoughts and ideas he expresses in this article were prompted by his preparation to attend and present a paper at the inaugural annual conference of the African Evaluation Association (AfrEA). As an African American evaluator, his participation was first and foremost personally significant. It was equally a historically significant moment in global evaluation history. He took this opportunity to write a paper that reflected on evaluation approaches he personally believed had a major influence on the evaluation of schools and other educational institutions in the United States. Since the 1999 AfrEA conference, conversations in the evaluation literature and the American Evaluation Association about the role of culture and cultural context have resulted in additional personal reflections that are shared in this article. The author first takes an historical look at selected program evaluation approaches and thoughts by a select group of evaluation theorists (in the case of some icons in the field) who have influenced his views as well as evaluative practices. He then considers, through the scholarly work of these individuals, the evaluation approaches they have articulated, and related discussions. He asserts that program evaluation approaches can and should be more culturally responsive if one is to fully understand the effectiveness, benefits, and outcomes of programs designed to help less powerful stakeholders. He goes on to discuss a few encouraging examples of emerging thinking about program evaluation approaches and standards offered by the evaluation community that may be conducive to achieving such objectives for this new millennium. One realistic remedy to advance the discourse and practice of educational evaluation is to increase the number of trained evaluatorsfrom racial and cultural groups that have typically been disenfranchised within the social fabric of the United States.

The case study method has not been as frequently used in international business (IB) research as quantitative methods. Moreover, it has been sometimes misused and quite often criticized. Still, it can be very useful for understanding such complex phenomena as the internationalization process or the management of multinational enterprises. Based on case study methodology literature and the author's personal experience from conducting case studies and reviewing case study articles, this paper proposes some ways for overcoming some of the critiques of the case study method and increasing its contribution to IB research. (Contains 1 table and 1 figure.)

The recent "Nationwide academic achievement and study situation survey" was clearly influenced by the idea of "authentic assessment", an educational assessment perspective focused on "quality" and "engagement". However, when "performance assessment", the assessment method corresponding to this focus, is adopted in academic achievement surveys, it runs the risk of turning into a rigid hollow structure. In this paper I will reflect on the ideal application of performance assessment in academic achievement surveys, and will investigate the concepts of "consequential validity", "equity", and "moderation" in regard to their potential to further develop the discourse. (Contains 1 note.)

In this paper, we introduce various graphical methods that can be used to represent data in mixed research. First, we present a broad taxonomy of visual representation. Next, we use this taxonomy to provide an overview of visual techniques for quantitative data display and qualitative data display. Then, we propose what we call "crossover" visual extensions to summarize and integrate both qualitative and quantitative results within the same framework. We provide several examples of crossover (mixed research) graphical displays that illustrate this natural extension. In so doing, we contend that the use of crossover (mixed research) graphical displays enhances researchers' understanding (i.e., increased Verstehen) of social and behavioral phenomena in general and the meaning that underlies these phenomena in particular. (Contains 2 tables and 5 figures.)

It is now a matter of routine that schools in England are able to demonstrate the value of their work in terms of "impact" and "outcomes." In the province of imaginative education this is problematic. While Government has sought to create a new relationship between inspection and school self-evaluation, this in effect has amounted to little more than a bureaucratic and performative form of "self-inspection." At the same time the teaching profession is reminded that it lacks a shared language to enable clarity and precision about its judgements (Hargreaves, 2004). Acknowledging the necessity for imaginative educators to make their work publicly demonstrable, and recognising the private imaginative lifeworld as a sacred space, this paper calls for a (re)focusing of educational evaluation in imaginative education. Drawing on phenomenological research approaches and ideas of connoisseurship and pupil voice, six "situated" imaginative practices, spanning the solitary and the collective, are proposed in an attempt to consider ways in which the imagination might be made amenable to communal educational evaluation. Before the development of a shared evaluative language can be entertained, the necessary conditions for educational evaluation must first be created, and these conditions involve educators in the cultivation of their own imaginative lifeworlds as a professional practice. Ultimately, through processes of interpretation and communalisation, educational evaluation of the imagination becomes an intrinsically transformative practice. (Contains 3 footnotes.)

The assessment of children in their years before school and their first years of school has been, traditionally, informal. Further, assessment of children's mathematical skills at this level has been infrequent compared to social, emotional and physical assessments. However, there are contexts where reliable, valid, standardised data from assessment in mathematics are required. This paper outlines the development of two assessment tools for mathematics that were originally developed for such contexts. Item Response Theory (IRT) analyses enabled the construction of assessment forms that address the range of abilities of 4- to 8-year-old children, and provided the scales used for constructing formative and summative reports of achievement. A description of the development of the assessment tools and the IRT analysis that provides the reporting formats are presented together with some research uses of the tools. (Contains 7 figures and 2 tables.)

Career and technical education research often centers around quantitative research designs. The Delphi Technique provides a structured communication process designed to produce a detailed examination of a topic and/or problem and discussion from the participating group. The contributions of individuals via this tool produce a group perspective not otherwise attainable. This paper provides the foundations of the Delphi Technique, discusses its strengths and weaknesses, explains the use and stages followed, discusses panel selection, and explains how consensus among participants is reached. Examples of how the Delphi Technique has been used in career and technical education are also included.

Metacognition has been accorded a role in both mathematical problem solving and in the learning of mathematics. There has been consistent advocacy of the need for the promotion of metacognitive activity in both domains. Such advocacy can only be effective if the advocated process is well understood. In this paper we have four goals: to describe a "multi-method" technique developed to study student mathematical metacognition; to set out the structural elements and configuration of a coherent model of metacognition in the domain of mathematical problem solving; to report on the empirical utility (and validity) of this model; and, to report the insights into student mathematical metacognition arising from the research. Multi-Method Interview Tasks are appended. (Contains 4 figures.)

In this paper, the authors depict performance ethnography as having multiple ethical dimensions and multiple implications for the relationships between the evaluator and program participants. They have found performance ethnography to be essentially an intimate form of representation that has tremendous implications for the ownership of qualitative data, particularly data associated with program evaluation. They also address implications for qualitative researchers who wish to use performance as part of evaluation studies. They speculate on how to create a performance that not only gives justice to the multiple voices in qualitative data but that also invites dialogue that is meaningful, non-threatening, and ethically defensible. A dialogic approach to program evaluation research can lead to a more ethical and equitable relationship between evaluators and program participants, which in turn can result in richer understandings of program implementation. It also allows participants to claim ownership of the evaluation and can provide a vehicle for disseminating results. (Contains 3 notes.)

The purpose of the "Evidence to Practice" commentary is to highlight an issue or research finding in selected articles, emphasizing the relevance to practice. This article introduces a new tool designed to summarize the most up-to-date evidence in an easily readable format, in order to provide clinically useful answers within minutes for assisting decision-making and determining next steps (Straus & Haynes, 2009). The tool is the Evidence Alert Traffic Light Grading System (Novak & McIntyre, 2010). The article also focuses on the two papers that evaluated the effectiveness of an intervention: Therapeutic effects of horseback riding therapy (Whalen & Case-Smith, 2011) and Neoprene hand splints effects on handwriting (Frohlich, Wesley, Wallen, & Bundy, 2011). It begins with a thumbnail sketch of the strength of evidence published (McCandless, 2010) and then provides a summary of each intervention for evidence-based decision-making.

Good evaluation requires that evaluation efforts themselves be evaluated. Many things can and often do go wrong in evaluation work. Accordingly, it is necessary to check evaluations for problems such as bias, technical error, administrative difficulties, and misuse. Such checks are needed both to improve ongoing evaluation activities and to assess the merits of completed evaluation efforts. The aim of this paper is to present both a logical structure and methodological suggestions for evaluating evaluation. This paper is based on work performed in the Ohio State University Evaluation Center between 1963 and 1973. That work provided many occasions for addressing meta-evaluation issues, including developing evaluation systems and assessing the work of such systems, designing and conducting evaluation studies, training graduate students and practitioners to conduct evaluation work, and critiquing many evaluation designs and reports. These experiences and the attendant problems are the basis for this paper. Part I of this paper discusses the need to develop a technology for evaluating evaluation; describes eleven meta-evaluation criteria; and delineates six classes of problems that plague evaluation efforts. Part II presents a definition, eight premises, and a logical structure for meta-evaluation work. Part III describes how the structure might be used through describing and illustrating five meta-evaluation designs. (Contains 8 figures.)

The process of constructing assessment scales for performance testing is complex and multi-dimensional. As a result, a number of different approaches, both empirically and intuitively based, are open to developers. In this paper we outline the approach taken in the revision of a set of assessment scales used with speaking tests, and present the value of combining methodologies to inform and refine scale development. We set the process in the context of the growing influence of the Common European Framework of Reference (Council of Europe 2001) and outline a number of stages in terms of the procedures followed and outcomes produced. The findings describe a range of data that was collected and analysed through a number of phases and used to inform the revision of the scales, including consultation with experts, and data-driven qualitative and quantitative research studies. The overall aim of the paper is to illustrate the importance of combining intuitive and data-driven scale construction methodologies, and to suggest a usable scale construction model for application or adaptation in a variety of contexts. (Contains 6 tables and 2 figures.)

This paper presents the findings from an evaluation research project undertaken by Henley Management College in 2006. This project followed an earlier research study that focused on identifying the leadership development needs for leaders of small and medium sized-enterprises (SMEs) in the UK, developed a leadership development model and made recommendations for a leadership development programme. The principle challenges facing the SME organisations overwhelmingly concerned leading change in their organisation. Their personal challenges concerned implementing change initiatives, communicating these and motivating the people in their organisation. In consideration of the challenges identified, one of the key recommendations was to pilot an action learning programme and evaluate the effectiveness of action learning to develop leadership skills in SME leaders. Henley Management College then conducted a research project to evaluate the pilot programme. This pilot consisted of three learning sets, run in three geographical areas in the UK and facilitated using three very different styles. Following the pilot programme, there were several wider reviews by academics, advisors and representatives of relevant government boards. This has since resulted in a significant programme of action learning for SME leadership development being implemented. This paper provides an overview of the relevant literature and research methodology and then presents the evaluation results from the pilot programme. The dominant theme was that action learning does indeed deliver significant business benefits and provides a real opportunity for SME leaders to develop their leadership skills. (Contains 6 tables and 3 figures.)

In this article, the authors clarify and provide additional tests of the key elements of their age-period-cohort analysis of non-marital birth rates in this March 2009 issue of "Social Forces." Where Steve Martin, in his commentary, has suggested specific alternative specifications or interpretations of their findings, the authors have re-estimated and performed explicit tests of the alternatives. Martin suggests two explanations for the cohort patterns found in the authors' data: (1) different trends in the relationships of age-groups to NFRs; and (2) differences in the proportion unmarried for age-groups in different periods. The authors enter variables representing both of these postulated relationships into their models and find that they do not significantly alter the coefficients associated with the cohort characteristics that they reported in their original paper. Thus, at least taking these possible explanations into consideration, the authors find that the cohort effects remain strong in their data. Martin claims that the coefficients for the cohort characteristics that the authors report are improbably large--the authors find no evidence of that in their reading of the results. Martin finds the pattern of residuals within cohorts to be "weird" given the authors' model. In fact, the pattern of the residuals is supportive of cohort effects. Finally, the authors find cohort effects for age-period-specific NFRs when they analyze only cohorts with complete data, examining cohorts born in the years 1933-1937 and 1958-1962. According to Martin, such an analysis should not be subject to the biases that form the core of his critique. The authors have demonstrated the robustness of their results and strengthened the case for their interpretations. The points Martin raises and the authors' responses should help to clarify the nature of cohort effects and the authors' interpretation of them. (Contains 1 table, 1 figure and 4 notes.)

This paper reports on the development, self-critique and evolution of research methods for interpreting and understanding students' metacognition that were developed through the Metacognition and Reflective Inquiry (MRI) collaborative study. The MRI collaborative was a multi-year, multi-case, research study that investigated the elusive nature and character of high school students' metacognition across formal and informal science learning contexts. The study's research design comprised a series of integrated, layered, interpretive case studies which were conducted in a hermeneutic fashion over a 3 year period. The implementation of each case study provided an opportunity for the researchers to reflect critically on the research methods used to elucidate metacognition and hence refine the individual and collective capacity, responsiveness and fruitfulness of the methods used. This paper discusses the evolution of these methods and the lessons that the entire study provides for the conceptualization of other qualitative-interpretivist studies.

This paper examines the practice of metaevaluation as indicated by the Metaevaluation standard of the Program Evaluation Standards, as the evaluation of a specific evaluation to inform stakeholders about the evaluation's strengths and weaknesses. The findings from an analysis of eighteen metaevaluations, including a description of the data sources and methods used to come to conclusions about the evaluation and the criteria of quality employed, are reported. A diverse set of practices were identified, ranging from the use of emergent criteria in a narrative review of information about an evaluation to the structured application of the Program Evaluation Standards using a checklist. The paper concludes that the evaluation field does not have a common understanding of metaevaluation practice. (Contains 2 tables and 3 footnotes.)

Background: When a journal receives a duplicate publication, the ability to identify the submitted work as previously published, and reject it, is an assay to publication ethics best practices. The aim of this study was to evaluate how three different types of journals, namely open access (OA) journals, subscription-based journals, and presumed predatory journals, responded to receiving a previously published manuscript for review.Methods: We performed a quasi-experimental study in which we submitted a previously published article to a random sample of 602 biomedical journals, roughly 200 journals from each journal type sampled: OA journals, subscription-based journals, and presumed predatory journals. Three hundred and three journals received a Word version in manuscript format, while 299 journals received the formatted publisher's PDF version of the published article. We then recorded responses to the submission received after approximately 1 month. Responses were reviewed, extracted, and coded in duplicate. Our primary outcome was the rate of rejection of the two types of submitted articles (PDF vs Word) within our three journal types.Results: We received correspondence back from 308 (51.1%) journals within our study timeline (32 days); (N = 46 predatory journals, N = 127 OA journals, N = 135 subscription-based journals). Of the journals that responded, 153 received the Word version of the paper, while 155 received the PDF version. Four journals (1.3%) accepted our paper, 291 (94.5%) journals rejected the paper, and 13 (4.2%) requested a revision. A chi-square test looking at journal type, and submission type, was significant (χ2 (4) = 23.50, p < 0.001). All four responses to accept our article came from presumed predatory journals, 3 of which received the Word format and 1 that received the PDF format. Less than half of journals that rejected our submissions did so because they identified ethical issues such as plagiarism with the manuscript (133 (45.7%)).Conclusion: Few journals accepted our submitted paper. However, our findings suggest that all three types of journals may not have adequate safeguards in place to recognize and act on plagiarism or duplicate submissions. [ABSTRACT FROM AUTHOR]

This paper is based on the author's experiences of interviewing seven men and seven women who graduated from a Danish high school in 1980, at a moment when the upper secondary system underwent a process of democratisation. She questioned the informants as to the role education has played in their lives, and as to the sense their choice of education makes to them. In addition to providing extensive narrative data about the experience of going through the educational system in the 1970s and 1980s, her interview process also raised two fundamental methodological issues. The first regards the relationship between the narrative and its historical context. The second relates to how narrated truths can be understood more as split and divided between the interviewee and the interviewer than as negotiated, as is frequently argued. In the paper it is suggested that these narratives act more as a testing ground for the development of interpretations and reflections about the early years of the so-called "educational explosion" that has taken place in Denmark from the 1960s until today than as ultimate truths. (Contains 40 footnotes.)

With the advent of the UN Literacy Decade launched in 2003, there is increased emphasis on monitoring and evaluation of adult literacy around the globe. The present paper presents an overview of the main approaches that have been taken to measure adult literacy within and across countries. A particular focus of the present review is to compare and contrast three models of literacy assessment: (1) the "traditional" model based largely on census enumeration, which has been used over recent decades to collect national and regional rates of illiteracy; (2) the large-scale survey techniques employed with the International Adult Literacy Survey and similar models; and (3) an intermediate type of assessment that borrows from the previous two models, and attempts to "tailor" the size and complexity of the survey methodology to the policy questions needing answers (called the SQC model). The present paper suggests that there is no single model or set of methods that are universally appropriate to monitoring and measuring in adult literacy around the world, but that blending both programmatic and comparative models through the SQC approach may bring greater involvement in, and insight into, adult literacy evaluations.

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers, respectively. Data collected has been analysed to investigate whether reliable standards can be achieved for small-scale, medical school-based assessments, to establish the minimum number of judges required and the effect of a discussion phase on reliability. The root mean squared error (RMSE) has been used as a measure of reliability and used to compute 95% confidence intervals for comparison to the examination statistics. The RMSE has been used to calculate the minimum number of judges required to obtain a predetermined minimum level of reliability, and the effect of the number of judges and number of items have been examined. Values of the RMSE obtained vary from 0.9 to 2.2%. Using average variances across each paper type, the minimum number of judges to obtain a RMSE of less than 2% is 10 or more judges before discussion or 6 or more judges after discussion. The results indicate that including a discussion phase improves the reliability and reduces the minimum number of judges required. Decision studies indicate that increasing the number of questions included in the assessments would not significantly improve the reliability of the standard setting.

Determining the potential of educational information and communications technology for enhancing teaching and learning is the subject of a great deal of research activity. This paper argues that the technologies employed to enhance learning can also be exploited for the enhancement of research. Recorded learner-system interactions, dynamically captured computer screen activity and visual attention tracking all provide sources of rich process data that can be triangulated with tradition learning outcome (product) data. Such multiple data source approaches can be used in various research design and methodological contexts--design experiments, microgenetic experiments, case studies and so on. Process-analytic methods can reveal the fine details of an individual's learning trajectory and permit the effects of short-timescale interventions upon longer term learning outcomes to be assessed. In this paper, three illustrations of technology-enhanced research (TER) are described. The first example ("switchER") illustrates an iterative learner-centered design cycle. The second example (Hyperproof) illustrates how process data can elucidate individual differences in learning. The third example (PATSy, VL-PATSy) demonstrates how an established online e-learning system can be used as a tool for identifying students' clinical reasoning difficulties and how it can then be extended by having additional learning support resources added to it together with a context-sensitive, rule-based delivery system. The paper concludes with a discussion of broader TER issues such as the need to anticipate and plan for the analysis of large amounts of rich data. It concludes by suggesting some further ways in which TER might be exploited and provides links to some potentially useful TER tools and resources. (Contains 8 figures and 18 notes.)

This paper describes a comprehensive model of alignment that provides a foundation for meaningful reporting of students' academic progress over time. The model includes both horizontal and vertical alignment as integral parts of the development of content standards, test blueprints, items, item pools, instruments, performance level descriptors, and performance standards. It also explains why comprehensively integrating alignment into development processes results in procedural efficiencies and gains in validity evidence for the measurement of student progress. Cautions against reducing focus on content in favor of alignment are given. All parts of the comprehensive alignment model described in this paper either have been or soon will be implemented in a large-scale assessment program.

Purpose: This study compared a tablet PC questionnaire with a paper method for reliability and patient preferences in the acquisition of patient-reported outcomes (PROs) for patients treated with radiotherapy. By comparing the two modes of PRO administration, we aimed to evaluate the adequacy of using tablet PC questionnaires in future clinical use.Methods: Patients were randomized in a crossover study design using two different methods for PRO entry. A group of 89 patients answered a paper questionnaire followed by the tablet PC version, whereas 89 patients in another group completed the tablet PC questionnaire followed by the paper version. Surveys were performed four times per patient throughout the course of the radiotherapy. The Korean versions of the M.D. Anderson Symptom Inventory (MDASI-K) and the Brief Fatigue Inventory (BFI-K) were used. The primary endpoint of our current study was an assessment of patient preference for the survey method. The proportions of patients preferring each mode of questionnaire were evaluated.Results: The proportion of patients who preferred the tablet PC version, paper form, or who had no preference was 52.2, 22.1, and 25.7 %, respectively. More than half of the patients preferred the tablet PC to the paper version in all four surveys. Age, gender, educational status, prior experience of using a tablet PC, and the order of paper to tablet PC administration did not impact patient preferences. Inter-class correlation coefficients (ICCs) between the modes were 0.92 for MDASI-K and 0.94 for BFI-K and ranged from 0.91 to 0.96 on both instruments during the four surveys.Conclusions: A tablet PC-based PRO is an acceptable and reliable method compared with paper-based data collection for Korean patients receiving radiotherapy. [ABSTRACT FROM AUTHOR]

The aim of this paper is to examine the role of methodology in action research. It begins by showing how, as a form of inquiry concerned with the development of practice, action research is nothing other than a modern 20th century manifestation of the pre-modern tradition of practical philosophy. It then draws in Gadamer's powerful vindication of the contemporary relevance of practical philosophy in order to show how, by embracing the idea of "methodology", action research functions to sustain a distorted understanding of what practice is. The paper concludes by outlining a non-methodological view of action research whose chief task is to promote the kind of historical self-consciousness that the development of practice presupposes and requires.

This paper considers some of the methodological implications of undertaking and representing multimedia and digital, ethnographic work. It explores some of the challenges and opportunities of working with and across a range of media, and explores some of the consequences of bringing hypermedia applications to ethnographic work. The paper draws on a project that sought to explore the learning opportunities and experiences of an interactive science discovery centre, and that utilised a multimedia, ethnographic approach. The paper locates this methodological approach alongside recent and ongoing developments within qualitative research, and aims to situate educational research within the context of these developments. The paper particularly addresses issues of integrating different forms of qualitative data, ethnographic "design", the ethics of research practice, communication and capacity building. (Contains 1 figure.)

Background: Those engaged in research might argue that we are involved in a process of explaining the world to others and to ourselves. The way in which we go about that process differs. The kind of questions we are concerned about answering will determine the approach we use, the information we gather and how we analyse the data. Some research questions are best answered using qualitative methods. Aims: To explain the place of qualitative research methods within the context of different approaches to scientific enquiry. To describe qualitative research through an examination of the characteristics that differentiate it from quantitative approaches, to describe the types of strategies used, and the roles the researcher may adopt. To show how these methods have been applied to the study of stuttering. Methods & Procedures: Stuttering research was reviewed with the aim of finding studies using a range of qualitative methods. Work for discussion was selected based on historical importance, the methods used and the contribution the work has made to our understanding of stuttering and qualitative methods. Main Contribution: Papers using explorative qualitative research and mixed methodologies are described and their various contributions to our understanding of stuttering are discussed. Conclusions: The qualitative research reviewed in this paper has added to our understanding of the experiences, thinking and views of people who stutter. The reports on people's experiences of therapy show that we still have much to learn and that we need to listen to our clients and involve them in therapy and in outcome planning. (Contains 1 table and 2 notes.)

In recent years, constructionist methodologies such as discursive psychology (Edwards & Potter, 1992) have begun to be used in sport research. This paper provides a practical guide to applying a discursive psychological approach to sport data. It discusses the assumptions and principles of discursive psychology and outlines the stages of a discursive study from choice of data through to transcription and analysis. Finally, the paper demonstrates a discursive psychological analysis on sport data where athletes are accounting for success and failure in competition. The analysis demonstrates that for both success and failure, there is an apparent dilution of personal agency, to either maintain their modesty in the case of success or to manage blame when talking about failure. It is concluded that discursive psychology has much to offer sport research as it provides a methodology for in-depth studies of supporting interactions.

Background: The essence of humanism in medicine and health care is relationships-caring relationships between clinicians and patients. While raising concerns regarding professional-patient boundaries has positively contributed to our understanding and prevention of potentially harmful boundary violations, there is controversy about which types of relationships, caring acts, and practices are acceptable versus cross boundary lines.Objective: To examine primary care physicians' practices and attitudes regarding acts that have been questioned as potentially "inappropriate" or "unethical" crossing of professional-patient boundaries.Design: Surveys conducted via in-person polling or electronic and mailed paper submissions from April 2016 to July 2017. We calculated descriptive statistics and examined associations with practices and attitudes using logistic regression.Participants: Random sample of all US primary care physicians who treat adult patients; convenience sample of attendees at medicine grand rounds presentations.Main Measures: Outcomes were self-reported practices and attitudes related to giving patients rides home, paying for patients' medication, helping patients find jobs, employing patients, going to dinner with patients, and providing care to personal friends.Key Results: Among 1563 total respondents, 34% had given a ride home, 34% had paid for medications, 15% helped patients find a job, 7% had employed a patient, 10% had dinner with patients, and 59% provided care to personal friends. A majority disapproved of dinner with a patient (75%) but approved of or were neutral on all other scenarios (61-90%).Conclusions: The medical profession is quite divided on questions related to drawing lines about appropriate boundaries. Contrary to official and widespread proscriptions against such practices (with exception of dinner dates), many have actually engaged in such practices and the majority found them acceptable. [ABSTRACT FROM AUTHOR]

Purpose: In the higher education sector, the evaluation of learning and teaching projects is assuming a role as a quality and accountability indicator. The purpose of this paper is to investigate how learning and teaching project evaluation is approached and critiques alignment between evaluation theory and practice. Design/Methodology/Approach: The emergent realism paradigm provides the theoretical framework with a pragmatic approach to mixed-methods data collection. Thematic analysis was used to analyze the transcripts of interviews with 15 project leaders. Findings: Four key themes on project evaluation emerged: how evaluation is conceptualized, particularly the overlap, even conflation, between evaluation and research; capability building within the sector; resourcing in terms of time and money; and the role of an action-oriented approach to evaluation. The authors conclude that misalignment exists between evaluation theory and the practice of project evaluation and that this relationship can be further inhibited by a project leader's perception of evaluation. Practical Implications: A series of strategies for developing capacity across the higher education sector for project evaluation are presented. These include the development and provision of: a time allocation for evaluation in future and ongoing project plans with procedures to revisit the project and assess impact; models of how to incorporate evaluation into the research cycle; constructive feedback on evaluation reports from the university funding body; and networking opportunities to disseminate learnings from project evaluations. Originality/Value: This study focusses on the under-researched area of evaluation of learning and teaching projects in higher education, providing research-based evidence for strategies to develop sector capacity.

Cysticercosis is an infestation of Cysticercus cellulosae. When it occurs in the brain, chronic neurological complications can ensue, most commonly seizures. Neurocysticercosis is usually diagnosed by neuroimaging, a technique not available in most endemic countries. Hence immunological tests are valuable for diagnosis and epidemiological surveys. We evaluated the suitability of paper for storing blood and cerebrospinal fluid (CSF) until subsequent testing by enzyme-linked immunosorbent assay (ELISA), by testing whole blood samples on filter paper from 305 patients and CSF samples from 117 patients stored on ordinary white typing paper and on filter paper. Optimal preservation of biological samples is achieved when whole blood is stored on filter paper, CSF on white paper, and when samples are frozen within 1 week after collection. Our results could improve diagnostic capabilities and facilitate epidemiological surveys in endemic countries where immunodiagnostic tests cannot be rapidly performed because of inadequate laboratory infrastructure. [ABSTRACT FROM AUTHOR]

Examines fundamental conceptions regarding the role and nature of evaluating and researching music education programs. Questions the common idea that evaluative criteria for programs should focus primarily on the students' level of musical skill. Argues that consideration should be given to personal growth and other issues. (MJP)