Search

Without careful dissection of the ways in which biases can be encoded into artificial intelligence (AI) health technologies, there is a risk of perpetuating existing health inequalities at scale. One major source of bias is the data that underpins such technologies. The STANDING Together recommendations aim to encourage transparency regarding limitations of health datasets and proactive evaluation of their effect across population groups. Draft recommendation items were informed by a systematic review and stakeholder survey. The recommendations were developed using a Delphi approach, supplemented by a public consultation and international interview study. Overall, more than 350 representatives from 58 countries provided input into this initiative. 194 Delphi participants from 25 countries voted and provided comments on 32 candidate items across three electronic survey rounds and one in-person consensus meeting. The 29 STANDING Together consensus recommendations are presented here in two parts. Recommendations for Documentation of Health Datasets provide guidance for dataset curators to enable transparency around data composition and limitations. Recommendations for Use of Health Datasets aim to enable identification and mitigation of algorithmic biases that might exacerbate health inequalities. These recommendations are intended to prompt proactive inquiry rather than acting as a checklist. We hope to raise awareness that no dataset is free of limitations, so transparent communication of data limitations should be perceived as valuable, and absence of this information as a limitation. We hope that adoption of the STANDING Together recommendations by stakeholders across the AI health technology lifecycle will enable everyone in society to benefit from technologies which are safe and effective. [ABSTRACT FROM AUTHOR]

Amanda L Graham,1,2 Kelly M Carpenter,3 Sarah Cha,1 Sam Cole,3 Megan A Jacobs,1 Margaret Raskob,3 Heather Cole-Lewis,4,51Schroeder Institute for Tobacco Research and Policy Studies, Truth Initiative, Washington, DC, 2Department of Oncology, Georgetown University Medical Center/Cancer Prevention and Control Program, Lombardi Comprehensive Cancer Center, Washington, DC, 3Alere Wellbeing, Seattle, WA, 4Johnson & Johnson Health and Wellness Solutions, Inc., New Brunswick, NJ, 5ICF International, Rockville, MD, USA Background: The aim of this systematic review was to determine the effectiveness of Internet interventions in promoting smoking cessation among adult tobacco users relative to other forms of intervention recommended in treatment guidelines. Methods: This review followed Cochrane Collaboration guidelines for systematic reviews. Combinations of “Internet,” “web-based,”and “smoking cessation intervention” and related keywords were used in both automated and manual searches. We included randomized trials published from January 1990 through to April 2015. A modified version of the Cochrane risk of bias assessment tool was used. We calculated risk ratios (RRs) for each study. Meta-analysis was conducted using random-effects method to pool RRs. Presentation of results follows the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Results: Forty randomized trials involving 98,530 participants were included. Most trials had a low risk of bias in most domains. Pooled results comparing Internet interventions to assessment-only/waitlist control were significant (RR 1.60, 95% confidence interval [CI] 1.15–2.21, I2=51.7%; four studies). Pooled results of largely static Internet interventions compared to print materials were not significant (RR 0.83, 95% CI 0.63–1.10, I2=0%; two studies), whereas comparisons of interactive Internet interventions to print materials were significant (RR 2.10, 95% CI 1.25–3.52, I2=41.6%; two studies). No significant effects were observed in pooled results of Internet interventions compared to face-to-face counseling (RR 1.35, 95% CI 0.97–1.87, I2=0%; four studies) or to telephone counseling (RR 0.95, 95% CI 0.79–1.13, I2=0%; two studies). The majority of trials compared different Internet interventions; pooled results from 15 such trials (24 comparisons) found a significant effect in favor of experimental Internet interventions (RR 1.16, 95% CI 1.03–1.31, I2=76.7%). Conclusion: Internet interventions are superior to other broad reach cessation interventions (ie, print materials), equivalent to other currently recommended treatment modes (telephone and in-person counseling), and they have an important role to play in the arsenal of tobacco-dependence treatments. Keywords: systematic review, meta-analysis, Internet, smoking cessation, tobacco control

Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications., Competing Interests: Competing interests: This study was funded by Alphabet Inc and/or a subsidiary thereof (‘Alphabet’). K.S., T.T., S.S.M., J.G., R.S., E.W., M.A., L.H., K.C., S.R.P., H.C.-L., D.N., Q.M.R., M.S., A.W., S.L., P.A.M., S.P., B.G., E.D., B.A.A., N.T., Y.L., R.W., C.S., J.K.B., D.R.W., G.S.C. and Y.M., S.A., A.K. and V.N. are employees of Alphabet and may own stock as part of the standard compensation package. J.H.C. is co-founder of Reaction Explorer LLC that develops and licenses organic chemistry education software; is paid consulting fees from Sutton Pierce, Younker Hyde MacFarlane and Sykes McAllister as a medical expert witness; and is paid consulting fees from ISHI Health. The remaining authors declare no competing interests., (© 2025. The Author(s).)

Without careful dissection of the ways in which biases can be encoded into artificial intelligence (AI) health technologies, there is a risk of perpetuating existing health inequalities at scale. One major source of bias is the data that underpins such technologies. The STANDING Together recommendations aim to encourage transparency regarding limitations of health datasets and proactive evaluation of their effect across population groups. Draft recommendation items were informed by a systematic review and stakeholder survey. The recommendations were developed using a Delphi approach, supplemented by a public consultation and international interview study. Overall, more than 350 representatives from 58 countries provided input into this initiative. 194 Delphi participants from 25 countries voted and provided comments on 32 candidate items across three electronic survey rounds and one in-person consensus meeting. The 29 STANDING Together consensus recommendations are presented here in two parts. Recommendations for Documentation of Health Datasets provide guidance for dataset curators to enable transparency around data composition and limitations. Recommendations for Use of Health Datasets aim to enable identification and mitigation of algorithmic biases that might exacerbate health inequalities. These recommendations are intended to prompt proactive inquiry rather than acting as a checklist. We hope to raise awareness that no dataset is free of limitations, so transparent communication of data limitations should be perceived as valuable, and absence of this information as a limitation. We hope that adoption of the STANDING Together recommendations by stakeholders across the AI health technology lifecycle will enable everyone in society to benefit from technologies which are safe and effective., Competing Interests: Declaration of interests JEA is a named researcher on grants funded by the Engineering and Physical Sciences Research Council (EPSRC), the National Institute for Health and Care Research (NIHR), and the Medical Research Council (MRC); is co-organiser of the Alan Turing Institute Clinical AI Interest Group; and was an NIHR Academic Clinical Fellow from 2017 to 2020. EL is employed as a research fellow at University Hospitals Birmingham NHS Foundation Trust and received support to attend the Symposium for Artificial Intelligence Learning Health Systems conference (May, 2023). JO is an employee of Roche Diagnostics but contributed while an employee of the UK MHRA; declares support from the AIRIS 2024 committee; and has shares in Roche Group. MG is a Canadian Institute for Advanced Research (CIFAR) AI Chair, CIFAR Azrieli Global Scholar, Herman L F von Helmholtz Career Development Professor, and Jameel Clinic Affiliate, and acknowledges support from these programmes. SRP is an employee of Google Research and has stock in Google. JS is an employee of Harrison.ai. NR is an employee of Google Research. HC-L is an employee of Google and has stock or stock options in Google. DS is Acting Deputy Editor of The Lancet Digital Health, Elsevier. BG was scientific advisor for Kheiron Medical Technologies (January, 2018, to September, 2021) and has had part-time employment with stock options as part of the standard compensation package (since October, 2021); has part-time employment at HeartFlow with stock options as part of the standard compensation package (since September, 2018); and is the Royal Academy of Engineering Research Chair in Safe Deployment of Medical Imaging AI. MCa is Director of the Birmingham Health Partners Centre for Regulatory Science and Innovation, Director of the Centre for Patient Reported Outcomes Research, and an NIHR Senior Investigator. MCa, NIHR Birmingham Biomedical Research Centre, Applied Research Collaboration West Midlands, UK SPINE, Research England, the European Regional Development Fund DEMAND Hub at the University of Birmingham, the University Hospitals Birmingham NHS Foundation Trust, and the NIHR Blood and Transplant Research Unit in Precision Transplant and Cellular Therapeutics; funding from Health Data Research UK, Innovate UK (part of UKRI), Macmillan Cancer Support, UCB Pharma, GSK, Anthony Nolan, Gilead Sciences, European Commission, European Federation of Pharmaceutical Industries and Associations, Janssen, Merck, and The Brain Tumor Charity; and personal fees from Aparito, ICON, CIS Oncology, Takeda Pharmaceuticals, Merck, Daiichi Sankyo, Glaukos, GSK, Halfloop, the Patient-Centered Outcomes Research Institute (PCORI), Pfizer, Genentech, and Vertex Pharmaceuticals, outside of the submitted work. MCa also declares royalties via revenue share from the commercial licence of Symptom Burden Questionnaire-Long COVID; payment or honoraria from the University of Maastricht, South-Eastern Norway Regional Health Authority, Cochrane Portugal, Singapore National Medical Research Council; and stock in GSK held by a family member. MCa has a leadership or fiduciary role in PROTEUS consortium (paid via a consultancy fee from Genetech and PCORI). TJP declares NIH awards (OT2OD032701 and R01EB030362), and grants or contracts from Google Cloud and Amazon Web Services. SK was an employee of Hardian Health. L-AF received consulting fees from Therapeutic Goods Administration, Australia, and WHO, and is a member of the AI in Healthcare Committee for Standards Australia and Director of Tethyan Consulting. CS received research funding from UKRI, NIHR, and the Wellcome Trust to support their research programme. BAM reports grants or contracts from MRC, the British Heart Foundation, the United States Agency for International Development, and HDRUK; consulting fees and support for attending meetings from the Bill & Melinda Gates Foundation; and was an employee of the Wellcome Trust at the time of the Delphi study. KH is an employee of Google Research. AK is an employee of Google and receives Google stock reimbursement. MM was an employee of Genomics England, Founder and Director of One HealthTech and Data Science for Health Equity (within One HealthTech), and received support from Nature Journals and Conde Nast. RP is an employee of MHRA. AKM is Deputy Editor of NEJM AI, NEJM Group. PM has a data custodian role as Director of the Clinical Practice Research Datalink and is an employee of MHRA. ZK was employed by Health Research Authority at the time of the consensus process. AAk received grants from the Economic and Social Care Research Council (ADR Wales ES/W012227/1) and HDRUK (HDR-9006). LB received support from the US National Institutes of Health (NIH) All of Us Research Programme to attend STANDING Together meetings. JWG is on the 2022 Robert Wood Johnson Foundation Harold Amos Medical Faculty Development Programme; declares support from the Radiological Society of North America Health Disparities grant (EIHD2204), Lacuna Fund (67), Gordon and Betty Moore Foundation, and NIH (NIBIB) Medical Imaging and Data Resource Center grant under contracts 75N92020C00008 and 75N92020C00021; has received honoraria from the National Bureau of Economic Research for authorship in their 2022 conference collection; has participated on the American Heart Association Deracing Algorithms advisory board; and is a board member of Hl7 and SIIM. LR is Senior Editor of the journal Nature Medicine, Nature Research, Springer Nature. MCh is an NIHR Academic Clinical Fellow. AAr is a panel member for various NIH Research committees, has received honoraria from HDRUK for attending panel meetings, and has an honorary affiliation with Moorfield's Eye Hospital. SP is employed by WHO. ES has had research funded by UK Research and Innovation (UKRI), NIHR, Wellcome Trust, Health Data Research UK (HDRUK), MRC, EPSRC, and Innovate UK, and declares support from the European Respiratory Society, British Thoracic Society, and Gilead. VT is Head of UK Approved Body as at the British Standards Institution. AD reports institutional research grants awarded by the NIHR, MRC, and EPSRC, and is an NIHR Senior Investigator, Director of the UK's incubator for Regulatory Science in AI and Digital Healthcare, deputy director of the Centre for Regulatory Science and Innovation, Birmingham Health Partners, system lead for AI diagnostics to NHS England, and a member of the UK Government's Regulatory Horizons Council. XL reports funding from the NIHR, the UK National Health Service AI Lab, The Health Foundation, Research England, the MHRA, the National Institute for Health and Care Excellence, and Hardian Health, and was previously a Health Studies Scientist at Apple; reports grants from the Medical Research Council, Wellcome Trust, NIHR, and Moorfields Eye Hospital Charity; received payment or honoraria for talks and book chapter reviews from the University of Turku and Elsevier; and received support from Harvard Medical School, SingHealth, and The Ada Lovelace Institute. All other authors declare no competing interests., (© 2024 World Health Organization. Published by Elsevier Ltd. This is an Open Access article published under the CC BY 3.0 IGO license which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. In any use of this article, there should be no suggestion that WHO endorses any specific organisation, products or services. The use of the WHO logo is not permitted. This notice should be preserved along with the article's original URL.)

Competing Interests: Competing interests: All authors are employed by Google and own equity in Alphabet.

Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessment of LLM-generated answers for biases and EquityMedQA, a collection of seven datasets enriched for adversarial queries. Both our human assessment framework and our dataset design process are grounded in an iterative participatory approach and review of Med-PaLM 2 answers. Through our empirical study, we find that our approach surfaces biases that may be missed by narrower evaluation approaches. Our experience underscores the importance of using diverse assessment methodologies and involving raters of varying backgrounds and expertise. While our approach is not sufficient to holistically assess whether the deployment of an artificial intelligence (AI) system promotes equitable health outcomes, we hope that it can be leveraged and built upon toward a shared goal of LLMs that promote accessible and equitable healthcare., Competing Interests: Competing interests: This study was funded by Google LLC and/or its subsidiary (Google). S.R.P., H.C.-L., R.S., D.N., M.A., A. Dieng, N.T., Q.M.R., S.A., N.R., Y.L., M.S., A.W., A.P., C.N., P.S., A. Dewitt, P.M., S.P., K.H., A.K., C.S., J.B., G.C., Y.M., J.S.-L., I.H. and K.S. are employees of Google and may own stock as a part of a standard compensation package. The other authors declare no competing interests., (© 2024. The Author(s).)

Background: AI models have shown promise in performing many medical imaging tasks. However, our ability to explain what signals these models have learned is severely lacking. Explanations are needed in order to increase the trust of doctors in AI-based models, especially in domains where AI prediction capabilities surpass those of humans. Moreover, such explanations could enable novel scientific discovery by uncovering signals in the data that aren't yet known to experts., Methods: In this paper, we present a workflow for generating hypotheses to understand which visual signals in images are correlated with a classification model's predictions for a given task. This approach leverages an automatic visual explanation algorithm followed by interdisciplinary expert review. We propose the following 4 steps: (i) Train a classifier to perform a given task to assess whether the imagery indeed contains signals relevant to the task; (ii) Train a StyleGAN-based image generator with an architecture that enables guidance by the classifier ("StylEx"); (iii) Automatically detect, extract, and visualize the top visual attributes that the classifier is sensitive towards. For visualization, we independently modify each of these attributes to generate counterfactual visualizations for a set of images (i.e., what the image would look like with the attribute increased or decreased); (iv) Formulate hypotheses for the underlying mechanisms, to stimulate future research. Specifically, present the discovered attributes and corresponding counterfactual visualizations to an interdisciplinary panel of experts so that hypotheses can account for social and structural determinants of health (e.g., whether the attributes correspond to known patho-physiological or socio-cultural phenomena, or could be novel discoveries)., Findings: To demonstrate the broad applicability of our approach, we present results on eight prediction tasks across three medical imaging modalities-retinal fundus photographs, external eye photographs, and chest radiographs. We showcase examples where many of the automatically-learned attributes clearly capture clinically known features (e.g., types of cataract, enlarged heart), and demonstrate automatically-learned confounders that arise from factors beyond physiological mechanisms (e.g., chest X-ray underexposure is correlated with the classifier predicting abnormality, and eye makeup is correlated with the classifier predicting low hemoglobin levels). We further show that our method reveals a number of physiologically plausible, previously-unknown attributes based on the literature (e.g., differences in the fundus associated with self-reported sex, which were previously unknown)., Interpretation: Our approach enables hypotheses generation via attribute visualizations and has the potential to enable researchers to better understand, improve their assessment, and extract new knowledge from AI-based models, as well as debug and design better datasets. Though not designed to infer causality, importantly, we highlight that attributes generated by our framework can capture phenomena beyond physiology or pathophysiology, reflecting the real world nature of healthcare delivery and socio-cultural factors, and hence interdisciplinary perspectives are critical in these investigations. Finally, we will release code to help researchers train their own StylEx models and analyze their predictive tasks of interest, and use the methodology presented in this paper for responsible interpretation of the revealed attributes., Funding: Google., Competing Interests: Declaration of interests OL, DYS, HCL, CS, DRW, GSC, AH, YM, YL, NH and BB are current or past Google employees and may own Alphabet stock. IT, CRB, and CL are paid consultants to Google. All other authors declare no competing interests., (Copyright © 2024 The Author(s). Published by Elsevier B.V. All rights reserved.)

Background: Artificial intelligence (AI) has repeatedly been shown to encode historical inequities in healthcare. We aimed to develop a framework to quantitatively assess the performance equity of health AI technologies and to illustrate its utility via a case study., Methods: Here, we propose a methodology to assess whether health AI technologies prioritise performance for patient populations experiencing worse outcomes, that is complementary to existing fairness metrics. We developed the Health Equity Assessment of machine Learning performance (HEAL) framework designed to quantitatively assess the performance equity of health AI technologies via a four-step interdisciplinary process to understand and quantify domain-specific criteria, and the resulting HEAL metric. As an illustrative case study (analysis conducted between October 2022 and January 2023), we applied the HEAL framework to a dermatology AI model. A set of 5420 teledermatology cases (store-and-forward cases from patients of 20 years or older, submitted from primary care providers in the USA and skin cancer clinics in Australia), enriched for diversity in age, sex and race/ethnicity, was used to retrospectively evaluate the AI model's HEAL metric, defined as the likelihood that the AI model performs better for subpopulations with worse average health outcomes as compared to others. The likelihood that AI performance was anticorrelated to pre-existing health outcomes was estimated using bootstrap methods as the probability that the negated Spearman's rank correlation coefficient (i.e., "R") was greater than zero. Positive values of R suggest that subpopulations with poorer health outcomes have better AI model performance. Thus, the HEAL metric, defined as p (R >0), measures how likely the AI technology is to prioritise performance for subpopulations with worse average health outcomes as compared to others (presented as a percentage below). Health outcomes were quantified as disability-adjusted life years (DALYs) when grouping by sex and age, and years of life lost (YLLs) when grouping by race/ethnicity. AI performance was measured as top-3 agreement with the reference diagnosis from a panel of 3 dermatologists per case., Findings: Across all dermatologic conditions, the HEAL metric was 80.5% for prioritizing AI performance of racial/ethnic subpopulations based on YLLs, and 92.1% and 0.0% respectively for prioritizing AI performance of sex and age subpopulations based on DALYs. Certain dermatologic conditions were significantly associated with greater AI model performance compared to a reference category of less common conditions. For skin cancer conditions, the HEAL metric was 73.8% for prioritizing AI performance of age subpopulations based on DALYs., Interpretation: Analysis using the proposed HEAL framework showed that the dermatology AI model prioritised performance for race/ethnicity, sex (all conditions) and age (cancer conditions) subpopulations with respect to pre-existing health disparities. More work is needed to investigate ways of promoting equitable AI performance across age for non-cancer conditions and to better understand how AI models can contribute towards improving equity in health outcomes., Funding: Google LLC., Competing Interests: This study was funded by Google LLC. MS, TS, HC, EW, SP, DM, RJ, GK, YL, SF, QX, CH, PS, FT, PB, LHP, CHM, YM, GSC, DW, SV, CS, YL, IH, PHCC are current or former employees of Google and own stock as part of the standard compensation package. MP and JL are paid consultants of Google., (© 2024 The Author(s).)

Advances in machine learning for health care have brought concerns about bias from the research community; specifically, the introduction, perpetuation, or exacerbation of care disparities. Reinforcing these concerns is the finding that medical images often reveal signals about sensitive attributes in ways that are hard to pinpoint by both algorithms and people. This finding raises a question about how to best design general purpose pretrained embeddings (GPPEs, defined as embeddings meant to support a broad array of use cases) for building downstream models that are free from particular types of bias. The downstream model should be carefully evaluated for bias, and audited and improved as appropriate. However, in our view, well intentioned attempts to prevent the upstream components-GPPEs-from learning sensitive attributes can have unintended consequences on the downstream models. Despite producing a veneer of technical neutrality, the resultant end-to-end system might still be biased or poorly performing. We present reasons, by building on previously published data, to support the reasoning that GPPEs should ideally contain as much information as the original data contain, and highlight the perils of trying to remove sensitive attributes from a GPPE. We also emphasise that downstream prediction models trained for specific tasks and settings, whether developed using GPPEs or not, should be carefully designed and evaluated to avoid bias that makes models vulnerable to issues such as distributional shift. These evaluations should be done by a diverse team, including social scientists, on a diverse cohort representing the full breadth of the patient population for which the final model is intended., Competing Interests: Declaration of interests W-HW, AS, APK, AD’A, JP, RP, SP, VN, SA, AK, HC-L, YM, GSC, DRW, SS, SP, KE, and YL are employees of Google and hold Alphabet stock. W-HW, AS, APK, AD’A, RP, SP, VN, SA, AK, HC-L, YM, GSC, DRW, SS, SP, KE, and YL have patents filed or in progress under Google, broadly related to machine learning and embedding models. CL performs work at Google as a medical consultant via Vituity and receives consulting fees for clinical perspective and guidance. LAGC receives support for educational events and meetings from the National Institutes of Health, Stanford University, University of California San Francisco, University of Toronto, College of Intensive Care Medicine of Australia and New Zealand, University of Bergen, Amsterdam University Medical Centers, Académie Nationale de Médecine (France), and the Doris Duke Foundation (for the Reconsidering Race in Clinical Algorithms workshop at the National Academy of Medicine, Washington, DC). LAGC is an Editor in Chief of PLOS Digital Health, and on the advisory board of the Lancet Digital Health., (Copyright © 2024 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license. Published by Elsevier Ltd.. All rights reserved.)

Artificial intelligence as a medical device is increasingly being applied to healthcare for diagnosis, risk stratification and resource allocation. However, a growing body of evidence has highlighted the risk of algorithmic bias, which may perpetuate existing health inequity. This problem arises in part because of systemic inequalities in dataset curation, unequal opportunity to participate in research and inequalities of access. This study aims to explore existing standards, frameworks and best practices for ensuring adequate data diversity in health datasets. Exploring the body of existing literature and expert views is an important step towards the development of consensus-based guidelines. The study comprises two parts: a systematic review of existing standards, frameworks and best practices for healthcare datasets; and a survey and thematic analysis of stakeholder views of bias, health equity and best practices for artificial intelligence as a medical device. We found that the need for dataset diversity was well described in literature, and experts generally favored the development of a robust set of guidelines, but there were mixed views about how these could be implemented practically. The outputs of this study will be used to inform the development of standards for transparency of data diversity in health datasets (the STANDING Together initiative)., (© 2023. The Author(s).)

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications., (© 2023. The Author(s).)

Digital health promises to increase intervention reach and effectiveness for a range of behavioral health outcomes. Behavioral scientists have a unique opportunity to infuse their expertise in all phases of a digital health intervention, from design to implementation. The aim of this study was to assess behavioral scientists' interests and needs with respect to digital health endeavors, as well as gather expert insight into the role of behavioral science in the evolution of digital health. The study used a two-phased approach: (a) a survey of behavioral scientists' current needs and interests with respect to digital health endeavors (n = 346); (b) a series of interviews with digital health stakeholders for their expert insight on the evolution of the health field (n = 15). In terms of current needs and interests, the large majority of surveyed behavioral scientists (77%) already participate in digital health projects, and from those who have not done so yet, the majority (65%) reported intending to do so in the future. In terms of the expected evolution of the digital health field, interviewed stakeholders anticipated a number of changes, from overall landscape changes through evolving models of reimbursement to more significant oversight and regulations. These findings provide a timely insight into behavioral scientists' current needs, barriers, and attitudes toward the use of technology in health care and public health. Results might also highlight the areas where behavioral scientists can leverage their expertise to both enhance digital health's potential to improve health, as well as to prevent the potential unintended consequences that can emerge from scaling the use of technology in health care., (© Society of Behavioral Medicine 2020. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

The rapid expansion of technology promises to transform the behavior science field by revolutionizing the ways in which individuals can monitor and improve their health behaviors. To fully live into this promise, the behavior science field must address distinct challenges, including: building interventions that are not only scientifically sound but also engaging; using evaluation methods to precisely assess intervention components for intervention optimization; and building personalized interventions that acknowledge and adapt to the dynamic ecosystem of individual and contextual variables that impact behavior change. The purpose of this paper is to provide a framework to address these challenges by leveraging behavior science, human-centered design, and data science expertise throughout the cycle of developing and evaluating digital behavior change interventions (DBCIs). To define this framework, we reviewed current models and practices for intervention development and evaluation, as well as technology industry models for product development. The framework promotes an iterative process, aiming to maximize outcomes by incorporating faster and more frequent testing cycles into the lifecycle of a DBCI. Within the framework provided, we describe each phase, from development to evaluation, to discuss the optimal practices, necessary stakeholders, and proposed evaluation methods. The proposed framework may inform practices in both academia and industry, as well as highlight the need to offer collaborative platforms to ensure successful partnerships that can lead to more effective DBCIs that reach broad and diverse populations., (© The Author(s) 2019. Published by Oxford University Press on behalf of the Society of Behavioral Medicine.)

Researchers and practitioners of digital behavior change interventions (DBCI) use varying and, often, incongruent definitions of the term "engagement," thus leading to a lack of precision in DBCI measurement and evaluation. The objective of this paper is to propose discrete definitions for various types of user engagement and to explain why precision in the measurement of these engagement types is integral to ensuring the intervention is effective for health behavior modulation. Additionally, this paper presents a framework and practical steps for how engagement can be measured in practice and used to inform DBCI design and evaluation. The key purpose of a DBCI is to influence change in a target health behavior of a user, which may ultimately improve a health outcome. Using available literature and practice-based knowledge of DBCI, the framework conceptualizes two primary categories of engagement that must be measured in DBCI. The categories are health behavior engagement, referred to as "Big E," and DBCI engagement, referred to as "Little e." DBCI engagement is further bifurcated into two subclasses: (1) user interactions with features of the intervention designed to encourage frequency of use (ie, simple login, games, and social interactions) and make the user experience appealing, and (2) user interactions with behavior change intervention components (ie, behavior change techniques), which influence determinants of health behavior and subsequently influence health behavior. Achievement of Big E in an intervention delivered via digital means is contingent upon Little e. If users do not interact with DBCI features and enjoy the user experience, exposure to behavior change intervention components will be limited and less likely to influence the behavioral determinants that lead to health behavior engagement (Big E). Big E is also dependent upon the quality and relevance of the behavior change intervention components within the solution. Therefore, the combination of user interactions and behavior change intervention components creates Little e, which is, in turn, designed to improve Big E. The proposed framework includes a model to support measurement of DBCI that describes categories of engagement and details how features of Little e produce Big E. This framework can be applied to DBCI to support various health behaviors and outcomes and can be utilized to identify gaps in intervention efficacy and effectiveness., (©Heather Cole-Lewis, Nnamdi Ezeanochie, Jennifer Turgiss. Originally published in JMIR Formative Research (http://formative.jmir.org), 10.10.2019.)

Background: The Millennium Villages Project facilitated technology-based health interventions in rural under-resourced areas of sub-Saharan Africa. Our study examined whether data entry using SMS compared with paper forms by community health workers (CHWs) led to higher proportion of timely follow-up visits for malnutrition screening in under-5 children in Ghana, Rwanda, Senegal, and Uganda., Methods: Children under 5 years were screened for malnutrition every 90 days by CHWs using mid-upper arm circumference (MUAC) readings. CHWs used either SMS texts or paper forms to enter MUAC data. Reminder texts were sent at 15 days before follow-up was needed. Chi-square tests assessed proportion of timely follow-up visits within 90 days between SMS and paper groups. Logistic regression analysis was conducted in a step-wise multivariate model. Post-hoc power calculations were conducted to verify strength of associations., Results: SMS data entry was associated with a higher proportion of timely malnutrition follow-up visits compared with paper forms across all sites. The association was strongest with consistent SMS use over consecutive visits. SMS use at the first of 2 consecutive visits was most effective, highlighting the importance of SMS reminder alerts., Conclusions: SMS technology with reminders increased timely CHW malnutrition screening visits for under-5 children in Ghana, Rwanda, Senegal, and Uganda, highlighting the importance of such technology for improving health worker behavior in low-resource settings., (© Sarma et al.)

Background: The rise in usage of and access to new technologies in recent years has led to a growth in digital health behavior change interventions. As the shift to digital platforms continues to grow, it is increasingly important to consider how the field of information architecture (IA) can inform the development of digital health interventions. IA is the way in which digital content is organized and displayed, which strongly impacts users' ability to find and use content. While many information architecture best practices exist, there is a lack of empirical evidence on the role it plays in influencing behavior change and health outcomes., Objective: Our aim was to conduct a systematic review synthesizing the existing literature on website information architecture and its effect on health outcomes, behavioral outcomes, and website engagement., Methods: To identify all existing information architecture and health behavior literature, we searched articles published in English in the following databases (no date restrictions imposed): ACM Digital Library, CINAHL, Cochrane Library, Google Scholar, Ebsco, and PubMed. The search terms used included information terms (eg, information architecture, interaction design, persuasive design), behavior terms (eg, health behavior, behavioral intervention, ehealth), and health terms (eg, smoking, physical activity, diabetes). The search results were reviewed to determine if they met the inclusion and exclusion criteria created to identify empirical research that studied the effect of IA on health outcomes, behavioral outcomes, or website engagement. Articles that met inclusion criteria were assessed for study quality. Then, data from the articles were extracted using a priori categories established by 3 reviewers. However, the limited health outcome data gathered from the studies precluded a meta-analysis., Results: The initial literature search yielded 685 results, which was narrowed down to three publications that examined the effect of information architecture on health outcomes, behavioral outcomes, or website engagement. One publication studied the isolated impact of information architecture on outcomes of interest (ie, website use and engagement; health-related knowledge, attitudes, and beliefs; and health behaviors), while the other two publications studied the impact of information architecture, website features (eg, interactivity, email prompts, and forums), and tailored content on these outcomes. The paper that investigated IA exclusively found that a tunnel IA improved site engagement and behavior knowledge, but it decreased users' perceived efficiency. The first study that did not isolate IA found that the enhanced site condition improved site usage but not the amount of content viewed. The second study that did not isolate IA found that a tailored site condition improved site usage, behavior knowledge, and some behavior outcomes., Conclusions: No clear conclusion can be made about the relationship between IA and health outcomes, given limited evidence in the peer-reviewed literature connecting IA to behavioral outcomes and website engagement. Only one study reviewed solely manipulated IA, and we therefore recommend improving the scientific evidence base such that additional empirical studies investigate the impact of IA in isolation. Moreover, information from the gray literature and expert opinion might be identified and added to the evidence base, in order to lay the groundwork for hypothesis generation to improve empirical evidence on information architecture and health and behavior outcomes., (©Jillian Pugatch, Emily Grenen, Stacy Surla, Mary Schwarz, Heather Cole-Lewis. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 21.03.2018.)

Objective: This observational study highlights key insights related to participant engagement and cessation among adults who voluntarily subscribed to the nationwide US-based SmokefreeTXT program, a 42-day mobile phone text message smoking cessation program., Methods: Point prevalence abstinence rates were calculated for subscribers who initiated treatment in the program (n=18 080). The primary outcomes for this study were treatment completion and point prevalence abstinence rate at the end of the 42-day treatment. Secondary outcomes were point prevalence abstinence rates at 7 days postquit, 3 months post-treatment and 6 months post-treatment, as well as response rates to point prevalence abstinence assessments., Results: Over half the sample completed the 42-day treatment (n=9686). The end-of-treatment point prevalence abstinence for subscribers who initiated treatment was 7.2%. Among those who completed the entire 42 days of treatment, the end-of-treatment point prevalence abstinence was 12.9%. For subscribers who completed treatment, point prevalence abstinence results varied: 7 days postquit (23.7%), 3 months post-treatment (7.3%) and 6 months post-treatment (3.7%). Response rates for abstinence assessment messages ranged from 4.36% to 34.48%., Conclusions: Findings from this study illuminate the need to more deeply understand reasons for subscriber non-response and opt out and, in turn, improve program engagement and our ability to increase the likelihood for participants to stop smoking and measure long-term outcomes. Patterns of opt out for the program mirror the relapse curve generally observed for smoking cessation, thus highlighting time points at which to increase efforts to retain participants and provide additional support or incentives., Competing Interests: Competing interests: None declared., (Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.)

Background: Health risk assessments (HRAs), which often screen for depressive symptoms, are administered to millions of employees and health plan members each year. HRA data provide an opportunity to examine longitudinal trends in depressive symptomatology, as researchers have done previously with other populations., Objective: The primary research questions were: (1) Can we observe longitudinal trajectories in HRA populations like those observed in other study samples? (2) Do HRA variables, which primarily reflect modifiable health risks, help us to identify predictors associated with these trajectories? (3) Can we make meaningful recommendations for population health management, applicable to HRA participants, based on predictors we identify?, Methods: This study used growth mixture modeling (GMM) to examine longitudinal trends in depressive symptomatology among 22,963 participants in a Web-based HRA used by US employers and health plans. The HRA assessed modifiable health risks and variables such as stress, sleep, and quality of life., Results: Five classes were identified: A "minimal depression" class (63.91%, 14,676/22,963) whose scores were consistently low across time, a "low risk" class (19.89%, 4568/22,963) whose condition remained subthreshold, a "deteriorating" class (3.15%, 705/22,963) who began at subthreshold but approached severe depression by the end of the study, a "chronic" class (4.71%, 1081/22,963) who remained highly depressed over time, and a "remitting" class (8.42%, 1933/22,963) who had moderate depression to start, but crossed into minimal depression by the end. Among those with subthreshold symptoms, individuals who were male (P<.001) and older (P=.01) were less likely to show symptom deterioration, whereas current depression treatment (P<.001) and surprisingly, higher sleep quality (P<.001) were associated with increased probability of membership in the "deteriorating" class as compared with "low risk." Among participants with greater symptomatology to start, those in the "severe" class tended to be younger than the "remitting" class (P<.001). Lower baseline sleep quality (P<.001), quality of life (P<.001), stress level (P<.001), and current treatment involvement (P<.001) were all predictive of membership in the "severe" class., Conclusions: The trajectories identified were consistent with trends in previous research. The results identified some key predictors: we discuss those that mirror prior studies and offer some hypotheses as to why others did not. The finding that 1 in 5 HRA participants with subthreshold symptoms deteriorated to the point of clinical distress during succeeding years underscores the need to learn more about such individuals. We offer additional recommendations for follow-up research, which should be designed to reflect changes in health plan demographics and HRA delivery platforms. In addition to utilizing additional variables such as cognitive style to refine predictive models, future research could also begin to test the impact of more aggressive outreach strategies aimed at participants who are likely to deteriorate or remain significantly depressed over time., (©Richard Bedrosian, Matt Hawrilenko, Heather Cole-Lewis. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 31.03.2017.)

A central goal of behavioral medicine is the creation of evidence-based interventions for promoting behavior change. Scientific knowledge about behavior change could be more effectively accumulated using "ontologies." In information science, an ontology is a systematic method for articulating a "controlled vocabulary" of agreed-upon terms and their inter-relationships. It involves three core elements: (1) a controlled vocabulary specifying and defining existing classes; (2) specification of the inter-relationships between classes; and (3) codification in a computer-readable format to enable knowledge generation, organization, reuse, integration, and analysis. This paper introduces ontologies, provides a review of current efforts to create ontologies related to behavior change interventions and suggests future work. This paper was written by behavioral medicine and information science experts and was developed in partnership between the Society of Behavioral Medicine's Technology Special Interest Group (SIG) and the Theories and Techniques of Behavior Change Interventions SIG. In recent years significant progress has been made in the foundational work needed to develop ontologies of behavior change. Ontologies of behavior change could facilitate a transformation of behavioral science from a field in which data from different experiments are siloed into one in which data across experiments could be compared and/or integrated. This could facilitate new approaches to hypothesis generation and knowledge discovery in behavioral science.

Background: Social media platforms are increasingly being used to support individuals in behavior change attempts, including smoking cessation. Examining the interactions of participants in health-related social media groups can help inform our understanding of how these groups can best be leveraged to facilitate behavior change., Objective: The aim of this study was to analyze patterns of participation, self-reported smoking cessation length, and interactions within the National Cancer Institutes' Facebook community for smoking cessation support., Methods: Our sample consisted of approximately 4243 individuals who interacted (eg, posted, commented) on the public Smokefree Women Facebook page during the time of data collection. In Phase 1, social network visualizations and centrality measures were used to evaluate network structure and engagement. In Phase 2, an inductive, thematic qualitative content analysis was conducted with a subsample of 500 individuals, and correlational analysis was used to determine how participant engagement was associated with self-reported session length., Results: Between February 2013 and March 2014, there were 875 posts and 4088 comments from approximately 4243 participants. Social network visualizations revealed the moderator's role in keeping the community together and distributing the most active participants. Correlation analyses suggest that engagement in the network was significantly inversely associated with cessation status (Spearman correlation coefficient = -0.14, P=.03, N=243). The content analysis of 1698 posts from 500 randomly selected participants identified the most frequent interactions in the community as providing support (43%, n=721) and announcing number of days smoke free (41%, n=689)., Conclusions: These findings highlight the importance of the moderator for network engagement and provide helpful insights into the patterns and types of interactions participants are engaging in. This study adds knowledge of how the social network of a smoking cessation community behaves within the confines of a Facebook group.

Objective: To investigate subjective experiences and patterns of engagement with a novel electronic tool for facilitating reflection and problem solving for individuals with type 2 diabetes, Mobile Diabetes Detective (MoDD)., Methods: In this qualitative study, researchers conducted semi-structured interviews with individuals from economically disadvantaged communities and ethnic minorities who are participating in a randomized controlled trial of MoDD. The transcripts of the interviews were analyzed using inductive thematic analysis; usage logs were analyzed to determine how actively the study participants used MoDD., Results: Fifteen participants in the MoDD randomized controlled trial were recruited for the qualitative interviews. Usage log analysis showed that, on average, during the 4 weeks of the study, the study participants logged into MoDD twice per week, reported 120 blood glucose readings, and set two behavioral goals. The qualitative interviews suggested that individuals used MoDD to follow the steps of the problem-solving process, from identifying problematic blood glucose patterns, to exploring behavioral triggers contributing to these patterns, to selecting alternative behaviors, to implementing these behaviors while monitoring for improvements in glycemic control., Discussion: This qualitative study suggested that informatics interventions for reflection and problem solving can provide structured scaffolding for facilitating these processes by guiding users through the different steps of the problem-solving process and by providing them with context-sensitive evidence and practice-based knowledge related to diabetes self-management on each of those steps., Conclusion: This qualitative study suggested that MoDD was perceived as a useful tool in engaging individuals in self-monitoring, reflection, and problem solving., (© The Author 2016. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please email: journals.permissions@oup.com.)

Background: Electronic cigarette (e-cigarette) use has increased in the United States, leading to active debate in the public health sphere regarding e-cigarette use and regulation. To better understand trends in e-cigarette attitudes and behaviors, public health and communication professionals can turn to the dialogue taking place on popular social media platforms such as Twitter., Objective: The objective of this study was to conduct a content analysis to identify key conversation trends and patterns over time using historical Twitter data., Methods: A 5-category content analysis was conducted on a random sample of tweets chosen from all publicly available tweets sent between May 1, 2013, and April 30, 2014, that matched strategic keywords related to e-cigarettes. Relevant tweets were isolated from the random sample of approximately 10,000 tweets and classified according to sentiment, user description, genre, and theme. Descriptive analyses including univariate and bivariate associations, as well as correlation analyses were performed on all categories in order to identify patterns and trends., Results: The analysis revealed an increase in e-cigarette-related tweets from May 2013 through April 2014, with tweets generally being positive; 71% of the sample tweets were classified as having a positive sentiment. The top two user categories were everyday people (65%) and individuals who are part of the e-cigarette community movement (16%). These two user groups were responsible for a majority of informational (79%) and news tweets (75%), compared to reputable news sources and foundations or organizations, which combined provided 5% of informational tweets and 12% of news tweets. Personal opinion (28%), marketing (21%), and first person e-cigarette use or intent (20%) were the three most common genres of tweets, which tended to have a positive sentiment. Marketing was the most common theme (26%), and policy and government was the second most common theme (20%), with 86% of these tweets coming from everyday people and the e-cigarette community movement combined, compared to 5% of policy and government tweets coming from government, reputable news sources, and foundations or organizations combined., Conclusions: Everyday people and the e-cigarette community are dominant forces across several genres and themes, warranting continued monitoring to understand trends and their implications regarding public opinion, e-cigarette use, and smoking cessation. Analyzing social media trends is a meaningful way to inform public health practitioners of current sentiments regarding e-cigarettes, and this study contributes a replicable methodology.

Background: Electronic cigarettes (e-cigarettes) continue to be a growing topic among social media users, especially on Twitter. The ability to analyze conversations about e-cigarettes in real-time can provide important insight into trends in the public's knowledge, attitudes, and beliefs surrounding e-cigarettes, and subsequently guide public health interventions., Objective: Our aim was to establish a supervised machine learning algorithm to build predictive classification models that assess Twitter data for a range of factors related to e-cigarettes., Methods: Manual content analysis was conducted for 17,098 tweets. These tweets were coded for five categories: e-cigarette relevance, sentiment, user description, genre, and theme. Machine learning classification models were then built for each of these five categories, and word groupings (n-grams) were used to define the feature space for each classifier., Results: Predictive performance scores for classification models indicated that the models correctly labeled the tweets with the appropriate variables between 68.40% and 99.34% of the time, and the percentage of maximum possible improvement over a random baseline that was achieved by the classification models ranged from 41.59% to 80.62%. Classifiers with the highest performance scores that also achieved the highest percentage of the maximum possible improvement over a random baseline were Policy/Government (performance: 0.94; % improvement: 80.62%), Relevance (performance: 0.94; % improvement: 75.26%), Ad or Promotion (performance: 0.89; % improvement: 72.69%), and Marketing (performance: 0.91; % improvement: 72.56%). The most appropriate word-grouping unit (n-gram) was 1 for the majority of classifiers. Performance continued to marginally increase with the size of the training dataset of manually annotated data, but eventually leveled off. Even at low dataset sizes of 4000 observations, performance characteristics were fairly sound., Conclusions: Social media outlets like Twitter can uncover real-time snapshots of personal sentiment, knowledge, attitudes, and behavior that are not as accessible, at this scale, through any other offline platform. Using the vast data available through social media presents an opportunity for social science and public health methodologies to utilize computational methodologies to enhance and extend research and practice. This study was successful in automating a complex five-category manual content analysis of e-cigarette-related content on Twitter using machine learning techniques. The study details machine learning model specifications that provided the best accuracy for data related to e-cigarettes, as well as a replicable methodology to allow extension of these methods to additional topics.

Despite growing evidence of the benefits of breastfeeding, rates of breastfeeding remain disproportionately low among adolescent mothers compared with older mothers in the United States. Current interventions primarily target adult women, and little evidence is available for breastfeeding promotion among young women. Accordingly, we aim to review interventions designed to improve breastfeeding rates among adolescents to make recommendations for future research and practice. We searched MEDLINE and PsycINFO for articles that evaluated interventions aiming to improve rates of breastfeeding initiation, duration, or exclusivity among adolescents. Inclusion criteria included interventions targeting pregnant or postpartum adolescents (mean/median age <22 years) that were conducted in high-income settings. Six interventions met our inclusion criteria; of these, 4 interventions aimed to increase breastfeeding initiation, 5 aimed to increase breastfeeding duration, and 4 aimed to increase breastfeeding exclusivity. Interventions included school-based programs, home visits, and telephone support that were implemented by a combination of peer counselors, nurse clinicians, doulas, and lactation consultants. Only 1 intervention, a combination of education and counseling provided by a lactation consultant-peer counselor team, significantly improved both breastfeeding initiation and duration. Other results were mixed, and studies were subject to several methodological limitations. We recommend that more interventions should be developed and evaluated. In addition, interventions should be less resource intensive, be more theoretically driven, and specifically include mothers and partners of adolescents to successfully promote breastfeeding among adolescent mothers., (© The Author(s) 2014.)

The aim of this systematic review of reviews is to identify mobile text-messaging interventions designed for health improvement and behavior change and to derive recommendations for practice. We have compiled and reviewed existing systematic research reviews and meta-analyses to organize and summarize the text-messaging intervention evidence base, identify best-practice recommendations based on findings from multiple reviews, and explore implications for future research. Our review found that the majority of published text-messaging interventions were effective when addressing diabetes self-management, weight loss, physical activity, smoking cessation, and medication adherence for antiretroviral therapy. However, we found limited evidence across the population of studies and reviews to inform recommended intervention characteristics. Although strong evidence supports the value of integrating text-messaging interventions into public health practice, additional research is needed to establish longer-term intervention effects, identify recommended intervention characteristics, and explore issues of cost-effectiveness.

The field of mHealth has made significant advances in a short period of time, demanding a more thorough and scientific approach to understanding and evaluating its progress. A recent review of mHealth literature identified two primary research needs in order for mHealth to strengthen health systems and promote healthy behaviors, namely health outcomes and cost-benefits (Mechael et al., 2010 ). In direct response to the gaps identified in mHealth research, the aim of this paper is to present the study design and highlight key observations and next steps from an evaluation of the mHealth activities within the electronic health (eHealth) architecture implemented by the Millennium Villages Project (MVP) by leveraging data generated through mobile technology itself alongside complementary qualitative research and costing assessments. The study, funded by the International Development and Research Centre (IDRC) as part of the Open Architecture Standards and Information Systems research project (OASIS II) (Sinha, 2009 ), is being implemented on data generated by 14 MVP sites in 10 Sub-Saharan African countries including more in-depth research in Ghana, Rwanda, Tanzania, and Uganda. Specific components of the study include rigorous quantitative case-control analyses and other epidemiological approaches (such as survival analysis) supplemented by in-depth qualitative interviews spread out over 18 months, as well as a costing study to assess the impact of mHealth on health outcomes, service delivery, and efficiency.

Objectives: Strong evidence exists to support an intergenerational cycle of adolescent fatherhood, yet such a cycle has not been studied. We examined whether paternal adolescent fatherhood (i.e., father of study participant was age 19 years or younger when his first child was born) and other factors derived from the ecological systems theory predicted participant adolescent fatherhood., Methods: Data included 1496 young males who were interviewed annually from the National Longitudinal Survey of Youth 1997. Cox regression survival analysis was used to determine the effect of paternal adolescent fatherhood on participant adolescent fatherhood., Results: Sons of adolescent fathers were 1.8 times more likely to become adolescent fathers than were sons of older fathers, after other risk factors were accounted for. Additionally, factors from each ecological domain-individual (delinquency), family (maternal education), peer (early adolescent dating), and environment (race/ethnicity, physical risk environment)-were independent predictors of adolescent fatherhood., Conclusions: These findings support the need for pregnancy prevention interventions specifically designed for young males who may be at high risk for continuing this cycle. Interventions that address multiple levels of risk will likely be most successful at reducing pregnancies among partners of young men.