Search

Agent-based Models (ABMs) are valuable tools for policy analysis. ABMs help analysts explore the emergent consequences of regulatory and policy interventions in multi-agent decision-making settings. But the validity of inferences drawn from ABM explorations depends on the quality of the ABM agents’ behavioral models. Prior approaches for specifying behaviors have limitations. This paper examines the value of reinforcement learning (RL) models as adaptive, high-performing, and behaviorally-valid models of agent decision-making in ABMs. We discuss the value of RL for modeling agents’ utility-maximizing behaviors in policy-relevant ABMs. We address the problem of adapting RL algorithms to handle multi-agency in games by adapting and extending methods from recent literature. We evaluate examples of such RL-based ABM agents via experiments on two policy-relevant ABMs: a Minority Game ABM, and an ABM of Influenza Transmission. The RL behavioral models can outperform the default adaptive behavioral models. We also run analytic experiments on our RL-equipped ABMs: explorations of the effects of dynamic behavioral heterogeneity in a population, the impact of social network factors on adaptability, and the emergence of synchronization in a community. Our results suggest that the RL formalism can be an efficient abstraction for behavioral models in ABMs.

Scenario discovery offers a new means to characterize and communicate the information in computer simulation models under conditions of deep uncertainty. The approach first defines scenarios as the future states of world where a proposed policy fails to meet its goals and then uses statistical algorithms to find concise descriptions of such regions in large databases of simulation model results. Current scenario discovery applications rely on the Patient Rule Induction Method (PRIM), a user-interactive bump-hunting algorithm that identifies hyper-rectangular regions in the input space of the simulation model. While often successful, scenario discovery applications have been limited because in general a policy's vulnerabilities are not well described by the PRIM's hyper-rectangular regions. This study proposes and evaluates improved scenario discovery algorithms that address this challenge with a Principal Component Analysis (PCA)-based preprocessing step that transforms the original model input parameters so that PRIM can then identify high quality hyper-rectangular scenarios in the new rotated coordination system. We explore two versions. PCA-PRIM allows rotations among all uncertain model input parameters and CPCA-PRIM (for constrained PCA-PRIM) only allows rotations among parameters within user-specified domains. The latter may provide more useful information to users, who may find scenario axes described by linear combinations of related domain parameters more interpretable than combinations of dissimilar parameters. We run two sets of tests on the PCA-PRIM and CPCA-PRIM algorithms, the first using simulated test date and the second results from a model used in a previous RAND study of the cost-effectiveness of renewable energy portfolio standards. We find that the new algorithms produce higher quality scenarios than PRIM alone as evaluated by several important measures of merit. In the test data, PCA-PRIM produces improvements averaging 37 percent, and CPCA-PRIM averaging 14 percent, over PRIM alone. In the renewable energy policy case study, PCA-PRIM and CPCA-PRIM exhibit similar improvements of about 16 percent over PRIM, and CPCA-PRIM generates scenarios interpretable by, and that provide useful information to, decision makers.

(See the Editorial Commentary by Jena on pages 1797–9.) Recent evidence suggests that antiretroviral therapy (ART) reduces the probability of human immunodeficiency virus (HIV) transmission, particularly if initiated at early stages of the disease [1–3]. Evidence also suggests that testing HIV positive causes dramatic reductions in sexual activity levels, which also reduces HIV transmission probability [4, 5]. In light of this evidence, health officials across the United States are taking steps toward scaling up testing programs and recommending immediate ART initiation to individuals testing positive, regardless of CD4 cell count [6]. Some argue a “test-and-treat” policy could lead to elimination of the HIV epidemic [7–9]. Although reasons exist for optimism regarding the role of increased testing and treating in HIV prevention agendas, some concerns remain unaddressed. One important concern is that increased ART use creates more multidrug-resistant strains, which might limit the benefits of test-and-treat [10]. Treatment resistance already poses a major public health problem in the United States [11]. Therefore, determining how to implement test-and-treat policies where HIV prevention is maximized and multidrug resistance (MDR) prevalence is minimized is important. Previous mathematical models that simulate the impact of scaling up test-and-treat policies show mixed results. Some find dramatic benefits [7, 12], whereas others find only modest effects [13–15]. There is also evidence from a natural experiment suggesting that some model results may be exaggerated [16]. The discrepancy in model findings reflects the sensitivity of test-and-treat outcomes to differing underlying assumptions about HIV prevalence, proportion of undiagnosed cases, risky sexual behavior, and other population or location-specific parameters [17, 18]. This highlights the critical need to calibrate mathematical models to mimic real-world HIV prevalence and incidence trends [19]. Additionally, none of these articles explicitly addresses potential effects of test-and-treat policies on the spread of MDR. However, an earlier strand of literature examining expanded ART use addresses this issue and reports mixed findings. Some find that increasing the percentage of people using ART would substantially reduce the HIV epidemic's severity even in the presence of high ART resistance levels [9, 20]. However, the authors note that emergence of highly transmissible resistant strains of HIV can significantly reduce the benefits of expanded use of ART [9]. Baggaley et al found that controlling sub-Saharan African HIV epidemics through treatment is ineffective, as increasing the proportion on treatment increases the emergence and spread of drug resistance [21]. Others find that benefits from expanded ART use are counterbalanced by modest increases in risky sexual behavior [21–23]. Our study contributes to this literature by using a mathematical model to simulate effects of increased testing and early initiation of treatment for men who have sex with men (MSM) in Los Angeles County (LAC). No previous study has focused on test-and-treat in LAC. In the United States, LAC has the largest incidence of HIV, and MSM account for 82% of all people living with HIV/AIDS (PLWHA) [24, 25]. We calibrate our model using HIV surveillance data from LAC for the years 2000 to 2009. Following calibration, we manipulate parameters relating to test-and-treat to simulate alternate test-and-treat policy scenarios. To assess the individual and complementary effects of testing and treating on epidemiologic outcomes, the intensity of both testing and treatment rates are varied. Finally, we also assess how the intensity of each mechanism affects the portion of the population with MDR. The results could help inform policy makers on best approaches for the test-and-treat policy and where to focus future HIV prevention efforts.

BackgroundData on effectiveness of preexposure prophylaxis (PrEP) for human immunodeficiency virus (HIV)-uninfected women attempting conception with HIV-infected male partners are limited to observational studies.MethodsTo explore the benefits of PrEP for conception, we developed a model to estimate the average annual probability of a woman remaining HIV-uninfected and having a child ("successful" outcome) via condomless sex with an HIV-infected male. The outcome likelihood is dependent upon parameters defining HIV-1 infectivity. We simulated 2 scenarios: optimal (condomless sex acts limited to the ovulation window), and suboptimal (acts not limited to ovulation).ResultsIn the optimal scenario when the male is on antiretroviral therapy (ART), the average annual probability of the successful outcome is 29.1%, increasing to 29.2% with the addition of PrEP (P = .45). In the suboptimal scenario, the probability is 26.8% with ART alone versus 27.3% with ART/PrEP (P < .0001). Older maternal age reduces the probability of success in both scenarios, particularly after age 30.ConclusionsIn our model, PrEP provides little added benefit when the HIV-infected male partner is on ART, condomless sex is limited to the ovulation window, and other modifiable transmission risks are optimized. Older female age decreases the probability of success by increasing the number of condomless sex acts required for conception.

The threat of suicide bombings in the United States and elsewhere prompted the Department of Homeland Security to commission the Naval Research Laboratory (NRL) to develop a method for predicting the determinants of suicide bombing attacks. As a test case, NRL chose to study bombings in Israel, where many such attacks have already taken place and where the data describing them are particularly good. NRL focused on four Israeli cities Jerusalem, Haifa, Tel Aviv, and Netanya and on three terrorist groups Hamas, Al-Aqsa Martyrs Brigade, and the Palestinian Islamic Jihad from 1993 to 2006. The first part of the NRL study identified spatial preference patterns. It found that areas at high risk for attack included those with bus routes, main traffic routes, and parking areas. For the second part of the study, NRL turned to RAND to enhance the predictive ability of the methodology by incorporating socioeconomic, demographic, and political aspects of suicide bomber attacks. RAND researchers tested these variables both quantitatively and qualitatively. The quantitative analysis focuses on Jerusalem, while the qualitative analysis includes all four cities.

We thank Gonzalez-Serna and his colleagues for initiating a discussion on our paper that models the impact of the test-and-treat policy on the human immunodeficiency virus (HIV) epidemic in Los Angeles County. Gonzalez-Serna et al contest the relevance of our findings that test-and-treat could potentially increase multidrug resistance (MDR) in Los Angeles by 89% [1]. Using observational data from British Columbia, Canada, they show that MDR and total drug resistance prevalence in British Columbia decreased over a period during which antiretroviral treatment (ART) prevalence increased by 60%. There are several reasons why this finding might not be particularly relevant for evaluating the impact of test-and-treat in Los Angeles or other regions of the world. First, British Columbia is a distinct setting with vastly different demographics and healthcare characteristics than Los Angeles. There is evidence that MDR prevalence in Canada is generally much lower than in the United States [2]. As with any mathematical model, our results are a product of the assumptions we make about parameter values and initial conditions, which are setting-specific. Because MDR prevalence in Los Angeles is higher at baseline than in British Columbia, the force of infection of transmitted resistance is higher, causing faster growth. Model results may be different with dynamics based on British Columbia characteristics. Second, these findings from British Columbia are inconsistent with findings in other parts of the world [3–6]. A recent World Health Organization report highlights the growth of drug-resistant HIV in low- and middle-income countries over the past decade, and shows a positive association between ART coverage and prevalence of transmitted drug-resistant HIV [7]. Third, a recent Canadian surveillance report shows that the results Gonzales-Serna et al report from British Columbia might not even generalize to other provinces in Canada. This report surveys 6 Canadian provinces and shows that prevalence of resistance increased by approximately 70% from 1999 to 2008 and there was no reduction in MDR [8]. Furthermore, a considerable proportion of transmitted drug resistance in both the United States and Canada remains undetected [9]. Fourth, early-stage HIV (ESH) treatment—a hallmark of test-and-treat—stayed relatively stable in British Columbia from 1995 to 2008 [10]. It is likely that ESH will coincide with lower levels of adherence, a strong predictor of increases in acquired drug resistance. Gonzalez-Serna et al also note a decrease in clinical significance of MDR, as dozens of different drug classes are now available. They suggest that pandrug resistance is the appropriate resistance measure instead of triple-therapy resistance, which we use. Even if this were true, MDR will likely raise HIV treatment costs for cash-strapped patients or healthcare systems. In addition, with expansion of early treatment, MDR detection might be more difficult as patients are asymptomatic and might not be monitored closely. Without close monitoring of resistance, patients could unknowingly develop MDR and thus delay initiation of second-line treatment. Therefore, the clinical significance of MDR may be greater with a larger ESH treatment prevalence. We agree with Gonzalez-Serna et al (and state in the main text) that test-and-treat is likely to bring epidemiologic benefits even with MDR growth, and we do not recommend abandoning this policy. However, Gonzalez-Serna et al downplay potential implications of MDR growth. We believe a prudent approach would be to evaluate the cost-effectiveness of test-and-treat compared to other policies, and if adoption of test-and-treat is warranted, it should be accompanied by initiatives to control and closely monitor MDR such as expanded MDR surveillance and interventions to improve adherence.