Evolutionarily stable strategies in stable and periodically fluctuating populations: The Rosenzweig-MacArthur predator-prey model

Significance Many evolutionary studies of ecological systems assume, explicitly or implicitly, ecologically stable population dynamics. Ecological analyses typically assume, on the other hand, no evolution. We study a model (using predator–prey dynamics as an example) combining ecology and evolution within the same framework. For this purpose, we use the evolutionarily stable strategies (ESSs) framework, emphasizing that evolutionary change, in general, will occur as a result of mutant strategies being able to invade a population. The significance of our contribution is to derive mathematical conditions for the existence of an ESS in a periodically limit-cycle ecological system.
An evolutionarily stable strategy (ESS) is an evolutionary strategy that, if adapted by a population, cannot be invaded by any deviating (mutant) strategy. The concept of ESS has been extensively studied and widely applied in ecology and evolutionary biology [M. Smith, On Evolution (1972)] but typically on the assumption that the system is ecologically stable. With reference to a Rosenzweig–MacArthur predator–prey model [M. Rosenzweig, R. MacArthur, Am. Nat. 97, 209–223 (1963)], we derive the mathematical conditions for the existence of an ESS when the ecological dynamics have asymptotically stable limit points as well as limit cycles. By extending the framework of Reed and Stenseth [J. Reed, N. C. Stenseth, J. Theoret. Biol. 108, 491–508 (1984)], we find that ESSs occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. These functions are identified and shown to have a similar form for both stable and fluctuating populations. We illustrate these results with a concrete example.