Shape of population interfaces as an indicator of mutational instability in coexisting cell populations

Cellular populations such as avascular tumors and microbial biofilms may "invade" or grow into surrounding populations. The invading population is often comprised of a heterogeneous mixture of cells with varying growth rates. The population may also exhibit mutational instabilities, such as a heavy deleterious mutation load in a cancerous growth. We study the dynamics of a heterogeneous, mutating population competing with a surrounding homogeneous population, as one might find in a cancerous invasion of healthy tissue. We find that the shape of the population interface serves as an indicator for the evolutionary dynamics within the heterogeneous population. In particular, invasion front undulations become enhanced when the invading population is near a mutational meltdown transition or when the surrounding "bystander" population is barely able to reinvade the mutating population. We characterize these interface undulations and the effective fitness of the heterogeneous population in one- and two-dimensional systems.