Adaptive speciation can occur when a population undergoes assortative mating and disruptive selection caused by frequency-dependent intraspecific competition. However, other interactions, such as mutualisms based on trait matching, may generate conflicting selective pressures that constrain species diversification. We used individual-based simulations to explore how different types of mutualism affect adaptive diversification. A magic trait was assumed to simultaneously mediate mate choice, intraspecific competition, and mutualisms. In scenarios of intimate, specialized mutualisms, individuals interact with one or few individual mutualistic partners, and diversification is constrained only if the mutualism is obligate. In other scenarios, increasing numbers of different partners per individual limit diversification by generating stabilizing selection. Stabilizing selection emerges from the greater likelihood of trait mismatches for rare, extreme phenotypes than for common intermediate phenotypes. Constraints on diversification imposed by increased numbers of partners decrease if the trait matching degree has smaller positive effects on fitness. These results hold after the relaxation of various assumptions. When trait matching matters, mutualism-generated stabilizing selection would thus often constrain diversification in obligate mutualisms, such as ant-myrmecophyte associations, and in low-intimacy mutualisms, including plant-seed disperser systems. Hence, different processes, such as trait convergence favoring the incorporation of nonrelated species, are needed to explain the higher richness of low-intimacy assemblages--shown here to be up to 1 order of magnitude richer than high-intimacy systems.