Mixture modeling for the detection of subpopulations in a pharmacokinetic/pharmacodynamic analysis

To be able to estimate accurately parameters entering a non-linear mixed effects model taking into account that one or more subpopulations of patients can exist rather than assuming that the entire population is best described by unimodal distributions for the random effects, we proposed a methodology based on the likelihood approximation using the Gauss–Hermite quadrature. The idea is to combine the estimation of the model parameters and the detection of homogeneous subgroups of patients in a given population using a Gaussian mixture for the distribution of the random effects. As the accuracy of the likelihood approximation is likely to govern the quality of the estimation of the different parameters entering the nonlinear mixed effects model, we based this approximation on the use of an adjustable Gauss– Hermite quadrature. Moreover, to complete this methodology, we propose a strategy allowing the detection and explanation of heterogeneity based on the Kullback–Leibler test, which was used to estimate the number of components in the Gaussian mixture. In order to evaluate the capability of the method to take into account heterogeneity, this strategy was performed in a PK/PD analysis using the database and the structural model selected in a previous analysis. In this analysis, non-responders were found out using NONMEM [Beal and Sheiner. NONMEM Users Guides. NONMEM Project Group, University of California, San Francisio, 1992] in a population of diabetic patients treated with a once-a-day new formulation of an antidiabetic drug. The authors looked for a subpopulation of patients for whom the therapeutic effect would vanish. In this paper, we looked for subpopulations of patients exhibiting specificities with respect to different parameters entering the description of the effect. The results obtained with our approach are compared in terms of parameter estimation and heterogeneity detection to those obtained in the previous analysis.