This study reviewed research on the effect of heterogeneous ability grouping in a cooperative learning environment on academic achievement in mathematics. A large number of studies has shown the importance of cooperative learning procedure in educational settings. The procedure include a teacher presentation followed by small-group interaction in which students work together in heterogeneous groups of four to six members. A variety of types of cooperative learning methods have been used in teaching many mathematics courses. The effect of homogeneity versus heterogeneity of group member ability on academic achievement in mathematics still remained unclear. Roots of the cooperative learning are examined. Six methods for conducting cooperative small-group learning in the classroom reviewed in this study. The six methods are: Student Team-Achievement Division (STAD), Teams-Games-Tournaments (TGT), Team Accelerated Instruction (TAI), Jigsaw II, Learning together, and Group investigation. Numerous studies have shown positive effects of cooperative learning on cognitive and affective outcomes in students. Much of the research is focused on achievement, generally identifying an advantage of cooperative learning settings over competitive settings. Some studies discussed effects of between- and within-class ability grouping on the achievement. Overall, findings of research have shown that ability grouping has harmful result on lower-ability and minority students and have not increased academic achievement. An alternative to ability grouping is cooperative learning method. The classes and groups of students should consist of learners of all ability levels. The most troublesome problems in mathematics instruction is heterogeneity in students preparation and learning rate. The findings of the research on the mathematics achievement effect of TAI, within-class ability grouping (AGAT), and whole-class instruction have shown positive effects on computation skills in TAI and AGAT, but no significant differences between both methods. Overall, the majority of findings of the research bases on comparison in student achievement while teaching mathematics in small-groups and in ability-grouped classrooms have shown no significant difference. When students assigned to work in mix-ability small groups during mathematics instruction high, middle, and low achievers demonstrate different levels of involvement. Low-ability students were less involved in group work than their high-achieving peers. Other studies have shown positive effect of heterogeneous grouping upon gifted students, but negative outcomes for non-gifted students. The research findings overall calls attention to the needs for additional studies on group composition during cooperative small-group instruction in mathematics.