Recently, due to the fast development in multimedia applications of wireless communication systems and demands of increasing the capacity and data rates to meet the requirements of 5G mobile communication systems, new transmission techniques are needed. One of the powerful enabling tools of the 5G communication systems is the multi-input multi-output (MIMO) antenna system employing multiple antennas and utilizing the multipath fading mobile environment to increase the channel capacity without any additional bandwidth and/or transmitted power. However, because of the limited space of mobile devices, the distances between antennas are decreasing. The mutual coupling between antennas severely affects the overall performance of the MIMO system. So, isolation improvement methods are applied to reduce these influences. In this review, different isolation techniques and their features and drawbacks are presented, and the techniques for applying them in modern MIMO antenna systems are illustrated. Besides, the performance characteristics of these MIMO systems are compared. The review also contains basic concepts of the MIMO antenna system and its performance metrics. Finally, the future directions and trends for the MIMO antenna system design are discussed briefly. [ABSTRACT FROM AUTHOR]