Purpose – The purpose of this paper is to analyze the frame of a missile formation cooperative control system, and present an optimal keeping controller of a missile formation in the cooperative engagement. Design/methodology/approach – A missile relative motion model is established directly based on the kinematics relationships in the relative coordinated frame, following that is the detailed process of designing an optimal formation controller, which is analyzed through the small disturbance linearized method and transforming control variables method, respectively, these two methods both have themselves properties. The equations and control variables are intuitive during the linearized analysis, but errors brought by the linearized method are unavoidable, which will reduce the control precision. As for the transforming method, the control accuracy is greatly increased although the control form is a little complex, so in this paper the transforming control variable method is mainly researched to design an optimal formation controller. Considering the states of a leader as input perturbation variables, we design an optimal formation controller based on the linear quadric theory, which has quadric optimal performances of the missile flight states and control quantity. In order to obtain a higher accurate solution, the precise integration algorithm is introduced to solve the Riccati Equation that significantly affects the accuracy of an optimal control problem. Findings – The relative motion model established directly in the relative coordinate frame has intuitive physical significance, and the optimal controller based on this relative motion model is capable of restraining the invariable or slowly varying perturbation brought by the velocity of a leader and the input perturbations caused by the maneuver of the leader, at the same time this optimal controller can implement formation reconfiguration and keeping to an expected states rapidly, steadily and exactly; the steady errors can be greatly decreased by analyzing the relative motion model through transforming control variables method compared to the small disturbance linearized operation. Practical implications – The main frame of a missile formation cooperative engagement system can be found in this paper, which shows a clear structure and relations of each part of this complex system. The relations between each subsystem including the specific input and output variables can also be used to guide and restrict how to design each subsystem. The emphasis of this paper is on designing an optimal formation keeping controller which can overcome slowly varying or invariable perturbations and implement quadric optimal keeping control rapidly, stably and accurately. Originality/value – This paper provides a new method to analyze the missile relative motion model. The proposed proportional and integral (PI) optimal controller based on this model, and utilizing the Precise Integration Algorithm to solve this optimal controller are also new thoughts for formation control problems. [ABSTRACT FROM AUTHOR]