This paper builds exact optical solitons to the coupled of nonlinear equations which describes the pulse propagation in birefringent optical fibers with the effects of ellipticity angle and coupling coefficient. By adopting the new Sardar sub-ODE method, diverse optical solitons as W-shaped bright, dark optical solitons, trigonometric function and singular function solutions have been secured. To show out the warranty of the stability of the acquired results, we examine the effects of the ellipticity angle and coupling coefficient on modulation instability. It is noted beside of the linear and circular birefringence, for θ = 45 o , it is exhibited unstable zone where MI growth rate. To enroll the effects of the coupling coefficient on MI growth rate, we fixed a value of the ellipticity angle θ = 45 o . In Figs. 6 and 10, we acquired some new behavior of the MI gain compared to Houwe et al. (2021). These results show out stable/unstable zones compared to Ablowitz and Horikis (2017), Houwe et al. (2021). Moreover, these results will be helpful in nonlinear optical fibers to explain the communication over long distance and optical fibers design. [ABSTRACT FROM AUTHOR]