Summary: Series compensation improves the power transfer capability and the stability of a transmission system. Inclusion of series compensating devices along with their protecting equipment introduces the problems to conventional protective relaying during power system faults. In this paper, a new protection strategy is proposed based on the instantaneous active and reactive power variations. The applicability of the proposed method to a series compensated transmission system is studied with numerous fault cases. Instantaneous power quantities are measured, and respective active and reactive power quantities are reproduced onto a two‐dimensional coordinated system with reactive power as ordinate and active power as abscissa. The locus of ordered pair of instantaneous power quantities (p(t), q(t)) forms a P‐Q trajectory. The quadrant in which the starting point of P‐Q trajectory (p0(t), q0(t)) lies is computed and taken as reference. This P‐Q portrait is analyzed to detect the fault depending on the condition that the operating point shifts its quadrant under fault condition. Fault will be classified based on the polarities of extracted powers. Classification tree is formed by a set of yes/no consequences to classify the fault. The proposed method is tested for different types of faults, at distinct locations and fault inceptions on 500‐kV, 50‐Hz test system. Results thus obtained show the potentiality of proposed strategy for the protection of series compensated transmission system concerning the speed, computational burden, and mathematical complexity. [ABSTRACT FROM AUTHOR]