Abstract: A new method for investigating mental fatigue based on P300 variability is presented here. In this approach a new coupled particle filtering for tracking variability of P300 subcomponents, i.e., P3a and P3b, across trials is developed. The latency, amplitude, and width of each subcomponent, as the main varying parameters, are modelled using state space system. In this model the observation is modelled as a linear function of amplitude and a nonlinear function of latency and width. Two Rao-blackwellised particle filters are then coupled and employed for recursive estimation of the state of the system across trials. By including some physiological based constraints, the proposed technique prevents generation of invalid particles during estimation of the state of the system. The main advantage of the algorithm compared with other single trial based methods is its robustness in the low signal-to-noise ratio situations. The method is applied to both simulated data and real mental fatigue data. The results demonstrate potential use of the method in event-related potential (ERP) based applications. [Copyright &y& Elsevier]