The present study focuses on the preparation of a novel hybrid epoxy nanocomposite with glycidyl polyhedral oligomeric silsesquioxane (POSS) as nanofiller, carboxyl terminated poly(acrylonitrile-co-butadiene) (CTBN) as modifying agent and diglycidyl ether of bisphenol A (DGEBA) as matrix polymer. The reaction between DGEBA, CTBN, and glycidyl POSS was carefully monitored and interpreted by using Fourier transform infrared (FTIR) and differential scanning calorimetry (DSC). An exclusive mechanism of the reaction between the modifier, nanofiller, and the matrix is proposed herein, which attempts to explains the chemistry behind the formation of an intricate network between POSS, CTBN, and DGEBA. The mechanical properties, such as tensile strength, and fracture toughness, were also carefully examined. The fracture toughness increases for epoxy/CTBN, epoxy/POSS, and epoxy/CTBN/POSS hybrid systems with respect to neat epoxy, but for hybrid composites toughening capability of soft rubber particles is lost by the presence of POSS. Field emission scanning electron micrographs (FESEM) of fractured surfaces were examined to understand the toughening mechanism. The viscoelastic properties of epoxy/CTBN, epoxy/POSS, and epoxy/CTBN/POSS hybrid systems were analyzed using dynamic mechanical thermal analysis (DMTA). The storage modulus shows a complex behavior for the epoxy/POSS composites due to the existence of lower and higher crosslink density sites. However, the storage modulus of the epoxy phase decreases with the addition of soft CTBN phase. The Tg corresponding to epoxy-rich phase was evident from the dynamic mechanical spectrum. For hybrid systems, the Tg is intermediate between the epoxy/rubber and epoxy/POSS systems. Finally, TGA (thermo gravimetric analysis) studies were employed to evaluate the thermal stability of prepared blends and composites. POLYM. COMPOS., 2015. © 2015 Society of Plastics Engineers