Fulde-Ferrell-Larkin-Ovchinnikov states and quantum oscillations in mesoscopic superconductors and superfluid ultracold Fermi gases

11 pages, 5 figures, Submitted to PRA; International audience; We have studied the distinctive features of the Fulde-Ferrel-Larkin-Ovchinnikov (FFLO) instability and phase transitions in two--dimensional (2D) mesoscopic superconductors placed in magnetic field of arbitrary orientation and rotating superfluid Fermi gases with imbalanced state populations. Using a generalized version of the phenomenological Ginzburg-Landau theory we have shown that the FFLO states are strongly modified by the effect of the trapping potential confining the condensate. The phenomenon of the inhomogeneous state formation is determined by the interplay of three length scales: (i) length scale of the FFLO instability; (ii) 2D system size; (iii) length scale associated with the orbital effect caused either by the Fermi condensate rotation or magnetic field component applied perpendicular to the superconducting disc. We have studied this interplay and resulting quantum oscillation effects in both superconducting and superfluid finite -- size systems with FFLO instability and described the hallmarks of the FFLO phenomenon in a restricted geometry. The finite size of the system is shown to affect strongly the conditions of the observability of switching between the states with different vorticities.