Partial up-up-down order with the continuously distributed order parameter in the triangular antiferromagnet TmMgGaO4

Frustrated quasidoublets without time-reversal symmetry can host highly unconventional magnetic structures with continuously distributed order parameters even in a single-phase crystal. Here, we report the comprehensive thermodynamic and neutron diffraction investigation on the single crystal of TmMgGaO$_4$, which entails non-Kramers Tm$^{3+}$ ions arranged on a geometrically perfect triangular lattice. The crystal electric field (CEF) randomness caused by the site-mixing disorder of the nonmagnetic Mg$^{2+}$ and Ga$^{3+}$ ions, merges two lowest-lying CEF singlets of Tm$^{3+}$ into a ground-state (GS) quasidoublet. Well below $T_c$ $\sim$ 0.7 K, a small fraction of the antiferromagnetically coupled Tm$^{3+}$ Ising quasidoublets with small inner gaps condense into two-dimensional (2D) up-up-down magnetic structures with continuously distributed order parameters, and give rise to the \emph{columnar} magnetic neutron reflections below $\mu_0H_c$ $\sim$ 2.6 T, with highly anisotropic correlation lengths, $\xi_{ab}$ $\geq$ 250$a$ in the triangular plane and $\xi_c$ $
Comment: Accepted in Physical Review X. The present manuscript is significantly different from the earlier preprint, arXiv:1804.00696, in the analysis, results, and contains additional neutron diffraction data