The initial aim of the present paper is to provide a complete description of the eigenvalue problem for the non-commutative harmonic oscillator (NcHO), which is defined by a (two-by-two) matrix-valued self-adjoint parity-preserving ordinary differential operator [28], in terms of Heun's ordinary differential equations, the second-order Fuchsian differential equations with four regular singularities in a complex domain. This description has been achieved for odd eigenfunctions in Ochiai [25] nicely but missing up to now for the even parity, which is more important from the viewpoint of determination of the ground state of the NcHO. As a by-product of this study, examining the monodromy data (characteristic exponents, etc.) of the Heun equation, we prove that the multiplicity of the eigenvalue of the NcHO is at most two. Moreover, we give a condition for the existence of a finite-type eigenfunction (i.e., given by essentially a finite sum of Hermite functions) for the eigenvalue problem and an explicit example of such eigenvalues, from which one finds that doubly degenerate eigenstates of the NcHO actually exist even in the same parity. Also, we determine the possible shape of (so-called) Heun polynomial solutions of the Heun equations, which are obtained by the eigenvalue problem of the NcHO corresponding to finite-type eigenfunctions. Furthermore, as the second purpose of this paper, we discuss a connection between the quantum Rabi model [2, 15, 20, 41] and a certain element of the universal enveloping algebra U(sl2) of the Lie algebra sl2 naturally arising from the NcHO through the oscillator representation. Precisely, an equivalent picture of the quantum Rabi model drawn by a confluent Heun equation is obtained from the Heun operator defined by that element in U(sl2) under a (flat picture of non-unitary) principal series representation of sl2 through an appropriate confluent procedure. [ABSTRACT FROM AUTHOR]