We study the convergence of the transport plans $\gamma_\epsilon$ towards $\gamma_0$ as well as the cost of the entropy-regularized optimal transport $(c,\gamma_\epsilon)$ towards $(c,\gamma_0)$ as the regularization parameter $\epsilon$ vanishes in the setting of finite entropy marginals. We show that under the assumption of infinitesimally twisted cost and compactly supported marginals the distance $W_2(\gamma_\epsilon,\gamma_0)$ is asymptotically greater than $C\sqrt{\epsilon}$ and the suboptimality $(c,\gamma_\epsilon)-(c,\gamma_0)$ is of order $\epsilon$. In the quadratic cost case the compactness assumption is relaxed into a moment of order $2+\delta$ assumption. Moreover, in the case of a Lipschitz transport map for the non-regularized problem, the distance $W_2(\gamma_\epsilon,\gamma_0)$ converges to $0$ at rate $\sqrt{\epsilon}$. Finally, if in addition the marginals have finite Fisher information, we prove $(c,\gamma_\epsilon)-(c,\gamma_0) \sim d\epsilon/2$ and we provide a companion expansion of $H(\gamma_\epsilon)$. These results are achieved by disentangling the role of the cost and the entropy in the regularized problem.