In assisted reproductive technology (ART), embryos produced by in vitro fertilization (IVF) are graded according to their live birth potential, and high-grade embryos are preferentially transplanted. However, the rate of live birth following clinical ART remains low worldwide, suggesting that grading is inaccurate. One explanation is that grading is classically based on the characteristic shape of embryos at a limited number of developmental stages and does not consider the shape of embryos and intracellular structures, e.g., nuclei, at various stages important for normal embryogenesis. Therefore, here we developed a Normalized Multi-View Attention Network (NVAN) that directly predicts live birth potential from nuclear structural features in live-cell fluorescence images taken of mouse embryos across a wide range of stages. The classification accuracy of our method was 83.87%, which greatly exceeded that of existing machine-learning methods and that of visual inspection by embryo culture specialists. By visualizing the features that contributed most to the prediction of live birth potential, we found that the size and shape of the cell nucleus at the morula stage and at the time of cell division were important for live birth prediction. We anticipate that our method will help ART and developmental engineering as a new basic technology for IVF embryo selection.