Concentrating solar power plants are a clean energy source capable of competitive electricity generation even during night time, as well as the production of carbon-neutral fuels, offering a complementary role alongside photovoltaic plants. In these power plants, thousands of mirrors (heliostats) redirect sunlight onto a receiver, potentially generating temperatures exceeding 1000°C. Practically, such efficient temperatures are never attained. Several unknown, yet operationally crucial parameters, e.g., misalignment in sun-tracking and surface deformations can cause dangerous temperature spikes, necessitating high safety margins. For competitive levelized cost of energy and large-scale deployment, in-situ error measurements are an essential, yet unattained factor. To tackle this, we introduce a differentiable ray tracing machine learning approach that can derive the irradiance distribution of heliostats in a data-driven manner from a small number of calibration images already collected in most solar towers. By applying gradient-based optimization and a learning non-uniform rational B-spline heliostat model, our approach is able to determine sub-millimeter imperfections in a real-world setting and predict heliostat-specific irradiance profiles, exceeding the precision of the state-of-the-art and establishing full automatization. The new optimization pipeline enables concurrent training of physical and data-driven models, representing a pioneering effort in unifying both paradigms for concentrating solar power plants and can be a blueprint for other domains., (© 2024. The Author(s).)