• Locate and classify broccoli heads based to its state of maturity. • The proposed system can be adapted, being able to work directly with natural light. • The cost of the required computational equipment is quite low for agriculture applications. Broccoli is a vegetable grown worldwide due to its good nutritional properties. The harvest of this product is done selectively by hand depending on their size and state of maturation for both fresh market and agri-food industry. The final aim of our work is the development of a machine that is able to automatically harvest only those broccoli heads that have the size and ripeness suitable for the agri-food industry, besides discarding those overripe or with diseases. One critical element in such a machine is a vision system that locates and classifies the broccoli heads present in photographic images, to trigger later a cutting module. In this paper, we present an approach to that vision system, based on deep learning techniques. The proposed algorithm, running in a relatively cheap hardware, is able to work in real time, locating broccoli heads in 640 × 480 px digital images, and classifying then into harvestable , immature and wasted classes. Tested with images taken in real conditions, with many heads partially hidden by leaves, the system was able to correctly locate and classify up to 97% of the cases presented in the test set. [ABSTRACT FROM AUTHOR]