Working with imperfect machines: managing error while using ML in the field

Our vast and dynamic ocean is largely undersampled and its inhabitants poorly understood. Imaging systems, supported by machine learning-based automated analysis, promise to help us establish a more consistent presence in this difficult to access habitat. But on-going challenges with ML-systems keep the dream of an ocean teeming with robots capable of autonomously reporting observations or tracking rare individual organisms just out of reach. Indeed, the very signals we wish to measure – rapidly fluctuating, spatially and temporally heterogeneous biological distributions – confound supervised machine learning systems that are encoded with a static view of the world. Bridging the gap between these imperfect systems and deployable instrumentation requires supervised autonomy: keeping a human-in-the-loop to ensure the machine is doing the right thing. Here we describe two supervised autonomy experiments that push the limits of allowing in situ platforms to make decisions base on imagery. In the first, we couple a multi-object detector and a 3D stereo tracker on a Remotely Operated Vehicle to find and track midwater organisms while a human observer confirms the chosen target. In the second, we demonstrate transmitting selected Regions of Interest, chosen by a real time binary detector, from an Autonomous Underwater Vehicle at depth via an acoustic modem. Together, these experiments represent steps toward allowing a computer to trigger a behaviour change based on visual data to track a salient feature. Both fundamentally rely on a human operator to confirm the machine’s choice. Enabling this type of constructive interaction between potentially inaccurate machines and human supervisors is critical to ensure that our in situ systems operate in an efficient and unbiased manner.