Comparison of Automated Crater Catalogs for Mars From Benedix et al. (2020) and Lee and Hogan (2021).

Crater mapping using neural networks and other automated methods has increased recently with automated Crater Detection Algorithms (CDAs) applied to planetary bodies throughout the solar system. A recent publication by Benedix et al. (2020, https://doi.org/10.1029/2019ea001005) showed high performance at small scales compared to similar automated CDAs but with a net positive diameter bias in many crater candidates. I compare the publicly available catalogs from Benedix et al. (2020, https://doi.org/10.1029/2019ea001005) and Lee and Hogan (2021, https://doi.org/10.1016/j.cageo.2020.104645) and show that the reported performance is sensitive to the metrics used to test the catalogs. I show how the more permissive comparison methods indicate a higher CDA performance by allowing worse candidate craters to match ground‐truth craters. I show that the Benedix et al. (2020, https://doi.org/10.1029/2019ea001005) catalog has a substantial performance loss with increasing latitude and identify an image projection issue that might cause this loss. Finally, I suggest future applications of neural networks in generating large scientific datasets be validated using secondary networks with independent data sources or training methods. Plain Language Summary: I have compared two computer programs that use neural networks to identify craters on solar system bodies. The crater catalogs created by these programs have been previously compared against an existing human‐made catalog to measure their performance, but each comparison used a different method making it challenging to understand the difference between the catalogs. In this paper, I use the comparison methods from these two independent papers, discuss where the catalogs agree and disagree, and suggest why they disagree. I emphasize the need for more accurate and consistent methods to validate automatic crater‐mapping programs, especially when it is impractical for humans to check the craters found by automated methods. Key Points: The apparent performance of automatic crater detection algorithms is sensitive to the choice of metric used to gauge the performanceFeature Detection Algorithms trained to find spatial features should be used with appropriately projected images where possibleAs neural networks extend crater detection beyond practical human limits, validating their results with a independent networks is critical [ABSTRACT FROM AUTHOR]