Toward a common language to facilitate reproducible research and technology transfer: challenges and solutions

The keynote presentation from the 1st ACM conference on reproducibility and replicability (ACM REP'23) Please don't hesitate to provide your feedback via the public Discord serverfrom the MLCommons Task Force on Automation and Reproducibility and GitHub issues. [ GitHub project] [ Public Collective Knowledge repository ] [ Related reproducibility initiatives ] [ cTuning.org ] [ cKnowledge.org ] During the past 10 years, we have considerably improved the reproducibility of experimental results from published papers by introducing the artifact evaluation process with a unified artifact appendix and reproducibility checklists, Jupyter notebooks, containers, and Git repositories. On the other hand, our experience reproducing more than 200 papers shows that it can take weeks and months of painful and repetitive interactions between teams to reproduce artifacts. This effort includes decrypting numerous README files, examining ad-hoc artifacts and containers, and figuring out how to reproduce computational results. Furthermore, snapshot containers pose a challenge to optimize algorithms’ performance, accuracy, power consumption and operational costs across diverse and rapidly evolving software, hardware, and data used in the real world. In this talk, Iexplain how our practical artifact evaluation experience and the feedback from researchers and evaluators motivated us to develop a simple, intuitive, technology agnostic, and English-like scripting language called Collective Mind (CM). It helps to automatically adapt any given experiment to any software, hardware, and data while automatically generating unified README files and synthesizing modular containers with a unified API. It is being developed by MLCommons to facilitate reproducible AI/ML Systems research and minimizing manual and repetitive benchmarking and optimization efforts, reduce time and costs for reproducible research, and simplify technology transfer to production. I also present several recent use cases of how CM helps MLCommons, the Student Cluster Competition, and artifact evaluation at ACM/IEEE conferences. I conclude with our development plans, new challenges, possible solutions, and upcoming reproducibility and optimization challenges powered by the MLCommons Collective Knowledge platform and CM:access.cKnowledge.org.