Search

FOLD-RM is an explainable machine learning classification algorithm that uses training data to create a set of classification rules. In this paper we introduce CON-FOLD which extends FOLD-RM in several ways. CON-FOLD assigns probability-based confidence scores to rules learned for a classification task. This allows users to know how confident they should be in a prediction made by the model. We present a confidence-based pruning algorithm that uses the unique structure of FOLD-RM rules to efficiently prune rules and prevent overfitting. Furthermore, CON-FOLD enables the user to provide pre-existing knowledge in the form of logic program rules that are either (fixed) background knowledge or (modifiable) initial rule candidates. The paper describes our method in detail and reports on practical experiments. We demonstrate the performance of the algorithm on benchmark datasets from the UCI Machine Learning Repository. For that, we introduce a new metric, Inverse Brier Score, to evaluate the accuracy of the produced confidence scores. Finally we apply this extension to a real world example that requires explainability: marking of student responses to a short answer question from the Australian Physics Olympiad.

In this paper we demonstrate how logic programming systems and Automated first-order logic Theorem Provers (ATPs) can improve the accuracy of Large Language Models (LLMs) for logical reasoning tasks where the baseline performance is given by direct LLM solutions. We first evaluate LLM reasoning on steamroller problems using the PRONTOQA benchmark. We show how accuracy can be improved with a neuro-symbolic architecture where the LLM acts solely as a front-end for translating a given problem into a formal logic language and an automated reasoning engine is called for solving it. However, this approach critically hinges on the correctness of the LLM translation. To assess this translation correctness, we secondly define a framework of syntactic and semantic error categories. We implemented the framework and used it to identify errors that LLMs make in the benchmark domain. Based on these findings, we thirdly extended our method with capabilities for automatically correcting syntactic and semantic errors. For semantic error correction we integrate first-order logic ATPs, which is our main and novel contribution. We demonstrate that this approach reduces semantic errors significantly and further increases the accurracy of LLM logical reasoning.

This study presents the first examination of the ability of Large Language Models (LLMs) to follow reasoning strategies that are used to guide Automated Theorem Provers (ATPs). We evaluate the performance of GPT4, GPT3.5 Turbo and Google's recent Gemini model on problems from a steamroller domain. In addition to determining accuracy we make use of the Natural Language Processing library spaCy to explore new methods of investigating LLM's reasoning capabilities. This led to one alarming result, the low correlation between correct reasoning and correct answers for any of the tested models. We found that the models' performance when using the ATP reasoning strategies was comparable to one-shot chain of thought and observe that attention to uncertainty in the accuracy results is critical when drawing conclusions about model performance. Consistent with previous speculation we confirm that LLMs have a preference for, and are best able to follow, bottom up reasoning processes. However, the reasoning strategies can still be beneficial for deriving small and relevant sets of formulas for external processing by a trusted inference engine.

We report on progress towards the development of an Action Concept Inventory (ACI), a test that measures student understanding of action principles in introductory mechanics and optics. The ACI also covers key concepts of many-paths quantum mechanics, from which classical action physics arises. We used a multi-stage iterative development cycle for incorporating expert and student feedback into successive revisions of the ACI. The student feedback, including think-aloud interviews, enabled us to identify their misconceptions about action physics., Comment: 6 pages. This paper started out as arXiv:1507.06075v1, then split into arXiv:1507.06075v2 and this paper as a result of feedback from referees and an editor. arXiv:1507.06075v2 presents the case for teaching action physics, while this paper presents an evaluation tool

More than a decade ago Edwin Taylor issued a "call to action" that presented the case for basing introductory university mechanics teaching around the principle of stationary action. We report on our response to that call in the form of an investigation of the teaching and learning of the stationary action formulation of physics in a first-year university course. Our action physics instruction proceeded from the many-paths approach to quantum physics through to ray optics, classical mechanics, and relativity. Despite the challenges presented by action physics, students reported it to be accessible, interesting, motivational and valuable., Comment: 13 pages, 1 figure. Substantially revised version accepted by the American Journal of Physics. Details of the Action Concept Inventory are now in a separate paper