Comment on 'Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics'

A recent paper evaluating a new rubric-based graduate admissions approach using generic methods tentatively suggested that its decisions differed noticeably from the previous approach in an unspecified way. Using prior knowledge that the often-stated specific goal was to open a path to increased diversity by reducing barriers to admission of applicants with low undergraduate grade point averages and GRE scores allows simple statistical tests of changes in the distributions of the metrics. The simple tests confirm with good statistical confidence that the barrier-reduction changes were achieved. Nevertheless, the paper's argument that the de-emphasized test and grades are not predictive of graduate outcomes is not supported by the prior literature. On technique, although a method used in some of the analysis for dropping data points before running the machine algorithm is likely to bias those results, it helps to clarify why models of the rubric-based system were only weakly predictive.
Comment: about 2700 words, should appear in Phys. Rev. Phys. Educ. Res. once the authors of the original paper provide a Response