Search

"Gold" and "ground truth" human-mediated labels have error. The effects of this error can escape commonly reported metrics of label quality or obscure questions of accuracy, bias, fairness, and usefulness during model evaluation. This study demonstrates methods for answering such questions even in the context of very low reliabilities from expert humans. We analyze human labels, GPT model ratings, and transformer encoder model annotations describing the quality of classroom teaching, an important, expensive, and currently only human task. We answer the question of whether such a task can be automated using two Large Language Model (LLM) architecture families--encoders and GPT decoders, using novel approaches to evaluating label quality across six dimensions: Concordance, Confidence, Validity, Bias, Fairness, and Helpfulness. First, we demonstrate that using standard metrics in the presence of poor labels can mask both label and model quality: the encoder family of models achieve state-of-the-art, even "super-human", results across all classroom annotation tasks. But not all these positive results remain after using more rigorous evaluation measures which reveal spurious correlations and nonrandom racial biases across models and humans. This study then expands these methods to estimate how model use would change to human label quality if models were used in a human-in-the-loop context, finding that the variance captured in GPT model labels would worsen reliabilities for humans influenced by these models. We identify areas where some LLMs, within the generalizability of the current data, could improve the quality of expensive human ratings of classroom instruction., Comment: 20 pages, 15 figures, 58 pages with references and appendices

How many permutations of the natural numbers are needed so that every conditionally convergent series of real numbers can be rearranged to no longer converge to the same sum? We define the \emph{rearrangement number}, a new cardinal characteristic of the continuum, as the answer to this question. We compare the rearrangement number with several natural variants, for example one obtained by requiring the rearranged series to still converge but to a new, finite limit. We also compare the rearrangement number with several well-studied cardinal characteristics of the continuum. We present some new forcing constructions designed to add permutations that rearrange series from the ground model in particular ways, thereby obtaining consistency results going beyond those that follow from comparisons with familiar cardinal characteristics. Finally, we deal briefly with some variants concerning rearrangements by a special sort of permutation and with rearranging some divergent series to become (conditionally) convergent.

This paper highlights the results of an investigation of preservice elementary teachers' perceptions of their ability and preparation to teach via technology. Results indicated that participants tended to perceive themselves as capable of teaching via technology. However, they had difficulty identifying specific technological resources for teaching. Further, participants tended to be less positive in their perceptions of both the extent to which their degree program prepared them to teach via technology and the extent to which their instructors modeled the effective use of technology. Accordingly, modifications to teacher education programs may be warranted and some suggested alterations are provided. (Contains 11 references.) (Author/SM)

The natural forms of the Leibniz rule for the $k$th derivative of a product and of Fa\`a di Bruno's formula for the $k$th derivative of a composition involve the differential operator $\partial^k/\partial x_1 ... \partial x_k$ rather than $d^k/dx^k$, with no assumptions about whether the variables $x_1,...,x_k$ are all distinct, or all identical, or partitioned into several distinguishable classes of indistinguishable variables. Coefficients appearing in forms of these identities in which some variables are indistinguishable are just multiplicities of indistinguishable terms (in particular, if all variables are distinct then all coefficients are 1). The computation of the multiplicities in this generalization of Fa\`a di Bruno's formula is a combinatorial enumeration problem that, although completely elementary, seems to have been neglected. We apply the results to cumulants of probability distributions., Comment: 13 pages

We give a visually appealing counterexample to the proposition that unbiased estimators are better than biased estimators., Comment: 6 pages, LaTeX, To appear in the American Mathematical Monthly

Scaled Boolean algebras are a category of mathematical objects that arose from attempts to understand why the conventional rules of probability should hold when probabilities are construed, not as frequencies or proportions or the like, but rather as degrees of belief in uncertain propositions. This paper separates the study of these objects from that not-entirely-mathematical problem that motivated them. That motivating problem is explicated in the first section, and the application of scaled Boolean algebras to it is explained in the last section. The intermediate sections deal only with the mathematics. It is hoped that this isolation of the mathematics from the motivating problem makes the mathematics clearer., Comment: 53 pages, 8 Postscript figures, Uses ajour.sty from Academic Press, To appear in Advances in Applied Mathematics