Search

We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e.g., photographs) and unlabeled data from a target domain (e.g., sketches) are used to learn a classifier for the target domain. Conventional UDA methods (e.g., domain adversarial training) learn domain-invariant features to improve generalization to the target domain. In this paper, we show that contrastive pre-training, which learns features on unlabeled source and target data and then fine-tunes on labeled source data, is competitive with strong UDA methods. However, we find that contrastive pre-training does not learn domain-invariant features, diverging from conventional UDA intuitions. We show theoretically that contrastive pre-training can learn features that vary subtantially across domains but still generalize to the target domain, by disentangling domain and class information. Our results suggest that domain invariance is not necessary for UDA. We empirically validate our theory on benchmark vision datasets., Comment: ICML 2022 (Long Talk)

When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer -- the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between ID and OOD accuracy arises even in a simple setting: fine-tuning overparameterized two-layer linear networks. We prove that the OOD error of fine-tuning is high when we initialize with a fixed or random head -- this is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine-tuning and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning)., Comment: ICLR (Oral) 2022

Consider a prediction setting with few in-distribution labeled examples and many unlabeled examples both in- and out-of-distribution (OOD). The goal is to learn a model which performs well both in-distribution and OOD. In these settings, auxiliary information is often cheaply available for every input. How should we best leverage this auxiliary information for the prediction task? Empirically across three image and time-series datasets, and theoretically in a multi-task linear regression setting, we show that (i) using auxiliary information as input features improves in-distribution error but can hurt OOD error; but (ii) using auxiliary information as outputs of auxiliary pre-training tasks improves OOD error. To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error., Comment: ICLR 2021

Enhancing motivation and learning attitudes in an introductory physics course is an important but difficult task that can be achieved through class blogging. We incorporated into an introductory course a blog operated by upper-level physics students. Using the Colorado Learning Attitudes about Science Survey (CLASS), periodic in-class surveys, analysis of student blog comments, and post-instructional interviews, we evaluate how the blog combined with class instruction provided the students with a better sense of relevance and confidence and outline recommendations for future use of this strategy., Comment: Submitted to American Journal of Physics

To determine if a relationship existed between physical skills and achievement in reading, mathematics, and listening comprehension of fifth-grade students, a study evaluated 334 fifth-graders in Starkville, Mississippi, on 20 physical fitness, motor fitness, and sports skills, as well as the Stanford Achievement Tests. The physical skills test battery included four measures of physical fitness (flexibility, muscular endurance, and muscular strength); eight measures of motor fitness (speed, agility, power, and dynamic and static balance); and eight items to assess the sports skills of baseball, basketball, soccer, badminton, and golf. The academic scores used were the total scores in reading, mathematics, and listening comprehension. Analysis revealed three physical measures--golf putting, left hand grip strength, and football kicking--which were common predictors for each of the three areas of academic achievement. The railwalk was a common predictor in the negative direction for each of the three areas. Findings support the view that results of relationship studies between academic ability and physical skills are conflicting. (Three tables of data are included.) (MM)