Search

Fine-tuning contextualized representations learned by pre-trained language models remains a prevalent practice in NLP. However, fine-tuning can lead to representation degradation (also known as representation collapse), which may result in instability, sub-optimal performance, and weak generalization. In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain the information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations. We study the empirical behavior of the proposed regularization in comparison to 5 comparable baselines across 13 language understanding tasks (GLUE benchmark and six additional datasets). When evaluating in-domain performance, REPINA consistently outperforms other baselines on most tasks (10 out of 13). We also demonstrate its effectiveness in few-shot settings and robustness to label perturbation. As a by-product, we extend previous studies of representation collapse and propose several metrics to quantify it. Our empirical findings show that our approach is significantly more effective at mitigating representation collapse., Comment: 41 pages, 6 figures

Sepsis is a life-threatening disease with high morbidity, mortality and healthcare costs. The early prediction and administration of antibiotics and intravenous fluids is considered crucial for the treatment of sepsis and can save potentially millions of lives and billions in health care costs. Professional clinical care practitioners have proposed clinical criterion which aid in early detection of sepsis; however, performance of these criterion is often limited. Clinical text provides essential information to estimate the severity of the sepsis in addition to structured clinical data. In this study, we explore how clinical text can complement structured data towards early sepsis prediction task. In this paper, we propose multi modal model which incorporates both structured data in the form of patient measurements as well as textual notes on the patient. We employ state-of-the-art NLP models such as BERT and a highly specialized NLP model in Amazon Comprehend Medical to represent the text. On the MIMIC-III dataset containing records of ICU admissions, we show that by using these notes, one achieves an improvement of 6.07 points in a standard utility score for Sepsis prediction and 2.89% in AUROC score. Our methods significantly outperforms a clinical criteria suggested by experts, qSOFA, as well as the winning model of the PhysioNet Computing in Cardiology Challenge for predicting Sepsis.

In his paper Almost-Primes Represented by Quadratic Polynomials, Iwaniec proved that the polynomial n^2 + 1 takes on values with at most two prime factors (counted with multiplicity) infinitely often. He states that "in order to avoid technical complications, we shall restrict our proof to the polynomial n^2 + 1.". In this exposition, we follow Iwaniec's proof and show that for any irreducible quadratic polynomial G(n) (satisfying some obviously necessary hypotheses), G(n) has at most two prime factors for infinitely many values of n., Comment: an essay submitted in partial fulfillment of the requirements for the degree of Master of Science (2006)

We compare the asymptotic behavior of Carmichael's lambda function composed with Euler's totient function to the asymptotic behavior of Carmichael's lambda function composed with itself. We establish the normal order of the logarithm of the ratio., Comment: 18 pages, 2 figures