Search

Automatic International Classification of Diseases (ICD) coding aims to assign multiple ICD codes to a medical note with average length of 3,000+ tokens. This task is challenging due to a high-dimensional space of multi-label assignment (tens of thousands of ICD codes) and the long-tail challenge: only a few codes (common diseases) are frequently assigned while most codes (rare diseases) are infrequently assigned. This study addresses the long-tail challenge by adapting a prompt-based fine-tuning technique with label semantics, which has been shown to be effective under few-shot setting. To further enhance the performance in medical domain, we propose a knowledge-enhanced longformer by injecting three domain-specific knowledge: hierarchy, synonym, and abbreviation with additional pretraining using contrastive learning. Experiments on MIMIC-III-full, a benchmark dataset of code assignment, show that our proposed method outperforms previous state-of-the-art method in 14.5% in marco F1 (from 10.3 to 11.8, P<0.001). To further test our model on few-shot setting, we created a new rare diseases coding dataset, MIMIC-III-rare50, on which our model improves marco F1 from 17.1 to 30.4 and micro F1 from 17.2 to 32.6 compared to previous method., Comment: Accepted by Findings of EMNLP 2022, code is available at https://github.com/whaleloops/KEPT

Suicide is an important public health concern and one of the leading causes of death worldwide. Suicidal behaviors, including suicide attempts (SA) and suicide ideations (SI), are leading risk factors for death by suicide. Information related to patients' previous and current SA and SI are frequently documented in the electronic health record (EHR) notes. Accurate detection of such documentation may help improve surveillance and predictions of patients' suicidal behaviors and alert medical professionals for suicide prevention efforts. In this study, we first built Suicide Attempt and Ideation Events (ScAN) dataset, a subset of the publicly available MIMIC III dataset spanning over 12k+ EHR notes with 19k+ annotated SA and SI events information. The annotations also contain attributes such as method of suicide attempt. We also provide a strong baseline model ScANER (Suicide Attempt and Ideation Events Retriever), a multi-task RoBERTa-based model with a retrieval module to extract all the relevant suicidal behavioral evidences from EHR notes of an hospital-stay and, and a prediction module to identify the type of suicidal behavior (SA and SI) concluded during the patient's stay at the hospital. ScANER achieved a macro-weighted F1-score of 0.83 for identifying suicidal behavioral evidences and a macro F1-score of 0.78 and 0.60 for classification of SA and SI for the patient's hospital-stay, respectively. ScAN and ScANER are publicly available., Comment: Paper accepted at NAACL 2022

Deep Neural Network (DNN) models have been shown to have high empirical privacy leakages. Clinical language models (CLMs) trained on clinical data have been used to improve performance in biomedical natural language processing tasks. In this work, we investigate the risks of training-data leakage through white-box or black-box access to CLMs. We design and employ membership inference attacks to estimate the empirical privacy leaks for model architectures like BERT and GPT2. We show that membership inference attacks on CLMs lead to non-trivial privacy leakages of up to 7%. Our results show that smaller models have lower empirical privacy leakages than larger ones, and masked LMs have lower leakages than auto-regressive LMs. We further show that differentially private CLMs can have improved model utility on clinical domain while ensuring low empirical privacy leakage. Lastly, we also study the effects of group-level membership inference and disease rarity on CLM privacy leakages.

Multi-phase materials, such as composite materials, exhibit multiple competing failure mechanisms during the growth of a macroscopic defect. For the simulation of the overall fracture process in such materials, we develop a two-phase spring network model that accounts for the architecture between the different components as well as the respective disorders in their failure characteristics. In the specific case of a plain weave architecture, we show that any offset between the layers reduces the delocalization of the stresses at the crack tip and thereby substantially lowers the strength and fracture toughness of the overall laminate. The avalanche statistics of the broken springs do not show a distinguishable dependence on the offsets between layers. The power law exponents are found to be much smaller than that of disordered spring network models in the absence of a crack. A discussion is developed on the possibility of the avalanche statistics being those near breakdown., Comment: 13 pages, 22 figures

In social-media platforms such as Twitter, Facebook, and Reddit, people prefer to use code-mixed language such as Spanish-English, Hindi-English to express their opinions. In this paper, we describe different models we used, using the external dataset to train embeddings, ensembling methods for Sentimix, and OffensEval tasks. The use of pre-trained embeddings usually helps in multiple tasks such as sentence classification, and machine translation. In this experiment, we haveused our trained code-mixed embeddings and twitter pre-trained embeddings to SemEval tasks. We evaluate our models on macro F1-score, precision, accuracy, and recall on the datasets. We intend to show that hyper-parameter tuning and data pre-processing steps help a lot in improving the scores. In our experiments, we are able to achieve 0.886 F1-Macro on OffenEval Greek language subtask post-evaluation, whereas the highest is 0.852 during the Evaluation Period. We stood third in Spanglish competition with our best F1-score of 0.756. Codalab username is asking28., Comment: Changed title and few more changes. This version will be published in SemEval2020. Added code Link

Conversational Machine Comprehension (CMC), a research track in conversational AI, expects the machine to understand an open-domain natural language text and thereafter engage in a multi-turn conversation to answer questions related to the text. While most of the research in Machine Reading Comprehension (MRC) revolves around single-turn question answering (QA), multi-turn CMC has recently gained prominence, thanks to the advancement in natural language understanding via neural language models such as BERT and the introduction of large-scale conversational datasets such as CoQA and QuAC. The rise in interest has, however, led to a flurry of concurrent publications, each with a different yet structurally similar modeling approach and an inconsistent view of the surrounding literature. With the volume of model submissions to conversational datasets increasing every year, there exists a need to consolidate the scattered knowledge in this domain to streamline future research. This literature review attempts at providing a holistic overview of CMC with an emphasis on the common trends across recently published models, specifically in their approach to tackling conversational history. The review synthesizes a generic framework for CMC models while highlighting the differences in recent approaches and intends to serve as a compendium of CMC for future researchers., Comment: Accepted to COLING 2020

We explore state-of-the-art neural models for question answering on electronic medical records and improve their ability to generalize better on previously unseen (paraphrased) questions at test time. We enable this by learning to predict logical forms as an auxiliary task along with the main task of answer span detection. The predicted logical forms also serve as a rationale for the answer. Further, we also incorporate medical entity information in these models via the ERNIE architecture. We train our models on the large-scale emrQA dataset and observe that our multi-task entity-enriched models generalize to paraphrased questions ~5% better than the baseline BERT model.

Pre-trained language models (LM) such as BERT, DistilBERT, and RoBERTa can be tuned for different domains (domain-tuning) by continuing the pre-training phase on a new target domain corpus. This simple domain tuning (SDT) technique has been widely used to create domain-tuned models such as BioBERT, SciBERT and ClinicalBERT. However, during the pretraining phase on the target domain, the LM models may catastrophically forget the patterns learned from their source domain. In this work, we study the effects of catastrophic forgetting on domain-tuned LM models and investigate methods that mitigate its negative effects. We propose continual learning (CL) based alternatives for SDT, that aim to reduce catastrophic forgetting. We show that these methods may increase the performance of LM models on downstream target domain tasks. Additionally, we also show that constraining the LM model from forgetting the source domain leads to downstream task models that are more robust to domain shifts. We analyze the computational cost of using our proposed CL methods and provide recommendations for computationally lightweight and effective CL domain-tuning procedures., Comment: Updated from a previous shorter version

We consider the problem of energy management in microgrid networks. A microgrid is capable of generating a limited amount of energy from a renewable resource and is responsible for handling the demands of its dedicated customers. Owing to the variable nature of renewable generation and the demands of the customers, it becomes imperative that each microgrid optimally manages its energy. This involves intelligently scheduling the demands at the customer side, selling (when there is a surplus) and buying (when there is a deficit) the power from its neighboring microgrids depending on its current and future needs. Typically, the transaction of power among the microgrids happens at a pre-decided price by the central grid. In this work, we formulate the problems of demand and battery scheduling, energy trading and dynamic pricing (where we allow the microgrids to decide the price of the transaction depending on their current configuration of demand and renewable energy) in the framework of stochastic games. Subsequently, we propose a novel approach that makes use of independent learners Deep Q-learning algorithm to solve this problem. Through extensive empirical evaluation, we show that our proposed framework is more beneficial to the majority of the microgrids and we provide a detailed analysis of the results.

We describe our approach towards building an efficient predictive model to detect emotions for a group of people in an image. We have proposed that training a Convolutional Neural Network (CNN) model on the emotion heatmaps extracted from the image, outperforms a CNN model trained entirely on the raw images. The comparison of the models have been done on a recently published dataset of Emotion Recognition in the Wild (EmotiW) challenge, 2017. The proposed method achieved validation accuracy of 55.23% which is 2.44% above the baseline accuracy, provided by the EmotiW organizers., Comment: This research paper has been accepted at Workshop on Computer Vision for Active and Assisted Living, WACV 2018

Perturb and Combine (P&C) group of methods generate multiple versions of the predictor by perturbing the training set or construction and then combining them into a single predictor (Breiman, 1996b). The motive is to improve the accuracy in unstable classification and regression methods. One of the most well known method in this group is Bagging. Arcing or Adaptive Resampling and Combining methods like AdaBoost are smarter variants of P&C methods. In this extended abstract, we lay the groundwork for a new family of methods under the P&C umbrella, known as Evolutionary Sampling (ES). We employ Evolutionary algorithms to suggest smarter sampling in both the feature space (sub-spaces) as well as training samples. We discuss multiple fitness functions to assess ensembles and empirically compare our performance against randomized sampling of training data and feature sub-spaces., Comment: 3 pages, 1 table, Data Efficient Machine Learning Workshop (DEML'16), ICML

The Chiral d-density wave (CDDW) order, at the anti-ferromagnetic wave vector (pi,pi), is assumed to represent the pseudo-gap (PG) state of a hole-doped cuprate superconductor. The pairing interaction required for the PG phase corresponds to a repulsive interaction of Coulombic origin. The d-wave superconductivity (DSC), driven by an appropriate assumed attractive interaction, is discussed within the mean-field framework together with the CDDW ordering. The single-particle excitation spectrum in the CDDW and DSC state is characterized by the Bogolubov quasi-particle bands-a characteristic feature of SC state. The coupled gap equations are solved self-consistently together with the equation to determine the chemical potential. With the pinning of the van Hove-singularities close to the chemical potential, one is able to calculate the thermodynamic and transport properties of the under-doped cuprates in a consistent manner. The electronic specific heat displays non-Fermi liquid feature in the CDDW state. The CDDW and DSC are found to represent two competing orders as the former brings about a depletion of the spectral weight (and Raman response function density) available for pairing in the anti-nodal region of momentum space.This is not in agreement with a preformed pairing scenario and justifies the different structures we have assumed for the pseudo-gap and the superconducting gap. Furthermore, these gaps in the excitation spectrum do not merge into a single quadrature gap. It is also shown that the depletion of the spectral weight below Tc at energies larger than the gap amplitude occurs. This is an indication of the strong-coupling superconductivity in cuprates. The calculation of the ratio of the quasi-particle thermal conductivity and temperature in the superconductiong phase is found to be constant in the limit of near-zero quasi-particle scattering rate., Comment: 17 pages, 8 figures. arXiv admin note: substantial text overlap with arXiv:1111.0928. This article is withdrawn due to a preexisting version online

We employ the Green's function technique to investigate the vacancy-induced quasi-localized magnetic moment formation in mono-layer graphene starting with the Dirac Hamiltonian, which focuses on the {\pi}- orbitals only, involving the nearest neighbor(NN)(t) and moderate second neighbor(SN)(t' < t/3) hopping integrals. The vacancy defect is modeled by the addition of the on-site perturbation potential to the Hamiltonian. We find that, when (t'/t) << 1, the vacancy induced {\pi}-state at the zero of energy(zero-mode state(ZMS)) does not inhabit the minority sub-lattice due to the strong scalar potential induced by the vacancy(the ZMSs get lodged in the majority sub-lattice) whereas, when (t'/t) is increased, the ZMS is somewhat suppressed. This shows that, not only the shift of the Fermi energy away from the linearly-dispersive Dirac points, the issue of this topological localization is also hinged on the ratio (t'/t). Furthermore, when a vacancy is present, the three sp2- hybridized {\sigma} states of each of the three nearest-neighbor carbon atoms, forming a carbon triangle surrounding the vacancy, are close to the Fermi energy (EF). The Hund's coupling between these {\sigma} electrons and the remaining electron which occupies the {\pi} state spin polarizes the {\pi} state leading to local moment formation close to EF. Since the system at the Fermi level has low electronic density, there is poor screening of such magnetic moments. This may lead to a high Curie temperature for such vacancy-induced moments., Comment: 13 pages, 4 figures

We calculate the electronic viscosity for a multiple quantum well structure in the presence of disorder potential(V = 4 meV), the electron-electron repulsion(U = 5-17 meV) and a strong magnetic field (B greater than or equal to 16.5 T) in the direction in which the electrons are trapped. This viscosity is different from the dissipation-less Hall viscosity which cannot take non-zero value in a time reversal invariant system. The Fermi energy density of states for the system has been calculated in the t-matrix approximation assuming low concentration of impurities. Our approach involves calculation of the density of viscosity, for temperature close to 0 K, on the Brillouin zone(BZ) followed by the numerical evaluation of the integral of viscosity density over the BZ. We show that (i) the viscosity coefficient is nearly proportional to B for given (V,U),and (ii) dissipation-less state, analogous to superfluidity, is possible for a critical value of U when (V, B) are given. We also calculate the entropy per particle (S) and show that the results comply with the KSS bound reported by Kovtun et al., Comment: 15 pages, 3 figures