Search

Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance. Previous studies have shown that DWT can be effective in the vision domain and natural language processing (NLP) pre-training stage. Specifically, DWT shows promise in practical scenarios, such as enhancing new generation or larger models using pre-trained yet older or smaller models and lacking a resource budget. However, the optimal conditions for using DWT have yet to be fully investigated in NLP pre-training. Therefore, this study examines three key factors to optimize DWT, distinct from those used in the vision domain or traditional knowledge distillation. These factors are: (i) the impact of teacher model quality on DWT effectiveness, (ii) guidelines for adjusting the weighting value for DWT loss, and (iii) the impact of parameter remapping as a student model initialization technique for DWT., Findings of ACL 2023

Neural Architecture Search (NAS) has emerged as a powerful technique for automating neural architecture design. However, existing NAS methods either require an excessive amount of time for repetitive training or sampling of many task-irrelevant architectures. Moreover, they lack generalization across different tasks and usually require searching for optimal architectures for each task from scratch without reusing the knowledge from the previous NAS tasks. To tackle such limitations of existing NAS methods, we propose a novel transferable task-guided Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. With the guidance of a surrogate model, such as a performance predictor for a given task, our DiffusionNAG can generate task-optimal architectures for diverse tasks, including unseen tasks. DiffusionNAG is highly efficient as it generates task-optimal neural architectures by leveraging the prior knowledge obtained from the previous tasks and neural architecture distribution. Furthermore, we introduce a score network to ensure the generation of valid architectures represented as directed acyclic graphs, unlike existing graph generative models that focus on generating undirected graphs. Extensive experiments demonstrate that DiffusionNAG significantly outperforms the state-of-the-art transferable NAG model in architecture generation quality, as well as previous NAS methods on four computer vision datasets with largely reduced computational cost.

This is the second study of the "Stereotypes, Common Ingroup Identity, and Outgroup Admiration" project. This study aims to replicate and extend the survey results using experimental methods.

Distillation-aware Neural Architecture Search (DaNAS) aims to search for an optimal student architecture that obtains the best performance and/or efficiency when distilling the knowledge from a given teacher model. Previous DaNAS methods have mostly tackled the search for the neural architecture for fixed datasets and the teacher, which are not generalized well on a new task consisting of an unseen dataset and an unseen teacher, thus need to perform a costly search for any new combination of the datasets and the teachers. For standard NAS tasks without KD, meta-learning-based computationally efficient NAS methods have been proposed, which learn the generalized search process over multiple tasks (datasets) and transfer the knowledge obtained over those tasks to a new task. However, since they assume learning from scratch without KD from a teacher, they might not be ideal for DaNAS scenarios. To eliminate the excessive computational cost of DaNAS methods and the sub-optimality of rapid NAS methods, we propose a distillation-aware meta accuracy prediction model, DaSS (Distillation-aware Student Search), which can predict a given architecture's final performances on a dataset when performing KD with a given teacher, without having actually to train it on the target task. The experimental results demonstrate that our proposed meta-prediction model successfully generalizes to multiple unseen datasets for DaNAS tasks, largely outperforming existing meta-NAS methods and rapid NAS baselines. Code is available at https://github.com/CownowAn/DaSS, Comment: ICLR 2023 (Notable-top-25%)

Additional file 1: Figure S1. The changes in AUROC with an increase in the number of features: (a) the classification of MCI-AD and CN and (b) the classification of MCI and CN. Figure S2. SHAP values from all 11 features: (a) the classification of MCI-AD and CN and (b) the classification of MCI and CN. Table S1. Baseline characteristics of participants at enrollment (additional trial n = 36). Table S2. The changes in accuracy with an increase in the number of features were examined in the MCI-AD dementia and CN groups. Table S3. The changes in accuracy with an increase in the number of features were examined in the MCI and CN groups. Table S4. Results of stacking techniques applied to ensemble-based models for MCI-AD and CN classification. Table S5. Mini-Mental State Examination for correlation with the AD diagnosis.

Raw data for the project of "Stereotypes, Common Ingroup Identity and Outgroup Admiration"

This project aims to investigate the relationship between ingroup moral- and image-shame and group members' self-construal and national identity. We will conduct a survey that asks participants about their independent-interdependent self-construal, national identification with multiple dimensions (e.g., commitment, superiority; Roccas et al., 2008), and their moral- and image-shame for recent national disaster in Korea (i.e., 10.29 Itaewon disaster). Further, we will examine the effect of moral- and image-shame on the defensiveness and corrective orientation toward ingroup misbehaviors.

Research for examining generalized prejudice (group-focused enmity) in Korea

Research for exploring generalized prejudice (group-focused enmity) in Korea

Dataset (raw data file in SPSS) for "Independent Self-Concept Promotes Group Creativity in a Collectivistic Cultural Context Only When the Group Norm Supports Collectivism" by Lee, H. and Choi, H-S. to appear in Group Dynamics: Theory, Research, and Practice.

This research examined the relationship between perceived walkability, use of public space, and civic & political engagement of Koreans.

This study investigates the relationship between perceived walkability, use of public space, and social well-being and community-oriented behaviors among international students in Korea.

Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on three different meta-learning methods and two benchmark datasets.

Previous research (Zick et al. 2008) suggested that animosity toward social minorities in Europe is intertwined, forming a syndrome of Group-Focused Enmity (GFE). In the current research, we extended the notion of GFE by identifying the GFE structure in a non-European context (South Korea). We also tested a novel hypothesis that stipulates an interplay between individuals’ self-concept and their value orientation in predicting the overall level of GFE. Two nation-wide surveys in South Korea showed that antagonism toward social minorities that have typically been marginalized and devalued in that country forms GFE while reflecting the unique intergroup context of Korean society. Further, we found as expected that independence in self-concept and a collectivistic value orientation jointly predict lower levels of GFE (Study 1). When political orientation and national identification were taken into account, the predicted interaction was observed only on antagonism toward ingroup minorities but not toward outgroup minorities by race or ethnicity (Study 2). Implications of these findings and directions for research on GFE are discussed., International Journal of Conflict and Violence (IJCV), Bd. 16 (2022)

This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.

Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on two different meta-learning methods and three benchmark datasets.

Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform human-designed networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets). Moreover, since such task-specific methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited. In this paper, we propose an efficient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search for a neural architecture for a novel dataset. The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning. Moreover, we also propose a meta-performance predictor to estimate and select the best architecture without direct training on target datasets. The experimental results demonstrate that our model meta-learned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple unseen datasets including CIFAR-10 and CIFAR-100, with an average search time of 33 GPU seconds. Even under MobileNetV3 search space, MetaD2A is 5.5K times faster than NSGANetV2, a transferable NAS method, with comparable performance. We believe that the MetaD2A proposes a new research direction for rapid NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years. Code is available at https://github.com/HayeonLee/MetaD2A., Comment: ICLR 2021

For deployment, neural architecture search should be hardware-aware, in order to satisfy the device-specific constraints (e.g., memory usage, latency and energy consumption) and enhance the model efficiency. Existing methods on hardware-aware NAS collect a large number of samples (e.g., accuracy and latency) from a target device, either builds a lookup table or a latency estimator. However, such approach is impractical in real-world scenarios as there exist numerous devices with different hardware specifications, and collecting samples from such a large number of devices will require prohibitive computational and monetary cost. To overcome such limitations, we propose Hardware-adaptive Efficient Latency Predictor (HELP), which formulates the device-specific latency estimation problem as a meta-learning problem, such that we can estimate the latency of a model's performance for a given task on an unseen device with a few samples. To this end, we introduce novel hardware embeddings to embed any devices considering them as black-box functions that output latencies, and meta-learn the hardware-adaptive latency predictor in a device-dependent manner, using the hardware embeddings. We validate the proposed HELP for its latency estimation performance on unseen platforms, on which it achieves high estimation performance with as few as 10 measurement samples, outperforming all relevant baselines. We also validate end-to-end NAS frameworks using HELP against ones without it, and show that it largely reduces the total time cost of the base NAS method, in latency-constrained settings. Code is available at https://github.com/HayeonLee/HELP., Comment: NeurIPS 2021 (Spotlight)

Recent metric-based meta-learning approaches, which learn a metric space that generalizes well over combinatorial number of different classification tasks sampled from a task distribution, have been shown to be effective for few-shot classification tasks of unseen classes. They are often trained with episodic training where they iteratively train a common metric space that reduces distance between the class representatives and instances belonging to each class, over large number of episodes with random classes. However, this training is limited in that while the main target is the generalization to the classification of unseen classes during training, there is no explicit consideration of generalization during meta-training phase. To tackle this issue, we propose a simple yet effective meta-learning framework for metricbased approaches, which we refer to as learning to generalize (L2G), that explicitly constrains the learning on a sampled classification task to reduce the classification error on a randomly sampled unseen classification task with a bilevel optimization scheme. This explicit learning aimed toward generalization allows the model to obtain a metric that separates well between unseen classes. We validate our L2G framework on mini-ImageNet and tiered-ImageNet datasets with two base meta-learning few-shot classification models, Prototypical Networks and Relation Networks. The results show that L2G significantly improves the performance of the two methods over episodic training. Further visualization shows that L2G obtains a metric space that clusters and separates unseen classes well., 9 pages, 3 figures

While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that the number of instances per task and class is fixed. Due to such restriction, they learn to equally utilize the meta-knowledge across all the tasks, even when the number of instances per task and class largely varies. Moreover, they do not consider distributional difference in unseen tasks, on which the meta-knowledge may have less usefulness depending on the task relatedness. To overcome these limitations, we propose a novel meta-learning model that adaptively balances the effect of the meta-learning and task-specific learning within each task. Through the learning of the balancing variables, we can decide whether to obtain a solution by relying on the meta-knowledge or task-specific learning. We formulate this objective into a Bayesian inference framework and tackle it using variational inference. We validate our Bayesian Task-Adaptive Meta-Learning (Bayesian TAML) on multiple realistic task- and class-imbalanced datasets, on which it significantly outperforms existing meta-learning approaches. Further ablation study confirms the effectiveness of each balancing component and the Bayesian learning framework.

This study describes South Koreans’ general perceptions of the integration of North and South Korea through a survey of 500 adults living in South Korea. The following multiple-choice questions were asked: one’s general ideas about the integration of North and South Korea; the type of Korean reunification which is mostly supported/opposed; the type of Korean reunification which is most probable; and the pros and cons of reunification as well as necessary factors for reunification. Furthermore, we examined the differences in the perception of Korean reunification among the subgroup based on participants’ demographic information (i.e., gender, age, political orientation). The main results are as follows. First, the most representative thought on integration was “geographical integration of the Korean Peninsula,” followed by “establishment of economic partnerships or communities” and “restoration of common identity.” Meanwhile, there were differences among participants with regard to the detailed representation of Korean reunification. It suggests that when the attitudes toward integration of North and South Korea society are discussed, differences in the perception among people should be considered., Historical Social Research / Historische Sozialforschung Vol. 44, No. 4 (2019): Forum: Social Integration or Mind Integration? An Alternative Approach for the Unification of Divided Systems. Starting Point and Frequency: Year: 1979, Issues per volume: 4, Volumes per year: 1, Historical Social Research , Transition (Online Supplement)

This study describes South Koreans’ general perceptions of the integration of North and South Korea through a survey of 500 adults living in South Korea. The following multiple-choice questions were asked: one’s general ideas about the integration of North and South Korea; the type of Korean reunification which is mostly supported/opposed; the type of Korean reunification which is most probable; and the pros and cons of reunification as well as necessary factors for reunification. Furthermore, we examined the differences in the perception of Korean reunification among the subgroup based on participants’ demographic information (i.e., gender, age, political orientation). The main results are as follows. First, the most representative thought on integration was “geographical integration of the Korean Peninsula,” followed by “establishment of economic partnerships or communities” and “restoration of common identity.” Meanwhile, there were differences among participants with regard to the detailed representation of Korean reunification. It suggests that when the attitudes toward integration of North and South Korea society are discussed, differences in the perception among people should be considered., Historical Social Research / Historische Sozialforschung Vol. 44, No. 4 (2019): Forum: Social Integration or Mind Integration? An Alternative Approach for the Unification of Divided Systems. Starting Point and Frequency: Year: 1979, Issues per volume: 4, Volumes per year: 1

This study describes South Koreans' general perceptions of the integration of North and South Korea through a survey of 500 adults living in South Korea. The following multiple-choice questions were asked: one's general ideas about the integration of North and South Korea; the type of Korean reunification which is mostly supported/opposed; the type of Korean reunification which is most probable; and the pros and cons of reunification as well as necessary factors for reunification. Furthermore, we examined the differences in the perception of Korean reunification among the subgroup based on participants' demographic information (i.e., gender, age, political orientation). The main results are as follows. First, the most representative thought on integration was "geographical integration of the Korean Peninsula," followed by "establishment of economic partnerships or communities" and "restoration of common identity." Meanwhile, there were differences among participants with regard to the detailed representation of Korean reunification. It suggests that when the attitudes toward integration of North and South Korea society are discussed, differences in the perception among people should be considered.

Learning objectives Background Findings and procedure details Conclusion Personal information References, Learning objectives: Development of pancreas is a complex process from formation of pancreatic dorsal and ventral primordials to rotation and fusion of these primordials. Subtle deviation in each process frequently gives rise to congenital anomalies in pancreas. In this...

Learning objectives Background Findings and procedure details Conclusion Personal information References, Learning objectives: -To learn the points for maintaining the reproducibility of breast US elastography examination technique. -To learn the points for maintaining elasticity score assessment...