Search

Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This paper presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods, and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field., Comment: 28 pages

Federated Learning (FL) since proposed has been applied in many fields, such as credit assessment, medical, etc. Because of the difference in the network or computing resource, the clients may not update their gradients at the same time that may take a lot of time to wait or idle. That's why Asynchronous Federated Learning (AFL) method is needed. The main bottleneck in AFL is communication. How to find a balance between the model performance and the communication cost is a challenge in AFL. This paper proposed a novel AFL framework VAFL. And we verified the performance of the algorithm through sufficient experiments. The experiments show that VAFL can reduce the communication times about 51.02\% with 48.23\% average communication compression rate and allow the model to be converged faster. The code is available at \url{https://github.com/RobAI-Lab/VAFL}, Comment: 8 pages

Federated Learning (FL) creates an ecosystem for multiple agents to collaborate on building models with data privacy consideration. The method for contribution measurement of each agent in the FL system is critical for fair credits allocation but few are proposed. In this paper, we develop a real-time contribution measurement method FedCM that is simple but powerful. The method defines the impact of each agent, comprehensively considers the current round and the previous round to obtain the contribution rate of each agent with attention aggregation. Moreover, FedCM updates contribution every round, which enable it to perform in real-time. Real-time is not considered by the existing approaches, but it is critical for FL systems to allocate computing power, communication resources, etc. Compared to the state-of-the-art method, the experimental results show that FedCM is more sensitive to data quantity and data quality under the premise of real-time. Furthermore, we developed federated learning open-source software based on FedCM. The software has been applied to identify COVID-19 based on medical images.

AI plays an important role in COVID-19 identification. Computer vision and deep learning techniques can assist in determining COVID-19 infection with Chest X-ray Images. However, for the protection and respect of the privacy of patients, the hospital's specific medical-related data did not allow leakage and sharing without permission. Collecting such training data was a major challenge. To a certain extent, this has caused a lack of sufficient data samples when performing deep learning approaches to detect COVID-19. Federated Learning is an available way to address this issue. It can effectively address the issue of data silos and get a shared model without obtaining local data. In the work, we propose the use of federated learning for COVID-19 data training and deploy experiments to verify the effectiveness. And we also compare performances of four popular models (MobileNet, ResNet18, MoblieNet, and COVID-Net) with the federated learning framework and without the framework. This work aims to inspire more researches on federated learning about COVID-19.

New coronavirus disease (COVID-19) has constituted a global pandemic and has spread to most countries and regions in the world. By understanding the development trend of a regional epidemic, the epidemic can be controlled using the development policy. The common traditional mathematical differential equations and population prediction models have limitations for time series population prediction, and even have large estimation errors. To address this issue, we propose an improved method for predicting confirmed cases based on LSTM (Long-Short Term Memory) neural network. This work compared the deviation between the experimental results of the improved LSTM prediction model and the digital prediction models (such as Logistic and Hill equations) with the real data as reference. And this work uses the goodness of fitting to evaluate the fitting effect of the improvement. Experiments show that the proposed approach has a smaller prediction deviation and a better fitting effect. Compared with the previous forecasting methods, the contributions of our proposed improvement methods are mainly in the following aspects: 1) we have fully considered the spatiotemporal characteristics of the data, rather than single standardized data; 2) the improved parameter settings and evaluation indicators are more accurate for fitting and forecasting. 3) we consider the impact of the epidemic stage and conduct reasonable data processing for different stage.

Distributed network of the computer and the design defects of the TCP protocol are given to the network attack to be multiplicative. Based on the simple and open assumptions of the TCP protocol in academic and collaborative communication environments, the protocol lacks secure authentication. In this paper, by adding RSA-based cryptography technology, RSA-based signature technology, DH key exchange algorithm, and HAMC-SHA1 integrity verification technology to the TCP protocol, and propose a security strategy which can effectively defend against TCP session hijacking.