Search

Your search keyword '"Wang, Jiayin"' showing total 160 results
Start Over Author "Wang, Jiayin" Publication Year Range This year
160 results on '"Wang, Jiayin"'

1. Interpret the Internal States of Recommendation Model with Sparse Autoencoder

Author

Wang, Jiayin, Zhang, Xiaoyu, Ma, Weizhi, and Zhang, Min

Subjects
Computer Science - Information Retrieval
Abstract

Explainable recommendation systems are important to enhance transparency, accuracy, and fairness. Beyond result-level explanations, model-level interpretations can provide valuable insights that allow developers to optimize system designs and implement targeted improvements. However, most current approaches depend on specialized model designs, which often lack generalization capabilities. Given the various kinds of recommendation models, existing methods have limited ability to effectively interpret them. To address this issue, we propose RecSAE, an automatic, generalizable probing method for interpreting the internal states of Recommendation models with Sparse AutoEncoder. RecSAE serves as a plug-in module that does not affect original models during interpretations, while also enabling predictable modifications to their behaviors based on interpretation results. Firstly, we train an autoencoder with sparsity constraints to reconstruct internal activations of recommendation models, making the RecSAE latents more interpretable and monosemantic than the original neuron activations. Secondly, we automated the construction of concept dictionaries based on the relationship between latent activations and input item sequences. Thirdly, RecSAE validates these interpretations by predicting latent activations on new item sequences using the concept dictionary and deriving interpretation confidence scores from precision and recall. We demonstrate RecSAE's effectiveness on two datasets, identifying hundreds of highly interpretable concepts from pure ID-based models. Latent ablation studies further confirm that manipulating latent concepts produces corresponding changes in model output behavior, underscoring RecSAE's utility for both understanding and targeted tuning recommendation models. Code and data are publicly available at https://github.com/Alice1998/RecSAE.

Published
2024

2. Beyond Utility: Evaluating LLM as Recommender

Author

Jiang, Chumeng, Wang, Jiayin, Ma, Weizhi, Clarke, Charles L. A., Wang, Shuai, Wu, Chuhan, and Zhang, Min

Subjects
Computer Science - Information Retrieval
Abstract

With the rapid development of Large Language Models (LLMs), recent studies employed LLMs as recommenders to provide personalized information services for distinct users. Despite efforts to improve the accuracy of LLM-based recommendation models, relatively little attention is paid to beyond-utility dimensions. Moreover, there are unique evaluation aspects of LLM-based recommendation models, which have been largely ignored. To bridge this gap, we explore four new evaluation dimensions and propose a multidimensional evaluation framework. The new evaluation dimensions include: 1) history length sensitivity, 2) candidate position bias, 3) generation-involved performance, and 4) hallucinations. All four dimensions have the potential to impact performance, but are largely unnecessary for consideration in traditional systems. Using this multidimensional evaluation framework, along with traditional aspects, we evaluate the performance of seven LLM-based recommenders, with three prompting strategies, comparing them with six traditional models on both ranking and re-ranking tasks on four datasets. We find that LLMs excel at handling tasks with prior knowledge and shorter input histories in the ranking setting, and perform better in the re-ranking setting, beating traditional models across multiple dimensions. However, LLMs exhibit substantial candidate position bias issues, and some models hallucinate non-existent items much more often than others. We intend our evaluation framework and observations to benefit future research on the use of LLMs as recommenders. The code and data are available at https://github.com/JiangDeccc/EvaLLMasRecommender.

Published
2024

3. PerSRV: Personalized Sticker Retrieval with Vision-Language Model

Author

Chee, Heng Er Metilda, Wang, Jiayin, Guo, Zhiqiang, Ma, Weizhi, and Zhang, Min

Subjects
Computer Science - Information Retrieval
Abstract

Instant Messaging is a popular means for daily communication, allowing users to send text and stickers. As the saying goes, "a picture is worth a thousand words", so developing an effective sticker retrieval technique is crucial for enhancing user experience. However, existing sticker retrieval methods rely on labeled data to interpret stickers, and general-purpose Vision-Language Models (VLMs) often struggle to capture the unique semantics of stickers. Additionally, relevant-based sticker retrieval methods lack personalization, creating a gap between diverse user expectations and retrieval results. To address these, we propose the Personalized Sticker Retrieval with Vision-Language Model framework, namely PerSRV, structured into offline calculations and online processing modules. The online retrieval part follows the paradigm of relevant recall and personalized ranking, supported by the offline pre-calculation parts, which are sticker semantic understanding, utility evaluation and personalization modules. Firstly, for sticker-level semantic understanding, we supervised fine-tuned LLaVA-1.5-7B to generate human-like sticker semantics, complemented by textual content extracted from figures and historical interaction queries. Secondly, we investigate three crowd-sourcing metrics for sticker utility evaluation. Thirdly, we cluster style centroids based on users' historical interactions to achieve personal preference modeling. Finally, we evaluate our proposed PerSRV method on a public sticker retrieval dataset from WeChat, containing 543,098 candidates and 12,568 interactions. Experimental results show that PerSRV significantly outperforms existing methods in multi-modal sticker retrieval. Additionally, our fine-tuned VLM delivers notable improvements in sticker semantic understandings.

Published
2024

4. An Enhanced Batch Query Architecture in Real-time Recommendation

Author

Zhang, Qiang, Teng, Zhipeng, Wu, Disheng, and Wang, Jiayin

Subjects
Computer Science - Information Retrieval, Computer Science - Machine Learning, C.3, H.3.3
Abstract

In industrial recommendation systems on websites and apps, it is essential to recall and predict top-n results relevant to user interests from a content pool of billions within milliseconds. To cope with continuous data growth and improve real-time recommendation performance, we have designed and implemented a high-performance batch query architecture for real-time recommendation systems. Our contributions include optimizing hash structures with a cacheline-aware probing method to enhance coalesced hashing, as well as the implementation of a hybrid storage key-value service built upon it. Our experiments indicate this approach significantly surpasses conventional hash tables in batch query throughput, achieving up to 90% of the query throughput of random memory access when incorporating parallel optimization. The support for NVMe, integrating two-tier storage for hot and cold data, notably reduces resource consumption. Additionally, the system facilitates dynamic updates, automated sharding of attributes and feature embedding tables, and introduces innovative protocols for consistency in batch queries, thereby enhancing the effectiveness of real-time incremental learning updates. This architecture has been deployed and in use in the bilibili recommendation system for over a year, a video content community with hundreds of millions of users, supporting 10x increase in model computation with minimal resource growth, improving outcomes while preserving the system's real-time performance., Comment: 8 pages, 10 figures, CIKM 2024 Applied Research Paper

Published
2024
Full Text
View/download PDF

5. Large Language Models as Evaluators for Recommendation Explanations

Author

Zhang, Xiaoyu, Li, Yishan, Wang, Jiayin, Sun, Bowen, Ma, Weizhi, Sun, Peijie, and Zhang, Min

Subjects
Computer Science - Information Retrieval, Computer Science - Computation and Language
Abstract

The explainability of recommender systems has attracted significant attention in academia and industry. Many efforts have been made for explainable recommendations, yet evaluating the quality of the explanations remains a challenging and unresolved issue. In recent years, leveraging LLMs as evaluators presents a promising avenue in Natural Language Processing tasks (e.g., sentiment classification, information extraction), as they perform strong capabilities in instruction following and common-sense reasoning. However, evaluating recommendation explanatory texts is different from these NLG tasks, as its criteria are related to human perceptions and are usually subjective. In this paper, we investigate whether LLMs can serve as evaluators of recommendation explanations. To answer the question, we utilize real user feedback on explanations given from previous work and additionally collect third-party annotations and LLM evaluations. We design and apply a 3-level meta evaluation strategy to measure the correlation between evaluator labels and the ground truth provided by users. Our experiments reveal that LLMs, such as GPT4, can provide comparable evaluations with appropriate prompts and settings. We also provide further insights into combining human labels with the LLM evaluation process and utilizing ensembles of multiple heterogeneous LLM evaluators to enhance the accuracy and stability of evaluations. Our study verifies that utilizing LLMs as evaluators can be an accurate, reproducible and cost-effective solution for evaluating recommendation explanation texts. Our code is available at https://github.com/Xiaoyu-SZ/LLMasEvaluator.

Published
2024

6. A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models

Author

Wang, Jiayin, Mo, Fengran, Ma, Weizhi, Sun, Peijie, Zhang, Min, and Nie, Jian-Yun

Subjects
Computer Science - Computation and Language
Abstract

Large language models (LLMs) are essential tools that users employ across various scenarios, so evaluating their performance and guiding users in selecting the suitable service is important. Although many benchmarks exist, they mainly focus on specific predefined model abilities, such as world knowledge, reasoning, etc. Based on these ability scores, it is hard for users to determine which LLM best suits their particular needs. To address these issues, we propose to evaluate LLMs from a user-centric perspective and design this benchmark to measure their efficacy in satisfying user needs under distinct intents. Firstly, we collect 1,846 real-world use cases from a user study with 712 participants from 23 countries. This first-hand data helps us understand actual user intents and needs in LLM interactions, forming the User Reported Scenarios (URS) dataset, which is categorized with six types of user intents. Secondly, based on this authentic dataset, we benchmark 10 LLM services with GPT-4-as-Judge. Thirdly, we show that benchmark scores align well with human preference in both real-world experience and pair-wise annotations, achieving Pearson correlations of 0.95 and 0.94, respectively. This alignment confirms that the URS dataset and our evaluation method establish an effective user-centric benchmark. The dataset, code, and process data are available at https://github.com/Alice1998/URS.

Published
2024

8. Adjunct Intraarterial or Intravenous Tirofiban Versus No Tirofiban After Successful Recanalization of Basilar Artery Occlusion Stroke: The BASILAR Registry.

Author

Li, Huagang, Ju, Dongsheng, Tao, Zhaojun, Wang, Jiayin, Nguyen, Thanh, Nogueira, Raul, Liu, Chang, Yang, Qingwu, Qiu, Zhongming, Yin, Congguo, Sun, Dong, Liu, Shudong, and Saver, Jeffrey

Subjects
basilar artery occlusion, endovascular treatment, intra‐arterial, prognosis, tirofiban, Humans, Tirofiban, Basilar Artery, Prospective Studies, Thrombolytic Therapy, Treatment Outcome, Stroke, Intracranial Hemorrhages, Arterial Occlusive Diseases, Registries, Endovascular Procedures, Thrombectomy
Abstract

BACKGROUND: Approximately half of patients who achieve successful reperfusion do not achieve functional independence. The present study sought to investigate the clinical outcomes and safety of intraarterial or intravenous tirofiban as adjunct therapy in patients with acute basilar artery occlusion who had achieved successful recanalization with endovascular treatment. METHODS AND RESULTS: In the national, prospective BASILAR (Endovascular Treatment for Acute Basilar Artery Occlusion Study) registry, 458 patients who met inclusion criteria were divided into 3 groups based on tirofiban administration (no tirofiban, n=262; intravenous tirofiban, n=101; intraarterial+intravenous tirofiban, n=95). Their clinical outcomes were compared with 90-day modified Rankin Scale scores. Adjusted odds ratios (aORs) and 95% CIs were obtained by logistic regression models and propensity score matching. Safety outcomes included any intracranial hemorrhage (ICH), symptomatic ICH, and mortality. Among 458 included patients, 184 (40.2%) achieved a favorable outcome (modified Rankin Scale score 0-3). There were no differences between the intravenous tirofiban group and the no tirofiban group in terms of safety and clinical outcomes (all P>0.05). Compared with the no tirofiban group, the intraarterial+intravenous tirofiban group had higher odds of 90-day modified Rankin Scale score 0 to 3 (aOR, 2.44 [95% CI, 1.30-4.64], P=0.006) and lower 3-month mortality (aOR, 0.38 [95% CI, 0.19-0.71], P=0.002) without an increase in any ICH (aOR, 0.34 [95% CI, 0.09-1.01], P=0.07) or symptomatic ICH (aOR, 0.23 [95% CI, 0.03-0.90], P=0.05). Similar results of intraarterial+intravenous tirofiban on improving clinical outcomes were detected in novel cohorts constructed by propensity score matching. CONCLUSIONS: Intraarterial+intravenous rather than intravenous tirofiban improved clinical outcomes without increasing the frequency of symptomatic ICH among patients with basilar artery occlusion after successful endovascular treatment. Further studies are needed to delineate the roles of intraarterial+intravenous tirofiban in patients with basilar artery occlusion receiving endovascular treatment.

Published
2024

11. Enhancing MET Copy Number Estimation by Factoring in Tumor Purity and Variant Types

Author

Zhao, Minchao, Wang, Jiayin, Chang, Zhili, Liu, Yuqian, Goos, Gerhard, Series Editor, Hartmanis, Juris, Founding Editor, Bertino, Elisa, Editorial Board Member, Gao, Wen, Editorial Board Member, Steffen, Bernhard, Editorial Board Member, Yung, Moti, Editorial Board Member, Rojas, Ignacio, editor, Ortuño, Francisco, editor, Rojas, Fernando, editor, Herrera, Luis Javier, editor, and Valenzuela, Olga, editor

Published
2024
Full Text
View/download PDF

34. Genome-Wide Identification of the Geranylgeranyl Pyrophosphate Synthase (GGPS) Gene Family Associated with Natural Rubber Synthesis in Taraxacum kok-saghyz L. Rodin.

Author

Wang, Lili, He, Huan, Wang, Jiayin, Meng, Zhuang, Wang, Lei, Jin, Xiang, Zhang, Jianhang, Du, Pingping, Zhang, Liyu, Wang, Fei, Li, Hongbin, and Xie, Quanliang

Subjects
RUBBER, GENE expression, GENE families, ENZYME metabolism, LATEX
Abstract

Taraxacum kok-saghyz Rodin (TKS) is a recognized alternative source of natural rubber comparable to the rubber tree. The geranylgeranyl pyrophosphate synthase (GGPS) catalyzed the synthesis of geranylgeranyl pyrophosphate (GGPP), which is an important enzyme in the secondary metabolism pathway. In this study, we present the first analysis of the GGPS gene family in TKS, where a total of seven TkGGPS family members were identified. Their core motifs, conserved structural domains, gene structures, and cis-acting elements were described. In addition, two phylogenetic trees were constructed based on the Neighbor-Joining and Maximum-Likelihood methods, and the TkGGPSs were highly conserved and exhibited good collinearity with the other species. Transcriptome data showed that seven TkGGPS gene members were expressed in all the 12 tissues measured, and TkGGPS1, TkGGPS3, and TkGGPS6 were highly expressed in latex, suggesting that they may be associated with natural rubber synthesis. Meanwhile, quantitative real-time PCR (qRT-PCR) showed that the expression levels of the TkGGPS genes were regulated by the ethylene and methyl jasmonate (MeJA) pathways. Subcellular localization results indicated that all the TkGGPS proteins were also located in chloroplasts involved in photosynthesis in plants. This study will provide valuable insights into the selection of candidate genes for molecular breeding and natural rubber biosynthesis in TKS. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

35. TCSR: Self‐attention with time and category for session‐based recommendation.

Author

Zhu, Xiaoyan, Zhang, Yu, Li, Jiaxuan, Wang, Jiayin, and Lai, Xin

Subjects
RESEARCH personnel, PROBLEM solving, ATTENTION
Abstract

Session‐based recommendation that uses sequence of items clicked by anonymous users to make recommendations has drawn the attention of many researchers, and a lot of approaches have been proposed. However, there are still problems that have not been well addressed: (1) Time information is either ignored or exploited with a fixed time span and granularity, which fails to understand the personalized interest transfer pattern of users with different clicking speeds; (2) Category information is either omitted or considered independent of the items, which defies the fact that the relationships between categories and items are helpful for the recommendation. To solve these problems, we propose a new session‐based recommendation method, TCSR (self‐attention with time and category for session‐based recommendation). TCSR uses a non‐linear normalized time embedding to perceive user interest transfer patterns on variable granularity and employs a heterogeneous SAN to make full use of both items and categories. Moreover, a cross‐recommendation unit is adapted to adjust recommendations on the item and category sides. Extensive experiments on four real datasets show that TCSR significantly outperforms state‐of‐the‐art approaches. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

36. Genome-Wide Identification and Characterization of the HMGR Gene Family in Taraxacum kok-saghyz Provide Insights into Its Regulation in Response to Ethylene and Methyl Jsamonate Treatments.

Author

Du, Pingping, He, Huan, Wang, Jiayin, Wang, Lili, Meng, Zhuang, Jin, Xiang, Zhang, Liyu, Wang, Fei, Li, Hongbin, and Xie, Quanliang

Subjects
GENE families, RUBBER, PLANT breeding, PROTEIN structure, ENDOPLASMIC reticulum
Abstract

HMGR (3-hydroxy-3-methylglutaryl-CoA reductase) plays a crucial role as the first rate-limiting enzyme in the mevalonate (MVA) pathway, which is the upstream pathway of natural rubber biosynthesis. In this study, we carried out whole-genome identification of Taraxacum kok-saghyz (TKS), a novel rubber-producing alternative plant, and obtained six members of the TkHMGR genes. Bioinformatic analyses were performed including gene structure, protein properties, chromosomal localization, evolutionary relationships, and cis-acting element analyses. The results showed that HMGR genes were highly conserved during evolution with a complete HMG-CoA reductase conserved domain and were closely related to Asteraceae plants during the evolutionary process. The α-helix is the most prominent feature of the secondary structure of the TkHMGR proteins. Collinearity analyses demonstrated that a whole-genome duplication (WGD) event and tandem duplication event play a key role in the expansion of this family and TkHMGR1 and TkHMGR6 have more homologous gene between other species. Cis-acting element analysis revealed that the TkHMGR gene family had a higher number of MYB-related, light-responsive, hormone-responsive elements. In addition, we investigated the expression patterns of family members induced by ethylene (ETH) and methyl jasmonate (MeJA), and their expression levels at different stages of T. kok-saghyz root development. Finally, subcellular localization results showed that six TkHMGR members were all located in the endoplasmic reticulum. In conclusion, the results of our study lay a certain theoretical basis for the subsequent improvement of rubber yield, molecular breeding of rubber-producing plants, and genetic improvement of T. kok-saghyz. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

37. Improved diagnostic efficiency of CRC subgroups revealed using machine learning based on intestinal microbes.

Author

Liu, Guang, Su, Lili, Kong, Cheng, Huang, Liang, Zhu, Xiaoyan, Zhang, Xuanping, Ma, Yanlei, and Wang, Jiayin

Subjects
RANDOM forest algorithms, GUT microbiome, MICROBIAL diversity, COLORECTAL cancer, MACHINE learning
Abstract

Background: Colorectal cancer (CRC) is a common cancer that causes millions of deaths worldwide each year. At present, numerous studies have confirmed that intestinal microbes play a crucial role in the process of CRC. Additionally, studies have shown that CRC can be divided into several consensus molecular subtypes (CMS) based on tumor gene expression, and CRC microbiomes have been reported related to CMS. However, most previous studies on intestinal microbiome of CRC have only compared patients with healthy controls, without classifying of CRC patients based on intestinal microbial composition. Results: In this study, a CRC cohort including 339 CRC samples and 333 healthy controls was selected as the discovery set, and the CRC samples were divided into two subgroups (234 Subgroup1 and 105 Subgroup2) using PAM clustering algorithm based on the intestinal microbial composition. We found that not only the microbial diversity was significantly different (Shannon index, p-value < 0.05), but also 129 shared genera altered (p-value < 0.05) between the two CRC subgroups, including several marker genera in CRC, such as Fusobacterium and Bacteroides. A random forest algorithm was used to construct diagnostic models, which showed significantly higher efficiency when the CRC samples were divided into subgroups. Then an independent cohort including 187 CRC samples (divided into 153 Subgroup1 and 34 Subgroup2) and 123 healthy controls was chosen to validate the models, and confirmed the results. Conclusions: These results indicate that the divided CRC subgroups can improve the efficiency of disease diagnosis, with various microbial composition in the subgroups. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

38. A Machine Learning Model Based on CT Imaging Metrics and Clinical Features to Predict the Risk of Hospital-Acquired Pneumonia After Traumatic Brain Injury.

Author

Li, Shaojie, Feng, Qiangqiang, Wang, Jiayin, Wu, Baofang, Qiu, Weizhi, Zhuang, Yiming, Wang, Yong, and Gao, Hongzhi

Subjects
MACHINE learning, BRAIN injuries, SUPPORT vector machines, DECISION making, COMPUTED tomography, NOMOGRAPHY (Mathematics), NAIVE Bayes classification
Abstract

To develop a validated machine learning (ML) algorithm for predicting the risk of hospital-acquired pneumonia (HAP) in patients with traumatic brain injury (TBI). Materials and Methods: We employed the Least Absolute Shrinkage and Selection Operator (LASSO) to identify critical features related to pneumonia. Five ML models—Logistic Regression (LR), Extreme Gradient Boosting (XGB), Random Forest (RF), Naive Bayes Classifier (NB), and Support Vector Machine (SVC)—were developed and assessed using the training and validation datasets. The optimal model was selected based on its performance metrics and used to create a dynamic web-based nomogram. Results: In a cohort of 858 TBI patients, the HAP incidence was 41.02%. LR was determined to be the optimal model with superior performance metrics including AUC, accuracy, and F1-score. Key predictive factors included Age, Glasgow Coma Score, Rotterdam Score, D-dimer, and the Systemic Immune Response to Inflammation Index (SIRI). The nomogram developed based on these predictors demonstrated high predictive accuracy, with AUCs of 0.818 and 0.819 for the training and validation datasets, respectively. Decision curve analysis (DCA) and calibration curves validated the model's clinical utility and accuracy. Conclusion: We successfully developed and validated a high-performance ML algorithm to assess the risk of HAP in TBI patients. The dynamic nomogram provides a practical tool for real-time risk assessment, potentially improving clinical outcomes by aiding in early intervention and personalized patient management. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

46. Adduct of Pesticide and Lignocellulosic Waste via Cleavable Silaketal Linkages for Agricultural Double Reduction

Author

Han, Wenwen, Zhang, Shoupeng, Wang, Jiayin, Xu, Xiao-Qi, and Wang, Yapei

Abstract

The abuse of hazardous agrochemicals leads to excessive toxic agricultural emissions, posing a tremendous threat to the natural surroundings and human well-being. In practice, the amount of pesticides in protecting crops is often far less than that lost into the environment through evaporation and leaching. Minimizing the use of pesticides as well as improving their use efficiency has been included in the policy of “agricultural double reduction,” besides replacing the chemical fertilizer with straw returning. Here, we establish a strategy for controlling pesticide release from the lignocellulosic waste based on the stimulus-responsive cleavage of silaketal linkages. Noting that the cleavage of the silaketal linkages relies heavily on the substituent groups on silicon atoms, this pesticide-releasing system has the advantages of predictable service life and less environmental pollution in a desired time window. Instead of lengthy laboratory synthesis, outdoor instant synthesis can be conveniently realized with the help of a photothermal heating apparatus. After utilization, both silaketal linkages and lignocellulosic residuals are eco-friendly and can be a source of nutrients for soil. Referring to agricultural double reduction, this type of pesticide formulation is coined as a competitive approach to minimize pesticide pollution along with straw returning.

Published
2024
Full Text
View/download PDF

47. Understanding User Experience in Large Language Model Interactions

Author

Wang, Jiayin, Ma, Weizhi, Sun, Peijie, Zhang, Min, Nie, Jian-Yun, Wang, Jiayin, Ma, Weizhi, Sun, Peijie, Zhang, Min, and Nie, Jian-Yun

Abstract

In the rapidly evolving landscape of large language models (LLMs), most research has primarily viewed them as independent individuals, focusing on assessing their capabilities through standardized benchmarks and enhancing their general intelligence. This perspective, however, tends to overlook the vital role of LLMs as user-centric services in human-AI collaboration. This gap in research becomes increasingly critical as LLMs become more integrated into people's everyday and professional interactions. This study addresses the important need to understand user satisfaction with LLMs by exploring four key aspects: comprehending user intents, scrutinizing user experiences, addressing major user concerns about current LLM services, and charting future research paths to bolster human-AI collaborations. Our study develops a taxonomy of 7 user intents in LLM interactions, grounded in analysis of real-world user interaction logs and human verification. Subsequently, we conduct a user survey to gauge their satisfaction with LLM services, encompassing usage frequency, experiences across intents, and predominant concerns. This survey, compiling 411 anonymous responses, uncovers 11 first-hand insights into the current state of user engagement with LLMs. Based on this empirical analysis, we pinpoint 6 future research directions prioritizing the user perspective in LLM developments. This user-centered approach is essential for crafting LLMs that are not just technologically advanced but also resonate with the intricate realities of human interactions and real-world applications., Comment: 15 pages + 3 page references + 2 page Appendix

Published
2024

48. A User-Centric Benchmark for Evaluating Large Language Models

Author

Wang, Jiayin, Mo, Fengran, Ma, Weizhi, Sun, Peijie, Zhang, Min, Nie, Jian-Yun, Wang, Jiayin, Mo, Fengran, Ma, Weizhi, Sun, Peijie, Zhang, Min, and Nie, Jian-Yun

Abstract

Large Language Models (LLMs) are essential tools to collaborate with users on different tasks. Evaluating their performance to serve users' needs in real-world scenarios is important. While many benchmarks have been created, they mainly focus on specific predefined model abilities. Few have covered the intended utilization of LLMs by real users. To address this oversight, we propose benchmarking LLMs from a user perspective in both dataset construction and evaluation designs. We first collect 1846 real-world use cases with 15 LLMs from a user study with 712 participants from 23 countries. These self-reported cases form the User Reported Scenarios(URS) dataset with a categorization of 7 user intents. Secondly, on this authentic multi-cultural dataset, we benchmark 10 LLM services on their efficacy in satisfying user needs. Thirdly, we show that our benchmark scores align well with user-reported experience in LLM interactions across diverse intents, both of which emphasize the overlook of subjective scenarios. In conclusion, our study proposes to benchmark LLMs from a user-centric perspective, aiming to facilitate evaluations that better reflect real user needs. The benchmark dataset and code are available at https://github.com/Alice1998/URS.

Published
2024

49. Selective C(sp3)–H bond aerobic oxidation enabled by a π-conjugated small molecule-oxygen charge transfer state.

Author

Huang, Panyi, Xu, Yan, Song, Haijing, Wang, Jiayin, Wang, Jiayang, Li, Jianjun, Sun, Bin, and Jin, Can

Subjects
SUSTAINABLE chemistry, ABSTRACTION reactions, OXIDIZING agents, REACTIVE oxygen species, METAL catalysts, CHARGE transfer
Abstract

Due to the high oxidation potential of arenes and alkenes, previous methods of benzylic and allylic C–H oxidation necessitated the use of strong oxidizing agents, metal catalysts, or free radical initiators (such as photocatalysts or hydrogen atom transfer reagents) to be effective. Herein, we employ oxygen to generate a triple-spin charge transfer state with π-conjugated compounds effectively accessing the triplet excited states of arenes, followed by energy transfer to form singlet oxygen facilitating C–H bond oxidation. Distinct from previous oxidations, it is the first time that a series of natural products and drug molecules (27 examples) are selectively oxidized without any catalysts or additives. The transformation can be implemented under sunlight to produce valuable unsaturated (aromatic) ketones and only water as waste, which meets every one of the 12 principles of green chemistry. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results