Search

Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees, by dynamically adjusting test questions based on their performance. Widely adopted across diverse fields like education, healthcare, sports, and sociology, CAT has revolutionized testing practices. While traditional methods rely on psychometrics and statistics, the increasing complexity of large-scale testing has spurred the integration of machine learning techniques. This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method. By examining the test question selection algorithm at the heart of CAT's adaptivity, we shed light on its functionality. Furthermore, we delve into cognitive diagnosis models, question bank construction, and test control within CAT, exploring how machine learning can optimize these components. Through an analysis of current methods, strengths, limitations, and challenges, we strive to develop robust, fair, and efficient CAT systems. By bridging psychometric-driven CAT research with machine learning, this survey advocates for a more inclusive and interdisciplinary approach to the future of adaptive testing.

Large language models (LLMs), like ChatGPT, have shown some human-like cognitive abilities. For comparing these abilities of different models, several benchmarks (i.e. sets of standard test questions) from different fields (e.g., Literature, Biology and Psychology) are often adopted and the test results under traditional metrics such as accuracy, recall and F1, are reported. However, such way for evaluating LLMs can be inefficient and inaccurate from the cognitive science perspective. Inspired by Computerized Adaptive Testing (CAT) used in psychometrics, we propose an adaptive testing framework for LLM evaluation. Rather than using a standard test set and simply reporting accuracy, this approach dynamically adjusts the characteristics of the test questions, such as difficulty, based on the model's performance. This allows for a more accurate estimation of the model's abilities, using fewer questions. More importantly, it allows LLMs to be compared with humans easily, which is essential for NLP models that aim for human-level ability. Our diagnostic reports have found that ChatGPT often behaves like a ``careless student'', prone to slip and occasionally guessing the questions. We conduct a fine-grained diagnosis and rank the latest 6 instruction-tuned LLMs from three aspects of Subject Knowledge, Mathematical Reasoning, and Programming, where GPT4 can outperform other models significantly and reach the cognitive ability of middle-level students. Different tests for different models using efficient adaptive testing -- we believe this has the potential to become a new norm in evaluating large language models.

Weizhe Huang,1 Xin Xu,2– 5 Mengyang Liu,2– 5 Weixue Cui,2– 5 Guilin Peng2– 5 1Department of Thoracic Surgery, The First Affiliated Hospital of Shantou University Medical College, Shantou 515041, Guangdong, People’s Republic of China; 2Department of Thoracic Surgery, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou 510120, Guangdong, People’s Republic of China; 3State Key Laboratory of Respiratory Disease, Guangzhou 510120, Guangdong, People’s Republic of China; 4National Clinical Research Center for Respiratory Disease, Guangzhou 510120, Guangdong, People’s Republic of China; 5Guangzhou Institute of Respiratory Health, Guangzhou 510120, Guangdong, People’s Republic of ChinaCorrespondence: Guilin Peng Tel +86-20-83062114Email pvlpl4@163.comBackground: Lung cancer is the most commonly diagnosed cancer and the major cause of cancer-related deaths worldwide. The increasing studies have demonstrated that circular RNA (circRNA) was involved in the progression of various cancers, including non-small-cell lung cancer (NSCLC). This study was designed to assess the expression, roles and functional mechanisms of circ_0000735 in NSCLC.Materials and Methods: The expression levels of circ_0000735, miR-940 and bone morphogenetic protein binding endothelial cell precursor-derived regulator (BMPER) were estimated by the real-time quantitative polymerase chain reaction (RT-qPCR). The biological behaviors of NSCLC cells such as proliferation, migration and invasion were analyzed by cell counting kit-8 (CCK-8), colony-forming assays and transwell assay, respectively. Furthermore, extracellular acid ratio and lactate production were tested to assess glycolysis levels of NSCLC cells. The interaction relationship among circ_0000735, BMPER and miR-940 was analyzed by bioinformatics database and dual-luciferase reporter assay. The protein expression level of BMPER was assessed b

Background: Models for predicting the survival outcomes of stage I non-small-cell lung cancer (NSCLC) defined by the newly released 8th edition TNM staging system are scarce. This study aimed to develop a nomogram for predicting the cancer-specific survival (CSS) of these patients and identifying individuals with a higher risk for CSS. Methods: A total of 30,475 NSCLC cases were extracted from the Surveillance, Epidemiology, and End Results (SEER) database. We identified and integrated the risk factors to build a nomogram. The model was subjected to bootstrap internal validation with the SEER database, and external validation with a multicenter cohort of 1133 patients from China. The difference in the impact of adjuvant chemotherapy on model-defined high- and low-risk patients was examined using the National Cancer Database (NCDB). Results: Eight independent prognostic factors were identified and integrated into the model. The calibration curves showed good agreement. The concordance index (C-index) of the nomogram was higher than that of the staging system (IA1, IA2, IA3, and IB) (internal validation set 0.63 vs. 0.56; external validation set 0.66 vs. 0.55; both p < 0.01). Specifically, 21.7% of stage IB patients (7.5% of all stage I) were categorized into the high-risk group (score > 30). There was a significant interaction effect between the adjuvant chemotherapy and risk groups in the NCDB cohort (p = 0.003). Conclusions: We established a practical nomogram to predict CSS for 8th edition stage I NSCLC. A prospective study is warranted to determine its role in identifying adjuvant chemotherapy candidates.

BACKGROUND: The optimal treatment for stage IIIA-N2 non-small cell lung cancer (NSCLC) is controversial. We aimed to address this important issue through a Bayesian network meta-analysis.METHODS: We performed a search of electronic databases for randomized controlled trials comparing the following treatments: surgery, radiotherapy, chemotherapy, and their multiple combinations before March 25, 2018. Pooled data on overall survival and treatment-related deaths were analyzed within the Bayesian framework.RESULTS: Eighteen eligible trials reporting 13 treatments were included. In terms of overall survival, neoadjuvant chemotherapy followed by surgery and adjuvant chemotherapy or radiotherapy, which tended to be consistent (hazard ratio [HR] 1.14, 95% credible interval [CrI] 0.21 to 5.93), ranked superior to other treatments. Notably, neoadjuvant chemotherapy followed by surgery and adjuvant radiotherapy was significantly more effective in prolonging survival than surgery alone (HR 0.38, 95% CrI 0.18 to 0.81), surgery plus adjuvant radiotherapy (HR 0.51, 95% CrI 0.29 to 0.92) and potentially surgery plus adjuvant chemotherapy (HR 0.49, 95% CrI 0.23 to 1.05). Overall, with 29% as the highest possibility of causing the fewest treatment-related deaths, neoadjuvant chemotherapy followed by surgery and adjuvant chemotherapy or radiotherapy was the safest treatment option.CONCLUSIONS: Neoadjuvant chemotherapy followed by surgery and adjuvant chemotherapy or radiotherapy has the greatest possibility to be the optimal treatment with the best overall survival and fewest treatment-related deaths for stage IIIA-N2 NSCLC.