Search

Language models (LMs) are increasingly used to simulate human-like responses in scenarios where accurately mimicking a population's behavior can guide decision-making, such as in developing educational materials and designing public policies. The objective of these simulations is for LMs to capture the variations in human responses, rather than merely providing the expected correct answers. Prior work has shown that LMs often generate unrealistically accurate responses, but there are no established metrics to quantify how closely the knowledge distribution of LMs aligns with that of humans. To address this, we introduce "psychometric alignment," a metric that measures the extent to which LMs reflect human knowledge distribution. Assessing this alignment involves collecting responses from both LMs and humans to the same set of test items and using Item Response Theory to analyze the differences in item functioning between the groups. We demonstrate that our metric can capture important variations in populations that traditional metrics, like differences in accuracy, fail to capture. We apply this metric to assess existing LMs for their alignment with human knowledge distributions across three real-world domains. We find significant misalignment between LMs and human populations, though using persona-based prompts can improve alignment. Interestingly, smaller LMs tend to achieve greater psychometric alignment than larger LMs. Further, training LMs on human response data from the target distribution enhances their psychometric alignment on unseen test items, but the effectiveness of such training varies across domains., Comment: Code and data: https://github.com/joyheyueya/psychometric-alignment

How (dis)similar are the learning trajectories of vision-language models and children? Recent modeling work has attempted to understand the gap between models' and humans' data efficiency by constructing models trained on less data, especially multimodal naturalistic data. However, such models are often evaluated on adult-level benchmarks, with limited breadth in language abilities tested, and without direct comparison to behavioral data. We introduce DevBench, a multimodal benchmark comprising seven language evaluation tasks spanning the domains of lexical, syntactic, and semantic ability, with behavioral data from both children and adults. We evaluate a set of vision-language models on these tasks, comparing models and humans not only on accuracy but on their response patterns. Across tasks, models exhibit variation in their closeness to human response patterns, and models that perform better on a task also more closely resemble human behavioral responses. We also examine the developmental trajectory of OpenCLIP over training, finding that greater training results in closer approximations to adult response patterns. DevBench thus provides a benchmark for comparing models to human language development. These comparisons highlight ways in which model and human language learning processes diverge, providing insight into entry points for improving language models.

Previous studies in predicting crash risk primarily associated the number or likelihood of crashes on a road segment with traffic parameters or geometric characteristics of the segment, usually neglecting the impact of vehicles' continuous movement and interactions with nearby vehicles. Advancements in communication technologies have empowered driving information collected from surrounding vehicles, enabling the study of group-based crash risks. Based on high-resolution vehicle trajectory data, this research focused on vehicle groups as the subject of analysis and explored risk formation and propagation mechanisms considering features of vehicle groups and road segments. Several key factors contributing to crash risks were identified, including past high-risk vehicle-group states, complex vehicle behaviors, high percentage of large vehicles, frequent lane changes within a vehicle group, and specific road geometries. A multinomial logistic regression model was developed to analyze the spatial risk propagation patterns, which were classified based on the trend of high-risk occurrences within vehicle groups. The results indicated that extended periods of high-risk states, increase in vehicle-group size, and frequent lane changes are associated with adverse risk propagation patterns. Conversely, smoother traffic flow and high initial crash risk values are linked to risk dissipation. Furthermore, the study conducted sensitivity analysis on different types of classifiers, prediction time intervalsss and adaptive TTC thresholds. The highest AUC value for vehicle-group risk prediction surpassed 0.93. The findings provide valuable insights to researchers and practitioners in understanding and prediction of vehicle-group safety, ultimately improving active traffic safety management and operations of Connected and Autonomous Vehicles., Comment: 14 pages, 8 figures

Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. Moreover, many tests require multiple distinct sets of questions administered throughout the school year to closely monitor students' progress, known as parallel tests. In this study, we focus on tests of silent sentence reading efficiency, used to assess students' reading ability over time. To generate high-quality parallel tests, we propose to fine-tune large language models (LLMs) to simulate how previous students would have responded to unseen items. With these simulated responses, we can estimate each item's difficulty and ambiguity. We first use GPT-4 to generate new test items following a list of expert-developed rules and then apply a fine-tuned LLM to filter the items based on criteria from psychological measurements. We also propose an optimal-transport-inspired technique for generating parallel tests and show the generated tests closely correspond to the original test's difficulty and reliability based on crowdworker responses. Our evaluation of a generated test with 234 students from grades 2 to 8 produces test scores highly correlated (r=0.93) to those of a standard test form written by human experts and evaluated across thousands of K-12 students., Comment: Accepted to EMNLP 2023 (Main)

It is commonly seen that buses are blocked by the ones in front serving passengers and have to queue outside a curbside bus stop although there are vacant berths at the stop. The resultant bus delays degrade the service level of urban public transportation. A potential solution is to reschedule the arrivals of the buses at the stop for full utilization of the berths with the aid of connected vehicle technologies. This study proposes a mixed-integer linear programming model to optimize the scheduling of bus arrivals and the bus-berth matching at a curbside stop under connected vehicle environment. The objective is the minimization of the bus delays weighted by the number of passengers on the buses. Bus arrival times at the stop and the assignment of berths are optimized together with bus departure times from the stop. Bus punctuality is also taken into consideration. The proposed model could be applied dynamically to cater to time-varying traffic conditions. Numerical studies validate the advantages of the proposed model over the first-come-first-service strategy and the relaxed model without bus punctuality in terms of weighted bus delays and bus punctuality. Sensitivity analyses show that: 1) the proposed model is robust to the fluctuation of bus service time; and 2) a smaller number of berths may be preferred on condition that the bus demand does not exceed the stop capacity., Comment: The results in Figure 6 are not completely consistant with the results in Table 4. The delay is only several seconds and M2 is overestimated. More experimetns is needed to validate the proposed model

Connected and automated vehicles (CAVs) and human-driven vehicles (HVs) are expected to coexist in the near future. CAV-dedicated lanes and phases have been explored to handle the uncertainty in the driving behavior of HVs in the mixed traffic environment. However, CAV-dedicated phases could significantly sacrifice HV benefits. This study proposes a shared-phase-dedicated-lane (SPDL)-based traffic control model at isolated intersections under the mixed traffic environment. Left-turn and through CAVs share CAV-dedicated lanes and cross the intersection during the shared phases with HVs. A three-level optimization model is developed. At the upper level, a standard NEMA (National Electrical Manufacturers Association) ring barrier structure is used for the signal optimization and barrier durations are optimized by dynamic programming to minimize the total vehicle delay. At the middle level, phase sequence and phase durations are optimized by enumeration for the given barrier from the upper level and the minimum vehicle delay is fed to the upper level. At the lower level, CAV platooning in the buffer zone and trajectory planning in the passing zone are conducted based on the signal timings of the barrier from the middle level and the travel time of CAVs is fed to the middle level. A rolling-horizon scheme is further designed for the dynamical implementation of the proposed model with time-varying traffic conditions. Numerical studies validate the advantages of the SPDL-based control over the blue-phase based control in previous studies in terms of average vehicle delay and intersection capacity. Further, the SPDL-based model is extended to serve as an alternative approach without the buffer zone.

Platooning on highways with connected and automated vehicles (CAVs) has attracted considerable attention, while how to mange and coordinate platoons in urban networks remains largely an open question. This scientific gap mainly results from the maneuver complexity on urban roads, making it difficult to model the platoon formation process. Inspired by flocking behaviors in nature, this paper proposed a two-dimensional model to describe CAV group dynamics. The model is formulated based on potential fields in planar coordinates, which is composed of the inter-vehicle potential field and the cross-section potential field. The inter-vehicle potential field enables CAVs to attract each other when vehicle gaps are larger than the equilibrium distance, and repel each other otherwise. It also generates incentives for lane change maneuvers to join a platoon or to comply with the traffic management layer. The cross-section potential field is able to mimic lane keeping behavior and it also creates resistance to avoid unnecessary lane changes at very low incentives. These modeling principles can also be applied to human-driven vehicles in the mixed traffic environment. Behavioral plausibility of the model in terms of car-following rationality and car-following safety is demonstrated analytically and further verified with simulation in typical driving scenarios. The model is computationally efficient and can provide insights into platoon operations in urban networks.

Trajectory planning and coordination for connected and automated vehicles (CAVs) have been studied at isolated ``signal-free'' intersections and in ``signal-free'' corridors under the fully CAV environment in the literature. Most of the existing studies are based on the definition of approaching and exit lanes. The route a vehicle takes to pass through an intersection is determined from its movement. That is, only the origin and destination arms are included. This study proposes a mixed-integer linear programming (MILP) model to optimize vehicle trajectories at an isolated ``signal-free'' intersection without lane allocation, which is denoted as ``lane-allocation-free'' (LAF) control. Each lane can be used as both approaching and exit lanes for all vehicle movements including left-turn, through, and right-turn. A vehicle can take a flexible route by way of multiple arms to pass through the intersection. In this way, the spatial-temporal resources are expected to be fully utilized. The interactions between vehicle trajectories are modeled explicitly at the microscopic level. Vehicle routes and trajectories (i.e., car-following and lane-changing behaviors) at the intersection are optimized in one unified framework for system optimality in terms of total vehicle delay. Considering varying traffic conditions, the planning horizon is adaptively adjusted in the implementation procedure of the proposed model to make a balance between solution feasibility and computational burden. Numerical studies validate the advantages of the proposed LAF control in terms of both vehicle delay and throughput with different demand structures and temporal safety gaps., Comment: 31 pages, 5 figures, for simulation video, see https://magic.tongji.edu.cn/en/index.php?catid=41