Search

Deep Neural Networks (DNNs) are increasingly being used in software engineering and code intelligence tasks. These are powerful tools that are capable of learning highly generalizable patterns from large datasets through millions of parameters. At the same time, their large capacity can render them prone to memorizing data points. Recent work suggests that the memorization risk manifests especially strongly when the training dataset is noisy, involving many ambiguous or questionable samples, and memorization is the only recourse. The goal of this paper is to evaluate and compare the extent of memorization and generalization in neural code intelligence models. It aims to provide insights on how memorization may impact the learning behavior of neural models in code intelligence systems. To observe the extent of memorization in models, we add random noise to the original training dataset and use various metrics to quantify the impact of noise on various aspects of training and testing. We evaluate several state-of-the-art neural code intelligence models and benchmarks based on Java, Python, and Ruby codebases. Our results highlight important risks: millions of trainable parameters allow the neural networks to memorize anything, including noisy data, and provide a false sense of generalization. We observed all models manifest some forms of memorization. This can be potentially troublesome in most code intelligence tasks where they rely on rather noise-prone and repetitive data sources, such as code from GitHub. To the best of our knowledge, we provide the first study to quantify memorization effects in the domain of software engineering and code intelligence systems. This work raises awareness and provides new insights into important issues of training neural models in code intelligence systems that are usually overlooked by software engineering researchers., Information and Software Technology, IST Journal 2022, Elsevier

A wide range of code intelligence (CI) tools, powered by deep neural networks, have been developed recently to improve programming productivity and perform program analysis. To reliably use such tools, developers often need to reason about the behavior of the underlying models and the factors that affect them. This is especially challenging for tools backed by deep neural networks. Various methods have tried to reduce this opacity in the vein of "transparent/interpretable-AI". However, these approaches are often specific to a particular set of network architectures, even requiring access to the network's parameters. This makes them difficult to use for the average programmer, which hinders the reliable adoption of neural CI systems. In this paper, we propose a simple, model-agnostic approach to identify critical input features for models in CI systems, by drawing on software debugging research, specifically delta debugging. Our approach, SIVAND, uses simplification techniques that reduce the size of input programs of a CI model while preserving the predictions of the model. We show that this approach yields remarkably small outputs and is broadly applicable across many model architectures and problem domains. We find that the models in our experiments often rely heavily on just a few syntactic features in input programs. We believe that SIVAND's extracted features may help understand neural CI systems' predictions and learned behavior., The 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'21)

Code completion is commonly used by software developers and is integrated into all major IDE's. Good completion tools can not only save time and effort but may also help avoid incorrect API usage. Many proposed completion tools have shown promising results on synthetic benchmarks, but these benchmarks make no claims about the realism of the completions they test. This lack of grounding in real-world data could hinder our scientific understanding of developer needs and of the efficacy of completion models. This paper presents a case study on 15,000 code completions that were applied by 66 real developers, which we study and contrast with artificial completions to inform future research and tools in this area. We find that synthetic benchmarks misrepresent many aspects of real-world completions; tested completion tools were far less accurate on real-world data. Worse, on the few completions that consumed most of the developers' time, prediction accuracy was less than 20% -- an effect that is invisible in synthetic benchmarks. Our findings have ramifications for future benchmarks, tool design and real-world efficacy: Benchmarks must account for completions that developers use most, such as intra-project APIs; models should be designed to be amenable to intra-project data; and real-world developer trials are essential to quantifying performance on the least predictable completions, which are both most time-consuming and far more typical than artificial data suggests. We publicly release our preprint [https://doi.org/10.5281/zenodo.2565673] and replication data and materials [https://doi.org/10.5281/zenodo.2562249].

Modern software development is increasingly collaborative. Open Source Software (OSS) are the bellwether; they support dynamic teams, with tools for code sharing, communication, and issue tracking. The success of an OSS project is reliant on team communication. E.g., in issue discussions, individuals rely on rhetoric to argue their position, but also maintain technical relevancy. Rhetoric and technical language are on opposite ends of a language complexity spectrum: the former is stylistically natural; the latter is terse and concise. Issue discussions embody this duality, as developers use rhetoric to describe technical issues. The style mix in any discussion can define group culture and affect performance, e.g., issue resolution times may be longer if discussion is imprecise. Using GitHub, we studied issue discussions to understand whether project-specific language differences exist, and to what extent users conform to a language norm. We built project-specific and overall GitHub language models to study the effect of perceived language complexity on multiple responses. We find that experienced users conform to project-specific language norms, popular individuals use overall GitHub language rather than project-specific language, and conformance to project-specific language norms reduces issue resolution times. We also provide a tool to calculate project-specific perceived language complexity.

The use of traffic control systems can potentially improve the traffic flows on traffic networks. However, for the implementation of such control systems—both in simulation and in practice—many steps should be taken, and many choices are to be made. In this paper a list of considerations is provided for developing model-based traffic control systems in general, with a more detailed discussion on the use of model-predictive control for traffic regulation. A case study of designing a traffic controller is provided for the Dutch A12 freeway.

Traffic control approaches based on on-line optimization require fast and accurate integrated models for traffic flow, emission, and fuel consumption. In this context, one may want to integrate macroscopic traffic flow models with microscopic emission and fuel consumption models, which can result in shorter simulation times with fairly accurate estimates of the emissions and fuel consumption. In general, however, macroscopic traffic flow models and microscopic emission and fuel consumption models cannot be integrated with each other. We provide a general framework to integrate these two kinds of models. We illustrate the approach by considering the macroscopic traffic flow model METANET 1 and the microscopic emission and fuel consumption model VT-micro, 2 resulting in the so called the “VT-macro” model. Moreover, we characterize analytically the error introduced by the VT-macro model relative to the original VT-micro model. We further present an empirical analysis of the error and the computation time based on calibrated models of the Dutch A12 freeway.

We present a freeway-traffic control strategy that continuously adapts traffic control measures to prevailing traffic conditions and features faster computation speed than conventional model-based predictive control (MPC). The control approach is based on the principles of state feedback control and MPC. Instead of computing the control input sequence, the proposed controller optimizes the parameters of control laws that parametrize the control input sequences. This way, the computational burden of the controller is substantially reduced. We demonstrate the proposed control approach on a calibrated model of part of the Dutch A12 freeway using variable speed limits and ramp-metering rate.

This paper considers a variant of the travelling salesman problem where both energy consumption and variable travel speeds are taken into account. The problem describes the scheduling of pick-ups and deliveries of passengers with micro-ferries, where the energy consumption is dependent on the speed of the ferries. The schedule should be such that the ferries do not run out of energy during a trip, and time-window misfits, travel times, and waiting times for passengers are minimised. Scheduling of many transportation requests is made possible by including the charging of the ferries in the scheduling procedure, whereas the inclusion of embarking and disembarking times ensures that the passengers can board the ferry comfortably.

In this paper a model-based traffic control is used to design variable speed limits and on-ramp metering rates in order to reduce road traffic generated area-wide emissions near freeways. First an area-wide emission model is proposed and next a nonlinear model predictive control (MPC) approach is applied. The objectives of the MPC controller considered are the emissions, dispersions of emissions in a public area near a freeway, travel times, or the combination of these performance indicators. We compare different controlled scenarios with respect to the uncontrolled case and with respect to each other. The simulation-based case studies indicate that balanced solutions can be obtained using the proposed nonlinear MPC control strategy.