Search

The emergence of large language models (LLMs) has significantly pushed the frontiers of program synthesis. Advancement of LLM-based program synthesis calls for a thorough evaluation of LLM-generated code. Most evaluation frameworks focus on the (functional) correctness of generated code; efficiency, as an important measure of code quality, has been overlooked in existing evaluations. In this work, we develop ENAMEL (EfficeNcy AutoMatic EvaLuator), a rigorous and high-standard benchmark for evaluating the capability of LLMs in generating efficient code. Firstly, we propose a new efficiency metric called eff@k, which generalizes the pass@k metric from correctness to efficiency and appropriately handles right-censored execution time. Furthermore, we derive an unbiased and variance-reduced estimator of eff@k via Rao--Blackwellization; we also provide a numerically stable implementation for the new estimator. Secondly, to set a high-standard for efficiency evaluation, we employ a human expert to design best algorithms and implementations as our reference solutions of efficiency, many of which are much more efficient than existing canonical solutions in HumanEval and HumanEval+. Moreover, to ensure a rigorous evaluation, we employ a human expert to curate strong test case generators to filter out wrong code and differentiate suboptimal algorithms. An extensive study across 30 popular LLMs using our benchmark ENAMEL shows that LLMs still fall short of generating expert-level efficient code. Using two subsets of our problem set, we demonstrate that such deficiency is because current LLMs struggle in designing advanced algorithms and are barely aware of implementation optimization. Our benchmark is publicly available at https://github.com/q-rz/enamel .

Recent works on machine learning for combinatorial optimization have shown that learning based approaches can outperform heuristic methods in terms of speed and performance. In this paper, we consider the problem of finding an optimal topological order on a directed acyclic graph with focus on the memory minimization problem which arises in compilers. We propose an end-to-end machine learning based approach for topological ordering using an encoder-decoder framework. Our encoder is a novel attention based graph neural network architecture called \emph{Topoformer} which uses different topological transforms of a DAG for message passing. The node embeddings produced by the encoder are converted into node priorities which are used by the decoder to generate a probability distribution over topological orders. We train our model on a dataset of synthetically generated graphs called layered graphs. We show that our model outperforms, or is on-par, with several topological ordering baselines while being significantly faster on synthetic graphs with up to 2k nodes. We also train and test our model on a set of real-world computation graphs, showing performance improvements., Comment: To appear in NeurIPS 2022

This paper investigates the linear precoder design that maximizes the average mutual information of multiple-input multiple-output channels with finite-alphabet inputs and statistical channel state information known at the transmitter. This linear precoder design is an important open problem and is extremely difficult to solve: First, average mutual information lacks closed-form expression and involves complicated computations; Second, the optimization problem over precoder is nonconcave. This study explores the solution to this problem and provides the following contributions: 1) A closed-form lower bound of average mutual information is derived. It achieves asymptotic optimality at low and high signal-to-noise ratio regions and, with a constant shift, offers an accurate approximation to the average mutual information; 2) The optimal structure of the precoder is revealed, and a unified two-step iterative algorithm is proposed to solve this problem. Numerical examples show the convergence and the efficacy of the proposed algorithm. Compared to its conventional counterparts, the proposed linear precoding method provides a significant performance gain., Comment: 5 pages, 3 figures, accepted by IEEE Global Communications Conference (GLOBECOM) 2011, Houston, TX

In this paper, we investigate the optimal precoding scheme for relay networks with finite-alphabet constraints. We show that the previous work utilizing various design criteria to maximize either the diversity order or the transmission rate with the Gaussian-input assumption may lead to significant loss for a practical system with finite constellation set constraint. A linear precoding scheme is proposed to maximize the mutual information for relay networks. We exploit the structure of the optimal precoding matrix and develop a unified two-step iterative algorithm utilizing the theory of convex optimization and optimization on the complex Stiefel manifold. Numerical examples show that this novel iterative algorithm achieves significant gains compared to its conventional counterpart., Comment: Accepted by IEEE Int. Conf. Commun. (ICC), Kyoto, Japan, 2011