Search

The integration of Large Language Models (LLMs) into Development Environments (IDEs) has become a focal point in modern software development. LLMs such as OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment developer productivity by serving as intelligent, chat-driven programming assistants. However, utilizing LLMs out of the box is unlikely to be optimal for any given scenario. Rather, each system requires the LLM to be honed to its set of heuristics to ensure the best performance. In this paper, we introduce the Copilot evaluation harness: a set of data and tools for evaluating LLM-guided IDE interactions, covering various programming scenarios and languages. We propose our metrics as a more robust and information-dense evaluation than previous state of the art evaluation systems. We design and compute both static and execution based success metrics for scenarios encompassing a wide range of developer tasks, including code generation from natural language (generate), documentation generation from code (doc), test case generation (test), bug-fixing (fix), and workspace understanding and query resolution (workspace). These success metrics are designed to evaluate the performance of LLMs within a given IDE and its respective parameter space. Our learnings from evaluating three common LLMs using these metrics can inform the development and validation of future scenarios in LLM guided IDEs.

Code coverage is a widely used metric for quantifying the extent to which program elements, such as statements or branches, are executed during testing. Calculating code coverage is resource-intensive, requiring code building and execution with additional overhead for the instrumentation. Furthermore, computing coverage of any snippet of code requires the whole program context. Using Machine Learning to amortize this expensive process could lower the cost of code coverage by requiring only the source code context, and the task of code coverage prediction can be a novel benchmark for judging the ability of models to understand code. We propose a novel benchmark task called Code Coverage Prediction for Large Language Models (LLMs). We formalize this task to evaluate the capability of LLMs in understanding code execution by determining which lines of a method are executed by a given test case and inputs. We curate and release a dataset we call COVERAGEEVAL by executing tests and code from the HumanEval dataset and collecting code coverage information. We report the performance of four state-of-the-art LLMs used for code-related tasks, including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's Claude, on the Code Coverage Prediction task. Finally, we argue that code coverage as a metric and pre-training data source are valuable for overall LLM performance on software engineering tasks.

Current research on hate speech analysis is typically oriented towards monolingual and single classification tasks. In this paper, we present a new multilingual hate speech analysis dataset for English, Hindi, Arabic, French, German and Spanish languages for multiple domains across hate speech - Abuse, Racism, Sexism, Religious Hate and Extremism. To the best of our knowledge, this paper is the first to address the problem of identifying various types of hate speech in these five wide domains in these six languages. In this work, we describe how we created the dataset, created annotations at high level and low level for different domains and how we use it to test the current state-of-the-art multilingual and multitask learning approaches. We evaluate our dataset in various monolingual, cross-lingual and machine translation classification settings and compare it against open source English datasets that we aggregated and merged for this task. Then we discuss how this approach can be used to create large scale hate-speech datasets and how to leverage our annotations in order to improve hate speech detection and classification in general.

We study the feasibility of a Data Science assistant powered by a sequence-to-sequence transformer by training a new model JuPyT5 on all publicly available Jupyter Notebook GitHub repositories and developing a new metric: Data Science Problems (DSP). DSP is a collection of 1119 problems curated from 306 pedagogical notebooks with 92 dataset dependencies, natural language and Markdown problem descriptions, and assert-based unit tests. These notebooks were designed to test university students' mastery of various Python implementations of Math and Data Science, and we now leverage them to study the ability of JuPyT5 to understand and pass the tests. We analyze the content of DSP, validate its quality, and we find that given 100 sampling attempts JuPyT5 is able to solve 77.5\% of the DSP problems. We further present various ablation and statistical analyses and compare DSP to other recent natural language to code benchmarks.

Fano resonance exhibiting an asymmetric spectral line shape is a universal phenomenon observed in diverse physical systems. Here we experimentally establish a direct link between the spectral asymmetry parameter and a physically realizable phase factor of interference between a continuum and a discrete mode that leads to Fano resonance. Using a specially designed metamaterial, namely waveguided plasmonic crystal with a spatially varying orientation axis of plasmonic grating, we demonstrate control on the spectral asymmetry of the Fano resonance through changes in the geometric phase of polarized light. In this scenario, the changes in the geometric phase for input left, and right circular polarized light arises due to varying orientation angle of the grating axis. The systematic changes in the geometric phase and the resulting q-parameter of Fano resonance is interpreted by an appropriate theoretical model connecting the two physical entities. The demonstrated control over the spectral line shape of Fano resonance achieved by tailoring geometric phase may open up novel routes for polarization-based photonic applications.

Fano resonance in plasmonic oligomers originating from the interference of a spectrally broad superradiant mode and a discrete subradiant mode is under intensive recent investigations due to numerous potential applications. In this regard, development of experimental means to understand and control the complex Fano interference process and to modulate the resulting asymmetric Fano spectral line shape is highly sought after. Here we present a polarization Mueller matrix measurement and inverse analysis approach for quantitative understanding and interpretation of the complex interference process that lead to Fano resonance in symmetry broken plasmonic oligomers. The spectral Mueller matrices of the plasmonic oligomers were recorded using a custom designed dark-field Mueller matrix spectroscopy system. These were subsequently analyzed using differential Mueller matrix decomposition technique to yield the quantitative sample polarimetry characteristics, namely, polarization diattenuation (d) and linear retardance ({\delta}) parameters. The unique signature of the interference of the superradiant dipolar plasmon mode and the subradiant quadrupolar mode of the symmetry broken plasmonic oligomers manifested as rapid spectral variation of the diattenuation and the linear retardance parameters across the Fano spectral dip. The polarization information contained in the Mueller matrix was further utilized to desirably control the Fano spectral line shape. The experimental Mueller matrix analysis was complemented with finite element based numerical simulations, which enabled quantitative understanding of the interference of the superradiant and the subradiant plasmon modes and its link with the polarization diattenuation and retardance parameters.

Weak measurement enables faithful amplification and high precision measurement of small physical parameters and is under intensive investigation as an effective tool in metrology and for addressing foundational questions in quantum mechanics. Most of the experimental reports on weak measurements till date have employed external symmetric Gaussian pointers. Here, we demonstrate its universal nature in a system involving asymmetric spectral response of Fano resonance as the pointer arising naturally in precisely designed metamaterials, namely, waveguided plasmonic crystals. The weak coupling arises due to a tiny shift in the asymmetric spectral response between two orthogonal linear polarizations. By choosing the pre- and post-selected polarization states to be nearly mutually orthogonal, we observe both real and imaginary weak value amplifications manifested as spectacular shift of the peak frequency of Fano resonance and narrowing (or broadening) of the resonance line width, respectively. Weak value amplification using asymmetric Fano spectral response broadens the domain of applicability of weak measurements using natural spectral line shapes as pointer in wide range of physical systems.

Information on the polarization properties of scattered light from plasmonic systems are of paramount importance due to fundamental interest and potential applications. However, such studies are severely compromised due to the experimental difficulties in recording full polarization response of plasmonic nanostructures. Here, we report on a novel Mueller matrix spectroscopic system capable of acquiring complete polarization information from single isolated plasmonic nanoparticle/nanostructure. The outstanding issues pertaining to reliable measurements of full 4X4 spectroscopic scattering Mueller matrices from single nanoparticle/nanostructures are overcome by integrating an efficient Mueller matrix measurement scheme and a robust calibration method with a dark-field microscopic spectroscopy arrangement.The spectral polarization responses of the required polarization state generator, analyzer units, the imaging and the detection systemsare taken care off by eigenvalue calibration, thus enabling recording of the spectral polarization response (Mueller matrix) exclusively of the plasmonic system. Feasibility of quantitative Mueller matrix polarimetryand its potential utility is illustrated on a simple plasmonic system, that of gold nanorods. The demonstrated novel ability to record full polarization information over a broad wavelength range and to quantify the intrinsic plasmon polarimetry characteristics via Mueller matrix inverse analysis should lead to a novel route towards quantitative understanding, analysis / interpretation of a number of intricate plasmonic effects and may also prove useful towards development of polarization-controlled novel sensing schemes.

Spin orbit interaction and the resulting Spin Hall effect of light are under recent intensive investigations because of their fundamental nature and potential applications. Here, we report an extraordinary spin specific beam shift of light and demonstrate its tunability in an inhomogeneous anisotropic medium exhibiting spatially varying retardance level. The spin specificity (shift occurs only for one circular polarization mode, keeping the other orthogonal mode unaffected) is shown to arise due to the combined spatial gradients of the geometric phase and the dynamical phase of light. The constituent two orthogonal circular polarization modes of an input linearly polarized light evolve in different trajectories, eventually manifesting as a large and tunable spin separation. The spin specificity of the beam shift and the demonstrated principle of simultaneously tailoring space-varying geometric and dynamical phase of light for achieving its tunability (of both magnitude and direction), may provide an attractive route towards development of spin-optical devices., Comment: 17 pages, 4 figures