Search

Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to an ad-hoc operating point, making evaluation and comparison across different studies relatively difficult. Our work leverages: (1) the concept of operating characteristics curves and (2) the notion of a gain over a null reference, to derive a novel operating point agnostic assessment methodology for prediction intervals. The paper defines the Uncertainty Characteristics Curve and demonstrates its utility in selected scenarios. We argue that the proposed method addresses the current need for comprehensive assessment of prediction intervals and thus represents a valuable addition to the uncertainty quantification toolbox., Comment: Published at Workshop on Distribution-Free Uncertainty Quantification, International Conference on Machine Learning (ICML), July 2022. arXiv admin note: substantial text overlap with arXiv:2106.00858

Photons excited into ground state modes at finite temperature display partitioning among photon phases, lifetimes and distances travelled since creation. These distributions set the distance from an interface a created photon has some chance of emission. Excited photons at each frequency have a phase velocity set by each mode propagation index which determines mode density and internal energy contribution. All photons emitted after striking an interface obliquely are refracted. Their exit intensities are then irreversible except when weak internal attenuation occurs. At low temperature attenuation index is small so reversibility is approximate. As temperature rises refraction direction varies. Total emission remains reversible after transitioning through a non equilibrium state with no other heat inputs. In equilibrium the densities of excitations that create and annihilate photons are in balance with photon densities while emissivity depends on both indices and internal incident direction. Modelled exit intensities from pure water and crystalline silica contain strong resonant intensities and match data accurately. Intrinsic resonances formed within liquids and compounds are due to photon modes hybridising with localized excitations, including molecular oscillations and the anharmonic component of lattice distortions. They explain the many resonant spectral intensities seen in remote sensing. Each hybrid oscillator is a photonic virtual bound state whose energy fluctuates between levels separated by mode energy. Refraction induces solid angle changes and often anomalous refraction while thermal recycling of internally reflected photons modifies intensities and internal energy. Enhanced internal heat flux from phonon drag by photon density gradients under an external temperature gradient is also predicted., Comment: 27 pages, 5 figures

We theoretically and numerically study the efficiency of Brillouin-based opto-acoustic data storage in a photonic waveguide in the presence of thermal noise and laser phase noise. We compare the physics of the noise processes and how they affect different storage techniques, examining both amplitude and phase storage schemes. We investigate the effects of storage time and pulse properties on the quality of the retrieved signal, and find that phase storage is less sensitive to thermal noise than amplitude storage., Comment: 12 pages, 9 figures

We present a numerical method for modelling noise in Stimulated Brillouin Scattering (SBS). The model applies to dynamic cases such as optical pulses, and accounts for both thermal noise and phase noise from the input lasers. Using this model, we compute the statistical properties of the optical and acoustic power in the pulsed spontaneous and stimulated Brillouin cases, and investigate the effects of gain and pulse width on noise levels. We find that thermal noise plays an important role in the statistical properties of the fields, and that laser phase noise impacts the SBS interaction when the laser coherence time is close to the time-scale of the optical pulses. This algorithm is applicable to arbitrary waveguide geometries and material properties, and thus presents a versatile way of performing noise-based SBS numerical simulations, which are important in signal processing, sensing, microwave photonics and opto-acoustic memory storage., Comment: 10 pages, 12 figures

Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to a specific operating point, making evaluation and comparison across different studies difficult. Our work leverages: (1) the concept of operating characteristics curves and (2) the notion of a gain over a simple reference, to derive a novel operating point agnostic assessment methodology for prediction intervals. The paper describes the corresponding algorithm, provides a theoretical analysis, and demonstrates its utility in multiple scenarios. We argue that the proposed method addresses the current need for comprehensive assessment of prediction intervals and thus represents a valuable addition to the uncertainty quantification toolbox., Comment: 10 pages main paper, 9 pages appendix

Understanding model performance on unlabeled data is a fundamental challenge of developing, deploying, and maintaining AI systems. Model performance is typically evaluated using test sets or periodic manual quality assessments, both of which require laborious manual data labeling. Automated performance prediction techniques aim to mitigate this burden, but potential inaccuracy and a lack of trust in their predictions has prevented their widespread adoption. We address this core problem of performance prediction uncertainty with a method to compute prediction intervals for model performance. Our methodology uses transfer learning to train an uncertainty model to estimate the uncertainty of model performance predictions. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines. We believe this result makes prediction intervals, and performance prediction in general, significantly more practical for real-world use., Comment: 7+6 pages, 5 figures, AAAI 2021

We theoretically and numerically study the effects of thermal noise on pulses in backwards Stimulated Brillouin Scattering (SBS). Using a combination of stochastic calculus and numerical methods, we derive a theoretical model that can be used to quantitatively predict noise measurements. We study how the optical pulse configuration, including the input powers of the pump and Stokes fields, pulse durations and interaction time, affects the noise in the output Stokes field. We investigate the effects on the noise of the optical loss and waveguide length, and we find that the signal-to-noise ratio can be significantly improved, or reduced, for specific combinations of waveguide properties and pulse parameters., Comment: 15 pages, 8 figures

Building and maintaining high-quality test sets remains a laborious and expensive task. As a result, test sets in the real world are often not properly kept up to date and drift from the production traffic they are supposed to represent. The frequency and severity of this drift raises serious concerns over the value of manually labeled test sets in the QA process. This paper proposes a simple but effective technique that drastically reduces the effort needed to construct and maintain a high-quality test set (reducing labeling effort by 80-100% across a range of practical scenarios). This result encourages a fundamental rethinking of the testing process by both practitioners, who can use these techniques immediately to improve their testing, and researchers who can help address many of the open questions raised by this new approach., Comment: International Workshop on Challenges in Deploying and Monitoring Machine Learning Systems in Conjunction with ICML 2020

Generating high quality uncertainty estimates for sequential regression, particularly deep recurrent networks, remains a challenging and open problem. Existing approaches often make restrictive assumptions (such as stationarity) yet still perform poorly in practice, particularly in presence of real world non-stationary signals and drift. This paper describes a flexible method that can generate symmetric and asymmetric uncertainty estimates, makes no assumptions about stationarity, and outperforms competitive baselines on both drift and non drift scenarios. This work helps make sequential regression more effective and practical for use in real-world applications, and is a powerful new addition to the modeling toolbox for sequential uncertainty quantification in general.

Today's AI deployments often require significant human involvement and skill in the operational stages of the model lifecycle, including pre-release testing, monitoring, problem diagnosis and model improvements. We present a set of enabling technologies that can be used to increase the level of automation in AI operations, thus lowering the human effort required. Since a common source of human involvement is the need to assess the performance of deployed models, we focus on technologies for performance prediction and KPI analysis and show how they can be used to improve automation in the key stages of a typical AI operations pipeline.

Effective transport properties of heterogeneous structures are predicted by geometric microstructural parameters, but these can be difficult to calculate. Here, a boundary element code with a recurrent series method accurately and efficiently determines the high order parameters of polygonal and conformal prisms in regular two-dimensional lattices and Voronoi tessellations (VT). This reveals that proximity to simpler estimates is associated with: centroidal VT (cf random VT), compactness, and VT structures (cf similarly compact semi-regular lattices). An error in previously reported values for triangular lattices is noted.

Building on the foundation work of Brown, Milton and Torquato, we present a tractable approach to analyse the effective permittivity of anisotropic two-phase structures. This methodology accounts for successive dipolar interactions, providing a recurrent series expansion of the effective permittivity to arbitrary order. Within this framework, we also demonstrate a progressive method to determine tight bounds that converge towards the exact solution. We illustrate the utility of these methods by using ensemble averaging to determine the micro-structural parameters of anisotropic level-cut Gaussian fields. We find that the depolarization factor of these structures is equivalent to that of an isolated ellipse with the same stretchingratio, and discuss the contribution of the fourth order term to the exact anisotropy.

Accuracy is an important concern for suppliers of artificial intelligence (AI) services, but considerations beyond accuracy, such as safety (which includes fairness and explainability), security, and provenance, are also critical elements to engender consumers' trust in a service. Many industries use transparent, standardized, but often not legally required documents called supplier's declarations of conformity (SDoCs) to describe the lineage of a product along with the safety and performance testing it has undergone. SDoCs may be considered multi-dimensional fact sheets that capture and quantify various aspects of the product and its development to make it worthy of consumers' trust. Inspired by this practice, we propose FactSheets to help increase trust in AI services. We envision such documents to contain purpose, performance, safety, security, and provenance information to be completed by AI service providers for examination by consumers. We suggest a comprehensive set of declaration items tailored to AI and provide examples for two fictitious AI services in the appendix of the paper., Comment: 31 pages