Search

The next generation of telescopes such as the SKA and the Rubin Observatory will produce enormous data sets, requiring automated anomaly detection to enable scientific discovery. Here, we present an overview and friendly user guide to the Astronomaly framework for active anomaly detection in astronomical data. Astronomaly uses active learning to combine the raw processing power of machine learning with the intuition and experience of a human user, enabling personalised recommendations of interesting anomalies. It makes use of a Python backend to perform data processing, feature extraction and machine learning to detect anomalous objects; and a JavaScript frontend to allow interaction with the data, labelling of interesting anomalous and active learning. Astronomaly is designed to be modular, extendable and run on almost any type of astronomical data. In this paper, we detail the structure of the Astronomaly code and provide guidelines for basic usage., Comment: 10 pages, 3 figures, to be published in Proc. ADASS XXXI (2021), coincides with v1.2 release of Astronomaly (https://github.com/MichelleLochner/astronomaly)

Radio Frequency Interference (RFI) is unwanted noise that swamps the desired astronomical signal. Radio astronomers have always had to deal with RFI detection and excision around telescope sites, but little has been done to understand the full scope, nature and evolution of RFI in a unified way. We undertake this for the MeerKAT array using a probabilistic multidimensional framework approach focussing on UHF-band and L-band data. In the UHF- band, RFI is dominated by the allocated Global System for Mobile (GSM) Communications, flight Distance Measuring Equipment (DME), and UHF-TV bands. The L-band suffers from known RFI sources such as DMEs, GSM, and the Global Positioning System (GPS) satellites. In the "clean" MeerKAT band, we noticed the RFI occupancy changing with time and direction for both the L-band and UHF band. For example, we saw a significant increase (300% increase) in the fraction of L-band flagged data in November 2018 compared to June 2018. This increase seems to correlate with construction activity on site. In the UHF-band, we found that the early morning is least impacted by RFI and other outliers. We also found a dramatic decrease in DME RFI during the hard lockdown due to the COVID-19 pandemic. The work presented here allows us to characterise the evolution of RFI at the MeerKAT site. Any observatory can adopt it to understand the behaviour of RFI within its surroundings.

Anomaly detection algorithms are typically applied to static, unchanging, data features hand-crafted by the user. But how does a user systematically craft good features for anomalies that have never been seen? Here we couple deep learning with active learning -- in which an Oracle iteratively labels small amounts of data selected algorithmically over a series of rounds -- to automatically and dynamically improve the data features for efficient outlier detection. This approach, AHUNT, shows excellent performance on MNIST, CIFAR10, and Galaxy-DESI data, significantly outperforming both standard anomaly detection and active learning algorithms with static feature spaces. Beyond improved performance, AHUNT also allows the number of anomaly classes to grow organically in response to Oracle's evaluations. Extensive ablation studies explore the impact of Oracle question selection strategy and loss function on performance. We illustrate how the dynamic anomaly class taxonomy represents another step towards fully personalized rankings of different anomaly classes that reflect a user's interests, allowing the algorithm to learn to ignore statistically significant but uninteresting outliers (e.g., noise). This should prove useful in the era of massive astronomical datasets serving diverse sets of users who can only review a tiny subset of the incoming data., Comment: 10 pages, 7 figures

Low-resource African languages have not fully benefited from the progress in neural machine translation because of a lack of data. Motivated by this challenge we compare zero-shot learning, transfer learning and multilingual learning on three Bantu languages (Shona, isiXhosa and isiZulu) and English. Our main target is English-to-isiZulu translation for which we have just 30,000 sentence pairs, 28% of the average size of our other corpora. We show the importance of language similarity on the performance of English-to-isiZulu transfer learning based on English-to-isiXhosa and English-to-Shona parent models whose BLEU scores differ by 5.2. We then demonstrate that multilingual learning surpasses both transfer learning and zero-shot learning on our dataset, with BLEU score improvements relative to the baseline English-to-isiZulu model of 9.9, 6.1 and 2.0 respectively. Our best model also improves the previous SOTA BLEU score by more than 10.

The Infection Fatality Rate (IFR) for COVID-19 is a poorly known, yet crucial, aspect of the disease. Counting only current deaths in a region and assuming everyone in that region is infected provides an absolute lower bound on the IFR. Using this estimator for New York City, Lombardy and Madrid yields strong bounds on the average IFR in overwhelmed health systems. Their combined 35, 152 deaths implies an IFR ≥ 0.14% averaged over 25.1 million people. This is the best-case scenario and conclusively demonstrates that COVID-19 is more deadly than influenza. The actual value of the average COVID-19 IFR is likely to be higher than this bound.

Radio Frequency Interference (RFI) has historically plagued radio astronomy, worsening with the rapid spread of electronics and increasing telescope sensitivity. We present a multi-dimensional probabilistic framework for characterising the RFI environment around a radio astronomy site that uses automatically flagged data from the array itself. We illustrate the framework using about 1500 hours of commissioning data from the MeerKAT radio telescope; producing a 6-dimensional array that yields both average RFI occupancy as well as confidence intervals around the mean as a function of key variables (frequency, direction, baseline, time). Our results provide the first detailed view of the MeerKAT RFI environment at high sensitivity as a function of direction, frequency, time of day and baseline. They allow us to track the historical evolution of the RFI and to quantify fluctuations which can be used for alerting on new RFI. As expected we find the major RFI contributors for MeerKAT site are from Global Positioning System (GPS) satellites, flight Distance Measurement Equipment (DME) and the Global System for Mobile (GSM) Communications. Beyond characterising RFI environments our approach allows observers access to the prior probability of RFI in any combination of tracked variables, allowing for more efficient observation planning and data excision.

Statistical uncertainties are rarely incorporated in machine learning algorithms, especially for anomaly detection. Here we present the Bayesian Anomaly Detection And Classification (BADAC) formalism, which provides a unified statistical approach to classification and anomaly detection within a hierarchical Bayesian framework. BADAC deals with uncertainties by marginalising over the unknown, true, value of the data. Using simulated data with Gaussian noise, BADAC is shown to be superior to standard algorithms in both classification and anomaly detection performance in the presence of uncertainties, though with significantly increased computational cost. Additionally, BADAC provides well-calibrated classification probabilities, valuable for use in scientific pipelines. We show that BADAC can work in online mode and is fairly robust to model errors, which can be diagnosed through model-selection methods. In addition it can perform unsupervised new class detection and can naturally be extended to search for anomalous subsets of data. BADAC is therefore ideal where computational cost is not a limiting factor and statistical rigour is important. We discuss approximations to speed up BADAC, such as the use of Gaussian processes, and finally introduce a new metric, the Rank-Weighted Score (RWS), that is particularly suited to evaluating the ability of algorithms to detect anomalies., Comment: 29 pages, 13 figures, Demo available: https://github.com/ethyroberts/BADAC

New telescopes like the Square Kilometre Array (SKA) will push into a new sensitivity regime and expose systematics, such as direction-dependent effects, that could previously be ignored. Current methods for handling such systematics rely on alternating best estimates of instrumental calibration and models of the underlying sky, which can lead to inadequate uncertainty estimates and biased results because any correlations between parameters are ignored. These deconvolution algorithms produce a single image that is assumed to be a true representation of the sky, when in fact it is just one realization of an infinite ensemble of images compatible with the noise in the data. In contrast, here we report a Bayesian formalism that simultaneously infers both systematics and science. Our technique, Bayesian Inference for Radio Observations (BIRO), determines all parameters directly from the raw data, bypassing image-making entirely, by sampling from the joint posterior probability distribution. This enables it to derive both correlations and accurate uncertainties, making use of the flexible software MEQTREES to model the sky and telescope simultaneously. We demonstrate BIRO with two simulated sets of Westerbork Synthesis Radio Telescope data sets. In the first, we perform joint estimates of 103 scientific (flux densities of sources) and instrumental (pointing errors, beamwidth and noise) parameters. In the second example, we perform source separation with BIRO. Using the Bayesian evidence, we can accurately select between a single point source, two point sources and an extended Gaussian source, allowing for 'super-resolution' on scales much smaller than the synthesized beam., Comment: Published in MNRAS. See https://vimeo.com/117391380 for a video of MultiNest converging to the correct source model

We outline a new method to compute the Bayes Factor for model selection which bypasses the Bayesian Evidence. Our method combines multiple models into a single, nested, Supermodel using one or more hyperparameters. Since the models are now nested the Bayes Factors between the models can be efficiently computed using the Savage-Dickey Density Ratio (SDDR). In this way model selection becomes a problem of parameter estimation. We consider two ways of constructing the supermodel in detail: one based on combined models, and a second based on combined likelihoods. We report on these two approaches for a Gaussian linear model for which the Bayesian evidence can be calculated analytically and a toy nonlinear problem. Unlike the combined model approach, where a standard Monte Carlo Markov Chain (MCMC) struggles, the combined-likelihood approach fares much better in providing a reliable estimate of the log-Bayes Factor. This scheme potentially opens the way to computationally efficient ways to compute Bayes Factors in high dimensions that exploit the good scaling properties of MCMC, as compared to methods such as nested sampling that fail for high dimensions., Comment: 24 pages, 11 Figures

There are two redshifts in cosmology: $z_{obs}$, the observed redshift computed via spectral lines, and the model redshift, $z$, defined by the effective FLRW scale factor. In general these do not coincide. We place observational constraints on the allowed distortions of $z$ away from $z_{obs}$ - a possibility we dub redshift remapping. Remapping is degenerate with cosmic dynamics for either $d_L(z)$ or $H(z)$ observations alone: for example, the simple remapping $z = \alpha_1 z_{obs} +\alpha_2 z_{obs}^2$ allows a decelerating Einstein de Sitter universe to fit the observed supernova Hubble diagram as successfully as $\Lambda$CDM, highlighting that supernova data alone cannot prove that the universe is accelerating. We show however, that redshift remapping leads to apparent violations of cosmic distance duality that can be used to detect its presence even when neither a specific theory of gravity nor the Copernican Principle are assumed. Combining current data sets favours acceleration but does not yet rule out redshift remapping as an alternative to dark energy. Future surveys, however, will provide exquisite constraints on remapping and any models -- such as backreaction -- that predict it., Comment: 12 pages 13 figures. Significantly generalised to discuss the case where distances and Hubble rate are arbitrarily deformed and General Relativity is not assumed

We present a measurement of the volumetric Type Ia supernova (SN Ia) rate based on data from the Sloan Digital Sky Survey II (SDSS-II) Supernova Survey. The adopted sample of supernovae (SNe) includes 516 SNe Ia at redshift z \lesssim 0.3, of which 270 (52%) are spectroscopically identified as SNe Ia. The remaining 246 SNe Ia were identified through their light curves; 113 of these objects have spectroscopic redshifts from spectra of their host galaxy, and 133 have photometric redshifts estimated from the SN light curves. Based on consideration of 87 spectroscopically confirmed non-Ia SNe discovered by the SDSS-II SN Survey, we estimate that 2.04+1.61-0.95 % of the photometric SNe Ia may be misidentified. The sample of SNe Ia used in this measurement represents an order of magnitude increase in the statistics for SN Ia rate measurements in the redshift range covered by the SDSS-II Supernova Survey. If we assume a SN Ia rate that is constant at low redshift (z < 0.15), then the SN observations can be used to infer a value of the SN rate of rV = (2.69+0.34+0.21-0.30-0.01) x10^{-5} SNe yr^{-1} Mpc-3 (H0 /(70 km s^{-1} Mpc^{-1}))^{3} at a mean redshift of ~ 0.12, based on 79 SNe Ia of which 72 are spectroscopically confirmed. However, the large sample of SNe Ia included in this study allows us to place constraints on the redshift dependence of the SN Ia rate based on the SDSS-II Supernova Survey data alone. Fitting a power-law model of the SN rate evolution, r_V(z) = A_p x ((1 + z)/(1 + z0))^��, over the redshift range 0.0 < z < 0.3 with z0 = 0.21, results in A_p = (3.43+0.15-0.15) x 10^{-5} SNe yr^{-1} Mpc-3 (H0 /(70 km s^{-1} Mpc^{-1}))^{3} and ��= 2.04+0.90-0.89., 80 pages. v2 updates a few references.

This is the Users' Manual for the Fisher Matrix software Fisher4Cast and covers installation, GUI help, command line basics, code flow and data structure, as well as cosmological applications and extensions. Finally we discuss the extensive tests performed on the software., Comment: 37 pages, 27 figures. Software available at http://www.cosmology.org.za. Cosmological applications described in the accompanying release paper

Baryon Acoustic Oscillations (BAO) are frozen relics left over from the pre-decoupling universe. They are the standard rulers of choice for 21st century cosmology, providing distance estimates that are, for the first time, firmly rooted in well-understood, linear physics. This review synthesises current understanding regarding all aspects of BAO cosmology, from the theoretical and statistical to the observational, and includes a map of the future landscape of BAO surveys, both spectroscopic and photometric., Comment: 42 Pages, 21 colour figures. Extended version of chapter contributed to the book "Dark Energy", Ed. P. Ruiz-Lapuente, Cambridge University Press. Comments and suggested references welcomed. Supplementary slides available at http://cosmology.org.za/BAO_review_slides.ppt

This paper describes the Sixth Data Release of the Sloan Digital Sky Survey. With this data release, the imaging of the northern Galactic cap is now complete. The survey contains images and parameters of roughly 287 million objects over 9583 deg², including scans over a large range of Galactic latitudes and longitudes. The survey also includes 1.27 million spectra of stars, galaxies, quasars, and blank sky (for sky subtraction) selected over 7425 deg². This release includes much more stellar spectroscopy than was available in previous data releases and also includes detailed estimates of stellar temperatures, gravities, and metallicities. The results of improved photometric calibration are now available, with uncertainties of roughly 1% in g, r, i, and z, and 2% in u, substantially better than the uncertainties in previous data releases. The spectra in this data release have improved wavelength and flux calibration, especially in the extreme blue and extreme red, leading to the qualitatively better determination of stellar types and radial velocities. The spectrophotometric fluxes are now tied to point-spread function magnitudes of stars rather than fiber magnitudes. This gives more robust results in the presence of seeing variations, but also implies a change in the spectrophotometric scale, which is now brighter by roughly 0.35 mag. Systematic errors in the velocity dispersions of galaxies have been fixed, and the results of two independent codes for determining spectral classifications and redshifts are made available. Additional spectral outputs are made available, including calibrated spectra from individual 15 minute exposures and the sky spectrum subtracted from each exposure. We also quantify a recently recognized underestimation of the brightnesses of galaxies of large angular extent due to poor sky subtraction; the bias can exceed 0.2 mag for galaxies brighter than r = 14 mag.

This paper presents spectroscopy of supernovae discovered in the first season of the Sloan Digital Sky Survey-II Supernova Survey. This program searches for and measures multi-band light curves of supernovae in the redshift range z = 0.05 - 0.4, complementing existing surveys at lower and higher redshifts. Our goal is to better characterize the supernova population, with a particular focus on SNe Ia, improving their utility as cosmological distance indicators and as probes of dark energy. Our supernova spectroscopy program features rapid-response observations using telescopes of a range of apertures, and provides confirmation of the supernova and host-galaxy types as well as precise redshifts. We describe here the target identification and prioritization, data reduction, redshift measurement, and classification of 129 SNe Ia, 16 spectroscopically probable SNe Ia, 7 SNe Ib/c, and 11 SNe II from the first season. We also describe our efforts to measure and remove the substantial host galaxy contamination existing in the majority of our SN spectra., Comment: Accepted for publication in The Astronomical Journal(47pages, 9 figures)

Vast sound waves traveling through the relativistic plasma during the first million years of the universe imprint a preferred scale in the density of matter. We now have the ability to detect this characteristic fingerprint in the clustering of galaxies at various redshifts and use it to measure the acceleration of the expansion of the Universe. The Wide-Field Multi-Object Spectrograph (WFMOS) would use this test to shed significant light on the true nature of dark energy, the mysterious source of this cosmic acceleration. WFMOS would also revolutionise studies of the kinematics of the Milky Way and provide deep insights into the clustering of galaxies at redshifts up to z~4. In this article we discuss the recent progress in large galaxy redshift surveys and detail how WFMOS will help unravel the mystery of dark energy., Comment: 6 pages, pure pdf. An introduction to WFMOS and Baryon Acoustic Oscillations for a general audience

Optimising the major next-generation cosmological surveys (such as {\em SNAP, KAOS etc...}) is a key problem given our ignorance of the physics underlying cosmic acceleration and the plethora of surveys planned. We propose a Bayesian design framework which (1) maximises the discrimination power of a survey without assuming any underlying dark energy model, (2) finds the best niche survey geometry given current data and future competing experiments, (3) maximises the cross-section for serendipitous discoveries and (4) can be adapted to answer specific questions (such as `is dark energy dynamical?'). Integrated Parameter Space Optimisation (IPSO) is a design framework that integrates projected parameter errors over an entire dark energy parameter space and then extremises a figure of merit (such as Shannon entropy gain which we show is stable to off-diagonal covariance matrix perturbations) as a function of survey parameters using analytical, grid or MCMC techniques. We discuss examples where the optimisation can be performed analytically. IPSO is thus a general, model-independent and scalable framework that allows us to appropriately use prior information to design the best possible surveys., Comment: 15 pages, 6 colour figures. Significantly more discussion and illustrative examples

There are two basic ways to measure physical distances in cosmology: One based on standard candles and one based on standard rulers. Comparing current data for each method allows us to rule out axion-photon mixing and dust-extinction as the sources of supernova dimming and generally protects the case for cosmic acceleration from attacks based on loss of photons. The combined data constrains the energy densities in a LCDM model to 0.19 < Omega_m < 0.32 and 0.47 < Omega_Lambda < 0.82 (at 2��) without recourse to any further data sets. Future data will improve on these limits and allow us to place constraints on more exotic physics., 4 pages, 1 figure, uses moriond.sty. To be published in the Proceedings of the XXXIX Rencontres de Moriond 'Exploring the Universe'

The dynamics of preheating after inflation has not been clearly understood yet.In particular, the issue of the generation of metric perturbations during preheating on super-horizon scale is still unsettled. Large scale perturbations may leave an imprint on the cosmic microwave background, or may become seeds for generation of primordial black holes. Hence, in order to make a connection between the particle physics models and cosmological observations, understanding the evolution of super-Hubble scale perturbations during preheating is important. Here, we propose an alternative treatment to handle this issue based on the so-called separate universe approach, which suggests less efficient amplification of super-Hubble modes during preheating than was expected before. We also point out an important issue which may have been overlooked in previous treatments., Comment: Proceedings of 12th JGRG meeting (Tokyo, Nov.25-28(2002))

Until now all well-studied dark energy models proposed have very large Compton wavelengths, $\lambda_c > 100 Mpc$ at all time and hence show no clustering on small scales. Evidence against this would disfavour both $\Lambda$CDM and quintessence. We show however, that large $\lambda_c$ is actually favoured by current CMB, LSS and SN1a data by comparing quintessence with a condensate scenario with the same background dynamics. The condensate scenario has a very small $\lambda_c$ before making a transition to negative pressure and large $\lambda_c$ and therefore exhibits a rich CMB phenomenology which roughly interpolates between $\Lambda$CDM and flat CDM models., Comment: 5 pages, 3 colour eps figures. Related to a talk given by BB at the XVIIIth IAP Colloquium "On the nature of Dark Energy", Paris, July 2002

We propose a new technique for detecting gravitational waves using Quantum Entangled STate (QUEST) technology. Gravitational waves reduce the non-locality of correlated quanta controlled by Bell's inequalities, distorting quantum encryption key statistics away from a pure white noise. Gravitational waves therefore act as shadow eavesdroppers. The resulting colour distortions can, at least in principle, be separated from noise and can differentiate both deterministic and stochastic sources., Comment: 10 pages, Revtex, 2 colour JPEG figures, high resolution versions of the figures and additional material available at http://sun1.sms.port.ac.uk/cosmos/users/bruce/mousetrap/

The gravitational wave equations form a parametric resonance system during oscillatory reheating after inflation, when cast in terms of the electric and magnetic parts of the Weyl tensor. This is in direct analogy with preheating. For chaotic inflation with a quadratic potential, this analogy is exact. The resulting amplification of the gravitational wave spectrum begins in the broad resonance regime for chaotic inflation but quickly moves towards narrow resonance. This increases the amplitude and breaks the scale invariance of the tensor spectrum generated during inflation, an effect which may be detectable. The parametric amplification is absent, however, in 1st-order phase transition models and "warm" inflationary models with continuous entropy production., Comment: 10 pages, Revtex, 3 ps figures. Matched to published version

We develop and apply a fully covariant 1+3 electromagnetic analogy for gravity. The free gravitational field is covariantly characterized by the Weyl gravito-electric and gravito-magnetic spatial tensor fields, whose dynamical equations are the Bianchi identities. Using a covariant generalization of spatial vector algebra and calculus to spatial tensor fields, we exhibit the covariant analogy between the tensor Bianchi equations and the vector Maxwell equations. We identify gravitational source terms, couplings and potentials with and without electromagnetic analogues. The nonlinear vacuum Bianchi equations are shown to be invariant under covariant spatial duality rotation of the gravito-electric and gravito-magnetic tensor fields. We construct the super-energy density and super-Poynting vector of the gravitational field as natural U(1) group invariants, and derive their super-energy conservation equation. A covariant approach to gravito-electric/magnetic monopoles is also presented., 14 pages. Version to appear in Class. Quant. Grav