Search

This review summarizes popular unsupervised learning methods, and gives an overview of their past, current, and future uses in astronomy. Unsupervised learning aims to organise the information content of a dataset, in such a way that knowledge can be extracted. Traditionally this has been achieved through dimensionality reduction techniques that aid the ranking of a dataset, for example through principal component analysis or by using auto-encoders, or simpler visualisation of a high dimensional space, for example through the use of a self organising map. Other desirable properties of unsupervised learning include the identification of clusters, i.e. groups of similar objects, which has traditionally been achieved by the k-means algorithm and more recently through density-based clustering such as HDBSCAN. More recently, complex frameworks have emerged, that chain together dimensionality reduction and clustering methods. However, no dataset is fully unknown. Thus, nowadays a lot of research has been directed towards self-supervised and semi-supervised methods that stand to gain from both supervised and unsupervised learning., Comment: 30 pages, 6 figures. Invited contribution to special issue in Astronomy & Computing

Driven by the imminent need to rapidly process and classify millions of AGN spectra drawn from next generation astronomical facilities, we present a spectral fitting routine for Type 2 AGN spectra optimised for high volume processing, using the Quasar Spectral Fitting library (QSFit). We analyse an optically selected sample of 813 luminous Type 2 AGN spectra at $z < 0.83$ from the Sloan Digital Sky Survey (SDSS) to qualify its performance. We report a median narrow line H$\alpha$/H$\beta$ Balmer decrement of 4.5$\pm$0.8, alluding to the presence of dust in the narrow line region (NLR). We publish a specialised QSFit fitting routine for high signal to noise ratio spectra and general fitting routine for double peaked Type 2 AGN spectra applied on a sub-sample of 45 spectra from our parent sample. We report a median red and blue peak velocity separation of 390$\pm$60kms$^{-1}$. No trend is found for red or blue peaks to exhibit systematically different luminosity or ionization properties. Emission line diagnostics show that the double peaks in all sources are illuminated by an AGN-powered ionizing continuum. Finally, we examine the morphology of host galaxies of our double peaked sample. We find double peaked Type 2 AGN reside in merging systems at a comparable frequency to single peaked AGN. This suggests that the double peaked AGN phenomenon is likely to have a bi-conical outflow origin in the majority of cases. We publicly release the code used for spectral analysis and produced catalogues used in this work., Comment: 19 pages, 19 figures, 2 appendices. Accepted for publication in MNRAS

To understand the Active Galactic Nuclei (AGN) phenomenon and their impact on the evolution of galaxies, a complete AGN census is required; however, finding heavily obscured AGNs is observationally challenging. Here we use the deep and extensive multi-wavelength data in the COSMOS field to select a complete sample of 578 infrared (IR) quasars ($L_{\rm AGN,IR}>10^{45}\rm \: erg\: s^{-1}$) at $z<3$, with minimal obscuration bias, using detailed UV-to-far IR spectral energy distribution (SED) fitting. We complement our SED constraints with X-ray and radio observations to further investigate the properties of the sample. Overall, 322 of the IR quasars are detected by Chandra and have individual X-ray spectral constraints. From a combination of X-ray stacking and $L_{\rm 2-10\rm keV}$ - $L_{\rm 6\: \mu m}$ analyses, we show that the majority of the X-ray faint and undetected quasars are heavily obscured (many are likely Compton thick), highlighting the effectiveness of the mid-IR band to find obscured AGNs. We find that 355 ($\approx$61%) IR quasars are obscured ($N_{\rm H}>10^{22}\rm \: cm^{-2}$) and identify differences in the average properties between the obscured and unobscured quasars: (1) obscured quasars have star-formation rates $\approx 3$ times higher than unobscured systems for no significant difference in stellar mass and (2) obscured quasars have stronger radio emission than unobscured systems, with a radio-loudness parameter $\approx 0.2 \rm \: dex$ higher. These results are inconsistent with a simple orientation model but in general agreement with either extreme host-galaxy obscuration towards the obscured quasars or a scenario where obscured quasars are an early phase in the evolution of quasars., Comment: Accepted for publication in MNRAS. 22 pages, 15 figures, 4 tables

This report is the result of a joint discussion between the Rubin and Euclid scientific communities. The work presented in this report was focused on designing and recommending an initial set of Derived Data products (DDPs) that could realize the science goals enabled by joint processing. All interested Rubin and Euclid data rights holders were invited to contribute via an online discussion forum and a series of virtual meetings. Strong interest in enhancing science with joint DDPs emerged from across a wide range of astrophysical domains: Solar System, the Galaxy, the Local Volume, from the nearby to the primaeval Universe, and cosmology., Comment: Report of the Rubin-Euclid Derived Data Products Working Group, 78 pages, 11 figures

AstronomicAL is a human-in-the-loop interactive labelling and training dashboard that allows users to create reliable datasets and robust classifiers using active learning. This technique prioritises data that offer high information gain, leading to improved performance using substantially less data. The system allows users to visualise and integrate data from different sources and deal with incorrect or missing labels and imbalanced class sizes. AstronomicAL enables experts to visualise domain-specific plots and key information relating both to broader context and details of a point of interest drawn from a variety of data sources, ensuring reliable labels. In addition, AstronomicAL provides functionality to explore all aspects of the training process, including custom models and query strategies. This makes the software a tool for experimenting with both domain-specific classifications and more general-purpose machine learning strategies. We illustrate using the system with an astronomical dataset due to the field's immediate need; however, AstronomicAL has been designed for datasets from any discipline. Finally, by exporting a simple configuration file, entire layouts, models, and assigned labels can be shared with the community. This allows for complete transparency and ensures that the process of reproducing results is effortless, Comment: 7 pages, 4 figures, Journal of Open Source Software

We present a pipeline based on a random forest classifier for the identification of high column-density clouds of neutral hydrogen (i.e. the Lyman limit systems, LLSs) in absorption within large spectroscopic surveys of z>3 quasars. We test the performance of this method on mock quasar spectra that reproduce the expected data quality of the Dark Energy Spectroscopic Instrument (DESI) and the WHT Enhanced Area Velocity Explorer (WEAVE) surveys, finding >90% completeness and purity for N(HI)> 10^17.2 cm^-2 LLSs against quasars of g < 23 mag at z~3.5-3.7. After training and applying our method on 10,000 quasar spectra at z~3.5-4.0 from the Sloan Digital Sky Survey (Data Release 16), we identify ~6600 LLSs with N(HI)>10^17.5 cm^-2 between z~3.1-4.0 with a completeness and purity of >90% for the classification of LLSs. Using this sample, we measure a number of LLSs per unit redshift of 2.32 +/- 0.08 at z=[3.3,3.6]. We also present results on the performance of random forest for the measurement of the LLS redshifts and HI column densities, and for the identification of broad absorption line quasars., Comment: 12 pages, 13 figures. MNRAS in press. The classification of the full dataset is available as supplementary data at MNARS online

Classification will be an important first step for upcoming surveys that will detect billions of new sources such as LSST and Euclid, as well as DESI, 4MOST and MOONS. The application of traditional methods of model fitting and colour-colour selections will face significant computational constraints, while machine-learning (ML) methods offer a viable approach to tackle datasets of that volume. While supervised learning methods can perform very well for classification tasks, the creation of representative and accurate training sets is a resource and time consuming task. We present a viable alternative using an unsupervised ML method to separate stars, galaxies and QSOs using photometric data. The heart of our work uses HDBSCAN to find the star, galaxy and QSO clusters in a multidimensional colour space. We optimized the hyperparameters and input attributes of three separate HDBSCAN runs, each to select a particular object class, and thus treat the output of each separate run as a binary classifier. We subsequently consolidate the output to give our final classifications, optimized on their F1 scores. We explore the use of Random Forest and PCA as part of the pre-processing stage for feature selection and dimensionality reduction. Using our dataset of 50,000 spectroscopically labelled objects we obtain an F1 score of 98.9, 98.9 and 93.13 respectively for star, galaxy and QSO selection using our unsupervised learning method. We find that careful attribute selection is a vital part of accurate classification with HDBSCAN. We applied our classification to a subset of the SDSS spectroscopic catalogue and demonstrate the potential of our approach in correcting misclassified spectra useful for DESI and 4MOST. Finally, we create a multiwavelength catalogue of 2.7 million sources using the KiDS, VIKING and ALLWISE surveys and publish corresponding classifications and photometric redshifts., Comment: Accepted to A&A 12 November 2019

A minority of the optically selected quasar population are red at optical wavelengths due to the presence of dust along the line-of-sight. A key focus of many red quasar studies is to understand their relationship with the overall quasar population: are they blue quasars observed at a (slight) inclination angle or do they represent a transitional phase in the evolution of quasars? Identifying fundamental differences between red and blue quasars is key to discriminate between these two paradigms. To robustly explore this, we have uniformly selected quasars from the Sloan Digital Sky Survey with mid-infrared counterparts, carefully controlling for luminosity and redshift effects. We take a novel approach to distinguish between colour-selected quasars in the redshift range of $0.2 < z < 2.4$ by constructing redshift-sensitive $g^* - i^*$ colour cuts. From cross-matching this sample to the Faint Images of the Radio Sky at Twenty-centimeters (FIRST) survey, we have found a factor $\approx$ 3 larger fraction of radio-detected red quasars with respect to that of blue quasars. Through a visual inspection of the FIRST images and an assessment of the radio luminosities (rest-frame ${L_{\rm 1.4~GHz}}$ and ${L_{\rm 1.4~GHz}}/{L_{\rm 6\mu m}}$) we find that the radio-detection excess for red quasars is primarily due to compact and radio-faint systems (around the radio quiet--radio loud threshold). We show that our results rule out orientation as the origin for the differences between red and blue quasars and argue that they provide broad agreement with an evolutionary model., Comment: Accepted to be published in MNRAS: 21 pages, 16 figures, and 6 tables

The Complete Calibration of the Color-Redshift Relation (C3R2) survey is a multi-institution, multi-instrument survey that aims to map the empirical relation of galaxy color to redshift to i~24.5 (AB), thereby providing a firm foundation for weak lensing cosmology with the Stage IV dark energy missions Euclid and WFIRST. Here we present 3171 new spectroscopic redshifts obtained in the 2016B and 2017A semesters with a combination of DEIMOS, LRIS, and MOSFIRE on the Keck telescopes. The observations come from all of the Keck partners: Caltech, NASA, the University of Hawaii, and the University of California. Combined with the 1283 redshifts published in DR1, the C3R2 survey has now obtained and published 4454 high quality galaxy redshifts. We discuss updates to the survey design and provide a catalog of photometric and spectroscopic data. Initial tests of the calibration method performance are given, indicating that the sample, once completed and combined with extensive data collected by other spectroscopic surveys, should allow us to meet the cosmology requirements for Euclid, and make significant headway toward solving the problem for WFIRST. We use the full spectroscopic sample to demonstrate that galaxy brightness is weakly correlated with redshift once a galaxy is localized in the Euclid or WFIRST color space, with potentially important implications for the spectroscopy needed to calibrate redshifts for faint WFIRST and LSST sources., Comment: ApJ accepted. Survey website with links to the C3R2 redshift catalog and spectroscopic data hosted by KOA can be found at https://sites.google.com/view/c3r2-survey/home

In this paper we present the optical, near-infrared (NIR) and X-ray identifications of the 6287 radio sources detected in the 2.1 GHz deep radio survey down to a median rms of ~ 41microJy/beam obtained with the Australia Telescope Compact Array (ATCA) in the XXL-S field. The goal of this paper is to provide a multi wavelength catalogue of the counterparts of the radio sources to be used in further studies. For the optical and NIR identification of the radio sources, we used the likelihood ratio (LR) technique, slightly modified in order to take into account the presence of a large number of relatively bright counterparts close to the radio sources. This procedure led to the identification of optical/NIR counterparts for 4770 different radio sources (~77% of the whole radio sample), 414 of which also have an X-ray counterpart. This fraction of identification is in agreement with previous radio-optical association studies at a similar optical magnitude depth, but is relatively low in comparison to recent work conducted in other radio fields using deeper optical and NIR data. The analysis of optical and NIR properties of radio sources shows that, regardless of the radio flux limit of a radio survey, the nature of the identified sources is strongly dependent on the depth of the optical/NIR used in the identification process. Only with deep enough optical/NIR data will we be able to identify a significant fraction of radio sources with red (z_{DEC}-K) counterparts whose radio emission is dominated by nuclear activity rather than starburst activity., Comment: 11 pages, 11 figures, Accepted for publication in A&A

The classification of the host galaxies of the radio sources in the 25 deg$^2$ ultimate XMM extragalactic survey south field (XXL-S) is presented. XXL-S was surveyed at 2.1 GHz with the Australia Telescope Compact Array (ATCA) and is thus far the largest area radio survey conducted down to rms flux densities of $\sigma \sim 41$ $\mu$Jy beam$^{-1}$. Of the 6287 radio sources in XXL-S, 4758 (75.7%) were cross-matched to an optical counterpart using the likelihood ratio technique. There are 1110 spectroscopic redshifts and 3648 photometric redshifts available for the counterparts, of which 99.4% exist out to $z \sim 4$. A number of multiwavelength diagnostics, including X-ray luminosities, mid-infrared colours, spectral energy distribution fits, radio luminosities, and optical emission lines and colours, were used to classify the sources into three types: low-excitation radio galaxies (LERGs), high-excitation radio galaxies (HERGs), and star-forming galaxies (SFGs). The final sample contains 1729 LERGs (36.3%), 1159 radio-loud HERGs (24.4%), 296 radio-quiet HERGs (6.2%), 558 SFGs (11.7%), and 1016 unclassified sources (21.4%). The LERGs tend to exist in the most massive galaxies with low star formation rates and redder colours, whereas the HERGs and SFGs exist in galaxies of lower mass, higher star formation rates, and bluer colours. The fraction of blue host galaxies is higher for radio-quiet HERGs than for radio-loud HERGs. LERGs and radio-loud HERGs are found at all radio luminosities, but radio-loud HERGs tend to be more radio luminous than LERGs at a given redshift. These results are consistent with the emerging picture in which LERGs exist in the most massive quiescent galaxies typically found in clusters with hot X-ray halos and HERGs are associated with ongoing star formation in their host galaxies via the accretion of cold gas., Comment: 27 pages, 31 figures

The 2.1 GHz radio source catalogue of the 25 deg$^2$ ultimate XMM extragalactic survey south (XXL-S) field, observed with the Australia Telescope Compact Array (ATCA), is presented. The final radio mosaic achieved a resolution of $\sim$$4.8"$ and a median rms noise of $\sigma \approx41$ $\mu$Jy/beam. To date, this is the largest area radio survey to reach this flux density level. A total of 6350 radio components above 5$\sigma$ are included in the component catalogue, 26.4% of which are resolved. Of these components, 111 were merged together to create 48 multiple-component radio sources, resulting in a total of 6287 radio sources in the source catalogue, 25.9% of which were resolved. A survival analysis revealed that the median spectral index of the Sydney University Molonglo Sky Survey (SUMSS) 843 MHz sources in the field is $\alpha$ = $-$0.75, consistent with the values of $-0.7$ to $-0.8$ commonly used to characterise radio spectral energy distributions of active galactic nuclei (AGN). The 2.1 GHz and 1.4 GHz differential radio source counts are presented and compared to other 1.4 GHz radio surveys. The XXL-S source counts show good agreement with the other surveys., Comment: 19 pages, 19 figures. Accepted for publication in A&A

Throughout the processing and analysis of survey data, a ubiquitous issue nowadays is that we are spoilt for choice when we need to select a methodology for some of its steps. The alternative methods usually fail and excel in different data regions, and have various advantages and drawbacks, so a combination that unites the strengths of all while suppressing the weaknesses is desirable. We propose to use a two-level hierarchy of learners. Its first level consists of training and applying the possible base methods on the first part of a known set. At the second level, we feed the output probability distributions from all base methods to a second learner trained on the remaining known objects. Using classification of variable stars and photometric redshift estimation as examples, we show that the hierarchical combination is capable of achieving general improvement over averaging-type combination methods, correcting systematics present in all base methods, is easy to train and apply, and thus, it is a promising tool in the astronomical "Big Data" era., Comment: 6 pages, 3 figures. To appear in the conference proceedings of the IAU Symposium 325 AstroInformatics (2016 October 20-24, Sorrento, Italy)

Euclid is a Europe-led cosmology space mission dedicated to a visible and near infrared survey of the entire extra-galactic sky. Its purpose is to deepen our knowledge of the dark content of our Universe. After an overview of the Euclid mission and science, this contribution describes how the community is getting organized to face the data analysis challenges, both in software development and in operational data processing matters. It ends with a more specific account of some of the main contributions of the Swiss Science Data Center (SDC-CH)., Comment: 10 pages, 4 figures, IAU Symposium 325 on Astroinformatics

The Astropy Project (http://astropy.org) is, in its own words, "a community effort to develop a single core package for Astronomy in Python and foster interoperability between Python astronomy packages." For five years this project has been managed, written, and operated as a grassroots, self-organized, almost entirely volunteer effort while the software is used by the majority of the astronomical community. Despite this, the project has always been and remains to this day effectively unfunded. Further, contributors receive little or no formal recognition for creating and supporting what is now critical software. This paper explores the problem in detail, outlines possible solutions to correct this, and presents a few suggestions on how to address the sustainability of general purpose astronomical software.

We present 2.1 GHz imaging with the Australia Telescope Compact Array (ATCA) of a 6.5 deg^2 region within the XXM-Newton XXL South field using a band of 1.1-3.1 GHz. We achieve an angular resolution of 4.7" x 4.2" in the final radio continuum map with a median rms noise level of 50 uJy/beam. We identify 1389 radio sources in the field with peak S/N >=5 and present the catalogue of observed parameters. We find that 305 sources are resolved, of which 77 consist of multiple radio components. These number counts are in agreement with those found for the COSMOS-VLA 1.4 GHz survey. We derive spectral indices by a comparison with the Sydney University Molongolo Sky Survey (SUMSS) 843MHz data. We find an average spectral index of -0.78 and a scatter of 0.28, in line with expectations. This pilot survey was conducted in preparation for a larger ATCA program to observe the full 25 deg^2 southern XXL field. When complete, the survey will provide a unique resource of sensitive, wide-field radio continuum imaging with complementary X-ray data in the field. This will facilitate studies of the physical mechanisms of radio-loud and radio-quiet AGNs and galaxy clusters, and the role they play in galaxy evolution. The source catalogue is publicly available online via the XXL Master Catalogue browser and the Centre de Donn\'ees astronomiques de Strasbourg (CDS)., Comment: 7 pages, 6 figures. Accepted by A&A 13th October 2015

Active Galactic Nuclei (AGN) play a decisive role in galaxy evolution, particularly so when operating in a radiatively inefficient mode, where they launch powerful jets that reshape their surroundings. However, identifying them is difficult, since radio observations commonly have resolutions of between 1 arcsec and 10 arcsec, which is equally sensitive to radio emission from star-forming activity and from AGN. Very Long Baseline Interferometry (VLBI) observations allow one to filter out all but the most compact non-thermal emission from radio survey data. The observational and computational demands to do this in large surveys have been, until recently, too high to make such undertakings feasible. Only the recent advent of wide-field observing techniques have facilitated such observations, and we here present the results from a survey of 217 radio sources in the Lockman Hole/XMM field. We describe in detail some new aspects of the calibration, including primary beam correction, multi-source self-calibration, and mosaicing. As a result, we detected 65 out of the 217 radio sources and were able to construct, for the first time, the source counts of VLBI-detected AGN. They indicate that at least 15%-25% of the sub-mJy radio sources are AGN-driven, consistent with recent findings using other AGN selection techniques. We have used ancillary data to investigate the AGN hosts. We find that among the sources nearby enough to be resolved in the optical images, 88% (23/26) could be classified as early-type or bulge-dominated galaxies. While 50% of these sources are correctly represented by the SED of an early-type galaxy, for the rest the best fit was obtained with a heavily extinct starburst template, an effect we ascribe to a degeneracy in the fit. Overall, the typical hosts of VLBI-detected sources are in good agreement with being early-type or bulge-dominated galaxies., Comment: 41 pages, accepted for publication in A&A

We have discovered an X-ray selected galaxy cluster with a spectroscopic redshift of 1.753. The redshift is of the brightest cluster galaxy (BCG), which is coincident with the peak of the X-ray surface brightness. We also have concordant photometric redshifts for seven additional candidate cluster members. The X-ray luminosity of the cluster is 3.68 +/- 0.70 x 10^43 erg s^-1 in the 0.1 - 2.4 keV band. The optical/IR properties of the BCG imply its formation redshift was ~5 if its stars formed in a short burst. This result continues the trend from lower redshift in which the observed properties of BCGs are most simply explained by a monolithic collapse at very high redshift instead of the theoretically preferred gradual hierarchical assembly at later times. However the models corresponding to different formation redshifts are more clearly separated as our observation epoch approaches the galaxy formation epoch. Although our infrared photometry is not deep enough to define a red sequence, we do identify a few galaxies at the cluster redshift that have the expected red sequence photometric properties., Comment: 18 pages, 8 figures, accepted for publication in the Astrophysical Journal