Search

Deep learning has revolutionized the field of hyperspectral image (HSI) analysis, enabling the extraction of complex and hierarchical features. While convolutional neural networks (CNNs) have been the backbone of HSI classification, their limitations in capturing global contextual features have led to the exploration of Vision Transformers (ViTs). This paper introduces AMBER, an advanced SegFormer specifically designed for multi-band image segmentation. AMBER enhances the original SegFormer by incorporating three-dimensional convolutions to handle hyperspectral data. Our experiments, conducted on the Indian Pines, Pavia University, and PRISMA datasets, show that AMBER outperforms traditional CNN-based methods in terms of Overall Accuracy, Kappa coefficient, and Average Accuracy on the first two datasets, and achieves state-of-the-art performance on the PRISMA dataset., Comment: submitted to Neural Computing & Applications (Springer). Currently under review

Modern spectroscopic surveys can only target a small fraction of the vast amount of photometrically cataloged sources in wide-field surveys. Here, we report the development of a generative AI method capable of predicting optical galaxy spectra from photometric broad-band images alone. This method draws from the latest advances in diffusion models in combination with contrastive networks. We pass multi-band galaxy images into the architecture to obtain optical spectra. From these, robust values for galaxy properties can be derived with any methods in the spectroscopic toolbox, such as standard population synthesis techniques and Lick indices. When trained and tested on 64x64-pixel images from the Sloan Digital Sky Survey, the global bimodality of star-forming and quiescent galaxies in photometric space is recovered, as well as a mass-metallicity relation of star-forming galaxies. The comparison between the observed and the artificially created spectra shows good agreement in overall metallicity, age, Dn4000, stellar velocity dispersion, and E(B-V) values. Photometric redshift estimates of our generative algorithm can compete with other current, specialized deep-learning techniques. Moreover, this work is the first attempt in the literature to infer velocity dispersion from photometric images. Additionally, we can predict the presence of an active galactic nucleus up to an accuracy of 82%. With our method, scientifically interesting galaxy properties, normally requiring spectroscopic inputs, can be obtained in future data sets from large-scale photometric surveys alone. The spectra prediction via AI can further assist in creating realistic mock catalogs., Comment: Accepted by The Astrophysical Journal. Code is available at https://github.com/LarsDoorenbos/generate-spectra

We present a novel method for detecting outliers in astronomical time series based on the combination of a deep neural network and a k-nearest neighbor algorithm with the aim of identifying and removing problematic epochs in the light curves of astronomical objects. We use an EfficientNet network pre-trained on ImageNet as a feature extractor and perform a k-nearest neighbor search in the resulting feature space to measure the distance from the first neighbor for each image. If the distance is above the one obtained for a stacked image, we flag the image as a potential outlier. We apply our method to time series obtained from the VLT Survey Telescope (VST) monitoring campaign of the Deep Drilling Fields of the Vera C. Rubin Legacy Survey of Space and Time (LSST). We show that our method can effectively identify and remove artifacts from the VST time series and improve the quality and reliability of the data. This approach may prove very useful in sight of the amount of data that will be provided by the LSST, which will prevent the inspection of individual light curves. We also discuss the advantages and limitations of our method and suggest possible directions for future work., Comment: accepted for publication in A&A

Humanity has wondered whether we are alone for millennia. The discovery of life elsewhere in the Universe, particularly intelligent life, would have profound effects, comparable to those of recognizing that the Earth is not the center of the Universe and that humans evolved from previous species. There has been rapid growth in the fields of extrasolar planets and data-driven astronomy. In a relatively short interval, we have seen a change from knowing of no extrasolar planets to now knowing more potentially habitable extrasolar planets than there are planets in the Solar System. In approximately the same interval, astronomy has transitioned to a field in which sky surveys can generate 1 PB or more of data. The Data-Driven Approaches to Searches for the Technosignatures of Advanced Civilizations_ study at the W. M. Keck Institute for Space Studies was intended to revisit searches for evidence of alien technologies in light of these developments. Data-driven searches, being able to process volumes of data much greater than a human could, and in a reproducible manner, can identify *anomalies* that could be clues to the presence of technosignatures. A key outcome of this workshop was that technosignature searches should be conducted in a manner consistent with Freeman Dyson's "First Law of SETI Investigations," namely "every search for alien civilizations should be planned to give interesting results even when no aliens are discovered." This approach to technosignatures is commensurate with NASA's approach to biosignatures in that no single observation or measurement can be taken as providing full certainty for the detection of life. Areas of particular promise identified during the workshop were (*) Data Mining of Large Sky Surveys, (*) All-Sky Survey at Far-Infrared Wavelengths, (*) Surveys with Radio Astronomical Interferometers, and (*) Artifacts in the Solar System., Comment: Final Report prepared for the W. M. Keck Institute for Space Studies (KISS), http://kiss.caltech.edu/workshops/technosignatures/technosignatures.html ; eds. Lazio, Djorgovski, Howard, & Cutler; The study leads gratefully acknowledge the outstanding support of Michele Judd, KISS Executive Director, and her dedicated staff, who made the study experience invigorating and enormously productive

A trade-off between speed and information controls our understanding of astronomical objects. Fast-to-acquire photometric observations provide global properties, while costly and time-consuming spectroscopic measurements enable a better understanding of the physics governing their evolution. Here, we tackle this problem by generating spectra directly from photometry, through which we obtain an estimate of their intricacies from easily acquired images. This is done by using multi-modal conditional diffusion models, where the best out of the generated spectra is selected with a contrastive network. Initial experiments on minimally processed SDSS galaxy data show promising results., Comment: Accepted at NeurIPS 2022 Machine Learning and the Physical Sciences workshop

Modern sky surveys are producing ever larger amounts of observational data, which makes the application of classical approaches for the classification and analysis of objects challenging and time-consuming. However, this issue may be significantly mitigated by the application of automatic machine and deep learning methods. We propose ULISSE, a new deep learning tool that, starting from a single prototype object, is capable of identifying objects sharing the same morphological and photometric properties, and hence of creating a list of candidate sosia. In this work, we focus on applying our method to the detection of AGN candidates in a Sloan Digital Sky Survey galaxy sample, since the identification and classification of Active Galactic Nuclei (AGN) in the optical band still remains a challenging task in extragalactic astronomy. Intended for the initial exploration of large sky surveys, ULISSE directly uses features extracted from the ImageNet dataset to perform a similarity search. The method is capable of rapidly identifying a list of candidates, starting from only a single image of a given prototype, without the need for any time-consuming neural network training. Our experiments show ULISSE is able to identify AGN candidates based on a combination of host galaxy morphology, color and the presence of a central nuclear source, with a retrieval efficiency ranging from 21% to 65% (including composite sources) depending on the prototype, where the random guess baseline is 12%. We find ULISSE to be most effective in retrieving AGN in early-type host galaxies, as opposed to prototypes with spiral- or late-type properties. Based on the results described in this work, ULISSE can be a promising tool for selecting different types of astrophysical objects in current and future wide-field surveys (e.g. Euclid, LSST etc.) that target millions of sources every single night., Comment: Accepted for publication in A&A

In the era of large sky surveys, photometric redshifts (photo-z) represent crucial information for galaxy evolution and cosmology studies. In this work, we propose a new Machine Learning (ML) tool called Galaxy morphoto-Z with neural Networks (GaZNet-1), which uses both images and multi-band photometry measurements to predict galaxy redshifts, with accuracy, precision and outlier fraction superior to standard methods based on photometry only. As a first application of this tool, we estimate photo-z of a sample of galaxies in the Kilo-Degree Survey (KiDS). GaZNet-1 is trained and tested on $\sim140 000$ galaxies collected from KiDS Data Release 4 (DR4), for which spectroscopic redshifts are available from different surveys. This sample is dominated by bright (MAG$\_$AUTO$<21$) and low redshift ($z < 0.8$) systems, however, we could use $\sim$ 6500 galaxies in the range $0.8 < z < 3$ to effectively extend the training to higher redshift. The inputs are the r-band galaxy images plus the 9-band magnitudes and colours, from the combined catalogs of optical photometry from KiDS and near-infrared photometry from the VISTA Kilo-degree Infrared survey. By combining the images and catalogs, GaZNet-1 can achieve extremely high precision in normalized median absolute deviation (NMAD=0.014 for lower redshift and NMAD=0.041 for higher redshift galaxies) and low fraction of outliers ($0.4$\% for lower and $1.27$\% for higher redshift galaxies). Compared to ML codes using only photometry as input, GaZNet-1 also shows a $\sim 10-35$% improvement in precision at different redshifts and a $\sim$ 45% reduction in the fraction of outliers. We finally discuss that, by correctly separating galaxies from stars and active galactic nuclei, the overall photo-z outlier fraction of galaxies can be cut down to $0.3$\%., Comment: Accepted for publication in A&A

This paper presents the results of the Rubin Observatory Dark Energy Science Collaboration (DESC) 3x2pt tomography challenge, which served as a first step toward optimizing the tomographic binning strategy for the main DESC analysis. The task of choosing an optimal tomographic binning scheme for a photometric survey is made particularly delicate in the context of a metacalibrated lensing catalogue, as only the photometry from the bands included in the metacalibration process (usually riz and potentially g) can be used in sample definition. The goal of the challenge was to collect and compare bin assignment strategies under various metrics of a standard 3x2pt cosmology analysis in a highly idealized setting to establish a baseline for realistically complex follow-up studies; in this preliminary study, we used two sets of cosmological simulations of galaxy redshifts and photometry under a simple noise model neglecting photometric outliers and variation in observing conditions, and contributed algorithms were provided with a representative and complete training set. We review and evaluate the entries to the challenge, finding that even from this limited photometry information, multiple algorithms can separate tomographic bins reasonably well, reaching figures-of-merit scores close to the attainable maximum. We further find that adding the g band to riz photometry improves metric performance by ~15% and that the optimal bin assignment strategy depends strongly on the science case: which figure-of-merit is to be optimized, and which observables (clustering, lensing, or both) are included., Comment: 30 pages (incl. 12 in appendix), 12 figures. Version accepted for publication in the Open Journal of Astrophysics

In order to answer the open questions of modern cosmology and galaxy evolution theory, robust algorithms for calculating photometric redshifts (photo-z) for very large samples of galaxies are needed. Correct estimation of the various photo-z algorithms' performance requires attention to both the performance metrics and the data used for the estimation. In this work, we use the supervised machine learning algorithm MLPQNA to calculate photometric redshifts for the galaxies in the COSMOS2015 catalogue and the unsupervised Self-Organizing Maps (SOM) to determine the reliability of the resulting estimates. We find that for spec-z<1.2, photo-z predictions are on the same level of quality as SED fitting photo-z. We show that the SOM successfully detects unreliable spec-z that cause biases in the estimation of the photo-z algorithms' performance. Additionally, we use SOM to select the objects with reliable photo-z predictions. Our cleaning procedures allow to extract the subset of objects for which the quality of the final photo-z catalogs is improved by a factor of two, compared to the overall statistics., Comment: 26 pages, 15 figures, accepted for publication in MNRAS

We present a study of the active galactic nucleus (AGN) activity in the local Universe (z < 0.33) and its correlation with the host galaxy properties, derived from a Sloan Digital Sky Survey (SDSS DR8) sample with spectroscopic star-formation rate (SFR) and stellar mass ($\mathcal{M}_{\ast}$) determination. To quantify the level of AGN activity we used X-ray information from the XMM-Newton Serendipitous Source Catalogue (3XMM DR8). Applying multiwavelength AGN selection criteria (optical BPT-diagrams, X-ray/optical ratio etc) we found that 24% of the detected sources are efficiently-accreting AGN with moderate-to-high X-ray luminosity, which are twice as likely to be hosted by star-forming galaxies than by quiescent ones. The distribution of the specific Black Hole accretion rate (sBHAR, $\lambda_{\mathrm{sBHAR}}$) shows that nuclear activity in local, non-AGN dominated galaxies peaks at very low accretion rates ($-4 \lesssim \log\lambda_{\mathrm{sBHAR}} \lesssim -3$) in all stellar mass ranges. However, we observe systematically larger values of sBHAR for galaxies with active star-formation than for quiescent ones, as well as an increase of the mean $\lambda_{\mathrm{sBHAR}}$ with SFR for both star-forming and quiescent galaxies. These findings confirm the decreased level of AGN activity with cosmic time and are consistent with a scenario where both star-formation and AGN activity are fuelled by a common gas reservoir., Comment: 20 pages, 20 figures, accepted for publication in MNRAS

The current role of data-driven science is constantly increasing its importance within Astrophysics, due to the huge amount of multi-wavelength data collected every day, characterized by complex and high-volume information requiring efficient and as much as possible automated exploration tools. Furthermore, to accomplish main and legacy science objectives of future or incoming large and deep survey projects, such as JWST, LSST and Euclid, a crucial role is played by an accurate estimation of photometric redshifts, whose knowledge would permit the detection and analysis of extended and peculiar sources by disentangling low-z from high-z sources and would contribute to solve the modern cosmological discrepancies. The recent photometric redshift data challenges, organized within several survey projects, like LSST and Euclid, pushed the exploitation of multi-wavelength and multi-dimensional data observed or ad hoc simulated to improve and optimize the photometric redshifts prediction and statistical characterization based on both SED template fitting and machine learning methodologies. But they also provided a new impetus in the investigation on hybrid and deep learning techniques, aimed at conjugating the positive peculiarities of different methodologies, thus optimizing the estimation accuracy and maximizing the photometric range coverage, particularly important in the high-z regime, where the spectroscopic ground truth is poorly available. In such a context we summarize what learned and proposed in more than a decade of research., Comment: Preprint version of the manuscript to appear in the Volume: "The Physical Challenges of Astrostatistics", special issue of "Frontiers in Astronomy and Space Sciences"

The Probability Density Function (PDF) provides an estimate of the photometric redshift (zphot) prediction error. It is crucial for current and future sky surveys, characterized by strict requirements on the zphot precision, reliability and completeness. The present work stands on the assumption that properly defined rejection criteria, capable of identifying and rejecting potential outliers, can increase the precision of zphot estimates and of their cumulative PDF, without sacrificing much in terms of completeness of the sample. We provide a way to assess rejection through proper cuts on the shape descriptors of a PDF, such as the width and the height of the maximum PDF's peak. In this work we tested these rejection criteria to galaxies with photometry extracted from the Kilo Degree Survey (KiDS) ESO Data Release 4, proving that such approach could lead to significant improvements to the zphot quality: e.g., for the clipped sample showing the best trade-off between precision and completeness, we achieve a reduction in outliers fraction of $\simeq 75\%$ and an improvement of $\simeq 6\%$ for NMAD, with respect to the original data set, preserving the $\simeq 93\%$ of its content., Comment: Preprint version of the manuscript to appear in the Volume "Intelligent Astrophysics" of the series "Emergence, Complexity and Computation", Book eds. I. Zelinka, D. Baron, M. Brescia, Springer Nature Switzerland, ISSN: 2194-7287

Every field of Science is undergoing unprecedented changes in the discovery process, and Astronomy has been a main player in this transition since the beginning. The ongoing and future large and complex multi-messenger sky surveys impose a wide exploiting of robust and efficient automated methods to classify the observed structures and to detect and characterize peculiar and unexpected sources. We performed a preliminary experiment on KiDS DR4 data, by applying to the problem of anomaly detection two different unsupervised machine learning algorithms, considered as potentially promising methods to detect peculiar sources, a Disentangled Convolutional Autoencoder and an Unsupervised Random Forest. The former method, working directly on images, is considered potentially able to identify peculiar objects like interacting galaxies and gravitational lenses. The latter instead, working on catalogue data, could identify objects with unusual values of magnitudes and colours, which in turn could indicate the presence of singularities., Comment: Preprint version of the manuscript to appear in the Volume "Intelligent Astrophysics" of the series "Emergence, Complexity and Computation", Book eds. I. Zelinka, D. Baron, M. Brescia, Springer Nature Switzerland, ISSN: 2194-7287

Among the many challenges posed by the huge data volumes produced by the new generation of astronomical instruments there is also the search for rare and peculiar objects. Unsupervised outlier detection algorithms may provide a viable solution. In this work we compare the performances of six methods: the Local Outlier Factor, Isolation Forest, k-means clustering, a measure of novelty, and both a normal and a convolutional autoencoder. These methods were applied to data extracted from SDSS stripe 82. After discussing the sensitivity of each method to its own set of hyperparameters, we combine the results from each method to rank the objects and produce a final list of outliers., Comment: Preprint version of the accepted manuscript to appear in the Volume "Intelligent Astrophysics" of the series "Emergence, Complexity and Computation", Book eds. I. Zelinka, D. Baron, M. Brescia, Springer Nature Switzerland, ISSN: 2194-7287

Ultra-compact massive galaxies UCMGs, i.e. galaxies with stellar masses $M_{*} > 8 \times 10^{10} M_{\odot}$ and effective radii $R_{e} < 1.5$ kpc, are very rare systems, in particular at low and intermediate redshifts. Their origin as well as their number density across cosmic time are still under scrutiny, especially because of the paucity of spectroscopically confirmed samples. We have started a systematic census of UCMG candidates within the ESO Kilo Degree Survey, together with a large spectroscopic follow-up campaign to build the largest possible sample of confirmed UCMGs. This is the third paper of the series and the second based on the spectroscopic follow-up program. Here, we present photometrical and structural parameters of 33 new candidates at redshifts $0.15 \lesssim z \lesssim 0.5$ and confirm 19 of them as UCMGs, based on their nominal spectroscopically inferred $M_{*}$ and $R_{e}$. This corresponds to a success rate of $\sim 58\%$, nicely consistent with our previous findings. The addition of these 19 newly confirmed objects, allows us to fully assess the systematics on the system selection, and finally reduce the number density uncertainties. Moreover, putting together the results from our current and past observational campaigns and some literature data, we build the largest sample of UCMGs ever collected, comprising 92 spectroscopically confirmed objects at $0.1 \lesssim z \lesssim 0.5$. This number raises to 116, allowing for a $3\sigma$ tolerance on the $M_{*}$ and $R_{e}$ thresholds for the UCMG definition. For all these galaxies we have estimated the velocity dispersion values at the effective radii which have been used to derive a preliminary mass-velocity dispersion correlation.

In the last years, Astroinformatics has become a well defined paradigm for many fields of Astronomy. In this work we demonstrate the potential of a multidisciplinary approach to identify globular clusters (GCs) in the Fornax cluster of galaxies taking advantage of multi-band photometry produced by the VLT Survey Telescope using automatic self-adaptive methodologies. The data analyzed in this work consist of deep, multi-band, partially overlapping images centered on the core of the Fornax cluster. In this work we use a Neural-Gas model, a pure clustering machine learning methodology, to approach the GC detection, while a novel feature selection method ($\Phi$LAB) is exploited to perform the parameter space analysis and optimization. We demonstrate that the use of an Astroinformatics based methodology is able to provide GC samples that are comparable, in terms of purity and completeness with those obtained using single band HST data (Brescia et al. 2012) and two approaches based respectively on a morpho-photometric (Cantiello et al. 2018b) and a PCA analysis (D'Abrusco et al. 2015) using the same data discussed in this work., Comment: 29 pages, 14 figures

Despite the high accuracy of photometric redshifts (zphot) derived using Machine Learning (ML) methods, the quantification of errors through reliable and accurate Probability Density Functions (PDFs) is still an open problem. First, because it is difficult to accurately assess the contribution from different sources of errors, namely internal to the method itself and from the photometric features defining the available parameter space. Second, because the problem of defining a robust statistical method, always able to quantify and qualify the PDF estimation validity, is still an open issue. We present a comparison among PDFs obtained using three different methods on the same data set: two ML techniques, METAPHOR (Machine-learning Estimation Tool for Accurate PHOtometric Redshifts) and ANNz2, plus the spectral energy distribution template fitting method, BPZ. The photometric data were extracted from the KiDS (Kilo Degree Survey) ESO Data Release 3, while the spectroscopy was obtained from the GAMA (Galaxy and Mass Assembly) Data Release 2. The statistical evaluation of both individual and stacked PDFs was done through quantitative and qualitative estimators, including a dummy PDF, useful to verify whether different statistical estimators can correctly assess PDF quality. We conclude that, in order to quantify the reliability and accuracy of any zphot PDF method, a combined set of statistical estimators is required., Comment: Accepted for publication by MNRAS, 20 pages, 14 figures

Global Stellar Formation Rates or SFRs are crucial to constrain theories of galaxy formation and evolution. SFR's are usually estimated via spectroscopic observations which require too much previous telescope time and therefore cannot match the needs of modern precision cosmology. We therefore propose a novel method to estimate SFRs for large samples of galaxies using a variety of supervised ML models., Comment: ESANN 2018 - Proceedings, ISBN-13 9782875870483

The explosion of data in recent years has generated an increasing need for new analysis techniques in order to extract knowledge from massive datasets. Machine learning has proved particularly useful to perform this task. Fully automatized methods have recently gathered great popularity, even though those methods often lack physical interpretability. In contrast, feature based approaches can provide both well-performing models and understandable causalities with respect to the correlations found between features and physical processes. Efficient feature selection is an essential tool to boost the performance of machine learning models. In this work, we propose a forward selection method in order to compute, evaluate, and characterize better performing features for regression and classification problems. Given the importance of photometric redshift estimation, we adopt it as our case study. We synthetically created 4,520 features by combining magnitudes, errors, radii, and ellipticities of quasars, taken from the SDSS. We apply a forward selection process, a recursive method in which a huge number of feature sets is tested through a kNN algorithm, leading to a tree of feature sets. The branches of the tree are then used to perform experiments with the random forest, in order to validate the best set with an alternative model. We demonstrate that the sets of features determined with our approach improve the performances of the regression models significantly when compared to the performance of the classic features from the literature. The found features are unexpected and surprising, being very different from the classic features. Therefore, a method to interpret some of the found features in a physical context is presented. The methodology described here is very general and can be used to improve the performance of machine learning models for any regression or classification task., Comment: 21 pages, 11 figures, accepted for publication on A&A, final version after language revision

Within scientific and real life problems, classification is a typical case of extremely complex tasks in data-driven scenarios, especially if approached with traditional techniques. Machine Learning supervised and unsupervised paradigms, providing self-adaptive and semi-automatic methods, are able to navigate into large volumes of data characterized by a multi-dimensional parameter space, thus representing an ideal method to disentangle classes of objects in a reliable and efficient way. In Astrophysics, the identification of candidate Globular Clusters through deep, wide-field, single band images, is one of such cases where self-adaptive methods demonstrated a high performance and reliability. Here we experimented some variants of the known Neural Gas model, exploring both supervised and unsupervised paradigms of Machine Learning for the classification of Globular Clusters. Main scope of this work was to verify the possibility to improve the computational efficiency of the methods to solve complex data-driven problems, by exploiting the parallel programming with GPU framework. By using the astrophysical playground, the goal was to scientifically validate such kind of models for further applications extended to other contexts., Comment: 15 pages, 3 figures, to appear in the Volume of Springer Communications in Computer and Information Science (CCIS). arXiv admin note: substantial text overlap with arXiv:1710.03900

Astronomy has entered the big data era and Machine Learning based methods have found widespread use in a large variety of astronomical applications. This is demonstrated by the recent huge increase in the number of publications making use of this new approach. The usage of machine learning methods, however is still far from trivial and many problems still need to be solved. Using the evaluation of photometric redshifts as a case study, we outline the main problems and some ongoing efforts to solve them., Comment: 13 pages, 3 figures, Springer's Communications in Computer and Information Science (CCIS), Vol. 822

In Astrophysics, the identification of candidate Globular Clusters through deep, wide-field, single band HST images, is a typical data analytics problem, where methods based on Machine Learning have revealed a high efficiency and reliability, demonstrating the capability to improve the traditional approaches. Here we experimented some variants of the known Neural Gas model, exploring both supervised and unsupervised paradigms of Machine Learning, on the classification of Globular Clusters, extracted from the NGC1399 HST data. Main focus of this work was to use a well-tested playground to scientifically validate such kind of models for further extended experiments in astrophysics and using other standard Machine Learning methods (for instance Random Forest and Multi Layer Perceptron neural network) for a comparison of performances in terms of purity and completeness., Comment: Proceedings of the XIX International Conference "Data Analytics and Management in Data Intensive Domains" (DAMDID/RCDL 2017), Moscow, Russia, October 10-13, 2017, 8 pages, 4 figures

Photometric redshifts (photo-z's) provide an alternative way to estimate the distances of large samples of galaxies and are therefore crucial to a large variety of cosmological problems. Among the various methods proposed over the years, supervised machine learning (ML) methods capable to interpolate the knowledge gained by means of spectroscopical data have proven to be very effective. METAPHOR (Machine-learning Estimation Tool for Accurate PHOtometric Redshifts) is a novel method designed to provide a reliable PDF (Probability density Function) of the error distribution of photometric redshifts predicted by ML methods. The method is implemented as a modular workflow, whose internal engine for photo-z estimation makes use of the MLPQNA neural network (Multi Layer Perceptron with Quasi Newton learning rule), with the possibility to easily replace the specific machine learning model chosen to predict photo-z's. After a short description of the software, we present a summary of results on public galaxy data (Sloan Digital Sky Survey - Data Release 9) and a comparison with a completely different method based on Spectral Energy Distribution (SED) template fitting., Comment: 2016 IEEE Symposium Series on Computational Intelligence, SSCI 2016 7849953

In the current data-driven science era, it is needed that data analysis techniques has to quickly evolve to face with data whose dimensions has increased up to the Petabyte scale. In particular, being modern astrophysics based on multi-wavelength data organized into large catalogues, it is crucial that the astronomical catalog cross-matching methods, strongly dependant from the catalogues size, must ensure efficiency, reliability and scalability. Furthermore, multi-band data are archived and reduced in different ways, so that the resulting catalogues may differ each other in formats, resolution, data structure, etc, thus requiring the highest generality of cross-matching features. We present $C^{3}$ (Command-line Catalogue Cross-match), a multi-platform application designed to efficiently cross-match massive catalogues from modern surveys. Conceived as a stand-alone command-line process or a module within generic data reduction/analysis pipeline, it provides the maximum flexibility, in terms of portability, configuration, coordinates and cross-matching types, ensuring high performance capabilities by using a multi-core parallel processing paradigm and a sky partitioning algorithm., Comment: 6 pages, 4 figures, proceedings of the IAU-325 symposium on Astroinformatics, Cambridge University press

We present METAPHOR (Machine-learning Estimation Tool for Accurate PHOtometric Redshifts), a method able to provide a reliable PDF for photometric galaxy redshifts estimated through empirical techniques. METAPHOR is a modular workflow, mainly based on the MLPQNA neural network as internal engine to derive photometric galaxy redshifts, but giving the possibility to easily replace MLPQNA with any other method to predict photo-z's and their PDF. We present here the results about a validation test of the workflow on the galaxies from SDSS-DR9, showing also the universality of the method by replacing MLPQNA with KNN and Random Forest models. The validation test include also a comparison with the PDF's derived from a traditional SED template fitting method (Le Phare)., Comment: proceedings of the International Astronomical Union, IAU-325 symposium, Cambridge University press

Euclid is a Europe-led cosmology space mission dedicated to a visible and near infrared survey of the entire extra-galactic sky. Its purpose is to deepen our knowledge of the dark content of our Universe. After an overview of the Euclid mission and science, this contribution describes how the community is getting organized to face the data analysis challenges, both in software development and in operational data processing matters. It ends with a more specific account of some of the main contributions of the Swiss Science Data Center (SDC-CH)., Comment: 10 pages, 4 figures, IAU Symposium 325 on Astroinformatics

In the modern galaxy surveys photometric redshifts play a central role in a broad range of studies, from gravitational lensing and dark matter distribution to galaxy evolution. Using a dataset of about 25,000 galaxies from the second data release of the Kilo Degree Survey (KiDS) we obtain photometric redshifts with five different methods: (i) Random forest, (ii) Multi Layer Perceptron with Quasi Newton Algorithm, (iii) Multi Layer Perceptron with an optimization network based on the Levenberg-Marquardt learning rule, (iv) the Bayesian Photometric Redshift model (or BPZ) and (v) a classical SED template fitting procedure (Le Phare). We show how SED fitting techniques could provide useful information on the galaxy spectral type which can be used to improve the capability of machine learning methods constraining systematic errors and reduce the occurrence of catastrophic outliers. We use such classification to train specialized regression estimators, by demonstrating that such hybrid approach, involving SED fitting and machine learning in a single collaborative framework, is capable to improve the overall prediction accuracy of photometric redshifts., Comment: 6 pages, 1 figure, proceedings of the International Astronomical Union, IAU-325 symposium, Cambridge University press

Photometric redshifts (photo-z's) are fundamental in galaxy surveys to address different topics, from gravitational lensing and dark matter distribution to galaxy evolution. The Kilo Degree Survey (KiDS), i.e. the ESO public survey on the VLT Survey Telescope (VST), provides the unprecedented opportunity to exploit a large galaxy dataset with an exceptional image quality and depth in the optical wavebands. Using a KiDS subset of about 25,000 galaxies with measured spectroscopic redshifts, we have derived photo-z's using i) three different empirical methods based on supervised machine learning, ii) the Bayesian Photometric Redshift model (or BPZ), and iii) a classical SED template fitting procedure (Le Phare). We confirm that, in the regions of the photometric parameter space properly sampled by the spectroscopic templates, machine learning methods provide better redshift estimates, with a lower scatter and a smaller fraction of outliers. SED fitting techniques, however, provide useful information on the galaxy spectral type which can be effectively used to constrain systematic errors and to better characterize potential catastrophic outliers. Such classification is then used to specialize the training of regression machine learning models, by demonstrating that a hybrid approach, involving SED fitting and machine learning in a single collaborative framework, can be effectively used to improve the accuracy of photo-z estimates., Comment: Accepted by MNRAS, 17 pages, 11 figures

The emerging need for efficient, reliable and scalable astronomical catalog cross-matching is becoming more pressing in the current data-driven science era, where the size of data has rapidly increased up to the Petabyte scale. C3 (Command-line Catalogue Cross-matching) is a multi-platform tool designed to efficiently cross-match massive catalogues from modern astronomical surveys, ensuring high-performance capabilities through the use of a multi-core parallel processing paradigm. The tool has been conceived to be executed as a stand-alone command-line process or integrated within any generic data reduction/analysis pipeline, providing the maximum flexibility to the end user, in terms of parameter configuration, coordinates and cross-matching types. In this work we present the architecture and the features of the tool. Moreover, since the modular design of the tool enables an easy customization to specific use cases and requirements, we present also an example of a customized C3 version designed and used in the FP7 project ViaLactea, dedicated to cross-correlate Hi-GAL clumps with multi-band compact sources., Comment: 4 pages, to appear in the Proceedings of ADASS 2016, Astronomical Society of the Pacific (ASP) Conference Series

The most valuable asset of a space mission like Euclid are the data. Due to their huge volume, the automatic quality control becomes a crucial aspect over the entire lifetime of the experiment. Here we focus on the design strategy for the Science Ground Segment (SGS) Data Quality Common Tools (DQCT), which has the main role to provide software solutions to gather, evaluate, and record quality information about the raw and derived data products from a primarily scientific perspective. The SGS DQCT will provide a quantitative basis for evaluating the application of reduction and calibration reference data, as well as diagnostic tools for quality parameters, flags, trend analysis diagrams and any other metadata parameter produced by the pipeline. In a large programme like Euclid, it is prohibitively expensive to process large amount of data at the pixel level just for the purpose of quality evaluation. Thus, all measures of quality at the pixel level are implemented in the individual pipeline stages, and passed along as metadata in the production. In this sense most of the tasks related to science data quality are delegated to the pipeline stages, even though the responsibility for science data quality is managed at a higher level. The DQCT subsystem of the SGS is currently under development, but its path to full realization will likely be different than that of other subsystems. Primarily because, due to a high level of parallelism and to the wide pipeline processing redundancy, for instance the mechanism of double Science Data Center for each processing function, the data quality tools have not only to be widely spread over all pipeline segments and data levels, but also to minimize the occurrences of potential diversity of solutions implemented for similar functions, ensuring the maximum of coherency and standardization for quality evaluation and reporting in the SGS., Comment: 4 pages, to appear in the Proceedings of ADASS 2016, Astronomical Society of the Pacific (ASP) Conference Series

Modern Astrophysics is based on multi-wavelength data organized into large and heterogeneous catalogues. Hence, the need for efficient, reliable and scalable catalogue cross-matching methods plays a crucial role in the era of the petabyte scale. Furthermore, multi-band data have often very different angular resolution, requiring the highest generality of cross-matching features, mainly in terms of region shape and resolution. In this work we present $C^{3}$ (Command-line Catalogue Cross-match), a multi-platform application designed to efficiently cross-match massive catalogues. It is based on a multi-core parallel processing paradigm and conceived to be executed as a stand-alone command-line process or integrated within any generic data reduction/analysis pipeline, providing the maximum flexibility to the end-user, in terms of portability, parameter configuration, catalogue formats, angular resolution, region shapes, coordinate units and cross-matching types. Using real data, extracted from public surveys, we discuss the cross-matching capabilities and computing time efficiency also through a direct comparison with some publicly available tools, chosen among the most used within the community, and representative of different interface paradigms. We verified that the $C^{3}$ tool has excellent capabilities to perform an efficient and reliable cross-matching between large datasets. Although the elliptical cross-match and the parametric handling of angular orientation and offset are known concepts in the astrophysical context, their availability in the presented command-line tool makes $C^{3}$ competitive in the context of public astronomical tools., Comment: 18 pages, 9 figures, Accepted for publication on PASP

A variety of fundamental astrophysical science topics require the determination of very accurate photometric redshifts (photo-z's). A wide plethora of methods have been developed, based either on template models fitting or on empirical explorations of the photometric parameter space. Machine learning based techniques are not explicitly dependent on the physical priors and able to produce accurate photo-z estimations within the photometric ranges derived from the spectroscopic training set. These estimates, however, are not easy to characterize in terms of a photo-z Probability Density Function (PDF), due to the fact that the analytical relation mapping the photometric parameters onto the redshift space is virtually unknown. We present METAPHOR (Machine-learning Estimation Tool for Accurate PHOtometric Redshifts), a method designed to provide a reliable PDF of the error distribution for empirical techniques. The method is implemented as a modular workflow, whose internal engine for photo-z estimation makes use of the MLPQNA neural network (Multi Layer Perceptron with Quasi Newton learning rule), with the possibility to easily replace the specific machine learning model chosen to predict photo-z's. We present a summary of results on SDSS-DR9 galaxy data, used also to perform a direct comparison with PDF's obtained by the Le Phare SED template fitting. We show that METAPHOR is capable to estimate the precision and reliability of photometric redshifts obtained with three different self-adaptive techniques, i.e. MLPQNA, Random Forest and the standard K-Nearest Neighbors models., Comment: Accepted from MNRAS, 17 pages, 16 figures

The VIALACTEA project has a work package dedicated to Tools and Infrastructure and, inside it, a task for the Database and Virtual Observatory Infrastructure. This task aims at providing an infrastructure to store all the resources needed by the, more purposely, scientific work packages of the project itself. This infrastructure includes a combination of: storage facilities, relational databases and web services on top of them, and has taken, as a whole, the name of VIALACTEA Knowledge Base (VLKB). This contribution illustrates the current status of this VLKB. It details the set of data resources put together; describes the database that allows data discovery through VO inspired metadata maintenance; illustrates the discovery, cutout and access services built on top of the former two for the users to exploit the data content., Comment: 11 pages, 1 figure, SPIE Astronomical Telescopes + Instrumentation 2016, Software and Cyberifrastructure for Astronomy IV, Conference Proceedings

Astronomy is undergoing through a methodological revolution triggered by an unprecedented wealth of complex and accurate data. DAMEWARE (DAta Mining & Exploration Web Application and REsource) is a general purpose, Web-based, Virtual Observatory compliant, distributed data mining framework specialized in massive data sets exploration with machine learning methods. We present the DAMEWARE (DAta Mining & Exploration Web Application REsource) which allows the scientific community to perform data mining and exploratory experiments on massive data sets, by using a simple web browser. DAMEWARE offers several tools which can be seen as working environments where to choose data analysis functionalities such as clustering, classification, regression, feature extraction etc., together with models and algorithms., Comment: User Manual of the DAMEWARE Web Application, 51 pages

Due to the necessity to evaluate photo-z for a variety of huge sky survey data sets, it seemed important to provide the astronomical community with an instrument able to fill this gap. Besides the problem of moving massive data sets over the network, another critical point is that a great part of astronomical data is stored in private archives that are not fully accessible on line. So, in order to evaluate photo-z it is needed a desktop application that can be downloaded and used by everyone locally, i.e. on his own personal computer or more in general within the local intranet hosted by a data center. The name chosen for the application is PhotoRApToR, i.e. Photometric Research Application To Redshift (Cavuoti et al. 2015, 2014; Brescia 2014b). It embeds a machine learning algorithm and special tools dedicated to preand post-processing data. The ML model is the MLPQNA (Multi Layer Perceptron trained by the Quasi Newton Algorithm), which has been revealed particularly powerful for the photo-z calculation on the base of a spectroscopic sample (Cavuoti et al. 2012; Brescia et al. 2013, 2014a; Biviano et al. 2013). The PhotoRApToR program package is available, for different platforms, at the official website (http://dame.dsf.unina.it/dame_photoz.html#photoraptor)., Comment: User Manual of the PhotoRaptor tool, 54 pages. arXiv admin note: substantial text overlap with arXiv:1501.06506

The exploitation of present and future synoptic (multi-band and multi-epoch) surveys requires an extensive use of automatic methods for data processing and data interpretation. In this work, using data extracted from the Catalina Real Time Transient Survey (CRTS), we investigate the classification performance of some well tested methods: Random Forest, MLPQNA (Multi Layer Perceptron with Quasi Newton Algorithm) and K-Nearest Neighbors, paying special attention to the feature selection phase. In order to do so, several classification experiments were performed. Namely: identification of cataclysmic variables, separation between galactic and extra-galactic objects and identification of supernovae., Comment: Accepted by MNRAS, 11 figures, 18 pages

The VIALACTEA project aims at building a predictive model of star formation in our galaxy. We present the innovative integrated framework and the main technologies and methodologies to reach this ambitious goal., Comment: Astronomical Data Analysis Software and Systems XXV. Proceedings of a Conference held from October 25th to 30th, 2015 at Rydges World Square in Sydney, Australia

Calibrating the photometric redshifts of >10^9 galaxies for upcoming weak lensing cosmology experiments is a major challenge for the astrophysics community. The path to obtaining the required spectroscopic redshifts for training and calibration is daunting, given the anticipated depths of the surveys and the difficulty in obtaining secure redshifts for some faint galaxy populations. Here we present an analysis of the problem based on the self-organizing map, a method of mapping the distribution of data in a high-dimensional space and projecting it onto a lower-dimensional representation. We apply this method to existing photometric data from the COSMOS survey selected to approximate the anticipated Euclid weak lensing sample, enabling us to robustly map the empirical distribution of galaxies in the multidimensional color space defined by the expected Euclid filters. Mapping this multicolor distribution lets us determine where - in galaxy color space - redshifts from current spectroscopic surveys exist and where they are systematically missing. Crucially, the method lets us determine whether a spectroscopic training sample is representative of the full photometric space occupied by the galaxies in a survey. We explore optimal sampling techniques and estimate the additional spectroscopy needed to map out the color-redshift relation, finding that sampling the galaxy distribution in color space in a systematic way can efficiently meet the calibration requirements. While the analysis presented here focuses on the Euclid survey, similar analysis can be applied to other surveys facing the same calibration challenge, such as DES, LSST, and WFIRST., Comment: ApJ accepted, 17 pages, 10 figures

We estimated photometric redshifts (zphot) for more than 1.1 million galaxies of the ESO Public Kilo-Degree Survey (KiDS) Data Release 2. KiDS is an optical wide-field imaging survey carried out with the VLT Survey Telescope (VST) and the OmegaCAM camera, which aims at tackling open questions in cosmology and galaxy evolution, such as the origin of dark energy and the channel of galaxy mass growth. We present a catalogue of photometric redshifts obtained using the Multi Layer Perceptron with Quasi Newton Algorithm (MLPQNA) model, provided within the framework of the DAta Mining and Exploration Web Application REsource (DAMEWARE). These photometric redshifts are based on a spectroscopic knowledge base which was obtained by merging spectroscopic datasets from GAMA (Galaxy And Mass Assembly) data release 2 and SDSS-III data release 9. The overall 1 sigma uncertainty on Delta z = (zspec - zphot) / (1+ zspec) is ~ 0.03, with a very small average bias of ~ 0.001, a NMAD of ~ 0.02 and a fraction of catastrophic outliers (| Delta z | > 0.15) of ~0.4%., Comment: MNRAS, 6 pages, 4 figures

The Kilo-Degree Survey (KiDS) is an optical wide-field imaging survey carried out with the VLT Survey Telescope and the OmegaCAM camera. KiDS will image 1500 square degrees in four filters (ugri), and together with its near-infrared counterpart VIKING will produce deep photometry in nine bands. Designed for weak lensing shape and photometric redshift measurements, the core science driver of the survey is mapping the large-scale matter distribution in the Universe back to a redshift of ~0.5. Secondary science cases are manifold, covering topics such as galaxy evolution, Milky Way structure, and the detection of high-redshift clusters and quasars. KiDS is an ESO Public Survey and dedicated to serving the astronomical community with high-quality data products derived from the survey data, as well as with calibration data. Public data releases will be made on a yearly basis, the first two of which are presented here. For a total of 148 survey tiles (~160 sq.deg.) astrometrically and photometrically calibrated, coadded ugri images have been released, accompanied by weight maps, masks, source lists, and a multi-band source catalog. A dedicated pipeline and data management system based on the Astro-WISE software system, combined with newly developed masking and source classification software, is used for the data production of the data products described here. The achieved data quality and early science projects based on the data products in the first two data releases are reviewed in order to validate the survey data. Early scientific results include the detection of nine high-z QSOs, fifteen candidate strong gravitational lenses, high-quality photometric redshifts and galaxy structural parameters for hundreds of thousands of galaxies. (Abridged), Comment: 26 pages, 26 figures, 2 appendices; two new figures, several textual clarifications, updated references; accepted for publication in A&A

We present an innovative method called FilExSeC (Filaments Extraction, Selection and Classification), a data mining tool developed to investigate the possibility to refine and optimize the shape reconstruction of filamentary structures detected with a consolidated method based on the flux derivative analysis, through the column-density maps computed from Herschel infrared Galactic Plane Survey (Hi-GAL) observations of the Galactic plane. The present methodology is based on a feature extraction module followed by a machine learning model (Random Forest) dedicated to select features and to classify the pixels of the input images. From tests on both simulations and real observations the method appears reliable and robust with respect to the variability of shape and distribution of filaments. In the cases of highly defined filament structures, the presented method is able to bridge the gaps among the detected fragments, thus improving their shape reconstruction. From a preliminary "a posteriori" analysis of derived filament physical parameters, the method appears potentially able to add a sufficient contribution to complete and refine the filament reconstruction., Comment: Proceeding of WIRN 2015 Conference, May 20-22, Vietri sul Mare, Salerno, Italy. Published in Smart Innovation, Systems and Technology, Springer, ISSN 2190-3018, 9 pages, 4 figures

We discuss whether modern machine learning methods can be used to characterize the physical nature of the large number of objects sampled by the modern multi-band digital surveys. In particular, we applied the MLPQNA (Multi Layer Perceptron with Quasi Newton Algorithm) method to the optical data of the Sloan Digital Sky Survey - Data Release 10, investigating whether photometric data alone suffice to disentangle different classes of objects as they are defined in the SDSS spectroscopic classification. We discuss three groups of classification problems: (i) the simultaneous classification of galaxies, quasars and stars; (ii) the separation of stars from quasars; (iii) the separation of galaxies with normal spectral energy distribution from those with peculiar spectra, such as starburst or starforming galaxies and AGN. While confirming the difficulty of disentangling AGN from normal galaxies on a photometric basis only, MLPQNA proved to be quite effective in the three-class separation. In disentangling quasars from stars and galaxies, our method achieved an overall efficiency of 91.31% and a QSO class purity of ~95%. The resulting catalogue of candidate quasars/AGNs consists of ~3.6 million objects, of which about half a million are also flagged as robust candidates, and will be made available on CDS VizieR facility., Comment: Accepted for publication by MNRAS, 13 pages, 6 figures

Photometric redshifts (photo-z) are crucial to the scientific exploitation of modern panchromatic digital surveys. In this paper we present PhotoRApToR (Photometric Research Application To Redshift): a Java/C++ based desktop application capable to solve non-linear regression and multi-variate classification problems, in particular specialized for photo-z estimation. It embeds a machine learning algorithm, namely a multilayer neural network trained by the Quasi Newton learning rule, and special tools dedicated to pre- and postprocessing data. PhotoRApToR has been successfully tested on several scientific cases. The application is available for free download from the DAME Program web site., Comment: To appear on Experimental Astronomy, Springer, 20 pages, 15 figures