Search

Building on previous Bayesian approaches, we introduce a novel formulation of probabilistic cross-identification, where detections are directly associated to (hypothesized) astronomical objects in a globally optimal way. We show that this new method scales better for processing multiple catalogs than enumerating all possible candidates, especially in the limit of crowded fields, which is the most challenging observational regime for new-generation astronomy experiments such as the Rubin Observatory Legacy Survey of Space and Time. Here we study simulated catalogs where the ground truth is known and report on the statistical and computational performance of the method. The paper is accompanied by a public software tool to perform globally optimal catalog matching based on directional data.

Cross-matching catalogues from radio surveys to catalogues of sources at other wavelengths is extremely hard, because radio sources are often extended, often consist of several spatially separated components, and often no radio component is coincident with the optical/infrared host galaxy. Traditionally, the cross-matching is done by eye, but this does not scale to the millions of radio sources expected from the next generation of radio surveys. We present an innovative automated procedure, using Bayesian hypothesis testing, that models trial radio-source morphologies with putative positions of the host galaxy. This new algorithm differs from an earlier version by allowing more complex radio source morphologies, and performing a simultaneous fit over a large field. We show that this technique performs well in an unsupervised mode., 9 pages, 7 figures

The spectroscopic features of white dwarfs are formed in the thin upper layer of their stellar photosphere. These features carry information about the white dwarf's surface temperature, surface gravity, and chemical composition (hereafter 'labels'). Existing methods to determine these labels rely on complex ab-initio theoretical models which are not always publicly available. Here we present two techniques to determine atmospheric labels from white dwarf spectra: a generative fitting pipeline that interpolates theoretical spectra with artificial neural networks, and a random forest regression model using parameters derived from absorption line features. We test and compare our methods using a large catalog of white dwarfs from the Sloan Digital Sky Survey (SDSS), achieving the same accuracy and negligible bias compared to previous studies. We package our techniques into an open-source Python module 'wdtools' that provides a computationally inexpensive way to determine stellar labels from white dwarf spectra observed from any facility. We will actively develop and update our tool as more theoretical models become publicly available. We discuss applications of our tool in its present form to identify interesting outlier white dwarf systems including those with magnetic fields, helium-rich atmospheres, and double-degenerate binaries., 11 pages, 7 figures. Accepted for publication in MNRAS; updated references

In this article, the authors examine fundamental linkages between geolocation (human movement) data and financial market equity price behavior. The geolocation positions were recorded intraday and cover the period from January 1, 2018 to July 17, 2018. The authors concentrate their study on a popular hedge fund trading strategy known as pairs trading. First, the authors collect data regarding Under Armour (UA) and Nike’s stock market price and volume. Second, they investigate the volume of people visiting each physical store of UA and Nike, as proxied from anonymous cell phone geolocation traffic. Third, they monitor the relative activities for tweets related to UA and Nike. Fourth, after combining all the data, the authors glean the following fascinating results: (1) Geolocation information is proven to be an important factor in a pairs trading strategy between UA and Nike, according to the results of feature selection and prediction accuracy for price ratio change; (2) both ensemble methods of machine learning and rolling analysis could significantly raise the prediction accuracy; and (3) a pairs trading strategy incorporating geolocation information can have a cumulative return of 13.72% from January 2018 to June 2018, with an annualized Sharpe ratio of 3.88. TOPICS:Statistical methods, simulations, big data/machine learning Key Findings • Geolocation data can encapsulate the relative influx and outflux of consumers from department stores. This is an extremely useful feature in training machine learning models that can be used in lucrative pairs-trading strategies. • In the era of big data, it is tempting to throw every feature into a model and assume that the model will always figure out which features are most important. However, the model will usually perform better when trained on a subset of distinct features. • Ensemble methods are an effective means of eliminating bias in machine learning models because they use the output from a series of independently trained submodels to generate one balanced result.

In this work we examine the largest 100 cryptocurrency return series ranging from 2015 to early 2018. We concentrate our analysis on daily returns and find several interesting stylized facts. First, principal components analysis reveals a complex return generating process. As we examine our data in the most recent year, we find that surprisingly more than one principal component appears to explain the cross-sectional variation in returns. Second, similar to hedge fund returns, cryptocurrency returns suffer from the “beta-in-the-tails” hidden risk. Third, we find that predicting cryptocurrency movements with machine learning and artificial intelligence algorithms is marginally attractive with variation in predictability power per cryptocurrency. Fourth, lower volatile cryptocurrencies are slightly more predictable than more volatile ones. Fifth, evidence exists that efficacy of distinct information sets varies across machine learning algorithms, showing that predictability may be much more complex given a set of machine learning algorithms. Finally, short-term predictability is very tenuous, which suggests that near-term cryptocurrency markets are semi-strong form efficient and therefore, day trading cryptocurrencies may be very challenging. Keywords: cryptocurrency, blockchain, machine learning, bitcoin, beta-in-the-tails, risks

We describe a computational framework for hierarchical Bayesian inference with simple (typically single-plate) parametric graphical models that uses graphics processing units (GPUs) to accelerate computations, enabling deployment on very large datasets. Its C++ implementation, CUDAHM (CUDA for Hierarchical Models) exploits conditional independence between instances of a plate, facilitating massively parallel exploration of the replication parameter space using the single instruction, multiple data architecture of GPUs. It provides support for constructing Metropolis-within-Gibbs samplers that iterate between GPU-accelerated robust adaptive Metropolis sampling of plate-level parameters conditional on upper-level parameters, and Metropolis-Hastings sampling of upper-level parameters on the host processor conditional on the GPU results. CUDAHM is motivated by demographic problems in astronomy, where density estimation and linear and nonlinear regression problems must be addressed for populations of thousands to millions of objects whose features are measured with possibly complex uncertainties. We describe a thinned latent point process framework for modeling such demographic data. We demonstrate accurate GPU-accelerated parametric conditional density deconvolution for simulated populations of up to 300,000 objects in ~1 hour using a single NVIDIA Tesla K40c GPU. Supplementary material provides details about the CUDAHM API and the demonstration problem.

The relative timing of sleep and of eating within the circadian day is important for human health. Despite much data on sleep and a growing data set for eating, there remains a need for an interpretative framework for the understanding of this data for health decisions. This study provides a new statistical and machine learning analysis of more than 500 participants in the Daily24 project. From their data, and the analysis, we propose a framework for determining the classification of participants into different chronotypes and with that the ability to realize the potential impact of daily circadian habits on health. We propose that our resulting distribution curves could be used, similar to NHANES (National Health and Nutrition Examination Survey) data for pediatric growth, as a measure for circadian misalignment and used to help guide re-entrainment schedules.

We introduce a new method to estimate the probability that an extragalactic transient source is associated with a candidate host galaxy. This approach relies solely on simple observables: sky coordinates and their uncertainties, galaxy fluxes and angular sizes. The formalism invokes Bayes' rule to calculate the posterior probability P(O_i|x) from the galaxy prior P(O), observables x, and an assumed model for the true distribution of transients in/around their host galaxies. Using simulated transients placed in the well-studied COSMOS field, we consider several agnostic and physically motivated priors and offset distributions to explore the method sensitivity. We then apply the methodology to the set of 13~fast radio bursts (FRBs) localized with an uncertainty of several arcseconds. Our methodology finds nine of these are securely associated to a single host galaxy, P(O_i|x)>0.95. We examine the observed and intrinsic properties of these secure FRB hosts, recovering similar distributions as previous works. Furthermore, we find a strong correlation between the apparent magnitude of the securely identified host galaxies and the estimated cosmic dispersion measures of the corresponding FRBs, which results from the Macquart relation. Future work with FRBs will leverage this relation and other measures from the secure hosts as priors for future associations. The methodology is generic to transient type, localization error, and image quality. We encourage its application to other transients where host galaxy associations are critical to the science, e.g. gravitational wave events, gamma-ray bursts, and supernovae. We have encoded the technique in Python on GitHub: https://github.com/FRBs/astropath., In press, ApJ; comments still welcome; Visit https://github.com/FRBs/astropath to use and build PATH

We discuss the history and lessons learned from a series of deployments of environmental sensors measuring soil parameters and CO2 fluxes over the last fifteen years, in an outdoor environment. We present the hardware and software architecture of our current Gen-3 system, and then discuss how we are simplifying the user facing part of the software, to make it easier and friendlier for the environmental scientist to be in full control of the system. Finally, we describe the current effort to build a large-scale Gen-4 sensing platform consisting of hundreds of nodes to track the environmental parameters for urban green spaces in Baltimore, Maryland., 12 pages, 16 figures, Sensornets 2021 Conference

Cosmological N -body simulations play a vital role in studying models for the evolution of the Universe. To compare to observations and make a scientific inference, statistic analysis on large simulation datasets, e.g., finding halos, obtaining multi-point correlation functions, is crucial. However, traditional in-memory methods for these tasks do not scale to the datasets that are forbiddingly large in modern simulations. Our prior paper (Liu et al., 2015) proposes memory-efficient streaming algorithms that can find the largest halos in a simulation with up to 1 0 9 particles on a small server or desktop. However, this approach fails when directly scaling to larger datasets. This paper presents a robust streaming tool that leverages state-of-the-art techniques on GPU boosting, sampling, and parallel I/O, to significantly improve performance and scalability. Our rigorous analysis of the sketch parameters improves the previous results from finding the centers of the 1 0 3 largest halos (Liu et al., 2015) to ∼ 1 0 4 − 1 0 5 , and reveals the trade-offs between memory, running time and number of halos. Our experiments show that our tool can scale to datasets with up to ∼ 1 0 12 particles while using less than an hour of running time on a single GPU Nvidia GTX 1080.

Astronomical catalogues use the concept of quality flags to identify poor quality detections. For those cases, there is no certainty that this detection corresponds or not to an astronomical source. Normally, with future surveys using more advanced instruments, this uncertainty may be resolved. Here we present a new approach to better exploit the knowledge that can be extracted from a survey. This approach involves the combination of appropriate artificial intelligences methodologies, i.e. Bayesian inference, Voronoi tessellation and computer vision all this integrated in an automatic processing pipeline. When this pipeline is used with the COSMOS catalogue, apparent false detections are re-discovered as real sources. This demonstrates the benefits of using artificial intelligence methodologies in the exploitation of Big Data and the consequent knowledge discovery. The paper is structured as follows: section 1 presents an introduction of the problem and its context; section 2 describes the automatic processing pipeline and its specific use for the problem indicated in section 1; finally section 3 presents relevant results obtained with real data.

In 2017, Baltimore City was awarded $75 million earmarked for the targeted demolition of a portion of its 16,000 vacant and abandoned buildings. Selecting an optimal set of demolition targets is difficult given that the cost per demolition is not independent of the overall demolition pattern; like many older cities, Baltimore’s abandoned housing includes a large number of attached row houses, which require the construction of retaining walls when a demolished home abuts a non-demolished home. In this paper, we present a method by which planners can use integer linear programming to identify optimal demolition targets for a number of potential objectives. The simplest objective, demolishing the maximum number of houses for a specific budget, is compared with more complex functions that attempt to proxy improved quality of life resulting from the demolitions. The results of different objective functions are then assessed in terms of equity and efficiency using the spatial distribution of proposed targets as a point of comparison.

Institutional investors may have an unclear understanding of the role of social media in asset price determination. Although some of the top quant hedge funds use crowd-based information gleaned from tweets, this relationship may be opaque to the rest of our community. In this article, the authors attempt to clarify the confusion regarding social media sentiment and security return behavior. They show that, surprisingly, Tweet sentiments have significant power in explaining the time-series contemporaneous variation in daily stock returns, even in the presence of well-known equity factors. By examining direct tweet sentiments as provided by StockTwits, the authors claim to have identified a Social Media Factor , the “sixth” factor, and they highlight the distinctions vis-a-vis Fama–French’s factors.

We investigate the fundamental linkages between “geolocation” (human movement) data and financial market’s equity price behavior. The geolocation positions were recorded intraday and cover the period from January 1, 2018 to July 17, 2018. Our initial data set spans over 54 billion observations and was provided by Fysical. We focus our study on a popular hedge fund trading strategy known as “pairs trading.” First, we collect Under Armour (UA) and Nike’s stock price and volume data. Second, we investigate the relative activity of people visiting a particular physical store of UA and Nike, as proxied from anonymous cell phone foot traffic. Third, we collect the relative sentiment for tweets to UA and Nike. After combining all the data, we glean the following fascinating results: (1) geolocation information is an important variable in pairs trading strategy between UA and Nike, as evidenced by the results from our feature selection popularity methodology; (2) surprisingly, pairs trading only using geolocation data yields positive returns, and employing machine learning methods and rolling analysis enhances the returns; and (3) pairs trading strategy incorporating geolocation information yields a cumulative return of 13.7% from January, 2018 to June, 2018, with an Annualized Sharpe Ratio of 3.9. Some caveats are in order – our results are very sensitive to transaction costs assumptions and our data set has serious limitations.

A new data science initiative in materials research has been launched at The Johns Hopkins University within the Materials in Extreme Dynamic Environments (MEDE) Collaborative Research Alliance (CRA). Our first goal is to build a solution that facilitates seamless data sharing among MEDE scientists. We expect to shorten the design and development cycle of new materials by providing integrated storage, database, and analysis services, building on proven components of the SciServer project developed at the Institute for Data Intensive Engineering and Science (IDIES). Here we present our system design and demonstrate the power of our approach through a use-case that enables easy comparison of simulations and measurements. This prototype effort, focusing on boron carbide (BC), brings together multiple materials research elements in the Ceramics group within the MEDE CRA. The SciServer platform offers single-sign on access to various general purpose data analysis tools familiar to materials scientists in MEDE. During the case study deployment, users appreciated the simple data file upload process, automated database ingestion, and platform applicability to both students of the art and power users. From our case study experience in aggregating data from both simulations and physical experiments, we developed a template workflow from which a user may run a common data comparison task outright or customize to another purpose. Next, we turn to acquiring data from more MEDE groups and expanding the user base to the Metals group.

Errors have occurred during the production process. On page 15, Sect. 7, third paragraph, tenth line, the sentence should read: “The number of data points in the HCV light curves ranges from five (our cut-olimit) to 120.” And the citation “Spetsieri et al. 2019” should be “Spetsieri et al., in prep.”, in the first paragraph of Sect. 7.4 (page 19), and in Sect. 8, second to last paragraph (page 21).

Refraction by the atmosphere causes the positions of sources to depend on the airmass through which an observation was taken. This shift is dependent on the underlying spectral energy of the source and the filter or bandpass through which it is observed. Wavelength-dependent refraction within a single passband is often referred to as differential chromatic refraction (DCR). With a new generation of astronomical surveys undertaking repeated observations of the same part of the sky over a range of different airmasses and parallactic angles, DCR should be a detectable and measurable astrometric signal. In this paper we introduce a novel procedure that takes this astrometric signal and uses it to infer the underlying spectral energy distribution of a source; we solve for multiple latent images at specific wavelengths via a generalized deconvolution procedure built on robust statistics. We demonstrate the utility of such an approach for estimating a partially deconvolved image, at higher spectral resolution than the input images, for surveys such as the Large Synoptic Survey Telescope (LSST).

We release the AllWISE counterparts and Gaia matches to 106,573 and 17,665 X-ray sources detected in the ROSAT 2RXS and XMMSL2 surveys with |b|>15. These are the brightest X-ray sources in the sky, but their position uncertainties and the sparse multi-wavelength coverage until now rendered the identification of their counterparts a demanding task with uncertain results. New all-sky multi-wavelength surveys of sufficient depth, like AllWISE and Gaia, and a new Bayesian statistics based algorithm, NWAY, allow us, for the first time, to provide reliable counterpart associations. NWAY extends previous distance and sky density based association methods and, using one or more priors (e.g., colors, magnitudes), weights the probability that sources from two or more catalogues are simultaneously associated on the basis of their observable characteristics. Here, counterparts have been determined using a WISE color-magnitude prior. A reference sample of 4524 XMM/Chandra and Swift X-ray sources demonstrates a reliability of ~ 94.7% (2RXS) and 97.4% (XMMSL2). Combining our results with Chandra-COSMOS data, we propose a new separation between stars and AGN in the X-ray/WISE flux-magnitude plane, valid over six orders of magnitude. We also release the NWAY code and its user manual. NWAY was extensively tested with XMM-COSMOS data. Using two different sets of priors, we find an agreement of 96% and 99% with published Likelihood Ratio methods. Our results were achieved faster and without any follow-up visual inspection. With the advent of deep and wide area surveys in X-rays (e.g. SRG/eROSITA, Athena/WFI) and radio (ASKAP/EMU, LOFAR, APERTIF, etc.) NWAY will provide a powerful and reliable counterpart identification tool., MNRAS, Paper accepted for publication. Updated catalogs are available at www.mpe.mpg.de/XraySurveys/2RXS_XMMSL2 . NWAY available at https://github.com/JohannesBuchner/nway

SPIDERS (SPectroscopic IDentification of eROSITA Sources) is an SDSS-IV survey running in parallel to the eBOSS cosmology project. SPIDERS will obtain optical spectroscopy for large numbers of X-ray-selected AGN and galaxy cluster members detected in wide area eROSITA, XMM-Newton and ROSAT surveys. We describe the methods used to choose spectroscopic targets for two sub-programmes of SPIDERS: X-ray selected AGN candidates detected in the ROSAT All Sky and the XMM-Newton Slew surveys. We have exploited a Bayesian cross-matching algorithm, guided by priors based on mid-IR colour-magnitude information from the WISE survey, to select the most probable optical counterpart to each X-ray detection. We empirically demonstrate the high fidelity of our counterpart selection method using a reference sample of bright well-localised X-ray sources collated from XMM-Newton, Chandra and Swift-XRT serendipitous catalogues, and also by examining blank-sky locations. We describe the down-selection steps which resulted in the final set of SPIDERS-AGN targets put forward for spectroscopy within the eBOSS/TDSS/SPIDERS survey, and present catalogues of these targets. We also present catalogues of ~12000 ROSAT and ~1500 XMM-Newton Slew survey sources which have existing optical spectroscopy from SDSS-DR12, including the results of our visual inspections. On completion of the SPIDERS program, we expect to have collected homogeneous spectroscopic redshift information over a footprint of ~7500 deg$^2$ for >85 percent of the ROSAT and XMM-Newton Slew survey sources having optical counterparts in the magnitude range 17, Comment: MNRAS, accepted

Probabilistic cross-identification has been successfully applied to a number of problems in astronomy from matching simple point sources to associating stars with unknown proper motions and even radio observations with realistic morphology. Here we study the Bayes factor for clustered objects and focus in particular on galaxies to assess the effect of typical angular correlations. Numerical calculations provide the modified relationship, which (as expected) suppresses the evidence for the associations at the shortest separations where the 2-point auto-correlation function is large. Ultimately this means that the matching probability drops at somewhat shorter scales than in previous models., Comment: Accepted for publication in Astronomy and Computing, 6 pages, 3 figures

Observational astronomy in the time-domain era faces several new challenges. One of them is the efficient use of observations obtained at multiple epochs. The work presented here addresses faint object detection with multi-epoch data, and describes an incremental strategy for separating real objects from artifacts in ongoing surveys, in situations where the single-epoch data are summaries of the full image data, such as single-epoch catalogs of flux and direction estimates for candidate sources. The basic idea is to produce low-threshold single-epoch catalogs, and use a probabilistic approach to accumulate catalog information across epochs; this is in contrast to more conventional strategies based on co-added or stacked image data across all epochs. We adopt a Bayesian approach, addressing object detection by calculating the marginal likelihoods for hypotheses asserting there is no object, or one object, in a small image patch containing at most one cataloged source at each epoch. The object-present hypothesis interprets the sources in a patch at different epochs as arising from a genuine object; the no-object (noise) hypothesis interprets candidate sources as spurious, arising from noise peaks. We study the detection probability for constant-flux objects in a simplified Gaussian noise setting, comparing results based on single exposures and stacked exposures to results based on a series of single-epoch catalog summaries. Computing the detection probability based on catalog data amounts to generalized cross-matching: it is the product of a factor accounting for matching of the estimated fluxes of candidate sources, and a factor accounting for matching of their estimated directions. We find that probabilistic fusion of multi-epoch catalog information can detect sources with only modest sacrifice in sensitivity and selectivity compared to stacking., 11 pages, 11 figures

The Hubble Space Telescope (HST) has obtained multi-epoch observations providing the opportunity for a comprehensive variability search aiming to uncover new variables. We have therefore undertaken the task of creating a catalog of variable sources based on version 3 of the Hubble Source Catalog (HSC), which relies on publicly available images obtained with the WFPC2, ACS, and WFC3 instruments onboard the HST. We adopted magnitude-dependent thresholding in median absolute deviation (a robust measure of light curve scatter) combined with sophisticated preprocessing techniques and visual quality control to identify and validate variable sources observed by Hubble with the same instrument and filter combination five or more times. The Hubble Catalog of Variables (HCV) includes 84,428 candidate variable sources (out of 3.7 million HSC sources that were searched for variability) with $V \leq 27$ mag; for 11,115 of them the variability is detected in more than one filter. The data points in the light curves of the variables in the HCV catalog range from five to 120 points (typically having less than ten points); the time baseline ranges from under a day to over 15 years; while $\sim$8% of all variables have amplitudes in excess of 1 mag. Visual inspection performed on a subset of the candidate variables suggests that at least 80% of the candidate variables that passed our automated quality control are true variable sources rather than spurious detections resulting from blending, residual cosmic rays, and calibration errors. The HCV is the first, homogeneous catalog of variable sources created from archival HST data and currently is the deepest catalog of variables available. The catalog includes variable stars in our Galaxy and nearby galaxies, as well as transients and variable active galactic nuclei. (abbreviated), Comment: 33 pages, 15 figures, 11 tables; accepted in A&A. The HCV is available from the ESA Hubble Science Archive (eHST) (http://archives.esac.esa.int/ehst) at ESAC, and can be easily explored using the HCV Explorer (http://archives.esac.esa.int/hcv-explorer). The HCV is also hosted at STScI in the form of a High Level Science Product (HLSP) in MAST (https://archive.stsci.edu/hlsp/hcv)

Future radio surveys will generate catalogues of tens of millions of radio sources, for which redshift estimates will be essential to achieve many of the science goals. However, spectroscopic data will be available for only a small fraction of these sources, and in most cases even the optical and infrared photometry will be of limited quality. Furthermore, radio sources tend to be at higher redshift than most optical sources and so a significant fraction of radio sources hosts differ from those for which most photometric redshift templates are designed. We therefore need to develop new techniques for estimating the redshifts of radio sources. As a starting point in this process, we evaluate a number of machine-learning techniques for estimating redshift, together with a conventional template-fitting technique. We pay special attention to how the performance is affected by the incompleteness of the training sample and by sparseness of the parameter space or by limited availability of ancillary multi-wavelength data. As expected, we find that the quality of the photometric-redshift degrades as the quality of the photometry decreases, but that even with the limited quality of photometry available for all sky-surveys, useful redshift information is available for the majority of sources, particularly at low redshift. We find that a template-fitting technique performs best with high-quality and almost complete multi-band photometry, especially if radio sources that are also X-ray emitting are treated separately. When we reduced the quality of photometry to match that available for the EMU all-sky radio survey, the quality of the template-fitting degraded and became comparable to some of the machine learning methods. Machine learning techniques currently perform better at low redshift than at high redshift, because of incompleteness of the currently available training data at high redshifts., Comment: Submitted to PASP

Matching astronomical catalogs in crowded regions of the sky is challenging both statistically and computationally due to the many possible alternative associations. Budav\'ari and Basu (2016) modeled the two-catalog situation as an Assignment Problem and used the famous Hungarian algorithm to solve it. Here we treat cross-identification of multiple catalogs by introducing a different approach based on integer linear programming. We first test this new method on problems with two catalogs and compare with the previous results. We then test the efficacy of the new approach on problems with three catalogs. The performance and scalability of the new approach is discussed in the context of large surveys.

Astronomical discoveries often happen at the edge of our observational capabilities. To fully analyze telescopic images, researchers must combine data from separate telescopes, but large volumes of data with intrinsic differences make this difficult. SkyQuery, a scalable query engine, helps with this process.

One of the outstanding challenges of cross-identification is multiplicity: detections in crowded regions of the sky are often linked to more than one candidate associations of similar likelihoods. We map the resulting maximum likelihood partitioning to the fundamental assignment problem of discrete mathematics and efficiently solve the two-way catalog-level matching in the realm of combinatorial optimization using the so-called Hungarian algorithm. We introduce the method, demonstrate its performance in a mock universe where the true associations are known, and discuss the applicability of the new procedure to large surveys., 6 pages, 4 figures, accepted for publication in the Astronomical Journal

In the lead-up to the Square Kilometre Array (SKA) project, several next-generation radio telescopes and upgrades are already being built around the world. These include APERTIF (The Netherlands), ASKAP (Australia), eMERLIN (UK), VLA (USA), e-EVN (based in Europe), LOFAR (The Netherlands), Meerkat (South Africa), and the Murchison Widefield Array (MWA). Each of these new instruments has different strengths, and coordination of surveys between them can help maximise the science from each of them. A radio continuum survey is being planned on each of them with the primary science objective of understanding the formation and evolution of galaxies over cosmic time, and the cosmological parameters and large-scale structures which drive it. In pursuit of this objective, the different teams are developing a variety of new techniques, and refining existing ones. Here we describe these projects, their science goals, and the technical challenges which are being addressed to maximise the science return., Accepted by PASA, 22 October 2012

We present the methodology and data behind the photometric redshift database of the Sloan Digital Sky Survey Data Release 12 (SDSS DR12). We adopt a hybrid technique, empirically estimating the redshift via local regression on a spectroscopic training set, then fitting a spectrum template to obtain K-corrections and absolute magnitudes. The SDSS spectroscopic catalog was augmented with data from other, publicly available spectroscopic surveys to mitigate target selection effects. The training set is comprised of $1,976,978$ galaxies, and extends up to redshift $z\approx 0.8$, with a useful coverage of up to $z\approx 0.6$. We provide photometric redshifts and realistic error estimates for the $208,474,076$ galaxies of the SDSS primary photometric catalog. We achieve an average bias of $\overline{\Delta z_{\mathrm{norm}}} = 5.84 \times 10^{-5}$, a standard deviation of $\sigma \left(\Delta z_{\mathrm{norm}}\right)=0.0205$, and a $3\sigma$ outlier rate of $P_o=4.11\%$ when cross-validating on our training set. The published redshift error estimates and photometric error classes enable the selection of galaxies with high quality photometric redshifts. We also provide a supplementary error map that allows additional, sophisticated filtering of the data., Comment: 12 pages, 7 figures. Original submitted to MNRAS on 2016 March 03, revision submitted on 2016 April 21