Search

Observing celestial objects and advancing our scientific knowledge about them involves tedious planning, scheduling, data collection and data post-processing. Many of these operational aspects of astronomy are guided and executed by expert astronomers. Reinforcement learning is a mechanism where we (as humans and astronomers) can teach agents of artificial intelligence to perform some of these tedious tasks. In this paper, we will present a state of the art overview of reinforcement learning and how it can benefit astronomy., Comment: To appear, Astronomy & Computing

Sky models used in radio interferometric data processing primarily consist of compact and discrete radio sources. When there is a need to model large scale diffuse structure such as the Galaxy, specialized source models are sought after for the sake of simplicity and computational efficiency. In this paper, we propose the use of shapelet basis functions for modeling large scale diffuse structure in various radio interferometric data processing pipelines. Conventional source model construction using shapelet basis functions is restricted to using images of smaller size due to limitations in computational resources such as memory. We propose a novel shapelet decomposition method to lift this restriction, enabling the use of images of millions of pixels (as well as a wide spectral bandwidth) for building models of large scale diffuse structure. Furthermore, the application of direction dependent errors onto diffuse sky models is an expensive operation often performed as a convolution. In this paper, we propose using some specific properties of shapelet basis functions to apply such direction dependent errors as a product of the model coefficients, thus the avoiding the need for convolution. We provide results based on simulations and real observations. In order to measure the efficacy of our proposed method in modeling large scale diffuse structure, we consider direction dependent calibration of simulated as well as real LOFAR observations that have a significant amount of large scale diffuse structure. Results show that by including large scale shapelet models of the diffuse sky, we are able to overcome a major problem faced by existing calibration techniques that do not model such large scale diffuse structure, i.e., the suppression of such large scale diffuse structure due to model incompleteness., Comment: Accepted - A&A

Model based reinforcement learning has proven to be more sample efficient than model free methods. On the other hand, the construction of a dynamics model in model based reinforcement learning has increased complexity. Data processing tasks in radio astronomy are such situations where the original problem which is being solved by reinforcement learning itself is the creation of a model. Fortunately, many methods based on heuristics or signal processing do exist to perform the same tasks and we can leverage them to propose the best action to take, or in other words, to provide a `hint'. We propose to use `hints' generated by the environment as an aid to the reinforcement learning process mitigating the complexity of model construction. We modify the soft actor critic algorithm to use hints and use the alternating direction method of multipliers algorithm with inequality constraints to train the agent. Results in several environments show that we get the increased sample efficiency by using hints as compared to model free methods.

Direction dependent calibration of widefield radio interferometers estimates the systematic errors along multiple directions in the sky. This is necessary because with most systematic errors that are caused by effects such as the ionosphere or the receiver beam shape, there is significant spatial variation. Fortunately, there is some deterministic behavior of these variations in most situations. We enforce this underlying smooth spatial behavior of systematic errors as an additional constraint onto spectrally constrained direction dependent calibration. Using both analysis and simulations, we show that this additional spatial constraint improves the performance of multi-frequency direction dependent calibration., Comment: MNRAS Accepted 2021 December 12. Received 2021 December 11; in original form 2021 October 9

Modern radio telescopes produce unprecedented amounts of data, which are passed through many processing pipelines before the delivery of scientific results. Hyperparameters of these pipelines need to be tuned by hand to produce optimal results. Because many thousands of observations are taken during a lifetime of a telescope and because each observation will have its unique settings, the fine tuning of pipelines is a tedious task. In order to automate this process of hyperparameter selection in data calibration pipelines, we introduce the use of reinforcement learning. We test two reinforcement learning techniques, twin delayed deep deterministic policy gradient (TD3) and soft actor-critic (SAC), to train an autonomous agent to perform this fine tuning. For the sake of generalization, we consider the pipeline to be a black-box system where the summarized state of the performance of the pipeline is used by the autonomous agent. The autonomous agent trained in this manner is able to determine optimal settings for diverse observations and is therefore able to perform 'smart' calibration, minimizing the need for human intervention., Comment: MNRAS Accepted 2021 May 12. Received 2021 May 11; in original form 2021 February 5

Mitigation of radio frequency interference (RFI) is essential to deliver science-ready radio interferometric data to astronomers. In this paper, using dual polarized radio interferometers, we propose to use the polarization information of post-correlation interference signals to detect and mitigate them. We use the directional statistics of the polarized signals as the detection criteria and formulate a distributed, wideband spectrum sensing problem. Using consensus optimization, we solve this in an online manner, working with mini-batches of data. We present extensive results based on simulations to demonstrate the feasibility of our method., Comment: EUSIPCO 2020 accepted

With ever increasing data rates produced by modern radio telescopes like LOFAR and future telescopes like the SKA, many data processing steps are overwhelmed by the amount of data that needs to be handled using limited compute resources. Calibration is one such operation that dominates the overall data processing computational cost, nonetheless, it is an essential operation to reach many science goals. Calibration algorithms do exist that scale well with the number of stations of an array and the number of directions being calibrated. However, the remaining bottleneck is the raw data volume, which scales with the number of baselines, and which is proportional to the square of the number of stations. We propose a 'stochastic' calibration strategy where we only read in a mini-batch of data for obtaining calibration solutions, as opposed to reading the full batch of data being calibrated. Nonetheless, we obtain solutions that are valid for the full batch of data. Normally, data need to be averaged before calibration is performed to accommodate the data in size-limited compute memory. Stochastic calibration overcomes the need for data averaging before any calibration can be performed, and offers many advantages including: enabling the mitigation of faint radio frequency interference; better removal of strong celestial sources from the data; and better detection and spatial localization of fast radio transients., Comment: MNRAS Accepted 2020 March 2. Received 2020 March 2 ; in original form 2020 January 27

SAGECal has been designed to find the most accurate calibration solutions for low radio frequency imaging observations, with minimum artefacts due to incomplete sky models. SAGECAL is developed to handle extremely large datasets, e.g., when the number of frequency bands greatly exceeds the number of available nodes on a compute cluster. Accurate calibration solutions are derived at the expense of large computational loads, which require distributed computing and modern compute devices, such as GPUs, to decrease runtimes. In this work, we investigate if the GPU version of SAGECal scales well enough to meet the requirements for the Square Kilometre Array and we compare its performance with the CPU version., Comment: 4 pages, 4 figures, presented at ADASS 2019

We present a stochastic, limited-memory Broyden Fletcher Goldfarb Shanno (LBFGS) algorithm that is suitable for handling very large amounts of data. A direct application of this algorithm is radio interferometric calibration of raw data at fine time and frequency resolution. Almost all existing radio interferometric calibration algorithms assume that it is possible to fit the dataset being calibrated into memory. Therefore, the raw data is averaged in time and frequency to reduce its size by many orders of magnitude before calibration is performed. However, this averaging is detrimental for the detection of some signals of interest that have narrow bandwidth and time duration such as fast radio bursts (FRBs). Using the proposed algorithm, it is possible to calibrate data at such a fine resolution that they cannot be entirely loaded into memory, thus preserving such signals. As an additional demonstration, we use the proposed algorithm for training deep neural networks and compare the performance against the mainstream first order optimization algorithms that are used in deep learning., Comment: Draft, final version in IEEE Data Science Workshop 2019 proceedings

Calibration is an essential step in radio interferometric data processing that corrects the data for systematic errors and in addition, subtracts bright foreground interference to reveal weak signals hidden in the residual. These weak and unknown signals are much sought after to reach many science goals but the effect of calibration on such signals is an ever present concern. The main reason for this is the incompleteness of the model used in calibration. Distributed calibration based on consensus optimization has been shown to mitigate the effect due to model incompleteness by calibrating data covering a wide bandwidth in a computationally efficient manner. In this paper, we study the statistical performance of direction dependent distributed calibration, i.e., the distortion caused by calibration on the residual statistics. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We derive an analytical relationship for the influence of the input on the residual and use this to find the relationship between the input and output probability density functions. Using simulations we show that the smallest eigenvalue of the Jacobian of this mapping is a reliable indicator of the statistical performance of calibration. The analysis developed in this paper can also be applied to other data processing steps in radio interferometry such as imaging and foreground subtraction as well as to many other machine learning problems., Comment: MNRAS Accepted 2019 April 30. Received 2019 April 9; in original form 2018 December 6

Distributed calibration based on consensus optimization is a computationally efficient method to calibrate large radio interferometers such as LOFAR and SKA. Calibrating along multiple directions in the sky and removing the bright foreground signal is a crucial step in many science cases in radio interferometry. The residual data contain weak signals of huge scientific interest and of particular concern is the effect of incomplete sky models used in calibration on the residual. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We derive an analytical relationship between the input and output probability density functions which can be used to study the performance of calibration., Comment: published in the Proceedings of IEEE Sensor Array and Multichannel Signal Processing Workshop (IEEE SAM 2018), published by IEEE

Calibration of a typical radio interferometric array yields thousands of parameters as solutions. These solutions contain valuable information about the systematic errors in the data (ionosphere and beam shape). This information could be reused in calibration to improve the accuracy and also can be fed into imaging to improve the fidelity. We propose a distributed optimization strategy to construct models for the systematic errors in the data using the calibration solutions. We formulate this as an elastic net regularized distributed optimization problem which we solve using the alternating direction method of multipliers (ADMM) algorithm. We give simulation results to show the feasibility of the proposed distributed model construction scheme., Comment: Draft, to be published in the Proceedings of the 2018 International Conference on Acoustics, Speech and Signal Processing (IEEE ICASSP 2018), published by IEEE

New and upcoming radio interferometers will produce unprecedented amounts of data that demand extremely powerful computers for processing. This is a limiting factor due to the large computational power and energy costs involved. Such limitations restrict several key data processing steps in radio interferometry. One such step is calibration where systematic errors in the data are determined and corrected. Accurate calibration is an essential component in reaching many scientific goals in radio astronomy and the use of consensus optimization that exploits the continuity of systematic errors across frequency significantly improves calibration accuracy. In order to reach full consensus, data at all frequencies need to be calibrated simultaneously. In the SKA regime, this can become intractable if the available compute agents do not have the resources to process data from all frequency channels simultaneously. In this paper, we propose a multiplexing scheme that is based on the alternating direction method of multipliers (ADMM) with cyclic updates. With this scheme, it is possible to simultaneously calibrate the full dataset using far fewer compute agents than the number of frequencies at which data are available. We give simulation results to show the feasibility of the proposed multiplexing scheme in simultaneously calibrating a full dataset when a limited number of compute agents are available., Comment: MNRAS Accepted 2017 November 28. Received 2017 November 28; in original form 2017 July 06

Distributed radio interferometric calibration based on consensus optimization has been shown to improve the estimation of systematic errors in radio astronomical observations. The intrinsic continuity of systematic errors across frequency is used by a consensus polynomial to penalize traditional calibration. Consensus is achieved via the use of alternating direction method of multipliers (ADMM) algorithm. In this paper, we extend the existing distributed calibration algorithms to use ADMM with an adaptive penalty parameter update. Compared to a fixed penalty, its adaptive update has been shown to perform better in diverse applications of ADMM. In this paper, we compare two such popular penalty parameter update schemes: residual balance penalty update and spectral penalty update (Barzilai-Borwein). We apply both schemes to distributed radio interferometric calibration and compare their performance against ADMM with a fixed penalty parameter. Simulations show that both methods of adaptive penalty update improve the convergence of ADMM but the spectral penalty parameter update shows more stability., Comment: Draft, to be published in the Proceedings of the 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) (IEEE CAMSAP 2017), published by IEEE

We recently proposed the use of consensus optimization as a viable and effective way to improve the quality of calibration of radio interferometric data. We showed that it is possible to obtain far more accurate calibration solutions and also to distribute the compute load across a network of computers by using this technique. A crucial aspect in any consensus optimization problem is the selection of the penalty parameter used in the alternating direction method of multipliers (ADMM) iterations. This affects the convergence speed as well as the accuracy. In this paper, we use the Hessian of the cost function used in calibration to appropriately select this penalty. We extend our results to a multi-directional calibration setting, where we propose to use a penalty scaled by the squared intensity of each direction., Comment: Draft, to be published in the Proceedings of the 24th European Signal Processing Conference (EUSIPCO-2016) in 2016, published by EURASIP

The redshifted 21 cm line of neutral hydrogen is a promising probe of the Epoch of Reionization (EoR). However, its detection requires a thorough understanding and control of the systematic errors. We study two systematic biases observed in the LOFAR EoR residual data after calibration and subtraction of bright discrete foreground sources. The first effect is a suppression in the diffuse foregrounds, which could potentially mean a suppression of the 21 cm signal. The second effect is an excess of noise beyond the thermal noise. The excess noise shows fluctuations on small frequency scales, and hence it can not be easily removed by foreground removal or avoidance methods. Our analysis suggests that sidelobes of residual sources due to the chromatic point spread function and ionospheric scintillation can not be the dominant causes of the excess noise. Rather, both the suppression of diffuse foregrounds and the excess noise can occur due to calibration with an incomplete sky model containing predominantly bright discrete sources. We show that calibrating only on bright sources can cause suppression of other signals and introduce an excess noise in the data. The levels of the suppression and excess noise depend on the relative flux of sources which are not included in the model with respect to the flux of modeled sources. We discuss possible solutions such as using only long baselines to calibrate the interferometric gain solutions as well as simultaneous multi-frequency calibration along with their benefits and shortcomings., Comment: Submitted to MNRAS, 14 pages, 11 figures

Increasing data volumes delivered by a new generation of radio interferometers require computationally efficient and robust calibration algorithms. In this paper, we propose distributed calibration as a way of improving both computational cost as well as robustness in calibration. We exploit the data parallelism across frequency that is inherent in radio astronomical observations that are recorded as multiple channels at different frequencies. Moreover, we also exploit the smoothness of the variation of calibration parameters across frequency. Data parallelism enables us to distribute the computing load across a network of compute agents. Smoothness in frequency enables us reformulate calibration as a consensus optimization problem. With this formulation, we enable flow of information between compute agents calibrating data at different frequencies, without actually passing the data, and thereby improving robustness. We present simulation results to show the feasibility as well as the advantages of distributed calibration as opposed to conventional calibration., Comment: MNRAS Accepted 2015 March 13. Received 2015 January 28; in original form 2014 November 6, low resolution figures

Radio interferometers observe the Fourier space of the sky, at locations determined by the array geometry. Before a real space image is constructed by a Fourier transform, the data is weighted to improve the quality of reconstruction. Two criteria for calculation of weights are maximizing sensitivity and minimizing point spread function (PSF) sidelobe levels. In this paper, we propose a novel weighting scheme suitable for ultra deep imaging experiments. The proposed weighting scheme is used to maximize sensitivity while minimizing PSF sidelobe variation across frequency and multiple epochs. We give simulation results that show the superiority of the proposed scheme compared with commonly used weighting schemes in achieving these objectives., Comment: MNRAS Accepted 2014 July 22. Received 2014 July 15; in original form 2014 June 26

Several experiments are underway to detect the cosmic redshifted 21-cm signal from neutral hydrogen from the Epoch of Reionization (EoR). Due to their very low signal-to-noise ratio, these observations aim for a statistical detection of the signal by measuring its power spectrum. We investigate the extraction of the variance of the signal as a first step towards detecting and constraining the global history of the EoR. Signal variance is the integral of the signal's power spectrum, and it is expected to be measured with a high significance. We demonstrate this through results from a simulation and parameter estimation pipeline developed for the Low Frequency Array (LOFAR)-EoR experiment. We show that LOFAR should be able to detect the EoR in 600 hours of integration using the variance statistic. Additionally, the redshift ($z_r$) and duration ($\Delta z$) of reionization can be constrained assuming a parametrization. We use an EoR simulation of $z_r = 7.68$ and $\Delta z = 0.43$ to test the pipeline. We are able to detect the simulated signal with a significance of 4 standard deviations and extract the EoR parameters as $z_r = 7.72^{+0.37}_{-0.18}$ and $\Delta z = 0.53^{+0.12}_{-0.23}$ in 600 hours, assuming that systematic errors can be adequately controlled. We further show that the significance of detection and constraints on EoR parameters can be improved by measuring the cross-variance of the signal by cross-correlating consecutive redshift bins., Comment: 13 pages, 14 figures, Accepted for publication in MNRAS

In order to cope with the increased data volumes generated by modern radio interferometers such as LOFAR (Low Frequency Array) or SKA (Square Kilometre Array), fast and efficient calibration algorithms are essential. Traditional radio interferometric calibration is performed using nonlinear optimization techniques such as the Levenberg-Marquardt algorithm in Euclidean space. In this paper, we reformulate radio interferometric calibration as a nonlinear optimization problem on a Riemannian manifold. The reformulated calibration problem is solved using the Riemannian trust-region method. We show that calibration on a Riemannian manifold has faster convergence with reduced computational cost compared to conventional calibration in Euclidean space., Comment: Draft version. Final version will appear in IEEE ICASSP 2013, http://www.icassp2013.com/

The new generation of radio synthesis arrays, such as LOFAR and SKA, have been designed to surpass existing arrays in terms of sensitivity, angular resolution and frequency coverage. This evolution has led to the development of advanced calibration techniques that ensure the delivery of accurate results at the lowest possible computational cost. However, the performance of such calibration techniques is still limited by the compact, bright sources in the sky, used as calibrators. It is important to have a bright enough source that is well distinguished from the background noise level in order to achieve satisfactory results in calibration. We present "clustered calibration" as a modification to traditional radio interferometric calibration, in order to accommodate faint sources that are almost below the background noise level into the calibration process. The main idea is to employ the information of the bright sources' measured signals as an aid to calibrate fainter sources that are nearby the bright sources. In the case where we do not have bright enough sources, a source cluster could act as a bright source that can be distinguished from background noise. We construct a number of source clusters assuming that the signals of the sources belonging to a single cluster are corrupted by almost the same errors, and each cluster is calibrated as a single source, using the combined coherencies of its sources simultaneously. This upgrades the power of an individual faint source by the effective power of its cluster. We give performance analysis of clustered calibration to show the superiority of this approach compared to the traditional unclustered calibration. We also provide analytical criteria to choose the optimum number of clusters for a given observation in an efficient manner., Comment: 18 pages, 21 figures, Accepted 2013 January 2. Abstract abridged

The Square Kilometre Array (SKA) will have a low frequency component (SKA-low) which has as one of its main science goals the study of the redshifted 21cm line from the earliest phases of star and galaxy formation in the Universe. This 21cm signal provides a new and unique window on both the formation of the first stars and accreting black holes and the later period of substantial ionization of the intergalactic medium. The signal will teach us fundamental new things about the earliest phases of structure formation, cosmology and even has the potential to lead to the discovery of new physical phenomena. Here we present a white paper with an overview of the science questions that SKA-low can address, how we plan to tackle these questions and what this implies for the basic design of the telescope., Comment: Accepted for publication in Experimental Astronomy, reformatted to 57 pages, some updated and improved figures and minor changes and updates to the text

Full polarimetric radio interferometric calibration is performed by estimating 2 by 2 Jones matrices representing instrumental and propagation effects. The solutions obtained in this way differ from the true solutions by a 2 by 2 unitary matrix ambiguity. This ambiguity is common to all stations for which a solution is obtained but it is different for solutions obtained at different time and frequency intervals. Therefore, straightforward interpolation of solutions obtained at different time and frequency intervals is not possible. In this paper, we propose to use the theory of quotient manifolds for obtaining correct interpolants that are immune to unitary matrix ambiguities., Comment: Accepted 2012 September 24. Received 2012 September 21; in original form 2012 June 12

The knowledge of receiver beam shapes is essential for accurate radio interferometric imaging. Traditionally, this information is obtained by holographic techniques or by numerical simulation. However, such methods are not feasible for an observation with time varying beams, such as the beams produced by a phased array radio interferometer. We propose the use of the observed data itself for the estimation of the beam shapes. We use the directional gains obtained along multiple sources across the sky for the construction of a time varying beam model. The construction of this model is an ill posed non linear optimization problem. Therefore, we propose to use Riemannian optimization, where we consider the constraints imposed as a manifold. We compare the performance of the proposed approach with traditional unconstrained optimization and give results to show the superiority of the proposed approach., Comment: Accepted on 18-09-2012. Draft version. The final publication is available at springerlink.com ; Experimental Astronomy, 2012

Interferometric calibration always yields non unique solutions. It is therefore essential to remove these ambiguities before the solutions could be used in any further modeling of the sky, the instrument or propagation effects such as the ionosphere. We present a method for LOFAR calibration which does not yield a unitary ambiguity, especially under ionospheric distortions. We also present exact ambiguities we get in our solutions, in closed form. Casting this as an optimization problem, we also present conditions for this approach to work. The proposed method enables us to use the solutions obtained via calibration for further modeling of instrumental and propagation effects. We provide extensive simulation results on the performance of our method. Moreover, we also give cases where due to degeneracy, this method fails to perform as expected and in such cases, we suggest exploiting diversity in time, space and frequency., Comment: Draft version. Final version published on 10 April 2012

The application of orthonormal basis functions such as Prolate Spheroidal Wave Functions (PSWF) for accurate source modeling in radio astronomy has been comprehensively studied. They are of great importance for high fidelity, high dynamic range imaging with new radio telescopes as well as conventional ones. But the construction of PSWF is computationally expensive compared to other closed form basis functions. In this paper, we suggest a solution to reduce its computational cost by more efficient construction of the matrix kernel which relates the image domain to visibility (or Fourier) domain. Radio astronomical images are mostly represented using a regular grid of rectangular pixels. This is required for efficient storage and display purposes and moreover, comes naturally as a by product of the Fast Fourier Transform (FFT) in imaging. We propose the use of Delaunay triangulation as opposed to regular gridding of an image for a finer selection of the region of interest (signal support) during the PSWF kernel construction. We show that the computational efficiency improves without loss of information. Once the PSWF basis is constructed using the irregular grid, we revert back to the regular grid by interpolation and thereafter, conventional imaging techniques can be applied., Comment: 5 pages, 6 figures, accepted for publication in ISSPIT (IEEE International Symposium on Signal Processing and Information Technology), Bilbao- Spain, 14th-17th Dec.2011

This paper introduces an amendment to radio interferometric calibration of sources below the noise level. The main idea is to employ the information of the stronger sources' measured signals as a plug-in criterion to solve for the weaker ones. For this purpose, we construct a number of source clusters, with centroids mainly near the strongest sources, assuming that the signals of the sources belonging to a single cluster are corrupted by almost the same errors. Due to this characteristic of clusters, each cluster is calibrated as a single source, using all the coherencies of its sources simultaneously. The obtained solutions for every cluster are assigned to all the cluster's sources. An illustrative example reveals the superiority of this calibration compared to the un-clustered calibration., Comment: 4 pages, 2 figures, 1 table, accepted in IEEE Workshop on Statistical Signal Processing (SSP2011)

In order to produce high dynamic range images in radio interferometry, bright extended sources need to be removed with minimal error. However, this is not a trivial task because the Fourier plane is sampled only at a finite number of points. The ensuing deconvolution problem has been solved in many ways, mainly by algorithms based on CLEAN. However, such algorithms that use image pixels as basis functions have inherent limitations and by using an orthonormal basis that span the whole image, we can overcome them. The construction of such an orthonormal basis involves fine tuning of many free parameters that define the basis functions. The optimal basis for a given problem (or a given extended source) is not guaranteed. In this paper, we discuss the use of generalized prolate spheroidal wave functions as a basis. Given the geometry (or the region of interest) of an extended source and the sampling points on the visibility plane, we can construct the optimal basis to model the source. Not only does this gives us the minimum number of basis functions required but also the artifacts outside the region of interest are minimized., Comment: 4 pages, 6 figures

Deconvolution is essential for radio interferometric imaging to produce scientific quality data because of finite sampling in the Fourier plane. Most deconvolution algorithms are based on CLEAN which uses a grid of image pixels, or clean components. A critical matter in this process is the selection of pixel size for optimal results in deconvolution. As a rule of thumb, the pixel size is chosen smaller than the resolution dictated by the interferometer. For images consisting of unresolved (or point like) sources, this approach yields optimal results. However, for sources that are not point like, in particular for partially resolved sources, the selection of right pixel size is still an open issue. In this paper, we investigate the limitations of pixelization in deconvolving extended sources. In particular, we pursue the usage of orthonormal basis functions to model extended sources yielding better results than by using clean components., Comment: 4 pages, 5 figures, the 6th IEEE Sensor Array and Multichannel Signal Processing workshop

One of the aims of the Low Frequency Array (LOFAR) Epoch of Reionization (EoR) project is to measure the power spectrum of variations in the intensity of redshifted 21-cm radiation from the EoR. The sensitivity with which this power spectrum can be estimated depends on the level of thermal noise and sample variance, and also on the systematic errors arising from the extraction process, in particular from the subtraction of foreground contamination. We model the extraction process using realistic simulations of the cosmological signal, the foregrounds and noise, and so estimate the sensitivity of the LOFAR EoR experiment to the redshifted 21-cm power spectrum. Detection of emission from the EoR should be possible within 360 hours of observation with a single station beam. Integrating for longer, and synthesizing multiple station beams within the primary (tile) beam, then enables us to extract progressively more accurate estimates of the power at a greater range of scales and redshifts. We discuss different observational strategies which compromise between depth of observation, sky coverage and frequency coverage. A plan in which lower frequencies receive a larger fraction of the time appears to be promising. We also study the nature of the bias which foreground fitting errors induce on the inferred power spectrum, and discuss how to reduce and correct for this bias. The angular and line-of-sight power spectra have different merits in this respect, and we suggest considering them separately in the analysis of LOFAR data., Comment: 13 pages, 11 figures, 1 table; accepted for publication in MNRAS

An obstacle to the detection of redshifted 21cm emission from the epoch of reionization (EoR) is the presence of foregrounds which exceed the cosmological signal in intensity by orders of magnitude. We argue that in principle it would be better to fit the foregrounds non-parametrically - allowing the data to determine their shape - rather than selecting some functional form in advance and then fitting its parameters. Non-parametric fits often suffer from other problems, however. We discuss these before suggesting a non-parametric method, Wp smoothing, which seems to avoid some of them. After outlining the principles of Wp smoothing we describe an algorithm used to implement it. We then apply Wp smoothing to a synthetic data cube for the LOFAR EoR experiment. The performance of Wp smoothing, measured by the extent to which it is able to recover the variance of the cosmological signal and to which it avoids leakage of power from the foregrounds, is compared to that of a parametric fit, and to another non-parametric method (smoothing splines). We find that Wp smoothing is superior to smoothing splines for our application, and is competitive with parametric methods even though in the latter case we may choose the functional form of the fit with advance knowledge of the simulated foregrounds. Finally, we discuss how the quality of the fit is affected by the frequency resolution and range, by the characteristics of the cosmological signal and by edge effects., Comment: 15 pages, 12 figures; lengthened and two figures added, to match version accepted by MNRAS

A number of experiments are set to measure the 21-cm signal of neutral hydrogen from the Epoch of Reionization (EoR). The common denominator of these experiments are the large data sets produced, contaminated by various instrumental effects, ionospheric distortions, RFI and strong Galactic and extragalactic foregrounds. In this paper, the first in a series, we present the Data Model that will be the basis of the signal analysis for the LOFAR (Low Frequency Array) EoR Key Science Project (LOFAR EoR KSP). Using this data model we simulate realistic visibility data sets over a wide frequency band, taking properly into account all currently known instrumental corruptions (e.g. direction-dependent gains, complex gains, polarization effects, noise, etc). We then apply primary calibration errors to the data in a statistical sense, assuming that the calibration errors are random Gaussian variates at a level consistent with our current knowledge based on observations with the LOFAR Core Station 1. Our aim is to demonstrate how the systematics of an interferometric measurement affect the quality of the calibrated data, how errors correlate and propagate, and in the long run how this can lead to new calibration strategies. We present results of these simulations and the inversion process and extraction procedure. We also discuss some general properties of the coherency matrix and Jones formalism that might prove useful in solving the calibration problem of aperture synthesis arrays. We conclude that even in the presence of realistic noise and instrumental errors, the statistical signature of the EoR signal can be detected by LOFAR with relatively small errors. A detailed study of the statistical properties of our data model and more complex instrumental models will be considered in the future., Comment: 20 pages, 9 figures

Radio Interferometry is an essential method for astronomical observations. Self-calibration techniques have increased the quality of the radio astronomical observations (and hence the science) by orders of magnitude. Recently, there is a drive towards sensor arrays built using inexpensive hardware and distributed over a wide area acting as radio interferometers. Calibration of such arrays poses new problems in terms of computational cost as well as in performance of existing calibration algorithms. We consider the application of the Space Alternating Generalized Expectation Maximization (SAGE) \cite{Fess94} algorithm for calibration of radio interferometric arrays. Application to real data shows that this is an improvement over existing calibration algorithms that are based on direct, deterministic non linear optimization. As presented in this paper, we can improve the computational cost as well as the quality of the calibration using this algorithm., Comment: 6 pages, 3 figures, Appearing in 13th IEEE DSP workshop (IEEE Signal Processing Society)

Detecting redshifted 21cm emission from neutral hydrogen in the early Universe promises to give direct constraints on the epoch of reionization (EoR). It will, though, be very challenging to extract the cosmological signal (CS) from foregrounds and noise which are orders of magnitude larger. Fortunately, the signal has some characteristics which differentiate it from the foregrounds and noise, and we suggest that using the correct statistics may tease out signatures of reionization. We generate mock datacubes simulating the output of the Low Frequency Array (LOFAR) EoR experiment. These cubes combine realistic models for Galactic and extragalactic foregrounds and the noise with three different simulations of the CS. We fit out the foregrounds, which are smooth in the frequency direction, to produce residual images in each frequency band. We denoise these images and study the skewness of the one-point distribution in the images as a function of frequency. We find that, under sufficiently optimistic assumptions, we can recover the main features of the redshift evolution of the skewness in the 21cm signal. We argue that some of these features - such as a dip at the onset of reionization, followed by a rise towards its later stages - may be generic, and give us a promising route to a statistical detection of reionization., Comment: 11 pages, 10 figures. One figure added. Other small changes to match accepted version

We present an efficient method to generate large simulations of the Epoch of Reionization (EoR) without the need for a full 3-dimensional radiative transfer code. Large dark-matter-only simulations are post-processed to produce maps of the redshifted 21cm emission from neutral hydrogen. Dark matter haloes are embedded with sources of radiation whose properties are either based on semi-analytical prescriptions or derived from hydrodynamical simulations. These sources could either be stars or power-law sources with varying spectral indices. Assuming spherical symmetry, ionized bubbles are created around these sources, whose radial ionized fraction and temperature profiles are derived from a catalogue of 1-D radiative transfer experiments. In case of overlap of these spheres, photons are conserved by redistributing them around the connected ionized regions corresponding to the spheres. The efficiency with which these maps are created allows us to span the large parameter space typically encountered in reionization simulations. We compare our results with other, more accurate, 3-D radiative transfer simulations and find excellent agreement for the redshifts and the spatial scales of interest to upcoming 21cm experiments. We generate a contiguous observational cube spanning redshift 6 to 12 and use these simulations to study the differences in the reionization histories between stars and quasars. Finally, the signal is convolved with the LOFAR beam response and its effects are analyzed and quantified. Statistics performed on this mock data set shed light on possible observational strategies for LOFAR., Comment: 18 pages, 21 figures, submitted to MNRAS For high-resolution images follow "http://www.astro.rug.nl/~thomas/eormap.pdf"

Radio astronomical observations have very poor signal to noise ratios, unlike in other disciplines. On the other hand, it is possible to observe the object of interest for long time intervals as well as using a wider bandwidth. Traditionally, by averaging in time and in frequency, it has been possible to improve the signal to noise ratio of astronomical observations to improve the dynamic range. This is possible due to the inherent assumption that the object of interest in the sky is invariant over time and the frequency range of observation. However, in reality this assumption does not hold, due to intrinsic variation of the sky as well as due to errors generated by the instrument. In this paper, we shall discuss an alternative to averaging of images, without ignoring subtle changes in the observed data over time and frequency, using subspace decomposition. By separation of data to signal and noise subspaces, not only would this improve the quality of the data, but also enable us to detect faint artifacts due to calibration errors, interference etc., Comment: 4 pages, 4 figures

Future high redshift 21-cm experiments will suffer from a high degree of contamination, due both to astrophysical foregrounds and to non-astrophysical and instrumental effects. In order to reliably extract the cosmological signal from the observed data, it is essential to understand very well all data components and their influence on the extracted signal. Here we present simulated astrophysical foregrounds datacubes and discuss their possible statistical effects on the data. The foreground maps are produced assuming 5 deg x 5 deg windows that match those expected to be observed by the LOFAR Epoch-of-Reionization (EoR) key science project. We show that with the expected LOFAR-EoR sky and receiver noise levels, which amount to ~52 mK at 150 MHz after 300 hours of total observing time, a simple polynomial fit allows a statistical reconstruction of the signal. We also show that the polynomial fitting will work for maps with realistic yet idealised instrument response, i.e., a response that includes only a uniform uv coverage as a function of frequency and ignores many other uncertainties. Polarized galactic synchrotron maps that include internal polarization and a number of Faraday screens along the line of sight are also simulated. The importance of these stems from the fact that the LOFAR instrument, in common with all current interferometric EoR experiments has an instrumentally polarized response., Comment: 18 figures, 3 tables, accepted to be published in MNRAS