Search

The unparalleled success of artificial intelligence (AI) in the technology sector has catalyzed an enormous amount of research in the scientific community. It has proven to be a powerful tool, but as with any rapidly developing field, the deluge of information can be overwhelming, confusing and sometimes misleading. This can make it easy to become lost in the same hype cycles that have historically ended in the periods of scarce funding and depleted expectations known as AI Winters. Furthermore, while the importance of innovative, high-risk research cannot be overstated, it is also imperative to understand the fundamental limits of available techniques, especially in young fields where the rules appear to be constantly rewritten and as the likelihood of application to high-stakes scenarios increases. In this perspective, we highlight the guiding principles of data-driven modeling, how these principles imbue models with almost magical predictive power, and how they also impose limitations on the scope of problems they can address. Particularly, understanding when not to use data-driven techniques, such as machine learning, is not something commonly explored, but is just as important as knowing how to apply the techniques properly. We hope that the discussion to follow provides researchers throughout the sciences with a better understanding of when said techniques are appropriate, the pitfalls to watch for, and most importantly, the confidence to leverage the power they can provide., Comment: 7 pages, 3 figures

We present Planck LFI frequency sky maps derived within the BeyondPlanck framework. This framework draws samples from a global posterior distribution that includes instrumental, astrophysical and cosmological parameters, and the main product is an entire ensemble of frequency sky map samples. This ensemble allows for computationally convenient end-to-end propagation of low-level instrumental uncertainties into higher-level science products. We show that the two dominant sources of LFI instrumental systematic uncertainties are correlated noise and gain fluctuations, and the products presented here support - for the first time - full Bayesian error propagation for these effects at full angular resolution. We compare our posterior mean maps with traditional frequency maps delivered by the Planck collaboration, and find generally good agreement. The most important quality improvement is due to significantly lower calibration uncertainties in the new processing, as we find a fractional absolute calibration uncertainty at 70 GHz of $\delta g_{0}/g_{0} =5 \cdot 10^{-5}$, which is nominally 40 times smaller than that reported by Planck 2018. However, the original Planck 2018 estimate has a non-trivial statistical interpretation, and this further illustrates the advantage of the new framework in terms of producing self-consistent and well-defined error estimates of all involved quantities without the need of ad hoc uncertainty contributions. We describe how low-resolution data products, including dense pixel-pixel covariance matrices, may be produced directly from the posterior samples without the need for computationally expensive analytic calculations or simulations. We conclude that posterior-based frequency map sampling provides unique capabilities in terms of low-level systematics modelling and error propagation, and may play an important role for future CMB B-mode experiments. (Abridged.), Comment: 32 pages, 23 figures, data available from https://www.cosmoglobe.uio.no/

We present posterior sample-based cosmic microwave background (CMB) constraints from Planck LFI and WMAP observations derived through global end-to-end Bayesian processing. We use these samples to study correlations between CMB, foreground, and instrumental parameters, and we identify a particularly strong degeneracy between CMB temperature fluctuations and free-free emission on intermediate angular scales, which is mitigated through model reduction, masking, and resampling. We compare our posterior-based CMB results with previous Planck products, and find generally good agreement, but with higher noise due to exclusion of HFI data. We find a best-fit CMB dipole amplitude of $3362.7\pm1.4{\mu}K$, in excellent agreement with previous Planck results. The quoted uncertainty is derived directly from the sampled posterior distribution, and does not involve any ad hoc contribution for systematic effects. Similarly, we find a temperature quadrupole amplitude of $\sigma^{TT}_2=229\pm97{\mu}K^2$, in good agreement with previous results in terms of the amplitude, but the uncertainty is an order of magnitude larger than the diagonal Fisher uncertainty. Relatedly, we find lower evidence for a possible alignment between $\ell = 2$ and $\ell = 3$ than previously reported due to a much larger scatter in the individual quadrupole coefficients, caused both by marginalizing over a more complete set of systematic effects, and by our more conservative analysis mask. For higher multipoles, we find that the angular temperature power spectrum is generally in good agreement with both Planck and WMAP. This is the first time the sample-based asymptotically exact Blackwell-Rao estimator has been successfully established for multipoles up to $\ell\le600$, and it now accounts for the majority of the cosmologically important information. Cosmological parameter constraints are presented in a companion paper. (Abriged), Comment: 26 pages, 24 figures. Submitted to A&A. Part of the BeyondPlanck paper suite

The BeyondPlanck and Cosmoglobe collaborations have implemented the first integrated Bayesian end-to-end analysis pipeline for CMB experiments. The primary long-term motivation for this work is to develop a common analysis platform that supports efficient global joint analysis of complementary radio, microwave, and sub-millimeter experiments. A strict prerequisite for this to succeed is broad participation from the CMB community, and two foundational aspects of the program are therefore reproducibility and Open Science. In this paper, we discuss our efforts toward this aim. We also discuss measures toward facilitating easy code and data distribution, community-based code documentation, user-friendly compilation procedures, etc. This work represents the first publicly released end-to-end CMB analysis pipeline that includes raw data, source code, parameter files, and documentation. We argue that such a complete pipeline release should be a requirement for all major future and publicly-funded CMB experiments, noting that a full public release significantly increases data longevity by ensuring that the data quality can be improved whenever better processing techniques, complementary datasets, or more computing power become available, and thereby also taxpayers' value for money; providing only raw data and final products is not sufficient to guarantee full reproducibility in the future.

We present cosmological parameter constraints as estimated using the Bayesian BeyondPlanck (BP) analysis framework. This method supports seamless end-to-end error propagation from raw time-ordered data to final cosmological parameters. As a first demonstration of the method, we analyze time-ordered Planck LFI observations, combined with selected external data (WMAP 33-61GHz, Planck HFI DR4 353 and 857GHz, and Haslam 408MHz) in the form of pixelized maps which are used to break critical astrophysical degeneracies. Overall, all results are generally in good agreement with previously reported values from Planck 2018 and WMAP, with the largest relative difference for any parameter of about 1 sigma when considering only temperature multipoles between 29

We describe the correction procedure for Analog-to-Digital Converter (ADC) differential non-linearities (DNL) adopted in the Bayesian end-to-end BeyondPlanck analysis framework. This method is nearly identical to that developed for the official LFI Data Processing Center (DPC) analysis, and relies on the binned rms noise profile of each detector data stream. However, rather than building the correction profile directly from the raw rms profile, we first fit a Gaussian to each significant ADC-induced rms decrement, and then derive the corresponding correction model from this smooth model. The main advange of this approach is that only samples which are significantly affected by ADC DNLs are corrected. The new corrections are only applied to data for which there is a clear detection of the non-linearities, and for which they perform at least comparably with the DPC corrections. Out of a total of 88 LFI data streams (sky and reference load for each of the 44 detectors) we apply the new minimal ADC corrections in 25 cases, and maintain the DPC corrections in 8 cases. All these correctsion are applited to 44 or 70 GHz channels, while, as in previous analyses, none of the 30 GHz ADCs show significant evidence of non-linearity. By comparing the BeyondPlanck and DPC ADC correction methods, we estimate that the residual ADC uncertainty is about two orders of magnitude below the total noise of both the 44 and 70 GHz channels, and their impact on current cosmological parameter estimation is small. However, we also show that non-idealities in the ADC corrections can generate sharp stripes in the final frequency maps, and these could be important for future joint analyses with HFI, WMAP, or other datasets. We therefore conclude that, although the existing corrections are adequate for LFI-based cosmological parameter analysis, further work on LFI ADC corrections is still warranted., Comment: 9 pages, 8 figures, mild language edits

We present the first application of the Cosmoglobe analysis framework by analyzing 9-year $\mathit{WMAP}$ time-ordered observations using similar machinery as BeyondPlanck utilizes for $\mathit{Planck}$ LFI. We analyze only the $\mathit Q$-band (41 GHz) data and report on the low-level analysis process from uncalibrated time-ordered data to calibrated maps. Most of the existing BeyondPlanck pipeline may be reused for $\mathit{WMAP}$ analysis with minimal changes to the existing codebase. The main modification is the implementation of the same preconditioned biconjugate gradient mapmaker used by the $\mathit{WMAP}$ team. Producing a single $\mathit{WMAP}$ $\mathit Q$1-band sample requires 22 CPU-hrs, which is slightly more than the cost of a $\mathit{Planck}$ 44 GHz sample of 17 CPU-hrs; this demonstrates that full end-to-end Bayesian processing of the $\mathit{WMAP}$ data is computationally feasible. In general, our recovered maps are very similar to the maps released by the $\mathit{WMAP}$ team, although with two notable differences. In temperature we find a $\sim2\,\mathrm{\mu K}$ quadrupole difference that most likely is caused by different gain modeling, while in polarization we find a distinct $2.5\,\mathrm{\mu K}$ signal that has been previously called poorly-measured modes by the $\mathit{WMAP}$ team. In the Cosmoglobe processing, this pattern arises from temperature-to-polarization leakage from the coupling between the CMB Solar dipole, transmission imbalance, and sidelobes. No traces of this pattern are found in either the frequency map or TOD residual map, suggesting that the current processing has succeeded in modelling these poorly measured modes within the assumed parametric model by using $\mathit{Planck}$ information to break the sky-synchronous degeneracies inherent in the $\mathit{WMAP}$ scanning strategy., Comment: 11 figures, submitted to A&A. Includes updated instrument model and changes addressing referee comments

We present the intensity foreground algorithms and model employed within the BeyondPlanck analysis framework. The BeyondPlanck analysis is aimed at integrating component separation and instrumental parameter sampling within a global framework, leading to complete end-to-end error propagation in the $Planck$ Low Frequency Instrument (LFI) data analysis. Given the scope of the BeyondPlanck analysis, a limited set of data is included in the component separation process, leading to foreground parameter degeneracies. In order to properly constrain the Galactic foreground parameters, we improve upon the previous $\texttt{Commander}$ component separation implementation by adding a suite of algorithmic techniques. These algorithms are designed to improve the stability and computational efficiency for weakly constrained posterior distributions. These are: 1) joint foreground spectral parameter and amplitude sampling, building on ideas from Miramare; 2) component-based monopole determination; 3) joint spectral parameter and monopole sampling; and 4) application of informative spatial priors for component amplitude maps. We find that the only spectral parameter with a significant signal-to-noise ratio using the current BeyondPlanck data set is the peak frequency of the anomalous microwave emission component, for which we find $\nu_{\mathrm{p}}=25.3\pm0.5$ GHz; all others must be constrained through external priors. Future works will be aimed at integrating many more data sets into this analysis, both map and time-ordered based, thereby gradually eliminating the currently observed degeneracies in a controlled manner with respect to both instrumental systematic effects and astrophysical degeneracies. When this happens, the simple LFI-oriented data model employed in the current work will need to be generalized to account for both a richer astrophysical model and additional instrumental effects., Comment: 27 pages, 23 figures, 3 tables, part 13 of the BeyondPlanck release. All BeyondPlanck products and software will be released publicly at http://beyondplanck.science. Paper version as accepted by A&A

We constrain the level of polarized anomalous microwave emission (AME) on large angular scales using $\textit{Planck}$ LFI and $\textit{WMAP}$ polarization data within a Bayesian CMB analysis framework. We model synchrotron emission with a power-law spectral energy distribution, and the sum of AME and thermal dust emission through linear regression with the $\textit{Planck}$ HFI 353 GHz data. This template-based dust emission model allows us to constrain the level of polarized AME while making minimal assumptions on its frequency dependence. We neglect cosmic microwave background fluctuations, but show through simulations that these have a minor impact on the results. We find that the resulting AME polarization fraction confidence limit is sensitive to the polarized synchrotron spectral index prior, and for priors steeper than $\beta_{\mathrm{s}} = -3.1\pm0.1$ we find an upper limit of $p_{\mathrm{AME}}^{\rm max}\lesssim 0.6\,\%$ ($95\,\%$ confidence). In contrast, for $\beta_{\mathrm{s}}=-3.0\pm0.1$, we find a nominal detection of $p_{\mathrm{AME}}=2.5\pm1.0\,\%$ ($95\,\%$ confidence). These data are thus not strong enough to simultaneously and robustly constrain both polarized synchrotron emission and AME, and our main result is therefore a constraint on the AME polarization fraction explicitly as a function of $\beta_\mathrm{s}$. Combining the current $\textit{Planck}$ and $\textit{WMAP}$ observations with measurements from high-sensitivity low-frequency experiments such as C-BASS and QUIJOTE will be critical to improve these limits further., Comment: 10 pages, 9 figures, part 15 of the BeyondPlanck release. All BeyondPlanck products and software will be released publicly at http://beyondplanck.science. Refereed, edited, and accepted by A&A. Added references, added arxiv IDs to BeyondPlanck papers. Figure update. Expanded discussion on AME models. Moved title from BeyondPlanck XV to BeyondPlanck XVI due to internal reordering

We describe the computational infrastructure for end-to-end Bayesian CMB analysis implemented by the BeyondPlanck collaboration. This code is called commander3, and provides a statistically consistent framework for global analysis of CMB and microwave observations, and may be useful for a wide range of legacy, current, and future experiments. The paper has three main goals. Firstly, we provide a high-level overview of the existing code base, aiming to guide readers who wish to extend and adapt the code according to their own needs, or to reimplement it from scratch in a different programming language. Secondly, we discuss some critical computational challenges that arise within any global CMB analysis framework, for instance in-memory compression of time-ordered data, FFT optimization, and parallelization and load-balancing. Thirdly, we quantify the CPU and RAM requirements for the current BeyondPlanck analysis, and find that a total of 1.5 TB of RAM is required for efficient analysis, and the total cost of a full Gibbs sample is 170 CPU-hrs, including both low-level processing and high-level component separation, which is well within the capabilities of current low-cost computing facilities. The existing code base is made publicly available under a GNU General Public Library (GPL) license., Comment: 16 Pages, 7 Figures. Part of the BeyondPlanck paper suite

We introduce a new formulation of the Conviqt convolution algorithm in terms of spin harmonics, and apply this to the problem of sidelobe correction for BeyondPlanck, the first end-to-end Bayesian Gibbs sampling framework for CMB analysis. We compare our implementation to the previous Planck LevelS implementation, and find good agreement between the two codes in terms of accuracy, but with a speed-up reaching a factor of 3--10, depending on the frequency bandlimits, $l_{\textrm{max}}$ and $m_{\textrm{max}}$. The new algorithm is significantly simpler to implement and maintain, since all low-level calculations are handled through an external spherical harmonic transform library. We find that our mean sidelobe estimates for Planck LFI agree well with previous efforts. Additionally, we present novel sidelobe rms maps that quantify the uncertainty in the sidelobe corrections due to variations in the sky model., Comment: 9 pages, 8 figures. Part of the BeyondPlanck paper suite

We discuss the treatment of bandpass and beam leakage corrections in the Bayesian BeyondPlanck CMB analysis pipeline as applied to the Planck LFI measurements. As a preparatory step, we first apply three corrections to the nominal LFI bandpass profiles including removal of a known systematic effect in the ground measuring equipment at 61 GHz; smoothing of standing wave ripples; and edge regularization. The main net impact of these modifications is an overall shift in the 70 GHz bandpass of +0.6 GHz; we argue that any analysis of LFI data products, either from Planck or BeyondPlanck, should use these new bandpasses. In addition, we fit a single free bandpass parameter for each radiometer of the form $\Delta_i = \Delta_0 + \delta_i$, where $\Delta_0$ represents an absolute frequency shift per frequency band and $\delta_i$ is a relative shift per detector. The absolute correction is only fitted at 30 GHz with a full $\chi^2$-based likelihood, resulting in a correction of $\Delta_{30}=0.24\pm0.03\,$GHz. The relative corrections are fitted using a spurious map approach, fundamentally similar to the method pioneered by the WMAP team, but without introducing many additional degrees of freedom. All bandpass parameters are sampled using a standard Metropolis sampler within the main BeyondPlanck Gibbs chain, and bandpass uncertainties are thus propagated to all other data products in the analysis. In total, we find that our bandpass model significantly reduces leakage effects. For beam leakage corrections, we adopt the official Planck LFI beam estimates without additional degrees of freedom, and only marginalize over the underlying sky model. We note that this is the first time leakage from beam mismatch has been included for Planck LFI maps., Comment: 14 pages, 11 figures, this is part 10 in the BeyondPlanck release. All BeyondPlanck products and software will be released publicly at http://beyondplanck.science. Submitted to A&A

We constrain polarized foreground emission between 30 and 70 GHz with the Planck Low Frequency Instrument (LFI) and WMAP data within the global Bayesian BeyondPlanck framework. We combine for the first time full-resolution Planck LFI time-ordered data with low-resolution WMAP sky maps at 33, 40 and 61 GHz. Spectral parameters are fit with a likelihood defined at the native resolution of each frequency channel. This analysis represents the first implementation of true multi-resolution component separation applied to CMB observations for both amplitude and spectral energy distribution (SED) parameters. For synchrotron emission, we approximate the SED as a power-law in frequency and find that the low signal-to-noise ratio of the current data strongly limits the number of free parameters that may be robustly constrained. We partition the sky into four large disjoint regions (High Latitude; Galactic Spur; Galactic Plane; and Galactic Center), each associated with its own power-law index. We find that the High Latitude region is prior-dominated, while the Galactic Center region is contaminated by residual instrumental systematics. The two remaining regions appear to be signal-dominated, and for these we derive spectral indices of $\beta_{\mathrm s}^{\mathrm{Spur}}=-3.17\pm0.06$ and $\beta_{\mathrm s}^{\mathrm{Plane}}=-3.03\pm0.07$, in good agreement with previous results. For thermal dust emission we assume a modified blackbody model and we fit a single power-law index across the full sky. We find $\beta_{\mathrm{d}}=1.64\pm0.03$, which is slightly steeper than reported from Planck HFI data, but still statistically consistent at the 2$\sigma$ confidence level., Comment: 18 pages, 15 figures. All BeyondPlanck products and software will be released publicly at http://beyondplanck.science. This paper has been submitted to A&A. Major revision, matches submitted version

We present a Bayesian calibration algorithm for CMB observations as implemented within the global end-to-end BeyondPlanck (BP) framework, and apply this to the Planck Low Frequency Instrument (LFI) data. Following the most recent Planck analysis, we decompose the full time-dependent gain into a sum of three orthogonal components: One absolute calibration term, common to all detectors; one time-independent term that can vary between detectors; and one time-dependent component that is allowed to vary between one-hour pointing periods. Each term is then sampled conditionally on all other parameters in the global signal model through Gibbs sampling. The absolute calibration is sampled using only the orbital dipole as a reference source, while the two relative gain components are sampled using the full sky signal, including the orbital and Solar CMB dipoles, CMB fluctuations, and foreground contributions. We discuss various aspects of the data that influence gain estimation, including the dipole/polarization quadrupole degeneracy and anomalous jumps in the instrumental gain. Comparing our solution to previous pipelines, we find good agreement in general, with relative deviations of -0.67% (-0.84%) for 30 GHz, 0.12% (-0.04%) for 44 GHz and -0.03% (-0.64%) for 70 GHz, compared to Planck DR4 (Planck 2018). The deviations we find are within expected error bounds, and we attribute them to differences in data usage and general approach between the pipelines. In particular, the BP calibration is performed globally, resulting in better inter-frequency consistency. Additionally, WMAP observations are used actively in the BP analysis, which breaks degeneracies in the Planck data set and results in better agreement with WMAP. Although our presentation and algorithm are currently oriented toward LFI processing, the procedure is fully generalizable to other experiments., Comment: 19 pages, 18 figures. All BeyondPlanck products and software will be released publicly at http://beyondplanck.science

We present a Bayesian method for estimating instrumental noise parameters and propagating noise uncertainties within the global BeyondPlanck Gibbs sampling framework, and apply this to Planck Low Frequency Instrument (LFI) time-ordered data. Following previous literature, we initially adopt a $1/f$ model for the noise power spectral density (PSD), but find the need for an additional lognormal component in the noise model for the 30 and 44\,GHz bands. We implement an optimal Wiener-filter (or constrained realization) gap-filling procedure to account for masked data. We then use this procedure to both estimate the gapless correlated noise in the time-domain, $n_\mathrm{corr}$, and to sample the noise PSD parameters, $\xi^n = \{\sigma_0, f_\mathrm{knee}, \alpha, A_\mathrm{p}\}$. In contrast to previous \textit{Planck} analyses, we assume piecewise stationary noise only within each pointing period (PID), not throughout the full mission, but we adopt the LFI Data Processing Center (DPC) results as priors on $\alpha$ and $f_\mathrm{knee}$. On average, we find best-fit correlated noise parameters that are mostly consistent with previous results, with a few notable exceptions. However, a detailed inspection of the time-dependent results reveals many important findings. First and foremost, we find strong evidence for statistically significant temporal variations in all noise PSD parameters, many of which are directly correlated with satellite housekeeping data. Second, while the simple $1/f$ model appears to be an excellent fit for the LFI 70 GHz channel, there is evidence for additional correlated noise not described by a $1/f$ model in the 30 and 44 GHz channels, including within the primary science frequency range of 0.1--1 Hz. (Abridged), Comment: 26 pages, 29 figures, as accepted in A&A

We present a Gibbs sampling solution to the map-making problem for CMB measurements, building on existing destriping methodology. Gibbs sampling breaks the computationally heavy destriping problem into two separate steps; noise filtering and map binning. Considered as two separate steps, both are computationally much cheaper than solving the combined problem. This provides a huge performance benefit as compared to traditional methods, and allows us for the first time to bring the destriping baseline length to a single sample. We apply the Gibbs procedure to simulated Planck 30 GHz data. We find that gaps in the time-ordered data are handled efficiently by filling them with simulated noise as part of the Gibbs process. The Gibbs procedure yields a chain of map samples, from which we may compute the posterior mean as a best-estimate map. The variation in the chain provides information on the correlated residual noise, without need to construct a full noise covariance matrix. However, if only a single maximum-likelihood frequency map estimate is required, we find that traditional conjugate gradient solvers converge much faster than a Gibbs sampler in terms of total number of iterations. The conceptual advantages of the Gibbs sampling approach lies in statistically well-defined error propagation and systematic error correction, and this methodology forms the conceptual basis for the map-making algorithm employed in the BeyondPlanck framework, which implements the first end-to-end Bayesian analysis pipeline for CMB observations., Comment: 11 pages, 10 figures. All BeyondPlanck products and software will be released publicly at http://beyondplanck.science during the online release conference (November 18-20, 2020). Connection details will be made available at the same website. Registration is mandatory for the online tutorial, but optional for the conference

We describe the BeyondPlanck project in terms of motivation, methodology and main products, and provide a guide to a set of companion papers that describe each result in fuller detail. We implement a complete end-to-end Bayesian analysis framework for the Planck LFI observations. The primary product is a full joint posterior distribution $P(\omega|d)$, where $\omega$ represents the set of all free instrumental, astrophysical, and cosmological parameters. Notable advantages of this approach are seamless end-to-end propagation of uncertainties; accurate modeling of both astrophysical and instrumental effects in the most natural basis for each uncertain quantity; optimized computational costs with little or no need for intermediate human interaction between various analysis steps; and a complete overview of the entire analysis process within one single framework. We focus in particular on low-$\ell$ CMB polarization reconstruction with Planck LFI. We identify several important new effects that have not been accounted for in previous pipelines, including gain over-smoothing and time-variable and non-$1/f$ correlated noise in the 30 and 44 GHz channels. We find that all results are consistent with the $\Lambda$CDM model, and we constrain the reionization optical depth to $\tau=0.066\pm0.013$, with a low-resolution $\chi^2$ probability-to-exceed of 32%. This uncertainty is about 30% larger than the official pipelines, arising from taking into account a more complete instrumental model. The marginal CMB Solar dipole amplitude is $3362.7\pm1.4\mu\mathrm{K}$, where the error bar is derived directly from the posterior distribution without the need of any ad-hoc instrumental corrections. We are currently not aware of any significant unmodelled systematic effects remaining in the Planck LFI data, and, for the first time, the 44 GHz channel is fully exploited. (Abridged.), Comment: 61 pages, 31 figures, submitted to A&A. Major revision: Previously reported unresolved effects are now understood and mitigated. This paper version concludes the BeyondPlanck project. All BeyondPlanck products and software are available at https://cosmoglobe.uio.no