Search

Coronal Mass Ejections~(CMEs) are extremely dynamical large scale events in which plasma-not only the coronal plasma-is ejected into the interplanetary space. Their interplanetary counterparts measured in-situ are Interplanetary Coronal Mass Ejections (ICMEs), which is also an important part of space weather.Even though the kinetic properties of the plasma might change because of dynamic effects occurring during the expansion of the CME, the heavy ion characteristics remain unchanged after it leaves the low corona. Charge states of heavy ions reflect important information about the coronal temperature profile due to the freeze-in effect, while elemental abundances indicate potential source regions of the plasma.With the help of the Pulse Height Analysis (PHA) data from the Solar Wind Ion Composition Spectromet (SWICS) on board the Advanced Composition Explorer (ACE), combined with a newly developed multi-population model, we are able to conduct a high time resolution (12 minutes) case study on a complex ejecta detected by ACE in May 2005. This case lasted more than 80 hours and caused a strong geomagnetic response, with a Dst index at -247.Multiple discontinuous periods with highly charged heavy ions are identified, elemental abundances also differ during those ”hot” periods. Heavy ion characteristics provide us an unique opportunity to see the boundaries of different parts of an ICME.

Since the 1970s it has been empirically known that the area of solar coronal holes a ects the properties of high-speed solar windstreams (HSSs) at Earth. We derive a simple analytical model for the propagation of HSSs from the Sun to Earth and thereby showhow the area of coronal holes and the size of their boundary regions a ect the HSS velocity, temperature, and density near Earth.We assume that velocity, temperature, and density profiles form across the HSS cross section close to the Sun and that these spatialprofiles translate into corresponding temporal profiles in a given radial direction due to the solar rotation. These temporal distributionsdrive the stream interface to the preceding slow solar wind plasma and disperse with distance from the Sun. The HSS properties at1AU are then given by all HSS plasma parcels launched from the Sun that did not run into the stream interface at Earth distance.We show that the velocity plateau region of HSSs as seen at 1AU, if apparent, originates from the center region of the HSS closeto the Sun, whereas the velocity tail at 1AU originates from the trailing boundary region. Small HSSs can be described to entirelyconsist of boundary region plasma, which intrinsically results in smaller peak velocities. The peak velocity of HSSs at Earth furtherdepends on the longitudinal width of the HSS close to the Sun. The shorter the longitudinal width of an HSS close to the Sun, themore of its “fastest” HSS plasma parcels from the HSS core and trailing boundary region have impinged upon the stream interfacewith the preceding slow solar wind, and the smaller is the peak velocity of the HSS at Earth. As the longitudinal width is statisticallycorrelated to the area of coronal holes, this also explains the well-known empirical relationship between coronal hole areas and HSSpeak velocities. Further, the temperature and density of HSS plasma parcels at Earth depend on their radial expansion from the Sunto Earth. The radial expansion is determined by the velocity gradient across the HSS boundary region close to the Sun and gives thevelocity-temperature and density-temperature relationships at Earth their specific shape. When considering a large number of HSSs,the assumed correlation between the HSS velocities and temperatures close to the Sun degrades only slightly up to 1AU, but thecorrelation between the velocities and densities is strongly disrupted up to 1AU due to the radial expansion. Finally, we show howthe number of particles of the piled-up slow solar wind in the stream interaction region depends on the velocities and densities of theHSS and preceding slow solar wind plasma.

The properties of the solar wind represent a mixture of indicators for solar origin and transport effects. Both are of interest for the understanding of heliophysics and space weather effects. Most available solar wind classifications focus on the solar origin, in part based on transport effected properties. We aim to identify the solar wind properties that are most important for solar wind classification. We select seven solar wind properties: proton density, proton speed, proton temperature, absolute magnetic field strength, proton-proton collisional age, the ratio between the densities of O6+ and O7+ and the mean charge state of Fe. We apply an unsupervised machine learning method, k-means, to each subset of the these parameters and compare the results to a reference case based on all seven solar wind properties. Two scenarios are considered which provide a simple and a detailed solar wind classification, respectively. We identified the proton density as the most important solar wind property for solar wind classification. Furthermore, we found that charge state composition is important to accurately identify the solar source region. This holds for the simple case of three solar wind types but is even more important for a more detailed classification. In comparison to proton density and proton temperature, the solar wind speed turns out to be a less influential property. Our results underscore the importance of highly accurate measurements, in particular for proton density, proton temperature and the charge state composition.

Context. Interplanetary collisionless shocks are known to be sources of energetic charged particles up to hundreds of MeV. However, the underlying acceleration mechanisms are still under debate.Aims. We determine the properties of suprathermal protons accelerated by the interplanetary shock on 2021 November 3 with the unprecedented high-resolution measurements by the SupraThermal Electron Proton sensor of the Energetic Particle Detector onboard the Solar Orbiter spacecraft, in order to constrain the potential shock acceleration mechanisms.Methods. We first reconstruct the pitch-angle distributions (PADs) of suprathermal protons in the solar wind frame. Then, we study the evolution of the PADs, flux temporal profile and velocity distribution function of this proton population close to the shock and compare the observations to theoretical predictions.Results. We find that the suprathermal proton fluxes peak ∼12 to ∼24 seconds before the shock in the upstream region. The proton fluxes rapidly decrease by ∼50% in a thin layer (∼8000 km) adjacent to the shock in the downstream region and become constant further downstream. Furthermore, the proton velocity distribution functions in the upstream (downstream) region fit to a double power law, f (v) ∼ v−γ, at ∼1000 − 3600 km s−1, with a γ of ∼3.4 ± 0.2 (∼4.3 ± 0.7) at velocities (v) below a break at ∼1800 ± 100 km s−1 (∼1600 ± 200 km s−1) and a γ of ∼5.8 ± 0.3 (∼5.8 ± 0.2) at velocities above. These indices are all smaller than predicted by first-order Fermi acceleration. In addition, the proton PADs show anisotropies in the direction away from the shock in the close upstream region and become nearly isotropic further upstream, while downstream of the shock, they show a clear tendency of anisotropies towards 90◦ PA.Conclusions. These results suggest that the acceleration of suprathermal protons at interplanetary shocks are dynamic on a time scale of ∼10 seconds, i.e., few proton gyro-periods. Furthermore, shock drift acceleration likely plays an important role in accelerating these suprathermal protons.

Context. Coronal mass ejections (CMEs) are extremely dynamical, large-scale events in which plasma – but not only the coronal plasma – is ejected into interplanetary space. If a CME is detected in situ by a spacecraft located in the interplanetary medium, it is then called an interplanetary coronal mass ejection (ICME). This solar activity has been studied widely since coronagraphs were first flown into space in the early 1970s. Aims. Charge states of heavy ions reflect important information about the coronal temperature profile due to the freeze-in effect and it is estimated that iron ions freeze in at heights of ∼5 solar radii. However, the measured charge-state distribution of iron ions cannot be composed of only one single group of plasma. To identify the different populations of iron charge-state composition of ICMEs and determine their sources, we developed a model that independently uses two, three, and four populations of iron ions to fit the measured charge-state distribution in ICMEs detected by the Advanced Composition Explorer (ACE) at 1 AU. Methods. Three parameters are used to identify a certain population, namely freeze-in temperature, relative abundance, and kappa value (κ), which together describe the potential non-Maxwellian kappa distributions of coronal electrons. Our method chooses the reduced chi-squared to describe the goodness of fit of the model to the observations. The parameters of our model are optimized with the covariance-matrix-adaptation evolution strategy (CMA-ES). Results. Two major types of ICMEs are identified according to the existence of hot material, and both, that is, the cool type and the hot type, have two main subtypes. Different populations in those types have their own features related to freeze-in temperature and κ. The electron velocity distribution function usually contains a significant hot tail in typical coronal material and hot material, while the Maxwellian distribution appears more frequently in mid-temperature material. Our model is also suitable for all types of solar wind and the existence of hot populations as well as the change of temperatures of individual populations may indicate boundaries between ICMEs and individual solar wind streams.

Since the 1970s it has been empirically known that the area of solar coronal holes affects the properties of high-speed solar wind streams (HSSs) at Earth. We derive a simple analytical model for the propagation of HSSs from the Sun to Earth and thereby show how the area of coronal holes and the size of their boundary regions affect the HSS velocity, temperature, and density near Earth. We assume that velocity, temperature, and density profiles form across the HSS cross section close to the Sun and that these spatial profiles translate into corresponding temporal profiles in a given radial direction due to the solar rotation. These temporal distributions drive the stream interface to the preceding slow solar wind plasma and disperse with distance from the Sun. The HSS properties at 1 AU are then given by all HSS plasma parcels launched from the Sun that did not run into the stream interface at Earth distance. We show that the velocity plateau region of HSSs as seen at 1 AU, if apparent, originates from the center region of the HSS close to the Sun, whereas the velocity tail at 1 AU originates from the trailing boundary region. Small HSSs can be described to entirely consist of boundary region plasma, which intrinsically results in smaller peak velocities. The peak velocity of HSSs at Earth further depends on the longitudinal width of the HSS close to the Sun. The shorter the longitudinal width of an HSS close to the Sun, the more of its “fastest” HSS plasma parcels from the HSS core and trailing boundary region have impinged upon the stream interface with the preceding slow solar wind, and the smaller is the peak velocity of the HSS at Earth. As the longitudinal width is statistically correlated to the area of coronal holes, this also explains the well-known empirical relationship between coronal hole areas and HSS peak velocities. Further, the temperature and density of HSS plasma parcels at Earth depend on their radial expansion from the Sun to Earth. The radial expansion is determined by the velocity gradient across the HSS boundary region close to the Sun and gives the velocity-temperature and density-temperature relationships at Earth their specific shape. When considering a large number of HSSs, the assumed correlation between the HSS velocities and temperatures close to the Sun degrades only slightly up to 1 AU, but the correlation between the velocities and densities is strongly disrupted up to 1 AU due to the radial expansion. Finally, we show how the number of particles of the piled-up slow solar wind in the stream interaction region depends on the velocities and densities of the HSS and preceding slow solar wind plasma.

Context. The potential field source surface (PFSS) model is an important tool that helps link the solar coronal magnetic field to the solar wind. Due to its simplicity, it allows for predictions to be computed rapidly and requires little input data, though at the cost of reduced accuracy compared to more complex models. So far, PFSS models have almost exclusively been computed for a spherical outer boundary or “source surface”. Changing this to an elliptical source surface holds promise to increase its prediction accuracy without the necessity of incorporating complex computations or additional model assumptions. Aims. The main goal of this work is to evaluate the merit of adding another parameter, namely the ellipticity of the source surface, to the PFSS model. In addition, the applicability of the PFSS model during different periods of the solar activity cycle as well as the impact of the source surface radius are analyzed. Methods. To evaluate the model, in situ spacecraft data are mapped back to the source surface via a ballistic approach. The in situ magnetic field polarity is compared to the magnetic field polarity predicted by the model at the source surface. This method is based on the assumption that better performing models provide better agreements between the prediction and the measured magnetic field polarity. We employ data from the Advanced Composition Explorer and the twin Solar Terrestrial Relations Observatory (STEREO) for this analysis. Results. We show that the PFSS model performs slightly better with oblate elliptical source surfaces elongated along the solar equatorial plane, although the best found ellipticity varies for different spacecraft and periods. In addition, it is demonstrated that the performance of the presented analysis degrades during the active times of the solar activity cycle.

Context. The properties of a solar wind stream are determined by its source region and by transport effects. Independently of the solar wind type, the solar wind measured in situ is always affected by both. This means that reliably determining the solar wind type from in situ observations is useful for the analysis of its solar origin and its evolution during the travel time to the spacecraft that observes the solar wind. In addition, the solar wind type also influences the interaction of the solar wind with other plasma such as Earth’s magnetosphere. Aims. We consider the proton-proton collisional age as an ordering parameter for the solar wind at 1 AU and explore its relation to the solar wind classification scheme developed by Xu & Borovsky (2015, J. Geophys. Res.: Space Phys., 120, 70). We use this to show that explicit magnetic field information is not required for this solar wind classification. Furthermore, we illustrate that solar wind classification schemes that rely on threshold values of solar wind parameters should depend on the phase in the solar activity cycle since the respective parameters change with the solar activity cycle. Methods. The categorization of the solar wind following Xu & Borovsky (2015, J. Geophys. Res.: Space Phys., 120, 70) was taken as our reference for determining the solar wind type. Based on the observation that the three basic solar wind types from this categorization cover different regimes in terms of proton-proton collisional age acol, p-p, we propose a simplified solar wind classification scheme that is only based on the proton-proton collisional age. We call the resulting method the PAC solar wind classifier. For this purpose, we derive time-dependent threshold values in the proton-proton collisional age for two variants of the proposed PAC scheme: (1) similarity-PAC is based on the similarity to the full Xu & Borovsky (2015, J. Geophys. Res.: Space Phys., 120, 70) scheme, and (2) distribution-PAC is based directly on the distribution of the proton-proton collisional age. Results. The proposed simplified solar wind categorization scheme based on the proton-proton collisional age represents an equivalent alternative to the full Xu & Borovsky (2015, J. Geophys. Res.: Space Phys., 120, 70) solar wind classification scheme and leads to a classification that is very similar to the full Xu & Borovsky (2015, J. Geophys. Res.: Space Phys., 120, 70) scheme. The proposed PAC solar wind categorization separates coronal hole wind from helmet-streamer plasma as well as helmet-streamer plasma (slow solar wind without a current sheet crossing) from sector-reversal plasma (slow solar wind with a current sheet crossing). Unlike the full Xu & Borovsky (2015, J. Geophys. Res.: Space Phys., 120, 70) scheme, PAC does not require information on the magnetic field as input. Conclusions. The solar wind is well ordered by the proton-proton collisional age. This implies underlying intrinsic relationships between the plasma properties, in particular, proton temperature and magnetic field strength in each plasma regime. We argue that sector-reversal plasma is a combination of particularly slow and dense solar wind and most stream interaction boundaries. Most solar wind parameters (e.g., the magnetic field strength, B, and the oxygen charge state ratio no7+/no6+) change with the solar activity cycle. Thus, all solar wind categorization schemes based on threshold values need to be adapted to the solar activity cycle as well. Because it does not require magnetic field information but only proton plasma measurements, the proposed PAC solar wind classifier can be applied directly to solar wind data from the Solar and Heliospheric Observatoty (SOHO), which is not equipped with a magnetometer.

Context. The elemental composition of the solar wind differs from the solar photospheric composition. Elements with low first ionization potential (FIP) appear enhanced compared to O in the solar wind relative to the respective photospheric abundances. This so-called FIP effect is different in the slow solar wind and the coronal hole wind. However, under the same plasma conditions, for elements with similar FIPs such as Mg, Si, and Fe, comparable enhancements are expected. Aims. We scrutinize the assumption that the FIP effect is always similar for different low FIP elements, namely Mg, Si, and Fe. Methods. Here we investigate the dependency of the FIP effect of low FIP elements on the O7+/O6+ charge state ratio depending on time, that is the solar activity cycle, and solar wind type. In addition, we order the observed FIP ratios with respect to the O7+/O6+ charge state ratio into bins and analyze separately the respective distributions of the FIP ratio of Mg, Si, and Fe for each O7+/O6+ charge state ratio bin. Results. We observe that the FIP effect shows the same qualitative yearly behavior for Mg and Si, while Fe shows significant differences during the solar activity maximum and its declining phase. In each year, the FIP effect for Mg and Si always increases with increasing O7+/O6+ charge state ratio, but for high O7+/O6+ charge state ratios the FIP effect for Fe shows a qualitatively different behavior. During the years 2001–2006, instead of increasing with the O7+/O6+ charge state ratio, the Fe FIP ratio exhibits a broad peak or plateau. In addition, the FIP distribution per O7+/O6+ charge state bin is significantly broader for Fe than for Mg and Si. Conclusions. These observations support the conclusion that the elemental fractionation is only partly determined by FIP. In particular, the qualitative difference in behavior with increasing O7+/O6+ charge state ratio between Fe on the one hand and Mg and Si on the other hand is not yet well explained by models of fractionation.

The Sun is a highly dynamic environment that exhibits dynamic behavior on many different timescales. In particular, coronal holes exhibit temporal and spatial variability. Signatures of these coronal dynamics are inherited by the coronal hole wind streams that originate in these regions and can effect the Earth's magnetosphere. Both the cause of the observed variabilities and how these translate to fluctuations in the in situ observed solar wind is not yet fully understood. During solar activity minimum the structure of the magnetic field typically remains stable over several Carrington rotations (CRs). But how stable is the solar magnetic field? Here, we address this question by analyzing the evolution of a coronal hole structure and the corresponding coronal hole wind stream emitted from this source region over 12 consecutive CRs in 2006. To this end, we link in situ observations of Solar Wind Ion Composition Spectrometer (SWICS) onboard the Advanced Composition Explorer (ACE) with synoptic maps of Michelson Doppler imager (MDI) on the Solar and Heliospheric Observatory (SOHO) at the photospheric level through a combination of ballistic back-mapping and a potential field source surface (PFSS) approach. Together, these track the evolution of the open field line region that is identified as the source region of a recurring coronal hole wind stream. We find that the shape of the open field line region and to some extent also the solar wind properties are influenced by surrounding more dynamic closed loop regions. We show that the freeze-in order can change within a coronal hole wind stream on small timescales and illustrate a mechanism that can cause changes in the freeze-in order. The inferred minimal temperature profile is variable even within coronal hole wind and is in particular most variable in the outer corona.

The solar wind represents the background into which other space weather phenomena are embedded. Interplanetary coronal mass ejections (ICMEs) travel through and interact with the background solar wind. The nature of this interaction depends on plasma structures, the solar wind, and the typically faster ICME. It is well known that the solar wind that is continuously emitted from the Sun comes in at least two varieties. These are historically called slow solar wind and fast solar wind, although their most distinctive property is not the solar wind speed, but the elemental and charge state compositions. So far, solar wind categorization has been approached with different combinations of hand-crafted classifiers based on expert knowledge and heuristic methods. As a result, the actual number of different solar wind types differs between typical approaches and, in particular, the decision boundaries between solar wind regimes differ considerably. In this situation, a purely data-driven approach that aims at dividing solar wind plasma into categories based on their similarity to other observations can be insightful. Such an approach is in machine learning, realized by unsupervised clustering methods such as k-means clustering. Given a sufficiently large data set, k-means clustering can improve and validate the available solar wind categorization schemes. Here, we apply k-means clustering to solar wind data measured by instruments on the Advanced Composition Explorer, compare the resulting solar wind types to existing heuristic solar wind categorization schemes, determine the probable number of distinct solar wind types that are supported by the observations, discuss the physical interpretation of the resulting solar wind types, and finally utilize the k-means approach to investigate feature selection for solar wind categorization.

Context. The interstellar flow longitude corresponds to the Sun’s direction of movement relative to the local interstellar medium. Thus, it constitutes a fundamental parameter for our understanding of the heliosphere and, in particular, its interaction with its surroundings, which is currently investigated by the Interstellar Boundary EXplorer (IBEX). One possibility to derive this parameter is based on pickup ions (PUIs) that are former neutral ions that have been ionized in the inner heliosphere. The neutrals enter the heliosphere as an interstellar wind from the direction of the Sun’s movement against the partially ionized interstellar medium. PUIs carry information about the spatial variation of their neutral parent population (density and flow vector field) in their velocity distribution function. From the symmetry of the longitudinal flow velocity distribution, the interstellar flow longitude can be derived.Aim. The aim of this paper is to identify and eliminate systematic errors that are connected to this approach of measuring the interstellar flow longitude; we want to minimize any systematic influences on the result of this analysis and give a reasonable estimate for the uncertainty.Methods. We use He+ data measured by the PLAsma and SupraThermal Ion Composition (PLASTIC) sensor on the Solar TErrestrial RElations Observatory Ahead (STEREO A) spacecraft. We analyze a recent approach, identify sources of systematic errors, and propose solutions to eliminate them. Furthermore, a method is introduced to estimate the error associated with this approach. Additionally, we investigate how the selection of interplanetary magnetic field angles, which is closely connected to the pickup ion velocity distribution function, affects the result for the interstellar flow longitude.Results. We find that the revised analysis used to address part of the expected systematic effects obtains significantly different results than presented in the previous study. In particular, the derived uncertainties are considerably larger. Furthermore, an unexpected systematic trend of the resulting interstellar flow longitude with the selection of interplanetary magnetic field orientation is uncovered.

Because of their convincing performance, there is a growing interest in using evolutionary algorithms for reinforcement learning. We propose learning of neural network policies by the covariance matrix adaptation evolution strategy (CMA-ES), a randomized variable-metric search algorithm for continuous optimization. We argue that this approach, which we refer to as CMA Neuroevolution Strategy (CMA-NeuroES), is ideally suited for reinforcement learning, in particular because it is based on ranking policies (and therefore robust against noise), efficiently detects correlations between parameters, and infers a search direction from scalar reinforcement signals. We evaluate the CMA-NeuroES on five different (Markovian and non-Markovian) variants of the common pole balancing problem. The results are compared to those described in a recent study covering several RL algorithms, and the CMA-NeuroES shows the overall best performance.

The solar wind originating from coronal holes is comparatively well-understood and is characterized by lower densities and average charge states compared to the so-called slow solar wind. Except for wave perturbations, the average properties of the coronal-hole solar wind are passably constant. In this case study, we {ocus on observations of the Solar Wind Ion Composition Spectrometer (SWICS) on the Advanced Composition Explorer (ACE) of individual streams of coronal-hole solar wind that illustrate that although the O and C charge states are low in coronal-hole wind, the Fe charge distribution is more variable. In particular, \markmeold{we illustrate that the Fe charge states in coronal-hole solar wind are frequently as high as in slow solar wind. We selected individual coronal-hole solar wind} streams based on their collisional age as well as their respective O and C charge states and analyzed their Fe charge-state distributions. Additionally, with a combination of simple ballistic back-mapping and the potential field source surface model, transitions between streams with high and low Fe charge states were mapped back to the photosphere. The relative frequency of high and low Fe charge-state streams is compared for the years 2004 and 2006. We found several otherwise typical coronal-hole streams that include Fe charge states either as high as or lower than in slow solar wind. Eight such transitions in 2006 were mapped back to equatorial coronal holes that were either isolated or connected to the northern coronal-hole. Attempts to identify coronal structures associated with the transitions were so far inconclusive.

A recent paper [Shearer et al., 2014] reported that during solar maximum Ne showed a surprisingly low abundance. This leads to the question whether other elements show the same behavior. The good mass resolution of Mass-Time-Of-Flight (MTOF) as part of the Charge ELement and Isotope Analysis System (CELIAS) on the Solar Helioshperic Observatory (SOHO) allows to investigate the composition of heavy minor elements in different types of solar wind. We restrict this study to slow solar wind, where the characterisation of slow solar wind is taken from Xu and Borovsky, 2014. This classification scheme requires magnet field information. Since SOHO does not carry a magnetometer, we use the Magnetometer (MAG) of the Advanced Composition Explorer (ACE) instead. The Solar Wind Ion Composition Spectrometer (ACE/SWICS) also provides composition data for cross-calibration and charge-state distributions as input for the transmission function of MTOF whenever the two spacecraft can be expected to observe the same type of wind. We illustrate the MTOF’s capability to determine the solar wind abundance compared to the photospheric abundance (called the FIP ratio in the following) for rare elements like Ti or Cr on long-time scales as a proof of concept for our analysis. And in this brief study, measurements with both ACE/SWICS indicate that the observed elements exhibit a (weak) dependence on the solar cycle, whereas the MTOF measurements are inconclusive.

Welcome to the Black-Box Optimization Benchmarking (BBOB) 2012 workshop!This workshop is a follow up of the BBOB 2009 workshop in Montreal and the BBOB 2010 workshop in Portland. The workshop focuses on benchmarking continuous optimizers, a crucial task to assess the performance of optimizers quantitatively, to understand the weaknesses and strengths of each algorithm and a compulsory path to evaluate new algorithm designs. Because this task is tedious and difficult to realize, we provide the participants with: the choice and implementation of two well-motivated single-objective benchmark function testbeds for noiseless and noisy optimization,the design of an experimental set-up,the generation of data output, andthe post-processing and presentation of the results in graphs and tables.Over the years, the BBOB workshop has established a standard for benchmarking continuous optimizers. The code needed to run the experiments and to post-process the data as well as the data collected during the previous editions of the workshop are available on the website http://coco.gforge.inria.fr/doku.php?id=start.We would like to thank all the participants for their contributions and are happy to announce that we have accepted 19 papers. For further news about the workshop please check our webpage, and subscribe to the mailing lists!

Verstärkungslernen (Reinforcement learning, RL) ist ein Untergebiet des Maschinellen Lernens. Ein Agent befindet sich in einer Umwelt und kommuniziert mit dieser durch fest vorgegebene Kanäle. Der Agent beobachtet den Zustand seiner Umwelt und wählt basierend auf dieser Beobachtung und seiner internen Handlungsstrategie eine Aktion, die von der Umwelt ausgeführt wird. Schließlich erhält der Agent von der Umwelt ein skalares Verstärkungslernsignal, das das Verhalten des Agenten bewertet. Im ersten Teil werden die konzeptuellen Gemeinsamkeiten und Unterschiede zwischen policy gradient Methoden und evolutionärem RL untersucht und experimentell nachgewiesen. Der zweite Teil der Arbeit befasst sich mit der weiteren Verbesserung der Rauschkontrolle für evolutionäre Algorithmen. Selektionsrennen werden hergeleitet, ihre theoretischen Eigenschaften dargestellt und ihr Verhalten wird experimentell untersucht und mit einem kompetitiven Verfahren verglichen.

Uncertainty arises in reinforcement learning from various sources, and therefore it is necessary to consider statistics based on several roll-outs for evaluating behavioral policies. We add an adaptive uncertainty handling based on Hoeffding and empirical Bernstein races to the CMA-ES, a variable metric evolution strategy proposed for direct policy search. The uncertainty handling adjusts individually the number of episodes considered for the evaluation of a policy. The performance estimation is kept just accurate enough for a sufficiently good ranking of candidate policies, which is in turn sufficient for the CMA-ES to find better solutions. This increases the learning speed as well as the robustness of the algorithm.

Two variable metric reinforcement learning methods, the natural actor-critic algorithm and the covariance matrix adaptation evolution strategy, are compared on a conceptual level and analysed experimentally on the mountain car benchmark task with and without noise.