Search

We present coniferest, an open source generic purpose active anomaly detection framework written in Python. The package design and implemented algorithms are described. Currently, static outlier detection analysis is supported via the Isolation forest algorithm. Moreover, Active Anomaly Discovery (AAD) and Pineforest algorithms are available to tackle active anomaly detection problems. The algorithms and package performance are evaluated on a series of synthetic datasets. We also describe a few success cases which resulted from applying the package to real astronomical data in active anomaly detection tasks within the SNAD project., Comment: 13 pages, 1 figure

In the task of anomaly detection in modern time-domain photometric surveys, the primary goal is to identify astrophysically interesting, rare, and unusual objects among a large volume of data. Unfortunately, artifacts -- such as plane or satellite tracks, bad columns on CCDs, and ghosts -- often constitute significant contaminants in results from anomaly detection analysis. In such contexts, the Active Anomaly Discovery (AAD) algorithm allows tailoring the output of anomaly detection pipelines according to what the expert judges to be scientifically interesting. We demonstrate how the introduction real-bogus scores, obtained from a machine learning classifier, improves the results from AAD. Using labeled data from the SNAD ZTF knowledge database, we train four real-bogus classifiers: XGBoost, CatBoost, Random Forest, and Extremely Randomized Trees. All the models perform real-bogus classification with similar effectiveness, achieving ROC-AUC scores ranging from 0.93 to 0.95. Consequently, we select the Random Forest model as the main model due to its simplicity and interpretability. The Random Forest classifier is applied to 67 million light curves from ZTF DR17. The output real-bogus score is used as an additional feature for two anomaly detection algorithms: static Isolation Forest and AAD. While results from Isolation Forest remained unchanged, the number of artifacts detected by the active approach decreases significantly with the inclusion of the real-bogus score, from 27 to 3 out of 100. We conclude that incorporating the real-bogus classifier result as an additional feature in the active anomaly detection pipeline significantly reduces the number of artifacts in the outputs, thereby increasing the incidence of astrophysically interesting objects presented to human experts., Comment: 8 pages, 6 figures

Identification of crystal structures is a crucial stage in the exploration of novel functional materials. This procedure is usually time-consuming and can be false-positive or false-negative. This necessitates a significant level of expert proficiency in the field of crystallography and, especially, requires deep experience in perovskite - related structures of hybrid perovskites. Our work is devoted to the machine learning classification of structure types of hybrid lead halides based on available X-ray diffraction data. Here, we proposed a simple approach to quickly identify of dimensionality of inorganic substructures, types of lead halide polyhedra connectivity and structure types using common powder XRD data and ML - decision tree classification model. The average accuracy of our ML algorithm in predicting the dimensionality of inorganic substructure, type of connection of lead halide and inorganic substructure topology by theoretically calculated XRD pattern among 14 most common structure types reaches 0.86+-0.05, 0.827+-0.028 and 0.71+-0.05, respectively. The validation of our decision tree classification ML model on experimental XRD data shows the accuracies of 1.0 and 0.82 for the dimension and structure type prediction. Thus, our approach can significantly simplify and accelerate the interpretation of highly complicated XRD data for hybrid lead halides.

Most of the stars in the Universe are M spectral class dwarfs, which are known to be the source of bright and frequent stellar flares. In this paper, we propose new approaches to discover M-dwarf flares in ground-based photometric surveys. We employ two approaches: a modification of a traditional method of parametric fit search and a machine learning algorithm based on active anomaly detection. The algorithms are applied to Zwicky Transient Facility (ZTF) data release 8, which includes the data from the ZTF high-cadence survey, allowing us to reveal flares lasting from minutes to hours. We analyze over 35 million ZTF light curves and visually scrutinize 1168 candidates suggested by the algorithms to filter out artifacts, occultations of a star by an asteroid, and other types of known variable objects. The result of this analysis is the largest catalogue of ZTF flaring stars to date, representing 134 flares with amplitudes ranging from -0.2 to -4.6 magnitudes, including repeated flares. Using Pan-STARRS DR2 colors, we assign a spectral subclass to each object in the sample. For 13 flares with well-sampled light curves and available geometric distances from Gaia DR3, we estimate the bolometric energy. This research shows that the proposed methods combined with the ZTF's cadence strategy are suitable for identifying M-dwarf flares and other fast transients, allowing for the extraction of significant astrophysical information from their light curves., Comment: 19 pages, 6 figures, 1 table, 1 appendix

We present Rainbow, a physically motivated framework which enables simultaneous multi-band light curve fitting. It allows the user to construct a 2-dimensional continuous surface across wavelength and time, even in situations where the number of observations in each filter is significantly limited. Assuming the electromagnetic radiation emission from the transient can be approximated by a black-body, we combined an expected temperature evolution and a parametric function describing its bolometric light curve. These three ingredients allow the information available in one passband to guide the reconstruction in the others, thus enabling a proper use of multi-survey data. We demonstrate the effectiveness of our method by applying it to simulated data from the Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC) as well as real data from the Young Supernova Experiment (YSE DR1). We evaluate the quality of the estimated light curves according to three different tests: goodness of fit, time of peak prediction and ability to transfer information to machine learning (ML) based classifiers. Results confirm that Rainbow leads to equivalent (SNII) or up to 75% better (SN Ibc) goodness of fit when compared to the Monochromatic approach. Similarly, accuracy when using Rainbow best-fit values as a parameter space in multi-class ML classification improves for all classes in our sample. An efficient implementation of Rainbow has been publicly released as part of the light curve package at https://github.com/light-curve/light-curve-python. Our approach enables straight forward light curve estimation for objects with observations in multiple filters and from multiple experiments. It is particularly well suited for situations where light curve sampling is sparse., Comment: 14 pages, 15 figures, submitted to A&A

We report here on results of experimental-theoretical investigation of high-order harmonic generation (HHG) in layers of CdSe semiconductor quantum dots of different sizes and a reference bulk CdSe thin film. We observe a strong decrease in the efficiency, up to complete suppression of HHG with energies of quanta above the bandgap for the smallest dots, whereas the intensity of below bandgap harmonics remains weakly affected by the dot size. In addition, it is observed that the ratio between suppression of above gap harmonics versus below gap harmonics increases with driving wavelength. We suggest that the reduction in the dot size below the classical electron oscillatory radius and the corresponding off the dots wall scattering limits the maximum acceleration by the laser field. Moreover, this scattering leads to a chaotization of motion, causing dephasing and a loss of coherence, therefore suppressing the efficiency of the emission of highest-order harmonics. Our results demonstrate a new regime of intense laser-nanoscale solid interaction, intermediate between the bulk and single molecule response., Comment: 32 pages, 19 Figures

We report the automatic detection of 11 transients (7 possible supernovae and 4 active galactic nuclei candidates) within the Zwicky Transient Facility fourth data release (ZTF DR4), all of them observed in 2018 and absent from public catalogs. Among these, three were not part of the ZTF alert stream. Our transient mining strategy employs 41 physically motivated features extracted from both real light curves and four simulated light curve models (SN Ia, SN II, TDE, SLSN-I). These features are input to a k-D tree algorithm, from which we calculate the 15 nearest neighbors. After pre-processing and selection cuts, our dataset contained approximately a million objects among which we visually inspected the 105 closest neighbors from seven of our brightest, most well-sampled simulations, comprising 89 unique ZTF DR4 sources. Our result illustrates the potential of coherently incorporating domain knowledge and automatic learning algorithms, which is one of the guiding principles directing the SNAD team. It also demonstrates that the ZTF DR is a suitable testing ground for data mining algorithms aiming to prepare for the next generation of astronomical data., Comment: 30 pages, 5 figures, 3 tables, preprint accepted to New Astronomy

We present results from applying the SNAD anomaly detection pipeline to the third public data release of the Zwicky Transient Facility (ZTF DR3). The pipeline is composed of 3 stages: feature extraction, search of outliers with machine learning algorithms and anomaly identification with followup by human experts. Our analysis concentrates in three ZTF fields, comprising more than 2.25 million objects. A set of 4 automatic learning algorithms was used to identify 277 outliers, which were subsequently scrutinised by an expert. From these, 188 (68%) were found to be bogus light curves -- including effects from the image subtraction pipeline as well as overlapping between a star and a known asteroid, 66 (24%) were previously reported sources whereas 23 (8%) correspond to non-catalogued objects, with the two latter cases of potential scientific interest (e. g. 1 spectroscopically confirmed RS Canum Venaticorum star, 4 supernovae candidates, 1 red dwarf flare). Moreover, using results from the expert analysis, we were able to identify a simple bi-dimensional relation which can be used to aid filtering potentially bogus light curves in future studies. We provide a complete list of objects with potential scientific application so they can be further scrutinised by the community. These results confirm the importance of combining automatic machine learning algorithms with domain knowledge in the construction of recommendation systems for astronomy. Our code is publicly available at https://github.com/snad-space/zwad, Comment: 31 pages, 28 figures, 4 tables, accepted for publication in MNRAS

A method is proposed for measurements of the transparency coefficient of the grids used in ionization chambers. The method is based on application of the Green's reciprocity theorem and gives an accuracy of (3 - 5)%. The experimental data are compared with the results of theoretical calculations for the case of flat grids.

We consider a multidimensional inhomogeneous birth-death process (BDP) and obtain bounds on the rate of convergence for the corresponding one-dimensional processes.

We present new mixture representations for the generalized Linnik distribution in terms of normal, Laplace, exponential and stable laws and establish the relationship between the mixing distributions in these representations. Based on these representations, we prove some limit theorems for a wide class of rather simple statistics constructed from samples with random sized including, e. g., random sums of independent random variables with finite variances and maximum random sums, in which the generalized Linnik distribution plays the role of the limit law. Thus we demonstrate that the scheme of geometric (or, in general, negative binomial) summation is far not the only asymptotic setting (even for sums of independent random variables) in which the generalized Linnik law appears as the limit distribution., Comment: 19 pages. arXiv admin note: text overlap with arXiv:1602.02480, arXiv:1506.02778

In this paper we study a non-stationary Markovian queueing model of a two-processor heterogeneous system with time-varying arrival and service rates. We obtain the bounds on the rate of convergence and find the main limiting characteristics of the queue-length process.

The approaches, based on the negative binomial model for the distribution of duration of the wet periods measured in days, are proposed to the definition of extreme precipitation. This model demonstrates excellent fit with real data and provides a theoretical base for the determination of asymptotic approximations to the distributions of the maximum daily precipitation volume within a wet period as well as the total precipitation volume over a wet period. The first approach to the definition (and determination) of extreme precipitation is based on the tempered Snedecor-Fisher distribution of the maximum daily precipitation. According to this approach, a daily precipitation volume is considered to be extreme, if it exceeds a certain (pre-defined) quantile of the tempered Snedecor--Fisher distribution. The second approach is based on that the total precipitation volume for a wet period has the gamma distribution. Hence, the hypothesis that the total precipitation volume during a certain wet period is extremely large can be formulated as the homogeneity hypothesis of a sample from the gamma distribution. Two equivalent tests are proposed for testing this hypothesis. Both of these tests deal with the relative contribution of the total precipitation volume for a wet period to the considered set (sample) of successive wet periods. Within the second approach it is possible to introduce the notions of relatively and absolutely extreme precipitation volumes. The results of the application of these tests to real data are presented yielding the conclusion that the intensity of wet periods with extreme large precipitation volume increases., Comment: 21 pages, 10 figures, 2 tables

Based on the negative binomial model for the duration of wet periods measured in days, an asymptotic approximation is proposed for the distribution of the maximum daily precipitation volume within a wet period. This approximation has the form of a scale mixture of the Frechet distribution with the gamma mixing distribution and coincides with the distribution of a positive power of a random variable having the Snedecor-Fisher distribution. The proof of this result is based on the representation of the negative binomial distribution as a mixed geometric (and hence, mixed Poisson) distribution and limit theorems for extreme order statistics in samples with random sizes having mixed Poisson distributions. Some analytic properties of the obtained limit distribution are described. In particular, it is demonstrated that under certain conditions the limit distribution is mixed exponential and hence, is infinitely divisible. It is shown that under the same conditions the limit distribution can be represented as a scale mixture of stable or Weibull or Pareto or folded normal laws. The corresponding product representations for the limit random variable can be used for its computer simulation. Several methods are proposed for the estimation of the parameters of the distribution of the maximum daily precipitation volume. The results of fitting this distribution to real data are presented illustrating high adequacy of the proposed model. The obtained mixture representations for the limit laws and the corresponding asymptotic approximations provide better insight into the nature of mixed probability ("Bayesian") models., Comment: 17 pages, 6 figures. arXiv admin note: text overlap with arXiv:1703.07276

In the present paper we demonstrate the results of a statistical analysis of some characteristics of precipitation events and propose a kind of a theoretical explanation of the proposed models in terms of mixed Poisson and mixed exponential distributions based on the information-theoretical entropy reasoning. The proposed models can be also treated as the result of following the popular Bayesian approach., Comment: 5 pages, 4 figures; ICNAAM 2016

In this paper we study a wide and flexible family of discrete distributions, the so-called generalized negative binomial (GNB) distributions that are mixed Poisson distributions in which the mixing laws belong to the class of generalized gamma (GG) distributions. The latter was introduced by E. W. Stacy as a special family of lifetime distributions containing gamma, exponential power and Weibull distributions. These distributions seem to be very promising in the statistical description of many real phenomena being very convenient and almost universal models for the description of statistical regularities in discrete data. Analytic properties of GNB distributions are studied. A GG distribution is proved to be a mixed exponential distribution if and only if the shape and exponent power parameters are no greater than one. The mixing distribution is written out explicitly as a scale mixture of strictly stable laws concentrated on the nonnegative halfline. As a corollary, the representation is obtained for the GNB distribution as a mixed geometric distribution. The corresponding scheme of Bernoulli trials with random probability of success is considered. Within this scheme, a random analog of the Poisson theorem is proved establishing the convergence of mixed binomial distributions to mixed Poisson laws. Limit theorems are proved for random sums of independent random variables in which the number of summands has the GNB distribution and the summands have both light- and heavy-tailed distributions. The class of limit laws is wide enough and includes the so-called generalized variance gamma distributions. Various representations for the limit laws are obtained in terms of mixtures of Mittag-Leffler, Linnik or Laplace distributions. Some applications of GNB distributions in meteorology are discussed.

Estimates are constructed for the deviation of the concentration functions of sums of independent random variables with finite variances from the folded normal distribution function without any assumptions concerning the existence of the moments of summands of higher orders. The obtained results are extended to Poisson-binomial, binomial and Poisson random sums. Under the same assumptions, the bounds are obtained for the approximation of the concentration functions of mixed Poisson random sums by the corresponding limit distributions. In particular, bounds are obtained for the accuracy of approximation of the concentration functions of geometric, negative binomial and Sichel random sums by the exponential, the folded variance gamma and the folded Student distribution. Numerical estimates of all the constants involved are written out explicitly., Comment: 14 pages. Research supported by the Russian Foundation for Basic Research, project 15-07-02984. arXiv admin note: substantial text overlap with arXiv:1507.00979

We present some product representations for random variables with the Linnik, Mittag-Leffler and Weibull distributions and establish the relationship between the mixing distributions in these representations. Based on these representations, we prove some limit theorems for a wide class of rather simple statistics constructed from samples with random sized including, e. g., random sums of independent random variables with finite variances, maximum random sums, extreme order statistics, in which the Linnik and Mittag-Leffler distributions play the role of limit laws. Thus we demonstrate that the scheme of geometric summation is far not the only asymptotic setting (even for sums of independent random variables) in which the Mittag-Leffler and Linnik laws appear as limit distributions. The two-sided Mittag-Leffler and one-sided Linnik distribution are introduced and also proved to be limit laws for some statistics constructed from samples with random sizes.