Search

Understanding what is normal is a key aspect of protecting a domain. Other domains invest heavily in observational science to develop models of normal behavior to better detect anomalies. Recent advances in high performance graph libraries, such as the GraphBLAS, coupled with supercomputers enables processing of the trillions of observations required. We leverage this approach to synthesize low-parameter observational models of anonymized Internet traffic with a high regard for privacy., Comment: Accepted to IEEE HPEC, 7 pages, 6 figures, 1 table, 41 references

One of the more complex tasks for researchers using HPC systems is performance monitoring and tuning of their applications. Developing a practice of continuous performance improvement, both for speed-up and efficient use of resources is essential to the long term success of both the HPC practitioner and the research project. Profiling tools provide a nice view of the performance of an application but often have a steep learning curve and rarely provide an easy to interpret view of resource utilization. Lower level tools such as top and htop provide a view of resource utilization for those familiar and comfortable with Linux but a barrier for newer HPC practitioners. To expand the existing profiling and job monitoring options, the MIT Lincoln Laboratory Supercomputing Center created LLoad, a tool that captures a snapshot of the resources being used by a job on a per user basis. LLload is a tool built from standard HPC tools that provides an easy way for a researcher to track resource usage of active jobs. We explain how the tool was designed and implemented and provide insight into how it is used to aid new researchers in developing their performance monitoring skills as well as guide researchers in their resource requests.

We study the phase diagram of the antiferromagnetic $J_1$-$J_2$ Heisenberg model on the pyrochlore lattice with $S=1$ spins at zero and finite temperatures. We use a combination of complementary state-of-the-art quantum many-body approaches such as density matrix renormalization group (DMRG), density-matrix purification and pseudo-Majorana functional renormalization group (PMFRG). We present an efficient approach to preserve the applicability of the PMFRG for spin-1 systems at finite temperatures despite the inevitable presence of unphysical spin states. The good performance of our methods is first demonstrated for the nearest-neighbor pyrochlore Heisenberg model where the finite temperature behavior of the specific heat and uniform susceptibility show excellent agreement within PMFRG and density-matrix purification. Including an antiferromagnetic second neighbor coupling we find that the non-magnetic ground-state phase of the nearest neighbor model extents up to $J_2/J_1 \sim 0.02 $ within DMRG, beyond which magnetic $\boldsymbol{k}=0$ long-range order sets in. Our PMFRG calculations find the phase transition in a similar regime $J_2/J_1\sim 0.035(8)$ which, together with the DMRG result, provides a strong argument for the existence of a small but finite non-magnetic ground-state phase in the spin-1 pyrochlore Heisenberg model. We also discuss the origin of discrepancies between different versions of the functional renormalization group concerning the location of this phase transition., Comment: 13 pages, 10 figures

Spiral spin liquids possess a subextensively degenerate ground-state manifold, represented by a continuum of energy minima in reciprocal space. Since a small change of the spiral state wavevector requires a global change of the spin configuration in real space, it is a priori unclear how such systems can fluctuate within the degenerate ground state manifold. Only recently it was proposed that momentum vortices are responsible for the liquidity of the spiral phase and that these systems are closely related to an emergent rank-2 U(1) gauge theory [H. Yan and J. Reuther, Phys. Rev. Research 4, 023175 (2022)]. As a consequence of this gauge structure, four-fold pinch-point singularities were found in a generalized spin correlator. In this article, we use classical Monte Carlo and molecular dynamics calculations to embed the previously studied spiral spin liquid into a broader phase diagram of the square lattice XY model. We find a multitude of unusual phases and phase transitions surrounding the spiral spin liquid such as an effective four-state Potts transition into a colinear double-striped phase resulting from the spontaneous breaking of two coupled $\mathbb{Z}_2$ symmetries. Since this phase is stabilized by entropic effects selecting the momenta away from the spiral manifold, it undergoes a re-entrance phenomenon at low temperatures into a nematic spiral phase. We also observe a region of parameters where the phase transition into the spiral spin liquid does not break any symmetries and where the critical exponents do not match those of standard universality classes. We study the importance of momentum vortices in driving this phase transition and discuss the possibility of a Kosterlitz-Thouless transition of momentum vortices. Finally, we explore the regime where the rank-2 U(1) gauge theory is valid by investigating the four-fold pinch point singularities across the phase diagram, Comment: 19 pages, 22 figures

As a broader set of applications from simulations to data analysis and machine learning require more parallel computational capability, the demand for interactive and urgent high performance computing (HPC) continues to increase. This paper overviews the progress made so far and elucidates the challenges and opportunities for greater integration of interactive and urgent HPC policies, techniques, and technologies into HPC ecosystems., Comment: 10 pages, submitted to IEEE CiSE journal

We implement the temperature flow scheme first proposed by Honerkamp and Salmhofer in Phys.~Rev.~B 64, 184516 (2001) into the pseudo-Majorana functional renormalization group method for quantum spin systems. Since the renormalization group parameter in this approach is a physical quantity -- the temperature $T$ -- the numerical efficiency increases significantly compared to more conventional renormalization group parameters, especially when computing finite temperature phase diagrams. We first apply this method to determine the finite temperature phase diagram of the $J_1$-$J_2$ Heisenberg model on the simple cubic lattice where our findings support claims of a vanishingly small nonmagnetic phase around the high frustration point $J_2=0.25J_1$. Perhaps most importantly, we find the temperature flow scheme to be advantageous in detecting finite temperature phase transitions as, by construction, a phase transition is never encountered at an artificial, unphysical cutoff parameter. Finally, we apply the temperature flow scheme to the dipolar XXZ model on the square lattice where we find a rich phase diagram with a large non-magnetic regime down to the lowest accessible temperatures. Wherever a comparison with error-controlled (quantum) Monte Carlo methods is applicable, we find excellent quantitative agreement with less than $5\%$ deviation from the numerically exact results., Comment: 12 pages, 7 figures

The pseudo-fermion functional renormalization group is generalized to treat spin Hamiltonians with finite magnetic fields, enabling its application to arbitrary spin lattice models with linear and bilinear terms in the spin operators. We discuss in detail an efficient numerical implementation of this approach making use of the system's symmetries. Particularly, we demonstrate that the inclusion of small symmetry breaking magnetic seed fields regularizes divergences of the susceptibility at magnetic phase transitions. This allows the investigation of spin models within magnetically ordered phases at $T=0$ in the physical limit of vanishing renormalization group parameter $\Lambda$. We explore the capabilities and limitations of this method extension for Heisenberg models on the square, honeycomb and triangular lattices. While the zero-field magnetizations of these systems are systematically overestimated, the types of magnetic orders are correctly captured, even if the local orientations of the seed field are chosen differently than the spin orientations of the realized magnetic order. Furthermore, the magnetization curve of the square lattice Heisenberg antiferromagnet shows good agreement with error controlled methods. In the future, the inclusion of magnetic fields in the pseudo-fermion functional renormalization group, which is also possible in three dimensional spin systems, will enable a variety of additional interesting applications such as the investigation of magnetization plateaus.

Gauge theories are powerful tools in theoretical physics, allowing complex phenomena to be reduced to simple principles, and are used in both high-energy and condensed matter physics. In the latter context, gauge theories are becoming increasingly popular for capturing the intricate spin correlations in spin liquids, exotic states of matter in which the dynamics of quantum spins never ceases, even at absolute zero temperature. We consider a spin system on a three-dimensional pyrochlore lattice where emergent gauge fields not only describe the spin liquid behaviour at zero temperature but crucially determine the system's temperature evolution, with distinct gauge fields giving rise to different spin liquid phases in separate temperature regimes. Focusing first on classical spins, in an intermediate temperature regime, the system shows an unusual coexistence of emergent vector and matrix gauge fields where the former is known from classical spin ice systems while the latter has been associated with fractonic quasiparticles, a peculiar type of excitation with restricted mobility. Upon cooling, the system transitions into a low-temperature phase where an entropic selection mechanism depopulates the degrees of freedom associated with the matrix gauge field, rendering the system spin ice like. We further provide numerical evidence that in the corresponding quantum model, a spin liquid with coexisting vector and matrix gauge fields has a finite window of stability in the parameter space of spin interactions down to zero temperature. Finally, we discuss the relevance of our findings for non-Kramers pyrochlore materials., Comment: 13 pages, 5 figures

This paper is an update of the survey of AI accelerators and processors from past four years, which is now called the Lincoln AI Computing Survey - LAICS (pronounced "lace"). As in past years, this paper collects and summarizes the current commercial accelerators that have been publicly announced with peak performance and peak power consumption numbers. The performance and power values are plotted on a scatter graph, and a number of dimensions and observations from the trends on this plot are again discussed and analyzed. Market segments are highlighted on the scatter plot, and zoomed plots of each segment are also included. Finally, a brief description of each of the new accelerators that have been added in the survey this year is included., Comment: 7 pages, 6 figures, 2023 IEEE High Performance Extreme Computing (HPEC) conference, September 2023

$\require{mhchem}$The square-kagome lattice Heisenberg antiferromagnet is a highly frustrated Hamiltonian whose material realizations have been scarce. We theoretically investigate the recently synthesized $\ce{Na6Cu7BiO4(PO4)4Cl3}$ where a Cu$^{2+}$ spin-$1/2$ square-kagome lattice (with six site unit cell) is decorated by a seventh magnetic site alternatingly above and below the layers. The material does not show any sign of long-range magnetic order down to 50 mK despite a Curie-Weiss temperature of $-212$ K indicating a quantum paramagnetic phase. Our DFT energy mapping elicits a purely antiferromagnetic Hamiltonian that features longer range exchange interactions beyond the pure square-kagome model and, importantly, we find the seventh site to be strongly coupled to the plane. We combine two variational Monte Carlo approaches, pseudo-fermion/Majorana functional renormalization group and Schwinger-Boson mean field calculations to show that the complex Hamiltonian of $\ce{Na6Cu7BiO4(PO4)4Cl3}$ still features a nonmagnetic ground state. We explain how the seventh Cu$^{2+}$ site actually aids the stabilization of the disordered state. We predict static and dynamic spin structure factors to guide future neutron scattering experiments., Comment: 5 pages, 4 figures. Supplement: 7 pages, 11 figures v2: minor revisions in text and figure labels

Expanding the scientific tools available to protect computer networks can be aided by a deeper understanding of the underlying statistical distributions of network traffic and their potential geometric interpretations. Analyses of large scale network observations provide a unique window into studying those underlying statistics. Newly developed GraphBLAS hypersparse matrices and D4M associative array technologies enable the efficient anonymized analysis of network traffic on the scale of trillions of events. This work analyzes over 100,000,000,000 anonymized packets from the largest Internet telescope (CAIDA) and over 10,000,000 anonymized sources from the largest commercial honeyfarm (GreyNoise). Neither CAIDA nor GreyNoise actively emit Internet traffic and provide distinct observations of unsolicited Internet traffic (primarily botnets and scanners). Analysis of these observations confirms the previously observed Cauchy-like distributions describing temporal correlations between Internet sources. The Gull lighthouse problem is a well-known geometric characterization of the standard Cauchy distribution and motivates a potential geometric interpretation for Internet observations. This work generalizes the Gull lighthouse problem to accommodate larger classes of coastlines, deriving a closed-form solution for the resulting probability distributions, stating and examining the inverse problem of identifying an appropriate coastline given a continuous probability distribution, identifying a geometric heuristic for solving this problem computationally, and applying that heuristic to examine the temporal geometry of different subsets of network observations. Application of this method to the CAIDA and GreyNoise data reveals a several orders of magnitude difference between known benign and other traffic which can lead to potentially novel ways to protect networks., Comment: 9 pages, 7 figures, IEEE HPEC 2023 (accepted)

pPython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in Python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (PythonMPI) in pure Python. pPython follows a SPMD (single program multiple data) model of computation. pPython runs on a single-node (e.g., a laptop) running Windows, Linux, or MacOS operating systems or on any combination of heterogeneous systems that support Python, including on a cluster through a Slurm scheduler interface so that pPython can be executed in a massively parallel computing environment. It is interesting to see what performance pPython can achieve compared to the traditional socket-based MPI communication because of its unique file-based messaging implementation. In this paper, we present the point-to-point and collective communication performances of pPython and compare them with those obtained by using mpi4py with OpenMPI. For large messages, pPython demonstrates comparable performance as compared to mpi4py., Comment: arXiv admin note: substantial text overlap with arXiv:2208.14908

Matrix/array analysis of networks can provide significant insight into their behavior and aid in their operation and protection. Prior work has demonstrated the analytic, performance, and compression capabilities of GraphBLAS (graphblas.org) hypersparse matrices and D4M (d4m.mit.edu) associative arrays (a mathematical superset of matrices). Obtaining the benefits of these capabilities requires integrating them into operational systems, which comes with its own unique challenges. This paper describes two examples of real-time operational implementations. First, is an operational GraphBLAS implementation that constructs anonymized hypersparse matrices on a high-bandwidth network tap. Second, is an operational D4M implementation that analyzes daily cloud gateway logs. The architectures of these implementations are presented. Detailed measurements of the resources and the performance are collected and analyzed. The implementations are capable of meeting their operational requirements using modest computational resources (a couple of processing cores). GraphBLAS is well-suited for low-level analysis of high-bandwidth connections with relatively structured network data. D4M is well-suited for higher-level analysis of more unstructured data. This work demonstrates that these technologies can be implemented in operational settings., Comment: Accepted to IEEE HPEC, 8 pages, 8 figures, 1 table, 69 references. arXiv admin note: text overlap with arXiv:2203.13934. text overlap with arXiv:2309.01806

Defending community-owned cyber space requires community-based efforts. Large-scale network observations that uphold the highest regard for privacy are key to protecting our shared cyberspace. Deployment of the necessary network sensors requires careful sensor placement, focusing, and calibration with significant volumes of network observations. This paper demonstrates novel focusing and calibration procedures on a multi-billion packet dataset using high-performance GraphBLAS anonymized hypersparse matrices. The run-time performance on a real-world data set confirms previously observed real-time processing rates for high-bandwidth links while achieving significant data compression. The output of the analysis demonstrates the effectiveness of these procedures at focusing the traffic matrix and revealing the underlying stable heavy-tail statistical distributions that are necessary for anomaly detection. A simple model of the corresponding probability of detection ($p_{\rm d}$) and probability of false alarm ($p_{\rm fa}$) for these distributions highlights the criticality of network sensor focusing and calibration. Once a sensor is properly focused and calibrated it is then in a position to carry out two of the central tenets of good cybersecurity: (1) continuous observation of the network and (2) minimizing unbrokered network connections., Comment: Accepted to IEEE HPEC, 9 pages, 12 figures, 1 table, 63 references, 2 appendices

Quantum spin liquids (QSLs) have become a key area of research in magnetism due to their remarkable properties, such as long-range entanglement, fractional excitations, pinch-point singularities, and topologically protected phenomena. In recent years, the search for QSLs has expanded into the three-dimensional world, where promising features have been found in materials that form pyrochlore and hyper-kagome lattices, despite the suppression of quantum fluctuations due to high dimensionality. One such material is the $S = 1$ K$_2$Ni$_2$(SO$_4$)$_3$ compound, which belongs to the langbeinite family consisting of two interconnected trillium lattices. Although magnetically ordered, K$_2$Ni$_2$(SO$_4$)$_3$ has been found to exhibit a highly dynamical and correlated state which can be driven into a pure quantum spin liquid under magnetic fields of only $B \simeq 4$~T. In this article, we combine inelastic neutron scattering measurements with pseudo-fermion functional renormalization group (PFFRG) and classical Monte Carlo (cMC) calculations to study the magnetic properties of K$_2$Ni$_2$(SO$_4$)$_3$, revealing a high level of agreement between the experiment and theory. We further reveal the origin of the dynamical state in K$_2$Ni$_2$(SO$_4$)$_3$ by studying a larger set of exchange parameters, uncovering an `island of liquidity' around a focal point given by a magnetic network composed of tetrahedra on a trillium lattice.

For decades, frustrated quantum magnets have been a seed for scientific progress and innovation in condensed matter. As much as the numerical tools for low-dimensional quantum magnetism have thrived and improved in recent years due to breakthroughs inspired by quantum information and quantum computation, higher-dimensional quantum magnetism can be considered as the final frontier, where strong quantum entanglement, multiple ordering channels, and manifold ways of paramagnetism culminate. At the same time, efforts in crystal synthesis have induced a significant increase in the number of tangible frustrated magnets which are generically three-dimensional in nature, creating an urgent need for quantitative theoretical modeling. We review the pseudo-fermion (PF) and pseudo-Majorana (PM) functional renormalization group (FRG) and their specific ability to address higher-dimensional frustrated quantum magnetism. First developed more than a decade ago, the PFFRG interprets a Heisenberg model Hamiltonian in terms of Abrikosov pseudofermions, which is then treated in a diagrammatic resummation scheme formulated as a renormalization group flow of $m$-particle pseudofermion vertices. The article reviews the state of the art of PFFRG and PMFRG and discusses their application to exemplary domains of frustrated magnetism, but most importantly, it makes the algorithmic and implementation details of these methods accessible to everyone. By thus lowering the entry barrier to their application, we hope that this review will contribute towards establishing PFFRG and PMFRG as the numerical methods for addressing frustrated quantum magnetism in higher spatial dimensions., Comment: Review article for Reports on Progress in Physics. Comments are welcome! (43+7) pages, (16+4) figures, 3 tables

We investigate classical Heisenberg models on the distorted windmill lattice and discuss their applicability to the spin-$1/2$ spin liquid candidate PbCuTe$_2$O$_6$. We first consider a general Heisenberg model on this lattice with antiferromagnetic interactions $J_n$ ($n=1,2,3,4$) up to fourth neighbors. Setting $J_1=J_2$ (as approximately realized in PbCuTe$_2$O$_6$) we map out the classical ground state phase diagram in the remaining parameter space and identify a competition between $J_3$ and $J_4$ that opens up interesting magnetic scenarios. Particularly, these couplings tune the ground states from coplanar commensurate or non-coplanar incommensurate magnetically ordered states to highly degenerate ground state manifolds with subextensive or extensive degeneracies. In the latter case, we uncover an unusual classical spin liquid defined on a lattice of corner sharing octahedra. We then focus on the particular set of interaction parameters $J_n$ that has previously been proposed for PbCuTe$_2$O$_6$ and investigate the system's incommensurate magnetic ground state order and finite temperature multistage ordering mechanism. We perform extensive finite temperature simulations of the system's dynamical spin structure factor and compare it with published neutron scattering data for PbCuTe$_2$O$_6$ at low temperatures. Our results demonstrate that thermal fluctuations in the classical model can largely explain the signal distribution in the measured spin structure factor but we also identify distinct differences. Our investigations make use of a variety of different analytical and numerical approaches for classical spin systems, such as Luttinger-Tisza, classical Monte Carlo, iterative minimization, and molecular dynamics simulations., Comment: 17 pages, 9 figures

In this paper, we present a fast, lightweight odometry method that uses the Doppler velocity measurements from a Frequency-Modulated Continuous-Wave (FMCW) lidar without data association. FMCW lidar is a recently emerging technology that enables per-return relative radial velocity measurements via the Doppler effect. Since the Doppler measurement model is linear with respect to the 6-degrees-of-freedom (DOF) vehicle velocity, we can formulate a linear continuous-time estimation problem for the velocity and numerically integrate for the 6-DOF pose estimate afterward. The caveat is that angular velocity is not observable with a single FMCW lidar. We address this limitation by also incorporating the angular velocity measurements from a gyroscope. This results in an extremely efficient odometry method that processes lidar frames at an average wall-clock time of 5.64ms on a single thread, well below the 10Hz operating rate of the lidar we tested. We show experimental results on real-world driving sequences and compare against state-of-the-art Iterative Closest Point (ICP)-based odometry methods, presenting a compelling trade-off between accuracy and computation. We also present an algebraic observability study, where we demonstrate in theory that the Doppler measurements from multiple FMCW lidars are capable of observing all 6 degrees of freedom (translational and angular velocity)., Comment: Accepted and presented at IROS 2023

We present a classification of bilinear Majorana representations for spin-$S$ operators, based on the real irreducible matrix representations of SU(2). We identify two types of such representations: While the first type can be straightforwardly mapped onto standard complex fermionic representations of spin-$S$ operators, the second type realizes spin amplitudes $S=s(s+1)/4$ with $s\in\mathbb{N}$ and can be considered particularly efficient in representing spins via fermions. We show that for $s=1$ and $s=2$ this second type reproduces known spin-$1/2$ and spin-$3/2$ Majorana representations and we prove that these are the only bilinear Majorana representations that do not introduce any unphysical spin sectors. While for $s>2$, additional unphysical spin spaces are unavoidable they are less numerous than for more standard complex fermionic representations and carry comparatively small spin amplitudes. We apply our Majorana representations to exactly solvable small spin clusters and confirm that their low energy properties remain unaffected by unphysical spin sectors, making our representations useful for auxiliary-particle based methods.

Fracton phases are a particularly exotic type of quantum spin liquids where the elementary quasiparticles are intrinsically immobile. These phases may be described by unconventional gauge theories known as tensor or multipolar gauge theories, characteristic for so-called type-I or type-II fracton phases, respectively. Both variants have been associated with distinctive singular patterns in the spin structure factor, such as multifold pinch points for type-I and quadratic pinch points for type-II fracton phases. Here, we assess the impact of quantum fluctuations on these patterns by numerically investigating the spin $S=1/2$ quantum version of a classical spin model on the octahedral lattice featuring exact realizations of multifold and quadratic pinch points, as well as an unusual pinch line singularity. Based on large scale pseudo fermion and pseudo Majorana functional renormalization group calculations, we take the intactness of these spectroscopic signatures as a measure for the stability of the corresponding fracton phases. We find that in all three cases, quantum fluctuations significantly modify the shape of pinch points or lines by smearing them out and shifting signal away from the singularities in contrast to effects of pure thermal fluctuations. This indicates possible fragility of these phases and allows us to identify characteristic fingerprints of their remnants., Comment: 5 pages, 4 figures (Supplementary: 4 pages, 6 figures)

We investigate the ground state and critical temperature phase diagrams of the classical and quantum $S=1/2$ pyrochlore lattice with nearest-neighbor Heisenberg and Dzyaloshinskii-Moriya interactions (DMI). We consider ferromagnetic and antiferromagnetic Heisenberg exchange as well as direct and indirect DMI. Classically, three ground states are found: all-in/all-out, ferromagnetic and a locally ordered $XY$ phase, known as $\Gamma_5$, which displays an accidental classical U(1) degeneracy. Quantum zero-point energy fluctuations are found to lift the classical ground state degeneracy and select the $\psi_3$ state in most parts of the $\Gamma_5$ regime. Likewise, thermal fluctuations treated classically, select the $\psi_3$ state at $T=0^+$. In contrast, classical Monte Carlo finds that the system orders at $T_c$ in the $\psi_2$ state of $\Gamma_5$ for antiferromagnetic Heisenberg exchange and indirect DMI with a transition from $\psi_2$ to $\psi_3$ at a temperature $T_{\Gamma_5}

We examine the computational energy requirements of different systems driven by the geometrical scaling law, and increasing use of Artificial Intelligence or Machine Learning (AI-ML) over the last decade. With more scientific and technology applications based on data-driven discovery, machine learning methods, especially deep neural networks, have become widely used. In order to enable such applications, both hardware accelerators and advanced AI-ML methods have led to the introduction of new architectures, system designs, algorithms, and software. Our analysis of energy trends indicates three important observations: 1) Energy efficiency due to geometrical scaling is slowing down; 2) The energy efficiency at the bit-level does not translate into efficiency at the instruction-level, or at the system-level for a variety of systems, especially for large-scale AI-ML accelerators or supercomputers; 3) At the application level, general-purpose AI-ML methods can be computationally energy intensive, off-setting the gains in energy from geometrical scaling and special purpose accelerators. Further, our analysis provides specific pointers for integrating energy efficiency with performance analysis for enabling high-performance and sustainable computing in the future., Comment: 8 pages, 9 figures, Submitted to Proceedings of IEEE Conference on High Performance Extreme Computing (HPEC) 2022

This paper updates the survey of AI accelerators and processors from past three years. This paper collects and summarizes the current commercial accelerators that have been publicly announced with peak performance and power consumption numbers. The performance and power values are plotted on a scatter graph, and a number of dimensions and observations from the trends on this plot are again discussed and analyzed. Two new trends plots based on accelerator release dates are included in this year's paper, along with the additional trends of some neuromorphic, photonic, and memristor-based inference accelerators., Comment: 10 pages, 4 figures, 2022 IEEE High Performance Extreme Computing (HPEC) Conference. arXiv admin note: substantial text overlap with arXiv:2009.00993, arXiv:2109.08957

Internet analysis is a major challenge due to the volume and rate of network traffic. In lieu of analyzing traffic as raw packets, network analysts often rely on compressed network flows (netflows) that contain the start time, stop time, source, destination, and number of packets in each direction. However, many traffic analyses benefit from temporal aggregation of multiple simultaneous netflows, which can be computationally challenging. To alleviate this concern, a novel netflow compression and resampling method has been developed leveraging GraphBLAS hyperspace traffic matrices that preserve anonymization while enabling subrange analysis. Standard multitemporal spatial analyses are then performed on each subrange to generate detailed statistical aggregates of the source packets, source fan-out, unique links, destination fan-in, and destination packets of each subrange which can then be used for background modeling and anomaly detection. A simple file format based on GraphBLAS sparse matrices is developed for storing these statistical aggregates. This method is scale tested on the MIT SuperCloud using a 50 trillion packet netflow corpus from several hundred sites collected over several months. The resulting compression achieved is significant (<0.1 bit per packet) enabling extremely large netflow analyses to be stored and transported. The single node parallel performance is analyzed in terms of both processors and threads showing that a single node can perform hundreds of simultaneous analyses at over a million packets/sec (roughly equivalent to a 10 Gigabit link)., Comment: arXiv admin note: text overlap with arXiv:2203.13934, arXiv:2108.06653, arXiv:2008.00307

Python has become a standard scientific computing language with fast-growing support of machine learning and data analysis modules, as well as an increasing usage of big data. The Dynamic Distributed Dimensional Data Model (D4M) offers a highly composable, unified data model with strong performance built to handle big data fast and efficiently. In this work we present an implementation of D4M in Python. $D4M.py$ implements all foundational functionality of D4M and includes Accumulo and SQL database support via Graphulo. We describe the mathematical background and motivation, an explanation of the approaches made for its fundamental functions and building blocks, and performance results which compare $D4M.py$'s performance to D4M-MATLAB and D4M.jl., Comment: 8 pages, 7 figures, accepted to HPEC 2022

pPython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in Python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (PythonMPI) in pure Python. The core data structure in pPython is a distributed numerical array whose distribution onto multiple processors is specified with a map construct. Communication operations between distributed arrays are abstracted away from the user and pPython transparently supports redistribution between any block-cyclic-overlapped distributions in up to four dimensions. pPython follows a SPMD (single program multiple data) model of computation. pPython runs on any combination of heterogeneous systems that support Python, including Windows, Linux, and MacOS operating systems. In addition to running transparently on single-node (e.g., a laptop), pPython provides a scheduler interface, so that pPython can be executed in a massively parallel computing environment. The initial implementation uses the Slurm scheduler. Performance of pPython on the HPC Challenge benchmark suite demonstrates both ease of programming and scalability., Comment: arXiv admin note: substantial text overlap with arXiv:astro-ph/0606464

We investigate the ground-state properties of the nearest-neighbor $S=1$ pyrochlore Heisenberg antiferromagnet using two complementary numerical methods, density-matrix renormalization group (DMRG) and pseudofermion functional renormalization group (PFFRG). Within DMRG, we are able to reliably study clusters with up to 48 spins by keeping 20 000 SU(2) states. The investigated 32-site and 48-site clusters both show indications of a robust $C_3$ rotation symmetry breaking of the ground-state spin correlations and the 48-site cluster additionally features inversion symmetry breaking. Our PFFRG analysis of various symmetry-breaking perturbations corroborates the findings of either $C_3$ or a combined $C_3$/inversion symmetry breaking. Moreover, in both methods the symmetry-breaking tendencies appear to be more pronounced than in the $S=1/2$ system., Comment: 8 pages, 6 figures

Pinch point singularities, associated with flat band magnetic excitations, are tell-tale signatures of Coulomb spin liquids. While their properties in the presence of quantum fluctuations have been widely studied, the fate of the complementary non-analytic features -- shaped as half-moons and stars -- arising from adjacent shallow dispersive bands has remained unexplored. Here, we address this question for the spin $S=1/2$ Heisenberg antiferromagnet on the kagome lattice with second and third neighbor couplings, which allows one to tune the classical ground state from flat bands to being governed by shallow dispersive bands for intermediate coupling strengths. Employing the complementary strengths of variational Monte Carlo, pseudo-fermion functional renormalization group, and density-matrix renormalization group, we establish the quantum phase diagram. The U(1) Dirac spin liquid ground state of the nearest-neighbor antiferromagnet remains remarkably robust till intermediate coupling strengths when it transitions into a pinwheel valence bond crystal displaying signatures of half-moons in its structure factor. Our work thus identifies a microscopic setting that realizes one of the proximate orders of the Dirac spin liquid identified in a recent work [Song, Wang, Vishwanath, He, Nat. Commun. 10, 4254 (2019)]. For larger couplings, we obtain a collinear magnetically ordered ground state characterized by star-like patterns., Comment: 7 pages, 5 figures. Supplemental Material appended

High-Performance Computing (HPC) centers and cloud providers support an increasingly diverse set of applications on heterogenous hardware. As Artificial Intelligence (AI) and Machine Learning (ML) workloads have become an increasingly larger share of the compute workloads, new approaches to optimized resource usage, allocation, and deployment of new AI frameworks are needed. By identifying compute workloads and their utilization characteristics, HPC systems may be able to better match available resources with the application demand. By leveraging datacenter instrumentation, it may be possible to develop AI-based approaches that can identify workloads and provide feedback to researchers and datacenter operators for improving operational efficiency. To enable this research, we released the MIT Supercloud Dataset, which provides detailed monitoring logs from the MIT Supercloud cluster. This dataset includes CPU and GPU usage by jobs, memory usage, and file system logs. In this paper, we present a workload classification challenge based on this dataset. We introduce a labelled dataset that can be used to develop new approaches to workload classification and present initial results based on existing approaches. The goal of this challenge is to foster algorithmic innovations in the analysis of compute workloads that can achieve higher accuracy than existing methods. Data and code will be made publicly available via the Datacenter Challenge website : https://dcc.mit.edu., Comment: Accepted at IPDPS ADOPT'22