Search

Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models. However, optimal techniques for upcycling at scale remain unclear. In this work, we conduct an extensive study of upcycling methods and hyperparameters for billion-parameter scale language models. We propose a novel "virtual group" initialization scheme and weight scaling approach to enable upcycling into fine-grained MoE architectures. Through ablations, we find that upcycling outperforms continued dense model training. In addition, we show that softmax-then-topK expert routing improves over topK-then-softmax approach and higher granularity MoEs can help improve accuracy. Finally, we upcycled Nemotron-4 15B on 1T tokens and compared it to a continuously trained version of the same model on the same 1T tokens: the continuous trained model achieved 65.3% MMLU, whereas the upcycled model achieved 67.6%. Our results offer insights and best practices to effectively leverage upcycling for building MoE language models.

The introduction of superconductivity to the Dirac surface states of a topological insulator leads to a topological superconductor, which may support topological quantum computing through Majorana zero modes. The development of a scalable material platform is key to the realization of topological quantum computing. Here we report on the growth and properties of high-quality (Bi,Sb)2Te3/graphene/gallium heterostructures. Our synthetic approach enables atomically sharp layers at both hetero-interfaces, which in turn promotes proximity-induced superconductivity that originates in the gallium film. A lithography-free, van der Waals tunnel junction is developed to perform transport tunneling spectroscopy. We find a robust, proximity-induced superconducting gap formed in the Dirac surface states in 5-10 quintuple-layer (Bi,Sb)2Te3/graphene/gallium heterostructures. The presence of a single Abrikosov vortex, where the Majorana zero modes are expected to reside, manifests in discrete conductance changes. The present material platform opens up opportunities for understanding and harnessing the application potential of topological superconductivity., Comment: 19 pages, 4 figures, with a Supplemental Information. Comments are welcome

A quantum anomalous Hall (QAH) insulator is a topological state of matter, in which the interior is insulating but electrical current flows along the edges of the sample, in either clockwise (right-handed) or counter-clockwise (left-handed) direction dictated by the spontaneous magnetization orientation. Such chiral edge current (CEC) eliminates any backscattering, giving rise to quantized Hall resistance and zero longitudinal resistance. In this work, we fabricate mesoscopic QAH sandwich (i.e. magnetic topological insulator (TI)/TI/magnetic TI) Hall bar devices and succeed in switching the CEC chirality in QAH insulators through spin-orbit torque (SOT) by applying a current pulse and suitably controlled gate voltage. The well-quantized QAH states with opposite CEC chiralities are demonstrated through four- and three-terminal measurements before and after SOT switching. Our theoretical calculations show that the SOT that enables the magnetization switching can be generated by both bulk and surface carriers in QAH insulators, in good agreement with experimental observations. Current pulse-induced switching of the CEC chirality in QAH insulators will not only advance our knowledge in the interplay between magnetism and topological states but also expedite easy and instantaneous manipulation of the QAH state in proof-of-concept energy-efficient electronic and spintronic devices as well as quantum information applications., Comment: 24 pages, 5 figures, comments are welcome

Two essential ingredients for the quantum anomalous Hall (QAH) effect, i.e. topological and magnetic orders, can be combined by doping magnetic ions into a topological insulator (TI) film. Through this approach, the QAH effect has been realized in chromium (Cr)- and/or vanadium (V)-doped TI (Bi,Sb)2Te3 thin films. In this work, we synthesize both V- and Cr-doped Bi2Te3 thin films with controlled dopant concentration using molecular beam epitaxy (MBE). By performing magneto-transport measurements, we find that both systems show an unusual but yet similar ferromagnetic response with respect to magnetic dopant concentration, specifically the Curie temperature does not increase monotonically but shows a local maximum at a critical dopant concentration. Our angle-resolved photoemission spectroscopy (ARPES) measurements show that the Cr/V doping introduces hole carriers into Bi2Te3, which consequently move the chemical potential toward the charge neutral point. In addition, the Cr/V doping also reduces the spin-orbit coupling of Bi2Te3 which drives it from a nontrivial TI to a trivial semiconductor. The unusual ferromagnetic response observed in Cr/V-doped Bi2Te3 thin films is attributed to the dopant-concentration-induced magnetic exchange interaction, which displays the evolution from the van Vleck-type ferromagnetism in a nontrivial magnetic TI to the Ruderman-Kittel-Kasuya-Yosida (RKKY)-type ferromagnetism in a trivial diluted magnetic semiconductor. Our work provides insights into the ferromagnetic properties of magnetically doped TI thin films and facilitates the pursuit of high-temperature QAH effect., Comment: 17 pages, 4 figures. Comments are welcome

Conventional optical tweezers are generated by the intensity gradient of highly focused laser beams, but the requirement of strong intensity gradient limits the tunability of optical traps. Here we show a new type of optical tweezers with tunable potential wells by manipulating the phase gradient of light. Using a new method to calculate holograms, we can design desirable phase profiles and intensity distributions of optical patterns. Optical force arising from the phase gradient creates tunable potential wells for versatile optical manipulation, such as trapping nanoparticles in peanut-shaped optical spots, and positioning and shifting nanoparticles in optical gears as demonstrated in our experiments. The phase-gradient-based optical tweezers have several merits including flexible design and easy control, which open a new way for optical trapping and manipulation., Comment: 20 pages, 12 figures

Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information, such as stochastic gradients, among different nodes. Recently, gradient sparsification techniques have been proposed to reduce communications cost and thus alleviate the network overhead. However, most of gradient sparsification techniques consider only synchronous parallelism and cannot be applied in asynchronous scenarios, such as asynchronous distributed training for federated learning at mobile devices. In this paper, we present a dual-way gradient sparsification approach (DGS) that is suitable for asynchronous distributed training. We let workers download model difference, instead of the global model, from the server, and the model difference information is also sparsified so that the information exchanged overhead is reduced by sparsifying the dual-way communication between the server and workers. To preserve accuracy under dual-way sparsification, we design a sparsification aware momentum (SAMomentum) to turn sparsification into adaptive batch size between each parameter. We conduct experiments at a cluster of 32 workers, and the results show that, with the same compression ratio but much lower communication cost, our approach can achieve better scalability and generalization ability., Comment: 8 pages

Holographic optical tweezers can be applied to manipulate microscopic particles in arbitrary optical patterns, which classical optical tweezers cannot do. This ability relies on accurate computer-generated holography (CGH), yet most CGH techniques can only shape the intensity profiles while the phase distributions are random. Here, we introduce a new method for fast generation of holograms that allows for simultaneously shaping both the intensity and phase distributions of light. The method uses a discrete inverse Fourier transform formula to directly calculate a hologram in one step, in which a random phase factor is introduced into the formula to enable simultaneous control of intensity and phase. Various optical patterns can be created, as demonstrated by the experimentally measured intensity and phase profiles projected from the holograms. The simultaneous shaping of intensity and phase of light provides new opportunities for optical trapping and manipulation, such as optical transportation of metal nanoparticles in ring traps with linear and nonlinear phase distributions.