Search

Large Language Models (LLMs) have dramatically advanced AI applications, yet their deployment remains challenging due to their immense inference costs. Recent studies ameliorate the computational costs of LLMs by increasing their activation sparsity but suffer from significant performance degradation on downstream tasks. In this work, we introduce a new framework for sparsifying the activations of base LLMs and reducing inference costs, dubbed Contextually Aware Thresholding for Sparsity (CATS). CATS is relatively simple, easy to implement, and highly effective. At the heart of our framework is a new non-linear activation function. We demonstrate that CATS can be applied to various base models, including Mistral-7B and Llama2-7B, and outperforms existing sparsification techniques in downstream task performance. More precisely, CATS-based models often achieve downstream task performance within 1-2% of their base models without any fine-tuning and even at activation sparsity levels of 50%. Furthermore, CATS-based models converge faster and display better task performance than competing techniques when fine-tuning is applied. Finally, we develop a custom GPU kernel for efficient implementation of CATS that translates the activation of sparsity of CATS to real wall-clock time speedups. Our custom kernel implementation of CATS results in a ~15% improvement in wall-clock inference latency of token generation on both Llama-7B and Mistral-7B.

Maximum Inner Product Search (MIPS) is a ubiquitous task in machine learning applications such as recommendation systems. Given a query vector and $n$ atom vectors in $d$-dimensional space, the goal of MIPS is to find the atom that has the highest inner product with the query vector. Existing MIPS algorithms scale at least as $O(\sqrt{d})$, which becomes computationally prohibitive in high-dimensional settings. In this work, we present BanditMIPS, a novel randomized MIPS algorithm whose complexity is independent of $d$. BanditMIPS estimates the inner product for each atom by subsampling coordinates and adaptively evaluates more coordinates for more promising atoms. The specific adaptive sampling strategy is motivated by multi-armed bandits. We provide theoretical guarantees that BanditMIPS returns the correct answer with high probability, while improving the complexity in $d$ from $O(\sqrt{d})$ to $O(1)$. We also perform experiments on four synthetic and real-world datasets and demonstrate that BanditMIPS outperforms prior state-of-the-art algorithms. For example, in the Movie Lens dataset ($n$=4,000, $d$=6,000), BanditMIPS is 20$\times$ faster than the next best algorithm while returning the same answer. BanditMIPS requires no preprocessing of the data and includes a hyperparameter that practitioners may use to trade off accuracy and runtime. We also propose a variant of our algorithm, named BanditMIPS-$\alpha$, which achieves further speedups by employing non-uniform sampling across coordinates. Finally, we demonstrate how known preprocessing techniques can be used to further accelerate BanditMIPS, and discuss applications to Matching Pursuit and Fourier analysis., Comment: 24 pages

Random forests are some of the most widely used machine learning models today, especially in domains that necessitate interpretability. We present an algorithm that accelerates the training of random forests and other popular tree-based learning methods. At the core of our algorithm is a novel node-splitting subroutine, dubbed MABSplit, used to efficiently find split points when constructing decision trees. Our algorithm borrows techniques from the multi-armed bandit literature to judiciously determine how to allocate samples and computational power across candidate split points. We provide theoretical guarantees that MABSplit improves the sample complexity of each node split from linear to logarithmic in the number of data points. In some settings, MABSplit leads to 100x faster training (an 99% reduction in training time) without any decrease in generalization performance. We demonstrate similar speedups when MABSplit is used across a variety of forest-based variants, such as Extremely Random Forests and Random Patches. We also show our algorithm can be used in both classification and regression tasks. Finally, we show that MABSplit outperforms existing methods in generalization performance and feature importance calculations under a fixed computational budget. All of our experimental results are reproducible via a one-line script at https://github.com/ThrunGroup/FastForest., Comment: Published at NeurIPS 2022, 30 pages

We investigate photoproduction of the scalar mesona off the nucleon, using the effective Lagrangians and the Regge approach. We first study $f_0(980)$ photoproduction, replacing the Feynman propagator with the Regge ones for the $\rho$-meson exchange in the $t$-channel. The model parameters are fixed by reproducing the experimental data on the differential cross section. We then apply the same method to $f_0(500)$ or $\sigma$ photoproduction with the same parameters for the Regge propagator. Since the threshold energy of the $f_0(500)$ production is rather low, $N^*$ resonances, which can decay into the nucleon and two pions in the isoscalar and scalar state, can come into play in the $s$-channel. To examine the effects of the $N^*$ resonances, we set up two models to look into the respective contribution of the $N^*$ resonances. We find that in particular $N^*(1440)$ and $N^*(1680)$ play important roles in $f_0(500)$ photoproduction. We discuss the physical implications of the present results., Comment: 15 pages, 8 figures. Final version accepted for publication in PTEP

In the present talk, we report a recent investigation on photoproduction of the $\gamma N \to f_0(500)N$ within a framework of the effective Lagrangian. We include the nucleon resonances with pin $1/2$ in the $s$ channel. The coupling constants have been etermined by assuming that the decay process $N^* \to (\pi\pi)_{I=0,J=0}N$ can be regarded as $N^* \to f_0(500) N$. We discuss the numerical results for the total cross sections and possible extension of the present work., Comment: 4 pages, 5 figures. A contribution to The 10th International Workshop on the Physics of Excited Nucleons (NSTAR2015) held in Osaka University, May 25(Mon)-28(Thu), 2015

We propose a neural connectomics strategy called Fluorescent In-Situ Sequencing of Barcoded Individual Neuronal Connections (FISSEQ-BOINC), leveraging fluorescent in situ nucleic acid sequencing in fixed tissue (FISSEQ). FISSEQ-BOINC exhibits different properties from BOINC, which relies on bulk nucleic acid sequencing. FISSEQ-BOINC could become a scalable approach for mapping whole-mammalian-brain connectomes with rich molecular annotations.

An organic polybutadiene (PB) rubberbased aerogel insulation material was developed that will provide superior thermal insulation and inherent radiation protection, exhibiting the flexibility, resiliency, toughness, and durability typical of the parent polymer, yet with the low density and superior insulation properties associated with the aerogels. The rubbery behaviors of the PB rubber-based aerogels are able to overcome the weak and brittle nature of conventional inorganic and organic aerogel insulation materials. Additionally, with higher content of hydrogen in their structure, the PB rubber aerogels will also provide inherently better radiation protection than those of inorganic and carbon aerogels. Since PB rubber aerogels also exhibit good hydrophobicity due to their hydrocarbon molecular structure, they will provide better performance reliability and durability as well as simpler, more economic, and environmentally friendly production over the conventional silica or other inorganic-based aerogels, which require chemical treatment to make them hydrophobic. Inorganic aerogels such as silica aerogels demonstrate many unusual and useful properties. There are several strategies to overcoming the drawbacks associated with the weakness and brittleness of silica aerogels. Development of the flexible fiber-reinforced silica aerogel composite blanket has proven one promising approach, providing a conveniently fielded form factor that is relatively robust toward handling in industrial environments compared to silica aerogel monoliths. However, the flexible silica aerogel composites still have a brittle, dusty character that may be undesirable, or even intolerable, in certain applications. Although the cross-linked organic aerogels such as resorcinol-formaldehyde (RF), polyisocyanurate, and cellulose aerogels show very high impact strength, they are also very brittle with little elongation (i.e., less rubbery). Also, silica and carbon aerogels are less efficient radiation shielding materials due to their lower content of hydrogen element. The present invention relates to maleinized polybutadiene (or polybutadiene adducted with maleic anhydride)- based aerogel monoliths and composites, and the methods for preparation. Hereafter, they are collectively referred to as polybutadiene aerogels. Specifically, the polybutadiene aerogels of the present invention are prepared by mixing a maleinized polybutadiene resin, a hardener containing a maleic anhydride reactive group, and a catalyst in a suitable solvent, and maintaining the mixture in a quiescent state for a sufficient period of time to form a polymeric gel. After aging at elevated temperatures for a period of time to provide uniformly stronger wet gels, the micro porous maleinized polybutadiene- based aerogel is then obtained by removing interstitial solvent by supercritical drying. The mesoporous maleinized polybutadiene-based aerogels contain an open-pore structure, which provides inherently hydrophobic, flexible, nearly unbreakable, less dusty aerogels with excellent thermal and physical properties. The materials can be used as thermal and acoustic insulation, radiation shielding, and vibration-damping materials. The organic PB-based rubber aerogels are very flexible, no-dust, and hydrophobic organics that demonstrated the following ranges of typical properties: densities of 0.08 to 0.255 grams per cubic centimeters, shrinkage factor (raerogel/rtarget) = 1.2 to 2.84, and thermal conductivity values of 20.0 to 35.0 mW/m-K.

aerogel insulation material was developed that will provide superior thermal insulation and inherent radiation protection for government and commercial applications. The rubbery polyureabased aerogel exhibits little dustiness, good flexibility and toughness, and durability typical of the parent polyurea polymer, yet with the low density and superior insulation properties associated with aerogels. The thermal conductivity values of polyurea-based aerogels at lower temperature under vacuum pressures are very low and better than that of silica aerogels. Flexible, rubbery polyurea-based aerogels are able to overcome the weak and brittle nature of conventional inorganic and organic aerogels, including polyisocyanurate aerogels, which are generally prepared with the one similar component to polyurethane rubber aerogels. Additionally, with higher content of hydrogen in their structures, the polyurea rubber-based aerogels will also provide inherently better radiation protection than those of inorganic and carbon aerogels. The aerogel materials also demonstrate good hydrophobicity due to their hydrocarbon molecular structure. There are several strategies to overcoming the drawbacks associated with the weakness and brittleness of silica aerogels. Development of the flexible fiber-reinforced silica aerogel composite blanket has proven to be one promising approach, providing a conveniently fielded form factor that is relatively robust in industrial environments compared to silica aerogel monoliths. However, the flexible, silica aerogel composites still have a brittle, dusty character that may be undesirable, or even intolerable, in certain application environments. Although the cross - linked organic aerogels, such as resorcinol- formaldehyde (RF), polyisocyanurate, and cellulose aerogels, show very high impact strength, they are also very brittle with little elongation (i.e., less rubbery). Also, silica and carbon aerogels are less efficient radiation shielding materials due to their lower content of hydrogen element. The invention involves mixing at least one isocyanate resin in solvent along with a specific amount of at least one polyamine hardener. The hardener is selected from a group of polyoxyalkyleneamines, amine-based polyols, or a mixture thereof. Mixing is performed in the presence of a catalyst and reinforcing inorganic and/or organic materials, and the system is then subjected to gelation, aging, and supercritical drying. The aerogels will offer exceptional flexibility, excellent thermal and physical properties, and good hydrophobicity. The rubbery polyurea-based aerogels are very flexible with no dust and hydrophobic organics that demonstrated the following ranges of typical properties: densities of 0.08 to 0.293 g/cu cm, shrinkage factor (raerogel/rtarget) = 1.6 to 2.84, and thermal conductivity values of 15.2 to 20.3 mW/m K.

The present invention relates to cross-linked polyolefin aerogels in simple and fiber-reinforced composite form. Of particular interest are polybutadiene aerogels. Especially aerogels derived from polybutadienes functionalized with anhydrides, amines, hydroxyls, thiols, epoxies, isocyanates or combinations thereof.