Search

The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance., Comment: ACL 2024

Recommender systems are often susceptible to well-crafted fake profiles, leading to biased recommendations. Among existing defense methods, data-processing-based methods inevitably exclude normal samples, while model-based methods struggle to enjoy both generalization and robustness. To this end, we suggest integrating data processing and the robust model to propose a general framework, Triple Cooperative Defense (TCD), which employs three cooperative models that mutually enhance data and thereby improve recommendation robustness. Furthermore, Considering that existing attacks struggle to balance bi-level optimization and efficiency, we revisit poisoning attacks in recommender systems and introduce an efficient attack strategy, Co-training Attack (Co-Attack), which cooperatively optimizes the attack optimization and model training, considering the bi-level setting while maintaining attack efficiency. Moreover, we reveal a potential reason for the insufficient threat of existing attacks is their default assumption of optimizing attacks in undefended scenarios. This overly optimistic setting limits the potential of attacks. Consequently, we put forth a Game-based Co-training Attack (GCoAttack), which frames the proposed CoAttack and TCD as a game-theoretic process, thoroughly exploring CoAttack's attack potential in the cooperative training of attack and defense. Extensive experiments on three real datasets demonstrate TCD's superiority in enhancing model robustness. Additionally, we verify that the two proposed attack strategies significantly outperform existing attacks, with game-based GCoAttack posing a greater poisoning threat than CoAttack., Comment: arXiv admin note: text overlap with arXiv:2210.13762

In single-cell RNA sequencing (scRNA-seq) analysis, a key challenge is inferring hidden cellular dynamics from static cell snapshots. Various computational methods have been developed to address this, focusing on perspectives like pseudotime trajectories, RNA velocities, and estimating the differentiation potential of cells, often referred to as "cell potency." This review summarizes 14 methods for defining cell potency from scRNA-seq data, categorizing them into average-based, entropy-based, and correlation-based methods based on how they summarize gene expression levels into a potency measure. We highlight the key similarities and differences within and between these categories, offering a high-level intuition for each method. Additionally, we use unified mathematical notations to detail each method's methodology and summarize their usage complexities, including parameters, required inputs, and differences between published descriptions and software implementations. We conclude that cell potency estimation remains an open question without a consensus on the optimal approach, emphasizing the need for benchmark datasets and studies. This review aims to provide a foundation for future benchmark studies, while also addressing the broader challenge of comparing methods that infer cellular dynamics from scRNA-seq data through various perspectives, including pseudotime trajectories, RNA velocities, and cell potency.

Video memorability is a measure of how likely a particular video is to be remembered by a viewer when that viewer has no emotional connection with the video content. It is an important characteristic as videos that are more memorable are more likely to be shared, viewed, and discussed. This paper presents results of a series of experiments where we improved the memorability of a video by selectively cropping frames based on image saliency. We present results of a basic fixed cropping as well as the results from dynamic cropping where both the size of the crop and the position of the crop within the frame, move as the video is played and saliency is tracked. Our results indicate that especially for videos of low initial memorability, the memorability score can be improved., Comment: 12 pages

In open quantum systems undergoing phase transitions, the intricate interplay between unitary and dissipative processes leaves many information-theoretic properties opaque. We are here interested in interparticle correlations within such systems, specifically examining quantum entanglement, quantum discord, and classical correlation within the steady state of a driven-dissipative collective spin model. This model is renowned for, counterintuitively, its transition from a high-purity to a low-purity state with decreasing dissipation. Our investigation, rooted in numerical analysis using PPT criteria, underscores that entanglement reaches its zenith precisely at the phase transition juncture. Intriguingly, even within the mesoscopic scale near the transition point, entanglement endures across both phases, despite the open nature of the model. Notably, when employing traditional pairwise entanglement measures tailored to this model, detecting entanglement in the low-purity phase has proven elusive. In stark contrast, quantum discord and its variations chart an alternate trajectory, ascending monotonically as the system progresses into the low-purity phase. Consequently, lowered dissipation amplifies quantum correlation, yet it engenders entanglement solely in proximity to the transition point.

Natural intelligences (NIs) thrive in a dynamic world - they learn quickly, sometimes with only a few samples. In contrast, artificial intelligences (AIs) typically learn with a prohibitive number of training samples and computational power. What design principle difference between NI and AI could contribute to such a discrepancy? Here, we investigate the role of weight polarity: development processes initialize NIs with advantageous polarity configurations; as NIs grow and learn, synapse magnitudes update, yet polarities are largely kept unchanged. We demonstrate with simulation and image classification tasks that if weight polarities are adequately set a priori, then networks learn with less time and data. We also explicitly illustrate situations in which a priori setting the weight polarities is disadvantageous for networks. Our work illustrates the value of weight polarities from the perspective of statistical and computational efficiency during learning., Comment: ICML camera-ready

Recommender systems are often susceptible to well-crafted fake profiles, leading to biased recommendations. The wide application of recommender systems makes studying the defense against attack necessary. Among existing defense methods, data-processing-based methods inevitably exclude normal samples, while model-based methods struggle to enjoy both generalization and robustness. Considering the above limitations, we suggest integrating data processing and robust model and propose a general framework, Triple Cooperative Defense (TCD), which cooperates to improve model robustness through the co-training of three models. Specifically, in each round of training, we sequentially use the high-confidence prediction ratings (consistent ratings) of any two models as auxiliary training data for the remaining model, and the three models cooperatively improve recommendation robustness. Notably, TCD adds pseudo label data instead of deleting abnormal data, which avoids the cleaning of normal data, and the cooperative training of the three models is also beneficial to model generalization. Through extensive experiments with five poisoning attacks on three real-world datasets, the results show that the robustness improvement of TCD significantly outperforms baselines. It is worth mentioning that TCD is also beneficial for model generalizations., Comment: 15 pages, 4 figures, 5 tables

Why do brains have inhibitory connections? Why do deep networks have negative weights? We propose an answer from the perspective of representation capacity. We believe representing functions is the primary role of both (i) the brain in natural intelligence, and (ii) deep networks in artificial intelligence. Our answer to why there are inhibitory/negative weights is: to learn more functions. We prove that, in the absence of negative weights, neural networks with non-decreasing activation functions are not universal approximators. While this may be an intuitive result to some, to the best of our knowledge, there is no formal theory, in either machine learning or neuroscience, that demonstrates why negative weights are crucial in the context of representation capacity. Further, we provide insights on the geometric properties of the representation space that non-negative deep networks cannot represent. We expect these insights will yield a deeper understanding of more sophisticated inductive priors imposed on the distribution of weights that lead to more efficient biological and machine learning., Comment: ICCV2023 camera-ready

Public blockchains have spurred the growing popularity of decentralized transactions and smart contracts, but they exhibit limitations on the transaction throughput, storage, and computation. To avoid transaction gridlock, public blockchains impose large fees and per-block resource limits, making it difficult to accommodate the ever-growing transaction demand. Previous research endeavors to improve the scalability of blockchain through various technologies, such as side-chaining, sharding, secured off-chain computation, communication network optimizations, and efficient consensus protocols. However, these approaches have not attained a widespread adoption due to their inability in delivering a cloud-like performance, in terms of the scalability in transaction throughput, storage, and compute capacity. In this work, we determine that the major obstacle to public blockchain scalability is their underlying unstructured P2P networks. We further show that a centralized network can support the deployment of decentralized smart contracts. We propose a novel approach for achieving scalable decentralization: instead of trying to make blockchain scalable, we deliver decentralization to already scalable cloud by using an Ethereum smart contract. We introduce Blockumulus, a framework that can deploy decentralized cloud smart contract environments using a novel technique called overlay consensus. Through experiments, we demonstrate that Blockumulus is scalable in all three dimensions: computation, data storage, and transaction throughput. Besides eliminating the current code execution and storage restrictions, Blockumulus delivers a transaction latency between 2 and 5 seconds under normal load. Moreover, the stress test of our prototype reveals the ability to execute 20,000 simultaneous transactions under 26 seconds, which is on par with the average throughput of worldwide credit card transactions., Comment: 41st IEEE International Conference on Distributed Computing Systems (ICDCS 2021)

Rare-earth niobates and tantalates possess low thermal conductivities, which can be further reduced in high-entropy compositions. Here, a large number of 40 compositions are synthesized to investigate the origin of low thermal conductivity. Amongst, 29 possess single (nominally cubic) fluorite phases and most of them are new compositionally complex (medium- or high-entropy) compositions. One new finding is that doping 2 % of light element cations can further reduce thermal conductivity. This large data set enables the discovery of a negative correlation between the thermal conductivity and averaged radius ratio of the 3+/5+ cations. While this ratio is still below the threshold for forming long-range ordered weberite phases, this correlation suggests the reduced thermal conductivity is related to short-range weberite order, which is indeed revealed by diffuse scattering in X-ray and neutron diffraction. Specifically, neutron diffraction is used characterize five selected specimens. A better fit to a weberite structure is found at nanoscale (~1 nm). The characteristic length (domain size) is smaller but with stronger short-range ordering in more insulative materials. As it approaches the Ioffe-Regel limit, the phonon limit breaks down and "diffusons" give rise to the observed amorphous-like thermal conductivity. Disordered oxygen sublattices are also confirmed by neutron diffraction.

Improved thermomechanical properties have been reported for various high-entropy oxides containing typically five metal cations. This study further investigates a series of duodenary (11 metals + oxygen) high-entropy oxides by mixing different fractions of a five-cation fluorite-structured niobate and a seven-cation pyrochlore (both containing Yb) with matching lattice constants. Nine compositions of duodenary high-entropy oxides have been examined. All of them exhibit single high-entropy phases of either disordered fluorite or ordered pyrochlore structure. An order-disorder transition (ODT) is evident with changing composition, accompanied by a reduction in thermal conductivity (k). In comparison with the ODT criteria developed from ternary oxides, these duodenary oxides are more prone to disorder, but the ODT is still controlled by similar factors (but at different thresholds). Interestingly, there are abrupt increases in Young's modulus (E) at low mixing concentrations near both endmembers. The E/k ratios are increased, in comparison with both endmembers. This study suggests a new route to tailor high-entropy ceramics via controlling cation ordering vs. disordering.

This paper presents the development of instrumentation for the measurement of high-temperature thermal conductivity of bulk and coatings using a modulated photothermal radiometry (MPR) method, where a sample is heated by an intensity-modulated laser to probe into different layers of the sample. While MPR has been previously established, most of the previous studies only focus on the measurement at room temperature. The MPR has not been well studied for measurements of bulk and coating materials at high temperatures, which are increasingly important for a multitude of applications, such as materials used in the concentrating solar power (CSP) plants and the nuclear reactors. MPR is a non-contact technique that utilizes the intrinsic thermal emission from the specimens for thermometry, which is favorable for measurements at high temperatures in harsh environment. The authors designed and utilized a sample holder suitable for high temperature measurement up to 973 K with good temperature uniformity within the sample. The high-temperature MPR setup was validated by measuring bulk materials with known thermal conductivity. The setup and technique were then extended to the measurement of black solar-absorbing coatings of 10 to 50 {\mu}m thick on various substrates by modulating the frequency of the laser heating beam and the thermal penetration depth. The studies showed that thermal conductivities of typical solar-absorbing coatings are 0.4 ~ 0.8 W m-1 K-1, indicating a possibly large temperature drop within the coating under high solar irradiation flux, such as over 1000-sun for central solar towers in CSP plants.

Polymer-air multilayer (PAM) was developed to decrease the heat loss through window glass panes. A PAM consists of a few polymer films separated from each other by air gaps. Thanks to the excellent optical properties of the polymer films, the visual transmittance of PAM is higher than 70%, and the haze is less than 2%. PAM not only has mechanisms to reduce the conductive and convective heat transfer, but also can obstruct the radiative heat transfer. With a 4~6 mm thick PAM coating, the U-factor of a glass pane can be lowered from above 1 Btu/{{h}{ft}^2{{\deg}F}} to 0.5~0.6 Btu/{{h}{ft}^2{{\deg}F}} .PAM is resilient and robust, relevant to the window retrofitting applications., Comment: 21 pages, 7 figures

In lifelong learning, data are used to improve performance not only on the present task, but also on past and future (unencountered) tasks. While typical transfer learning algorithms can improve performance on future tasks, their performance on prior tasks degrades upon learning new tasks (called forgetting). Many recent approaches for continual or lifelong learning have attempted to maintain performance on old tasks given new tasks. But striving to avoid forgetting sets the goal unnecessarily low. The goal of lifelong learning should be to use data to improve performance on both future tasks (forward transfer) and past tasks (backward transfer). In this paper, we show that a simple approach -- representation ensembling -- demonstrates both forward and backward transfer in a variety of simulated and benchmark data scenarios, including tabular, vision (CIFAR-100, 5-dataset, Split Mini-Imagenet, and Food1k), and speech (spoken digit), in contrast to various reference algorithms, which typically failed to transfer either forward or backward, or both. Moreover, our proposed approach can flexibly operate with or without a computational budget.

Nanoporous structures including single nanopores and nanoporous membranes have been utilized as a platform to study fundamental liquid-vapor phase change heat transfer (PCHT) processes as well as a promising candidate for high flux heat dissipation. Both thin film boiling and evaporation through nanoporous structures have been demonstrated to achieve high heat flux, but they are usually considered two mutually exclusive regimes operated under vastly different conditions, and the factors dictating how close the PCHT process is to the kinetic limit are elusive. In this work, we utilized a unique transition between thin film boiling and evaporation through nanoporous membranes to clarify the factors determining the heat flux and heat transfer coefficient (HTC) with respect to the kinetic limit conditions. We unambiguously showed the controllable transition from boiling to evaporation, when the liquid receded into the nanopores and provided additional driving force from capillary pumping sustained in the nanoscale pores. We showed that this transition is universal and can be understood from a simple fluid transport model for all the four types of fluids we studied, which cover a wide span of surface tension (water, ethanol, isopropanol (IPA), FC-72). More importantly, PCHT conditions at the transition points between boiling and evaporation were close to those of the kinetic limit of all these fluids. However, further increase of the heat flux beyond the transition points led to decreasing HTC and deviation from the kinetic limit, which can be attributed to the increasing vapor resistance in the vapor space and inside the nanopores. This increasing vapor resistance was also confirmed by experiments on IPA with different vapor pressures.

High-entropy ceramics generally exhibit reduced thermal conductivity, but little is known about what controls this suppression and which descriptor can predict it. Herein, 18 medium- and high-entropy pyrochlores were synthesized to measure their thermal conductivity and Young's modulus. Up to 35% reductions in thermal conductivity were achieved with retained moduli, thereby attaining insulative yet stiff properties for potential thermal barrier coating applications. Notably, the measured thermal conductivity correlates well with a modified size disorder parameter. Thus, this modified size disorder parameter is suggested as a useful descriptor for designing thermally-insulative medium- and high-entropy ceramics (broadly defined as "compositionally-complex ceramics").