Search

Fine-tuning pre-trained large language models (LLMs) with limited hardware presents challenges due to GPU memory constraints. Various distributed fine-tuning methods have been proposed to alleviate memory constraints on GPU. However, determining the most effective method for achieving rapid fine-tuning while preventing GPU out-of-memory issues in a given environment remains unclear. To address this challenge, we introduce LLMem, a solution that estimates the GPU memory consumption when applying distributed fine-tuning methods across multiple GPUs and identifies the optimal method. We conduct GPU memory usage estimation prior to fine-tuning, leveraging the fundamental structure of transformer-based decoder models and the memory usage distribution of each method. Experimental results show that LLMem accurately estimates peak GPU memory usage on a single GPU, with error rates of up to 1.6%. Additionally, it shows an average error rate of 3.0% when applying distributed fine-tuning methods to LLMs with more than a billion parameters on multi-GPU setups., Comment: 9 pages, 9 figures, accepted to IJCAI 2024

Eye-tracking research has proven valuable in understanding numerous cognitive functions. Recently, Frey et al. provided an exciting deep learning method for learning eye movements from fMRI data. However, it needed to co-register fMRI into standard space to obtain eyeballs masks, and thus required additional templates and was time consuming. To resolve this issue, in this paper, we propose a framework named MRGazer for predicting eye gaze points from fMRI in individual space. The MRGazer consisted of eyeballs extraction module and a residual network-based eye gaze prediction. Compared to the previous method, the proposed framework skips the fMRI co-registration step, simplifies the processing protocol and achieves end-to-end eye gaze regression. The proposed method achieved superior performance in a variety of eye movement tasks than the co-registration-based method, and delivered objective results within a shorter time (~ 0.02 Seconds for each volume) than prior method (~0.3 Seconds for each volume).

Composite materials with 3D architectures are desirable in a variety of applications for the capability of tailoring their properties to meet multiple functional requirements. By the arrangement of materials' internal components, structure design is of great significance in tuning the properties of the composites. However, most of the composite structures are proposed by empirical designs following existing patterns. Hindered by the complexity of 3D structures, it is hard to extract customized structures with multiple desired properties from large design space. Here we report a multi-objective driven Wasserstein generative adversarial network (MDWGAN) to implement inverse designs of 3D composite structures according to given geometrical, structural and mechanical requirements. Our framework consists a GAN based network which generates 3D composite structures possessing with similar geometrical and structural features to the target dataset. Besides, multiple objectives are introduced to our framework for the control of mechanical property and isotropy of the composites. Real time calculation of the properties in training iterations is achieved by an accurate surrogate model. We constructed a small and concise dataset to illustrate our framework. With multiple objectives combined by their weight, and the 3D-GAN act as a soft constraint, our framework is proved to be capable of tuning the properties of the generated composites in multiple aspects, while keeping the selected features of different kinds of structures. The feasibility on small dataset and potential scalability on objectives of other properties make our work a novel, effective approach to provide fast, experience free composite structure designs for various functional materials.

Many composite materials, both natural and fabricated, process a Voronoi like architecture or microstructure. Furthermore, the stochasticity and connectivity of Voronoi tessellation endow the composite materials constructed by this kind of structures a wide range of desired properties including high and tuneable stiffness, high strength and good manufacturability. Thus, Voronoi-based fibre network structures are regarded as promising designs of composite reinforcements, as well as powerful tools in composite mechanics simulations. In this paper, the elastic properties of composites reinforced by a 3D Voronoi fibre network are systemically investigated based on the precise control of the Voronoi cell regularity. The regularity of the reinforcement fibre networks, according to our definition, is found positively related to the Young's moduli of the composites. Interestingly, 20% percent of defects in total reinforcement fibres only causes a less than 6.5% of Young's moduli drop in Voronoi fibre network reinforced composite. The Voronoi fibre network reinforced composites also shows higher Young's moduli than those of conventional composites with discrete reinforcements. The possibility of simulating aerogels by Voronoi fibre network structures are also presented., Comment: 29 pages, 15 figures

Nanoscale polymeric thin films are widely used in diverse applications such as energy devices, flexible electronics and biosensors, where a satisfactory mechanical performance is of vital importance to realize their full functionality. It has been evidenced that the elastic properties of polymer films are often strongly affected by their thickness; however, the underlying mechanism of this phenomenon, especially a thorough understanding at the microscopic level, has yet to be achieved. Here we established a coarse-grained molecular dynamics (CGMD) based computational framework, combining with experimental verifications, aiming to reveal the conformational origin of the stiffening behavior of crosslinked polymeric thin films. By imposing systematic controls over the polymer network structures, we found that the bi-axial modulus changes are essentially consequent of the alteration of polymer conformations. A unified theory was then proposed, to quantitatively clarify the correlation between the elastic properties of the system and the distributional variations of the chain end-to-end distances, with predicting a significant hardening effect on top of the conventional entropic elasticity with largely uncoiled chains. Adopting processing protocols inspired by the modeling, our experiments showed that PDMS films at approximately the same thickness may exhibit a two order of magnitude difference in their moduli. The good agreement between experiments and simulations illustrated our findings as an effective guideline for tailoring the elastic properties of polymer films at nanoscale.

The vapor-liquid-solid (VLS) method is considered a plausible technique for synthesizing germanium (Ge) nanostructures (e.g. nanowires), which have a broad range of applications due to their unique electronic properties and intrinsic compatibility with silicon. However, crystallization failures and material defects are still frequently observed in VLS processes, with insufficient understanding of their underlying mechanisms due to instrumental limitations for high-resolution in-situ characterizations. Employing an accurate interatomic potential well fitted to the gold-germanium (Au-Ge) phase diagram, we performed molecular dynamics simulations for a systematic investigation on the Au-catalyzed growth process of Ge crystals. From the simulations, relationships were established between the overall Ge growth rate and several main synthesis conditions, including substrate crystallographic orientation, temperature and Ge supersaturation in liquid. The dynamical behaviors of Ge atoms near the liquid-solid growing interface were captured, from which the atom surface stability and exchange rate were estimated for quantifying the atomistic details of the growth. These interface properties were further linked to the surface morphologies, to explain the observed orientation-dependent growing modes. This study sheds new lights into the understanding of the VLS growth mechanisms of Ge crystals, and provides scientific guidelines for designing innovative synthesis methods for similar nanomaterials.

Decoding brain cognitive states from neuroimaging signals is an important topic in neuroscience. In recent years, deep neural networks (DNNs) have been recruited for multiple brain state decoding and achieved good performance. However, the open question of how to interpret the DNN black box remains unanswered. Capitalizing on advances in machine learning, we integrated attention modules into brain decoders to facilitate an in-depth interpretation of DNN channels. A 4D convolution operation was also included to extract temporo-spatial interaction within the fMRI signal. The experiments showed that the proposed model obtains a very high accuracy (97.4%) and outperforms previous researches on the 7 different task benchmarks from the Human Connectome Project (HCP) dataset. The visualization analysis further illustrated the hierarchical emergence of task-specific masks with depth. Finally, the model was retrained to regress individual traits within the HCP and to classify viewing images from the BOLD5000 dataset, respectively. Transfer learning also achieves good performance. A further visualization analysis shows that, after transfer learning, low-level attention masks remained similar to the source domain, whereas high-level attention masks changed adaptively. In conclusion, the proposed 4D model with attention module performed well and facilitated interpretation of DNNs, which is helpful for subsequent research.

We introduce Tuna, a static analysis approach to optimizing deep neural network programs. The optimization of tensor operations such as convolutions and matrix multiplications is the key to improving the performance of deep neural networks. Many deep learning model optimization mechanisms today use dynamic analysis, which relies on experimental execution on a target device to build a data-driven cost model of the program. The reliance on dynamic profiling not only requires access to target hardware at compilation time but also incurs significant cost in machine resources. We introduce an approach that profiles the program by constructing features based on the target hardware characteristics in order. We use static analysis of the relative performance of tensor operations to optimize the deep learning program. Experiments show that our approach can achieve up to 11x performance compared to dynamic profiling based methods with the same compilation time.

Polymer electrolytes are promising candidates for the next generation lithium-ion battery technology. Large scale screening of polymer electrolytes is hindered by the significant cost of molecular dynamics (MD) simulation in amorphous systems: the amorphous structure of polymers requires multiple, repeated sampling to reduce noise and the slow relaxation requires long simulation time for convergence. Here, we accelerate the screening with a multi-task graph neural network that learns from a large amount of noisy, unconverged, short MD data and a small number of converged, long MD data. We achieve accurate predictions of 4 different converged properties and screen a space of 6247 polymers that is orders of magnitude larger than previous computational studies. Further, we extract several design principles for polymer electrolytes and provide an open dataset for the community. Our approach could be applicable to a broad class of material discovery problems that involve the simulation of complex, amorphous materials., Comment: 29 pages, 6 figures + supplementary information

We propose a novel non-randomized anytime orienteering algorithm for finding k-optimal goals that maximize reward on a specialized graph with budget constraints. This specialized graph represents a real-world scenario which is analogous to an orienteering problem of finding k-most optimal goal states.