Search

The emergence and growing popularity of multimodal large language models (MLLMs) have significant potential to enhance various aspects of daily life, from improving communication to facilitating learning and problem-solving. Mobile phones, as essential daily companions, represent the most effective and accessible deployment platform for MLLMs, enabling seamless integration into everyday tasks. However, deploying MLLMs on mobile phones presents challenges due to limitations in memory size and computational capability, making it difficult to achieve smooth and real-time processing without extensive optimization. In this paper, we present BlueLM-V-3B, an algorithm and system co-design approach specifically tailored for the efficient deployment of MLLMs on mobile platforms. To be specific, we redesign the dynamic resolution scheme adopted by mainstream MLLMs and implement system optimization for hardware-aware deployment to optimize model inference on mobile phones. BlueLM-V-3B boasts the following key highlights: (1) Small Size: BlueLM-V-3B features a language model with 2.7B parameters and a vision encoder with 400M parameters. (2) Fast Speed: BlueLM-V-3B achieves a generation speed of 24.4 token/s on the MediaTek Dimensity 9300 processor with 4-bit LLM weight quantization. (3) Strong Performance: BlueLM-V-3B has attained the highest average score of 66.1 on the OpenCompass benchmark among models with $\leq$ 4B parameters and surpassed a series of models with much larger parameter sizes (e.g., MiniCPM-V-2.6, InternVL2-8B)., Comment: 21 pages

The Lyman-alpha Solar Telescope (LST) on board the Advanced Space-based Solar Observatory (ASO-S) is the first payload to image the full solar disk and the solar corona in both white-light (WL) and ultraviolet (UV) H I Lya, extending up to 2.5 solar radii (Rs). Since the launch of the ASO-S on 9 October 2022, LST has captured various significant solar activities including flares, prominences, coronal mass ejections (CMEs). LST covers different passbands of 121.6 nm, 360 nm and 700 nm. The Lya Solar Disk Imager (SDI) has a field of view (FOV) of 38.4 arcmin and a spatial resolution of around 9.5 arcsec, while the White-Light Solar Telescope (WST) has a FOV of 38.43 arcmin and a spatial resolution of around 3.0 arcsec. The FOV of the Lya Solar Corona Imager (SCI) reaches 81.1 arcmin and its spatial resolution is 4.3 arcsec. The stray-light level in the 700 nm waveband is about 7.8e-6 MSB (mean solar brightness) at 1.1 Rs and 7.6e-7 MSB at 2.5 Rs, and in the Lya waveband it is around 4.3e-3 MSB at 1.1 Rs and 4.1e-4 MSB at 2.5 Rs. This article will detail the results from on-orbit tests and calibrations., Comment: Solar Physics (ASO-S mission topical collection), accepted

AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents. Their capabilities of completing complex tasks by directly interacting with the graphical user interface (GUI) on mobile devices are trained and evaluated with the proposed dataset. AMEX comprises over 104K high-resolution screenshots from 110 popular mobile applications, which are annotated at multiple levels. Unlike existing mobile device-control datasets, e.g., MoTIF, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions, each averaging 13 steps with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we develop a baseline model SPHINX Agent and compare its performance across state-of-the-art agents trained on other datasets. To facilitate further research, we open-source our dataset, models, and relevant evaluation tools. The project is available at https://yuxiangchai.github.io/AMEX/

Recent developments in large-scale pre-trained text-to-image diffusion models have significantly improved the generation of high-fidelity images, particularly with the emergence of diffusion models based on transformer architecture (DiTs). Among these diffusion models, diffusion transformers have demonstrated superior image generation capabilities, boosting lower FID scores and higher scalability. However, deploying large-scale DiT models can be expensive due to their extensive parameter numbers. Although existing research has explored efficient deployment techniques for diffusion models such as model quantization, there is still little work concerning DiT-based models. To tackle this research gap, in this paper, we propose TerDiT, a quantization-aware training (QAT) and efficient deployment scheme for ternary diffusion models with transformers. We focus on the ternarization of DiT networks and scale model sizes from 600M to 4.2B. Our work contributes to the exploration of efficient deployment strategies for large-scale DiT models, demonstrating the feasibility of training extremely low-bit diffusion transformer models from scratch while maintaining competitive image generation capacities compared to full-precision models. Code will be available at https://github.com/Lucky-Lance/TerDiT., Comment: 18 pages, 13 figures

We present ImageBind-LLM, a multi-modality instruction tuning method of large language models (LLMs) via ImageBind. Existing works mainly focus on language and image instruction tuning, different from which, our ImageBind-LLM can respond to multi-modality conditions, including audio, 3D point clouds, video, and their embedding-space arithmetic by only image-text alignment training. During training, we adopt a learnable bind network to align the embedding space between LLaMA and ImageBind's image encoder. Then, the image features transformed by the bind network are added to word tokens of all layers in LLaMA, which progressively injects visual instructions via an attention-free and zero-initialized gating mechanism. Aided by the joint embedding of ImageBind, the simple image-text training enables our model to exhibit superior multi-modality instruction-following capabilities. During inference, the multi-modality inputs are fed into the corresponding ImageBind encoders, and processed by a proposed visual cache model for further cross-modal embedding enhancement. The training-free cache model retrieves from three million image features extracted by ImageBind, which effectively mitigates the training-inference modality discrepancy. Notably, with our approach, ImageBind-LLM can respond to instructions of diverse modalities and demonstrate significant language generation quality. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter., Comment: Code is available at https://github.com/OpenGVLab/LLaMA-Adapter

Moire patterns appear frequently when taking photos of digital screens, drastically degrading the image quality. Despite the advance of CNNs in image demoireing, existing networks are with heavy design, causing redundant computation burden for mobile devices. In this paper, we launch the first study on accelerating demoireing networks and propose a dynamic demoireing acceleration method (DDA) towards a real-time deployment on mobile devices. Our stimulus stems from a simple-yet-universal fact that moire patterns often unbalancedly distribute across an image. Consequently, excessive computation is wasted upon non-moire areas. Therefore, we reallocate computation costs in proportion to the complexity of image patches. In order to achieve this aim, we measure the complexity of an image patch by designing a novel moire prior that considers both colorfulness and frequency information of moire patterns. Then, we restore image patches with higher-complexity using larger networks and the ones with lower-complexity are assigned with smaller networks to relieve the computation burden. At last, we train all networks in a parameter-shared supernet paradigm to avoid additional parameter burden. Extensive experiments on several benchmarks demonstrate the efficacy of our proposed DDA. In addition, the acceleration evaluated on the VIVO X80 Pro smartphone equipped with a chip of Snapdragon 8 Gen 1 shows that our method can drastically reduce the inference time, leading to a real-time image demoireing on mobile devices. Source codes and models are released at https://github.com/zyxxmu/DDA, Comment: To appear in the eleventh International Conference on Learning Representations (ICLR 2023)

Processing maps are key to guiding the thermo-mechanical processing (TMP) of superalloys. However, traditional processing maps are incapable of delimiting failure, which is an essential factor to be concerned about during the TMP of superalloys. Employing isothermal hot compression experiments and finite element analysis (FEA), the present study examined the failure behaviors of a powder metallurgy (P/M) Ni-based superalloy and constructed processing maps with failure domains based on the predicted failure threshold. The micromechanical Gurson-Tvergaard-Needleman (GTN) damage model was employed in the FEA to model the cavity-driven intergranular fracture of the superalloy. Deformation temperature and strain rate were considered in the range of 1050 ~ 1150 C and 0.001 ~ 1 s-1, respectively. The FEA results reveal that the maximum tensile stress locates at the outer budging surfaces of the samples, which causes failure initiation and subsequent propagation into longitudinal cracks, being consistent with the experiments. It is further demonstrated that the failure is strain-controlled and the critical failure strain remains nearly insensitive to the range of strain rates considered while increasing with the increase of temperature in a third-order polynomial. Finally, an optimized processing window for hot deformation of the superalloy is formulated to warrant good hot workability while avoiding flow instability and failure. The present study offers direct insights into the failure behaviors of P/M Ni-based superalloys and details a modeling strategy to delineate optimized parametric spaces for the TMP of superalloys.

Spherical-cap-shaped interfacial nanobubbles (NBs) forming on hydrophobic surfaces in aqueous solutions have extensively been studied both from a fundamental point of view and due to their relevance for various practical applications. In this study, the nucleation mechanism of spontaneously generated NBs at solid-liquid interfaces of immersed nanostructured hydrophobic surfaces is studied. Depending on the size and density of the surface nanostructures, NBs with different size and density were reproducibly and deterministically obtained. A two-step process can explain the NB nucleation, based on the crevice model, i.e., entrapped air pockets in surface cavities which grow by diffusion. The results show direct evidence for the spontaneous formation of NBs on a surface at its immersion. Next, the influence of size and shape of the nanostructures on the nucleated NBs are revealed. In particular, on non-circular nanopits we obtain NBs with a non-circular footprint, demonstrating the strong pinning forces at the three-phase contact line., Comment: 17 pages, 7 figures

Radiation damage to the steel material of reactor pressure vessels is a major threat to the nuclear reactor safety. It is caused by the metal atom cascade collision, initialized when the atoms are struck by a high-energy neutron. The paper presents MISA-MD, a new implementation of molecular dynamics, to simulate such cascade collision with EAM potential. MISA-MD realizes (1) a hash-based data structure to efficiently store an atom and find its neighbors, and (2) several acceleration and optimization strategies based on SW26010 processor of Sunway Taihulight supercomputer, including an efficient potential table storage and interpolation method, a coloring method to avoid write conflicts, and double-buffer and data reuse strategies. The experimental results demonstrated that MISA-MD has good accuracy and scalability, and obtains a parallel efficiency of over 79% in an 655-billion-atom system. Compared with a state-of-the-art MD program LAMMPS, MISA-MD requires less memory usage and achieves better computational performance.

We study the free-surface deformation dynamics of an immersed glassy thin polymer film supported on a substrate, induced by an air nanobubble at the free surface.We combine analytical and numerical treatments of the glassy thin film equation, resulting from the lubrication approximation applied to the surface mobile layer of the glassy film, under the driving of an axisymmetric step function in the pressure term accounting for the nanobubble's Laplace pressure. Using the method of Green's functions, we derive a general solution for the film profile. We show that the lateral extent of the surface perturbation follows an asymptotic viscocapillary power-law behaviour in time, and that the film's central height decays logarithmically in time in this regime. This process eventually leads to film rupture and dewetting at finite time, for which we provide an analytical prediction exhibiting explicitly the dependencies in surface mobility, film thickness and bubble size, among others. Finally, using finite-element numerical integration, we discuss how non-linear effects induced by the curvature and film profile can affect the evolution.