Search

Bio-inspired multi-joint snake robots offer the advantages of terrain adaptability due to their limbless structure and high flexibility. However, a series of dozens of motor units in typical multiple-joint snake robots results in a heavy body structure and hundreds of watts of high power consumption. This paper presents a joint-repositionable, inner-wireless snake robot that enables multi-joint-like locomotion using a low-powered underactuated mechanism. The snake robot, consisting of a series of flexible passive links, can dynamically change its joint coupling configuration by repositioning motor-driven joint units along rack gears inside the robot. Additionally, a soft robot skin wirelessly powers the internal joint units, avoiding the risk of wire tangling and disconnection caused by the movable joint units. The combination of the joint-repositionable mechanism and the wireless-charging-enabled soft skin achieves a high degree of bending, along with a lightweight structure of 1.3 kg and energy-efficient wireless power transmission of 7.6 watts.

The observation of a resonance structure in the opening angle of the electron-positron pairs in the $^{7}$Li(p,\ee) $^{8}$Be reaction was claimed and interpreted as the production and subsequent decay of a hypothetical particle (X17). Similar excesses, consistent with this particle, were later observed in processes involving $^{4}$He and $^{12}$C nuclei with the same experimental technique. The MEG II apparatus at PSI, designed to search for the $\mu^+ \rightarrow \mathrm{e}^+ \gamma$ decay, can be exploited to investigate the existence of this particle and study its nature. Protons from a Cockroft-Walton accelerator, with an energy up to 1.1 MeV, were delivered on a dedicated Li-based target. The $\gamma$ and the e$^{+}$e$^{-}$ pair emerging from the $^8\mathrm{Be}^*$ transitions were studied with calorimeters and a spectrometer, featuring a broader angular acceptance than previous experiments. We present in this paper the analysis of a four-week data-taking in 2023 with a beam energy of 1080 keV, resulting in the excitation of two different resonances with Q-value \SI{17.6}{\mega\electronvolt} and \SI{18.1}{\mega\electronvolt}. No significant signal was found, and limits at \SI{90}{\percent} C.L. on the branching ratios (relative to the $\gamma$ emission) of the two resonances to X17 were set, $R_{17.6} < 1.8 \times 10^{-6} $ and $R_{18.1} < 1.2 \times 10^{-5} $., Comment: 11 pages, 9 figures, submitted to EPJC

Large Language Models (LLMs) have attracted significant attention due to their human-like language understanding and generation capabilities, as well as their applicability across various domains. These models, characterized by their massive scale and extensive training data, continue to push the boundaries of what is possible in natural language processing. The Llama 3 series, for instance, exemplifies this trend with its flagship model boasting 405 billion parameters trained on 15.6 trillion tokens. The immense computational demands associated with training such models have spurred ongoing research into optimizing the efficiency of the training process, particularly through the use of lower-precision formats. NVIDIA's H100 GPU, which introduces support for FP8 in addition to the more conventional FP16 and BF16 formats, has emerged as a focal point in this optimization effort. Preliminary studies suggest that FP8 could offer substantial reductions in training time without sacrificing model performance when compared to BF16, making it a promising candidate for large-scale model training. However, the broader implications of adopting FP8, particularly in terms of training stability and downstream task performance, have yet to be fully understood. In this study, we delve into the practical trade-offs involved in adopting FP8 over BF16 for training LLMs., Comment: 2 pages,extended abstract

In large language model (LLM) training, several parallelization strategies, including Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), as well as Sequence Parallelism (SP) and Context Parallelism (CP), are employed to distribute model parameters, activations, and optimizer states across devices. Identifying the optimal parallelization configuration for each environment while avoiding GPU memory overflow remains a challenging task. In this study, we provide precise formulas to estimate the memory consumed by parameters, gradients, optimizer states, and activations for 4D parallel training (DP, TP, PP, CP) in the Llama architecture. We conducted 454 experiments on A100 and H100 GPUs, incorporating often neglected factors such as temporary buffers and memory fragmentation into our analysis. Results indicate that when the estimated memory usage is below 80\% of the available GPU memory, the training never encounters out-of-memory errors. This simple yet effective formula allows us to identify parallelization configurations that could lead to memory overflow in advance, significantly reducing the configuration search space. Additionally, through a comprehensive exploration of optimal configurations in 4D parallelism, our analysis of the 454 experimental results provides empirical insights into optimal 4D parallelism configurations.

We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models. The code is available at https://github.com/team-approx-bayes/ivon-lora., Comment: Published at 38th Workshop on Fine-Tuning in Machine Learning (NeurIPS 2024). Code available at https://github.com/team-approx-bayes/ivon-lora. In version 2 we fixed a typo in the equation of prior in section 2

In fluids under temperature gradients, long-range correlations (LRCs) emerge generically, leading to enhanced density fluctuations. This phenomenon, characterized by the $\boldsymbol{q}^{-4}$ divergence in the static structure factor (where $\boldsymbol{q}$ is the wavenumber), has been extensively studied both theoretically and experimentally. However, they remain unexplored in Hamiltonian particle systems using molecular dynamics (MD) simulations. This Letter reports the first MD study to provide unambiguous observations of the LRCs. We demonstrate this by three distinct approaches: (1) measuring the static structure factor and directly observing the $\boldsymbol{q}^{-4}$ divergence characterizing the LRCs; (2) detecting the corresponding $\boldsymbol{q}^{-4}$ divergence in the dynamic structure factor; (3) establishing a quantitative agreement between MD results and predictions from fluctuating hydrodynamics, the phenomenological theory that predicts the LRCs. Our findings demonstrate that MD simulations offer a powerful complementary tool to theoretical and experimental investigations of LRCs., Comment: 7+5 pages, 4+3 figures

The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, it brings together researchers and expertise from the document analysis, privacy, and federated learning communities. Participants fine-tuned a pre-trained, state-of-the-art Document Visual Question Answering model provided by the organizers for this new domain, mimicking a typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold in track 1 and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition community. Ultimately, the competition analysis provides best practices and recommendations for successfully running privacy-focused federated learning challenges in the future., Comment: 27 pages, 6 figures

Local learning, which trains a network through layer-wise local targets and losses, has been studied as an alternative to backpropagation (BP) in neural computation. However, its algorithms often become more complex or require additional hyperparameters because of the locality, making it challenging to identify desirable settings in which the algorithm progresses in a stable manner. To provide theoretical and quantitative insights, we introduce the maximal update parameterization ($\mu$P) in the infinite-width limit for two representative designs of local targets: predictive coding (PC) and target propagation (TP). We verified that $\mu$P enables hyperparameter transfer across models of different widths. Furthermore, our analysis revealed unique and intriguing properties of $\mu$P that are not present in conventional BP. By analyzing deep linear networks, we found that PC's gradients interpolate between first-order and Gauss-Newton-like gradients, depending on the parameterization. We demonstrate that, in specific standard settings, PC in the infinite-width limit behaves more similarly to the first-order gradient. For TP, even with the standard scaling of the last layer, which differs from classical $\mu$P, its local loss optimization favors the feature learning regime over the kernel regime.

We propose the first learning scheme for functional differential equations (FDEs). FDEs play a fundamental role in physics, mathematics, and optimal control. However, the numerical analysis of FDEs has faced challenges due to its unrealistic computational costs and has been a long standing problem over decades. Thus, numerical approximations of FDEs have been developed, but they often oversimplify the solutions. To tackle these two issues, we propose a hybrid approach combining physics-informed neural networks (PINNs) with the \textit{cylindrical approximation}. The cylindrical approximation expands functions and functional derivatives with an orthonormal basis and transforms FDEs into high-dimensional PDEs. To validate the reliability of the cylindrical approximation for FDE applications, we prove the convergence theorems of approximated functional derivatives and solutions. Then, the derived high-dimensional PDEs are numerically solved with PINNs. Through the capabilities of PINNs, our approach can handle a broader class of functional derivatives more efficiently than conventional discretization-based methods, improving the scalability of the cylindrical approximation. As a proof of concept, we conduct experiments on two FDEs and demonstrate that our model can successfully achieve typical $L^1$ relative error orders of PINNs $\sim 10^{-3}$. Overall, our work provides a strong backbone for physicists, mathematicians, and machine learning experts to analyze previously challenging FDEs, thereby democratizing their numerical analysis, which has received limited attention. Code is available at \url{https://github.com/TaikiMiyagawa/FunctionalPINN}., Comment: Accepted at NeurIPS 2024. Both authors contributed equally. Some contents are omitted due to arXiv's storage limit. Please refer to the full paper at OpenReview (NeurIPS 2024) or https://github.com/TaikiMiyagawa/FunctionalPINN

This paper provides an overview of the techniques employed by Hibikino-Musashi@Home, which intends to participate in the domestic standard platform league. The team has developed a dataset generator for training a robot vision system and an open-source development environment running on a Human Support Robot simulator. The large language model powered task planner selects appropriate primitive skills to perform the task requested by users. The team aims to design a home service robot that can assist humans in their homes and continuously attends competitions to evaluate and improve the developed system.

We propose a new operator defined between two tensors, the broadcast product. The broadcast product calculates the Hadamard product after duplicating elements to align the shapes of the two tensors. Complex tensor operations in libraries like \texttt{numpy} can be succinctly represented as mathematical expressions using the broadcast product. Finally, we propose a novel tensor decomposition using the broadcast product, highlighting its potential applications in dimensionality reduction.

Throughout the history of computer vision, while research has explored the integration of images (visual) and point clouds (geometric), many advancements in image and 3D object recognition have tended to process these modalities separately. We aim to bridge this divide by integrating images and point clouds on a unified transformer model. This approach integrates the modality-specific properties of images and point clouds and achieves fundamental downstream tasks in image and 3D object recognition on a unified transformer model by learning visual-geometric representations. In this work, we introduce Formula-Supervised Visual-Geometric Pre-training (FSVGP), a novel synthetic pre-training method that automatically generates aligned synthetic images and point clouds from mathematical formulas. Through cross-modality supervision, we enable supervised pre-training between visual and geometric modalities. FSVGP also reduces reliance on real data collection, cross-modality alignment, and human annotation. Our experimental results show that FSVGP pre-trains more effectively than VisualAtom and PC-FractalDB across six tasks: image and 3D object classification, detection, and segmentation. These achievements demonstrate FSVGP's superior generalization in image and 3D object recognition and underscore the potential of synthetic pre-training in visual-geometric representation learning. Our project website is available at https://ryosuke-yamada.github.io/fdsl-fsvgp/., Comment: Accepted to ECCV2024

Signed graphs are equipped with both positive and negative edge weights, encoding pairwise correlations as well as anti-correlations in data. A balanced signed graph has no cycles of odd number of negative edges. Laplacian of a balanced signed graph has eigenvectors that map simply to ones in a similarity-transformed positive graph Laplacian, thus enabling reuse of well-studied spectral filters designed for positive graphs. We propose a fast method to learn a balanced signed graph Laplacian directly from data. Specifically, for each node $i$, to determine its polarity $\beta_i \in \{-1,1\}$ and edge weights $\{w_{i,j}\}_{j=1}^N$, we extend a sparse inverse covariance formulation based on linear programming (LP) called CLIME, by adding linear constraints to enforce ``consistent" signs of edge weights $\{w_{i,j}\}_{j=1}^N$ with the polarities of connected nodes -- i.e., positive/negative edges connect nodes of same/opposing polarities. For each LP, we adapt projections on convex set (POCS) to determine a suitable CLIME parameter $\rho > 0$ that guarantees LP feasibility. We solve the resulting LP via an off-the-shelf LP solver in $\mathcal{O}(N^{2.055})$. Experiments on synthetic and real-world datasets show that our balanced graph learning method outperforms competing methods and enables the use of spectral filters and graph convolutional networks (GCNs) designed for positive graphs on signed graphs., Comment: 5 pages, 1 figure. Submitted to ICASSP 2025

In this work, we investigate the understudied effect of the training data used for image super-resolution (SR). Most commonly, novel SR methods are developed and benchmarked on common training datasets such as DIV2K and DF2K. However, we investigate and rethink the training data from the perspectives of diversity and quality, {thereby addressing the question of ``How important is SR training for SR models?''}. To this end, we propose an automated image evaluation pipeline. With this, we stratify existing high-resolution image datasets and larger-scale image datasets such as ImageNet and PASS to compare their performances. We find that datasets with (i) low compression artifacts, (ii) high within-image diversity as judged by the number of different objects, and (iii) a large number of images from ImageNet or PASS all positively affect SR performance. We hope that the proposed simple-yet-effective dataset curation pipeline will inform the construction of SR datasets in the future and yield overall better models., Comment: Accepted to ECCV2024

Pre-training and transfer learning are an important building block of current computer vision systems. While pre-training is usually performed on large real-world image datasets, in this paper we ask whether this is truly necessary. To this end, we search for a minimal, purely synthetic pre-training dataset that allows us to achieve performance similar to the 1 million images of ImageNet-1k. We construct such a dataset from a single fractal with perturbations. With this, we contribute three main findings. (i) We show that pre-training is effective even with minimal synthetic images, with performance on par with large-scale pre-training datasets like ImageNet-1k for full fine-tuning. (ii) We investigate the single parameter with which we construct artificial categories for our dataset. We find that while the shape differences can be indistinguishable to humans, they are crucial for obtaining strong performances. (iii) Finally, we investigate the minimal requirements for successful pre-training. Surprisingly, we find that a substantial reduction of synthetic images from 1k to 1 can even lead to an increase in pre-training performance, a motivation to further investigate ''scaling backwards''. Finally, we extend our method from synthetic images to real images to see if a single real image can show similar pre-training effect through shape augmentation. We find that the use of grayscale images and affine transformations allows even real images to ''scale backwards''., Comment: Accepted to ECCV2024

Ordered states for a classical Heisenberg model on fcc lattice structure with ABC stacked kagome planes of magnetic ions are investigated by numerically solving the Landau-Lifshitz (LL) equation. Both the nearest-neighbor exchange interaction and the magnetic dipolar interactions are included in the model. The model with only the nearest-neighbor antiferromagnetic exchange interaction is known to show the layered 120 degree spin structure. On the other hand, the model with only the magnetic dipolar interactions is known to exhibit a continuous degeneracy expressed by six sublattice spin vectors, which is reduced by an order-by-disorder process with thermal fluctuations. In the present study, other ordered states appear for various values of relative strength of the two kinds of the interactions. A vortex spin structure on hexagonal lattice points in the kagomeplanes is a novel one. Another ordered state may be glassy state in which apparent translational symmetry cannot be seen. Layered 120 degree spin structures but not uniform in the direction perpendicular to the kagome planes with various period in the direction appear depending on the relative strength of the two kinds of the interactions.

Ozawa and Shioya proposed the limit formula for observable diameters of pyramids under weak convergence. However, we find a constructive counterexample to an inequality used in their proof. In this paper, we correct the inequality and verify the limit formula., Comment: 9 pages

The influence of space weathering on the observed spectra of C-complex asteroids remains uncertain. This has long hindered our understanding of their composition through telescope observations. Multi-band imaging of Ryugu by ONC-T on Hayabusa2 and that of Bennu by MapCam on OSIRIS-REx found opposite spectral trends of space weathering; Ryugu darkened/reddened while Bennu brightened/blued. How the spectra of Ryugu and Bennu evolved relative to each other would place a constraint for understanding their origins and evolutions. In this study, we compared the space weathering trends on Ryugu and Bennu by applying the results of cross calibration between ONC-T and MapCam. We show that the average Bennu surface is brighter by 18.0 $\pm$ 1.5% at 550 nm and bluer by 0.18 $\pm$ 0.03 $\mu$m$^{-1}$ (480-850 nm slope) than Ryugu. The spectral slopes of surface materials are more uniform on Bennu than on Ryugu at spatial scales $\gtrsim$1 m, but Bennu is more heterogeneous at $\lesssim$1 m. This suggests that lateral mixing due to resurfacing may have been more efficient on Bennu. The reflectance-spectral slope distributions of craters on Ryugu and Bennu appeared to follow two trend lines with an offset before cross calibration, but they converged to a single straight trend without a bend after cross calibration. We show that the spectra of the freshest craters on Ryugu and Bennu are indistinguishable within the uncertainty of cross calibration. These results suggest that Ryugu and Bennu initially had similar spectra before space weathering and that they evolved in completely opposite directions along the same trend line, subsequently evolving into asteroids with different disk-averaged spectra. These findings further suggest that space weathering likely expanded the spectral slope variation of C-complex asteroids, implying that they may have formed from materials with more uniform spectral slopes.

This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its activities, and technical reports on the LLMs developed by LLM-jp. For the latest activities, visit https://llm-jp.nii.ac.jp/en/.

Physics-informed Neural Networks (PINNs) is a method for numerical simulation that incorporates a loss function corresponding to the governing equations into a neural network. While PINNs have been explored for their utility in inverse analysis, their application in acoustic analysis remains limited. This study presents a method to identify loss parameters in acoustic tubes using PINNs. We categorized the loss parameters into two groups: one dependent on the tube's diameter and another constant, independent of it. The latter were set as the trainable parameters of the neural network. The problem of identifying the loss parameter was formulated as an optimization problem, with the physical properties being determined through this process. The neural network architecture employed was based on our previously proposed ResoNet, which is designed for analyzing acoustic resonance. The efficacy of the proposed method is assessed through both forward and inverse analysis, specifically through the identification of loss parameters. The findings demonstrate that it is feasible to accurately identify parameters that significantly impact the sound field under analysis. By merely altering the governing equations in the loss function, this method could be adapted to various sound fields, suggesting its potential for broad application., Comment: 10 pages, 7 figures, The following article has been submitted to Mechanical Engineering Journal. After it is published, it will be found at https://www.jstage.jst.go.jp/browse/mej

Iron-based superconductor Sr$_2$VFeAsO$_3$ is composed of alternate stacking of a superconducting FeAs layer and an insulating vanadium-oxide layer with a perovskite-type structure. Electronic orders stemming from the spin and orbital degrees of freedom of V $3d$ electrons can arise in the vanadium-oxide layer, but such orders have not been confirmed so far. Here, we systematically investigate the electronic state of Sr$_2$VFeAsO$_{3-\delta}$ with oxygen deficiency and demonstrate the phase diagram of Sr$_2$VFeAsO$_{3-\delta}$ as a function of the $c$-axis lattice parameter, which has turned out to be a suitable measure of the amount of oxygen deficiency. We found a magnetic and structural anomaly at $\sim 100$ K with a thermal hysteresis, which is manifested with the introduction of oxygen deficiency. The presence of orthorhombic distortion was revealed below the temperature at which the anomaly appears, suggestive of V orbital ordering involving the $d_{xz}$ and $d_{yz}$ orbitals. It seems that substantial fluctuations associated with the orthorhombic distortion significantly influence the electronic state of the FeAs layer. Our findings indicate that the vanadium-oxide layer plays a significant role in the electronic state of Sr$_2$VFeAsO$_{3-\delta}$., Comment: 10 pages, 9 figures

Solid-state nanopore gating inspired by biological ion channels is gaining increasing traction due to a large range of applications in biosensing and drug delivery. Integration of stimuli-responsive molecules such as poly(N-isopropylacrylamide) (PNIPAM) inside nanopores can enable temperature-dependent gating, which so far has only been demonstrated using external heaters. In this work, we combine plasmonic resonators inside the nanopore architecture with PNIPAM to enable optical gating of individual or multiple nanopores with micrometer resolution and a switching speed of few milliseconds by thermo-plasmonics. We achieve a temperature change of 40 kelvin per millisecond and demonstrate the efficacy of this method using nanopore ionic conductivity measurements that enables selective activation of individual nanopores in an array. Moreover, the selective gating of specific nanopores in an array can set distinct ionic conductance levels: low, medium, and high (i.e., 0, 1, and 2), which could be exploited for logical gating with optical signal control. Such selective optical gating in nanopore arrays marks a breakthrough in nanofluidics, as it paves the way towards smart devices that offer multifunctional applications including biosensing, targeted drug delivery, and fluidic mixing.

Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual pre-training, and the performance monotonically increased with the amount of training data up to 100B tokens. Consequently, Swallow achieved superior performance compared to other LLMs that were trained from scratch in English and Japanese. An analysis of the effects of continual pre-training revealed that it was particularly effective for Japanese question answering tasks. Furthermore, to elucidate effective methodologies for cross-lingual continual pre-training from English to Japanese, we investigated the impact of vocabulary expansion and the effectiveness of incorporating parallel corpora. The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.

Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.10 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora., Comment: 17 pages

Pretrained language models underpin several AI applications, but their high computational cost for training limits accessibility. Initiatives such as BLOOM and StarCoder aim to democratize access to pretrained models for collaborative community development. However, such existing models face challenges: limited multilingual capabilities, continual pretraining causing catastrophic forgetting, whereas pretraining from scratch is computationally expensive, and compliance with AI safety and development laws. This paper presents Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. Continually pretrained from StarCoderPlus on 435 billion additional tokens, Aurora-M surpasses 2 trillion tokens in total training token count. It is the first open-source multilingual model fine-tuned on human-reviewed safety instructions, thus aligning its development not only with conventional red-teaming considerations, but also with the specific concerns articulated in the Biden-Harris Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Aurora-M is rigorously evaluated across various tasks and languages, demonstrating robustness against catastrophic forgetting and outperforming alternatives in multilingual settings, particularly in safety evaluations. To promote responsible open-source LLM development, Aurora-M and its variants are released at https://huggingface.co/collections/aurora-m/aurora-m-models-65fdfdff62471e09812f5407 ., Comment: Preprint