Search

As an effective alternative to the direct fine-tuning on target tasks in specific languages, cross-lingual transfer addresses the challenges of limited training data by decoupling ''task ability'' and ''language ability'' by fine-tuning on the target task in the source language and another selected task in the target language, respectively. However, they fail to fully separate the task ability from the source language or the language ability from the chosen task. In this paper, we acknowledge the mutual reliance between task ability and language ability and direct our attention toward the gap between the target language and the source language on tasks. As the gap removes the impact of tasks, we assume that it remains consistent across tasks. Based on this assumption, we propose a new cross-lingual transfer method called $\texttt{AdaMergeX}$ that utilizes adaptive adapter merging. By introducing a reference task, we can determine that the divergence of adapters fine-tuned on the reference task in both languages follows the same distribution as the divergence of adapters fine-tuned on the target task in both languages. Hence, we can obtain target adapters by combining the other three adapters. Furthermore, we propose a structure-adaptive adapter merging method. Our empirical results demonstrate that our approach yields new and effective cross-lingual transfer, outperforming existing methods across all settings.

Large language models (LLMs) have demonstrated impressive capabilities across diverse languages. This study explores how LLMs handle multilingualism. Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow ($\texttt{MWork}$): LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures, respectively. In the final layers, LLMs generate responses aligned with the original language of the query. To verify $\texttt{MWork}$, we introduce Parallel Language-specific Neuron Detection ($\texttt{PLND}$) to identify activated neurons for inputs in different languages without any labeled data. Using $\texttt{PLND}$, we validate $\texttt{MWork}$ through extensive experiments involving the deactivation of language-specific neurons across various layers and structures. Moreover, $\texttt{MWork}$ allows fine-tuning of language-specific neurons with a small dataset, enhancing multilingual abilities in a specific language without compromising others. This approach results in an average improvement of $3.6\%$ for high-resource languages and $2.3\%$ for low-resource languages across all tasks with just $400$ documents.

As a dominant force in text-to-image generation tasks, Diffusion Probabilistic Models (DPMs) face a critical challenge in controllability, struggling to adhere strictly to complex, multi-faceted instructions. In this work, we aim to address this alignment challenge for conditional generation tasks. First, we provide an alternative view of state-of-the-art DPMs as a way of inverting advanced Vision-Language Models (VLMs). With this formulation, we naturally propose a training-free approach that bypasses the conventional sampling process associated with DPMs. By directly optimizing images with the supervision of discriminative VLMs, the proposed method can potentially achieve a better text-image alignment. As proof of concept, we demonstrate the pipeline with the pre-trained BLIP-2 model and identify several key designs for improved image generation. To further enhance the image fidelity, a Score Distillation Sampling module of Stable Diffusion is incorporated. By carefully balancing the two components during optimization, our method can produce high-quality images with near state-of-the-art performance on T2I-Compbench.

Cell alignment often forms nematic order, which can lead to anomalous collective cell flow due to the so-called active force. Although it is appreciated that cell migration is driven by traction force, a quantitative evaluation of the relationships between the traction force, the nematic patterning, and the cell flow velocity is still elusive. Here we have found that cellular traction force aligns almost perfectly and is proportional in amplitude to the gradient of the nematic order tensor, not only near the topological defects but also globally. Furthermore, the flow in the monolayer was best described by adding nonlinear forces and a diffusion term derived from symmetry considerations. These nonlinear active forces enhance density instability but suppress bending instability, explaining why cell accumulation and dispersion can occur in neural progenitor cell culture while their ordering pattern is stable., Comment: 5 pages, 4 figures, Supplementary 14 pages, 11 figures

With the incorporation of the UNet architecture, diffusion probabilistic models have become a dominant force in image generation tasks. One key design in UNet is the skip connections between the encoder and decoder blocks. Although skip connections have been shown to improve training stability and model performance, we reveal that such shortcuts can be a limiting factor for the complexity of the transformation. As the sampling steps decrease, the generation process and the role of the UNet get closer to the push-forward transformations from Gaussian distribution to the target, posing a challenge for the network's complexity. To address this challenge, we propose Skip-Tuning, a simple yet surprisingly effective training-free tuning method on the skip connections. Our method can achieve 100% FID improvement for pretrained EDM on ImageNet 64 with only 19 NFEs (1.75), breaking the limit of ODE samplers regardless of sampling steps. Surprisingly, the improvement persists when we increase the number of sampling steps and can even surpass the best result from EDM-2 (1.58) with only 39 NFEs (1.57). Comprehensive exploratory experiments are conducted to shed light on the surprising effectiveness. We observe that while Skip-Tuning increases the score-matching losses in the pixel space, the losses in the feature space are reduced, particularly at intermediate noise levels, which coincide with the most effective range accounting for image quality improvement.

We provide an alternative view of some results in [1, 3, 11]. In particular, we prove that (1) if a continuous self-map of a compact metric space has the shadowing, then the union of the basins of terminal chain components is a dense $G_\delta$-subset of the space; and (2) if a continuous self-map of a locally connected compact metric space has the shadowing, and if the chain recurrent set is totally disconnected, then the map is almost chain continuous., Comment: 8 pages

We selected sources with a steep soft-X-ray-band spectrum with a photon index larger than 2.5 -- measured by eROSITA on board the Spectrum-Roentgen-Gamma (SRG) -- from the eFEDS AGN catalogue as candidates of highly accreting supermassive black holes, and investigated their multi-wavelength properties. Among 601 bright AGN with 0.2-5 keV counts of greater than 100, 83 sources (~14%) are classified as steep-spectrum sources. These sources have typical 0.5-2 keV luminosities of L(SX) ~ 1e44 erg/s and the majority of them are found at redshifts below z=1. In comparison with sources with flatter spectra, these sources have, on average, a UV (or optical) to 2 keV luminosity ratio that is larger by ~0.3 dex and bluer optical-to-UV continuum emission. They also appear to be radio quiet based on the detection rate in the FIRST and VLASS surveys. Their host galaxies -- at least in the redshift range of z=0.2-0.8, where the AGN-galaxy decomposition results from the Subaru Hyper Suprime-Cam imaging are available -- tend to be late-type and have smaller stellar masses than those of sources with flatter spectra. These properties are similar to those found in nearby narrow-line Seyfert 1 galaxies, in agreement with the picture that they are AGN with elevated accretion rates and are in the early growth phase of black hole and galaxy co-evolution. However, the steep-spectrum sources are not exclusively narrow-line Seyfert 1 galaxies; indeed many are broad-line Seyfert 1 galaxies, as found by a catalogue search. This suggests that these steep-spectrum sources may be black holes generally with high accretion rates but of a wide mass range, including a few objects emitting at L(SX)>1e45 erg/s, of which black hole masses can be close to 10^9 M_sun., Comment: 8 pages, accepted for publication in A&A

Models prone to spurious correlations in training data often produce brittle predictions and introduce unintended biases. Addressing this challenge typically involves methods relying on prior knowledge and group annotation to remove spurious correlations, which may not be readily available in many applications. In this paper, we establish a novel connection between unsupervised object-centric learning and mitigation of spurious correlations. Instead of directly inferring subgroups with varying correlations with labels, our approach focuses on discovering concepts: discrete ideas that are shared across input samples. Leveraging existing object-centric representation learning, we introduce CoBalT: a concept balancing technique that effectively mitigates spurious correlations without requiring human labeling of subgroups. Evaluation across the benchmark datasets for sub-population shifts demonstrate superior or competitive performance compared state-of-the-art baselines, without the need for group annotation. Code is available at https://github.com/rarefin/CoBalT.

The Fokker-Planck (FP) equation is a foundational PDE in stochastic processes. However, curse of dimensionality (CoD) poses challenge when dealing with high-dimensional FP PDEs. Although Monte Carlo and vanilla Physics-Informed Neural Networks (PINNs) have shown the potential to tackle CoD, both methods exhibit numerical errors in high dimensions when dealing with the probability density function (PDF) associated with Brownian motion. The point-wise PDF values tend to decrease exponentially as dimension increases, surpassing the precision of numerical simulations and resulting in substantial errors. Moreover, due to its massive sampling, Monte Carlo fails to offer fast sampling. Modeling the logarithm likelihood (LL) via vanilla PINNs transforms the FP equation into a difficult HJB equation, whose error grows rapidly with dimension. To this end, we propose a novel approach utilizing a score-based solver to fit the score function in SDEs. The score function, defined as the gradient of the LL, plays a fundamental role in inferring LL and PDF and enables fast SDE sampling. Three fitting methods, Score Matching (SM), Sliced SM (SSM), and Score-PINN, are introduced. The proposed score-based SDE solver operates in two stages: first, employing SM, SSM, or Score-PINN to acquire the score; and second, solving the LL via an ODE using the obtained score. Comparative evaluations across these methods showcase varying trade-offs. The proposed method is evaluated across diverse SDEs, including anisotropic OU processes, geometric Brownian, and Brownian with varying eigenspace. We also test various distributions, including Gaussian, Log-normal, Laplace, and Cauchy. The numerical results demonstrate the score-based SDE solver's stability, speed, and performance across different settings, solidifying its potential as a solution to CoD for high-dimensional FP equations., Comment: 22 pages

Recently, the peak structure of the sound velocity was observed in the lattice simulation of two-color and two-flavor QCD at the finite quark chemical potential. The comparison with the chiral perturbation theory (ChPT) result was undertaken, however, the ChPT failed in reproducing the peak structure. In this study, to extend the ChPT framework, we incorporate contributions of the $\sigma$ meson, that is identified as the chiral partner of pions, on top of the low-energy pion dynamics by using the linear sigma model (LSM). Based on the LSM we derive analytic expressions of the thermodynamic quantities as well as the sound velocity within a mean-field approximation. As a result, we find that those quantities are provided by sums of the ChPT results and corrections, where the latter is characterized by a mass difference between the chiral partners, the $\sigma$ meson and pion. The chiral partner contributions are found to yield a peak in the sound velocity successfully. We furthermore show that the sound velocity peak emerges only when $m_\sigma >\sqrt{3}m_\pi$ and $\mu_q > m_\pi$, with $m_{\sigma(\pi)}$ and $\mu_q$ being the $\sigma$ meson (pion) mass and the quark chemical potential, respectively. The correlation between the sound velocity peak and the sign of the trace anomaly is also addressed., Comment: 14 pages, 4 figures; some references added; version accepted in PRD

Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, to enhance 3D-MoLM's ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties. We release our codes and datasets at https://github.com/lsh0520/3D-MoLM.

Understanding how people use area in a city can be a valuable information in a wide range of fields, from marketing to urban planning. Area usage is subject to change over time due to various events including seasonal shifts and pandemics. Before the spread of smartphones, this data had been collected through questionnaire survey. However, this is not a sustainable approach in terms of time to results and cost. There are many existing studies on area modeling, which characterize an area with some kind of information, using Point of Interest (POI) or inter-area movement data. However, since POI is data that is statically tied to space, and inter-area movement data ignores the behavior of people within an area, existing methods are not sufficient in terms of capturing area usage changes. In this paper, we propose a novel area modeling method named Area2Vec, inspired by Word2Vec, which models areas based on people's location data. This method is based on the discovery that it is possible to characterize an area based on its usage by using people's stay information in the area. And it is a novel method that can reflect the dynamically changing people's behavior in an area in the modeling results. We validated Area2vec by performing a functional classification of areas in a district of Japan. The results show that Area2Vec can be usable in general area analysis. We also investigated area usage changes due to COVID-19 in two districts in Japan. We could find that COVID-19 made people refrain from unnecessary going out, such as visiting entertainment areas., Comment: This paper is an English translation of the paper published in the Transactions of the Information Processing Society of Japan (http://doi.org/10.20729/00213190)

Cells contain multiple condensates which spontaneously form due to the heterotypic interactions between their components. Although the proteins and disordered region sequences that are responsible for condensate formation have been extensively studied, the rule of interactions between the components that allow demixing, i.e., the coexistence of multiple condensates, is yet to be elucidated. Here we construct an effective theory of the interaction between heteropolymers by fitting it to the molecular dynamics simulation results obtained for more than 200 sequences sampled from the disordered regions of human proteins. We find that the sum of amino acid pair interactions across two heteropolymers predicts the Boyle temperature qualitatively well, which can be quantitatively improved by the dimer pair approximation, where we incorporate the effect of neighboring amino acids in the sequences. The improved theory, combined with the finding of a metric that captures the effective interaction strength between distinct sequences, allowed the selection of up to three disordered region sequences that demix with each other in multicomponent simulations, as well as the generation of artificial sequences that demix with a given sequence.The theory points to a generic sequence design strategy to demix or hypermix thanks to the low dimensional nature of the space of the interactions that we identify. As a consequence of the geometric arguments in the space of interactions, we find that the number of distinct sequences that can demix with each other is strongly constrained, irrespective of the choice of the coarse-grained model. Altogether, we construct a theoretical basis for methods to estimate the effective interaction between heteropolymers, which can be utilized in predicting phase separation properties as well as rules of assignment in the localization and functions of disordered proteins., Comment: 24 pages, 24 figures

Model Predictive Control (MPC) is a versatile approach capable of accommodating diverse control requirements, holding significant promise for a broad spectrum of industrial applications. Noteworthy challenges associated with MPC include the substantial computational burden and the inherent difficulty in ensuring system stability. Recently, a rapid computation technique has been introduced as a potential solution. This method guides the input toward convergence with the optimal control problem solution by employing the primal-dual gradient (PDG) dynamics as a controller. Furthermore, stability assurances grounded in dissipativity theory have been established. However, these assurances are applicable solely to continuous-time feedback systems. As a consequence, when the controller undergoes discretization and is implemented as a sampled-data system, stability cannot be guaranteed. In this paper, we propose a discrete-time dynamical controller, incorporating specific modifications to the PDG approach, and present stability conditions relevant to the resulting sampled-data system. Additionally, we introduce an extension designed to enhance control performance. Numerical examples substantiate that our proposed method not only enhances control effectiveness but also effectively discerns stability degradation resulting from discretization, a nuance often overlooked by conventional methods.

We investigate the effect of charm quark on the chiral phase transition of light quarks at finite temperature based on the four-flavor soft-wall holographic QCD model. In the massless limit, we find that the thermal chiral phase transition is of the second order in the four-quark flavor system. In the case with the massive charm quark and the massless light and strange quarks, the order of the phase transition changes to the first order. This is due to the quark flavor symmetry breaking which is associated with the violation of the $U(1)$ axial symmetry. Once the light and strange quarks get massive, the explicit chiral symmetry breaking becomes eminent, then the crossover phase transition is realized at the physical quark masses. We also map the order of the phase transition on a phase diagram in the quark mass plane where the light- and strange-quark masses are degenerate but differ from the value of the charm quark mass. This phase diagram is an extension of the conventional Columbia plot to the four-quark flavor system. The critical exponents related to the chiral phase transition are also addressed., Comment: We have updated some parts of the text and added a subsection on critical exponents

The commercialization of text-to-image diffusion models (DMs) brings forth potential copyright concerns. Despite numerous attempts to protect DMs from copyright issues, the vulnerabilities of these solutions are underexplored. In this study, we formalized the Copyright Infringement Attack on generative AI models and proposed a backdoor attack method, SilentBadDiffusion, to induce copyright infringement without requiring access to or control over training processes. Our method strategically embeds connections between pieces of copyrighted information and text references in poisoning data while carefully dispersing that information, making the poisoning data inconspicuous when integrated into a clean dataset. Our experiments show the stealth and efficacy of the poisoning data. When given specific text prompts, DMs trained with a poisoning ratio of 0.20% can produce copyrighted images. Additionally, the results reveal that the more sophisticated the DMs are, the easier the success of the attack becomes. These findings underline potential pitfalls in the prevailing copyright protection strategies and underscore the necessity for increased scrutiny to prevent the misuse of DMs., Comment: Accepted for presentation at ICML 2024

Diffusion-based generative methods have proven effective in modeling trajectories with offline datasets. However, they often face computational challenges and can falter in generalization, especially in capturing temporal abstractions for long-horizon tasks. To overcome this, we introduce the Hierarchical Diffuser, a simple, fast, yet surprisingly effective planning method combining the advantages of hierarchical and diffusion-based planning. Our model adopts a "jumpy" planning strategy at the higher level, which allows it to have a larger receptive field but at a lower computational cost -- a crucial factor for diffusion-based planning methods, as we have empirically verified. Additionally, the jumpy sub-goals guide our low-level planner, facilitating a fine-tuning stage and further improving our approach's effectiveness. We conducted empirical evaluations on standard offline reinforcement learning benchmarks, demonstrating our method's superior performance and efficiency in terms of training and planning speed compared to the non-hierarchical Diffuser as well as other hierarchical planning methods. Moreover, we explore our model's generalization capability, particularly on how our method improves generalization capabilities on compositional out-of-distribution tasks.

We successfully measured the trigonometric parallax of Sagittarius A* (Sgr A*) to be $117\pm17$ micro-arcseconds ($\mu$as) using the VLBI Exploration of Radio Astrometry (VERA) with the newly developed broad-band signal-processing system named OCTAVE-DAS. The measured parallax corresponds to a Galactocentric distance at the Sun of $R_0 = 8.5^{+1.5}_{-1.1}$ kpc. By combining the astrometric results with VERA and the Very Long Baseline Array (VLBA) over a monitoring period of 25 years, the proper motion of Sgr A* is obtained to be $(\mu_\alpha, \mu_\delta) = (-3.133\pm0.003, -5.575\pm0.005)$ mas yr$^{-1}$ in equatorial coordinates, corresponding to $(\mu_l, \mu_b) = (-6.391\pm0.005, -0.230\pm0.004)$ mas yr$^{-1}$ in Galactic coordinates. This gives an angular orbital velocity of the Sun of $\Omega_\odot = 30.30 \pm 0.02$ km s$^{-1}$ kpc$^{-1}$. We find upper limits to the core wander, $\Delta \theta < 0.20$ mas (1.6 AU), peculiar motion, $\Delta \mu < 0.10$ mas yr$^{-1}$ (3.7 km s$^{-1}$), and acceleration, $a < 2.6$ $\mu$as yr$^{-2}$ (0.10 km s$^{-1}$ yr$^{-1}$) for Sgr A*. Thus, we obtained upper mass limits of $\approx$ 3 $\times$ 10$^{4}$$M_{\odot}$ and $\approx$ 3 $\times$ 10$^{3}$$M_{\odot}$ for the supposed intermediate-mass black holes at 0.1 and 0.01 pc from the Galactic center, respectively., Comment: 15 pages, 9 figures, accepted for publication in the Publications of the Astronomical Society of Japan

Creativity serves as a cornerstone for societal progress and innovation. With the rise of advanced generative AI models capable of tasks once reserved for human creativity, the study of AI's creative potential becomes imperative for its responsible development and application. In this paper, we prove in theory that AI can be as creative as humans under the condition that it can properly fit the data generated by human creators. Therefore, the debate on AI's creativity is reduced into the question of its ability to fit a sufficient amount of data. To arrive at this conclusion, this paper first addresses the complexities in defining creativity by introducing a new concept called Relative Creativity. Rather than attempting to define creativity universally, we shift the focus to whether AI can match the creative abilities of a hypothetical human. The methodological shift leads to a statistically quantifiable assessment of AI's creativity, term Statistical Creativity. This concept, statistically comparing the creative abilities of AI with those of specific human groups, facilitates theoretical exploration of AI's creative potential. Our analysis reveals that by fitting extensive conditional data without marginalizing out the generative conditions, AI can emerge as a hypothetical new creator. The creator possesses the same creative abilities on par with the human creators it was trained on. Building on theoretical findings, we discuss the application in prompt-conditioned autoregressive models, providing a practical means for evaluating creative abilities of generative AI models, such as Large Language Models (LLMs). Additionally, this study provides an actionable training guideline, bridging the theoretical quantification of creativity with practical model training., Comment: The paper examines AI's creativity, introducing Relative and Statistical Creativity for theoretical and practical analysis, along with practical training guidelines. Project Page: ai-relative-creativity.github.io

Physics-Informed Neural Networks (PINNs) have proven effective in solving partial differential equations (PDEs), especially when some data are available by seamlessly blending data and physics. However, extending PINNs to high-dimensional and even high-order PDEs encounters significant challenges due to the computational cost associated with automatic differentiation in the residual loss. Herein, we address the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE). Starting with the second-order high-dimensional PDEs ubiquitous in scientific computing, HTE transforms the calculation of the entire Hessian matrix into a Hessian vector product (HVP). This approach alleviates the computational bottleneck via Taylor-mode automatic differentiation and significantly reduces memory consumption from the Hessian matrix to HVP. We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions. Comparisons with Stochastic Dimension Gradient Descent (SDGD) highlight the distinct advantages of HTE, particularly in scenarios with significant variance among dimensions. We further extend HTE to higher-order and higher-dimensional PDEs, specifically addressing the biharmonic equation. By employing tensor-vector products (TVP), HTE efficiently computes the colossal tensor associated with the fourth-order high-dimensional biharmonic equation, saving memory and enabling rapid computation. The effectiveness of HTE is illustrated through experimental setups, demonstrating comparable convergence rates with SDGD under memory and speed constraints. Additionally, HTE proves valuable in accelerating the Gradient-Enhanced PINN (gPINN) version as well as the Biharmonic equation. Overall, HTE opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs., Comment: Published in Computer Methods in Applied Mechanics and Engineering

We propose a new method, Adversarial In-Context Learning (adv-ICL), to optimize prompt for in-context learning (ICL) by employing one LLM as a generator, another as a discriminator, and a third as a prompt modifier. As in traditional adversarial learning, adv-ICL is implemented as a two-player game between the generator and discriminator, where the generator tries to generate realistic enough output to fool the discriminator. In each round, given an input prefixed by task instructions and several exemplars, the generator produces an output. The discriminator is then tasked with classifying the generator input-output pair as model-generated or real data. Based on the discriminator loss, the prompt modifier proposes possible edits to the generator and discriminator prompts, and the edits that most improve the adversarial loss are selected. We show that adv-ICL results in significant improvements over state-of-the-art prompt optimization techniques for both open and closed-source models on 11 generation and classification tasks including summarization, arithmetic reasoning, machine translation, data-to-text generation, and the MMLU and big-bench hard benchmarks. In addition, because our method uses pre-trained models and updates only prompts rather than model parameters, it is computationally efficient, easy to extend to any LLM and task, and effective in low-resource settings., Comment: ACL 2024

Studying a centi-parsec supermassive black hole binary (SMBHB) would allow us to explore a new parameter space in active galactic nuclei, and these objects are also potential sources of gravitational waves. We report evidence that an SMBHB with an orbital period of about 30 yr may be resident in the nearby galactic nucleus M81. This orbital period and the known mass of M81 imply an orbital separation of about 0.02 pc. The jet emanating from the primary black hole showed a short period of jet wobbling at about 16.7 yr, superposing a long-term precession at a timescale of several hundred years. Periodic radio and X-ray outbursts were also found two times per orbital period, which could be explained by a double-peaked mass accretion rate variation per binary orbit. If confirmed, M81 would be one of the closest SMBHB candidates, providing a rare opportunity to study the final parsec problem., Comment: 10 pages, 4 figures, published in the ApJ

Estimating 3D rotations is a common procedure for 3D computer vision. The accuracy depends heavily on the rotation representation. One form of representation -- rotation matrices -- is popular due to its continuity, especially for pose estimation tasks. The learning process usually incorporates orthogonalization to ensure orthonormal matrices. Our work reveals, through gradient analysis, that common orthogonalization procedures based on the Gram-Schmidt process and singular value decomposition will slow down training efficiency. To this end, we advocate removing orthogonalization from the learning process and learning unorthogonalized `Pseudo' Rotation Matrices (PRoM). An optimization analysis shows that PRoM converges faster and to a better solution. By replacing the orthogonalization incorporated representation with our proposed PRoM in various rotation-related tasks, we achieve state-of-the-art results on large-scale benchmarks for human pose estimation.