Search

Full Waveform Inversion (FWI) stands as a nonlinear, high-resolution technology for subsurface imaging via surface-recorded data. This paper introduces an augmented Lagrangian dual formulation for FWI, rooted in the viewpoint that Lagrange multipliers serve as fundamental unknowns for the accurate linearization of the FWI problem. Once these multipliers are estimated, the determination of model parameters becomes simple. Therefore, unlike traditional primal algorithms, the proposed dual method circumvents direct engagement with model parameters or wavefields, instead tackling the estimation of Lagrange multipliers through a gradient ascent iteration. This approach yields two significant advantages: i) the background model remains fixed, requiring only one LU matrix factorization for each frequency inversion. ii) Convergence of the algorithm can be improved by leveraging techniques like quasi-Newton l-BFGS methods and Anderson acceleration. Numerical examples from elastic and acoustic FWI utilizing different benchmark models are provided, showing that the dual algorithm converges quickly and requires fewer computations than the standard primal algorithm.

In this paper we find several teleparallel $F(T,B)$ solutions for a Robertson-Walker (TRW) cosmological spacetime. We first set and solve the $F(T,B)$-type field equations for a linear perfect fluid. Using similar techniques, we then find new $F(T,B)$ solutions for non-linear perfect fluids with a weak-quadratic correction term to the linear equation of state (EoS). Finally, we solve for new classes of $F(T,B)$ solutions for a scalar field source by assuming a power-law scalar field and then an exponential scalar field in terms of the time-coordinate. For flat cosmological cases ($k=0$ cases), we find new exact and approximate $F(T,B)$ solutions. For non-flat cases ($k=\pm 1$ cases), we only find new teleparallel $F(T,B)$ solutions for some specific and well-defined cosmological expansion subcases. We conclude by briefly discussing the impact of these new teleparallel solutions on cosmological processes such as dark energy quintessence and phantom energy models., Comment: 35 pages, no figure

With the rapid advancement of devices requiring intensive computation, such as Internet of Things (IoT) devices, smart sensors, and wearable technology, the computational demands on individual platforms with limited resources have escalated, necessitating the offloading of the generated tasks by the devices to edge. These tasks are often real-time with strict response time requirements. Among these devices, autonomous vehicles present unique challenges due to their critical need for timely and accurate processing to ensure passenger safety. Selecting suitable servers in a heterogeneous mobile edge computing (MEC) architecture is vital to optimizing real-time task processing rates for such applications. To address this, we present an algorithmic solution to improve the allocation of heterogeneous servers to real-time tasks, aiming to maximize the number of processed tasks. By analyzing task and server characteristics in the MEC architecture, we develop the suitability-based adaptive resource selection (SARS) algorithm, which evaluates server suitability based on factors like time constraints and server capabilities. Additionally, we introduce the proactive on-demand resource allocation (PORA) algorithm, which strategically reserves computational resources to ensure availability for critical real-time tasks. We compare the proposed algorithms with several classical and state-of-the-art algorithms. Computational results demonstrate that our approach outperforms existing algorithms, processes more tasks, and effectively prioritizes urgent tasks, particularly in autonomous driving applications., Comment: Accepted to the Proceedings of the 2024 International Conference on Computational Science and Computational Intelligence (CSCI'24), December 11-13, 2024, Las Vegas, USA

Boundary Representation (B-Rep) is the de facto representation of 3D solids in Computer-Aided Design (CAD). B-Rep solids are defined with a set of NURBS (Non-Uniform Rational B-Splines) surfaces forming a closed volume. To represent a surface, current works often employ the UV-grid approximation, i.e., sample points uniformly on the surface. However, the UV-grid method is not efficient in surface representation and sometimes lacks precision and regularity. In this work, we propose NeuroNURBS, a representation learning method to directly encode the parameters of NURBS surfaces. Our evaluation in solid generation and segmentation tasks indicates that the NeuroNURBS performs comparably and, in some cases, superior to UV-grids, but with a significantly improved efficiency: for training the surface autoencoder, GPU consumption is reduced by 86.7%; memory requirement drops by 79.9% for storing 3D solids. Moreover, adapting BrepGen for solid generation with our NeuroNURBS improves the FID from 30.04 to 27.24, and resolves the undulating issue in generated surfaces.

Emerging Large Language Model (LLM) applications require long input prompts to perform complex downstream tasks like document analysis and code generation. For these long context length applications, the length of the input prompt poses a significant challenge in terms of inference efficiency since the inference costs increase linearly with sequence length. However, for many of these applications, much of the context in the prompt is fixed across different user inputs, thereby providing the opportunity to perform offline optimizations to process user inputs quickly, as they are received. In this work, we propose Squeezed Attention as a mechanism to accelerate LLM applications where a large portion of the input prompt is fixed. We first leverage K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. During inference, we compare query tokens from the user input with the centroids to predict which of the keys from the fixed context are semantically relevant and need to be loaded during inference. We then compute exact attention using only these important keys from the fixed context, thereby reducing bandwidth and computational costs. We also extend our method to use a hierarchical centroid lookup to identify important keys, which can reduce the complexity of attention from linear to logarithmic with respect to the context length. We implement optimized Triton kernels for centroid comparison and sparse FlashAttention with important keys, achieving more than 4x speedups during both the prefill and generation phases for long-context inference. Furthermore, we have extensively evaluated our method on various long-context benchmarks including LongBench, where it achieves a 3x reduction in KV cache budget without accuracy loss and up to an 8x reduction with <0.5 point accuracy gap for various models.

We determine parameters of the renormalization group-consistent three-flavor color-superconducting Nambu-Jona-Lasinio (NJL) model that are suited to investigate possible compact-star configurations. Our goal is to provide quark-matter equations of state (EoS) that can be used for hadron-quark hybrid-star constructions. To that end, we mainly focus on the parameters of the quark-matter model. By varying the vector and diquark coupling constants, we analyze their impact on the EoS, the speed of sound, the maximum diquark gap, and the mass-radius relation. In almost all configurations, a stable color-flavor-locked (CFL) phase appears in the core of the maximum-mass configurations, typically spanning several kilometers in radius. In other cases, the star's two-flavor color-superconducting (2SC) branch of the EoS becomes unstable before reaching the CFL transition density. At neutron-star densities, the speed of sound squared reaches up to $c_s^2 \sim 0.6$ and the CFL gap up to $\Delta\sim250\,$MeV. We argue that adding a hadronic EoS at lower densities by performing a Maxwell construction does not increase the maximum mass substantially. Thus we use the $2.0 M_{\odot}$ constraint to constrain the NJL model parameters that are suited for the construction of hybrid-star EoS. We construct three examples of the hybrid-star model, demonstrating that there is room for different color-superconducting compositions. The hybrid EoSs obtained in this way can have no 2SC matter or different ratios of 2SC and CFL quark matter in the core. We show that early hadron-quark transitions are possible that can modify the tidal deformability at 1.4 $M_\odot$. We find that these EoSs are consistent with the imposed constraints from astrophysics and perturbative QCD. They allow for different hybrid-star scenarios with a hadronic EoS that is soft at low to intermediate densities ($\sim 1-3\, n_{\text{sat}}$)., Comment: 21 pages, 12 figures, 2 tables, comments are welcomed

In this paper, a novel approach for wireless localization is proposed and experimentally validated that leverages space-time coded reconfigurable intelligent surfaces (RIS). It is demonstrated that applying proper single-bit codes to each RIS element, enables accurate determination of the direction of arrival (AOA) at the receiver. Moreover, we introduce different scenarios that such technique can be used for localization. By incorporating RIS, a passive component, the method significantly reduces the complexity found in previous localization techniques. Additionally, the use of 1-bit codes minimizes hardware requirements, offering a reliable, low-cost solution for localization in advanced telecommunications networks.

This paper presents experimental realization of a reconfigurable intelligent surface (RIS) using space-time coding metasurfaces to enable concurrent beam steering and data modulation. The proposed approach harnesses the capabilities of metasurfaces, allowing precise temporal control over individual unit cells of the RIS. We show that by employing proper binary codes manipulating the state of unit cells, the RIS can act as a digital data modulator with beam steering capability. We describe the experimental setup and computational tools, followed by validation through harmonic generation and investigation of beam steering and data modulation. Additionally, four digital modulation schemes are evaluated. By implementing customized binary codes, constellations under varying conditions are compared, showcasing the potential for real-world applications. This study offers new insights into the practical implementation of RIS for advanced wireless communication systems.

The high-luminosity upgrade of the LHC (HL-LHC) brings unprecedented requirements for precision bunch-by-bunch luminosity measurement and beam-induced background monitoring in real time. A key component of the CMS Beam Radiation Instrumentation and Luminosity detector system is a stand-alone luminometer, the Fast Beam Condition Monitor (FBCM), which is able to operate independently at all times with a triggerless asynchronous readout. FBCM utilizes a dedicated front-end ASIC to amplify the signals from CO$_2$-cooled silicon-pad sensors with 1 ns timing resolution. Front-end (FE) electronics are subject to high-radiation conditions, thus all components are radiation hardened: sensors, ASICs, transceivers, etc. The FBCM ASIC contains 6 channels, each outputting a high-speed binary signal carrying the time-of-arrival and time-over-threshold information. This signal is sent via a gigabit optical link to the back-end electronics for analysis. A dedicated test system is designed for the FBCM FE electronics with a modular setup for all testing needs of the project from initial ASIC validation test to system-level testing with the full read-out chain. The paper reports on the design, read-out architecture, and testing program for the FBCM electronics., Comment: 6 pages, 5 figures. Presented at 6th International Conference on Technology and Instrumentation in Particle Physics (TIPP2023), 4-8 Sep 2023, Cape Town, Western Cape, South Africa. Submitted to PoS(TIPP2023)

User equipment (UE) devices with high compute performance acting on data with dynamic and stochastic nature to train Artificial Intelligence/Machine Learning (AI/ML) models call for real-time, agile distributed machine learning (DL) algorithms. Consequently, we focus on UE-centric DL algorithms where UEs initiate requests to adapt AI/ML models for better performance, e.g., locally refined AI/ML models among a set of headsets or smartphones. This new setup requires selecting a set of UEs as aggregators (here called leaders) and another set as followers, where all UEs update their models based on their local data, and followers share theirs with leaders for aggregation. From a networking perspective, the first question is how to select leaders and associate followers efficiently. This results in a high dimensional mixed integer programming problem and involves internal UE state information and state information among UEs, called external state information in this paper. To address this challenge, we introduce two new indices: a Leader Internal Index (LII), which is a function of the internal states of each device, demonstrating the willingness to be a leader such as battery life and AI/hardware accelerators, and a Leader eXternal Index (LXI), which is a function of external state information among UEs, such as trust, channel condition, and any aspect relevant for associating a follower with a leader. These two indices transform the highly complex leader selection and follower association problem into a better tractable formulation. More importantly, LIIs and LXIs allow to keep the internal and external state information of this problem inside of each device without compromising users' privacy.

In the medical domain, acquiring large datasets poses significant challenges due to privacy concerns. Nonetheless, the development of a robust deep-learning model for retinal disease diagnosis necessitates a substantial dataset for training. The capacity to generalize effectively on smaller datasets remains a persistent challenge. The scarcity of data presents a significant barrier to the practical implementation of scalable medical AI solutions. To address this issue, we've combined a wide range of data sources to improve performance and generalization to new data by giving it a deeper understanding of the data representation from multi-modal datasets and developed a self-supervised framework based on large language models (LLMs), SwinV2 to gain a deeper understanding of multi-modal dataset representations, enhancing the model's ability to extrapolate to new data for the detection of eye diseases using optical coherence tomography (OCT) images. We adopt a two-phase training methodology, self-supervised pre-training, and fine-tuning on a downstream supervised classifier. An ablation study conducted across three datasets employing various encoder backbones, without data fusion, with low data availability setting, and without self-supervised pre-training scenarios, highlights the robustness of our method. Our findings demonstrate consistent performance across these diverse conditions, showcasing superior generalization capabilities compared to the baseline model, ResNet-50., Comment: 25 pages, 9 tables, 10 figures

Understanding the orientation of geological structures is crucial for analyzing the complexity of the Earths' subsurface. For instance, information about geological structure orientation can be incorporated into local anisotropic regularization methods as a valuable tool to stabilize the solution of inverse problems and produce geologically plausible solutions. We introduce a new variational method that employs the alternating direction method of multipliers within an alternating minimization scheme to jointly estimate orientation and model parameters in both 2D and 3D inverse problems. Specifically, the proposed approach adaptively integrates recovered information about structural orientation, enhancing the effectiveness of anisotropic Tikhonov regularization in recovering geophysical parameters. The paper also discusses the automatic tuning of algorithmic parameters to maximize the new method's performance. The proposed algorithm is tested across diverse 2D and 3D examples, including structure-oriented denoising and trace interpolation. The results show that the algorithm is robust in solving the considered large and challenging problems, alongside efficiently estimating the associated tilt field in 2D cases and the dip, strike, and tilt fields in 3D cases. Synthetic and field examples show that the proposed anisotropic regularization method produces a model with enhanced resolution and provides a more accurate representation of the true structures.

Recent advancements in function calling and tool use have significantly enhanced the capabilities of large language models (LLMs) by enabling them to interact with external information sources and execute complex tasks. However, the limited context window of LLMs presents challenges when a large number of tools are available, necessitating efficient methods to manage prompt length and maintain accuracy. Existing approaches, such as fine-tuning LLMs or leveraging their reasoning capabilities, either require frequent retraining or incur significant latency overhead. A more efficient solution involves training smaller models to retrieve the most relevant tools for a given query, although this requires high quality, domain-specific data. To address those challenges, we present a novel framework for generating synthetic data for tool retrieval applications and an efficient data-driven tool retrieval strategy using small encoder models. Empowered by LLMs, we create ToolBank, a new tool retrieval dataset that reflects real human user usages. For tool retrieval methodologies, we propose novel approaches: (1) Tool2Vec: usage-driven tool embedding generation for tool retrieval, (2) ToolRefiner: a staged retrieval method that iteratively improves the quality of retrieved tools, and (3) MLC: framing tool retrieval as a multi-label classification problem. With these new methods, we achieve improvements of up to 27.28 in Recall@K on the ToolBench dataset and 30.5 in Recall@K on ToolBank. Additionally, we present further experimental results to rigorously validate our methods. Our code is available at \url{https://github.com/SqueezeAILab/Tool2Vec}

Recent large language models (LLMs) have enabled the development of advanced agentic systems that can integrate various tools and APIs to fulfill user queries through function calling. However, the deployment of these LLMs on the edge has not been explored since they typically require cloud-based infrastructure due to their substantial model size and computational demands. To this end, we present TinyAgent, an end-to-end framework for training and deploying task-specific small language model agents capable of function calling for driving agentic systems at the edge. We first show how to enable accurate function calling for open-source models via the LLMCompiler framework. We then systematically curate a high-quality dataset for function calling, which we use to fine-tune two small language models, TinyAgent-1.1B and 7B. For efficient inference, we introduce a novel tool retrieval method to reduce the input prompt length and utilize quantization to further accelerate the inference speed. As a driving application, we demonstrate a local Siri-like system for Apple's MacBook that can execute user commands through text or voice input. Our results show that our models can achieve, and even surpass, the function-calling capabilities of larger models like GPT-4-Turbo, while being fully deployed at the edge. We open-source our dataset, models, and installable package and provide a demo video for our MacBook assistant agent., Comment: EMNLP 2024 Demo

The Nambu-Jona-Lasinio (NJL) model and specifically its extension to color superconductivity (CSC) is a popular effective model for investigating dense quark matter. However, the reliability of its results is challenged by cutoff artifacts, which emerge if temperature or chemical potential are of the order of the cutoff energy scales. In this work, we generalize an idea from [Braun et al. SciPost Phys., 6:056, 2019], which is based on the requirement of renormalization-group (RG) consistency and has successfully been applied to the two-flavor Quark-Meson-Diquark model, to the NJL model for electrically and color-neutral three-flavor color-superconducting quark matter. To this end, we analyze the medium divergences of the model and eliminate them by appropriate counterterms, introducing three different schemes. We show that the RG-consistent treatment removes the cutoff artifacts of the conventional regularization and enables the investigation of CSC matter at higher densities by the model. Our studies reveal the emergence of a so-called dSC phase within the melting pattern of the Color-Flavor Locked (CFL) phase at high chemical potentials, consistent with earlier Ginzburg-Landau analyses., Comment: 28 pages, 8 figures, 1 table

Flood-induced deformation of the bed topography of fluvial meandering rivers could lead to river bank displacement, structural failure of the infrastructures, and the propagation of scour or deposition features. The assessment of sediment transport in large-scale meanders is, therefore, a key environmental issue. High-fidelity numerical models provide powerful tools for such assessments. However, high-fidelity simulations of large-scale rivers using the coupled flow and morphodynamics modules can be computationally expensive, owing to the costly two-way coupling between turbulence and bed morphodynamics. This study seeks to present a novel machine learning approach, which is trained using coupled large-eddy simulation (LES) and morphodynamics results. The proposed machine learning approach predicts bed shear stress and equilibrium bed morphology of large-scale meanders under bankfull flow conditions at a fraction of the cost of coupled LES-morphodynamics. We developed and evaluated the performance of a convolutional neural network autoencoder (CNNAE) algorithm to generate high-fidelity bed shear stress and equilibrium morphology of large-scale meandering rivers. The CNNAE algorithm utilizes instantaneous shear stress distribution and change of bed elevation of high-fidelity simulation results, along with geometric parameters of meanders as inputs to predict mean bed shear stress distribution and equilibrium bed elevation of rivers. The results demonstrated the feasibility, accuracy, and efficiency of the proposed CNNAE algorithm.

Despite recent advances in human pose estimation (HPE), poor generalization to out-of-distribution (OOD) data remains a difficult problem. While previous works have proposed Test-Time Adaptation (TTA) to bridge the train-test domain gap by refining network parameters at inference, the absence of ground-truth annotations makes it highly challenging and existing methods typically increase inference times by one or more orders of magnitude. We observe that 1) not every test time sample is OOD, and 2) HPE errors are significantly larger on distal keypoints (wrist, ankle). To this end, we propose ESCAPE: a lightweight correction and selective adaptation framework which applies a fast, forward-pass correction on most data while reserving costly TTA for OOD data. The free energy function is introduced to separate OOD samples from incoming data and a correction network is trained to estimate the errors of pretrained backbone HPE predictions on the distal keypoints. For OOD samples, we propose a novel self-consistency adaptation loss to update the correction network by leveraging the constraining relationship between distal keypoints and proximal keypoints (shoulders, hips), via a second ``reverse" network. ESCAPE improves the distal MPJPE of five popular HPE models by up to 7% on unseen data, achieves state-of-the-art results on two popular HPE benchmarks, and is significantly faster than existing adaptation methods., Comment: 32 pages, 8 figures

Long context inference presents challenges at the system level with increased compute and memory requirements, as well as from an accuracy perspective in being able to reason over long contexts. Recently, several methods have been proposed to compress the prompt to reduce the context length. However, there has been little work on comparing the different proposed methods across different tasks through a standardized analysis. This has led to conflicting results. To address this, here we perform a comprehensive characterization and evaluation of different prompt compression methods. In particular, we analyze extractive compression, summarization-based abstractive compression, and token pruning methods. Surprisingly, we find that extractive compression often outperforms all the other approaches, and enables up to 10x compression with minimal accuracy degradation. Interestingly, we also find that despite several recent claims, token pruning methods often lag behind extractive compression. We only found marginal improvements on summarization tasks., Comment: Es-FoMo @ ICML 2024

Extreme data rate scientific experiments create massive amounts of data that require efficient ML edge processing. This leads to unique validation challenges for VLSI implementations of ML algorithms: enabling bit-accurate functional simulations for performance validation in experimental software frameworks, verifying those ML models are robust under extreme quantization and pruning, and enabling ultra-fine-grained model inspection for efficient fault tolerance. We discuss approaches to developing and validating reliable algorithms at the scientific edge under such strict latency, resource, power, and area requirements in extreme experimental environments. We study metrics for developing robust algorithms, present preliminary results and mitigation strategies, and conclude with an outlook of these and future directions of research towards the longer-term goal of developing autonomous scientific experimentation methods for accelerated scientific discovery., Comment: IEEE VLSI Test Symposium 2024 (VTS)

Far from equilibrium, chemical and biological systems can form complex patterns and waves through reaction-diffusion coupling. Fluid motion often tends to disrupt these self-organized concentration patterns. In this study, we investigate the influence of Marangoni-driven flows in a thin layer of fluid ascending the outer surfaces of hydrophilic obstacles on the spatio-temporal dynamics of chemical waves in the modified Belousov-Zhabotinsky reaction. Our observations reveal that circular waves originate nearly simultaneously at the obstacles and propagate outward. In a covered setup, where evaporation is minimal, the wavefronts maintain their circular shape. However, in an uncovered setup with significant evaporative cooling, the interplay between surface tension-driven Marangoni flows and gravity destabilizes the wavefronts, creating distinctive flower-like patterns around the obstacles. Our experiments further show that the number of petals formed increases linearly with the obstacle's diameter, though a minimum diameter is required for these instabilities to appear. These findings demonstrate the potential to 'engineer' specific wave patterns, offering a method to control and direct reaction dynamics. This capability is especially important for developing microfluidic devices requiring precise control over chemical wave propagation.

Large language models (LLMs) demonstrate impressive capabilities in mathematical reasoning. However, despite these achievements, current evaluations are mostly limited to specific mathematical topics, and it remains unclear whether LLMs are genuinely engaging in reasoning. To address these gaps, we present the Mathematical Topics Tree (MaTT) benchmark, a challenging and structured benchmark that offers 1,958 questions across a wide array of mathematical subjects, each paired with a detailed hierarchical chain of topics. Upon assessing different LLMs using the MaTT benchmark, we find that the most advanced model, GPT-4, achieved a mere 54\% accuracy in a multiple-choice scenario. Interestingly, even when employing Chain-of-Thought prompting, we observe mostly no notable improvement. Moreover, LLMs accuracy dramatically reduced by up to 24.2 percentage point when the questions were presented without providing choices. Further detailed analysis of the LLMs' performance across a range of topics showed significant discrepancy even for closely related subtopics within the same general mathematical area. In an effort to pinpoint the reasons behind LLMs performances, we conducted a manual evaluation of the completeness and correctness of the explanations generated by GPT-4 when choices were available. Surprisingly, we find that in only 53.3\% of the instances where the model provided a correct answer, the accompanying explanations were deemed complete and accurate, i.e., the model engaged in genuine reasoning.

Regularization techniques are necessary to compute meaningful solutions to discrete ill-posed inverse problems. The well-known 2-norm Tikhonov regularization method equipped with a discretization of the gradient operator as regularization operator penalizes large gradient components of the solution to overcome instabilities. However, this method is homogeneous, i.e., it does not take into account the orientation of the regularized solution and therefore tends to smooth the desired structures, textures and discontinuities, which often contain important information. If the local orientation field of the solution is known, a possible way to overcome this issue is to implement local anisotropic regularization by penalizing weighted directional derivatives. In this paper, considering problems that are inherently two-dimensional, we propose to automatically and simultaneously recover the regularized solution and the local orientation parameters (used to define the anisotropic regularization term) by solving a bilevel optimization problem. Specifically, the lower level problem is Tikhonov regularization equipped with local anisotropic regularization, while the objective function of the upper level problem encodes some natural assumptions about the local orientation parameters and the Tikhonov regularization parameter. Application of the proposed algorithm to a variety of inverse problems in imaging (such as denoising, deblurring, tomography and Dix inversion), with both real and synthetic data, shows its effectiveness and robustness.

Prize-Collecting Steiner Tree (PCST) is a generalization of the Steiner Tree problem, a fundamental problem in computer science. In the classic Steiner Tree problem, we aim to connect a set of vertices known as terminals using the minimum-weight tree in a given weighted graph. In this generalized version, each vertex has a penalty, and there is flexibility to decide whether to connect each vertex or pay its associated penalty, making the problem more realistic and practical. Both the Steiner Tree problem and its Prize-Collecting version had long-standing $2$-approximation algorithms, matching the integrality gap of the natural LP formulations for both. This barrier for both problems has been surpassed, with algorithms achieving approximation factors below $2$. While research on the Steiner Tree problem has led to a series of reductions in the approximation ratio below $2$, culminating in a $\ln(4)+\epsilon$ approximation by Byrka, Grandoni, Rothvo{\ss}, and Sanit\`a, the Prize-Collecting version has not seen improvements in the past 15 years since the work of Archer, Bateni, Hajiaghayi, and Karloff, which reduced the approximation factor for this problem from $2$ to $1.9672$. Interestingly, even the Prize-Collecting TSP approximation, which was first improved below $2$ in the same paper, has seen several advancements since then. In this paper, we reduce the approximation factor for the PCST problem substantially to 1.7994 via a novel iterative approach.

Denoising diffusion models have recently gained prominence as powerful tools for a variety of image generation and manipulation tasks. Building on this, we propose a novel tool for real-time editing of images that provides users with fine-grained region-targeted supervision in addition to existing prompt-based controls. Our novel editing technique, termed Layered Diffusion Brushes, leverages prompt-guided and region-targeted alteration of intermediate denoising steps, enabling precise modifications while maintaining the integrity and context of the input image. We provide an editor based on Layered Diffusion Brushes modifications, which incorporates well-known image editing concepts such as layer masks, visibility toggles, and independent manipulation of layers; regardless of their order. Our system renders a single edit on a 512x512 image within 140 ms using a high-end consumer GPU, enabling real-time feedback and rapid exploration of candidate edits. We validated our method and editing system through a user study involving both natural images (using inversion) and generated images, showcasing its usability and effectiveness compared to existing techniques such as InstructPix2Pix and Stable Diffusion Inpainting for refining images. Our approach demonstrates efficacy across a range of tasks, including object attribute adjustments, error correction, and sequential prompt-based object placement and manipulation, demonstrating its versatility and potential for enhancing creative workflows., Comment: arXiv admin note: text overlap with arXiv:2306.00219

This paper focuses on reducing the communication cost of federated learning by exploring generalization bounds and representation learning. We first characterize a tighter generalization bound for one-round federated learning based on local clients' generalizations and heterogeneity of data distribution (non-iid scenario). We also characterize a generalization bound in R-round federated learning and its relation to the number of local updates (local stochastic gradient descents (SGDs)). Then, based on our generalization bound analysis and our representation learning interpretation of this analysis, we show for the first time that less frequent aggregations, hence more local updates, for the representation extractor (usually corresponds to initial layers) leads to the creation of more generalizable models, particularly for non-iid scenarios. We design a novel Federated Learning with Adaptive Local Steps (FedALS) algorithm based on our generalization bound and representation learning analysis. FedALS employs varying aggregation frequencies for different parts of the model, so reduces the communication cost. The paper is followed with experimental results showing the effectiveness of FedALS.

Elastic full-waveform inversion (EFWI) is a process used to estimate subsurface properties by fitting seismic data while satisfying wave propagation physics. The problem is formulated as a least-squares data fitting minimization problem with two sets of constraints: Partial-differential equation (PDE) constraints governing elastic wave propagation and physical model constraints implementing prior information. The alternating direction method of multipliers is used to solve the problem, resulting in an iterative algorithm with well-conditioned subproblems. Although wavefield reconstruction is the most challenging part of the iteration, sparse linear algebra techniques can be used for moderate-sized problems and frequency domain formulations. The Hessian matrix is blocky with diagonal blocks, making model updates fast. Gradient ascent is used to update Lagrange multipliers by summing PDE violations. Various numerical examples are used to investigate algorithmic components, including model parameterizations, physical model constraints, the role of the Hessian matrix in suppressing interparameter cross-talk, computational efficiency with the source sketching method, and the effect of noise and near-surface effects.

In this paper, a new method is given for counting cycles in the Tanner graph of a (Type-I) quasi-cyclic (QC) low-density parity-check (LDPC) code which the complexity mainly is dependent on the base matrix, independent from the CPM-size of the constructed code. Interestingly, for large CPM-sizes, in comparison of the existing methods, this algorithm is the first approach which efficiently counts the cycles in the Tanner graphs of QC-LDPC codes. In fact, the algorithm recursively counts the cycles in the parity-check matrix column-by-column by finding all non-isomorph tailless backtrackless closed (TBC) walks in the base graph and enumerating theoretically their corresponding cycles in the same equivalent class. Moreover, this approach can be modified in few steps to find the cycle distributions of a class of LDPC codes based on Affine permutation matrices (APM-LDPC codes). Interestingly, unlike the existing methods which count the cycles up to $2g-2$, where $g$ is the girth, the proposed algorithm can be used to enumerate the cycles of arbitrary length in the Tanner graph. Moreover, the proposed cycle searching algorithm improves upon various previously known methods, in terms of computational complexity and memory requirements., Comment: 18 pages, 4 figures

Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks. The code is available.

Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a Llama-2-7B student model. Our code is available at https://github.com/SqueezeAILab/LLM2LLM ., Comment: ACL 2024

The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training LLMs. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0x/2yrs, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. Here, we analyze encoder and decoder Transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation., Comment: Published in IEEE Micro Journal

High penetration of renewable generation in the electricity grid presents power system operators with challenges including voltage instability mainly due to fluctuating power generation. To cope with intermittent generation, community batteries introduce an elegant solution for storing excess generation of renewable resources and reverting to the grid in peak demand periods. The question of the right battery type and size coupled with the right investment is challenging. Furthermore, the growth in adapting EVs imposes additional demand challenges on the power system compared to traditional industrial and household demand. This paper introduces long-term planning for community batteries to capture the surplus generation of PV resources for a given area and redirect these resources to charge EVs, without direct injection to the grid. For long-term investment planning on batteries, we consider 15 years' worth of historical data associated with solar irradiance, temperature, EV demands, and household demands. A novel stochastic mathematical model is proposed for decision-making on battery specifications (the type and capacity per year) based on the four standard battery types provided by the CSIRO in Australia. Uncertainties related to the EVs and RESs are captured by a non-parametric robust technique, named information gap decision theory, from optimistic and pessimistic perspectives. The investment decision-making part is formulated as mixed-integer linear programming taking advantage of the powerful commercial solver -- GUROBI -- which leads to finding feasible global solutions with low computational burden. The outcomes of this investigation not only detect optimal battery installation strategies to improve the stability profile of the grid by capturing the excess generation of PV resources but also facilitate EV integration in the community toward reaching net-zero emissions targets.

Both energy-aware, batteryless intermittent systems and signal-aware adaptive sampling algorithms (ASA) aim to maximize sensor data accuracy under energy constraints in edge devices. Intuitively, combining both into a signal- & energy-aware solution would yield even better accuracy. Unfortunately, ASAs and intermittent systems rely on conflicting energy availability assumptions. So, a straightforward combination cannot achieve their combined benefits. Therefore, we propose SEASONS, the first framework for signal- and energy-aware intermittent systems. SEASONS buffers signal data in time, monitoring queue dynamics to ensure the data is reported within a user-specified latency constraint. SEASONS improves sensor data accuracy by 31% without increasing energy.

The high-luminosity upgrade of the LHC brings unprecedented requirements for real-time and precision bunch-by-bunch online luminosity measurement and beam-induced background monitoring. A key component of the CMS Beam Radiation, Instrumentation and Luminosity system is a stand-alone luminometer, the Fast Beam Condition Monitor (FBCM), which is fully independent from the CMS central trigger and data acquisition services and able to operate at all times with a triggerless readout. FBCM utilizes a dedicated front-end application-specific integrated circuit (ASIC) to amplify the signals from CO$_2$-cooled silicon-pad sensors with a timing resolution of a few nanoseconds, which enables the measurement of the beam-induced background. FBCM uses a modular design with two half-disks of twelve modules at each end of CMS, with four service modules placed close to the outer edge to reduce radiation-induced aging. The electronics system design adapts several components from the CMS Tracker for power, control and read-out functionalities. The dedicated FBCM23 ASIC contains six channels and adjustable shaping time to optimize the noise with regards to sensor leakage current. Each ASIC channel outputs a single binary high-speed asynchronous signal carrying time-of-arrival and time-over-threshold information. The chip output signal is digitized, encoded and sent via a radiation-hard gigabit transceiver and an optical link to the back-end electronics for analysis. This paper reports on the updated design of the FBCM detector and the ongoing testing program., Comment: 16th Topical Seminar on Innovative Particle and Radiation Detectors (IPRD23) 2023 Sept 25-29 Siena, Italy

LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference. Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision. Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods: (i) Per-Channel Key Quantization, where we adjust the dimension along which we quantize the Key activations to better match the distribution; (ii) Pre-RoPE Key Quantization, where we quantize Key activations before the rotary positional embedding to mitigate its impact on quantization; (iii) Non-Uniform KV Cache Quantization, where we derive per-layer sensitivity-weighted non-uniform datatypes that better represent the distributions; and (iv) Per-Vector Dense-and-Sparse Quantization, where we isolate outliers separately for each vector to minimize skews in quantization ranges. By applying our method to the LLaMA, Llama-2, Llama-3, and Mistral models, we achieve < 0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4, outperforming existing approaches. Our method enables serving LLaMA-7B with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system. We develop custom CUDA kernels for KVQuant, showing that we can achieve up to ~1.7x speedups, compared to baseline fp16 matrix-vector multiplications, for the LLaMA-7B model., Comment: NeurIPS 2024

Despite the revolutionary impact of AI and the development of locally trained algorithms, achieving widespread generalized learning from multi-modal data in medical AI remains a significant challenge. This gap hinders the practical deployment of scalable medical AI solutions. Addressing this challenge, our research contributes a self-supervised robust machine learning framework, OCT-SelfNet, for detecting eye diseases using optical coherence tomography (OCT) images. In this work, various data sets from various institutions are combined enabling a more comprehensive range of representation. Our method addresses the issue using a two-phase training approach that combines self-supervised pretraining and supervised fine-tuning with a mask autoencoder based on the SwinV2 backbone by providing a solution for real-world clinical deployment. Extensive experiments on three datasets with different encoder backbones, low data settings, unseen data settings, and the effect of augmentation show that our method outperforms the baseline model, Resnet-50 by consistently attaining AUC-ROC performance surpassing 77% across all tests, whereas the baseline model exceeds 54%. Moreover, in terms of the AUC-PR metric, our proposed method exceeded 42%, showcasing a substantial increase of at least 10% in performance compared to the baseline, which exceeded only 33%. This contributes to our understanding of our approach's potential and emphasizes its usefulness in clinical settings., Comment: 12 pages, 7 figures, 6 tables

This paper proposes an end-to-end framework for generating 3D human pose datasets using Neural Radiance Fields (NeRF). Public datasets generally have limited diversity in terms of human poses and camera viewpoints, largely due to the resource-intensive nature of collecting 3D human pose data. As a result, pose estimators trained on public datasets significantly underperform when applied to unseen out-of-distribution samples. Previous works proposed augmenting public datasets by generating 2D-3D pose pairs or rendering a large amount of random data. Such approaches either overlook image rendering or result in suboptimal datasets for pre-trained models. Here we propose PoseGen, which learns to generate a dataset (human 3D poses and images) with a feedback loss from a given pre-trained pose estimator. In contrast to prior art, our generated data is optimized to improve the robustness of the pre-trained model. The objective of PoseGen is to learn a distribution of data that maximizes the prediction error of a given pre-trained model. As the learned data distribution contains OOD samples of the pre-trained model, sampling data from such a distribution for further fine-tuning a pre-trained model improves the generalizability of the model. This is the first work that proposes NeRFs for 3D human data generation. NeRFs are data-driven and do not require 3D scans of humans. Therefore, using NeRF for data generation is a new direction for convenient user-specific data generation. Our extensive experiments show that the proposed PoseGen improves two baseline models (SPIN and HybrIK) on four datasets with an average 6% relative improvement.

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.

A novel method has been devised to compute the Local Integrals of Motion (LIOMs) for a one-dimensional many-body localized system. In this approach, a class of optimal unitary transformations is deduced in a tensor-network formalism to diagonalize the Hamiltonian of the specified system. To construct the tensor network, we utilize the eigenstates of the subsystems Hamiltonian to attain the desired unitary transformations. Subsequently, we optimize the eigenstates and acquire appropriate unitary localized operators that will represent the LIOMs tensor network. The efficiency of the method was assessed and found to be both fast and almost accurate. In framework of the introduced tensor-network representation, we examine how the entanglement spreads along the considered many-body localized system and evaluate the outcomes of the approximations employed in this approach. The important and interesting result is that in the proposed tensor network approximation, if the length of the blocks is greater than the length of localization, then the entropy growth will be linear in terms of the logarithmic time. Also, it has been demonstrated that, the entanglement can be calculated by only considering two blocks next to each other, if the Hamiltonian has been diagonalized using the unitary transformation made by the provided tensor-network representation.

The reasoning capabilities of the recent LLMs enable them to execute external function calls to overcome their inherent limitations, such as knowledge cutoffs, poor arithmetic skills, or lack of access to private data. This development has allowed LLMs to select and coordinate multiple functions based on the context to tackle more complex problems. However, current methods for function calling often require sequential reasoning and acting for each function which can result in high latency, cost, and sometimes inaccurate behavior. To address this, we introduce LLMCompiler, which executes functions in parallel to efficiently orchestrate multiple function calls. Drawing inspiration from the principles of classical compilers, LLMCompiler enables parallel function calling with three components: (i) a Function Calling Planner, formulating execution plans for function calling; (ii) a Task Fetching Unit, dispatching function calling tasks; and (iii) an Executor, executing these tasks in parallel. LLMCompiler automatically generates an optimized orchestration for the function calls and can be used with both open-source and closed-source models. We have benchmarked LLMCompiler on a range of tasks with different patterns of function calling. We observe consistent latency speedup of up to 3.7x, cost savings of up to 6.7x, and accuracy improvement of up to ~9% compared to ReAct. Our code is available at https://github.com/SqueezeAILab/LLMCompiler., Comment: ICML 2024

Full-waveform inversion (FWI) is an effective method for imaging subsurface properties using sparsely recorded data. It involves solving a wave propagation problem to estimate model parameters that accurately reproduce the data. Recent trends in FWI have led to the development of extended methodologies, among which source extension methods leveraging reconstructed wavefields to solve penalty or augmented Lagrangian (AL) formulations have emerged as robust algorithms, even for inaccurate initial models. Despite their demonstrated robustness, challenges remain, such as the lack of a clear physical interpretation, difficulty in comparison, and reliance on difficult-to-compute least squares (LS) wavefields. This paper is divided into two critical parts. In the first, a novel formulation of these methods is explored within a unified Lagrangian framework. This novel perspective permits the introduction of alternative algorithms that employ LS multipliers instead of wavefields. These multiplier-oriented variants appear as regularizations of the standard FWI, are adaptable to the time domain, offer tangible physical interpretations, and foster enhanced convergence efficiency. The second part of the paper delves into understanding the underlying mechanisms of these techniques. This is achieved by solving the FWI equations using iterative linearization and inverse scattering methods. The paper provides insight into the role and significance of Lagrange multipliers in enhancing the linearization of FWI equations. It explains how different methods estimate multipliers or make approximations to increase computing efficiency. Additionally, it presents a new physical understanding of the Lagrange multiplier used in the AL method, highlighting how important it is for improving algorithm performance when compared to penalty methods.

A novel wireless microstrip-based RF sensor designed for detecting changes in ionic content of water and the addition of solid contaminant objects is proposed and demonstrated. The sensor can be installed on the exterior wall of dielectric containers and customized according to the material of the container to enable wireless sensing. Its operation within the lower microwave frequency range (670-730 MHz) serves to minimize signal attenuation in water and streamlines circuitry design. The most significant feature of this sensor is its unique design, rendering it impervious to its surrounding environment. This not only shields it from environmental noise but also maximizes its sensitivity by efficiently utilizing incoming power for sensing purposes. The sensor exhibits remarkable sensitivity, capable of detecting solute concentrations as low as 0.003125 M in water. It can also detect the insertion of foreign solid objects into the container from the exterior wirelessly and distinguish them from liquids being added. As a proof-of-concept demonstration, the sensor in this study was optimized for a porcelain wall of 10-12 mm thickness. The sensor's small size and the materials used for its fabrication make it adaptable to a wide range of applications in industries such as food, pharmaceuticals, and bathroom fixtures. The aforementioned properties position the sensor as an ideal choice for various smart bathroom applications, where accurate and reliable water use monitoring is essential for efficient water conservation., Comment: The paper is submitted to IEEE Sensors Journal

In teleparallel geometries the coframe and corresponding spin-connection are the principal geometric objects and consequently the appropriate definition of a symmetry is that of an affine symmetry. The set of invariant coframes and their corresponding spin connections that respect the full six dimensional Lie algebra of Robertson-Walker affine symmetries are displayed and discussed. We will refer to such geometries as teleparallel Robertson-Walker (TRW) geometries, where the corresponding derived metric is of Robertson-Walker form and is characterized by the parameter $k = (-1,0,1)$. The field equations are explicitly presented for the $F(T)$ class of teleparallel TRW spacetimes. We are primarily interested in investigating the $k \neq 0$ TRW models. After first studying the $k=0$ models and, in particular, writing their governing field equations in an appropriate form, we then study their late time stability with respect to perturbations in $k$ in both the cases of a vanishing and non-vanishing effective cosmological constant term. As an illustration we consider both quadratic $F(T)$ theories and power-law solutions., Comment: 18 pages, no figure, Published in Universe MDPI

Spatially coupled low-density parity-check (SC-LDPC) codes are a class of capacity approaching LDPC codes with low message recovery latency when a sliding window decoding is used. In this paper, we first present a new method for the construction of a class of SC-LDPC codes by the incidence matrices of a given non-negative integer matrix $E$, and then the relationship of 4-cycles between matrix $E$ and the corresponding SC-LDPC code are investigated. Finally, by defining a new class of integer finite sequences, called {\it good sequences}, for the first time, we give an explicit method for the construction of a class of 4-cycle free SC-LDPC codes that can achieve (in most cases) the minimum coupling width.

Generative Large Language Models (LLMs) based on the Transformer architecture have recently emerged as a dominant foundation model for a wide range of Natural Language Processing tasks. Nevertheless, their application in real-time scenarios has been highly restricted due to the significant inference latency associated with these models. This is particularly pronounced due to the autoregressive nature of generative LLM inference, where tokens are generated sequentially since each token depends on all previous output tokens. It is therefore challenging to achieve any token-level parallelism, making inference extremely memory-bound. In this work, we propose SPEED, which improves inference efficiency by speculatively executing multiple future tokens in parallel with the current token using predicted values based on early-layer hidden states. For Transformer decoders that employ parameter sharing, the memory operations for the tokens executing in parallel can be amortized, which allows us to accelerate generative LLM inference. We demonstrate the efficiency of our method in terms of latency reduction relative to model accuracy and demonstrate how speculation allows for training deeper decoders with parameter sharing with minimal runtime overhead., Comment: NeurIPS Workshop on Efficient Natural Language and Speech Processing (2023)

Locomotive soft robots (SoRos) have gained prominence due to their adaptability. Traditional locomotive SoRo design is based on limb structures inspired by biological organisms and requires human intervention. Evolutionary robotics, designed using evolutionary algorithms (EAs), have shown potential for automatic design. However, EA-based methods face the challenge of high computational cost when considering multiphysics in locomotion, including materials, actuations, and interactions with environments. Here, we present a design approach for pneumatic SoRos that integrates gradient-based topology optimization with multiphysics material point method (MPM) simulations. This approach starts with a simple initial shape (a cube with a central cavity). The topology optimization with MPM then automatically and iteratively designs the SoRo shape. We design two SoRos, one for walking and one for climbing. These SoRos are 3D printed and exhibit the same locomotion features as in the simulations. This study presents an efficient strategy for designing SoRos, demonstrating that a purely mathematical process can produce limb-like structures seen in biological organisms., Comment: 36 total pages (27 pages, 9 supplementary pages), 5 Figures, 9 Supplementary figures. 1 Supplementary table

Natural Language Processing (NLP) has undergone transformative changes with the advent of deep learning methodologies. One challenge persistently confronting researchers is the scarcity of high-quality, annotated datasets that drive these models. This paper explores the nuances of synthetic data generation in NLP, with a focal point on template-based question generation. By assessing its advantages, including data augmentation potential and the introduction of structured variety, we juxtapose these benefits against inherent limitations, such as the risk of overfitting and the constraints posed by pre-defined templates. Drawing from empirical evaluations, we demonstrate the impact of template-based synthetic data on the performance of modern transformer models. We conclude by emphasizing the delicate balance required between synthetic and real-world data, and the future trajectories of integrating synthetic data in model training pipelines. The findings aim to guide NLP practitioners in harnessing synthetic data's potential, ensuring optimal model performance in diverse applications.

Raman scattering of photons into phonons gives rise to entangled photon pairs when the phonon emitted in a Stokes process is coherently absorbed in antiStokes scattering, forming the photonic analog of Cooper pairs. We present a nonperturbative theory for the time evolution of photonic Cooper pairs that treats interacting photons and phonons as a hybrid excitation, the Ramaniton. As the Ramaniton propagates in a wave guide it displays quantum oscillations between photon and phonon occupation, leading to resonant squeezed Stokes-antiStokes light when the phonon occupation becomes equal to zero without recurring back to the photon vacuum. This phenomenon is predicted to generate up to 28 dB of squeezed light even in standard silicon on insulator waveguides., Comment: Accepted for publication in Phys. Rev. Research

Transformer models have revolutionized natural language processing with their unparalleled ability to grasp complex contextual relationships. However, the vast number of parameters in these models has raised concerns regarding computational efficiency, environmental impact, and deployability on resource-limited platforms. To address these challenges, this paper investigates the application of weight pruning-a strategic reduction of model parameters based on their significance-as an optimization strategy for Transformer architectures. Through extensive experimentation, we explore various pruning methodologies, highlighting their impact on model performance, size, and computational demands. Our findings suggest that with judicious selection of pruning hyperparameters, significant reductions in model size are attainable without considerable compromise on performance. Moreover, when coupled with post-pruning fine-tuning strategies, some pruned models even exhibit enhanced generalization capabilities. This work seeks to bridge the gap between model efficiency and performance, paving the way for more scalable and environmentally responsible deep learning applications.

The burgeoning complexity of contemporary deep learning models, while achieving unparalleled accuracy, has inadvertently introduced deployment challenges in resource-constrained environments. Knowledge distillation, a technique aiming to transfer knowledge from a high-capacity "teacher" model to a streamlined "student" model, emerges as a promising solution to this dilemma. This paper provides a comprehensive overview of the knowledge distillation paradigm, emphasizing its foundational principles such as the utility of soft labels and the significance of temperature scaling. Through meticulous examination, we elucidate the critical determinants of successful distillation, including the architecture of the student model, the caliber of the teacher, and the delicate balance of hyperparameters. While acknowledging its profound advantages, we also delve into the complexities and challenges inherent in the process. Our exploration underscores knowledge distillation's potential as a pivotal technique in optimizing the trade-off between model performance and deployment efficiency.

This paper presents novel systems and methodologies for the development of efficient large language models (LLMs). It explores the trade-offs between model size, performance, and computational resources, with the aim of maximizing the efficiency of these AI systems. The research explores novel methods that allow different parts of the model to share parameters, reducing the total number of unique parameters required. This approach ensures that the model remains compact without sacrificing its ability to learn and represent complex language structures. This study provides valuable insights and tools for creating more efficient and effective LLMs, contributing to a more sustainable and accessible future for AI language modeling.