Search

GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.

Rogue waves are known to occur on the ocean surface leading to significant damage to marine installations and compromising ship safety. Understanding the physical mechanisms responsible for extreme wave focusing is crucial in order to predict and prevent their formation and impact. Two intensively discussed wave amplification frameworks are the linear and nonlinear focusing mechanisms. These are also known as superposition principle and modulation instability, respectively. We report an experimental study investigating the formation mechanism in a unidirectional representative JONSWAP-type sea state and show that the nonlinear focusing mechanism can be sub-categorized into either a localized amplitude or a so far less-studied phase-related frequency modulation, or both being at play. The frequency modulation-type mechanism occurs at a lower probability, as suggested from the distribution of more than 200 extreme events, however, it cannot be underrated or disregarded.

Compared to classical optical coherence theory based on Maxwell's electromagnetic theory and Glauber's quantum optical coherence theory based on matrix mechanics formulation of quantum mechanics, quantum optical coherence theory based on Feynman's path integral formulation of quantum mechanics provides a novel tool to study optical coherence. It has the advantage of understanding the connection between mathematical calculations and physical interpretations better. Quantum optical coherence theory based on Feynman's path integral is introduced and reviewed in this paper. Based on the results of transient first-order interference of two independent light beams, it is predicted that the classical model for electric field of thermal light introduced by classical optical textbooks may not be accurate. The physics of two-photon bunching of thermal light and Hong-Ou-Mandel dip of entangled photon pairs is the same, which can be interpreted by constructive and destructive two-photon interference, respectively. Quantum optical coherence theory based on Feynman's path integral is helpful to understand the coherence properties of light, which may eventually lead us to the answer of the question: what is a photon?, Comment: 40 pages, 35 figures

Remote heart rate measurement is an increasingly concerned research field, usually using remote photoplethysmography (rPPG) to collect heart rate information through video data collection. However, in certain specific scenarios (such as low light conditions, intense lighting, and non-line-of-sight situations), traditional imaging methods fail to capture image information effectively, that may lead to difficulty or inability in measuring heart rate. To address these limitations, this study proposes using ghost imaging as a substitute for traditional imaging in the aforementioned scenarios. The mean absolute error between experimental measurements and reference true values is 4.24 bpm.Additionally, the bucket signals obtained by the ghost imaging system can be directly processed using digital signal processing techniques, thereby enhancing personal privacy protection., Comment: 4 pages, 6 figures

We study the stochastic multi-armed bandit problem in the $P$-pass streaming model. In this problem, the $n$ arms are present in a stream and at most $m

The modulation instability (MI) is responsible for the disintegration of a regular nonlinear wave train and can lead to strong localizations in a from of rogue waves. This mechanism has been studied in a variety of nonlinear dispersive media, such as hydrodynamics, optics, plasma, mechanical systems, electric transmission lines, and Bose-Einstein condensates, while its impact on applied sciences is steadily growing. Following the linear stability analysis of weakly nonlinear waves, the classical MI dynamics, can be triggered when a pair of small-amplitude sidebands are excited within a particular frequency range around the main peak frequency. That is, a three-wave system is usually required to initiate the wave focusing process. Breather solutions of the nonlinear Schr\"odinger equation (NLSE) revealed that MI can generate much more complex localized structures, beyond the three-wave system initialization approach or by means of a continuous spectrum. In this work, we report an experimental study for deep-water surface gravity waves asserting that a MI process can be triggered by a single unstable sideband only, and thus, from a two-wave process when including the contribution of the peak frequency. The experimental data are validated against fully nonlinear hydrodynamic numerical wave tank simulations and show very good agreement. The long-term evolution of such unstable wave trains shows a distinct shift in the recurrent Fermi-Pasta-Ulam-Tsingou focusing cycles, which are captured by the NLSE and fully nonlinear hydrodynamic simulations with minor distinctions.

Rogue waves (RWs) can form on the ocean surface due to quasi-four wave resonant interaction or superposition principle. Both mechanisms have been acutely studied. The first of the two is known as the nonlinear focusing mechanism and leads to an increased probability of rogue waves when wave conditions are favourable, i.e., when unidirectionality and high narrowband energy of the wave field are satisfied. This work delves into the dynamics of extreme wave focusing in crossing seas, revealing a distinct type of nonlinear RWs, characterized by a decisive longevity compared to those generated by the dispersive focusing mechanism. In fact, through fully nonlinear hydrodynamic numerical simulations, we show that the interactions between two crossing unidirectional wave beams can trigger fully localized and robust development of RWs. These coherent structures, characterized by a typical spectral broadening then spreading in the form of dual bimodality and recurrent wave group focusing, not only defy the weakening expectation of quasi-four wave resonant interaction in directionally spread wave fields, but also differ from classical focusing mechanisms already mentioned. This has been determined following a rigorous lifespan-based statistical analysis of extreme wave events in our fully nonlinear simulations. Utilizing the coupled nonlinear Schr\"odinger framework, we also show that such intrinsic focusing dynamics can also be captured by weakly nonlinear wave evolution equations. This opens new research avenues for further explorations of these complex and intriguing wave phenomena in hydrodynamics as well as other nonlinear and dispersive multi-wave systems.

The immense popularity of racket sports has fueled substantial demand in tactical analysis with broadcast videos. However, existing manual methods require laborious annotation, and recent attempts leveraging video perception models are limited to low-level annotations like ball trajectories, overlooking tactics that necessitate an understanding of stroke techniques. State-of-the-art action segmentation models also struggle with technique recognition due to frequent occlusions and motion-induced blurring in racket sports videos. To address these challenges, We propose ViSTec, a Video-based Sports Technique recognition model inspired by human cognition that synergizes sparse visual data with rich contextual insights. Our approach integrates a graph to explicitly model strategic knowledge in stroke sequences and enhance technique recognition with contextual inductive bias. A two-stage action perception model is jointly trained to align with the contextual knowledge in the graph. Experiments demonstrate that our method outperforms existing models by a significant margin. Case studies with experts from the Chinese national table tennis team validate our model's capacity to automate analysis for technical actions and tactical strategies. More details are available at: https://ViSTec2024.github.io/., Comment: accepted by AAAI-24 Main Track

Learning with expert advice and multi-armed bandit are two classic online decision problems which differ on how the information is observed in each round of the game. We study a family of problems interpolating the two. For a vector $\mathbf{m}=(m_1,\dots,m_K)\in \mathbb{N}^K$, an instance of $\mathbf{m}$-MAB indicates that the arms are partitioned into $K$ groups and the $i$-th group contains $m_i$ arms. Once an arm is pulled, the losses of all arms in the same group are observed. We prove tight minimax regret bounds for $\mathbf{m}$-MAB and design an optimal PAC algorithm for its pure exploration version, $\mathbf{m}$-BAI, where the goal is to identify the arm with minimum loss with as few rounds as possible. We show that the minimax regret of $\mathbf{m}$-MAB is $\Theta\left(\sqrt{T\sum_{k=1}^K\log (m_k+1)}\right)$ and the minimum number of pulls for an $(\epsilon,0.05)$-PAC algorithm of $\mathbf{m}$-BAI is $\Theta\left(\frac{1}{\epsilon^2}\cdot \sum_{k=1}^K\log (m_k+1)\right)$. Both our upper bounds and lower bounds for $\mathbf{m}$-MAB can be extended to a more general setting, namely the bandit with graph feedback, in terms of the clique cover and related graph parameters. As consequences, we obtained tight minimax regret bounds for several families of feedback graphs.

Auction-based recommender systems are prevalent in online advertising platforms, but they are typically optimized to allocate recommendation slots based on immediate expected return metrics, neglecting the downstream effects of recommendations on user behavior. In this study, we employ reinforcement learning to optimize for long-term return metrics in an auction-based recommender system. Utilizing temporal difference learning, a fundamental reinforcement learning algorithm, we implement an one-step policy improvement approach that biases the system towards recommendations with higher long-term user engagement metrics. This optimizes value over long horizons while maintaining compatibility with the auction framework. Our approach is grounded in dynamic programming ideas which show that our method provably improves upon the existing auction-based base policy. Through an online A/B test conducted on an auction-based recommender system which handles billions of impressions and users daily, we empirically establish that our proposed method outperforms the current production system in terms of long-term user engagement metrics.

We propose an effective and robust algorithm for identifying partial differential equations (PDEs) with space-time varying coefficients from a single trajectory of noisy observations. Identifying unknown differential equations from noisy observations is a difficult task, and it is even more challenging with space and time varying coefficients in the PDE. The proposed algorithm, GP-IDENT, has three ingredients: (i) we use B-spline bases to express the unknown space and time varying coefficients, (ii) we propose Group Projected Subspace Pursuit (GPSP) to find a sequence of candidate PDEs with various levels of complexity, and (iii) we propose a new criterion for model selection using the Reduction in Residual (RR) to choose an optimal one among the pool of candidates. The new GPSP considers group projected subspaces which is more robust than existing methods in distinguishing correlated group features. We test GP-IDENT on a variety of PDEs and PDE systems, and compare it with the state-of-the-art parametric PDE identification algorithms under different settings to illustrate its outstanding performance. Our experiments show that GP-IDENT is effective in identifying the correct terms from a large dictionary and the model selection scheme is robust to noise.

We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4., Comment: 100 pages; updated authors list; fixed author names and added citation

Optical phased arrays (OPA) with high integration, fast speed, low power consumption, and high steering resolution are critical components in the emerging photonic integrated circuit (PIC), LiDAR, free space optical communication, 3D printing, and so on. According to the OPA working principle, its function is generally achieved by independently controlling the phase of the array elements. In practice, this presents a major challenge to overcome critical trade-offs of the element numbers, the control electronics, and the power consumption. Here, we give an alternative OPA solution to overcome this technical limitation, in the form of a cascaded periodically poled electro-optical crystal structure. Compared with the existing OPA scheme, only one control electronics is used to control the entire array elements in the current proposal, regardless of the number of array elements, implying higher integration and lower power consumption. With the help of the fast response properties of electro-optic crystal materials, a high-speed and high-resolution beam steering device is demonstrated. Simulating results show that the angular resolution can be improved by several orders of magnitude when the number of the cascaded-layer increases. An OPA prototype of a 6-layer cascaded periodically poled LiNbO$_3$ (cascaded-PPLN) was designed, fabricated, and characterized. The experiment that observed beam deflection in cascaded-PPLN OPA agrees well with the simulation results. Meantime, by demonstrating dynamic beam steering continually, its capability of continuous scanning and continually active phase tunability has been verified. Therefore, this cascaded periodically poled electro-optical crystal OPA offers a feasible direction of miniaturization and low power consumption for the optical system, such as the PIC system.

Due to the complex of mixed spectral point spread function within memory effect range, it is unreliable and slow to use speckle correlation technology for non-invasive imaging through scattering medium under broadband illumination. The contrast of the speckles will drastically drop as the light source's spectrum width increases. Here, we propose a method for producing the optical transfer function with several speckle frames within memory effect range to image under broadband illumination. The method can be applied to image amplitude and color objects under white LED illumination. Compared to other approaches of imaging under broadband illumination, such as deep learning and modified phase retrieval, our method can provide more stable results with faster convergence speed, which can be applied in high speed scattering imaging under natural light illumination.

The rise of deep neural networks offers new opportunities in optimizing recommender systems. However, optimizing recommender systems using deep neural networks requires delicate architecture fabrication. We propose NASRec, a paradigm that trains a single supernet and efficiently produces abundant models/sub-architectures by weight sharing. To overcome the data multi-modality and architecture heterogeneity challenges in the recommendation domain, NASRec establishes a large supernet (i.e., search space) to search the full architectures. The supernet incorporates versatile choice of operators and dense connectivity to minimize human efforts for finding priors. The scale and heterogeneity in NASRec impose several challenges, such as training inefficiency, operator-imbalance, and degraded rank correlation. We tackle these challenges by proposing single-operator any-connection sampling, operator-balancing interaction modules, and post-training fine-tuning. Our crafted models, NASRecNet, show promising results on three Click-Through Rates (CTR) prediction benchmarks, indicating that NASRec outperforms both manually designed models and existing NAS methods with state-of-the-art performance. Our work is publicly available at https://github.com/facebookresearch/NasRec., Comment: Proceedings of the ACM Web Conference 2023 (WWW'23)

Aggregation equations are broadly used to model population dynamics with nonlocal interactions, characterized by a potential in the equation. This paper considers the inverse problem of identifying the potential from a single noisy spatial-temporal process. The identification is challenging in the presence of noise due to the instability of numerical differentiation. We propose a robust model-based technique to identify the potential by minimizing a regularized data fidelity term, and regularization is taken as the total variation and the squared Laplacian. A split Bregman method is used to solve the regularized optimization problem. Our method is robust to noise by utilizing a Successively Denoised Differentiation technique. We consider additional constraints such as compact support and symmetry constraints to enhance the performance further. We also apply this method to identify time-varying potentials and identify the interaction kernel in an agent-based system. Various numerical examples in one and two dimensions are included to verify the effectiveness and robustness of the proposed method.

The modulation instability is a focusing mechanism responsible for the formation of strong wave localizations not only on the water surface, but also in a variety of nonlinear dispersive media. Such dynamics is initiated from the injection of side-bands, which translate into an amplitude modulation of the wave field. The nonlinear stage of unstable wave evolution can be described by exact solutions of the nonlinear Schr\"odinger equation (NLSE). In that case, the amplitude modulation of such coherent extreme wave structures is connected to a particular phase-shift seed in the carrier wave. In this letter, we show that phase-shifts localization applied to the background, excluding any amplitude modulation excitation, can indeed trigger extreme events. Such rogue waves can be for instance generated by considering the parametrization of fundamental breathers and thus, by seeding only the local phase-shift information to the regular carrier wave. Our wave tank experiments show an excellent agreement with the expected NLSE hydrodynamics and confirm that even though delayed in their evolution, breather-type extreme waves can be generated from a purely regular wave train. Such novel focusing mechanism awaits experimental confirmation in other nonlinear media, such optics, plasma, and Bose-Einstein condensates.

The problem of bandit with graph feedback generalizes both the multi-armed bandit (MAB) problem and the learning with expert advice problem by encoding in a directed graph how the loss vector can be observed in each round of the game. The mini-max regret is closely related to the structure of the feedback graph and their connection is far from being fully understood. We propose a new algorithmic framework for the problem based on a partition of the feedback graph. Our analysis reveals the interplay between various parts of the graph by decomposing the regret to the sum of the regret caused by small parts and the regret caused by their interaction. As a result, our algorithm can be viewed as an interpolation and generalization of the optimal algorithms for MAB and learning with expert advice. Our framework unifies previous algorithms for both strongly observable graphs and weakly observable graphs, resulting in improved and optimal regret bounds on a wide range of graph families including graphs of bounded degree and strongly observable graphs with a few corrupted arms.

In this work we study the problem about learning a partial differential equation (PDE) from its solution data. PDEs of various types are used as examples to illustrate how much the solution data can reveal the PDE operator depending on the underlying operator and initial data. A data driven and data adaptive approach based on local regression and global consistency is proposed for stable PDE identification. Numerical experiments are provided to verify our analysis and demonstrate the performance of the proposed algorithms., Comment: 44 pages

Nonlinear wave focusing originating from the universal modulation instability (MI) is responsible for the formation of strong wave localizations on the water surface and in nonlinear wave guides, such as optical Kerr media and plasma. Such extreme wave dynamics can be described by breather solutions of the nonlinear Schr\"odinger equation (NLSE) like by way of example the famed doubly-localized Peregrine breathers (PB), which typify particular cases of MI. On the other hand, it has been suggested that the MI relevance weakens when the wave field becomes broadband or directional. Here, we provide experimental evidence of nonlinear and distinct PB-type focusing in standing water waves describing the scenario of two counter-propagating wave trains. The collected collinear wave measurements are in excellent agreement with the hydrodynamic coupled NLSE (CNLSE) and suggest that MI can undisturbedly prevail during the interplay of several wave systems and emphasize the potential role of exact NLSE solutions in extreme wave formation beyond the formal narrowband and uni-directional limits. Our work may inspire further experimental investigations in various nonlinear wave guides governed by CNLSE frameworks as well as theoretical progress to predict strong wave coherence in directional fields.

The Galilean transformation is a universal operation connecting the coordinates of a dynamical system, which move relative to each other with a constant speed. In the context of exact solutions of the universal nonlinear Schr\"odinger equation (NLSE), inducing a Galilean velocity (GV) to the pulse involves a frequency shift to satisfy the symmetry of the wave equation. As such, the Galilean transformation has been deemed to be not applicable to wave groups in nonlinear dispersive media. In this paper, we demonstrate that in a wave tank generated Galilean transformed envelope and Peregrine solitons show clear variations from their respective pure dynamics on the water surface. The type of deviations depends on the sign of the GV and can be captured by the modified NLSE or the Euler equations. Moreover, we show that positive Galilean-translated envelope soliton pulses exhibit self-modulation. While designated GS and wave steepness values expedite multi-soliton dynamics, the strong focusing of such higher-order coherent waves inevitably lead to the generation of supercontinua as a result of soliton fission. We anticipate that kindred experimental and numerical studies might be implemented in other dispersive wave guides governed by nonlinearity.

Stronger light intensity fluctuations are pursued by related applications such as optical resolution, image enhancement, and beam positioning. In this paper, an Nth-order light intensity fluctuation amplifier is proposed, which was demonstrated by a four-wave mixing process with different statistical distribution coupling lights. Firstly, its amplification mechanism is revealed both theoretically and experimentally. The ratio $R$ of statistical distributions and the degree of second-order coherence ${g^{(2)}}(0)$ of beams are used to characterize the affected modulations and the increased light intensity fluctuations through the four-wave mixing process. The results show that the amplification of light intensity fluctuations is caused by not only the fluctuating light fields of incident coupling beams, but also the fluctuating nonlinear coefficient of interaction. At last, we highlight the potentiality of applying such amplifier to other N-order nonlinear optical effects.

Benefit from the promising features of second-order correlation, ghost imaging (GI) has received extensive attentions in recent years. Simultaneously, GI is affected by the poor trade-off between sampling rate and imaging quality. The traditional image reconstruction method in GI is to accumulate the action result of each speckle and the corresponding bucket signal. We found that the image reconstruction process of GI is very similar to the Recurrent Neural Network (RNN), which is one of the deep learning algorithm. In this paper, we proposed a novel method that effectively implements GI on the RNN architecture, called GI-RNN. The state of each layer in RNN is determined by the output of the previous layer and the input of this layer, and the output of the network is the sum of all previous states. Therefore, we take the speckle of each illumination and the corresponding bucket signal as the input of each layer, and the output of the network is the sum of all previous speckle and bucket signal, which is the image of the target. The testing results show that the proposed method can achieve image reconstruction at a very low sampling rate (0.38$\%$). Moreover, we compare GI-RNN with traditional GI algorithm and compressed sensing algorithm. The results of different targets show that GI-RNN is 6.61 dB higher than compressed sensing algorithm and 12.58 dB higher than traditional GI algorithm on average. In our view, the proposed method makes an important step to applications of GI.

Multi-party dialogue machine reading comprehension (MRC) raises an even more challenging understanding goal on dialogue with more than two involved speakers, compared with the traditional plain passage style MRC. To accurately perform the question-answering (QA) task according to such multi-party dialogue, models have to handle fundamentally different discourse relationships from common non-dialogue plain text, where discourse relations are supposed to connect two far apart utterances in a linguistics-motivated way.To further explore the role of such unusual discourse structure on the correlated QA task in terms of MRC, we propose the first multi-task model for jointly performing QA and discourse parsing (DP) on the multi-party dialogue MRC task. Our proposed model is evaluated on the latest benchmark Molweni, whose results indicate that training with complementary tasks indeed benefits not only QA task, but also DP task itself. We further find that the joint model is distinctly stronger when handling longer dialogues which again verifies the necessity of DP in the related MRC., Comment: Accepted by PACLIC 2021

The potential diagnostic applications of magnet-actuated capsules have been greatly increased in recent years. For most of these potential applications, accurate position control of the capsule have been highly demanding. However, the friction between the robot and the environment as well as the drag force from the tether play a significant role during the motion control of the capsule. Moreover, these forces especially the friction force are typically hard to model beforehand. In this paper, we first designed a magnet-actuated tethered capsule robot, where the driving magnet is mounted on the end of a robotic arm. Then, we proposed a learning-based approach to model the friction force between the capsule and the environment, with the goal of increasing the control accuracy of the whole system. Finally, several real robot experiments are demonstrated to showcase the effectiveness of our proposed approach., Comment: Because it overlaps with the previous article arvix:2108.07151, we apply for return.Thank you

With the potential applications of capsule robots in medical endoscopy, accurate dynamic control of the capsule robot is becoming more and more important. In the scale of a capsule robot, the friction between capsule and the environment plays an essential role in the dynamic model, which is usually difficult to model beforehand. In the paper, a tethered capsule robot system driven by a robot manipulator is built, where a strong magnetic Halbach array is mounted on the robot's end-effector to adjust the state of the capsule. To increase the control accuracy, the friction between capsule and the environment is learned with demonstrated trajectories. With the learned friction model, experimental results demonstrate an improvement of 5.6% in terms of tracking error., Comment: ICRAE 2021 Conference paper

Differing from the traditional method of achieving subwavelength interference, we have demonstrated the two-photon subwavelength interference effect of broadband chaotic light in a polarization-selective Michelson interferometer with an ultrafast two-photon absorption detector the first time, which is achieved by manipulating two-photon probability amplitudes involved in the interference. In theory, the two-photon polarization coherence matrix and probability amplitudes matrix are combined to develop polarized two-photon interference terms, which explains the experimental results well. In order to make better use of this interferometer to produce the subwavelength effect, we also make a series of error analyses to find out the relationship between the visibility and the degree of polarization error. Our experimental and theoretical results are helpful to understand the two-photon subwavelength interference, which sheds light on the development of the two-photon interference theory of vector light field based on quantum mechanics. These experimental results may help to develop future optical interferometry, optical polarimetry, and subwavelength lithography., Comment: 10 pages, 5 figures