Search

Despite the recognized potential of multimodal data to improve model accuracy, many large-scale industrial recommendation systems, including Taobao display advertising system, predominantly depend on sparse ID features in their models. In this work, we explore approaches to leverage multimodal data to enhance the recommendation accuracy. We start from identifying the key challenges in adopting multimodal data in a manner that is both effective and cost-efficient for industrial systems. To address these challenges, we introduce a two-phase framework, including: 1) the pre-training of multimodal representations to capture semantic similarity, and 2) the integration of these representations with existing ID-based models. Furthermore, we detail the architecture of our production system, which is designed to facilitate the deployment of multimodal representations. Since the integration of multimodal representations in mid-2023, we have observed significant performance improvements in Taobao display advertising system. We believe that the insights we have gathered will serve as a valuable resource for practitioners seeking to leverage multimodal data in their systems., Comment: Accepted at CIKM 2024

The dielectric loss of silicon nitride (Si$_3$N$_4$) limits the performance of microwave-frequency devices that rely on this material for sensing, signal processing, and quantum communication. Using superconducting resonant circuits, we measure the cryogenic loss tangent of either as-deposited or high-temperature annealed stoichiometric Si$_3$N$_4$ as a function of drive strength and temperature. The internal loss behavior of the electrical resonators is largely consistent with the standard tunneling model of two-level systems (TLS), including damping caused by resonant energy exchange with TLS and by the relaxation of non-resonant TLS. We further supplement the TLS model with a self-heating effect to explain an increase in the loss observed in as-deposited films at large drive powers. Critically, we demonstrate that annealing remedies this anomalous power-induced loss, reduces the relaxation-type damping by more than two orders of magnitude, and reduces the resonant-type damping by a factor of three. Employing infrared absorption spectroscopy, we find that annealing reduces the concentration of hydrogen in the Si$_3$N$_4$, suggesting that hydrogen impurities cause substantial dissipation., Comment: 11 pages, 7 figures

In machine learning systems, privileged features refer to the features that are available during offline training but inaccessible for online serving. Previous studies have recognized the importance of privileged features and explored ways to tackle online-offline discrepancies. A typical practice is privileged features distillation (PFD): train a teacher model using all features (including privileged ones) and then distill the knowledge from the teacher model using a student model (excluding the privileged features), which is then employed for online serving. In practice, the pointwise cross-entropy loss is often adopted for PFD. However, this loss is insufficient to distill the ranking ability for CTR prediction. First, it does not consider the non-i.i.d. characteristic of the data distribution, i.e., other items on the same page significantly impact the click probability of the candidate item. Second, it fails to consider the relative item order ranked by the teacher model's predictions, which is essential to distill the ranking ability. To address these issues, we first extend the pointwise-based PFD to the listwise-based PFD. We then define the calibration-compatible property of distillation loss and show that commonly used listwise losses do not satisfy this property when employed as distillation loss, thus compromising the model's calibration ability, which is another important measure for CTR prediction. To tackle this dilemma, we propose Calibration-compatible LIstwise Distillation (CLID), which employs carefully-designed listwise distillation loss to achieve better ranking ability than the pointwise-based PFD while preserving the model's calibration ability. We theoretically prove it is calibration-compatible. Extensive experiments on public datasets and a production dataset collected from the display advertising system of Alibaba further demonstrate the effectiveness of CLID., Comment: This paper has been accepted by WSDM'24

Conversion rate (CVR) prediction is an essential task for large-scale e-commerce platforms. However, refund behaviors frequently occur after conversion in online shopping systems, which drives us to pay attention to effective conversion for building healthier shopping services. This paper defines the probability of item purchasing without any subsequent refund as an effective conversion rate (ECVR). A simple paradigm for ECVR prediction is to decompose it into two sub-tasks: CVR prediction and post-conversion refund rate (RFR) prediction. However, RFR prediction suffers from data sparsity (DS) and sample selection bias (SSB) issues, as the refund behaviors are only available after user purchase. Furthermore, there is delayed feedback in both conversion and refund events and they are sequentially dependent, named cascade delayed feedback (CDF), which significantly harms data freshness for model training. Previous studies mainly focus on tackling DS and SSB or delayed feedback for a single event. To jointly tackle these issues in ECVR prediction, we propose an Entire space CAscade Delayed feedback modeling (ECAD) method. Specifically, ECAD deals with DS and SSB by constructing two tasks including CVR prediction and conversion \& refund rate (CVRFR) prediction using the entire space modeling framework. In addition, it carefully schedules auxiliary tasks to leverage both conversion and refund time within data to alleviate CDF. Experimental results on the offline industrial dataset and online A/B testing demonstrate the effectiveness of ECAD. In addition, ECAD has been deployed in one of the recommender systems in Alibaba, contributing to a significant improvement of ECVR., Comment: Accepted to CIKM'23

Cascading architecture has been widely adopted in large-scale advertising systems to balance efficiency and effectiveness. In this architecture, the pre-ranking model is expected to be a lightweight approximation of the ranking model, which handles more candidates with strict latency requirements. Due to the gap in model capacity, the pre-ranking and ranking models usually generate inconsistent ranked results, thus hurting the overall system effectiveness. The paradigm of score alignment is proposed to regularize their raw scores to be consistent. However, it suffers from inevitable alignment errors and error amplification by bids when applied in online advertising. To this end, we introduce a consistency-oriented pre-ranking framework for online advertising, which employs a chunk-based sampling module and a plug-and-play rank alignment module to explicitly optimize consistency of ECPM-ranked results. A $\Delta NDCG$-based weighting mechanism is adopted to better distinguish the importance of inter-chunk samples in optimization. Both online and offline experiments have validated the superiority of our framework. When deployed in Taobao display advertising system, it achieves an improvement of up to +12.3\% CTR and +5.6\% RPM.

Conversion rate (CVR) prediction is one of the core components in online recommender systems, and various approaches have been proposed to obtain accurate and well-calibrated CVR estimation. However, we observe that a well-trained CVR prediction model often performs sub-optimally during sales promotions. This can be largely ascribed to the problem of the data distribution shift, in which the conventional methods no longer work. To this end, we seek to develop alternative modeling techniques for CVR prediction. Observing similar purchase patterns across different promotions, we propose reusing the historical promotion data to capture the promotional conversion patterns. Herein, we propose a novel \textbf{H}istorical \textbf{D}ata \textbf{R}euse (\textbf{HDR}) approach that first retrieves historically similar promotion data and then fine-tunes the CVR prediction model with the acquired data for better adaptation to the promotion mode. HDR consists of three components: an automated data retrieval module that seeks similar data from historical promotions, a distribution shift correction module that re-weights the retrieved data for better aligning with the target promotion, and a TransBlock module that quickly fine-tunes the original model for better adaptation to the promotion mode. Experiments conducted with real-world data demonstrate the effectiveness of HDR, as it improves both ranking and calibration metrics to a large extent. HDR has also been deployed on the display advertising system in Alibaba, bringing a lift of $9\%$ RPM and $16\%$ CVR during Double 11 Sales in 2022., Comment: Accepted at KDD 2023. This work has already been deployed on the display advertising system in Alibaba, bringing substantial economic gains

Deep learning techniques have been applied widely in industrial recommendation systems. However, far less attention has been paid to the overfitting problem of models in recommendation systems, which, on the contrary, is recognized as a critical issue for deep neural networks. In the context of Click-Through Rate (CTR) prediction, we observe an interesting one-epoch overfitting problem: the model performance exhibits a dramatic degradation at the beginning of the second epoch. Such a phenomenon has been witnessed widely in real-world applications of CTR models. Thereby, the best performance is usually achieved by training with only one epoch. To understand the underlying factors behind the one-epoch phenomenon, we conduct extensive experiments on the production data set collected from the display advertising system of Alibaba. The results show that the model structure, the optimization algorithm with a fast convergence rate, and the feature sparsity are closely related to the one-epoch phenomenon. We also provide a likely hypothesis for explaining such a phenomenon and conduct a set of proof-of-concept experiments. We hope this work can shed light on future research on training more epochs for better performance., Comment: Accepted by CIKM2022

Despite the development of ranking optimization techniques, pointwise loss remains the dominating approach for click-through rate prediction. It can be attributed to the calibration ability of the pointwise loss since the prediction can be viewed as the click probability. In practice, a CTR prediction model is also commonly assessed with the ranking ability. To optimize the ranking ability, ranking loss (e.g., pairwise or listwise loss) can be adopted as they usually achieve better rankings than pointwise loss. Previous studies have experimented with a direct combination of the two losses to obtain the benefit from both losses and observed an improved performance. However, previous studies break the meaning of output logit as the click-through rate, which may lead to sub-optimal solutions. To address this issue, we propose an approach that can Jointly optimize the Ranking and Calibration abilities (JRC for short). JRC improves the ranking ability by contrasting the logit value for the sample with different labels and constrains the predicted probability to be a function of the logit subtraction. We further show that JRC consolidates the interpretation of logits, where the logits model the joint distribution. With such an interpretation, we prove that JRC approximately optimizes the contextualized hybrid discriminative-generative objective. Experiments on public and industrial datasets and online A/B testing show that our approach improves both ranking and calibration abilities. Since May 2022, JRC has been deployed on the display advertising platform of Alibaba and has obtained significant performance improvements., Comment: Accepted at KDD 2023

As the field of superconducting quantum computing approaches maturity, optimization of single-device performance is proving to be a promising avenue towards large-scale quantum computers. However, this optimization is possible only if performance metrics can be accurately compared among measurements, devices, and laboratories. Currently such comparisons are inaccurate or impossible due to understudied errors from a plethora of sources. In this Perspective, we outline the current state of error analysis for qubits and resonators in superconducting quantum circuits, and discuss what future investigations are required before superconducting quantum device optimization can be realized.

One of the difficulties of conversion rate (CVR) prediction is that the conversions can delay and take place long after the clicks. The delayed feedback poses a challenge: fresh data are beneficial to continuous training but may not have complete label information at the time they are ingested into the training pipeline. To balance model freshness and label certainty, previous methods set a short waiting window or even do not wait for the conversion signal. If conversion happens outside the waiting window, this sample will be duplicated and ingested into the training pipeline with a positive label. However, these methods have some issues. First, they assume the observed feature distribution remains the same as the actual distribution. But this assumption does not hold due to the ingestion of duplicated samples. Second, the certainty of the conversion action only comes from the positives. But the positives are scarce as conversions are sparse in commercial systems. These issues induce bias during the modeling of delayed feedback. In this paper, we propose DElayed FEedback modeling with Real negatives (DEFER) method to address these issues. The proposed method ingests real negative samples into the training pipeline. The ingestion of real negatives ensures the observed feature distribution is equivalent to the actual distribution, thus reducing the bias. The ingestion of real negatives also brings more certainty information of the conversion. To correct the distribution shift, DEFER employs importance sampling to weigh the loss function. Experimental results on industrial datasets validate the superiority of DEFER. DEFER have been deployed in the display advertising system of Alibaba, obtaining over 6.0% improvement on CVR in several scenarios. The code and data in this paper are now open-sourced {https://github.com/gusuperstar/defer.git}., Comment: Accepted at KDD 2021

Traditional industrial recommenders are usually trained on a single business domain and then serve for this domain. However, in large commercial platforms, it is often the case that the recommenders need to make click-through rate (CTR) predictions for multiple business domains. Different domains have overlapping user groups and items. Thus, there exist commonalities. Since the specific user groups have disparity and the user behaviors may change in various business domains, there also have distinctions. The distinctions result in domain-specific data distributions, making it hard for a single shared model to work well on all domains. To learn an effective and efficient CTR model to handle multiple domains simultaneously, we present Star Topology Adaptive Recommender (STAR). Concretely, STAR has the star topology, which consists of the shared centered parameters and domain-specific parameters. The shared parameters are applied to learn commonalities of all domains, and the domain-specific parameters capture domain distinction for more refined prediction. Given requests from different business domains, STAR can adapt its parameters conditioned on the domain characteristics. The experimental result from production data validates the superiority of the proposed STAR model. Since 2020, STAR has been deployed in the display advertising system of Alibaba, obtaining averaging 8.0% improvement on CTR and 6.0% on RPM (Revenue Per Mille)., Comment: Accepted at CIKM 2021

Feature interaction has been recognized as an important problem in machine learning, which is also very essential for click-through rate (CTR) prediction tasks. In recent years, Deep Neural Networks (DNNs) can automatically learn implicit nonlinear interactions from original sparse features, and therefore have been widely used in industrial CTR prediction tasks. However, the implicit feature interactions learned in DNNs cannot fully retain the complete representation capacity of the original and empirical feature interactions (e.g., cartesian product) without loss. For example, a simple attempt to learn the combination of feature A and feature B as the explicit cartesian product representation of new features can outperform previous implicit feature interaction models including factorization machine (FM)-based models and their variations. In this paper, we propose a Co-Action Network (CAN) to approximate the explicit pairwise feature interactions without introducing too many additional parameters. More specifically, giving feature A and its associated feature B, their feature interaction is modeled by learning two sets of parameters: 1) the embedding of feature A, and 2) a Multi-Layer Perceptron (MLP) to represent feature B. The approximated feature interaction can be obtained by passing the embedding of feature A through the MLP network of feature B. We refer to such pairwise feature interaction as feature co-action, and such a Co-Action Network unit can provide a very powerful capacity to fitting complex feature interactions. Experimental results on public and industrial datasets show that CAN outperforms state-of-the-art CTR models and the cartesian product method. Moreover, CAN has been deployed in the display advertisement system in Alibaba, obtaining 12\% improvement on CTR and 8\% on Revenue Per Mille (RPM), which is a great improvement to the business., Comment: WSDM 2022

The self-powered sensing system could harness ambient energy to power the sensor without the need for external electrical energy. Recently, the concept of photovoltaic (PV) self-powered gas sensing has aroused wider attentions due to room-temperature operation, low power consumption, small size and potential applications. The PV self-powered gas sensors integrate the photovoltaic effects and the gas sensing function into a single chip, which could truly achieve the goal of zero power consumption for an independent gas sensing device. As an emerging concept, the PV self-powered gas sensing has been achieved by using different strategies, including integrated gas sensor and solar cell, integrated light filter and solar cell, gas-sensitive heterojunction photovoltaics, and gas-sensitive lateral photovoltaics, respectively. The purpose of this review is to summarize recent advances of PV self-powered gas sensing and also remark on the directions for future research in this topic., Comment: 16 pages, 9 figures, 2 tables

Optical memory based on electromagnetically induced transparency (EIT) in a double-{\Lambda}atomic system provides a convenient way to convert the frequency, bandwidth or polarization of optical pulses by storing in one{\Lambda}channel and retrieving in another.This memory-based optical converter can be used to bridge quantum nodes of different physical properties in a quantum network. However, in real atoms, each energy level usually contains degenerate Zeeman states and this may lead to additional energy loss, as have been discussed in our recent theoretical paper (Phys. Rev. A 100, 063843). Here, we present an experimental study on the efficiency variation in the EIT-memory-based optical polarization conversion in cold cesium atomsunder the Zeeman-state optical pumping. The experimental results support the theoretical predictions. Our works provide quantitative knowledge and physical insight useful to the practical implementation of EIT-memory-based optical converter.

We present a theoretical study on the efficiency variation of coherent light conversion based on optical memories using the electromagnetically induced transparency (EIT) protocol in an atomic system with degenerate Zeeman states. Based on the Maxwell-Bloch equation, we obtain an approximate analytic solution for the converted light pulses which clarifies that two major factors affecting the efficiency of the converted pulses. The first one is the finite bandwidth effect of the pulses and the difference in the delay-bandwidth product of the writing and reading channel due to the difference in the transition dipole moment. The second one is the mismatch between the stored ground-state coherence and the ratio of the Clebsch-Gordan coefficient for the probe and control transition in the reading channel which results in a non-adiabatic energy loss. To correspond to real experimental conditions, we also perform a numerical calculation of the variation in conversion efficiency versus the Zeeman population distribution under the Zeeman-state optical pumping in storing a $\sigma^{+}$-polarized pulse and retrieving with $\sigma^{-}$ polarization in cesium atoms. Our work provides essential physical insights and quantitative knowledge for the development of a coherent optical converter based on EIT-memory., Comment: 12 pages, 10 figures

Since the science white paper of the Large High Altitude Air Shower Observatory (LHAASO) published on arXiv in 2019 [e-Print: 1905.02773 (astro-ph.HE)], LHAASO has completed the transition from a project to an operational gamma-ray astronomical observatory LHAASO is a new generation multi-component facility located in Daocheng, Sichuan province of China, at an altitude of 4410 meters. It aims at measuring with unprecedented sensitivity the spectrum, composition, and anisotropy of cosmic rays in the energy range between 10$^{12}$ and 10$^{18}$~eV, and acting simultaneously as a wide aperture (one stereoradiant) continuously operating gamma-ray telescope in the energy range between 10$^{11}$ and $10^{15}$~eV with the designed sensitivity of 1.3\% of the Crab Unit (CU) above 100 TeV. LHAASO's capability of measuring simultaneously different shower components (electrons, muons, and Cherenkov/fluorescence light), will allow it to investigate the origin, acceleration, and propagation of CR through measurement of the energy spectrum, elemental composition, and anisotropy with unprecedented resolution. The remarkable sensitivity of LHAASO will play a key role in CR physics and gamma-ray astronomy for a general and comprehensive exploration of the high energy universe and will allow important studies of fundamental physics (such as indirect dark matter search, Lorentz invariance violation, quantum gravity) and solar and heliospheric physics. The LHAASO Collaboration organized an editorial working group and finished all editorial work of this science book, to summarize the instrumental features and outline the prospects of scientific researches with the LHAASO experiment., Comment: This document is a collaborative effort, 185 pages, 110 figures