Search

Modern recommendation systems frequently employ online learning to dynamically update their models with freshly collected data. The most commonly used optimizer for updating neural networks in these contexts is the Adam optimizer, which integrates momentum ($m_t$) and adaptive learning rate ($v_t$). However, the volatile nature of online learning data, characterized by its frequent distribution shifts and presence of noises, poses significant challenges to Adam's standard optimization process: (1) Adam may use outdated momentum and the average of squared gradients, resulting in slower adaptation to distribution changes, and (2) Adam's performance is adversely affected by data noise. To mitigate these issues, we introduce CAdam, a confidence-based optimization strategy that assesses the consistence between the momentum and the gradient for each parameter dimension before deciding on updates. If momentum and gradient are in sync, CAdam proceeds with parameter updates according to Adam's original formulation; if not, it temporarily withholds updates and monitors potential shifts in data distribution in subsequent iterations. This method allows CAdam to distinguish between the true distributional shifts and mere noise, and adapt more quickly to new data distributions. Our experiments with both synthetic and real-world datasets demonstrate that CAdam surpasses other well-known optimizers, including the original Adam, in efficiency and noise robustness. Furthermore, in large-scale A/B testing within a live recommendation system, CAdam significantly enhances model performance compared to Adam, leading to substantial increases in the system's gross merchandise volume (GMV).

Deep Learning Recommendation Model(DLRM)s utilize the embedding layer to represent various categorical features. Traditional DLRMs adopt unified embedding size for all features, leading to suboptimal performance and redundant parameters. Thus, lots of Automatic Embedding size Search (AES) works focus on obtaining mixed embedding sizes with strong model performance. However, previous AES works can hardly address several challenges together: (1) The search results of embedding sizes are unstable; (2) Recommendation effect with AES results is unsatisfactory; (3) Memory cost of embeddings is uncontrollable. To address these challenges, we propose a novel one-shot AES framework called AdaS&S, in which a supernet encompassing various candidate embeddings is built and AES is performed as searching network architectures within it. Our framework contains two main stages: In the first stage, we decouple training parameters from searching embedding sizes, and propose the Adaptive Sampling method to yield a well-trained supernet, which further helps to produce stable AES results. In the second stage, to obtain embedding sizes that benefits the model effect, we design a reinforcement learning search process which utilizes the supernet trained previously. Meanwhile, to adapt searching to specific resource constraint, we introduce the resource competition penalty to balance the model effectiveness and memory cost of embeddings. We conduct extensive experiments on public datasets to show the superiority of AdaS&S. Our method could improve AUC by about 0.3% while saving about 20% of model parameters. Empirical analysis also shows that the stability of searching results in AdaS&S significantly exceeds other methods.

Large language models (LLMs) have achieved great success in many fields, and recent works have studied exploring LLMs for graph discriminative tasks such as node classification. However, the abilities of LLMs for graph generation remain unexplored in the literature. Graph generation requires the LLM to generate graphs with given properties, which has valuable real-world applications such as drug discovery, while tends to be more challenging. In this paper, we propose LLM4GraphGen to explore the ability of LLMs for graph generation with systematical task designs and extensive experiments. Specifically, we propose several tasks tailored with comprehensive experiments to address key questions regarding LLMs' understanding of different graph structure rules, their ability to capture structural type distributions, and their utilization of domain knowledge for property-based graph generation. Our evaluations demonstrate that LLMs, particularly GPT-4, exhibit preliminary abilities in graph generation tasks, including rule-based and distribution-based generation. We also observe that popular prompting methods, such as few-shot and chain-of-thought prompting, do not consistently enhance performance. Besides, LLMs show potential in generating molecules with specific properties. These findings may serve as foundations for designing good LLMs based models for graph generation and provide valuable insights and further research.

Given a Hyperparameter Optimization(HPO) problem, how to design an algorithm to find optimal configurations efficiently? Bayesian Optimization(BO) and the multi-fidelity BO methods employ surrogate models to sample configurations based on history evaluations. More recent studies obtain better performance by integrating BO with HyperBand(HB), which accelerates evaluation by early stopping mechanism. However, these methods ignore the advantage of a suitable evaluation scheme over the default HyperBand, and the capability of BO is still constrained by skewed evaluation results. In this paper, we propose FlexHB, a new method pushing multi-fidelity BO to the limit as well as re-designing a framework for early stopping with Successive Halving(SH). Comprehensive study on FlexHB shows that (1) our fine-grained fidelity method considerably enhances the efficiency of searching optimal configurations, (2) our FlexBand framework (self-adaptive allocation of SH brackets, and global ranking of configurations in both current and past SH procedures) grants the algorithm with more flexibility and improves the anytime performance. Our method achieves superior efficiency and outperforms other methods on various HPO tasks. Empirical results demonstrate that FlexHB can achieve up to 6.9X and 11.1X speedups over the state-of-the-art MFES-HB and BOHB respectively.

Recently, researchers have attempted to investigate the capability of LLMs in handling videos and proposed several video LLM models. However, the ability of LLMs to handle video grounding (VG), which is an important time-related video task requiring the model to precisely locate the start and end timestamps of temporal moments in videos that match the given textual queries, still remains unclear and unexplored in literature. To fill the gap, in this paper, we propose the LLM4VG benchmark, which systematically evaluates the performance of different LLMs on video grounding tasks. Based on our proposed LLM4VG, we design extensive experiments to examine two groups of video LLM models on video grounding: (i) the video LLMs trained on the text-video pairs (denoted as VidLLM), and (ii) the LLMs combined with pretrained visual description models such as the video/image captioning model. We propose prompt methods to integrate the instruction of VG and description from different kinds of generators, including caption-based generators for direct visual description and VQA-based generators for information enhancement. We also provide comprehensive comparisons of various VidLLMs and explore the influence of different choices of visual models, LLMs, prompt designs, etc, as well. Our experimental evaluations lead to two conclusions: (i) the existing VidLLMs are still far away from achieving satisfactory video grounding performance, and more time-related video tasks should be included to further fine-tune these models, and (ii) the combination of LLMs and visual models shows preliminary abilities for video grounding with considerable potential for improvement by resorting to more reliable models and further guidance of prompt instructions.

Overfitting is a common problem in machine learning, which means the model too closely fits the training data while performing poorly in the test data. Among various methods of coping with overfitting, dropout is one of the representative ways. From randomly dropping neurons to dropping neural structures, dropout has achieved great success in improving model performances. Although various dropout methods have been designed and widely applied in past years, their effectiveness, application scenarios, and contributions have not been comprehensively summarized and empirically compared by far. It is the right time to make a comprehensive survey. In this paper, we systematically review previous dropout methods and classify them into three major categories according to the stage where dropout operation is performed. Specifically, more than seventy dropout methods published in top AI conferences or journals (e.g., TKDE, KDD, TheWebConf, SIGIR) are involved. The designed taxonomy is easy to understand and capable of including new dropout methods. Then, we further discuss their application scenarios, connections, and contributions. To verify the effectiveness of distinct dropout methods, extensive experiments are conducted on recommendation scenarios with abundant heterogeneous information. Finally, we propose some open problems and potential research directions about dropout that worth to be further explored., Comment: 26 pages

We present Distributed Equivalent Substitution (DES) training, a novel distributed training framework for large-scale recommender systems with dynamic sparse features. DES introduces fully synchronous training to large-scale recommendation system for the first time by reducing communication, thus making the training of commercial recommender systems converge faster and reach better CTR. DES requires much less communication by substituting the weights-rich operators with the computationally equivalent sub-operators and aggregating partial results instead of transmitting the huge sparse weights directly through the network. Due to the use of synchronous training on large-scale Deep Learning Recommendation Models (DLRMs), DES achieves higher AUC(Area Under ROC). We successfully apply DES training on multiple popular DLRMs of industrial scenarios. Experiments show that our implementation outperforms the state-of-the-art PS-based training framework, achieving up to 68.7% communication savings and higher throughput compared to other PS-based recommender systems., Comment: Accepted by SIGIR '2020. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020

Although attention-based Neural Machine Translation (NMT) has achieved remarkable progress in recent years, it still suffers from issues of repeating and dropping translations. To alleviate these issues, we propose a novel key-value memory-augmented attention model for NMT, called KVMEMATT. Specifically, we maintain a timely updated keymemory to keep track of attention history and a fixed value-memory to store the representation of source sentence throughout the whole translation process. Via nontrivial transformations and iterative interactions between the two memories, the decoder focuses on more appropriate source word(s) for predicting the next target word at each decoding step, therefore can improve the adequacy of translations. Experimental results on Chinese=>English and WMT17 German<=>English translation tasks demonstrate the superiority of the proposed model., Comment: Accepted at IJCAI 2018

ICESat-2 is an exciting new multi-beam, photon-counting, satellite lidar designed to acquire simultaneous high-resolution measurements of the earth's surface and atmospheric structure. Launched in September, 2018 the Advanced Topographic Lidar Altimeter System (ATLAS) on board ICESat-2 has been acquiring data continuously for nearly 1 year. The 3 laser beams comprising the atmospheric channel are providing excellent back scatter measurements that enable the retrieval of global cloud fraction, cloud and aerosol layer height, column optical depth, and blowing snow properties. Public release of the data products began with version 1in May, 2019 and continues with a significantly improved version 2 in October, 2019. This presentation will give an overview of the ICESat-2 atmospheric channel and show examples of cloud, aerosol and blowings now properties as well as comparisons of ATLAS and CALIOP measurements.

UNKNOWN

UNKNOWN

Satellite lidar remote sensing of the atmosphere has been ongoing for more than a decade providing the opportunity to study atmospheric processes in great detail. Here we use 12 years of Cloud‐Aerosol Lidar with Orthogonal Polarization measurements to derive a climatology of blowing snow layer height, optical depth, and frequency over Antarctica for the period 2006–2017. Limited to the vertical resolution of the Cloud‐Aerosol Lidar with Orthogonal Polarization data, our climatology includes all blowing snow layers greater than about 30 m in thickness for clear or optically thin cloud regions. Our results show that blowing snow occurs over 50% of the time over large regions with frequencies often exceeding 70%. The overall pattern of blowing snow frequency is fairly consistent from year to year, but there are regional differences. We examined the data for temporal trends in blowing snow properties and found significant trends only in blowing snow frequency. A small area of East Antarctica with generally low blowing snow frequency shows a statistically significant increase in blowing snow frequency ranging from 10% to 100% per decade. No significant trends in frequency were found in regions of high (>50%) blowing snow frequency, and only isolated small areas exhibited a decrease in frequency through the study period.

In this Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) study, we present a simple way of determining laser pulse bidirectional reflectance over snow/ice surface using the Cloud-Aerosol LIdar with Orthogonal Polarization (CALIOP) 532 nanometer polarization channels' measurements. The saturated laser pulse returns from snow and ice surfaces are recovered based on surface tail information. The method overview and initial assessment of the method performance will be presented. The retrieved snow surface bidirectional reflectance is compared with reflectance from both CALIOP cloud cover regions and Moderate Resolution Imaging Spectroradiometer (Earth Observing System (EOS)) (MODIS) Bi-directional Reflectance Distribution Function (BRDF) / Albedo model parameters. The comparisons show that the snow surface bidirectional reflectance over Antarctica for saturation region are generally reliable with a mean value of about 0.90 plus or minus 0.10, while the mean surface reflectance from cloud cover region is about 0.84 plus or minus 0.13 and the calculated MODIS reflectance at 555 nanometers from the BRDF / Albedo model with near nadir illumination and viewing angles is about 0.96 plus or minus 0.04. The comparisons here demonstrate that the snow surface reflectance underneath the cloud with cloud optical depth of about 1 is significantly lower than that for a clear sky condition.

Blowing snow over Antarctica is a widespread and frequent event. Satellite remote sensing using lidar has shown that blowing snow occurs over 70% of the time over large areas of Antarctica in winter. The transport and sublimation of blowing snow are important terms in the ice sheet mass balance equation and the latter is also an important part of the hydrological cycle. Until now the only way to estimate the magnitude of these processes was through model parameterization. We present a technique that uses direct satellite observations of blowing snow and model (MERRA-2) temperature and humidity fields to compute both transport and sublimation of blowing snow over Antarctica for the period 2006 to 2016. The results show a larger annual continent-wide integrated sublimation than current published estimates and a significant transport of snow from continent to ocean. The talk will also include the lidar backscatter structure of blowing snow layers that often reach heights of 200 to 300 m as well as the first dropsonde measurements of temperature, moisture and wind through blowing snow layers.

Blowing snow processes commonly occur over the earth's ice sheets when the 10 mile wind speed exceeds a threshold value. These processes play a key role in the sublimation and redistribution of snow thereby influencing the surface mass balance. Prior field studies and modeling results have shown the importance of blowing snow sublimation and transport on the surface mass budget and hydrological cycle of high-latitude regions. For the first time, we present continent-wide estimates of blowing snow sublimation and transport over Antarctica for the period 2006-2016 based on direct observation of blowing snow events. We use an improved version of the blowing snow detection algorithm developed for previous work that uses atmospheric backscatter measurements obtained from the CALIOP (Cloud-Aerosol Lidar with Orthogonal Polarization) lidar aboard the CALIPSO (Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation) satellite. The blowing snow events identified by CALIPSO and meteorological fields from MERRA-2 are used to compute the blowing snow sublimation and transport rates. Our results show that maximum sublimation occurs along and slightly inland of the coastline. This is contrary to the observed maximum blowing snow frequency which occurs over the interior. The associated temperature and moisture reanalysis fields likely contribute to the spatial distribution of the maximum sublimation values. However, the spatial pattern of the sublimation rate over Antarctica is consistent with modeling studies and precipitation estimates. Overall, our results show that the 2006-2016 Antarctica average integrated blowing snow sublimation is about 393 +/- 196 Gt yr(exp -1), which is considerably larger than previous model-derived estimates. We find maximum blowing snow transport amount of 5 Mt km-1 yr(exp -1) over parts of East Antarctica and estimate that the average snow transport from continent to ocean is about 3.7 Gt yr(exp -1). These continent-wide estimates are the first of their kind and can be used to help model and constrain the surface mass budget over Antarctica.

This is the second part of a study on how temporal sampling frequency affects satellite retrievals in support of the Deep Space Climate Observatory (DSCOVR) mission. Continuing from Part 1, which looked at Earth's radiation budget, this paper presents the effect of sampling frequency on DSCOVR-derived cloud fraction. The output from NASA's Goddard Earth Observing System version 5 (GEOS-5) Nature Run is used as the "truth". The effect of temporal resolution on potential DSCOVR observations is assessed by subsampling the full Nature Run data. A set of metrics, including uncertainty and absolute error in the subsampled time series, correlation between the original and the subsamples, and Fourier analysis have been used for this study. Results show that, for a given sampling frequency, the uncertainties in the annual mean cloud fraction of the sunlit half of the Earth are larger over land than over ocean. Analysis of correlation coefficients between the subsamples and the original time series demonstrates that even though sampling at certain longer time intervals may not increase the uncertainty in the mean, the subsampled time series is further and further away from the "truth" as the sampling interval becomes larger and larger. Fourier analysis shows that the simulated DSCOVR cloud fraction has underlying periodical features at certain time intervals, such as 8, 12, and 24 h. If the data is subsampled at these frequencies, the uncertainties in the mean cloud fraction are higher. These results provide helpful insights for the DSCOVR temporal sampling strategy.