Search

Dense features, customized for different business scenarios, are essential in short video classification. However, their complexity, specific adaptation requirements, and high computational costs make them resource-intensive and less accessible during online inference. Consequently, these dense features are categorized as `Privileged Dense Features'.Meanwhile, end-to-end multi-modal models have shown promising results in numerous computer vision tasks. In industrial applications, prioritizing end-to-end multi-modal features, can enhance efficiency but often leads to the loss of valuable information from historical privileged dense features. To integrate both features while maintaining efficiency and manageable resource costs, we present Confidence-aware Privileged Feature Distillation (CPFD), which empowers features of an end-to-end multi-modal model by adaptively distilling privileged features during training. Unlike existing privileged feature distillation (PFD) methods, which apply uniform weights to all instances during distillation, potentially causing unstable performance across different business scenarios and a notable performance gap between teacher model (Dense Feature enhanced multimodal-model DF-X-VLM) and student model (multimodal-model only X-VLM), our CPFD leverages confidence scores derived from the teacher model to adaptively mitigate the performance variance with the student model. We conducted extensive offline experiments on five diverse tasks demonstrating that CPFD improves the video classification F1 score by 6.76% compared with end-to-end multimodal-model (X-VLM) and by 2.31% with vanilla PFD on-average. And it reduces the performance gap by 84.6% and achieves results comparable to teacher model DF-X-VLM. The effectiveness of CPFD is further substantiated by online experiments, and our framework has been deployed in production systems for over a dozen models., Comment: Camera ready for CIKM 2024

Electrons in graphene are described by relativistic Dirac-Weyl spinors with a two-component pseudospin1-12. The unique pseudospin structure of Dirac electrons leads to emerging phenomena such as the massless Dirac cone2, anomalous quantum Hall effect2, 3, and Klein tunneling4, 5 in graphene. The capability to manipulate electron pseudospin is highly desirable for novel graphene electronics, and it requires precise control to differentiate the two graphene sub-lattices at atomic level. Graphene/boron nitride (graphene/BN) Moire superlattice, where a fast sub-lattice oscillation due to B-N atoms is superimposed on the slow Moire period, provides an attractive approach to engineer the electron pseudospin in graphene13-18. This unusual Moire superlattice leads to a spinor potential with unusual hybridization of electron pseudospins, which can be probed directly through infrared spectroscopy because optical transitions are very sensitive to excited state wavefunctions. Here, we perform micro-infrared spectroscopy on graphene/BN heterostructure and demonstrate that the Moire superlattice potential is dominated by a pseudospin-mixing component analogous to a spatially varying pseudomagnetic field. In addition, we show that the spinor potential depends sensitively on the gate-induced carrier concentration in graphene, indicating a strong renormalization of the spinor potential from electron-electron interactions. Our study offers deeper understanding of graphene pseudospin structure under spinor Moire potential, as well as exciting opportunities to control pseudospin in two-dimensional heterostructures for novel electronic and photonic nanodevices.