Search

Knowledge-based visual question answering (KB-VQA) is a challenging task, which requires the model to leverage external knowledge for comprehending and answering questions grounded in visual content. Recent studies retrieve the knowledge passages from external knowledge bases and then use them to answer questions. However, these retrieved knowledge passages often contain irrelevant or noisy information, which limits the performance of the model. To address the challenge, we propose two synergistic models: Knowledge Condensation model and Knowledge Reasoning model. We condense the retrieved knowledge passages from two perspectives. First, we leverage the multimodal perception and reasoning ability of the visual-language models to distill concise knowledge concepts from retrieved lengthy passages, ensuring relevance to both the visual content and the question. Second, we leverage the text comprehension ability of the large language models to summarize and condense the passages into the knowledge essence which helps answer the question. These two types of condensed knowledge are then seamlessly integrated into our Knowledge Reasoning model, which judiciously navigates through the amalgamated information to arrive at the conclusive answer. Extensive experiments validate the superiority of the proposed method. Compared to previous methods, our method achieves state-of-the-art performance on knowledge-based VQA datasets (65.1% on OK-VQA and 60.1% on A-OKVQA) without resorting to the knowledge produced by GPT-3 (175B).

In recent years, text-to-video retrieval methods based on CLIP have experienced rapid development. The primary direction of evolution is to exploit the much wider gamut of visual and textual cues to achieve alignment. Concretely, those methods with impressive performance often design a heavy fusion block for sentence (words)-video (frames) interaction, regardless of the prohibitive computation complexity. Nevertheless, these approaches are not optimal in terms of feature utilization and retrieval efficiency. To address this issue, we adopt multi-granularity visual feature learning, ensuring the model's comprehensiveness in capturing visual content features spanning from abstract to detailed levels during the training phase. To better leverage the multi-granularity features, we devise a two-stage retrieval architecture in the retrieval phase. This solution ingeniously balances the coarse and fine granularity of retrieval content. Moreover, it also strikes a harmonious equilibrium between retrieval effectiveness and efficiency. Specifically, in training phase, we design a parameter-free text-gated interaction block (TIB) for fine-grained video representation learning and embed an extra Pearson Constraint to optimize cross-modal representation learning. In retrieval phase, we use coarse-grained video representations for fast recall of top-k candidates, which are then reranked by fine-grained video representations. Extensive experiments on four benchmarks demonstrate the efficiency and effectiveness. Notably, our method achieves comparable performance with the current state-of-the-art methods while being nearly 50 times faster.

The proliferation of short video and live-streaming platforms has revolutionized how consumers engage in online shopping. Instead of browsing product pages, consumers are now turning to rich-content e-commerce, where they can purchase products through dynamic and interactive media like short videos and live streams. This emerging form of online shopping has introduced technical challenges, as products may be presented differently across various media domains. Therefore, a unified product representation is essential for achieving cross-domain product recognition to ensure an optimal user search experience and effective product recommendations. Despite the urgent industrial need for a unified cross-domain product representation, previous studies have predominantly focused only on product pages without taking into account short videos and live streams. To fill the gap in the rich-content e-commerce area, in this paper, we introduce a large-scale cRoss-dOmain Product Ecognition dataset, called ROPE. ROPE covers a wide range of product categories and contains over 180,000 products, corresponding to millions of short videos and live streams. It is the first dataset to cover product pages, short videos, and live streams simultaneously, providing the basis for establishing a unified product representation across different media domains. Furthermore, we propose a Cross-dOmain Product rEpresentation framework, namely COPE, which unifies product representations in different domains through multimodal learning including text and vision. Extensive experiments on downstream tasks demonstrate the effectiveness of COPE in learning a joint feature space for all product domains., Comment: ICCV23

Live commerce is the act of selling products online through live streaming. The customer's diverse demands for online products introduce more challenges to Livestreaming Product Recognition. Previous works have primarily focused on fashion clothing data or utilize single-modal input, which does not reflect the real-world scenario where multimodal data from various categories are present. In this paper, we present LPR4M, a large-scale multimodal dataset that covers 34 categories, comprises 3 modalities (image, video, and text), and is 50x larger than the largest publicly available dataset. LPR4M contains diverse videos and noise modality pairs while exhibiting a long-tailed distribution, resembling real-world problems. Moreover, a cRoss-vIew semantiC alignmEnt (RICE) model is proposed to learn discriminative instance features from the image and video views of the products. This is achieved through instance-level contrastive learning and cross-view patch-level feature propagation. A novel Patch Feature Reconstruction loss is proposed to penalize the semantic misalignment between cross-view patches. Extensive experiments demonstrate the effectiveness of RICE and provide insights into the importance of dataset diversity and expressivity. The dataset and code are available at https://github.com/adxcreative/RICE, Comment: Accepted to ICCV2023

One striking feature of molecular rotors is their ability to change conformation with detectable optical signals through molecular motion when stimulated. However, due to the strong intermolecular interactions, synthetic molecular rotors have often relied on fluid environments. Here, we take advantage of the solid-state intramolecular motion of aggregation-induced emission (AIE) molecular rotors and one-dimensional fibers, developing highly sensitive optical fiber sensors that respond to ambient humidity rapidly and reversibly with observable chromatic fluorescence change. Moisture environments induce the swelling of the polymer fibers, activating intramolecular motions of AIE molecules to result in red-shifted fluorescence and linear response to ambient humidity. In this case, polymer fiber provides a process-friendly architecture and a physically tunable medium for the embedded AIE molecules to manipulate their fluorescence response characteristics. Assembly of sensor fibers could be built into hierarchical structures, which are adaptive to diverse-configuration for spatial-temporal humidity mapping, and suitable for device integration to build light-emitting sensors as well as touchless positioning interfaces for intelligence systems.

One striking feature of molecular rotors is their ability to change conformation with detectable optical signals through molecular motion when stimulated. However, due to the strong intermolecular interactions, synthetic molecular rotors have often relied on fluid environments. Here, we take advantage of the solid-state intramolecular motion of aggregation-induced emission (AIE) molecular rotors and one-dimensional fibers, developing highly sensitive optical fiber sensors that respond to ambient humidity rapidly and reversibly with observable chromatic fluorescence change. Moisture environments induce the swelling of the polymer fibers, activating intramolecular motions of AIE molecules to result in red-shifted fluorescence and linear response to ambient humidity. In this case, polymer fiber provides a process-friendly architecture and a physically tunable medium for the embedded AIE molecules to manipulate their fluorescence response characteristics. Assembly of sensor fibers could be built into hierarchical structures, which are adaptive to diverse-configuration for spatial-temporal humidity mapping, and suitable for device integration to build light-emitting sensors as well as touchless positioning interfaces for intelligence systems.

One striking feature of molecular rotors is their ability to change conformation with detectable optical signals through molecular motion when stimulated. However, due to the strong intermolecular interactions, synthetic molecular rotors have often relied on fluid environments. Here, we take advantage of the solid-state intramolecular motion of aggregation-induced emission (AIE) molecular rotors and one-dimensional fibers, developing highly sensitive optical fiber sensors that respond to ambient humidity rapidly and reversibly with observable chromatic fluorescence change. Moisture environments induce the swelling of the polymer fibers, activating intramolecular motions of AIE molecules to result in red-shifted fluorescence and linear response to ambient humidity. In this case, polymer fiber provides a process-friendly architecture and a physically tunable medium for the embedded AIE molecules to manipulate their fluorescence response characteristics. Assembly of sensor fibers could be built into hierarchical structures, which are adaptive to diverse-configuration for spatial-temporal humidity mapping, and suitable for device integration to build light-emitting sensors as well as touchless positioning interfaces for intelligence systems.

Circularly polarized luminescence (CPL) switching has attracted great attention due to the potential applications in chiral photonics and electronics. However, the lack of examples to achieve switchable CPL within a single material in the dry solid state hampers the scope of applications. Herein, we demonstrate a crystalline chiral polymer film as a polarizing medium consisting of radially assembled twisted crystallites, where achiral aggregation-induced emissive luminogens (AIEgens) are confined between the twisted crystalline stacks, eventually yielding handedness-switchable CPL by simple film tilting. Hierarchically organized twisted crystallites create the selective reflection activity of the polarizing medium. Upon film tilting, enantiomeric switching is realized by selectively collecting transmitted and reflected CPL components. The confined AIEgens in the crystalline polarizing system show a great enhancement of the luminescence efficiency. Moreover, the approach is general with broadband activity, and various AIEgens could be applied to generate full-color-tunable CPL. Additionally, the rigid and continuous nature of this polarizing system affords enhanced optical stability and facile modulation, developing a general route for designing chiroptical materials. © 2020 American Chemical Society.

Circularly polarized luminescence (CPL) switching has attracted great attention due to the potential applications in chiral photonics and electronics. However, the lack of examples to achieve switchable CPL within a single material in the dry solid state hampers the scope of applications. Herein, we demonstrate a crystalline chiral polymer film as a polarizing medium consisting of radially assembled twisted crystallites, where achiral aggregation-induced emissive luminogens (AIEgens) are confined between the twisted crystalline stacks, eventually yielding handedness-switchable CPL by simple film tilting. Hierarchically organized twisted crystallites create the selective reflection activity of the polarizing medium. Upon film tilting, enantiomeric switching is realized by selectively collecting transmitted and reflected CPL components. The confined AIEgens in the crystalline polarizing system show a great enhancement of the luminescence efficiency. Moreover, the approach is general with broadband activity, and various AIEgens could be applied to generate full-color-tunable CPL. Additionally, the rigid and continuous nature of this polarizing system affords enhanced optical stability and facile modulation, developing a general route for designing chiroptical materials. © 2020 American Chemical Society.

Despite the huge progress of luminescent molecular assemblies over the past decade, it is still challenging to understand their confined behavior in semi-crystalline polymers for constrained space recognition. Here, we report a polymorphic luminogen with aggregation-induced emission (AIE), capable of selective growth in polymer amorphous and crystalline phases with distinct color. The polymorphic behaviors of the AIE luminogen embedded within the polymer network are dependent on the size of nano-confinement: a thermodynamically stable polymorph of the AIE luminogen with green emission is stabilized in the amorphous phase, while a metastable polymorph with yellow emission is confined in the crystalline phase. The information on polymer crystalline and amorphous phases is transformed into distinct fluorescence colors, allowing a single AIE luminogen as a fluorescent marker for visualization of polymer microstructures in terms of amorphous and crystalline phase distribution, quantitative polymer crystallinity measurement, and spatial morphological arrangement. Our findings demonstrate that confinement of the AIE luminogen in the polymer network can achieve free space recognition and also provide a correlation between microscopic morphologies and macroscopic optical signals. We envision that our strategy will inspire the development of other materials with spatial confinement to incorporate AIE luminogens for various applications. This journal is © The Royal Society of Chemistry.

Circularly polarized luminescence (CPL) switching has attracted great attention due to the potential applications in chiral photonics and electronics. However, the lack of examples to achieve switchable CPL within a single material in the dry solid state hampers the scope of applications. Herein, we demonstrate a crystalline chiral polymer film as a polarizing medium consisting of radially assembled twisted crystallites, where achiral aggregation-induced emissive luminogens (AIEgens) are confined between the twisted crystalline stacks, eventually yielding handedness-switchable CPL by simple film tilting. Hierarchically organized twisted crystallites create the selective reflection activity of the polarizing medium. Upon film tilting, enantiomeric switching is realized by selectively collecting transmitted and reflected CPL components. The confined AIEgens in the crystalline polarizing system show a great enhancement of the luminescence efficiency. Moreover, the approach is general with broadband activity, and various AIEgens could be applied to generate full-color-tunable CPL. Additionally, the rigid and continuous nature of this polarizing system affords enhanced optical stability and facile modulation, developing a general route for designing chiroptical materials. © 2020 American Chemical Society.

Despite the huge progress of luminescent molecular assemblies over the past decade, it is still challenging to understand their confined behavior in semi-crystalline polymers for constrained space recognition. Here, we report a polymorphic luminogen with aggregation-induced emission (AIE), capable of selective growth in polymer amorphous and crystalline phases with distinct color. The polymorphic behaviors of the AIE luminogen embedded within the polymer network are dependent on the size of nano-confinement: a thermodynamically stable polymorph of the AIE luminogen with green emission is stabilized in the amorphous phase, while a metastable polymorph with yellow emission is confined in the crystalline phase. The information on polymer crystalline and amorphous phases is transformed into distinct fluorescence colors, allowing a single AIE luminogen as a fluorescent marker for visualization of polymer microstructures in terms of amorphous and crystalline phase distribution, quantitative polymer crystallinity measurement, and spatial morphological arrangement. Our findings demonstrate that confinement of the AIE luminogen in the polymer network can achieve free space recognition and also provide a correlation between microscopic morphologies and macroscopic optical signals. We envision that our strategy will inspire the development of other materials with spatial confinement to incorporate AIE luminogens for various applications. This journal is © The Royal Society of Chemistry.

Organic–inorganic (O–I) heterostructures, consisting of atomically thin inorganic semiconductors and organic molecules, present synergistic and enhanced optoelectronic properties with a high tunability. Here, we develop a class of air-stable vertical O–I heterostructures comprising a monolayer of transition-metal dichalcogenides (TMDs), including WS2, WSe2, and MoSe2, on top of tetraphenylethylene (TPE) core-based aggregation-induced emission (AIE) molecular rotors. The created O–I heterostructures yields a photoluminescence (PL) enhancement of up to ∼950%, ∼500%, and ∼330% in the top monolayer WS2, MoSe2, and WSe2 as compared to PL in their pristine monolayers, respectively. The strong PL enhancement is mainly attributed to the efficient photogenerated carrier process in the AIE luminogens (courtesy of their restricted intermolecular motions in the solid state) and the charge-transfer process in the created type I O–I heterostructures. Moreover, we observe an improvement in photovoltaic properties of the TMDs in the heterostructures including the quasi-Fermi level splitting, minority carrier lifetime, and light absorption. This work presents an inspiring example of combining stable, highly luminescent AIE-based molecules, with rich photochemistry and versatile applications, with atomically thin inorganic semiconductors for multifunctional and efficient optoelectronic devices.

Despite the huge progress of luminescent molecular assemblies over the past decade, it is still challenging to understand their confined behavior in semi-crystalline polymers for constrained space recognition. Here, we report a polymorphic luminogen with aggregation-induced emission (AIE), capable of selective growth in polymer amorphous and crystalline phases with distinct color. The polymorphic behaviors of the AIE luminogen embedded within the polymer network are dependent on the size of nano-confinement: a thermodynamically stable polymorph of the AIE luminogen with green emission is stabilized in the amorphous phase, while a metastable polymorph with yellow emission is confined in the crystalline phase. The information on polymer crystalline and amorphous phases is transformed into distinct fluorescence colors, allowing a single AIE luminogen as a fluorescent marker for visualization of polymer microstructures in terms of amorphous and crystalline phase distribution, quantitative polymer crystallinity measurement, and spatial morphological arrangement. Our findings demonstrate that confinement of the AIE luminogen in the polymer network can achieve free space recognition and also provide a correlation between microscopic morphologies and macroscopic optical signals. We envision that our strategy will inspire the development of other materials with spatial confinement to incorporate AIE luminogens for various applications. This journal is © The Royal Society of Chemistry.

Image retrieval is a fundamental problem in computer vision. This paper presents our 3rd place detailed solution to the Google Landmark Retrieval 2020 challenge. We focus on the exploration of data cleaning and models with metric learning. We use a data cleaning strategy based on embedding clustering. Besides, we employ a data augmentation method called Corner-Cutmix, which improves the model's ability to recognize multi-scale and occluded landmark images. We show in detail the ablation experiments and results of our method.

Designing aerogel materials featuring both high thermal insulation property and excellent mechanical robustness is of great interest for applications in superior integrated energy management systems. To meet the above requirements, composite aerogels based on hierarchical “stiff–soft” binary networks are reported, in which secondary mesoporous polymethylsilsesquioxane domains intertwined by bacterial cellulose nanofibrillar networks are connected in tandem. The resulting composite aerogels are characterized by highly porous (93.6%) and nanosized structure with a surface area of 660 m2 g−1, leading to the excellent thermal insulation performance with a low thermal conductivity of 15.3 mW m−1 K−1. The integrated “stiff–soft” binary nature also endows the composite aerogels with high flexibility that can conform to various substrates as well as large tensile strength that can withstand more than 2.70 × 104 times its own weight. These composite aerogels show multifunctionality in terms of efficient wearable protection, controllable thermal management, and ultrafast oil/water separation. These favorable multifeatures present composite aerogels ideal for aerospace, industrial, and commercial applications.

Development of new aggregation-induced emission (AIE) luminogens has been a hot research topic because they thoroughly solve the notorious aggregation-caused quenching effect confronted in conventional fluorogens and their promising applications in, for example, organic light-emitting diodes, chemo-and biosensors and bioimaging. Many AIE luminogens (AIEgens) have been prepared but most of them are silole, tetraphenylethene, distyrylanthracene, and their derivatives. In this work, based on the skeleton of tetraphenylpyrazine (TPP), a new AIEgen, named TPP-PDCV, is generated by functionalizing TPP with malonitrile group. TPP-PDCV can serve as a sensitive ratiometric fluorescent probe for detecting hydrogen sulfide with high speciality and low detection limit of down to 0.5 x 10(-6) m. The mechanism for such detection is fully investigated and deciphered. Unlike most reported mechanochromic AIEgens, which undergo turn-off or -on emission or emission bathochromic shift in the presence of external stimuli, TPP-PDCV exhibits an abnormal and reversible mechanochromism with hypsochromic effect. These indicate that TPP-PDCV possesses a huge potential for high-tech applications through rational modification of TPP core.

Development of new aggregation-induced emission (AIE) luminogens has been a hot research topic because they thoroughly solve the notorious aggregation-caused quenching effect confronted in conventional fluorogens and their promising applications in, for example, organic light-emitting diodes, chemo-and biosensors and bioimaging. Many AIE luminogens (AIEgens) have been prepared but most of them are silole, tetraphenylethene, distyrylanthracene, and their derivatives. In this work, based on the skeleton of tetraphenylpyrazine (TPP), a new AIEgen, named TPP-PDCV, is generated by functionalizing TPP with malonitrile group. TPP-PDCV can serve as a sensitive ratiometric fluorescent probe for detecting hydrogen sulfide with high speciality and low detection limit of down to 0.5 x 10(-6) m. The mechanism for such detection is fully investigated and deciphered. Unlike most reported mechanochromic AIEgens, which undergo turn-off or -on emission or emission bathochromic shift in the presence of external stimuli, TPP-PDCV exhibits an abnormal and reversible mechanochromism with hypsochromic effect. These indicate that TPP-PDCV possesses a huge potential for high-tech applications through rational modification of TPP core.

Graphene fiber-based electrodes for supercapacitors are promising candidates for wearable energy storage. Their main limitation, although, is the low electrochemical performance caused by the restacking of graphene sheets and their hydrophobicity to electrolytes. Incorporation of nanofillers into graphene is an efficient way to overcome the challenges, however, often leading to a severe deterioration in their mechanical property and/or conductivity, thus significantly influences the practical applications and rate performance of the device. Herein, an approach of fabricating hybrid fibers from graphene oxide (GO) and cellulose nanocrystal (CNC) via non-liquid-crystal spinning and followed by chemical reduction is presented to collectively work around the problems. The resultant hybrid GO/CNC fibers demonstrated a high capacitive performance, enhanced mechanical property, and improved hydrophilicity simultaneously. Furthermore, the conductivity kept at a high value. Sample with a GO/CNC weight ratio of 100/20 possessed a high capacitance of 208.2 F cm−3, a strength of 199.8 MPa, a contact angle of 63.3°, and conductivity of 64.7 S cm−1. Moreover, the supercapacitor assembled from this fiber exhibited a high energy density and power density (5.1 mW h cm−3 and 496.4 mW cm−3), excellent flexibility and bending stability, which has a great potential for use as a flexible power storage.

Development of new aggregation-induced emission (AIE) luminogens has been a hot research topic because they thoroughly solve the notorious aggregation-caused quenching effect confronted in conventional fluorogens and their promising applications in, for example, organic light-emitting diodes, chemo-and biosensors and bioimaging. Many AIE luminogens (AIEgens) have been prepared but most of them are silole, tetraphenylethene, distyrylanthracene, and their derivatives. In this work, based on the skeleton of tetraphenylpyrazine (TPP), a new AIEgen, named TPP-PDCV, is generated by functionalizing TPP with malonitrile group. TPP-PDCV can serve as a sensitive ratiometric fluorescent probe for detecting hydrogen sulfide with high speciality and low detection limit of down to 0.5 x 10(-6) m. The mechanism for such detection is fully investigated and deciphered. Unlike most reported mechanochromic AIEgens, which undergo turn-off or -on emission or emission bathochromic shift in the presence of external stimuli, TPP-PDCV exhibits an abnormal and reversible mechanochromism with hypsochromic effect. These indicate that TPP-PDCV possesses a huge potential for high-tech applications through rational modification of TPP core.

Building humidity sensors possessing the features of diverse-configuration compatibility, and capability of measurement of spatial and temporal humidity gradients is of great interest for highly integrated electronics and wearable monitoring systems. Herein, a visual sensing approach based on fluorescent imaging is presented, by assembling aggregation-inducedemission (AIE)-active molecular rotors into a moisture-captured network; the resulting AIE humidity sensors are compatible with diverse applications, having tunable geometries and desirable architectures. The invisible information of relative humidity (RH) is transformed into different fluorescence colors that enable direct observation by the naked eyes based on the twisted intramolecular charge-transfer effect of the AIE-active molecular rotors. The resulting AIE humidity sensors show excellent performance in terms of good sensitivity, precise quantitative measurement, high spatial-temporal resolution, and fast response/recovery time. Their multiscale applications, such as regional environmental RH detection, internal humidity mapping, and sensitive human-body humidity sensing are demonstrated. The proposed humidity visualization strategy may provide a new insight to develop humidity sensors for various applications.

Building humidity sensors possessing the features of diverse-configuration compatibility, and capability of measurement of spatial and temporal humidity gradients is of great interest for highly integrated electronics and wearable monitoring systems. Herein, a visual sensing approach based on fluorescent imaging is presented, by assembling aggregation-inducedemission (AIE)-active molecular rotors into a moisture-captured network; the resulting AIE humidity sensors are compatible with diverse applications, having tunable geometries and desirable architectures. The invisible information of relative humidity (RH) is transformed into different fluorescence colors that enable direct observation by the naked eyes based on the twisted intramolecular charge-transfer effect of the AIE-active molecular rotors. The resulting AIE humidity sensors show excellent performance in terms of good sensitivity, precise quantitative measurement, high spatial-temporal resolution, and fast response/recovery time. Their multiscale applications, such as regional environmental RH detection, internal humidity mapping, and sensitive human-body humidity sensing are demonstrated. The proposed humidity visualization strategy may provide a new insight to develop humidity sensors for various applications.