Search

Controllability and proactivity are crucial properties of autonomous conversational agents (CAs). Controllability requires the CAs to follow the standard operating procedures (SOPs), such as verifying identity before activating credit cards. Proactivity requires the CAs to guide the conversation towards the goal during user uncooperation, such as persuasive dialogue. Existing research cannot be unified with controllability, proactivity, and low manual annotation. To bridge this gap, we propose a new framework for planning-based conversational agents (PCA) powered by large language models (LLMs), which only requires humans to define tasks and goals for the LLMs. Before conversation, LLM plans the core and necessary SOP for dialogue offline. During the conversation, LLM plans the best action path online referring to the SOP, and generates responses to achieve process controllability. Subsequently, we propose a semi-automatic dialogue data creation framework and curate a high-quality dialogue dataset (PCA-D). Meanwhile, we develop multiple variants and evaluation metrics for PCA, e.g., planning with Monte Carlo Tree Search (PCA-M), which searches for the optimal dialogue action while satisfying SOP constraints and achieving the proactive of the dialogue. Experiment results show that LLMs finetuned on PCA-D can significantly improve the performance and generalize to unseen domains. PCA-M outperforms other CoT and ToT baselines in terms of conversation controllability, proactivity, task success rate, and overall logical coherence, and is applicable in industry dialogue scenarios. The dataset and codes are available at XXXX.

Non-autoregressive (NAR) automatic speech recognition (ASR) models predict tokens independently and simultaneously, bringing high inference speed. However, there is still a gap in the accuracy of the NAR models compared to the autoregressive (AR) models. In this paper, we propose a single-step NAR ASR architecture with high accuracy and inference speed, called EffectiveASR. It uses an Index Mapping Vector (IMV) based alignment generator to generate alignments during training, and an alignment predictor to learn the alignments for inference. It can be trained end-to-end (E2E) with cross-entropy loss combined with alignment loss. The proposed EffectiveASR achieves competitive results on the AISHELL-1 and AISHELL-2 Mandarin benchmarks compared to the leading models. Specifically, it achieves character error rates (CER) of 4.26%/4.62% on the AISHELL-1 dev/test dataset, which outperforms the AR Conformer with about 30x inference speedup., Comment: Submitted to ICASSP 2025

Dual-Encoders is a promising mechanism for answer retrieval in question answering (QA) systems. Currently most conventional Dual-Encoders learn the semantic representations of questions and answers merely through matching score. Researchers proposed to introduce the QA interaction features in scoring function but at the cost of low efficiency in inference stage. To keep independent encoding of questions and answers during inference stage, variational auto-encoder is further introduced to reconstruct answers (questions) from question (answer) embeddings as an auxiliary task to enhance QA interaction in representation learning in training stage. However, the needs of text generation and answer retrieval are different, which leads to hardness in training. In this work, we propose a framework to enhance the Dual-Encoders model with question answer cross-embeddings and a novel Geometry Alignment Mechanism (GAM) to align the geometry of embeddings from Dual-Encoders with that from Cross-Encoders. Extensive experimental results show that our framework significantly improves Dual-Encoders model and outperforms the state-of-the-art method on multiple answer retrieval datasets., Comment: Findings of EMNLP 2021(10 pages)

The Conformer model is an excellent architecture for speech recognition modeling that effectively utilizes the hybrid losses of connectionist temporal classification (CTC) and attention to train model parameters. To improve the decoding efficiency of Conformer, we propose a novel connectionist temporal summarization (CTS) method that reduces the number of frames required for the attention decoder fed from the acoustic sequences generated by the encoder, thus reducing operations. However, to achieve such decoding improvements, we must fine-tune model parameters, as cross-attention observations are changed and thus require corresponding refinements. Our final experiments show that, with a beamwidth of 4, the LibriSpeech's decoding budget can be reduced by up to 20% and for FluentSpeech data it can be reduced by 11%, without losing ASR accuracy. An improvement in accuracy is even found for the LibriSpeech "test-other" set. The word error rate (WER) is reduced by 6\% relative at the beam width of 1 and by 3% relative at the beam width of 4., Comment: Submitted to INTERSPEECH 2022 (5 pages, 2 figures)

SLU combines ASR and NLU capabilities to accomplish speech-to-intent understanding. In this paper, we compare different ways to combine ASR and NLU, in particular using a single Conformer model with different ways to use its components, to better understand the strengths and weaknesses of each approach. We find that it is not necessarily a choice between two-stage decoding and end-to-end systems which determines the best system for research or application. System optimization still entails carefully improving the performance of each component. It is difficult to prove that one direction is conclusively better than the other. In this paper, we also propose a novel connectionist temporal summarization (CTS) method to reduce the length of acoustic encoding sequences while improving the accuracy and processing speed of end-to-end models. This method achieves the same intent accuracy as the best two-stage SLU recognition with complicated and time-consuming decoding but does so at lower computational cost. This stacked end-to-end SLU system yields an intent accuracy of 93.97% for the SmartLights far-field set, 95.18% for the close-field set, and 99.71% for FluentSpeech., Comment: Submitted to INTERSPEECH 2022. (5 pages, 1 figure.)

In this work, we address the Text-to-Speech (TTS) task by proposing a non-autoregressive architecture called EfficientTTS. Unlike the dominant non-autoregressive TTS models, which are trained with the need of external aligners, EfficientTTS optimizes all its parameters with a stable, end-to-end training procedure, while allowing for synthesizing high quality speech in a fast and efficient manner. EfficientTTS is motivated by a new monotonic alignment modeling approach (also introduced in this work), which specifies monotonic constraints to the sequence alignment with almost no increase of computation. By combining EfficientTTS with different feed-forward network structures, we develop a family of TTS models, including both text-to-melspectrogram and text-to-waveform networks. We experimentally show that the proposed models significantly outperform counterpart models such as Tacotron 2 and Glow-TTS in terms of speech quality, training efficiency and synthesis speed, while still producing the speeches of strong robustness and great diversity. In addition, we demonstrate that proposed approach can be easily extended to autoregressive models such as Tacotron 2., Comment: 15 pages, 9 figures

Optically induced Mie resonances in dielectric nanoantennas feature low dissipative losses and large resonant enhancement of both electric and magnetic fields. They offer an alternative platform to plasmonic resonances to study light-matter interactions from the weak to the strong coupling regimes. Here, we experimentally demonstrate the strong coupling of bright excitons in monolayer WS$_2$ with Mie surface lattice resonances (Mie-SLRs). We resolve both electric and magnetic Mie-SLRs of a Si nanoparticle array in angular dispersion measurements. At the zero detuning condition, the dispersion of electric Mie-SLRs (e-SLRs) exhibits a clear anti-crossing and a Rabi-splitting of 32 meV between the upper and lower polariton bands. The magnetic Mie-SLRs (m-SLRs) nearly cross the energy band of excitons. These results suggest that the field of m-SLRs is dominated by out-of-plane components that do not efficiently couple with the in-plane excitonic dipoles of the monolayer WS$_2$. In contrast, e-SLRs in dielectric nanoparticle arrays with relatively high quality factors (Q $\sim$ 120) facilitate the formation of collective Mie exciton-polaritons, and may allow the development of novel polaritonic devices which can tailor the optoelectronic properties of atomically thin two-dimensional semiconductors., Comment: 27 pages, 7 figures

Excitons spread through diffusion and interact through exciton-exciton annihilation. Nanophotonics can counteract the resulting decrease in light emission. However, conventional enhancement treats emitters as immobile and noninteracting. Here, we go beyond the localized Purcell effect to exploit exciton dynamics. As interacting excitons diffuse through optical hotspots, the balance of excitonic and nanophotonic properties leads to either enhanced or suppressed photoluminescence. We identify the dominant enhancement mechanisms in the limits of high and low diffusion and annihilation to turn their detrimental impact into additional emission. Our guidelines are relevant for efficient and high-power light-emitting diodes and lasers based on monolayer semiconductors, perovskites, or organic crystals.

Controlling the momentum of carriers in semiconductors, known as valley polarization, is a new resource for optoelectronics and information technologies. Materials exhibiting high polarization are needed for valley-based devices. Few-layer WS$_2$ shows a remarkable spin-valley polarization above 90%, even at room temperature. In stark contrast, polarization is absent for few-layer WSe$_2$ despite the expected material similarities. Here, we explain the origin of valley polarization in both materials based on the interplay between two indirect optical transitions. We show that the relative energy minima at the $\Lambda$- and K-valleys in the conduction band determine the spin-valley polarization of the direct K-K transition. Polarization appears as the energy of the K-valley rises above the $\Lambda$-valley as a function of temperature and number of layers. Our results advance the understanding of the high spin-valley polarization in WS$_2$. This insight will impact the design of both passive and tunable valleytronic devices operating at room temperature., Comment: 36 pages, including supplementary information

One-Shot methods have evolved into one of the most popular methods in Neural Architecture Search (NAS) due to weight sharing and single training of a supernet. However, existing methods generally suffer from two issues: predetermined number of channels in each layer which is suboptimal; and model averaging effects and poor ranking correlation caused by weight coupling and continuously expanding search space. To explicitly address these issues, in this paper, a Broadening-and-Shrinking One-Shot NAS (BS-NAS) framework is proposed, in which `broadening' refers to broadening the search space with a spring block enabling search for numbers of channels during training of the supernet; while `shrinking' refers to a novel shrinking strategy gradually turning off those underperforming operations. The above innovations broaden the search space for wider representation and then shrink it by gradually removing underperforming operations, followed by an evolutionary algorithm to efficiently search for the optimal architecture. Extensive experiments on ImageNet illustrate the effectiveness of the proposed BS-NAS as well as the state-of-the-art performance., Comment: 14 pages

Owing to its unique literal and aesthetical characteristics, automatic generation of Chinese poetry is still challenging in Artificial Intelligence, which can hardly be straightforwardly realized by end-to-end methods. In this paper, we propose a novel iterative polishing framework for highly qualified Chinese poetry generation. In the first stage, an encoder-decoder structure is utilized to generate a poem draft. Afterwards, our proposed Quality-Aware Masked Language Model (QAMLM) is employed to polish the draft towards higher quality in terms of linguistics and literalness. Based on a multi-task learning scheme, QA-MLM is able to determine whether polishing is needed based on the poem draft. Furthermore, QAMLM is able to localize improper characters of the poem draft and substitute with newly predicted ones accordingly. Benefited from the masked language model structure, QAMLM incorporates global context information into the polishing process, which can obtain more appropriate polishing results than the unidirectional sequential decoding. Moreover, the iterative polishing process will be terminated automatically when QA-MLM regards the processed poem as a qualified one. Both human and automatic evaluation have been conducted, and the results demonstrate that our approach is effective to improve the performance of encoder-decoder structure., Comment: accepted by AAAI-2020

Margin infused relaxed algorithms (MIRAs) dominate model tuning in statistical machine translation in the case of large scale features, but also they are famous for the complexity in implementation. We introduce a new method, which regards an N-best list as a permutation and minimizes the Plackett-Luce loss of ground-truth permutations. Experiments with large-scale features demonstrate that, the new method is more robust than MERT; though it is only matchable with MIRAs, it has a comparatively advantage, easier to implement.

List-wise based learning to rank methods are generally supposed to have better performance than point- and pair-wise based. However, in real-world applications, state-of-the-art systems are not from list-wise based camp. In this paper, we propose a new non-linear algorithm in the list-wise based framework called ListMLE, which uses the Plackett-Luce (PL) loss. Our experiments are conducted on the two largest publicly available real-world datasets, Yahoo challenge 2010 and Microsoft 30K. This is the first time in the single model level for a list-wise based system to match or overpass state-of-the-art systems in real-world datasets.

In learning to rank area, industry-level applications have been dominated by gradient boosting framework, which fits a tree using least square error principle. While in classification area, another tree fitting principle, weighted least square error, has been widely used, such as LogitBoost and its variants. However, there is a lack of analysis on the relationship between the two principles in the scenario of learning to rank. We propose a new principle named least objective loss based error that enables us to analyze the issue above as well as several important learning to rank models. We also implement two typical and strong systems and conduct our experiments in two real-world datasets. Experimental results show that our proposed method brings moderate improvements over least square error principle.

We demonstrate experimentally an enhanced delayed fluorescence in tetracene single crystals strongly coupled to optical modes in open cavities formed by arrays of plasmonic nanoparticles. Hybridization of singlet excitons with collective plasmonic resonances in the arrays leads to the splitting of the material dispersion into a lower and an upper polariton band. This splitting significantly modifies the dynamics of the photo-excited tetracene crystal, resulting in an increase of the delayed fluorescence by a factor of four. The enhanced delayed fluorescence is attributed to the emergence of an additional radiative decay channel, where the lower polariton band harvests long-lived triplet states. There is also an increase in total emission, which is wavelength dependent, and can be explained by the direct emission from the lower polariton band, the more effcient light out-coupling and the enhancement of the excitation intensity. The observed enhanced fluorescence opens the possibility of effcient radiative triplet harvesting in open optical cavities, to improve the performance of organic light emitting diodes.

In this paper, we comprehensively study on context-aware generation of Chinese song lyrics. Conventional text generative models generate a sequence or sentence word by word, failing to consider the contextual relationship between sentences. Taking account into the characteristics of lyrics, a hierarchical attention based Seq2Seq (Sequence-to-Sequence) model is proposed for Chinese lyrics generation. With encoding of word-level and sentence-level contextual information, this model promotes the topic relevance and consistency of generation. A large Chinese lyrics corpus is also leveraged for model training. Eventually, results of automatic and human evaluations demonstrate that our model is able to compose complete Chinese lyrics with one united topic constraint., Comment: accepted by The 16th Pacific Rim International Conference on AI

In recent years, deep learning technique has received intense attention owing to its great success in image recognition. A tendency of adaption of deep learning in various information processing fields has formed, including music information retrieval (MIR). In this paper, we conduct a comprehensive study on music audio classification with improved convolutional neural networks (CNNs). To the best of our knowledge, this the first work to apply Densely Connected Convolutional Networks (DenseNet) to music audio tagging, which has been demonstrated to perform better than Residual neural network (ResNet). Additionally, two specific data augmentation approaches of time overlapping and pitch shifting have been proposed to address the deficiency of labelled data in the MIR. Moreover, an ensemble learning of stacking is employed based on SVM. We believe that the proposed combination of strong representation of DenseNet and data augmentation can be adapted to other audio processing tasks., Comment: accepted by The 16th Pacific Rim International Conference on AI

This paper presents a novel, syllable-structured Chinese lyrics generation model given a piece of original melody. Most previously reported lyrics generation models fail to include the relationship between lyrics and melody. In this work, we propose to interpret lyrics-melody alignments as syllable structural information and use a multi-channel sequence-to-sequence model with considering both phrasal structures and semantics. Two different RNN encoders are applied, one of which is for encoding syllable structures while the other for semantic encoding with contextual sentences or input keywords. Moreover, a large Chinese lyrics corpus for model training is leveraged. With automatic and human evaluations, results demonstrate the effectiveness of our proposed lyrics generation model. To the best of our knowledge, there is few previous reports on lyrics generation considering both music and linguistic perspectives., Comment: accepted by The 16th Pacific Rim International Conference on AI

As one of the quintessence of Chinese traditional culture, couplet compromises two syntactically symmetric clauses equal in length, namely, an antecedent and subsequent clause. Moreover, corresponding characters and phrases at the same position of the two clauses are paired with each other under certain constraints of semantic and/or syntactic relatedness. Automatic couplet generation is recognized as a challenging problem even in the Artificial Intelligence field. In this paper, we comprehensively study on automatic generation of acrostic couplet with the first characters defined by users. The complete couplet generation is mainly divided into three stages, that is, antecedent clause generation pipeline, subsequent clause generation pipeline and clause re-ranker. To realize semantic and/or syntactic relatedness between two clauses, attention-based Sequence-to-Sequence (S2S) neural network is employed. Moreover, to provide diverse couplet candidates for re-ranking, a cluster-based beam search approach is incorporated into the S2S network. Both BLEU metrics and human judgments have demonstrated the effectiveness of our proposed method. Eventually, a mini-program based on this generation system is developed and deployed on Wechat for real users., Comment: accepted by The 16th Pacific Rim International Conference on AI

We demonstrate the strong coupling of direct transition excitons in tungsten disulfide (WS2) with collective plasmonic resonances at room temperature. We use open plasmonic cavities formed by periodic arrays of metallic nanoparticles. We show clear anti-crossings with monolayer, bilayer and thicker multilayer WS2 on top of the nanoparticle array. The Rabi energy of such hybrid system varies from 50 to 110 meV from monolayer to sixteen layers, while it does not scale with the square root of the number of layers as expected for collective strong coupling. We prove that out-of-plane coupling components can be disregarded since the normal field is screened due to the high refractive index contrast of the dielectric layers. Even though the in-plane dipole moments of the excitons decrease beyond monolayers, the strong in-plane field distributed in the flake can still enhance the coupling strength with multilayers. However, the screened out-of-plane field leads to the saturation of the Rabi energy. The achieved coherent coupling of TMD multilayers with open cavities could be exploited for manipulating the dynamics and transport of excitons in 2D semiconductors and developing ultrafast valley/spintronic devices.

Recurrent neural language models are the state-of-the-art models for language modeling. When the vocabulary size is large, the space taken to store the model parameters becomes the bottleneck for the use of recurrent neural language models. In this paper, we introduce a simple space compression method that randomly shares the structured parameters at both the input and output embedding layers of the recurrent neural language models to significantly reduce the size of model parameters, but still compactly represent the original input and output embedding layers. The method is easy to implement and tune. Experiments on several data sets show that the new method can get similar perplexity and BLEU score results while only using a very tiny fraction of parameters., Comment: To appear at AAAI 2018

We demonstrate room temperature chiral strong coupling of valley excitons in a transition metal dichalcogenide monolayer with spin-momentum locked surface plasmons. In this regime, we measure spin-selective excitation of directional flows of polaritons. Operating under strong light-matter coupling, our platform yields robust intervalley contrasts and coherences, enabling us to generate coherent superpositions of chiral polaritons propagating in opposite directions. Our results reveal the rich and easy to implement possibilities offered by our system in the context of chiral optical networks., Comment: 5 pages, 4 figures, plus Supporting Information