Search

This paper works on streaming automatic speech recognition (ASR). Mamba, a recently proposed state space model, has demonstrated the ability to match or surpass Transformers in various tasks while benefiting from a linear complexity advantage. We explore the efficiency of Mamba encoder for streaming ASR and propose an associated lookahead mechanism for leveraging controllable future information. Additionally, a streaming-style unimodal aggregation (UMA) method is implemented, which automatically detects token activity and streamingly triggers token output, and meanwhile aggregates feature frames for better learning token representation. Based on UMA, an early termination (ET) method is proposed to further reduce recognition latency. Experiments conducted on two Mandarin Chinese datasets demonstrate that the proposed model achieves competitive ASR performance in terms of both recognition accuracy and latency., Comment: Submitted to ICASSP 2025

The training of deep learning-based multichannel speech enhancement and source localization systems relies heavily on the simulation of room impulse response and multichannel diffuse noise, due to the lack of large-scale real-recorded datasets. However, the acoustic mismatch between simulated and real-world data could degrade the model performance when applying in real-world scenarios. To bridge this simulation-to-real gap, this paper presents a new relatively large-scale Real-recorded and annotated Microphone Array speech&Noise (RealMAN) dataset. The proposed dataset is valuable in two aspects: 1) benchmarking speech enhancement and localization algorithms in real scenarios; 2) offering a substantial amount of real-world training data for potentially improving the performance of real-world applications. Specifically, a 32-channel array with high-fidelity microphones is used for recording. A loudspeaker is used for playing source speech signals (about 35 hours of Mandarin speech). A total of 83.7 hours of speech signals (about 48.3 hours for static speaker and 35.4 hours for moving speaker) are recorded in 32 different scenes, and 144.5 hours of background noise are recorded in 31 different scenes. Both speech and noise recording scenes cover various common indoor, outdoor, semi-outdoor and transportation environments, which enables the training of general-purpose speech enhancement and source localization networks. To obtain the task-specific annotations, speaker location is annotated with an omni-directional fisheye camera by automatically detecting the loudspeaker. The direct-path signal is set as the target clean speech for speech enhancement, which is obtained by filtering the source speech signal with an estimated direct-path propagation filter., Comment: accepted by NIPS 2024

Spiking neural networks (SNNs) have garnered interest due to their energy efficiency and superior effectiveness on neuromorphic chips compared with traditional artificial neural networks (ANNs). One of the mainstream approaches to implementing deep SNNs is the ANN-SNN conversion, which integrates the efficient training strategy of ANNs with the energy-saving potential and fast inference capability of SNNs. However, under extreme low-latency conditions, the existing conversion theory suggests that the problem of misrepresentation of residual membrane potentials in SNNs, i.e., the inability of IF neurons with a reset-by-subtraction mechanism to respond to residual membrane potentials beyond the range from resting potential to threshold, leads to a performance gap in the converted SNNs compared to the original ANNs. This severely limits the possibility of practical application of SNNs on delay-sensitive edge devices. Existing conversion methods addressing this problem usually involve modifying the state of the conversion spiking neurons. However, these methods do not consider their adaptability and compatibility with neuromorphic chips. We propose a new approach based on explicit modeling of residual errors as additive noise. The noise is incorporated into the activation function of the source ANN, which effectively reduces the residual error. Our experiments on the CIFAR10/100 dataset verify that our approach exceeds the prevailing ANN-SNN conversion methods and directly trained SNNs concerning accuracy and the required time steps. Overall, our method provides new ideas for improving SNN performance under ultra-low-latency conditions and is expected to promote practical neuromorphic hardware applications for further development.

Since the implementation of the Materials Genome Project by the Obama administration in the United States, the development of various computational materials databases has fundamentally expanded the choices of industries such as materials and energy. In the field of thermoelectric materials, the thermoelectric figure of merit ZT quantifies the performance of the material. From the viewpoint of calculations for vast materials, the ZT values are not easily obtained due to their computational complexity. Here, we show how to build a database of thermoelectric materials based on first-principles calculations for the electronic and heat transport of materials. Firstly, the initial structures are classified according to the values of bandgap and other basic properties using the clustering algorithm K-means in machine learning, and high-throughput first principles calculations are carried out for narrow-bandgap semiconductors which exhibiting potential thermoelectric application. The present framework of calculations mainly includes deformation potential module, electrical transport performance module, mechanical and thermodynamic properties module. We have also set up a search webpage for the calculated database of thermoelectric materials, providing searching and viewing the related physical properties of materials. Our work may inspire the construction of more computational databases of first-principle thermoelectric materials and accelerate research progress in the field of thermoelectrics., Comment: 13 pages, 5 figures

In this work, we propose Mel-FullSubNet, a single-channel Mel-spectrogram denoising and dereverberation network for improving both speech quality and automatic speech recognition (ASR) performance. Mel-FullSubNet takes as input the noisy and reverberant Mel-spectrogram and predicts the corresponding clean Mel-spectrogram. The enhanced Mel-spectrogram can be either transformed to speech waveform with a neural vocoder or directly used for ASR. Mel-FullSubNet encapsulates interleaved full-band and sub-band networks, for learning the full-band spectral pattern of signals and the sub-band/narrow-band properties of signals, respectively. Compared to linear-frequency domain or time-domain speech enhancement, the major advantage of Mel-spectrogram enhancement is that Mel-frequency presents speech in a more compact way and thus is easier to learn, which will benefit both speech quality and ASR. Experimental results demonstrate a significant improvement in both speech quality and ASR performance achieved by the proposed model.

This paper works on non-autoregressive automatic speech recognition. A unimodal aggregation (UMA) is proposed to segment and integrate the feature frames that belong to the same text token, and thus to learn better feature representations for text tokens. The frame-wise features and weights are both derived from an encoder. Then, the feature frames with unimodal weights are integrated and further processed by a decoder. Connectionist temporal classification (CTC) loss is applied for training. Compared to the regular CTC, the proposed method learns better feature representations and shortens the sequence length, resulting in lower recognition error and computational complexity. Experiments on three Mandarin datasets show that UMA demonstrates superior or comparable performance to other advanced non-autoregressive methods, such as self-conditioned CTC. Moreover, by integrating self-conditioned CTC into the proposed framework, the performance can be further noticeably improved., Comment: Accepted by ICASSP 2024

One stream of reinforcement learning research is exploring biologically plausible models and algorithms to simulate biological intelligence and fit neuromorphic hardware. Among them, reward-modulated spike-timing-dependent plasticity (R-STDP) is a recent branch with good potential in energy efficiency. However, current R-STDP methods rely on heuristic designs of local learning rules, thus requiring task-specific expert knowledge. In this paper, we consider a spiking recurrent winner-take-all network, and propose a new R-STDP method, spiking variational policy gradient (SVPG), whose local learning rules are derived from the global policy gradient and thus eliminate the need for heuristic designs. In experiments of MNIST classification and Gym InvertedPendulum, our SVPG achieves good training performance, and also presents better robustness to various kinds of noises than conventional methods., Comment: Accepted to BMVC 2022

A common goal of Intelligent Tutoring Systems (ITS) is to provide learning environments that adapt to the varying abilities and characteristics of users. This type of adaptivity is possible only if the ITS has information that characterizes the learning behaviors of its users and can adjust its pedagogy accordingly. This study investigated an intelligent tutoring system with computer agents ("AutoTutor") designed to improve comprehension skills in adults with low reading literacy. One goal of this study was to classify adults into different clusters based on their behavioral patterns (accuracy and response time to answer questions) while they interacted with AutoTutor to help them improve their reading comprehension skills. A second goal was to investigate whether adults' behaviors were associated with different reading components. A third goal was to assess improvements in reading comprehension skills, based on psychometric tests, of different clusters of readers. Performance on AutoTutor was collected in a targeted 100-hour hybrid intervention for adult readers (n = 252) that included both human teachers and the AutoTutor system. The adults' average accuracy and response time in AutoTutor were used to cluster the adults into four categories: higher performers (comparatively fast and accurate), conscientious readers (slow but accurate), under-engaged readers (fast at the expense of somewhat lower accuracy) and struggling readers (slow and inaccurate). Two psychometric tests of comprehension were used to assess comprehension. Gains in comprehension scores were highest for conscientious readers, lowest for struggling readers, with higher performing readers and under-engaged readers in between. The results provide guidance to enhance the adaptivity of AutoTutor. [This is the online version of an article published in "International Journal of Artificial Intelligence in Education."]

The booming haptic data significantly improves the users'immersion during multimedia interaction. As a result, the study of Haptic,Audio-Visual Environment(HAVE)has attracted attentions of multimedia community. To realize such a system, a challenging tack is the synchronization of multiple sensorial signals that is critical to user experience. Despite of audio-visual synchronization efforts, there is still a lack of haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic-visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a haptic-visual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics, and a key frame extraction of visual signals based on deep object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic-visual signals. Without timestamps in signal flow, the proposed method is still effective and more robust to complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.

Artificial Intelligence (AI) based assessments are commonly used in a variety of settings including business, healthcare, policing, manufacturing, and education. In education, AI-based assessments undergird intelligent tutoring systems as well as many tools used to evaluate students and, in turn, guide learning and instruction. This chapter provides a framework for AI-based Assessment (AIBA) in Education that describes systems and tools. The framework comprises five interrelated dimensions that broadly address the purposes and procedures of AIBA for educational technologies: purposes (i.e., summative feedback, formative feedback, training, and instruction), constructs (e.g., knowledge, skills, strategies, and emotion), data sources (e.g., task performance, behaviors, language, and multimodal data), computational methods (e.g., Bayesian statistics, machine learning, and natural language processing), and visibility (e.g., task-related vs. stealth). This chapter describes systems and tools used to provide AIBA and instantiate these tools within these dimensions. Framing systems within these five dimensions of AIBA enables understanding the landscape of tools and systems, and, in turn, pointing to gaps in the literature and future directions. [This chapter was published in: "Handbook of Artificial Intelligence in Education," edited by Benedict du Boulay et al., Edward Elgar Publishing Limited, 2023, pp. 485-504.]

Standard Model $ZZ$ production is an important background for many searches at the LHC, especially in final states with missing transverse momentum. In this article, boson substitution is applied to estimate $ZZ \rightarrow \ell^+ \ell^- \nu \bar{\nu}$ yields from $Z\gamma \rightarrow \ell^+ \ell^- \gamma$ events. The cross-section ratio of the two processes and its uncertainties are evaluated at NNLO with the MATRIX generator as a function of the transverse momentum of the substituted boson. Uncertainties due to higher-order QCD corrections, parton distribution functions, photon isolation criteria, and electroweak corrections are evaluated. They depend strongly on the applied event selections and the considered transverse momentum range. For minimal selections, their size is 3$-$4%, dominated by QCD-related uncertainties for transverse momenta below 500 GeV and by uncertainties due to the factorization of QCD and electroweak corrections at higher transverse momenta., Comment: 32 pages, 18 figures

The Interactive Strategy Training for Active Reading and Thinking (iSTART) game-based intelligent tutoring system (ITS) was developed with a foundation of comprehension theory and principles of learning science to improve students' comprehension of complex scientific texts. iSTART has been shown to improve reading comprehension for learners from middle school through adulthood, particularly lower knowledge readers, through strategy instruction and game-based practice. This paper describes iSTART, the theoretical foundations that have guided iSTART development, and evidence for the feasibility of game-based practice to improve learning outcomes. This paper also introduces a novel method of assessing students' reading comprehension through game-based literacy assessments that have been incorporated in iSTART. The development of these stealth assessments was guided by recent work emphasizing the need for rapid, dynamic, and low stakes assessments that evaluate students' reading skills in the context of brief, dynamic games. Stealth assessments can generate estimates of multiple aspects of students' reading comprehension quickly and within a motivating environment. The work described in this paper is a promising method to assess students' literacy in an unobtrusive and authentic way that may lead to improved learning outcomes for students. [This is the online version of an article published in "International Journal of Human-Computer Interaction."]

Reading comprehension is often assessed by having students read passages and administering a test that assesses their understanding of the text. Shorter assessments may fail to give a full picture of comprehension ability while more thorough ones can be time consuming and costly. This study used data from a conversational intelligent tutoring system (AutoTutor) to assess reading comprehension ability in 52 low-literacy adults who interacted with the system. We analyzed participants' accuracy and time spent answering questions in conversations in lessons that targeted four theoretical components of comprehension: Word, Textbase, Situation Model, and Rhetorical Structure. Accuracy and answer response time were analyzed to track adults' proficiency for comprehension components, and we analyzed whether the four components predicted reading grade level. We discuss the results with respect to the advantages that a conversational intelligent tutoring system assessment may provide over traditional assessment tools and the linking of theory to practice in adult literacy. [This is the online version of an article published in "Behaviormetrika."]

To have a superior generalization, a deep learning neural network often involves a large size of training sample. With increase of hidden layers in order to increase learning ability, neural network has potential degradation in accuracy. Both could seriously limit applicability of deep learning in some domains particularly involving predictions of continuous variables with a small size of samples. Inspired by residual convolutional neural network in computer vision and recent findings of crucial shortcuts in the brains in neuroscience, we propose an autoencoder-based residual deep network for robust prediction. In a nested way, we leverage shortcut connections to implement residual mapping with a balanced structure for efficient propagation of error signals. The novel method is demonstrated by multiple datasets, imputation of high spatiotemporal resolution non-randomness missing values of aerosol optical depth, and spatiotemporal estimation of fine particulate matter <2.5 \mu m, achieving the cutting edge of accuracy and efficiency. Our approach is also a general-purpose regression learner to be applicable in diverse domains., Comment: including supplementary materials

Monolayer lateral heterostructure of graphene and hexagonal boron nitride (h-BNC) has attracted a growing attention mainly due to its tunable band-gap character and unique physical properties at interface. Hereby, we reported the first-time synthesis of a nearly freestanding h-BNC hybrid on a weakly coupled substrate of Ir (111), where graphene and h-BN possessing different surface heights and corrugations formed a perfect monolayer hybrid. With the aid of scanning tunneling microscopy/spectroscopy (STM/STS), we demonstrated that h-BN can patch alongside the boundary of pre-deposited graphene domains and vice versa to form a seamless monolayer hybrid, with the realization of predominant zigzag type chiral boundaries at the interface. Density-functional theory calculations and STM/STS measurements aided us to reveal that this interface between graphene and h-BN were atomically sharp in aspects of the chemical bonding as well as the local electronic property from both theoretical and experimental points of view., Comment: 17 pages, 5 figures

We have simultaneously measured conductance and thermoelectric power (TEP) of individual silicon and germanium/silicon core/shell nanowires in the field effect transistor device configuration. As the applied gate voltage changes, the TEP shows distinctly different behaviors while the electrical conductance exhibits the turn-off, subthreshold, and saturation regimes respectively. At room temperature, peak TEP value of $\sim 300 \mu$V/K is observed in the subthreshold regime of the Si devices. The temperature dependence of the saturated TEP values are used to estimate the carrier doping of Si nanowires., Comment: 4 pages, 3 figures

We report low-temperature electrical transport studies of molecule-scale silicon nanowires. Individual nanowires exhibit well-defined Coulomb blockade oscillations characteristic of charge addition to a single nanostructure with length scales up to at least 400 nm. Further studies demonstrate coherent charge transport through discrete single particle quantum levels extending the whole device, and show that the ground state spin configuration follows the Lieb-Mattis theorem. In addition, depletion of the nanowires suggests that phase coherent single-dot characteristics are accessible in a regime where correlations are strong., Comment: 4 pages and 4 figures

A systematic analysis is presented of the $B_{s}\to K$ form factor $f(q^{2}) $ in the whole range of momentum transfer $q^{2}$, which would be useful to analyzing the future data on $B_{s}\to K$ decays and extracting $| V_{ub}|$. With a modified QCD light cone sum rule (LCSR) approach, in which the contributions cancel out from the twist 3 wavefunctions of $K$ meson, we investigate in detail the behavior of $f(q^{2})$ at small and intermediate $q^{2}$ and the nonperturbative quantity $f_{B^{\ast}}g_{B^{\ast}B_{s}K}$ $(f_{B^{\ast}}$ is the decay constant of $B^{\ast}$ meson and $g_{B^{\ast}B_{s}K}$ the $B^{\ast}B_{s}K$ strong coupling), whose numerical result is used to study $q^{2}$ dependence of $f(q^{2})$ at large $q^{2}$ in the single pole approximation. Based on these findings, a form factor model from the best fit is formulated, which applies to the calculation on $f(q^{2})$ in the whole kinematically accessible range. Also, a comparison is made with the standard LCSR predictions., Comment: 11 pages, Latex, 1 eps figure, Final version to appear in Phys.Rev.D