Search

Word embeddings such as ELMo and BERT have been shown to model word usage in language with greater efficacy through contextualized learning on large-scale language corpora, resulting in significant performance improvement across many natural language processing tasks. In this work we integrate acoustic information into contextualized lexical embeddings through the addition of a parallel stream to the bidirectional language model. This multimodal language model is trained on spoken language data that includes both text and audio modalities. We show that embeddings extracted from this model integrate paralinguistic cues into word meanings and can provide vital affective information by applying these multimodal embeddings to the task of speaker emotion recognition.

Breakthroughs in transformer-based models have revolutionized not only the NLP field, but also vision and multimodal systems. However, although visualization and interpretability tools have become available for NLP models, internal mechanisms of vision and multimodal transformers remain largely opaque. With the success of these transformers, it is increasingly critical to understand their inner workings, as unraveling these black-boxes will lead to more capable and trustworthy models. To contribute to this quest, we propose VL-InterpreT, which provides novel interactive visualizations for interpreting the attentions and hidden representations in multimodal transformers. VL-InterpreT is a task agnostic and integrated tool that (1) tracks a variety of statistics in attention heads throughout all layers for both vision and language components, (2) visualizes cross-modal and intra-modal attentions through easily readable heatmaps, and (3) plots the hidden representations of vision and language tokens as they pass through the transformer layers. In this paper, we demonstrate the functionalities of VL-InterpreT through the analysis of KD-VLP, an end-to-end pretraining vision-language multimodal transformer-based model, in the tasks of Visual Commonsense Reasoning (VCR) and WebQA, two visual question answering benchmarks. Furthermore, we also present a few interesting findings about multimodal transformer behaviors that were learned through our tool., Comment: Best Demo Award at CVPR 2022

Suicide is a major societal challenge globally, with a wide range of risk factors, from individual health, psychological and behavioral elements to socio-economic aspects. Military personnel, in particular, are at especially high risk. Crisis resources, while helpful, are often constrained by access to clinical visits or therapist availability, especially when needed in a timely manner. There have hence been efforts on identifying whether communication patterns between couples at home can provide preliminary information about potential suicidal behaviors, prior to intervention. In this work, we investigate whether acoustic, lexical, behavior and turn-taking cues from military couples' conversations can provide meaningful markers of suicidal risk. We test their effectiveness in real-world noisy conditions by extracting these cues through an automatic diarization and speech recognition front-end. Evaluation is performed by classifying 3 degrees of suicidal risk: none, ideation, attempt. Our automatic system performs significantly better than chance in all classification scenarios and we find that behavior and turn-taking cues are the most informative ones. We also observe that conditioning on factors such as speaker gender and topic of discussion tends to improve classification performance., submitted to ICASSP 2020

In this work we analyze the importance of lexical and acoustic modalities in behavioral expression and perception. We demonstrate that this importance relates to the amount of therapy, and hence communication training, that a person received. It also exhibits some relationship to gender. We proceed to provide an analysis on couple therapy data by splitting the data into clusters based on gender or stage in therapy. Our analysis demonstrates the significant difference between optimal modality weights per cluster and relationship to therapy stage. Given this finding we propose the use of communication-skill aware fusion models to account for these differences in modality importance. The fusion models operate on partitions of the data according to the gender of the speaker or the therapy stage of the couple. We show that while most multimodal fusion methods can improve mean absolute error of behavioral estimates, the best results are given by a model that considers the degree of communication training among the interlocutors.

Appropriate embedding transformation of sentences can aid in downstream tasks such as NLP and emotion and behavior analysis. Such efforts evolved from word vectors which were trained in an unsupervised manner using large-scale corpora. Recent research, however, has shown that sentence embeddings trained using in-domain data or supervised techniques, often through multitask learning, perform better than unsupervised ones. Representations have also been shown to be applicable in multiple tasks, especially when training incorporates multiple information sources. In this work we aspire to combine the simplicity of using abundant unsupervised data with transfer learning by introducing an online multitask objective. We present a multitask paradigm for unsupervised learning of sentence embeddings which simultaneously addresses domain adaption. We show that embeddings generated through this process increase performance in subsequent domain-relevant tasks. We evaluate on the affective tasks of emotion recognition and behavior analysis and compare our results with state-of-the-art general-purpose supervised sentence embeddings. Our unsupervised sentence embeddings outperform the alternative universal embeddings in both identifying behaviors within couples therapy and in emotion recognition.

The worldwide population of people over the age of 65 has been predicted to more than double from 1990 to 2025. Therefore, ubiquitous health-care systems have become an important topic of research in recent years. In this paper, an integrated system for portable electrocardiography (ECG) monitoring, with an on-board processor for time-frequency analysis of heart rate variability (HRV), is presented. The main function of proposed system comprises three parts, namely, an analog-to-digital converter (ADC) controller, an HRV processor, and a lossless compression engine. At the beginning, ECG data acquired from front-end circuits through the ADC controller is passed through the HRV processor for analysis. Next, the HRV processor performs real-time analysis of time-frequency HRV using the Lomb periodogram and a sliding window configuration. The Lomb periodogram is suited for spectral analysis of unevenly sampled data and has been applied to time-frequency analysis of HRV in the proposed system. Finally, the ECG data are compressed by 2.5 times using the lossless compression engine before output using universal asynchronous receiver/transmitter (UART). Bluetooth is employed to transmit analyzed HRV data and raw ECG data to a remote station for display or further analysis. The integrated ECG health-care system design proposed has been implemented using UMC 90nm CMOS technology.

In this paper, a portable ECG processor designed for health monitoring applications is proposed. The ECG processor acquires three-channel ECG raw data through a front-end circuit, and measures the time between successive heart beats on lead II as RR intervals for heart rate variability (HRV) analysis. Functions such as R-peak detection, RR interval calculation, sliding memory window scheme, and time-frequency analysis of HRV have also been developed. A real-time HRV analysis processor is realized by employing a Lomb periodogram for time-frequency power spectral density (PSD) analysis of the heart rate. The Lomb time-frequency distribution (TFD) is suited for deriving the PSD of unevenly spaced data sets. The system has been implemented in hardware and verified on FPGA.

In this paper, a highly-integrated multiprocessor chip design enabling the real-time processing of biomedical signals in portable brain-heart monitoring systems is presented. The architecture comprises a novel diffuse optical tomography (DOT) processor for taking brain imaging, an independent component analysis (ICA) processor for removing artifacts of brain electroencephalogram (EEG) signals, and a heart rate variability (HRV) analysis processor for monitoring heart electrocardiogram (ECG) signals. The multiprocessor chip implemented in 65nm CMOS technology comprises 368k gates and occupies a core area of 462k µm2. Simulated power consumption using a full operation test case reports 3.6mW under the condition of 1.0V core supply voltage and 24MHz clock operating frequency.

This paper presents an EKG system-on-chip (SOC) for portable health care and home monitoring applications. The EKG system acquires three channel EKG from front-end circuits and includes functions such as beat detection, interval calculation, and time-frequency analysis of heart rate variability (HRV) in real-time. An HRV analysis engine has also been developed using Lomb periodogram for time-frequency power spectral density (PSD) analysis of heart rate. HRV analysis as well as raw data can be transmitted via Bluetooth to a cell phone or remote station through a UART interface.

In this paper, a system for time-frequency analysis of heart rate variability (HRV) using a fast windowed Lomb periodogram is proposed. Time-frequency analysis of HRV is achieved through a de-normalized fast Lomb periodogram with a sliding window configuration. The Lomb time-frequency distribution (TFD) is suited for spectral analysis of unevenly spaced data and has been applied to the analysis of HRV. The system has been implemented in hardware as an HRV processor and verified on FPGA. Simulations show that the proposed Lomb TFD is able to achieve better frequency resolution than short-time Fourier transform of the same hardware size. The proposed system is suitable for portable monitoring devices and as a biomedical signal processor on an system-on-chip (SOC) design.

This paper presents a complexity-efficient architecture for an EEG signal separation processor incorporating ICA with lossless data compression. An average correlation result of 0.9044 is achieved while transmitted EEG data bandwidth and power consumption are reduced by 41.6%. The chip area, operating frequency, and estimated power consumption of the proposed EEG architecture in UMC 90nm SP-HVT CMOS technology are 1,133 by 1,133 um2, up to 32MHz, and approximately 0.70mW at 0.9V supply voltage and 5 MHz operating frequency, respectively.

In this paper, an overview of a brain-heart monitoring system is first given. The latest development in miniature brain-heart monitoring system for emerging health applications is highlighted. Finally, the development of a low power biomedical signal processing and image reconstruction SoC design is presented. The significance of this SoC is to enable practical developments of portable real-time brain-heart monitoring systems. The proposed architecture comprises a novel functional near-infrared (fNIR) diffuse optical tomography system for brain imaging, an independent component analysis (ICA) processor for electroencephalogram (EEG) signal analysis, and a heart rate variability (HRV) analysis processor for electrocardiogram (ECG) signal analysis. Biomedical signals acquired from front-end sensor modules are processed in real-time or bypassed according to user settings. The processed data or biomedical signals is then losslessly compressed and sent to a remote science station for further analysis and 3D visualization. The final SoC is fabricated in UMC 90nm CMOS technology.

A portable brain-heart monitoring system is proposed to integrate and miniaturize those heavy equipments in the hospitals. The system comprises a 4-channel independent component analysis (ICA) engine for artifact removal from EEG, a heart-rate variability (HRV) analysis engine for on-line HRV analysis and a diffuse optical tomography (DOT) engine for reconstruction of the absorption coefficient image of the brain tissue. A lossless compression module achieves 2.5 compression ratio is also employed to reduce the power consumption of the wireless transmission. EEG, EKG and near-infrared signals acquired from the analog front-end IC are processed in real-time or bypassed according to user configurations. Processed data and raw data are compressed and sent to a remote science station by a commercial Bluetooth module for further analysis and 3-D visualization and remote diagnosis. The ICA and HRV engine are verified by real EEG and EKG signals while the DOT engine is verified by an experimental model. The system is implemented using UMC 65nm CMOS technology, and the core size is 680x680 um2, and the estimated power consumption of the chip working at 24 MHz under full mode is 3.6 mW.

In this paper, a system for time-frequency analysis of heart rate variability (HRV) using windowed Lomb periodogram is proposed. The system is designed with considerations in SOC implementation for portable applications. Time-frequency analysis of HRV is achieved through a de-normalized Lomb periodogram with a sliding window configuration. The Lomb time-frequency distribution (TFD) is suited for power spectral density (PSD) analysis of unevenly spaced data and has been applied to the analysis of heart rate variability. The system has been implemented in hardware as an HRV processor and verified on FPGA. Artificial heart rate was used to evaluate the system as well as data from the MIT-BIH arrhythmia database and real EKG data. Simulations show that the proposed Lomb TFD is able to achieve better frequency resolution than short-time Fourier transform of the same hardware size.

The advancements of wireless body area networks (WBAN) and wireless personal area networks (WPAN) has led to a recent increase of viable applications in wireless medical and healthcare devices. Developing biomedical sensors such as electroencephalograph (EEG) and electrocardiogram (ECG) sensors often require numerous connecting wires which may introduce noise and increase patient discomfort. In this paper we propose a system design and realization of a wireless EEG and ECG sensor network focusing on issues such as time synchronization, bandwidth, and power constraints constituent of WBANs. Our WSN comprises three transmitting nodes for a total of four EEG channels and an ECG channel. We solve problems such as data throughput requirements for EEG and ECG signal processing as well as time synchronization of received data at the base station. This paper keeps in consideration the possible implementation of our proposed system onto a system-on-chip (SOC) by putting focus on the chip size and low power consumption of the analog-front-end system.

In this paper, a contention free algorithm for solving memory collision problem of parallel Turbo decoder architecture using the simulated annealing algorithm is presented. Furthermore, we proposed two area-efficient extrinsic memory schemes based on the parallel contention free Turbo decoder. One of the proposed schemes employs only multiple single port memories with one temporary buffer instead of the original dual port or two port memories and the other scheme further employs an additional non-linear extrinsic mapping architecture. The proposed schemes lead to approximately 37% and 46% memory area reduction, respectively, for 16-parallel Turbo decoder in comparison to the conventional dual port memory scheme under the UMC 0.13µm CMOS process.