The Spoken Language Processing Lab (SLPL) focuses on the development of AI systems capable of understanding and generating human-like speech and text. Research includes enhancing deep learning models through compression and optimization for resource-limited settings. The lab also develops multimodal processing systems that combine visual, auditory, and textual data to improve interaction and decision-making. The lab is focused on advancing speech recognition and synthesis to enhance human-machine interactions across various languages and dialects and enhancing natural language understanding to better interpret context and nuanced meanings.

고려대학교 SLPL 연구실은 인간과 유사한 음성 및 텍스트를 이해하고 생성하는 AI 시스템 개발에 중점을 둔다. 딥러닝 모델의 압축 및 최적화를 통해 리소스가 제한된 환경에서의 활용성을 높이는 연구를 수행한다. 시각, 청각, 텍스트 데이터를 결합하여 상호 작용 및 의사 결정을 개선하는 멀티모달 처리 시스템을 개발한다. 다양한 언어 및 방언에 걸쳐 인간-기계 상호 작용을 향상시키기 위한 음성 인식 및 합성 기술을 발전시키고, 맥락과 미묘한 의미를 더 잘 해석하기 위한 자연어 이해 능력을 향상시키는 데 주력한다.

Mels-Tts: Multi-Emotion Multi-Lingual Multi-Speaker Text-To-Speech System Via Disentangled Style Tokens

Latent Filling: Latent Space Data Augmentation for Zero-Shot Speech Synthesis

Hierarchical Timbre-Cadence Speaker Encoder for Zero-shot Speech Synthesis

Self-Supervised Accent Learning for Under-Resourced Accents Using Native Language Data

Conformer-Based on-Device Streaming Speech Recognition with KD Compression and Two-Pass Architecture

Macro-Block Dropout for Improved Regularization in Training End-to-End Speech Recognition Models

Power-normalized cepstral coefficients (PNCC) for robust speech recognition

Multichannel signal processing with deep neural networks for automatic speech recognition

Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in Google Home

Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis

Sound source estimation using neural networks

Acoustic Modeling for Google Home

Delta-spectral cepstral coefficients for robust speech recognition

Feature extraction for robust speech recognition based on maximizing the sharpness of the power distribution and on power flooring

Feature extraction for robust speech recognition using a power-law nonlinearity and power-bias subtraction

Mobile device and method for preventing undesired key depression in the same

Apparatus and method for reducing power consumption in a mobile communication terminal

Attention based on-device streaming speech recognition with large speech corpus

End-end Speech-to-Text Translation with Modality Agnostic Meta-Learning

Signal separation for robust speech recognition based on phase difference information obtained in the frequency domain

Nonlinear enhancement of onset for robust speech recognition

Improved vocal tract length perturbation for a state-of-the-art end-to-end speech recognition system

Robust DTW-based recognition algorithm for hand-held consumer devices

A review of on-device fully neural end-to-end automatic speech recognition algorithms

Signal processing for robust speech recognition motivated by auditory processing

Physiologically-motivated synchrony-based processing for robust automatic speech recognition

Richard M. Stern

Dhananjaya Gowda

Michiel Bacchiani

Abhinav Garg

Ananya Misra

Arun Narayanan

Tara Sainath

R. C. Nongpiur

kean chin

Ehsan Variani

Izhak Shafran

Ron J Weiss

Bo Li

Kevin Wilson

Kyungmin Lee

Kshitiz Kumar

Beomseok Lee

Ankur Kumar

Sathish Indurthi

Wonyong Sung

Chanwoo Kim

This paper presents a new feature extraction algorithm called power normalized Cepstral coefficients (PNCC) that is motivated by auditory processing. Major new features of PNCC processing include the use of a power-law nonlinearity that replaces the traditional log nonlinearity used in MFCC coefficients, a noise-suppression algorithm based on asymmetric filtering that suppresses background excitation, and a module that accomplishes temporal masking. We also propose the use of medium-time power analysis in which environmental parameters are estimated over a longer duration than is commonly used for speech, as well as frequency smoothing. Experimental results demonstrate that PNCC processing provides substantial improvements in recognition accuracy compared to MFCC and PLP processing for speech in the presence of various types of additive noise and in reverberant environments, with only …

Multichannel automatic speech recognition (ASR) systems commonly separate speech enhancement, including localization, beamforming, and postfiltering, from acoustic modeling. In this paper, we perform multichannel enhancement jointly with acoustic modeling in a deep neural network framework. Inspired by beamforming, which leverages differences in the fine time structure of the signal at different microphones to filter energy arriving from different directions, we explore modeling the raw time-domain waveform directly. We introduce a neural network architecture, which performs multichannel filtering in the first layer of the network, and show that this network learns to be robust to varying target speaker direction of arrival, performing as well as a model that is given oracle knowledge of the true target speaker direction. Next, we show how performance can be improved by factoring the first layer to separate the …

We describe the structure and application of an acoustic room simulator to generate large-scale simulated data for training deep neural networks for far-field speech recognition. The system simulates millions of different room dimensions, a wide distribution of reverberation time and signal-to-noise ratios, and a range of microphone and sound source locations. We start with a relatively clean training set as the source and artificially create simulated data by randomly sampling a noise configuration for every new training example. As a result, the acoustic model is trained using examples that are virtually never repeated. We evaluate performance of this approach based on room simulation using a factored complex Fast Fourier Transform (CFFT) acoustic model introduced in our earlier work, which uses CFFT layers and LSTM AMs for joint multichannel processing and acoustic modeling. Results show that the simulator-driven approach is quite effective in obtaining large improvements not only in simulated test conditions, but also in real/rerecorded conditions. This room simulation system has been employed in training acoustic models including the ones for the recently released Google Home.

In this paper, we introduce a new algorithm for estimating the signal-to-noise ratio (SNR) of speech signals, called WADASNR (Waveform Amplitude Distribution Analysis). In this algorithm we assume that the amplitude distribution of clean speech can be approximated by the Gamma distribution with a shaping parameter of 0.4, and that an additive noise signal is Gaussian. Based on this assumption, we can estimate the SNR by examining the amplitude distribution of the noisecorrupted speech. We evaluate the performance of the WADA-SNR algorithm on databases corrupted by white noise, background music, and interfering speech. The WADA-SNR algorithm shows significantly less bias and less variability with respect to the type of noise compared to the standard NIST STNR algorithm. In addition, the algorithm is quite computationally efficient.

(57) ABSTRACT A system for estimating the location of a stationary or moving sound source includes multiple microphones, which need not be physically aligned in a linear array or a regular geometric pattern in a given environment, an auralizer that generates auralized multi-channel signals based at least on array-related transfer functions and room impulse responses of the microphones as well as signal labels corresponding to the auralized multi-channel signals, a feature extractor that extracts features from the auralized multi-channel signals for efficient processing, and a neural network that can be trained to estimate the location of the sound source based at least on the features extracted from the auralized multi-channel signals and the corresponding signal labels.

This paper describes the technical and system building advances made to the Google Home multichannel speech recognition system, which was launched in November 2016. Technical advances include an adaptive dereverberation frontend, the use of neural network models that do multichannel processing jointly with acoustic modeling, and Grid-LSTMs to model frequency variations. On the system level, improvements include adapting the model using Google Home specific data. We present results on a variety of multichannel sets. The combination of technical and system advances result in a reduction of WER of 8-28% relative compared to the current production system.

Almost all current automatic speech recognition (ASR) systems conventionally append delta and double-delta cepstral features to static cepstral features. In this work we describe a modified feature-extraction procedure in which the time-difference operation is performed in the spectral domain, rather than the cepstral domain as is generally presently done. We argue that this approach based on "delta-spectral" features is needed because even though delta-cepstral features capture dynamic speech information and generally greatly improve ASR recognition accuracy, they are not robust to noise and reverberation. We support the validity of the delta-spectral approach both with observations about the modulation spectrum of speech and noise, and with objective experiments that document the benefit that the delta-spectral approach brings to a variety of currently popular feature extraction algorithms. We found that the …

This paper presents a new robust feature extraction algorithm based on a modified approach to power bias subtraction combined with applying a threshold to the power spectral density. Power bias level is selected as a level above which the signal power distribution is sharpest. The sharpness is measured using the ratio of arithmetic mean to the geometric mean of medium-duration power. When subtracting this bias level, power flooring is applied to enhance robustness. These new ideas are employed to enhance our recently introduced feature extraction algorithm PNCC (Power Normalized Cepstral Coefficient). While simpler than our previous PNCC, experimental results show that this new PNCC is showing better performance than our previous implementation.

This paper presents a new feature extraction algorithm called Power-Normalized Cepstral Coefficients (PNCC) that is based on auditory processing. Major new features of PNCC processing include the use of a power-law nonlinearity that replaces the traditional log nonlinearity used for MFCC coefficients, and a novel algorithm that suppresses background excitation by estimating SNR based on the ratio of the arithmetic to geometric mean power, and subtracts the inferred background power. Experimental results demonstrate that the PNCC processing provides substantial improvements in recognition accuracy compared to MFCC and PLP processing for various types of additive noise. The computational cost of PNCC is only slightly greater than that of conventional MFCC processing.

Undesired key depression of a mobile device is prevented using a predetermined key activation pattern. This pattern includes successively pressing one or more keys of the mobile device a predetermined number of times. When this occurs, the mobile device converts from a key depression preventing mode to a half-operation mode. During this latter mode, func tions corresponding to key inputs are automatically per formed, provided they are input within a prescribed period of time. Otherwise, the mobile device converts back to the key depression preventing mode. As a result, undesired operation and power consumption of the mobile device is prevented.

Apparatus and method for reducing power consumption in a mobile communication device is accomplished by removing power from the display when the display is not visible to the user of the device. Specifically, contact between the face of the user and the device or between two or more portions of the device is detected to determine that the display is not visible to the user.

In this paper, we present a new on-device automatic speech recognition (ASR) system based on monotonic chunk-wise attention (MoChA) models trained with large (> 10K hours) corpus. We attained around 90% of a word recognition rate for general domain mainly by using joint training of connectionist temporal classifier (CTC) and cross entropy (CE) losses, minimum word error rate (MWER) training, layer-wise pretraining and data augmentation methods. In addition, we compressed our models by more than 3.4 times smaller using an iterative hyper low-rank approximation (LRA) method while minimizing the degradation in recognition accuracy. The memory footprint was further reduced with 8-bit quantization to bring down the final model size to lower than 39 MB. For on-demand adaptation, we fused the MoChA models with statistical n-gram models, and we could achieve a relatively 36% improvement on …

Collecting large amounts of data to train end-to-end Speech Translation (ST) models is more difficult compared to the ASR and MT tasks. Previous studies have proposed the use of transfer learning approaches to overcome the above difficulty. These approaches benefit from weakly supervised training data, such as ASR speech-to-transcript or MT text-to-text translation pairs. However, the parameters in these models are updated independently of each task, which may lead to sub-optimal solutions. In this work, we adopt a meta-learning algorithm to train a modality agnostic multi-task model that transfers knowledge from source tasks=ASR+MT to target task=ST where the ST task severely lacks data. In the meta-learning phase, parameters are updated in such a way that they act as a good ini-tialization point for the target ST task. We evaluate the proposed meta-learning approach for ST tasks on English-German (En …

In this paper, we present a new two-microphone approach that improves speech recognition accuracy when speech is masked by other speech. The algorithm improves on previous systems that have been successful in separating signals based on differences in arrival time of signal components from two microphones. The present algorithm differs from these efforts in that the signal selection takes place in the frequency domain. We observe that additional smoothing of the phase estimates over time and frequency is needed to support adequate speech recognition performance. We demonstrate that the algorithm described in this paper provides better recognition accuracy than timedomain-based signal separation algorithms, and at less than 10 percent of the computation cost.

In this paper we present a novel algorithm called Suppression of Slowly-varying components and the Falling edge of the power envelope (SSF) to enhance spectral features for robust speech recognition, especially in reverberant environments. This algorithm is motivated by the precedence effect and by the modulation frequency characteristics of the human auditory system. We describe two slightly different types of processing that differ in whether or not the falling edges of power trajectories are suppressed using a lowpassed power envelope signal. The SSF algorithms can be implemented for online processing. Speech recognition results show that this algorithm provides especially good robustness in reverberant environments. 1

In this paper, we present an improved vocal tract length perturbation (VTLP) algorithm as a data augmentation technique. VTLP is usually accomplished by adjusting the center frequencies of mel filterbank in [1]. Compared to the conventional approach, we re-synthesize waveforms from the frequencywarped spectra using overlap and addition (OLA). This approach had two advantages: First, we can apply an “acoustic simulator”[2, 3] after performing the VTLP-based frequency warping. Second, we may use a different window length for frequency warping from that used in feature processing. We observe that the best performance was obtained when the warping coefficient distribution is between 0.8 and 1.2, and the window length is 50 ms. We obtained 3.66% WER and 12.39% WER on the Librispeech test-clean and test-other using an attentionbased end-to-end speech recognition system without using any Language Models (LMs). Using the shallow-fusion technique with a Transformer LM, we achieved 2.44% WER and 8.29% WER on the Librispeech test-clean and test-other sets. To the best of our knowledge, the 2.44% WER on the test-clean is the best result ever reported on this test set.

This paper presents a new robust dynamic programming (DP)-based recognition algorithm that is quite suitable for menu-driven recognition applications with small vocabulary size (typically less than 50). When compared to the conventional dynamic time warping (DTW)-based recognizer, the proposed algorithm shows significantly improved recognition accuracy in speaker-independent cases. In addition, when compared to the conventional hidden Markov model (HMM)-based recognizer, the proposed algorithm requires smaller computational amount and parameter file size, while maintaining almost the same recognition rate for command recognition applications with small vocabulary size in hand-held consumer devices.

In this paper, we review various end-to-end automatic speech recognition algorithms and their optimization techniques for on-device applications. Conventional speech recognition systems comprise a large number of discrete components such as an acoustic model, a language model, a pronunciation model, a text-normalizer, an inverse-text normalizer, a decoder based on a Weighted Finite State Transducer (WFST), and so on. To obtain sufficiently high speech recognition accuracy with such conventional speech recognition systems, a very large language model (up to 100 GB) is usually needed. Hence, the corresponding WFST size becomes enormous, which prohibits their on-device implementation. Recently, fully neural network end-to-end speech recognition algorithms have been proposed. Examples include speech recognition systems based on Connectionist Temporal Classification (CTC), Recurrent Neural Network Transducer (RNN-T), Attention-based Encoder-Decoder models (AED), Monotonic Chunk-wise Attention (MoChA), transformer-based speech recognition systems, and so on. These fully neural network-based systems require much smaller memory footprints compared to conventional algorithms, therefore their on-device implementation has become feasible. In this paper, we review such end-to-end speech recognition models. We extensively discuss their structures, performance, and advantages compared to conventional algorithms.

Although automatic speech recognition systems have dramatically improved in recent decades, speech recognition accuracy still significantly degrades in noisy environments. While many algorithms have been developed to deal with this problem, they tend to be more effective in stationary noise such as white or pink noise than in the presence of more realistic degradations such as background music, background speech, and reverberation. At the same time, it is widely observed that the human auditory system retains relatively good performance in the same environments. The goal of this thesis is to use mathematical representations that are motivated by human auditory processing to improve the accuracy of automatic speech recognition systems. In our work we focus on five aspects of auditory processing. We first note that nonlinearities in the representation, and especially the nonlinear threshold effect, appear to play an important role in speech recognition. The second aspect of our work is a reconsideration of the impact of time-frequency resolution based on the observations that the best estimates of attributes of noise are obtained using relatively long observation windows, and that frequency smoothing provide significant improvements to robust recognition. Third, we note that humans are largely insensitive to the slowly-varying changes in the signal components that are most likely to arise from noise components of the input. We also consider the effects of temporal masking and the precedence effect for the processing of speech in reverberant environments and in the presence of a single interfering speaker. Finally, we exploit the excellent …

This paper describes the structure and performance of a new signal processing scheme, motivated by the physiology of the peripheral auditory system, that improves speech recognition accuracy in the presence of broadband noise. An important attribute of the peripheral processing is a novel mechanism to represent the cycle-by-cycle synchrony in the response of low-frequency auditory-nerve fibers, in addition to the more conventional processing based on mean rate of response. It is shown that the use of the physiologically-motivated peripheral processing improves recognition accuracy in the presence of both broadband and transient noise, and that the use of the synchrony mechanism provides further improvement beyond that which is provided by the mean rate mechanism.

Speech and Language Processing Lab

교수님 소개

김찬우

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Mels-Tts: Multi-Emotion Multi-Lingual Multi-Speaker Text-To-Speech System Via Disentangled Style Tokens

Latent Filling: Latent Space Data Augmentation for Zero-Shot Speech Synthesis

Hierarchical Timbre-Cadence Speaker Encoder for Zero-shot Speech Synthesis

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

김찬우

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Mels-Tts: Multi-Emotion Multi-Lingual Multi-Speaker Text-To-Speech System Via Disentangled Style Tokens

Latent Filling: Latent Space Data Augmentation for Zero-Shot Speech Synthesis

Hierarchical Timbre-Cadence Speaker Encoder for Zero-shot Speech Synthesis

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집