Joint Speech and Speaker Recognition using Hidden Markov Model/Vector Quantization for speaker independent Speech Recognition and Gaussian Mixture Model for speech independent speaker recognition- used MFCC (Mel-Frequency Cepstral Coefficient) for Feature Extraction (delta,delta delta and energy - 39 coefficients). Developed in JAVA with client/server Architecture, web interface developed in Adobe Flex. This project was done at TU, IOE - Pulchowk Campus, Nepal. For more details visit http://ganeshtiwaridotcomdotnp.blogspot.com ABSTRACT OF PROJECT>>> Biometric is physical characteristic unique to each individual. It has a very useful application in authentication and access control. The designed system is a text-prompted version of voice biometric which incorporates text-independent speaker verification and speaker-independent speech verification system implemented independently. The foundation for this joint system is that the speech signal conveys both the speech content and speaker identity. Such systems are more-secure from playback attack, since the word to speak during authentication is not previously set. During the course of the project various digital signal processing and pattern classification algorithms were studied. Short time spectral analysis was performed to obtain MFCC, energy and their deltas as feature. Feature extraction module is same for both systems. Speaker modeling was done by GMM and Left to Right Discrete HMM with VQ was used for isolated word modeling. And results of both systems were combined to authenticate the user. The speech model for each word was pre-trained by using utterance of 45 English words. The speaker model was trained by utterance of about 2 minutes each by 15 speakers. While uttering the individual words, the recognition rate of the speech recognition system is 92 % and speaker recognition system is 66%. For longer duration of utterance (>5sec) the recognition rate of speaker recognition system improves to 78%.

1. Major Project Mid-Term Presentation :Speaker Verification for Remote Authentication Members: Ganesh Tiwari (063BCT510) MadhavPandey(063BCT514) ManojShrestha(063BCT518) Supervisor : Dr. SubarnaShakya Associate Professor

2. Introduction Voice biometric system user login Text-Prompted system The claimant is asked to speak a prompted text Speech and Speaker Recognition/Verification More secure to playback attack. Web Application Client (Adobe Flex) : Voice Capture, preprocessing and feature extraction Server (JAVA) : Training / Classification BlazeDS RPC for JAVA-Flex Connectivity

3. Block Diagram of Speaker / Speech Recognition System

4. Signal Capture and Pre-Processing

5. Capture and Preprocessing Get the audio signal i.e., ADC Make suitable for feature extraction

6. Capture and Preprocessing :Capture 22050 Hz 16-bits,Signed Little Endian Mono Uncompressed PCM

7. Capture and Preprocessing :PCM Extract

8. Capture and Preprocessing : Silence Removal Algorithm described in paper ‘a new method for silence removal and endpoint detection’ † †G. Saha, SandipanChakroborty, SumanSenapati of Department of Electronics and Electrical Communication Engineering, Indian Institute of Technology, Khragpur, India

9. Capture and Preprocessing :Pre-Emphasis Boosting the high frequency energy In time domain, y[n] = x[n]−αx[n−1], 0.9 ≤ α≤ 1.0

10. Capture and Preprocessing : Framing Speech Signal is stationary (statistical properties) for 10-30 ms 50% overlapped frames each of 23ms is used

11. Capture and Preprocessing :Windowing Windowing is done on the frame blocked signal Hamming window

12. Feature Extraction

13. Feature Extraction Transform the input audio signal into a sequence of acoustic feature vectors MFCC : Mel Filter CepstralCoefficients as Feature Perceptual approach Human Ear processes audio signal in Mel scale Mel scale : linear up to 1KHz and logarithmic after 1KHz MFCC gives distribution of energy in Mel frequency band Calculated for each frame

14. Feature Extraction : Fourier Transform Gives information about the amount of energy at each frequency band FFT used

15. Feature Extraction : Mel Filter We used filter bank of triangular filters spaced in Mel scale

16. Feature Extraction : Mel Filter (contd..) Mel Filter Where,

17. Feature Extraction :Log, IFT(DCT) Log DCT MFCC

18. Feature Extraction : Cepstral Mean Subtraction CMS: for minimizing channel effect

19. Feature Extraction : Energy and Deltas For completeness of feature vector and for achieving high recognition rate A Energy Feature A delta or velocity feature, and a double delta or acceleration feature Calculated by linear regression of regression window M

20. Composition of Feature Vector 12 MFCC Features 12 Delta MFCC 12 Delta-Delta MFCC 1 Energy Feature 1 Delta Energy Feature 1 Delta-Delta Energy Feature  39 Features from each frame

21. Speaker Recognition/Verification by GMM

22. Gaussian Mixture Model Parametric probability density function Based on clustering technique M Gaussian components 𝑝(𝑥/)= 𝑚=1𝑀𝑤𝑚 . 𝑔𝑚(𝑥/𝜇𝑚 , 𝐶𝑚) 𝑥: a k-dimensional random vector 𝑤𝑚: mixture weight of mth component 𝑔𝑚 : k-dimensional Gaussian function (pdf) 𝑔𝑚𝑥/𝜇𝑚 , 𝐶𝑚 = 12𝜋𝐾.|𝐶𝑚| exp{−12𝑥−𝜇𝑚 .(𝐶𝑚−1(𝑥−𝜇𝑚 ))}  = (𝑤𝑚, 𝜇𝑚 ,𝐶𝑚)

23. GMM Training Goal: estimate the parameters Method: Maximum Likelihood estimation Input: X = {𝑥1,𝑥1,…,𝑥𝑇} P(X/) =𝑡=1𝑇𝑝(𝑥𝑡/) Maximize with Expectation Maximization algorithm Iterative process: initial model: 𝑖 new model: 𝑖+1 P(X/ 𝑖+1) ≥ P(X/ 𝑖)

24. Verification Decision: Hypothesis Test H0: the speaker is the claimed speaker H1: the speaker is an imposter Based on likelihood ratio  = P(X/)P(X/) Decision by threshold < 𝜃𝑇reject identity claim > 𝜃𝑇 accept identity claim

25. Speech Recognition by HMM/VQ

26. Hidden Markov Model :Definition Hidden Markov Model (HMM) is the statistical model HMM is the extension of Markov Process HMM has hidden states and observable symbols per states HMM Model : Observed data : feature vector Hidden states : phonemes

27. Codebook Generation K-Means Clustering Clustering the whole database & Codebook Generation VQ : Vector Quantization is used for mapping each input feature vector to discrete quantized symbols Codebook for each incoming feature vector is built Compare it to each of the prototype vectors in codebook Select the one which is closest (by some distancemetric) Replace the input vector by the index of this prototype vector observation sequence

28. Speech Recognition System: By : HMM / VQ

29. Hidden Markov Model :Training Training by: Forward backward (Baum-Welch) algorithm Forward-backward algorithm iteratively re-estimates the parameters and improves the probability that given observation are generated by the new parameters Three parameters need to be re-estimated: Initial state distribution: πi Transition probabilities: ai,j Emission probabilities: bi(ot) Input is observation sequence, given by VQ

30. Hidden Markov Model :Verification/Matching Viterbi algorithm is used Input is Observation sequence, given by VQ HMM model of the word Best matched word is returned

31. Problem Faced Learning curve Complex Mathematics Flex & Java Connectivity (initially) Data conversion

32. Remaining Tasks Speech Training Data Collection Model Training (HMM, GMM) Module Integration Testing

33. Thanks