Noise suppression during audio calls is expected from any real-time communication platform. Good denoising means exhaustive use of modern neural networks. AI implies heavy processing, real-time requires speed, and a large number of calls means a large need for resources. Therefore, the questions are: can clients do the necessary processing and can AI solutions be integrated on the client side? We will present a neural network for noise suppression implemented as part of the Infobip WebRTC platform. A generic way of integrating AI solutions with client-side JavaScript will be described, with a special focus on real-time requirements. The final solution based on RNNoise will be presented.

1. Make some noise
for AI in JavaScript
Sead Delalić, PhD
University of Sarajevo & Infobip

2. Agenda
• About us
• Problem description
• Proposed AI solution
• WebRTC solution
• Examples & Conclusion

3. Infobip

5. Sead Delalić
• PhD in Computer Science
• University of Sarajevo
• Optimization, Data mining, AI
• Senior Teaching Assistant at Faculty of Science
• AI Consultant at Infobip
• 5+ years of experience in AI industry
• 25 research/scientific papers

6. Problem description

7. Remove noise from audio calls.

8. Remove noise from audio calls.
Don't remove anything useful.

9. Questions
Which signal is
noise? Types of noise?
Which signal is useful?
How much time do we
have for processing?

10. Answers
• Interested in
‣ hearing the voice of the person speaking.

11. Answers
• Interested in
‣ hearing the voice of the person speaking.
• We don’t want
‣ static noise;
‣ air conditioner sound and similar noises;
‣ babble noise;
‣ typing sounds.

12. Answers
• Interested in
‣ hearing the voice of the person speaking.
• We don’t want
‣ static noise;
‣ air conditioner sound and similar noises;
‣ babble noise;
‣ typing sounds.
• Everything should be in real-time, while the call is ongoing.
• The total delay must be below 200 ms, ideally below 100 ms.

13. WebRTC
Calls (one-on-one)
Conferences

16. DENOISING

17. Solution

18. Preliminary
research
Standard
algorithms
Filters
Artificial
intelligence
Recurrent
networks &
Autoencoders

19. Noise Suppression with
ANNs

20. Process
Create dataset
Implement preprocessing
Model neural network
Train neural network
Use trained network

21. Dataset
• Synthesis: Clean speech + Noise = Noisy Speech
• MS-SNSD and other datasets
• Network Input: Noisy Speech
• Network Output: Clean Speech

22. Preprocessing
• Preprocessing depends on the chosen approach.
• A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement
‣ Jean-Marc Valin
‣ Proceedings of IEEE Multimedia Signal Processing (MMSP) Workshop, arXiv:1709.08243,
2018.
‣ https://arxiv.org/abs/1709.08243

23. Preprocessing
1. Audio signal divided into chunks
2. Fast Fourier Transform (FFT)
3. Mel Filter
4. Mel-frequency Cepstrum (MFCC)
5. Calculate deltas

29. FFT

30. How Fourier did it

31. How we did it

33. Mel Filter Bank

34. MFCC

35. RNNoise

36. Training

37. WebRTC implementation

38. Web Audio API
• Controlling audio on the Web
‣ Choose sources
‣ Add effects
‣ Create visualizations
‣ etc.
• Audio nodes to create routing graph
• Custom processor nodes

39. Flow
• Stream
• Audio Worklet
• Main Thread
• Worker Thread

40. Audio
Worklet
Main
Thread
Audio
Worker
Audio stream
Input Buffer &
Windowing
Noisy Window Noisy Window
Preprocessing
& RNNoise &
Postprocessing
Noise-Suppressed Window
Noise-Suppressed Window
Output Buffer
&
Reproduction

41. NumJs & TensorFlow.js
• NumJs
‣ Scientific computing with JS
‣ FFT & matrix operations
• TensorFlow.js
‣ Hardware-accelerated JS library for DL
‣ Convert Keras format to TFJS & run model in JS

42. Examples

43. Example – Original audio

44. Example – Original audio

45. Version 1.0

46. Version 1.0

47. Version 1.1

48. Version 1.1

49. Version 1.n

50. Version 1.n

51. Once again

52. Once again

53. More WebRTC examples

54. More WebRTC examples

55. Conclusion

56. Conclusion
• A hot topic in the WebRTC world
• A solution for real-time usage
‣ Different types of noises and generalization
‣ Speech enhancement
• Great room for improvement

57. Make some noise
for AI in JavaScript
Sead Delalić, PhD
delalic.sead@infobip.com | delalic.sead@pmf.unsa.ba