This is a presentation specifically created for the GSMA interest group on WebRTC. This presentations introduces Kurento from the perspective of operators. Kurento is a multimedia development framework. It has been created to ease the life of multimedia application developers. Using multimedia capabilities such as embedding a video onto your app or establishing a video conferencing link between two clients may be tricky, but there is no rocket science there. However, for applications requiring more advanced features things quickly get unmanageable. If you have been involved in multimedia projects, you probably know that features such as interoperable group communications, different communication roles (e. g.. publishers/viewers), video transforming and transcoding, video storage and tagging, integration into legacy video/voice infrastructures, computer vision, augmented reality, integration with external systems and databases and many others, pose quite a complex challenge, which usually requires huge expertise and effort. Specially when real-time communications are involved. If this is your case, Kurento will help you.

1. www.kurento.org
Presentation for the GSMA WebRTC
interest group on April 7th, 2014
The future of multimedia
communications and services:
Kurento and it’s role
Luis Lopez
lulop@kurento.org
http://www.kurento.org

2. www.kurento.org
Who am I?
• Luis Lopez
– Associate Professor at ETSIT/URJC
• http://www.urjc.es
• http://www.etsit.urjc.es
– Director of the FUN-LAB research
group
• Future Networks Laboratory
• http://www.gsyc.es
– Leader of the Kurento.org initiative
• http://www.kurento.org
– Software engineer, software
developer and Open Source
Software Advocate
2

3. www.kurento.org
What does Kurento mean?
3

4. www.kurento.org
Kurento’s History
2004
•Kurento is
born as a tool
for
transferring
research
results to
industry
2012
•Kurento goes
open source
LGPL v2.1.
Professional
services
provided by
Naevatec
(startup)
June’13
•1M€ of
investment for
providing
multimedia
capabilities to
FI-WARE
Dec13
•Kurento
Featured at
WebRTC Expo
(Paris)
Feb’14
•3.3M€ of
investment for
making
Kurento an
elastic cloud
platform
(NUBOMEDIA)
April’14
•Kurento
featured at
IMS World
Conference
(Barcelone)
July’14
•Kurento
featured at
DevCon5 (New
York)
Sep’14
•500K€ of
investment for
maintenance
of Kurento FI-
WARE APIs
4

5. www.kurento.org
Kurento team
• 18 people: 15 developers, 1 sysadmin, 2 coordination
5

6. www.kurento.org
Real-time communications: our vision
Numberofusers
General purpose
RTC multimedia Services
Specific purpose RTC
multimedia services
Commodity, no business
opportunities here
Who cares about the immense minority?
Many business opportunities here
6

7. www.kurento.org
The opportunity: from services to
platforms and APIs
7

8. www.kurento.org
Who shares this vision: The WebRTC
movement
Numberofusers
General purpose
RTC multimedia Services
Specific purpose RTC
multimedia services
Commodity, no business
opportunities here
Who cares about the immense minority?
Many business opportunities here
Disruption direction
of WebRTC
8

9. www.kurento.org
Before and after WebRTC: the
perspective of the application developer
Developing
the client side
Before WebRTC After WebRTC
Begin End
• Unified APIs
• Standards
• FOSS
• Multiplatform
9

10. www.kurento.org
Why developers are important for the
WebRTC movement?
The effect of WWW developers on different technological areas
10

11. www.kurento.org
Should telcos take the hint?
WebRTC
WWW development community RCS/IMS development community
WebRTC makes multimedia RTC to be just another feature of your WWW application
11

12. www.kurento.org
What’s the relationship of WebRTC
with Kurento?
Developing
the client side
Developing the
infrastructure
side
Before WebRTC After WebRTC Next natural step…
Begin End Begin End
Begin End
• Unified APIs
• Standards
• FOSS
• Multiplatform
• Unified APIs
• Standards
• FOSS
• Multiplatform
• Unified APIs
• Standards
• FOSS
• Multiplatform
12

13. www.kurento.org
But is the infrastructure important?
What RTC applications will be demanded by the market on the next few years?
13

14. www.kurento.org
Kurento vision for Future RTC
multimedia services
Media is
here
Media got
there
Media got
there
Analyze
Transform
Store
Transport
Enrich
Augment
Adapt
Sensors
Context
Events
Media is
here
14

15. www.kurento.org
Enrich, analyze, adapt, augment:
What’s the problem?
Complexity
15

16. www.kurento.org
Future
Internet
Multimedia
Infrastructure
Simple
Development
APIs
The Kurento equation for Future
Internet infrastructures
16

17. www.kurento.org
What’s Kurento?
• Kurento.org
– Developer platform
• Infrastructures + APIs
• Kurento is not a service
– Future Internet
• Generalized multimedia
– Audio/Video/Sensor
• Advanced media processing
– Computer Vision, Augmented reality
• Interoperability
– WebRTC, RTP, HTTP, all codecs, etc.
– Open Source Software
• LGPL v2.1
17

18. www.kurento.org
Kurento Media Server: The nucleus of
Kurento
• KMS is a middleware for media streams
– Receives the stream
– Process the stream
– Issues the stream
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
18

19. www.kurento.org
Media API: The API for accessing KMS
capabilities
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
Java
Media API
JavaScript
Media API
Other
languages
Applications define the processing
of streams getting through KMS
19

20. www.kurento.org
Media API: media elements and media
pipelines
SinkSRC
Sink
SRC
SRCSink
Sink
Media Element
• Provides a specific media
functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
Media pipeline
• Chain of media elements
implementing the desired media
logic.
• The Media API provides the
capability of creating media
pipelines by joining media
elements of the toolbox
Media Element
Sink
SRC
20

21. www.kurento.org
The Lego Game: Developers create
applications connecting media elements
Protocols
& Codecs
Computer
Vision
Augmented
Reality
Mul sensory
Mul media
Media
Repository
IPTV
Integra on
Group
Communica ons
Sink%
H pEndpoint
Sink%SRC%
RtpEndpoint
SRC$
DataChannelEndpointSink%SRC%
WebRtcEndpoint
Sink%
SRC%
BarCodeReader
Sink%
SRC%
PointerTracker
Sink%
SRC%
FaceDetector
Sink%
SRC%
FaceOverlay
Sink%
SRC%
ChromaFilter
Sink%
SRC%
HeartRateBlender
Sink%
RecorderEndpoint
SRC$
PlayerEndpoint
Sink%
CdnUpload
Sink%
IPTVConnector Mixer
SinkSRC
Sink
SRC
Sink
SRCSink
Sink
Application 2
Sink
SRC
Sink
SRCSink
SinkSRC
Sink
Application 3
SinkSRC
SRCSink
SRC Sink
SinkSRC
Application 1
Toolbox of media elements
21

22. www.kurento.org
Media API trivial example
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build();
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(httpEndpoint);
httpEndpoint.getUrl(); //URL where the media is made available
Media Pipeline
HttpGetEndpoint
Media from
file or URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
22

23. www.kurento.org
The signaling: adapting the WWW
model to the multimedia services
Process WWW request
- DDBB access
- Authentication
- XML processing
- Etc.
HTTP request:
I want this resource
HTTP response:
The resource
Process media request
- Media API
- DDBB access
- Authentication
- XML processing
- Etc.
Signaling request:
I want this media
Signaling response:
The media is here
This is how you
create WWW
applications
(Servlets, ASP, PHP,
Rails, etc.)
This is how you create
Kurento enabled
applications:
Multimedia RTC is just
another feature of your
application
Developers
create this
23

24. www.kurento.org
The Content Handler: reacting to
signaling events
KMS
SinkSRC
Sink
SRC
Sink
SRCSink
Media API
REST API (Open API protocol) The Content Handler
Equivalent to a Servlet/ASP/PHP script
Exposes APIs for specifying:
- when receiving “this request”…
- … execute “this logic”
Developer can use the media API
Code building the media pipeline and
executing the application logic the
developer wants
24

25. www.kurento.org
Kurento Application Server, the
container of Handlers
• Is an extension of Java EE
technologies.
• Compatible with all Java EE Servlet
containers
• Hold Handlers
– Your specific application code
• Receives signaling requests:
– I want “this media” in this way …
• Dispatches request to the appropriate
handler
– @annotations based mapping
• Generate an answer showing “where
and how to find the media”
– URL where media is located
– Negotiated SDP
Java EE compatible container
HTTP
Servlet
SIP
Servlet
Web
service
s
Kurento
REST API
Specific handler implementations
Media API DD.BB.
Kurento Application Server (KAS)
Other java
APIs.
25

26. www.kurento.org
Putting it altogether: the Architecture
Kurento Media Server (KMS)
Receive
Video
Augmented
Reality
Send
Video
Computer
Vision
Video Playing
and Recording
Java EE compatible container
HTTP
Servlet
SIP
Servlet
Web
service
s
Kurento
REST API
Specific handler implementations
Signaling and
WWW traffic
Media
Media
Signaling and
WWW traffic
Media API DD.BB.
Kurento Application Server (KAS)
Other java
APIs.
26

27. www.kurento.org
Application execution flow
Client
Code
Application
Server (KAS)
Media
Server (KMS)
I want this media in this way …
(JSON-RPC request)
Commands requesting
the creation of a pipeline
What you want is here …
(JSON-RPC answer)
Media
negotiation
phase
Media
exchange
phase
1
2
Specific application
logic at the
server-side
(Content Handler)
Media
pipeline
creation
Media exchange between client and server
27

28. www.kurento.org
Kurento in the context of IMS
KMS = MRF
(Media Resource Function)
KAS = AS
(Application Server)
S-CSCF
(Serving- Call Session
Control Function
Media
Control
Media
Events
Context and
other information
services
UE
(User Equipment)
Application logic & orchestration
Media capabilities (pipeline)
Media elements can generate
events through deep analysis of
media
• IVR
• Sentiment analysis
• Face recognition
• Object tracking
• Etc.
Application logic can use context
and other external information
services for controlling media
elements or dynamically modifying
media pipelines
28

29. www.kurento.org
Kurento use case: WebRTC video
conference with “secret” participant• The “connection” primitive acts from source to destination
– You can connect how many sinks as you to a source
– Not-connected sources or sinks don’t break anything.
• In this example
– User 1 and 2 maintain a video conference
• webRtcEndpoint1.connect(webRtcEndpoint2), webRtcEndpoint2.connect(webRtcEndpoint1)
– User 3 can “see” that video conference, but he cannot participate
• webRtcEndpoint1.connect(webRtcEndpoint3), webRtcEndpoint2.connect(webRtcEndpoint3’)
29
Media Pipeline
WebRTC audio
streaming
SinkSRC
WebRTC video
streaming
SRCSinkSRCSink
WebRTC audio
streaming
WebRtcEndpoint 1
WebRtcEndpoint 3
WebRtcEndpoint 2SRCSink
WebRTC audio
streaming
WebRtcEndpoint 3’
User 1
User 2
User 3

30. www.kurento.org
Kurento use case: WebRTC bridge/SBC
between WebRTC and RTP domains
• Kurento transparently adapts codecs and formats. Just connect:
– webRtcEndpoint.connect(rtpEndpoint)
– rtpEndpoint.connect(webRtcEndpoint)
• Support for the most common audio and video codecs
– VP8, H.264, H.263, OPUS, AMR, Speex, G.711, etc.
30
Media Pipeline
WebRTC audio
streaming
SinkSRC
SRCSink
Media Pipeline
SinkSRC
SRCSink
WebRTC video
streaming
RTP audio
streaming
RTP video
streaming

31. www.kurento.org 31
SinkSRC
SRC Sink
SinkSRC
Kurento use case: interoperable MCU
SRCSink
• Group
communications
– Dispatcher
• Media router
• N to M
– Mixer
• Composite
mixed media
• N to 1

32. www.kurento.org
Kurento use case: Video conference
recorded and distributed (CDN)
• Kurento makes possible to interconnect different types of endpoints in your
application
– RecorderEndpoint
• Records media in the desired format
– HttpEndpoint
• Exposes the media flow through HTTP pseudostreaming (HTML5 <video> tag)
32
SinkSRC
Media Pipeline
Sink
SRCSink
SinkSRC
Sink
Media
to
file or
repository
CDN or
Web
browser

33. www.kurento.org
Kurento use case: video-conference
with augmented reality
• The infrastructure exposes advances processing capabilities
• You can connect them following the same scheme
– playerEndpoint.connect(jackVaderFilter)
– jackVaderFilter.connect(httpEndpoint)
• Use RTC endpoints if you want real-time media processing
– WebRtcEndpoint, RtpEndpoint
33
Media Pipeline
Media
from
file or
URI
Sink
SRC
SRC
Sink
JackVaderFilter
Result: https://www.youtube.com/watch?v=yJAQs23eoXw

34. www.kurento.org
Kurento use case: WebRTC video game
Result: https://www.youtube.com/watch?v=PYCw9-4oWWA
SinkSRC
Sink
Sink
SRC
Sink
SRC
Sink
SRC
Sink
SRC
WebRtcEndpoint
MirrorFilter PointerDetectorFilter
ChromaFilter FaceOverlayFilter
Media Pipeline
34
• Beyond video-conferencing: receiving events from the media stream
– Immersive and personalized advertising
– Games and entertainment
Events (buttons)
Game or advertisement logic
Change faceChange background

35. www.kurento.org
Media Pipeline
Media Pipeline
IoT and context aware applications:
hot IVR plugin
35
• Pipelines are dynamic, you can hot modify them using context or sensor
information
– Ex. when device is moving faster than 40 Kh, add IVR system
• “Record this call”, “Redirect to my secretary”, “Play file one”, Etc.
• Voice commands can modify the pipeline themselves
– “Record this call”  add a RecorderEndpoint
– “Play file 1”  add a player
– Ex. When security sensor fires, activate face-recognition/person-detection, etc
SinkSRC
SRCSink
Sink
SRC
SinkSRC
SRCSink
IvrFilter
VoiceEvents
Dynamic pipeline modification
Depending on context or other events

36. www.kurento.org
Media Pipeline
Smart-cities applications
36
• Advanced computer vision filters for
– Crowd detection
– Car plate recognition
– Intrusion detection
– Traffic control
– Etc.
CrowdEvents
Sink
SRC
Sink
SRC
CrowdDetector
RtspPlayer

37. www.kurento.org
Other scenarios
37
• There are opportunities on many verticals
– eHealth
– Logistics
– Security
– Social networks
– Jounalisms
• Creation of additional Media Elements is simple
– Integrate your own IVRs
– Integrate your own Computer Vision capabilities
– Integrate your own repositories
– Etc.
• Potential applications are unlimited
– Just combine the media elements

38. www.kurento.org
Thank you
Suggestions, comments and complains:
lulop@kurento.org
http://www.kurento.org
http://www.github.com/kurento
Twitter: @Kurentoms
38