This presentation contains a tutorial devoted to showing how Java developers can create rich multimedia applications with Kurento. Java developers will find natural Kurento development model, which is based on standard Java EE technologies and is inspired on the WWW Servlet model. If you have ever developed a Web application, you may be familiar with this scheme. At the browser, HTML and JavaScript code is in charge of user interaction and generates HTTP requests to the server. This code is usually programmed with the help of APIs such as jQuery, DOM, XHR or others. Upon reception, HTTP requests are processed by some kind of server side technology (e. g.. PHP, Java, Ruby, etc.) using service APIs providing features such as DB access, communications, transactions, XML parsing, and others. As a result, an HTTP response is issued and sent back to the client. Following this scheme, both server and client side APIs are just capabilities simplifying developer work and providing abstractions for programming faster and more efficiently. Kurento technologies adapt to the Web development model so that, from a programmer perspective, Kurento can be seen just as an additional set of APIs. Developers does not need to learn novel programming schemes and can reuse all their knowledge and previous background on WWW application development. When you need multimedia, just use Kurento APIs. For the rest, use your preferred APIs or reuse previous code. Kurento APIs have been designed for simplicity and Web developers will find them familiar and intuitive. Most of the low level details related to codecs, formats, protocols, profiles and containers are abstracted by the framework. Programmers just concentrate on specifying the sequence of processing steps that they want to execute on the media flows.

1. www.kurento.org
Real-time Multimedia Stream Processing
Developing rich multimedia applications with KurentoDeveloping rich multimedia applications
with Kurento: a tutorial for Java
developers
lulop@kurento.org
http://www.kurento.org

2. www.kurento.org
Multimedia infrastructures for the
Future Internet
Media is
here
Media got
there
Media got
there
Analyze
Transform
Store
Transport
Enrich
Augment
Adapt
Sensors
Context
Events
Media is
here
2

3. www.kurento.org
Enrich, augment, adapt, analyze,
transform, store: what’s the problem?
Complexity
3

4. www.kurento.org
Future
Internet
Multimedia
Infrastructure
Simple
Development
APIs
Kurento: the equation
4

5. www.kurento.org
•Interoperable media exchange (multiplatform/multiprotocol)
•WebRTC, RTP, HTTP (video tag), etc.
•Process media (Computer vision, augmented reality, media indexing, etc.)
•Media and metadata recording and recovery
•Transform and adapt media (H.264, H.263, VP8, Ogg, and others)
•Media routing and mixing
•Etc.
Multimedia infrastructure
• REST API
• JavaScript API
• Java API
APIs
• LGPL 2.1
Is distributed through a flexible FOSS license
What’s Kurento
5

6. www.kurento.org
Kurento Media Server (KMS): the
nucleus of Kurento
• KMS is a middleware for media streams
– Receives the stream
– Process the stream
– Issues the stream
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
6

7. www.kurento.org
The Media API: The API for accessing
KMS capabilities
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
Java
Media API
JavaScript
Media API
REST
API
Applications define the processing
of streams getting through KMS
7

8. www.kurento.org
Media API: Media Elements and Media
Pipelines
SinkSRC
Sink
SRC
SRCSink
Sink
Media Element
• Provides a specific media
functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
Media pipeline
• Chain of media elements
implementing the desired media
logic.
• The Media API provides the
capability of creating media
pipelines by joining media
elements of the toolbox
Media Element
Sink
SRC
8

9. www.kurento.org
Media API: trivial example (Java)
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build();
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(httpEndpoint);
httpEndpoint.getUrl(); //URL where the media is made available
Media Pipeline (inside KMS)
HttpGetEndpoint
Media from
file or URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
9

10. www.kurento.org
Is that enough? Think about the WWW
development model
Process WWW request
- DDBB access
- Authentication
- XML processing
- Etc.
HTTP request:
I want this resource
HTTP response:
The resource
• Intuition behind the WWW model
– Client asks what it wants
• GET /this_resource HTTP/1.1
– Server side APIs execute the associated processing
You develop this using …
• Servlet
• JSP
• ASP
• PHP
• Ruby on Rails
• CGI
• Etc.
10

11. www.kurento.org
We need an equivalent model: the role
of the Signaling Plane
• We need “something” with the capability of negotiating what,
when and how media is exchanged
• A protocol for “saying”
– I want “this media” …
• Identification of the media to exchange
– File in hard-drive, IP camera, user, etc.
– in “this way” …
• Identification of the processing of media
– Augmented, analyzed, etc.
– with “this format” …
• Quality
– Codec, screen-size, frame-rate, etc.
– at “this moment”
• Stream control
– Play, stop, start, pause, etc.
11

12. www.kurento.org
Don’t get it? think about WWW
development again …
Process WWW request
- DDBB access
- Authentication
- XML processing
- Etc.
HTTP request:
I want this resource
HTTP response:
The resource
Process media request
- Media API
- DDBB access
- Authentication
- XML processing
- Etc.
Signaling request:
I want this media
Signaling response:
The media is here
This is how you
create WWW
applications
(Servlets, ASP, PHP,
Rails, etc.)
This is how you create
Kurento enabled
applications:
Multimedia RTC is just
another feature of your
application
You, as a
programmer,
create this
12

13. www.kurento.org
Abstracting the signaling plane: The
Content Handler
KMS
SinkSRC
Sink
SRC
Sink
SRCSink
Media API
REST API (Open API protocol) The Content Handler
Equivalent to a Servlet/ASP/PHP script
Exposes APIs for specifying:
- when receiving “this request”…
- … execute “this logic”
Developer can use the media API
Code building the media pipeline and
executing the application logic the
developer wants
13

14. www.kurento.org
Kurento Application Server: The
container of Handlers
• Is an extension of Java EE technologies.
• Compatible with all Java EE Servlet
containers
• Hold Handlers
– Your specific application code
• Receives signaling requests:
– I want “this media” in this way …
• Dispatches request to the appropriate
handler
– @annotations based mapping
• Generate an answer showing “where
and how to find the media”
– URL where media is located
– Negotiated SDP
Java EE compatible container
HTTP
Servlet
SIP
Servlet
Web
service
s
Kurento
REST API
Specific handler implementations
Media API DD.BB.
Kurento Application Server (KAS)
Other java
APIs.
14

15. www.kurento.org
Kurento Architecture: putting it all
together
Kurento Media Server (KMS)
Receive
Video
Augmented
Reality
Send
Video
Computer
Vision
Video Playing
and Recording
Java EE compatible container
HTTP
Servlet
SIP
Servlet
Web
service
s
Kurento
REST API
Specific handler implementations
Signaling and
WWW traffic
Media
Media
Signaling and
WWW traffic
Media API DD.BB.
Kurento Application Server (KAS)
Other java
APIs.
15

16. www.kurento.org
Application execution flow
Client
Code
Application
Server (KAS)
Media
Server (KMS)
I want this media in this way …
(JSON-RPC request)
Commands requesting
the creation of a pipeline
What you want is here …
(JSON-RPC answer)
Media
negotiation
phase
Media
exchange
phase
1
2
Specific application
logic at the
server-side
(Content Handler)
Media
pipeline
creation
Media exchange between client and server
16

17. www.kurento.org
Content Handler: trivial example
//Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc.
@HttpPlayerService(path = "/player”) //Mapping of handler specified in path
public class MyPlayerHandler extends HttpPlayerHandler {
@Override
public void onContentRequest(HttpPlayerSession contentSession) {
//Thie client wants the media this handler provides
//Create the pipeline for providing the media
}
@Override
public void onContentStarted(HttpPlayerSession contentSession) {
//Media started flowing, you can execute additional actions
}
@Override
Public void onSessionTerminated(HttpPlayerSession contentSenssion){
//Media exchange termianted, you can collect your resources
}
17

18. www.kurento.org
Let’s develop with Kurento
• What you need
– A Kurento instance
• You can install your own Kurento instance
• You can launch a Kurento instance at the FI-LAB (FI-WARE project)
– http://lab.fi-ware.org
• Getting help
– Kurento web site
• http://www.kurento.org
– Kurento mailing list
• https://groups.google.com/forum/#!forum/kurento
– Twitter
• @kurentoms
18

19. www.kurento.org
Kurento Hello World: Playing a file
with an HTML5 client
Media Pipeline
HttpGetEndpoint
Media from
file or URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
Media API
REST API (Open API protocol)
Handler codeI want
“this media”
Media is
“at this URL”
19

20. www.kurento.org
Playing a file with an HTML5 client:
Handler code@HttpPlayerService(path = "/player”)
public class MyPlayerHandler extends HttpPlayerHandler {
@Override
public void onContentRequest(HttpPlayerSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(
"http://media.w3.org/2010/05/sintel/trailer.webm").build();
contentSession.setAttribute("player", playerEndpoint);
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(httpEndpoint);
contentSession.start(httpEndpoint);
}
@Override
public void onContentStarted(HttpPlayerSession contentSession) {
PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player");
playerEndpoint.play();
}
}
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyPlayerHandler.java
20

21. www.kurento.org
Playing a file with an HTML5 client:
client code<!DOCTYPE html>
<html>
<head>
<script src="./js/kws-content-api.js"></script>
<script>
var conn;
function start() {
var options = {
remoteVideoTag : "remoteVideo"
};
conn = new kwsContentApi.KwsContentPlayer("./player", options);
}
function terminate() {
conn.terminate();
}
</script>
</head>
<body>
<button onclick="start();">Start</button>
<button onclick="terminate();">Terminate</button>
<br />
<video id="remoteVideo" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/webapp/player.html
21

22. www.kurento.org
Playing a file with an HTML5 client:
See the example working
https://www.youtube.com/watch?v=n5BQlhYgGSo
22

23. www.kurento.org
Media Pipeline
Adding Computer Vision
HttpGetEndpoint
Media
from
file or
URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
SRC
Sink
JackVaderFilter
Media API
REST API (Open API protocol)
Handler codeI want
“this media”
Media is
“at this URL”
23

24. www.kurento.org
Adding Computer Vision: Handler code
@HttpPlayerService(path = "/playerWithFilter”)
public class MyPlayerHandler extends HttpPlayerHandler {
@Override
public void onContentRequest(HttpPlayerSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(
"http://media.w3.org/2010/05/sintel/trailer.webm").build();
contentSession.setAttribute("player", playerEndpoint);
JackVaderFilter filter = mp.newJackVaderFilter().build();
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
filter.connect(httpEndpoint);
playerEndpoint.connect(filter);
contentSession.start(httpEndpoint);
}
@Override
public void onContentStarted(HttpPlayerSession contentSession) {
PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player");
playerEndpoint.play();
}
}
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java
24

25. www.kurento.org
Adding Computer Vision: Client code
<!DOCTYPE html>
<html>
<head>
<script src="./js/kws-content-api.js"></script>
<script>
var conn;
function start() {
var options = {
remoteVideoTag : "remoteVideo"
};
conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", options);
}
function terminate() {
conn.terminate();
}
</script>
</head>
<body>
<button onclick="start();">Start</button>
<button onclick="terminate();">Terminate</button>
<br />
<video id="remoteVideo" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/webapp/playerFilter.html
25

26. www.kurento.org
Adding Computer Vision: See the
example working
https://www.youtube.com/watch?v=yJAQs23eoXw
26

27. www.kurento.org
WebRTC loopback
Media Pipeline
WebRTC
Streaming
Media API
REST API (Open API protocol)
Handler codeI want
“this media (SDP)”
Media is
“at here (SDP)”
SinkSRC
27

28. www.kurento.org
WebRTC loopback: Handler code
28
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyWebRtcLoopback.java
@WebRtcContentService(path = "/webRtcLoopback")
public class MyWebRtcLoopback extends WebRtcContentHandler {
@Override
public void onContentRequest(WebRtcContentSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();
webRtcEndpoint.connect(webRtcEndpoint);
contentSession.start(webRtcEndpoint);
}
}
28

29. www.kurento.org
WebRTC loopback: client code
29
<!DOCTYPE html>
<html>
<head>
<script src="./js/kws-content-api.js"></script>
<script
src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script>
<script>
var conn;
function start() {
var options = {
localVideoTag : "localVideo",
remoteVideoTag : "remoteVideo"
};
conn = new kwsContentApi.KwsWebRtcContent("./webRtcLoopback", options);
}
function terminate() {
conn.terminate();
}
</script>
</head>
<body>
<button id="start" onclick="start();">Start</button>
<button id="terminate" onclick="terminate();">Terminate</button>
<br />
<video id="localVideo" autoplay></video>
<video id="remoteVideo" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/webapp/webrtcLoopback.html
29

30. www.kurento.org
WebRTC loopback: see the example
working
30
https://www.youtube.com/watch?v=HaVqO06uuNA
30

31. www.kurento.org
WebRTC recording
Media Pipeline
WebRTC
Streaming
Media API
REST API (Open API protocol)
Handler codeI want
“this media (SDP)”
Media is
“at here (SDP)”
SinkSRC
Sink
Media
to
file or
URI
31

32. www.kurento.org
WebRTC recorder: Handler code
32
Source https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java
Source of handler playing the recorded video:
https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyPlayerRecording.java
@WebRtcContentService(path = "/webRtcRecorder")
public class MyWebRtcRecorder extends WebRtcContentHandler {
@Override
public void onContentRequest(WebRtcContentSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();
RecorderEndpoint recorderEndpoint = mp.newRecorderEndpoint("file:///tmp/recording").build();
contentSession.setAttribute("recorder", recorderEndpoint);
webRtcEndpoint.connect(webRtcEndpoint);
webRtcEndpoint.connect(recorderEndpoint);
contentSession.start(webRtcEndpoint);
}
@Override
public void onContentStarted(WebRtcContentSession contentSession) {
RecorderEndpoint recorderEndpoint = (RecorderEndpoint) contentSession.getAttribute("recorder");
recorderEndpoint.record();
}
}
32

33. www.kurento.org
WebRTC recording: client code
33
<!DOCTYPE html>
<html>
<head>
<script src="./js/kws-content-api.js"></script>
<script
src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script>
<script>
var conn;
function start() {
var options = {
localVideoTag : "localVideo",
remoteVideoTag : "remoteVideo"
};
conn = new kwsContentApi.KwsWebRtcContent("./webRtcRecorder", options);
}
function terminate() {
conn.terminate();
}
</script>
</head>
<body>
<button id="start" onclick="start();">Start</button>
<button id="terminate" onclick="terminate();">Terminate</button>
<a href="./playerRecording">Play recorded content</a>
<br />
<video id="localVideo" autoplay></video>
<video id="remoteVideo" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/webapp/webrtcRecorder.html
33

34. www.kurento.org
WebRTC recording: see the example
working
34
https://www.youtube.com/watch?v=ogN81PGkMuE
34

35. www.kurento.org
WebRTC one-to-many (Handler
omitted for simplicity)
Media Pipeline
SinkSRC
SRCSinkSRCSinkSRCSink
35

36. www.kurento.org
WebRTC one-to-many: Handler code
36
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyWebRtcOneToMany.java
@WebRtcContentService(path = "/webRtcOneToMany")
public class MyWebRtcOneToMany extends WebRtcContentHandler {
private WebRtcEndpoint firstWebRtcEndpoint;
private String sessionId;
@Override
public synchronized void onContentRequest(WebRtcContentSession contentSession) throws Exception {
if (firstWebRtcEndpoint == null) {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
firstWebRtcEndpoint = mp.newWebRtcEndpoint().build();
sessionId = contentSession.getSessionId();
contentSession.releaseOnTerminate(firstWebRtcEndpoint);
firstWebRtcEndpoint.connect(firstWebRtcEndpoint); //Loopback
contentSession.start(firstWebRtcEndpoint);
} else {
MediaPipeline mp = firstWebRtcEndpoint.getMediaPipeline();
WebRtcEndpoint newWebRtcEndpoint = mp.newWebRtcEndpoint().build();
contentSession.releaseOnTerminate(newWebRtcEndpoint);
newWebRtcEndpoint.connect(firstWebRtcEndpoint);
firstWebRtcEndpoint.connect(newWebRtcEndpoint); //Latest client gives media to the master endpoint
contentSession.start(newWebRtcEndpoint);
}
}
}
36

37. www.kurento.org
WebRTC one-to-many: client code
37
<!DOCTYPE html>
<html>
<head>
<script src="./js/kws-content-api.js"></script>
<script
src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script>
<script>
var conn;
function start() {
var options = {
localVideoTag : "localVideo",
remoteVideoTag : "remoteVideo"
};
conn = new kwsContentApi.KwsWebRtcContent("./webRtcOneToMany", options);
}
function terminate() {
conn.terminate();
}
</script>
</head>
<body>
<button id="start" onclick="start();">Start</button>
<button id="terminate" onclick="terminate();">Terminate</button>
<br />
<video id="localVideo" autoplay></video>
<video id="remoteVideo" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/webapp/webrtcOneToMany.html
37

38. www.kurento.org
WebRTC one-to-many: see the
example working
38
https://www.youtube.com/watch?v=TBkrl3fmHWI
38

39. www.kurento.org
WebRTC game (Handler omitted for
simplicity)
Media Pipeline
SinkSRC
Sink
SRC
Sink
SRC
39

40. www.kurento.org
WebRTC game: Handler code
40
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyWebRtcWithFilters.java
@WebRtcContentService(path = "/webRtcWithFilters")
public class MyWebRtcWithFilters extends WebRtcContentHandler {
@Override
public void onContentRequest(WebRtcContentSession contentSession)throws Exception {
final MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
final WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();
final PointerDetectorFilter pointerDetectorFilter = mp.newPointerDetectorFilter().build();
final FaceOverlayFilter faceOverlayFilter = mp.newFaceOverlayFilter().build();
PointerDetectorWindowMediaParam start = new PointerDetectorWindowMediaParam("start", 100, 100, 280, 380);
start.setImage("http://ci.kurento.com/imgs/start.png");
pointerDetectorFilter.addWindow(start);
pointerDetectorFilter.addWindowInListener(new MediaEventListener<WindowInEvent>() {
public void onEvent(WindowInEvent event) {
faceOverlayFilter.setOverlayedImage(
"http://ci.kurento.com/imgs/mario-wings.png",
-0.35F, -1.2F, 1.6F, 1.6F);
}
});
webRtcEndpoint.connect(pointerDetectorFilter);
pointerDetectorFilter.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);
contentSession.start(webRtcEndpoint);
}
}
40

41. www.kurento.org
WebRTC game: client code
41
<!DOCTYPE html>
<html>
<head>
<script src="./js/kws-content-api.js"></script>
<script
src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script>
<script>
var conn;
function start() {
var options = {
remoteVideoTag : "remoteVideo"
};
conn = new kwsContentApi.KwsWebRtcContent("./webRtcWithFilters", options);
}
function terminate() {
conn.terminate();
}
</script>
</head>
<body>
<button id="start" onclick="start();">Start</button>
<button id="terminate" onclick="terminate();">Terminate</button>
<br />
<video id="remoteVideo" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/webapp/webrtcFilters.html
41

42. www.kurento.org
WebRTC game: see the example
working
42
https://www.youtube.com/watch?v=5eJRnwKxgbY
42

43. www.kurento.org
Collaborations welcome
http://www.github.com/kurento
Thank you very much for your attention
Suggestions, comments and complains:
lulop@kurento.org
43