The document discusses the Stream Oriented Generic Enabler (SO GE) which provides multimedia capabilities to the FI-WARE infrastructure through a simple API. It allows sending and receiving media, processing media through elements like computer vision and augmented reality filters, and transforming media formats. Examples are given of media elements like HttpEndpoint, RtpEndpoint, and filters like JackVaderFilter. The SO GE API allows dynamically creating media pipelines by connecting these reusable elements. An overview is also given of developing applications using this API in JavaScript or Java.

1. Open APIs for Open Minds
The Stream Oriented Generic Enabler – SO GE
Developing rich multimedia applications with FI-WARE.

2. Speakers
1
 Luis López
• Leader of the Kurento project
• As. Prof. at URJC
• Coordinator of the Stream Oriented GE at
FI-WARE
 Jose Antonio Santos
• Chief Architect of Kurento project
• Senior Developer at Naevatec
• Researcher at FI-WARE project

3. The Stream Oriented Generic Enabler – SO GE
2
• Send and receive media (Multidevice/multiprotocol)
• Process media (Computer vision, augmented reality, media indexing, etc.)
• Transform and adapt media (H.264, H.263, VP8, Ogg, and others)
Provides multimedia capabilities to the FI-WARE infrastructure
• No need of protocol/codec low level understanding
Exposes those capabilities through a simple to use API
• LGPL 2.1
Is distributed through a flexible FOSS licence
• Sorry for the instabilities…
Is still work in progress

4. How media APIs (usually) work
3
Client side
- WWW browser
- Native API
Server side
- Media repositories
- Media capabilities
Hey, I want to send/receive media in this way
Here you have the media you requested
API API

5. The SO-GE API is a WWW inspired API
4
 Dynamic WWW page development
• Lots of capabilities (ex. Java EE)
 Multimedia development with the SO-GE
• Don’t want to loose any capability
Create HTML
- DDBB access
- Transactions
- Security tools
- Etc.
HTTP request:
I want this resource
HTTP response:
The resource
Create Media
- Media API
- DDBB access
- Transactions
- Security tools
- Etc.
HTTP request:
I want this media
HTTP response:
The media is here

6. The SO-GE Media API: media elements and pipelines
5
 Media Element
• Provides a specific media functionality
› Building block
› Send/receive media
› Process media
› Transform media
• The Media API provides a toolbox of
media elements ready to be used.
• New media elements can be added
 Media pipeline
• Chain of media elements implementing
the desired media logic.
• The Media API provides the capability of
creating media pipelines by joining media
elements of the toolbox
Media Element
Sink
SRC
File Source
SRC
Ogg-demuxer
Sink
SRC
Vorbis-decoder
Sink
SRC
Audio-sink
Sink
Theora-decoder
Sink
SRC
Video-sink
Sink

7. The global vision of a SO-GE application
6
SO-GE Media Server Infrastructure
Decode
Video
Augmented
Reality
Encode
Video
Computer
Vision
Video Playing
and Recording

8. Examples of media elements: HttpEndPoint
7
HttpEndPoint
 Function
• Sends/receives media basing on HTTP
transports (HTML5 video tag)
• Associates media to a temporary-one-
time-use URL
 Formats
• WebM
• H.264 support under development
 Input parameters
• None
HttpEndPoint
Media
Source
Media
Sink
HTTP POST
requests
HTTP
answers

9. Example of media elements: RtpEndPoint
8
RtpEndPoint
 Function
• Sends/receives media basing on RTP
transports (Real Time Communications).
• Negotiates media parameterization
through SDPs
 Formats
• H.263, MPEG4, H.264, VP8
 Input parameters
• Input SDP
RtpEndPoint
Media
Source
Media
Sink
RTP input
media
RTP output
media

10. Example of media elements: PlayerEndPoint
9
PlayerEndPoint
 Function
• Plays media from file or URI.
 Formats
• WebM
• MP4 (H.264)
• 3GPP
• AVI
• Etc.
 Input parameters
• File URI
PlayerEndPoint
Media
Source
Media from
file or URI

11. Example of media elements: RecorderEndPoint
10
RecorderEndPoint
 Function
• Records media into a file.
 Formats
• WebM
• H.264 formats under development
 Input parameters
• File path
RecorderEndPoint
Media
Source
Media
Sink
Media to
file

12. Example of media elements: JackVaderFilter
11
JackVaderFilter
 Function
• Example of filter
• Recognizes faces
• Adds a Jack Sparrow or Darth Vader hat
onto faces
 Formats
• Raw
 Input parameters
• None
JackVaderFilter
Sink
SRC
Media
Stream
Augmented
Media stream

13. Example of media elements: ZBarFilter
12
ZBarFilter
 Function
• Example of filter
• Recognizes Bar and QR codes
• Generates events
 Formats
• Raw
 Input parameters
• None
ZBarFilter
Sink
SRC
Media
Stream
Media
stream
Event

14. Developing media application with the SO-GE
 Runtime environment
• Install Kurento Media Server (soon available at FI-LAB)
• Install JBoss Java EE container (soon available at FI-LAB)
 Development environment
• Maven (repository: http://repository.kurento.com/archiva/repository/snapshots)
• Soon available at maven central
› kmf-content-api
› Provides handers
› kmf-media-api
› Provides media elements and pipelines
• Eclipse
• Javascript libraries (available at the same repository)
› kws-content-api
13

15. My first application with the SO-GE: play a video
14
HttpEndPoint
Media
Source
Media
Sink
PlayerEndPoint
Media
Source
Media from
file or URI
HTTP media
streaming
Client side Server side

16. My first application with the SO-GE: play a video
 Source code …
15

17. My second application: including a simple filter
16
HttpEndPoint
Media
Source
Media
Sink
PlayerEndPoint
Media
Source
HTTP media
streaming
JackVaderFilter
Sink
SRC
Client side Server side

18. My second application: including a simple filter
 Source code …
17

19. Making it more fun: introducing real time media
18
Media
Source
Media
Sink
Real time
media
HTTP media
streaming
JackVaderFilter
Sink
SRC
RtpEndPoint
Media
Source
Media
Sink
Client side Client sideServer Side

20. Making it more fun: introducing real time media
 Source code …
19

21. Programming with media events: the ZBarFilter
20
Real time
media
ZBarFilter
Sink
SRC
RtpEndPoint
Media
Source
Media
Sink
HttpEndPoint
Media
Source
Media
Sink
Media events to browser
Client side Client sideServer Side

22. Programming with media events: the ZBarFilter
 Source code …
21

23. Digging into the media API: implementing your own
filters: requirements
 Requirements
• GStreamer
› Based on GObject and GLib
› C programing skills needed
• OpenCV or your preferred image processing software
• C++ basic knowledge
• Thrift basic knowledge
22

24. Digging into the media API: implementing your own
filters: GStreamer media element implementation
 Download media server source code
 Look for sub-module gst-kurento-plugins
 Enter into src/filters folder
 Create your filter as couple of filtername.c filtername.h files
• Your filter must inherit from GstVideoFilter
• You have to implement virtual method transform_frame_ip
› This method gives you each frame to be processed
• Be aware of init and finalize functions if you use attributes that need to be freed
23

25. Digging into the media API: implementing your own
filters: Integration in media server
 Modify kms-interface/mediaServer.thrift file adding a new value to enum FilterType
 Create a C++ file to manage your GStreamer filter (by convention we use the same name in camel
case)
• This class inherits from kurento::Filter class
• Its constructor creates a “filterelement” gstreamer element and add your filter name as a
parameter (it will be created automatically)
• Destructor should free this element
 Modify kurento::MediaPipeline::createFilter method in order to add your filter type in the switch
statement
24

26. Facing the FI-WARE challenge using the SO-GE
 Getting started
• Use the FI-WARE Catalogue (ready since 15 minutes ago hopefully!)
› http://catalogue.fi-ware.eu/enablers/stream-oriented-kurento/
• Ask the SO-GE team around the FI-WARE stand
 Augmenting videos in novel ways
• Sensor information on live stream videos
› Pollution/noise/traffic  Red-filtering level
• Combine audio-visual & sensor information through augmented reality
› Thermometers, charts, gauges on top of a media flow.
 Make a camera to become an advanced sensor using computer vision
• Traffic status
› Car counter
• Weather
› Sky/ground color
 Conceive novel WWW and Smartphone interaction GUIs
• Up to your imagination
25

27.  http://fi-ppp.eu
 http://fi-ware.eu
 Follow @Fiware on Twitter !
Thanks !
26

28. Technical details: Javascript KwsContent API
 Creating a media connection
• conn = new KwsContent(uri, options);
• Options
› remoteVideoTag: id of the video tag rendering the remote stream
› Others for WebRtc (currently not available)
• Semantics
› Connects to URI and requests media
› Upon media reception, renders it on the specified video tag
 Adding event handlers
• conn.onstart( function …
• conn.onremotestream (function …
• conn.onterminate( function …
• conn.onmediaevent( function …
27

29. The SO-GE Content API: handlers
 Handler
• Is implemented by the application developer (equivalent to a Servlet)
• Defines the sequence of instructions executed when a request for media is received
 Four types
• PlayerHandler
› Handles requests for playing media with HTTP transport
› Compatible with the HTML5 video tag
• RecorderHandler
› Handlers request for recording media with HTML transport
› Compatible with HTTP file upload mechanism
• RtpMediaHandler
› Handlers request for establishing RTP media sessions
• WebRtcMediaHandler (under development)
28

30. Applica on logic
Applica on Server (JBosss/Mobicents)Media Server
Flash Video
Server
HTTP ServeletSIP Servlet
Media
element
proxies
Media Session (signaling)
Server side signaling plane
IP Network (remote clients)
Media Session (media plane)
SIPSignaling
Media
events
RESTfulSignaling
RAWHTTPSignaling
OtherSignaling
Other JEE
SOAPSignaling
Media applica on (App. logic)
Media
Repository
Input
Element
Decoder
AR Filter
CV Filter
Encoder
Adaptor
Output
Element
Incomingmediastream
Outgoingmediastream
Media message bus
Control
commands
Technical details: the SO-GE Architecture
29

31. PlayerHanlderinternaldetails:
sequencediagram
30
Player Handler Servlet Media ServerClient
Player Handler
(Applica on logic)
PlayRequest
Applica on Server
POST Content URL
create()
onMediaRequest(playHandler)
Addi onal logic
play(source)
create(h pEndPoint)
connect(source,
h pEndPoint)
h pEndPoint.getURL
Assigned URL
(JSON object)
200 OK
(JSON object)
build media
pipeline
createPipeline
create(media-element)
create, connect, etc.
GET URL
Media