This document summarizes the key components of a digital media production infrastructure, including the network, hard disks, and file systems.
The network must support high throughput without packet loss or delays to support real-time media workflows. Point-to-point switched Ethernet and layer 3 routing address limitations of shared Ethernet.
Hard disks are most efficient with large sequential access rather than small random access. Disk striping and intelligent file systems can reduce random access.
File systems must support non-blocking access across many clients and disks without oversubscription to ensure resources are available for media production needs.
2. Research and Development +/- 22 experts Member of international associations EBU (P/SCAIE, P/MAG, P/NP, P/HDTV,…) AMWA, SMPTE, IPTC, W3C Deliverables Expertise Proof of Concept Technology transfer VRT-medialab http://medialab.vrt.be
7. Future Media - Assumptions 5 Non-linear Distribution IP(<> FM, DAB, DVB) enables seamless integration of live and on-demand Software-based Receivers API’s and application frameworks enable independant software development 3rd party integration increasses application intelligence Versatile Content From mobile, standard-def and high-definition to scalable content http://labs.vrt.be/myradio
15. 9 Digital Media Production Suprastructure – Metadata management Production and distribution Production and distribution Editing Mastering Media AssetMgnt System Ingest Playout Infrastructure - Networks and Storage Production Platform
16. 10 Challenge 1 - Infrastructure Suprastructure – Metadata management Production and distribution Editing Mastering Media Asset Mgnt Ingest Playout Infrastructure - Networks and Storage Production Platform
17. 11 Challenge 2 – Digital Asset Management Meta Data Meta Data Communication (Information) Suprastructure – Metadata management Production and distribution Infrastructure - Networks and Storage Production Platform
18. 12 Challenge 2 - Digital AssetManagement Suprastructure – MetadataMgnt Production and distribution Infrastructure - Networks and Storage Production Platform
29. By using generic technology, the central media infrastructure and the local media specific work-centres should be integrated agnostically from each others technical implementations.
37. 19 ~ 200 // H-Res video clients simultaneous access at 500 Mbit/s / client (H-Res video: 50 Mbit/s) (= 10 x 50 Mbit/s / client) ~ 3000 // L-Res video clients Simultaneous access + // Throughput ~ 10 GB/s Ingest: >= 10 feeds simultaneous 10 x faster than real time transcoding -> browse key-frame Playout # hrs H-Res video: ~150.000 hrs On disk Total storage: 4 PetaB (net) on-line = on disk Hiërarchical storage mgmt. (on-line/near-line/off-line) near-line = on data-tape off-line = on data-tape (shelf) Design Requirements Media Asset Management File-based / (video)tape-less media environment High Availability
38. 20 Target architectural model: ~ 200 // H-Res video clients simultaneous access at 500 Mbit/s / client (H-Res video: 50 Mbit/s) (= 10 x 50 Mbit/s / client) Media Application Behaviour ~ 3000 // L-Res video clients Simultaneous access + // Throughput ~ 10 GB/s Ingest: >= 10 feeds simultaneous 10 x faster than real time transcoding -> browse key-frame Playout # hrs H-Res video: ~150.000 hrs On disk Total storage: 4 PetaB (net) on-line = on disk 0% 100% Load Hiërarchical storage mgmt. (on-line/near-line/off-line) near-line = on data-tape off-line = on data-tape (shelf) Media Asset Management File-based / (video)tape-less media environment High Availability Design Requirements
39. High Definition Media Production Infrastructure Media File System ( GPFS, AVID-ISIS, Isilon, Omneon Media Grid, XSAN, … ) Network ( Disk network, SAN ) Hard Disk Storage Architecture: 3 Key Components: 100 % Efficiency Geisha results IBC 2009
63. 28 or is this behavior typicalto the way it is implemented? > 60 % load Is this behavior inherent to IP 60 % load 30 % load 5 % load Conventional IT network Network Behavior
92. Max 30% efficiencyMedia network – L2 switching and L3 routing
93. 34 Classical IT (IP) Network -> Media IP Network 100 % load > 60 % load ≈ 60 % load 50 % load 30 % load 5 % load 5 % load Media IP network Classical IT (IP) network #p t #p t collision #p t #p t
121. 45 Essence Media essence Media Applications Media Clients Media Essence Files OSI 7 layer model File File System Protocol (FTP,NFS,SMB) Data network File I/O Requests Application Layer Application Layer File System File Presentation Layer Presentation Layer Session Layer Session Layer LUN I/O blocks Storage Network Transport Layer Transport Layer SCSI Protocol Block I/O Requests Network Layer Network Layer Data Link Layer Data Link Layer Storage Controller LUN Physical Layer Physical Layer Physisal Disk Logical segments Virtual connection Disk Network SCSI Protocol Segment I/O Requests Hard Disk Physical Disk Physical sectors Bytes Layered Model for Storage Infrastructure
122. 46 FC-client Based File servers Clients FC Network Avid-Unity / Quantel / SGI-CXFS / Thomson GV / … Video Proprietaire oplossingen Schaalbaarheid??? Media Infrastructure File System
123. 47 FC-client Based IP-client based Clients Clients Clients File servers Clients Media IP Network IBM-GPFS / SGI-CXFS / HP-Lustre / ADIC-Stornext / Nexsan / … FC Network File servers cluster Parallel access file system Avid-Unity / Quantel / SGI-CXFS / Thomson GV / … Video SAN Proprietaire oplossingen Schaalbaarheid??? MXF-AAF video/audio file Media Infrastructure File System
126. 49 3 x Pinnacle chrome connected to GPFS 1 x AVID connected to local storage NLE NLE NLE NLE ARDCAP Playout server XP 2004 - Proof Of Concept - (VRT/IBM) VTR Admin Clients (XP) PC clients (XP) Dart/Easycut/Ardlog/… SDI -> VTR -> VTR File -> Application server + Web server +DB2 ARDCAP Ingest system XP 1 Gbit/s per client SuSE SuSE -> SDI SUPERNET II Media Gb Back End 2x OS8800 10 Gbit/s 4x 1 Gbit/s per server IP ARDENC Transcoding system 4 x GPFS i-node x365 (Intel/Linux) EDL Conform SuSE SuSE 4 CPU 2 CPU 2 CPU 2 CPU Tivoli TSM+Client x365 (Intel/Linux) FC network 2x MDS9216 2x 200 MB/s per server FC FC 4x 200 MB/s per storage array media-file LTO-2 taperobot 72 slots (14 TB) FAStT 900 84 x 146 GB RAID 5 FAStT 900 84 x 146 GB RAID 5 GPFS Soft mirror 10 TB net (400 hrs)
127. 50 Media Applications Essence Media essence Files Protocol Efficiency Over IP network File System Capability I/O blocks FC Oversubscription Flow Control I/O block size vs. segment size + CPU limit Controllers Logical segments Segment size vs. Efficiency of FC-AL Physical sectors Disk Segment size vs. Disk efficiency Media Essence Media Clients File IP network (SUPERNET II) File I/O Requests File GPFS File System LUN FC Storage Network Block I/O Requests LUN DS4500 Physical Disk FC-AL Loops DiskNetwork Segment I/O Requests Physical Disk FC Hard Disk Bytes Original Proof of Concept – Standard IT Systems
128. 51 Media Applications Essence Media essence MPx3 PoC Media Clients Files Media IP Network IP network GPFS File System GPFS I/O blocks FC SAN DS4500 Standard IT Storage System Standard IT Storage System Logical segments FC-AL Hard Disk Model 10 MB/s FC- Hard Disk Physical sectors Original Proof of Concept – Standard IT Systems Media Essence Media Clients File File System Protocol (FTP,NFS,SMB) Data network Optimised File I/O Requests File File System Optimised LUN SCSI Protocol Storage Network Optimised Block I/O Requests LUN Storage Controller Physical Disk SCSI Protocol Disk Network Optimised Segment I/O Requests Physical Disk Hard Disk Bytes
140. 56 Problem 1 - Digital AssetManagement Suprastructure – Metadata Mgnt Production and distribution Infrastructure - Networks and Storage Production Platform
141. 57 The Solution - Media Asset Management System archiefnummer : ALG 20010813 1 fragmentnummer : 1 reeks : 1000 ZONNEN EN GARNALEN bandnummer : E03024404 formaat : DBCM fragmenttitel : 1000 ZONNEN & GARNALEN beeld : KL/PALPLUS fragmentduur : 18 20 tekst : 0'00" TOERISTISCH REPORTAGEMAGAZINE OVERZICHT ONDERWERPEN GENERIEK TOERISTISCH REPORTAGEMAGAZINE, OVERZICHT ONDERWERPEN 0'50" VANDAAG : KUNSTENAAR LUC HOFKENS ONTWIERP EEN OASE OP ZIJN DAKTERRAS IN BORGERHOUT DIE DOET DENKEN AAN DE GRAND CANYON INTERVIEW MET LUC EN ZIJN VROUW MARILOU BUITENBEELD DAK MET OMGEVING BUITENKANT ARBEIDERSWONING, PANO OVER ROTSWANDEN, KRATEN MET WATER, BEPANTING, FOTOALBUM MET VERLOOP WERKEN 4'00" JUNIOR : KLAARTJE ALAERTS, 13 JAAR WIL ASTRONAUTEN WORDEN ZE BEZOEKT HETEUROSPACE CENTER METRUIMTEVEREN, RAKETTEN SIMULATIE IN RUIMTEVEER, INTERVIEW, HEEFT EEN UFO GEZIEN MAAKT ZELF KLEIN RAKETJE, SCHIET HET AF 7'50" DE SCHEURKALENDER : ARCHIEF RECLAMEFILM IBM INTERVIEW MAURICE DE WILDE, EERSTE PERSOONLIJKECOMPUTER trefwoorden : BELGIE; BORGERHOUT; ARTIEST; OASE; KUNST; GRAND CANYON (NATUURGEBIED); DAK; TERRAS; INTERVIEW; EURO SPACE CENTER; RUIMTEVAART; PC; BOOTTOCHT; RIJKDOM; PASSAGIER; GASTRONOMIE; RESTAURANT; PERSONEEL; VAKANTIE; BINNENBEELD; SCHIP; BECKERS LEEN; VRT; LOTTO; RADIOOMROEPSTER; KLANKSTUDIO; UITVINDING; BARBECUE; BETONMOLEN; IBM; RECLAMESPOT rechthebbende : VRT Opzoekscherm FILM Set: 16 Aantal: 1 blz 1 van 3 trefwoorden: ibm and vrt archiefnummer: - uitzendjaar: maand: dag: fragmentnummer: fragmentduur: reeks: formaat: bandnummer: aflevering: afleveringsnummer: programma: uitzenddatum: fragmenttitel: tekst: kategorie: opnamedatum: opnamenummer: journalist: rechthebbende: SETS The strings required for the operation are not defined F11 F12 F13 F14 F17 F18 F19 F20 Ent Eindigen Sets Refset Toon Vorige Volg/Leeg Thesaurus Commando Opzoeken
149. Ad hoc plumbing (“best of breed”) compromises stability and quality of the product
150. The worst-case scenario is ad hoc plumbing by using an Enterprise Service BusMedia Asset Mgnt
151.
152. “The ESB should replace all application specific interfaces by an general purpose abstract interface”(IBM). At the contrary, it should implement some major interfaces without further custom development (NewsML-G2, TVAnytime, Mpeg7,…);
153. It is assumed that “the SOA and the ESB simplify your service portfolio” (e.g. a single video transcoder), but leads to massive overconsumption of IT infrastructure(!);Supply Chain Engine More, better and structuredmetadataenable the evolutionfrombare system integration to information integration and supplychain optimisation.
163. Information integration has priority on application integration(!!!). Semantic technology will enable(!) normalisation of semi-structured information.An architecture based on an extensible data-model and a consistent application framework (≠ Digitizing analogue and disintegrated information flows)
164. Lessons Learned – Principles of System Integration Everything should be made as simple as possible, but not one bit simpler.
172. APICS (www.apics.org) is een industrieassociatie die het logistiek vocabularium standaardiseert.
173. Implementatie - Logistieke informatie vormt de basis van een ERP systeemen wordt meestel gecommuniceerd als inhoud van een document.Sales and Operations Planning
174.
175. Per industrie en per type product zijn er vaak verschillende methodes met bijhorende documentatiestandaarden:
176. Nieuws maakt gebruik van NewsML(www.iptc.org/newsml)
177. Movie Script Markup Language (MSML) is in ontwikkeling
178.
179. Wanneer feature extraction algoritmes worden gebruikt om diverse visuele, aurale of conceptuele entiteiten te herkennen, maken we bij voorkeur gebruik van ISO/IEC Mpeg7 (http://en.wikipedia.org/wiki/mpeg7).
180.
181.
182. De EPG informatie wordt meestal gecentraliseerd in een distribution scheduling systeem, en dit publiceert op haar beurt het zendschema naar diverse platformen. We kunnen bijvoorbeeld P/META (www.ebu.ch) recupereren om deze interface te implementeren.
193. Timestamped metadata enables direct random access to audiovisual materialSearch Client (Custom Development) Legacy Video Library (Basisplus) Raw Material (EBU Superpop) Media Asset Management System (Ardome) Search Engine (Lucene/SOLR) Actual news items (Ardome)
208. In Italy for 30 years under the Borgias they had warfare, terror, murder and bloodshed but they enabled Michelangelo, Leonardo da Vinci and eventually the Renaissance. In Switzerland they ‘ve had brotherly love and piece for 500 years and they’ve been able to produce…
SMPTE: Society of Motion Picture and Television Engineers.SMPTE serves its members with the latest technology information and education on a rapidly changing industry. SMPTE also is an accredited and globally-respected industry standards-setting body.
In principe: - Automatisch verwerking metadata- Automatisch generatie metadata- Beter indexering en zoeken in (bestaande) meta- & mediadata