Cost, Risk, Loss and other fun things

Cost, Risk, Loss and other fun things Screening the Future 15 March 2011 Matthew Addis, IT Innovation Centre [email_address]

Themes: cost of compromise Costs Opportunities Risks

Themes: storage and online access ,[object Object],[object Object]

Costs ,[object Object],[object Object],[object Object]," Preservation is the totality of the steps necessary to ensure the permanent access ibility – forever - of an audiovisual document with the maximum integrity ". (CCAAA)

British Library: LIFE cost model L T Total cost Aq Acquisition cost I Ingest cost M metadata cost Ac Access cost S Storage cost P Preservation cost. The subscript T means that costs have to be calculated over the lifetime of the items being preserved.

[object Object],[object Object],[object Object],[object Object]

NASA Cost Estimation Toolkit ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Some ‘numbers’ for AV ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

COSTS: STORAGE AND ONLINE ACCESS

Trend: increasing storage capacity ,[object Object],[object Object],[object Object]

Trend: increasing recording density http://www.americanscientist.org/issues/pub/2010/3/avoiding-a-digital-dark-age

Trend: storage cost improvement ,[object Object]

TCO of storage + processing (Google)

Access costs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

TCO over time: migration ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

TCO over time: falls more slowly ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Implications: ‘Forever’ costs of storage ,[object Object],[object Object],Half life of annual cost Multiplier for ‘forever’ cost 1 2 2 3.3 3 4.9 5 7.7 10 14.9

Implications: outsourced archive hosting ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Risks ,[object Object],[object Object],[object Object],[object Object]

Lots of risks ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Files: 37 risks from ‘IT’ ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Example risks Risk ID Title Example R30 Hardware Failure A storage system corrupts files (bit rot) or loses data due to component failures (e.g. hard drives). R31 Software Failure A software upgrade to the system looses or corrupts the index used to locate files. R32 Systems fail to meet archive needs The system can ’ t cope with the data volumes and the backups fail. R33 Obsolescence of hardware or software A manufacturer stops support for a tape drive and there is insufficient head life left in existing drives owned by the archive to allow migration R34 Media degradation or obsolescence The BluRay optical discs used to store XDCAM files develop data loss. R35-R38 Security Insufficient security measures allow unauthorised access that results undetected modification of files.

Loss of data authenticity and integrity (origins) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Loss of data authenticity and integrity (things at risk) ,[object Object],[object Object],[object Object]

Loss of data authenticity and integrity (consequences) ,[object Object],[object Object],[object Object],[object Object],[object Object]

Loss of data authenticity and integrity (counter measures) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

RISKS: STORAGE AND ONLINE ACCESS

Trend: obsolescence ,[object Object],[object Object],Medium Storage Density bits/cm² Life, years Stone 10 10000 Paper 10 4 1000 Film 10 7 100 Disc 10 10 10

Data tape (LTO) 6 years 2 years Ultrium LTO roadmap

Data tape ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Eggs in one basket ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HDD error rates ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HDD lifetime ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HDD failure rates ,[object Object]

The IT Industry knows this already

But systems bring their own problems ,[object Object],[object Object]

‘ bit rot’ ,[object Object],[object Object],[object Object],[object Object],2007 study into data corruption by CERN David Rosenthal’s blog http://blog.dshr.org/

Cost of reducing chance of loss ,[object Object],[object Object],[object Object],[object Object]

Cost of not reducing loss (1) JPEG2000 with one error per 100KB Volker Heydegger study on file format sensitivity to corrupton ,[object Object],[object Object]

COST OF RISK: STORAGE AND ONLINE ACCESS

Storage: cost of risk of loss ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Comparing ‘cost of risk of loss’ ,[object Object],[object Object],[object Object],[object Object]

Approaches ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Detailed Comparison ,[object Object]

Tipping points ,[object Object],[object Object],[object Object],[object Object],[object Object],Increasing archive size

Two tools ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Challenges ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Approach ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Examples Data tape on shelves HDD in servers Storage as a Service Storage Cost Low (media, shelves, climate control) High (servers, power, cooling, maintenance) High (fully managed service) Access Cost High (people retrieve and load media) Low (internal network, automated) High (bandwidth, charges for i/o) Latent Failures Low (data tape is reliable) Med (‘bit rot’) Low (replication and monitoring) Access Failures Medium (drives eat tapes) Low/Medium (depends on system) Low (automated checks)

Use of resources ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Can I use the tools? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Making the plan happen ,[object Object],[object Object],[object Object],[object Object],[object Object]

Trust Questionnaire ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Questionnaire results Governance AV Material Management Utility

SLA Terms ,[object Object],ID Name Description Metrics Bounds Monitoring frequency QS-01 Availability The guarantee that the service will be available (up and exploitable) ME-01 The availability should never go below a specific threshold, there could be more than a threshold (e.g. for business hours and night) Once per fixed period like month or year OR on a sliding window with an appropriate width and moving with a convenient step QS-03 SIP ingestion time Total elapsed time from the SIP submission to the confirmation from the system that everything has been correctly acquired. ME-05 The SIP ingestion time should never go above a specific threshold, there could be more than a threshold (e.g. for business hours and night). It can be given as percentage, e.g. 90% of deliveries are done under a threshold 1 and the rest under threshold 2. Every time there is a SIP ingestion or periodically if percentage check is assumed on the base of precalculated statistics

SLA Terms ,[object Object],ID Name Description Units Metrics Bounds C-02 Maximum amount of storage The maximum amount allowed to a specific customer by contract GBytes ME-04 The occupied storage should never cross a specific threshold, partial exceeding for a limited period of time could be tolerated C-03 Maximum number of simultaneous users Maximum number of users logged in at the same time Positive integer ME-14 The actual number of logged users should never cross a specific threshold, partial exceeding for a limited period of time could be tolerated

SLA Terms ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Automation ,[object Object],[object Object]

Management Services and Resources Ingest, Access, Quality Control, Metadata validation… Management Modelling Set policies Service Manager Resources, SLAs: Monitoring, Managing, Automation

Content Producers and Consumers Suppliers e.g. 3 rd party storage Modelling Set policies QoE and QoS Workflow and Access control

Cost, Risk, Loss, Opportunity ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Costs Opportunities Risks

Some reference documents from PrestoPRIME ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

More ,[object Object],[object Object],[object Object],[object Object]

Cost, Risk, Loss and other fun things

Recomendados

Recomendados

Más contenido relacionado

Similar a Cost, Risk, Loss and other fun things

Similar a Cost, Risk, Loss and other fun things (20)

Más de PrestoCentre

Más de PrestoCentre (6)

Último

Último (20)

Cost, Risk, Loss and other fun things

Notas del editor