SlideShare una empresa de Scribd logo
1 de 56
Descargar para leer sin conexión
Research Paper
Data Storage Considerations for the Tactical Field Collection of Digital Imagery
INTL625 Imagery Intelligence
20 April 2010
Robert L. Watson
2
Table of Contents
Introduction................................................................................................................................................. 3
Chapter 1 Introduction to Storage Media ................................................................................................ 5
Magnetic Tape ........................................................................................................................................... 6
Optical Disks.............................................................................................................................................. 7
Magnetic Hard Disks ............................................................................................................................... 11
Solid State Drives .................................................................................................................................... 12
Redundant Array of Inexpensive/Independent Disks (RAID) ................................................................. 17
Chapter 2 Introduction to Video Compression...................................................................................... 25
Video Compression Standards................................................................................................................. 28
Metadata................................................................................................................................................... 30
Multimedia Players and Container Formats............................................................................................. 34
Chapter 3 Recommendations................................................................................................................... 37
Hardware Recommendations................................................................................................................... 37
Software Recommendations .................................................................................................................... 39
Video Retention and Deletion Schemes................................................................................................... 41
The Future................................................................................................................................................ 44
Chapter 4 Conclusion ............................................................................................................................... 46
Bibliography.............................................................................................................................................. 53
3
INTRODUCTION
As a result of the Global War on Terror, the need for timely intelligence, both on a global
as well as local scale has facilitated the need for more sophisticated imagery intelligence
gathering systems that provide useful information from satellites, unmanned aerial vehicles and
stationary surveillance systems, to name but a few. This large data gathering effort is
unprecedented in history and presents new challenges to our country and its government.
A recent article in the New York Times stated: "Military deluged in intelligence from
drones: Remote-controlled planes produce about 24 years' worth of video in 2009."1
Other
recent news has described the U.S. Intelligence community’s inability to preempt the Christmas
Day 2009 terrorist attempt as a byproduct of "information overload."2
Out of necessity, the need for timely imagery intelligence that provides support to both
our military and government has created the requirement for large groups of skilled analysts to
evaluate the enormous amount of imagery data that is being collected to determine what is
significant to the intelligence effort. In light of this, large-scale digital storage systems are
necessary to provide data storage and retrieval.
However, not all of the imagery that is collected is useable or of significant intelligence
value. In some instances, the time or resources may not exist to have the imagery reviewed and
analyzed in a timely manner, so its value as an intelligence source may diminish.
1
Christopher Drew, “NYT: Military deluged in intelligence from drones - Remote-controlled planes produced
about 24 years’ worth of video in 2009,” New York Times, January 10, 2010, Sunday;
http://www.msnbc.msn.com/id/34798080/ns/world_news-he_new_york_times/ (accessed January 11, 2010).
2
Dan De Luce, “US spy agencies face information overload: experts,” AFP (via Yahoo News), January 7, 2010,
Thursday; http://news.yahoo.com/s/afp/20100107/pl_afp/ usattacksintelligence (accessed January 25, 2010).
4
The paper will explore the current storage technologies and methodologies in use, their
advantages and shortcomings, and provide some insights and possible solutions to this ever-
growing issue.
The research will focus primarily on solutions for real-time digital video gathering
systems, specifically stand-alone, tactical systems currently used in the field environment. Since
a system’s storage capacity and the means of increasing this capability may be limited due to
system design and mission constraints, newer storage technologies and methods will be
suggested to include specific metrics, both user-initiated and software-based, that could be used
as criteria in determining imagery retention and disposal.
5
CHAPTER 1
INTRODUCTION TO STORAGE MEDIA
Digital storage devices are the building blocks of storage in disk subsystems and
standalone server systems. Functioning in the microscopic realm, they perform the reading and
writing functions necessary for the storage of data on nonvolatile media. Digital image
processing of remote sensor data and its associated geographic information system (GIS) data
requires significant storage resources, which are also necessary for the collection of large
imagery databases.3
Table 1 illustrates some of the more common digital mass storage devices and their
average time to physical obsolescence – the point at which the media begins to deteriorate and
data loss can occur.
Table 1. LONGEVITY OF DIGITAL STORAGE MEDIA.
AVERAGE LONGEVITY OF DIGITAL STORAGE MEDIA
MEDIA TYPE
AVERAGE OBSOLESCENCE
(IN YEARS)
REMARKS
Optical Disk >100 Very cost efficient, presently best means of long-term, digital
storage
Magnetic Disk
(Hard Disk Drive)
20 Main system storage medium, but storage longevity normally
less than 20 years.
Magnetic tape 10-15 Cost efficient, but may become unreadable if not rewound and
properly stored in a cool, dry place.
Flash-Solid State
Disk*
10 Expensive, but provides faster seek time than magnetic hard
disks. Data retention for up to 10 years without power applied.
Jensen, John R., Introductory Digital Image Processing: A remote Sensing Perspective, New Jersey: Prentice-Hall, 2005, 118-119.
*Tudor, Marius, “Are Flash Solid-State Disk Drives Ready for the Enterprise?,” Bit Micro Networks, Inc., 2009, 1 (accessed March
2, 2010).
The storage of remote sensor data can significantly impact system reliability and costs.
Substantial resources are required for even the most simple of systems. In light of this, mass
3
Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 69.
6
storage media should permit rapid retrieval of required imagery data, provide longevity, and be
cost effective.4
MAGNETIC TAPE
Tapes are low cost and find frequent use as storage and backup media. They are linear
access devices as all data is written to or read from them sequentially.5
They come in several
types varying in size, storage capacity, density, length, thickness, number of tracks and reels, and
speed. They are composed of magnetic tape with single or dual reels contained in a plastic
enclosure.6
Data in older tape drive technology was written by multiple heads in a parallel track
across the entire tape with some drives using a helical scanning method which wrote the data
diagonally. A linear serpentine recording method is used for modern tape drives, which requires
more tracks and fewer tape drive heads. Data is written using the linear method except that data
is continually written with the head being adjusted and reversed once the end of the tape is
arrived at.13
Tape media is not without its drawbacks. Since access to data stored on magnetic tape is
linear or sequential, it is not considered efficient for random data access. The linear nature of
tape also makes recovery and back up operations time consuming. This is the reason why tape is
not considered for use as primary system storage, but is used mainly for offline data storage and
vaulting.7, 8
4
John R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey:
Prentice-Hall, 2005), 117.
5
Shrivastava. Op. Cit. 269.
6
Ibid. 33.
7
Ibid.
7
Additionally, magnetic tapes slowly deteriorate over time developing surface cracks,
tears, and corrosion of the metal oxide coatings. Whether they are being used or stored for future
use, they should be maintained in environments having moderate temperatures and low
humidity.9
Improper maintenance (e.g., not rewinding the tape) and storage can cause magnetic
tape media to become unreadable within ten to fifteen years.10
OPTICAL DISKS
An optical disk is another recordable storage media which comes in a variety of types to
include compact disks (CDs), digital video/versatile disks (DVDs), magneto-optical (MO) disks,
and Blu-Ray disks (BDs).11
Optical disks comprise three broad categories, which determine their
usability. These categories are as follows12
:
1. Read only optical disks, which are recorded when they are manufactured and cannot
be altered or erased. They include Compact Disk (CD), Compact Disk – Read Only
Memory (CD-ROM), Digital Versatile/Video Disk – Read Only Memory (DVD-
ROM), Digital Versatile/Video Disk – Video (DVD-Video), and Blu-Ray Disk (BD).
2. Write Once Read Many (WORM) optical disks can be recorded once and cannot be
erased. These include Compact Disk – Recordable (CD-R), Digital Versatile/Video
Disk – Recordable (DVD-R), and Blu-Ray Disk – Recordable (BD-R).
3. Rewriteable/Magneto optical disks, which can be written, erased and read from any
number of times. These include Compact Disk – Rewriteable (CD-RW), Digital
Versatile/Video Disk – Rewriteable, Blu-Ray, and magneto-optical (MO) disks.
Regardless of type, they are manufactured using similar technologies which incorporate a
thin polycarbonate disk which is impressed with microscopic bumps arranged in a continuous,
8
Josh Judd, Principles of SAN Design: Design Build and Manage SANS, (Pennsylvania: Infinity Publishing,
2007), 18-19.
9
Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 83.
10
John R. Jensen, Introductory Digital Image Processing: A remote Sensing Perspective, (New Jersey:
Prentice-Hall, 2005), 118.
11
Encyclopedia.com, "Optical Disk," The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010).
12
Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006,
http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
8
spiral track of data. Following this, the disk is coated with a thin, reflective layer of aluminum
(barium ferrite for magneto-optical disks) to cover the bumps. A thin acrylic layer is then
sprayed over the metallic coating for protection.13, 14
Common optical disk media and their
typical storage capacities are shown in Table 2.
Table 2. COMMON OPTICAL STORAGE MEDIA.
OPTICAL MEDIA TYPICAL STORAGE
CAPACITY
REMARKS
CD/CD-ROM 700Mb Recorded at time of manufacture and cannot be
altered/erased
CD-R 650Mb Write Once Read Many (WORM) media. Once
written, can only be read from.
CD-RW 650Mb Data can be written/erased/read from the disk
any number of times.
DVD/DVD-ROM 9.4Gb
(4.7Gb per side)
Recorded at time of manufacture and cannot be
altered/erased
DVD-R 9.4Gb
(4.7Gb per side)
Write Once Read Many (WORM) media. Once
written, can only be read from.
DVD-RW 9.4Gb
(4.7Gb per side)
Data can be written/erased/read from the disk
any number of times.
BD*
25Gb (single layer)
50Gb (double layer)
Recorded at time of manufacture and cannot be
altered/erased
BD-R*
25Gb (single layer)
50Gb (double layer)
Write Once Read Many (WORM) media. Once
written, can only be read from.
BD-RE*
25Gb (single layer)
50Gb (double layer)
Data can be written/erased/read from the disk
any number of times.
MO**
128Mb to 9.2Gb
depending on disk size
Data can be written/erased/read from the disk
any number of times.
Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006,
http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
*Wikipedia contributors, "Blu-ray Disc recordable," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc_recordable&oldid=346269932 (accessed March 12, 2010).
**Wikipedia contributors, "Magneto-optical drive," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Magneto-optical_drive&oldid=338693032 (accessed March 12, 2010).
Optical disk systems write the data to the media using a low-power laser that etches
binary bits into the reflective layer. In this technique, the bits are heated to 150 degrees
centigrade, which are then realigned when a magnetic field is applied, creating a binary bit one.
13
Encyclopedia.com, "Optical Disk," The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010).
14
Mediatechnics Systems, Inc, "How a CD is Made," Mediatechnics Systems, Inc., FAQ,
https://www.mediatechnics.com/cdfaqs.htm (accessed March 5, 2010).
9
The recording of new data requires that existing bits be reset to a binary bit zero.15
To read
information from the disk, polarized light from a low-power laser is rotated according to the
direction of the magnetic field and the original binary signal is reproduced.16
Together with an optical disk drive, optical disks function much the same way as hard
disk drives. Access time components (of the optical disk drives laser seeking a target track and
acquiring the target sector ) are similar to the seek and rotational latencies present in hard disk
drives.17
The use of optical disks over magnetic storage media offers many advantages to include
higher storage capacity, lower cost, high data stability, environmental tolerance, and long shelf
life.18
The long-term storage potential exceeds 100 years and provides the means of storing large
amounts of data on a relatively small media footprint.19
Optical disks have a constant linear velocity which provides a constant read/write
bandwidth when access is sequential. This makes the media appropriate for multimedia backup
and restore operations, which are generally sequential in nature.20
As such, rewriteable CD-RWs
15
R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey: Prentice-
Hall, 2005), 118.
16
Daintith, John, "Magneto-optic Storage," A Dictionary of Computing. 2004, Encyclopedia.com,
http://www.encyclopedia.com/doc/1O11-magnetoopticstorage.html (accessed March 5, 2010).
17
Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 123.
18
Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006,
http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
19
R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey:
Prentice-Hall, 2005), 118.
20
Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003).
10
and DVD-RWs have replaced tapes as the primary backup system in most remote sensing
laboratories.21
However, the recent advent of Blu-Ray, deemed as a future replacement for the DVD
format, offers even greater storage capacity. Having the same physical dimensions as CDs and
DVDs, a standard, double layer Blu-Ray disk can store up to 50GB of data. The Blu-Ray
standard is open-ended with theoretical storage limits left unspecified. Larger disk capacities of
100 and 200 GB are currently available.22
The use of optical disks is not without its drawbacks. In relation to hard disk drives, the
data access latencies are much higher in optical drives. The typical seek time for hard disk
drives is 10msec compared to 100-300msec for optical drives.23, 24
Constant improvements in hard disk drive technology (i.e., price, capacity, and speed)
make the decision to use optical disks less inviting. The development of DVDs did provide
some improvement in the storage capability, but applications are limited with many
organizations still relying on tapes and hard disk drives for business and scientific data storage.
However, the Blu-Ray disk may provide the impetus necessary to facilitate a greater move
towards the use of optical media for large database storage.25
21
Jensen. Op Cit.
22
Wikipedia contributors, "Blu-ray Disc," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc&oldid=349002833 (accessed March 12, 2010).
23
Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 123.
24
The Free On-line Dictionary of Computing, "Average Seek Time,"
http://encyclopedia2.thefreedictionary.com/average+seek+time (accessed March 12 2010).
25
Simitci. Op Cit.
11
MAGNETIC HARD DISKS
The Magnetic Hard Disk, also referred to as a Hard Disk Drive (HDD), is an
electromechanical device which controls the performance of the storage system environment,
providing the primary function of reading and writing the data that is stored on the media. They
are the primary storage medium used on computers for the storage and access of data and
software applications.26, 27
A magnetic hard disk consists of round, magnetic platters which encode digital data with
magnetically charged media. Each platter is arranged in cylinders/tracks with sectors on which
respective data is stored. The platters spin at a high rate of speed, up to 15,000 revolutions per
minute (RPM). Several platters, a read/write head, and a controller constitute the main
components of a HDD. The servo-controlled read/write head is attached to a rapidly moving arm
and is positioned over specific sectors to provide access to data. Since the HDD is a read/write
mechanism, data may be consistently written to and removed from the media.28, 29
HDDs provide quick and simultaneous access to arbitrary data locations and have large
capacities. Multiple disks may be configured into storage arrays providing increased capacity
and improved performance.30
In relation to other storage media like optical disks and tape,
26
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 33.
27
Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 69.
28
Dane Nelson and Sam Siewart, PhD, “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, 30,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf (accessed March 2, 2010).
29
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 27.
30
Ibid.
12
HDDs provide high data throughput and low latency data retrieval.31
However, as degradation
may occur in less than twenty years, HDDs have a comparatively short storage life. The
continual use of a HDD will reduce its life expectancy significantly.32
SOLID STATE DRIVES
Solid state drives (SSDs) use semiconductor flash memory chips for data storage and
retrieval. Unlike mechanical HDDs, they have no moving parts. All processes are operated via a
controller assembly that is also composed of semiconductor materials. SSDs have improved in
recent years and are used when high performance is required for mission-critical
applications.33,34
They use nonvolatile memory which supports persistent data storage and are constructed
in either single-level cell (SLC) or multi-level cell (MLC) configurations, which are used to store
data bits on respective memory cells. SLC storage is used in high performance memory and
stores one bit per cell. MLC memory stores multiple bits per cell providing slower data transfer
rates, but is cheaper to manufacture than SLC memory.35
SSDs using SLC technology and high reliability controllers imitate HDDs via a
traditional storage interface and provide ultra-fast read/write performance, high reliability, high
31
Lawrence D. Bergman, and Vittorio Castelli, Image Databases: Search and Retrieval of Digital Imagery
(New York: John Wiley & Sons, Inc., 2002), 144.
32
Jensen, John R. Introductory Digital Image Processing: A remote Sensing Perspective. (New Jersey:
Prentice-Hall, 2005), 118.
33
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 80.
34
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,”
Intel Technology Journal, Volume 13, Issue 1, 2009, 30,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf
(accessed 03-02-10).
35
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 80.
13
data integrity, and minimized power requirements. This makes them ideal for supporting
applications that require the rapid processing of large amounts of information such as real-time
data feeds.36
HHDs, on the other hand, use rotating media and servo-actuated read/write heads, which
introduce seek and rotation latencies. The device is also prone to mechanical failure. Their ever-
increasing capacity due to advances in the areal density of the recording surfaces and their low
cost are their main advantages, but fast random access limitations have always been an issue.37
As a rule of thumb, central processing units (CPUs) are able to process data much faster
than mechanical HDDs can transfer it. HDDs have been the mainstay of storage for decades
with a storage capacity that has increased over 200,000 times that of the first HDDs developed in
the 1950’s. During that same time period, decreases in price have also made the media more
affordable.38
However, HDD performance has not managed to keep up with processor development.
Within the last 30 years, processors have seen exponential increases in speed with only marginal
increases in the read/write response time of HHDs. This has resulted in a major break between
CPU access capabilities and those of HDDs.39
The performance differences between HDDs and SSDs concern the number of random
reads/writes per second that the device can perform and the time necessary to resume full
36
Ibid.
37
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf , 30 (accessed 03-02-10).
38
Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf , 1 (accessed March 13, 2010).
39
Ibid, 2.
14
operational power from a low power state.40
The number of read/write operations that can be
completed in one second is called IOPs, or inputs/outputs per second. This metric is used to
measure the random read/write performance of different storage media. High performance
HDDs (15K RPM) normally perform 300 IOPs of random 4-kilobyte (KB) data.41
On the other
hand, SSDs are presently capable of processing 25,000+ 4KB IOPs, which is a tremendous
increase.42
Performance in HDDs is also affected adversely by the resumption of operation following
inactivity. When a HDD is inactive for a certain period of time, it will assume a low power state,
moving the read/write head off to the side and stopping the spinning platters. Upon the next
read/write request, the read/write head must be moved back in place and the platters must be
reactivated. This normally takes on the order of a few seconds to occur. However, when an SSD
is inactive and goes into a low power state, reactivation takes only a few milliseconds.43
The
following characteristics are indicative of solid state storage devices44
:
1. Lowest possible access times: 100-1000 times faster than mechanical drives.
2. High bandwidth: provides multiple gigabytes (GB) per second of random data
throughput.
3. High IOPS: provides extremely high random input/output (I/O) performance due to
low access times and high bandwidth.
40
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 30, (accessed 03-02-10).
41
Ibid.
42
Brook Crothers, “Seagate Enters Solid-State Drive Market,” CNet News, December 7, 2009,
http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010).
43
Nelson. Op. Cit.
44
Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf, 10 (accessed March 13, 2010).
15
4. Low price for performance: provides the best possible price/performance of all
storage devices.
5. High reliability: provides the same levels of data integrity/endurance as other
semiconductor devices.
6. Provides more consistent I/O response times.
7. Provides predictable wear and lifespan characteristics.
These characteristics certainly favor solid state drives over mechanical hard disk drives.
However, barriers to the large-scale adoption of SSDs do exist. First, the cost of SSDs is more
than HDDs. These costs are decreasing with the increased demand by organizations that require
the performance, reliability, lower power, and/or resistance to shock and vibration that they
provide.45
Second, the capacity of SDDs is usually much smaller than that of mechanical HDDs.
However, flash density is improving rapidly to meet user demand for increased capacity with
200GB SSDs already commercially available.46,47
Finally, SSDs have a limited number of write
cycles per storage cell. After many program/erase operations, flash memory loses the capability
to retain data. However, increased density and improved write wear-leveling algorithms have
greatly improved the longevity of SSDs, with enterprise-grade SSDs providing as many as 1
million program/erase cycles.48,49
45
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 31(accessed 03-02-10).
46
Ibid.
47
Brook Crothers, “Seagate Enters Solid-State Drive Market,” CNet News, December 7, 2009,
http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010).
48
Nelson. Op. Cit.
49
Jonathan Thatcher, “NAND Flash Solid State Storage Reliability and Data Integrity: An In-Depth Look,”
Storage Networking Industry Association (SNIA) Solid State Storage Initiative, 2009,
http://www.snia.org/forums/sssi/knowledge/education/Solid_State_Storage_Reliability_and_Data_Integrity--An_In-
depth_Look.pdf, 26 (accessed March 13, 2010).
16
Notwithstanding the shortcomings in SSDs, they are finding increased usage in
environments requiring enhanced response times to include high performance computing,
military applications, and data storage.50
The improving performance of SSD technology
permits them to complement or replace HDDs in computer, communications, and consumer
electronics. They have become the storage media of choice in avionics, industrial, medical, and
military equipment, which requires higher reliability under adverse mechanical and
environmental conditions.51
Direct attached storage (DAS) is typically used in aerospace, industrial, government, and
military applications, which require small form factor, durability, reliability, performance, and
the ability to endure extreme field environments. Currently, SSDs are the only practical solution
for these requirements, finding use in aircraft, weather balloons, spacecraft, missiles, ships,
submarines, trains, tanks and other armored vehicles, and portable military computers to name
but a few.52
The increasing requirements for reliability and performance in military, industrial, and
business applications is driving standards for data storage to extraordinary levels with faster
access, mobility, and extreme reliability becoming more imperative. As the performance
limitations of HDDs are reached and the costs of SSDs continue to decrease, SSDs will play a
more critical role in an environment that demands faster, more reliable data storage.53
50
Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf , 10 (accessed March 13, 2010).
51
Marius Tudor, “Are Flash Solid-State Disk Drives Ready for the Enterprise?,” BitMicro, 2009,
http://www.bitmicro.com/press_resources_flash_ssd_enterprise.php, 1 (accessed March 2, 2010).
52
Ibid. 3.
53
Ibid.
17
At the present, the mechanical HDD is the main obstacle to storage performance today,
due to rotational and seek actuation latencies when accessing data. This problem is complicated
when data access is random and of small, distributed I/O. Most hard drives can deliver from a
few hundred to around 100 MB/sec of IOPs when accessing sequential large blocks of data.54
However, this issue can be alleviated via two methods. The first method involves the
direct replacement of HDDs with SSDs, which can provide a tenfold increase in performance.55
The second method involves the use of a Redundant Array of Inexpensive/Independent Disks
(RAID), which can improve HDD performance by allowing concurrent access to data over an
entire array of disks, thereby increasing I/O throughput.56
As such, it is expected that HDDs will
see continued usage where high capacity storage is necessary and SSDs will be used where
higher performance capability is required.57
REDUNDANT ARRAY OF INEXPENSIVE/INDEPENDENT DISKS (RAID)
The term “RAID” stands for “Redundant Array of Inexpensive (or Independent) Disks,”
and it is one of the most important technologies in storage networking. RAID is often associated
with hardware such as disk subsystems and RAID adapters, but it is actually a set of software
algorithms that combine storage input/output (I/O) operations across multiple storage address
locations. RAID is usually applied with disk drives in disk subsystems, but it can also be applied
54
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,”
Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 38 (accessed 03-02-10).
55
Ibid.
56
Ibid.
57
Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf, 10 (accessed March 13, 2010).
18
across multiple disk subsystems in a storage network. It should be noted that these algorithms
are autonomous and apply to any storage media as long as the storage capacities are the same.58
Original research defined five levels of RAID with varying characteristics of
performance, capacity, and redundancy. Of these, levels one and five are the most common.
RAID Level 1 (RAID-1) encompasses simple disk mirroring without the use of parity or striping.
This level does not provide the benefits of performance and scalability related to other RAID
levels. RAID Level 5 (RAID-5) calculates parity values for stored data resulting in more
efficient data redundancy than mirroring.59
Present day has seen the development of other RAID
levels. Table 3 describes the current RAID levels and their main characteristics.60
Table 3. RAID LEVELS.
RAID Type Characteristics
RAID 0 Striping, no redundancy
RAID 1 Mirroring, full redundancy
RAID 10 Mirrored stripes, full redundancy
RAID 2 ECC protection, not used
RAID 3 Byte-interleaved parity, single parity disk
RAID 4 Block-interleaved parity, single parity disk
RAID 5 Block-interleaved parity, rotated parity blocks
RAID 5DP RAID 5 with double parity blocks per stripe
Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 117.
RAID provides four main benefits for data storage. These are as follows61
:
1. Data redundancy.
2. Large capacity storage.
3. Management consolidation of devices and subsystems.
4. Parallel processing for enhanced performance.
58
Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 191.
59
Ibid. 196.
60
Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 117.
61
Farley. Op. Cit. 193.
19
The main building block of the RAID system is the array. RAID architecture allows
combining multiple storage entities, called array members, into a single array that functions as a
single, virtual storage device. A RAID can have two or more arrays assembled from member
disk partitions as shown in Figure 1.62
3 U
RAID
CONTROLLER
MEMBER
MEMBER
MEMBER
MEMBER
MEMBER
ARRAY MEMBER
DISK PARTITIONS
Figure 1. RAID ARRAY WITH DISK PARTITION MEMBERS.
Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 192.
The use of RAID in image and video servers provides fault tolerance, which is the
primary concern for imagery/video databases. These systems attain fault tolerance by either disk
mirroring or parity encoding. Disk mirroring provides fault tolerance by the duplication of data
62
Ibid. 192.
20
onto separate disks. Parity encoding techniques use error-correcting codes to reduce storage
overhead.63
In disk mirroring, data is stored on two separate disks creating two copies of the data – a
mirrored pair, as shown in Figure 2. If one disk fails, the data remains intact on the other
(redundant) disk and the controller continues to provide requested data from the surviving disk.
Replacement of the failed disk automatically instructs the controller to copy data from the
remaining disk to the new one – an operation transparent to the host system.64
4
2 2
1
4
3
1
3
DATA 1
DATA 2
DATA 3
DATA 4
MIRRORING
Figure 2. MIRRORING.
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 55.
Mirroring provides complete data redundancy and enables faster recovery time following
disk failure. On the other hand, it only provides data protection and is not to be confused as a
63
Lawrence D. Bergman, and Vittorio Castelli, eds., Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 145.
64
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 55.
21
substitute for data backup. In this instance, mirroring continuously captures data changes, as
opposed to backups, which capture point-in-time images of the stored data.65
The amount of required storage is twice the amount of data needing to be stored, so
mirroring is considered expensive, being used on mission-critical applications which cannot
suffer data loss. The technique does provide improved read performance since read requests can
be read from both disks. However, there is an obvious deterioration in write performance since
each write request manifests itself as two writes on the disks.66
To understand the parity method of fault tolerance, the concept of “striping” must be
explained. In a RAID array, each disk has a predefined number of contiguously addressable
blocks called “strips.” The series of aligned strips spanning across all of the disks in the array is
called a “stripe.”67
This division and distribution of data across multiple disks is called
striping.68
RAID using this technique does not provide data protection unless mirroring or parity is
used in the process. However, use of striping can considerably increase I/O performance if the
RAID controller is configured to simultaneously access data across multiple disks.69
Parity is a mathematical technique used to re-create missing data. Parity RAID protects
striped data from disk failure through the addition of an extra disk. This disk is added to the
65
Ibid. 55.
66
Ibid. 56.
67
Ibid. 54.
68
Ibid. 433.
69
Ibid. 54.
22
stripe width and is used to hold parity, which provides a redundancy check that ensures
protection of data without having to maintain a full set of duplicate data.70
As opposed to mirroring, the use of parity significantly reduces data protection costs. For
example, in a RAID configuration using five disks (Figure 3), one disk retains the parity
information while the other four retain data. In this case, parity would require only 25 percent
extra disk usage compared to mirroring, which would need 100 percent extra disk usage.71
However, the use of parity is not without its disadvantages. Since parity information is
created by the data currently residing in the disk, it is recalculated for every change in data. This
process is time consuming and negatively affects the performance of the RAID controller.72
3
13
1
3
1
11 2
2
3
1
2
1
1
2
7
5
7
9
PARITY DISK
DATA DISKS
Figure 3. PARITY RAID.
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 56.
Additionally, in the case of a disk failure, the load increase on the surviving disks results
in deadline violations during video stream playback operations. To prevent this occurrence using
70
Ibid. 56.
71
Ibid. 56-57.
72
Ibid.
23
conventional fault-tolerance methods, disk use must be decreased during the fault-free state.
This reduction may be accomplished prior to imagery compression by using partitioning
techniques that take advantage of the characteristics of imagery through the use of two general
methods.73
First, the sequential nature of video access may be used to reduce the system overhead of
on-line recovery in a RAID. This may be accomplished by the computation of parity over a
video stream block sequence, ensuring that retrieved data used to recover a block that was stored
on the failed disk would be requested by the system in the future. The blocks are temporarily
stored in and serviced from the buffer, minimizing the time necessary for the failure recovery
process.74
Second, since human perception is tolerant to minor image distortion, the inherent
redundancies in imagery may be used to reconstruct lost imagery data to a fairly accurate degree.
This method uses error-correcting codes to accomplish this. The imagery is divided into sub-
images that are stored across the array members. The failure of a single disk will result in
fractional losses for several images.75
However, if the sub-images are generated using pixels, and none of the neighboring
pixels belong to the same sub-image, all elements of the lost pixels will be accessible in the event
of a single disk failure. The relationship between the neighboring pixels will allow for an
73
Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 146.
74
Ibid.
75
Ibid.
24
approximate reconstruction of the original imagery without the need for additional information
from any of the other surviving disks members.76
It should be noted that use of the aforementioned techniques will adversely affect the
compression of the imagery due to the reduction of the correlation between pixels allocated to
the same sub-images. This will increase the bit rate, imposing greater load requirements on all
disks in the RAID and reduce the number of video streams that may be retrieved
simultaneously.77
Post-compression partitioning algorithms have been developed to address this
limitation and are discussed in the following section.
76
Ibid.
77
Ibid.
25
CHAPTER 2
INTRODUCTION TO VIDEO COMPRESSION
Present day imagery technology has provided an endless amount of photographic and
video data, which presents a major challenge in its capture, processing and storage. As this
technology has evolved, the need to improve the quality and to reduce the scale of the imagery
has become of primary importance. This is particularly true since the development of high
resolution photography and video has created more complex files, which can be of extremely
large size, requiring more resources to use. This ever-growing database has required the
development of techniques to control the size and complexity of the imagery and to allow ease of
use. The science of digital compression technology was born out of this need.
Compression is the science of reducing the amount of data used to convey information
about an object. Because information has order and patterns, these can be extracted and
reconstructed to provide the fundamental nature of the original information using less data for
transmission and reception.78
Certain restrictions are required for the compression of
multimedia. The encoded/decoded data should provide the best possible quality. Complexity
should be minimal to provide limited data link delay and cost-effective implementation. Modern
compression techniques must compromise between these requirements.79
The goal of compression technology is the elimination of redundant data while retaining
only the information that is essential for the effective reproduction of the original data. In other
words, the number of bits represented by a signal is reduced to provide a smaller data file that
78
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 2.
79
Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke
Wöhrmann B.V., 1998), 45.
26
still represents the fundamental nature of the original data. This technique is accomplished using
either “lossless” compression, “lossy” compression, or a combination of both methods.80
Lossless data compression algorithms permit exact reconstruction of original data
following compression and allow for compression ratios of about four to one.81, 82
It is used
when it is critical that the original data be identical following decompression. This is important
for executable files, source code, and some image file formats like portable network graphics
(PNG) and graphics interchange format (GIF). It has many applications to include ZIP utilities,
and is often used in conjunction with lossy compression methods.83
As opposed to lossless compression, lossy compression does not allow for complete
recovery of the original signal. Only a near approximation is available due to the tradeoff
required to provide reasonable compression of the signal.84
Compression ratios ranging from
eight to one and twenty to one can provide images that are visually comparable to the original. It
is possible to produce higher ratios for lossy compression, but a noticeable difference will exist
between the original and the compressed image.85
The most common use of lossy compression
80
Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 211.
81
Ibid.
82
Wikipedia contributors, "Lossless data compression," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossless_data_compression&oldid=346709055 (accessed March 10,
2010).
83
Ibid.
84
Ibid.
85
Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 211.
27
is for the compression of multimedia data like audio, video, and still imagery. So, it finds greater
use in streaming media and internet telephony applications.86
A standard image compression method consists of three components: pixel-level
redundancy reduction, data discarding, and bit-level redundancy reduction. The normal image
compression process is shown in Figure 4. Note that lossless image compression systems do not
use the data discarding process as compression is achieved solely from the redundancy reduction
processes. Lossy image compression generally uses all three processes, but may omit bit-level
redundancy.87
Figure 4. IMAGE COMPRESSION COMPONENTS.
Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital Imagery (New
York: John Wiley & Sons, Inc., 2002), 212.
The pixel-level redundancy process reverses the mapping of the image, disassociating the
original pixels from the output. This process enumerates the image pixilation (converts it to data
bits) in a way that more corresponds to the human visual system frequency response. This data
is then transmitted to the data discarding process.88
The data discarding process removes insignificant data from the original data stream that
has been divided into separate data bits during the pixel-level redundancy process. It then
discards the data bits that are deemed unnecessary, and retains those that are minimally required
86
Wikipedia contributors, "Lossy compression," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossy_compression&oldid=347940008 (accessed March 10, 2010).
87
Op Cit. 212.
88
Ibid.
28
to produce a near-perfect rendition of the original, without loss of image fidelity. Statistical
image properties and human visual characteristics are considered in this approximation. This
data is then sent for bit-level redundancy reduction for final processing.89
The bit-level redundancy reduction process is considered a lossless process. It is used to
remove and/or reduce any unnecessary dependencies left over from the data discarding process.90
Following compression of the imagery by one or more compression standards such as those
developed by the Motion Pictures Experts Group (MPEG), the imagery is then sent to the
applicable storage location.
VIDEO COMPRESSION STANDARDS
Several video compression techniques exist that use different techniques to reduce data
size.91
The most commonly used family of digital video compression techniques is the Motion
Pictures Experts Group (MPEG) standards, which are briefly described in Table 4.
Table 4. MPEG STANDARDS.
STANDARD OBJECTIVE CHARACTERISTICS APPLICATIONS
MPEG-1
To provide a standard for encoding motion video at
bit rates transportable over T1 data circuits and for
replay on CD-ROM.
Provided audio and video recording at the same
data rate.
Broadcasting in any form and
large-distribution CD-ROMs.
MPEG-2
To provide a standard that included broadcast
quality video.
Provides video interlacing, scalable syntax, a
system layer to handle multiple program
streams.
Most popular standard presently
in use. Used for HDTV and for
compression of other
video/audio data.
MPEG-3
To provide a compression system suitable for high-
definition television (HDTV) broadcasting.
Abandoned when determined that MPEG-2 was
able to accommodate the need for HDTV.
N/A
MPEG-4
To provide for encoding of video and audio at very
low bit rates.
Provides for the combining of multiple video
streams. Increased efficiency and error
forgiveness over MPEG-2.
Provides end-user interaction
with specific applications like
games, interactive TV, and
educational systems.
MPEG-7
To provide a standard for applying metadata
technology to recorded video and audio.
Uses metadata technology to catalog/ index
video/audio files to accommodate search engine
retrieval.
In work, but will provide ease of
retrieval of data in large-scale
databases.
MPEG-21
To provide a complete structure for the management
and use of digital assets.
Provides all infrastructure support for
commercial transactions and rights
management.
In work, but will provide
intellectual property
management and protection.
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 152,172, 196, 268, 276.
89
Ibid.
90
Ibid.
91
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 2.
29
MPEG is a committee that was formed in 1988 under the Joint Technical Committee of
the International Standards Organization (ISO) and the International Electro-technical
Commission (IEC). Their formation was facilitated to derive a standard for encoding full motion
video at rates suitable for transport over T1 data circuits and for replay from CD-ROM at about
1.5Mbits/sec. This was the birth of the MPEG-1 standard, which ultimate goal was to record
sampled audio at 48 kHz together with video, at the same data rate.92
Following the development of MPEG-1, a standard was required that allowed for the
compression, storage, and digital transmission of television broadcast quality video. This project
became MPEG-2, which extended and improved MPEG-1.93
Among other things, MPEG-2
included the provision of video interlacing, a scalable syntax, and the addition of a system layer
to handle multiple program streams. In light of this, MPEG-2 has become the worldwide
standard for both standard- and high-definition transmissions.94
After development of MPEG-2, a standard that would support compression suitable for
high-definition television (HDTV) was deemed necessary. This effort became MPEG-3.
However, it was discarded very early into the program when it became clear that the new
compression algorithms and schemes used in MPEG-2 were capable of handling the HDTV
requirement.95
With the demise of MPEG-3, MPEG-4 was started to accommodate the encoding of
video/audio at very low data rates. Later amendments also provided applications that allowed
92
Ibid. 152.
93
Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke
Wöhrmann B.V., 1998), 45.
94
Symes. Op. Cit. 172-173.
95
Effelsberg. Op. Cit. 45.
30
end-user interaction with the video. This was particularly useful for gaming, interactive
television, educational systems, and high-end data retrieval systems. However, the standard has
not seen wide acceptance due to the popularity of MPEG-2.96
The current amount of digital video and audio information presently available in the
public domain is enormous and continues to grow. To sort, maintain, and access required
information in a timely manner, MPEG-7 has been proposed and may provide the future means
by which digital video is cataloged, stored, and retrieved. The MPEG-7 standard uses metadata
technology to both catalog and index audio-video files using methods that complement data
retrieval.97
The latest effort in video compression by the MPEG organization is MPEG-21, which
also accommodates metadata technology. The standard was undertaken to provide a complete
framework for the use and management of digital resources. Infrastructure support for
commercial transactions and rights management is also an important part of the standard, which
focus is “to enable transparent and augmented use of multimedia resources across a wide range
of networks and devices.”98
METADATA
To better understand what the MPEG-7 and MPEG-21 standards can provide to the
organization and retrieval of multimedia data, the concept of metadata requires further
explanation. Metadata is the “data about the data” or the “information about the information.” It
96
Op. Cit. 196, 222.
97
Ibid. 268-269.
98
Ibid. 276.
31
describes the various attributes of a given piece of information by providing meaning and
context. It also aids in the location, retrieval, use and management of the information.99, 100
Metadata provides many benefits to include101
:
1. Aids in the search for required information.
2. Aids in organization of digital resources.
3. Facilitates interoperability and legacy resource integration.
4. Provides digital identification.
5. Supports data archiving and preservation.
Typical remote sensing systems have enormous amounts of imagery that must be
maintained. To keep track of this data, information about the images is necessary. This
metadata can be stored with the imagery or in a separate database using pointers to link the
applicable imagery with its respective data set. Various metadata may be used in search and
retrieval operations while other metadata may provide user information.102
Metadata, as used in remote sensing systems, consists of several characteristics with
some elements associated with a collection of images and others that specify smaller subsets
called “granules,” which can be a single image or a small set of images. Examples of these
characteristics are as follows103
:
1. General Description, which includes the name, topic, collection version, etc.
2. Data Origin, which describes the platform, instrument, sensor used, etc
3. Spatial Coverage, which specifies the geographic area covered by the imagery in the
collection (e.g., location coordinates of the corners of a bounding rectangle).
99
Cornell University Library, “Moving Theory into Practice: Digital Imaging Tutorial, Chapter 5,
Metadata: Definition, Types, and Functions,”
http://www.library.cornell.edu/preservation/tutorial/metadata/metadata-01.html (accessed January 14, 2010).
100
National Information Standards Organization, Understanding Metadata (Maryland: NISO Press, 2004),
http://www.niso.org/publications/press/UnderstandingMetadata.pdf (accessed January 20, 2010), 1.
101
Ibid.
102
Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 62.
103
Ibid.
32
4. Temporal Coverage, which specifies the time range during which the imagery was
obtained (e.g., start/end date/time).
Standards for the collection of geographic data have been established by the U.S. Federal
Geographic Data Committee (FGDC). All geographic data that is managed by federal
government agencies must comply with this standard. The standards serve as a guideline for the
use of metadata in remote sensing applications.104
The ISO has also established a working group
on metadata for the purpose of developing standards for the recording of metadata on a separate
track than the audio-video. This would provide a powerful means of retrieval for later access to
the digitized data and illustrates the importance of the MPEG-7 and MPEG-21 standards.105
The
rest of this section will discuss metadata in regards to the MPEG-7 standard.
The scope of the MPEG-7 standard is illustrated in Figure 5. From the figure, it is clear
that only the syntax (Standard Description) is standardized. Devices that might generate
metadata and how that data is represented are specified by the standard. Generation of metadata
(i.e., Feature Extraction) and the applications that might use it (i.e., Search Engines) are
unspecified and left to commercial developers to provide.106
Several groups are presently
developing video content analysis algorithms to automatically extract semantic information from
video data with the hopes of partially automating the metadata creation process.107
104
Ibid. 63.
105
Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke
Wöhrmann B.V., 1998), 58.
106
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 270.
107
Effelsberg. Op. Cit.
33
FEATURE EXTRACTION STANDARD DESCRIPTION SEARCH ENGINE
MPEG-7 SCOPE
Figure 5. SCOPE OF THE MPEG-7 STANDARD.
Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke Wöhrmann
B.V., 1998), 58.
Assuming metadata is recorded simultaneously with the audio-visual data and timing
information, search engines can be developed that can access specific audio or video data from
large digital archives or databases.108
This timely access to required material can provide
immediate use – an important consideration especially for mission critical applications.
The generation of metadata poses two main issues. The first is the amount of data that is
generated. Human interaction in the indexing process will result in only a fraction of the total
created content being indexed. In this case, the automated analysis of digital records is critical
and presents many technical challenges. The second issue regards the standardization of
metadata descriptions. The situation has improved with the widespread availability of
information.109
Conversely, practical application of metadata has been made problematic by the creation
of different standardization methods. The MPEG-7 standard calls for extensible markup
language (XML) schemas as opposed to the Society of Motion Picture and Television Engineers
(SMPTE) which uses key-length-value (KLV) coding. Other organizations have created their
108
Ibid.
109
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 270.
34
own metadata sets that are based on their own needs. This proliferation of different metadata
methodologies creates interoperability issues, which was never the intent of MPEG-7. Efforts
are being led by MPEG and SMPTE to synchronize the different methodologies to allow
portability between them.110
MULTIMEDIA PLAYERS AND CONTAINER FORMATS
This discussion is not complete without providing a clear distinction between multimedia
players and multimedia container formats. Multimedia players are the software applications that
are used to playback video and audio data files. There are many of these applications available
to include Windows Media Player®
, Real Player®
, and QuickTime®
, to name but a few.
Multimedia container formats provide for the actual formatting of the recorded video and audio.
Different methods exist and are signified by the actual data file extensions. These include, but
are not limited to the following format types: AVI, ASF, and MOV.
Many of the currently fielded video collection systems use digital video recorder utilities
that apply the audio video interleave (AVI) container format for playback operations. Playback
of applicable video is then performed using one of the many available multimedia players like
Windows Media Player®
. The AVI format is proven over some years of usage and has continued
to be supported by many multimedia players. For purposes of this paper, only the AVI container
format will be discussed.
The AVI format has its origins in the Resource Interchange File Format (RIFF), which
divides the data into “blocks” that are each identified by a tag. The AVI file is a single block,
RIFF formatted file that is further divided into two mandatory sub-blocks and one optional sub-
block. The first sub-block is the file header, which contains video metadata such as frame width,
110
Ibid. 275-276.
35
height and rate. The second sub-block contains the actual audio-visual data that is the AVI
movie file. The third, optional, sub-block is used to index data block offsets within the file.111
Via the RIFF format, the sub-block containing the actual audio-visual data is encoded
and/or decoded by a software algorithm called a “codec,” an abbreviation for (en) coder/decoder.
Upon file creation, the codec interprets between raw data and the compressed data format used
within the sub-block. AVI files can hold audio-visual information inside the data blocks in
almost any compression scheme to include full frame (uncompressed), Motion JPEG, editable
MPEG, and MPEG-4 to name a few.112
AVI was not originally intended to contain compressed video, which requires access to
video frame data beyond the currently used frame. Methods do exist that support modern video
compression use within the AVI framework (MPEG-4, et. al.), even though this is outside the
scope of the original specification. Problems can occur during playback with utilities that do not
anticipate any issues like not having the correct codec required to play the video file.113
However, most audio-video files can be reviewed for playback provided the necessary
codec files are installed on the computing machinery that is being utilized for this purpose. As
such, the AVI format continues to see major use in both commercial as well as military
applications.
The previous sections have described some of the concepts related to data storage. In
particular, storage media, data redundancy, and video compression technologies were discussed
111
Wikipedia contributors, "Audio Video Interleave," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Audio_Video_Interleave&oldid=347331467 (accessed March 10, 2010).
112
Ibid.
113
Ibid.
36
in relation to the large scale storage of recorded audio-video data files. The discussion will now
turn to possible solutions to the storage of this data in relation to tactical video acquisition.
37
CHAPTER 3
RECOMMENDATIONS
HARDWARE CONSIDERATIONS
Although somewhat outside the original scope of this paper, it is important that the
system developer/user consider the computing machinery to be used in the tactical video
collection system. The personal computer (PC) has advanced to the point that powerful, robust
systems are available commercially at a fraction of the cost of high performance systems from
years past. Most PCs available for commercial-off-the-shelf (COTS) purchase are quite suitable
for digital video recording functions. Notwithstanding the obvious need for robust computing
machinery, given the critical nature of tactically fielded video collection systems, some minimal
hardware recommendations are as follows:
1. Robust PC-based system using a dual-processor, 64-bit bus motherboard with large
random access memory (RAM) capacity.
2. Best commercially available video graphics adapter using the most video RAM.
3. Main internal storage consisting of solid state hard drives (SSDs).
4. Redundant backup using an external (or internal) RAID-based system.
5. CD-RW/DVD-RW/Blu-Ray-RW for archiving purposes.
Following these deliberations, the primary, internal system storage should consist of solid
state drives (SSDs) of sufficient size to accommodate the operating system (OS), applicable
system and recorder applications, and onboard storage. The use of SSDs will allow for enhanced
system performance during mission critical operations. External (or internal) RAID storage is
recommended using standard hard disk drives, which provide more efficient mass storage
properties and should be sufficient for normal backup operations.
Regarding archival storage, the use of Blu-Ray optical disks is recommended given both
the large capacity and the storage longevity of optical disks. Compared to other optical storage
mediums as well as magnetic tape, Blu-Ray best accommodates the requirements necessary for
38
the storage of large amounts of captured video data, and it will also reduce hard storage
requirements. A simple hierarchy depicting these guidelines is shown in Figure 6 followed by
brief explanations for each level.
LEVEL 3
ANCILLARY STORAGE
(STORED ON-SITE)
LEVEL 2
SECONDARY STORAGE
(EXTERNAL/INTERNAL RAID)
LEVEL 1
PRIMARY STORAGE
(INTERNAL)
Figure 6. SIMPLE STORAGE HIERARCHY.
LEVEL 1:
Primary storage, internal to the computing machinery and using flash Solid State Drives (SSDs)
for their enhanced performance and data retrieval.
LEVEL 2:
Secondary storage for database back up (automated or manual) using RAID-based architecture
and consisting of mechanical Hard Disk Drives (HDDs) for their mass storage benefits.
LEVEL 3:
Ancillary storage using Blu-Ray optical disks for their large storage capacity, environmental
durability, and long-term storage properties.
39
SOFTWARE CONSIDERATIONS
Once the computing machinery is in place, the system developer/user must next consider
the software operating system (OS) and the required application software to be used with the
system. The Windows®
OS is recommended for its ready availability and widespread use. The
digital video recorder (DVR) application may be selected from open or proprietary sources. It
should be robust, and its selection based on system considerations. In addition to its recording
function it must also provide post-mission playback and video editing utilities.
For backup operations, an automated backup and restore system utility is recommended.
It should have a built in scheduling capability and operate in the background, transparent to the
host system. It should provide the capability for different types of backup operations, the most
common of which are explained in Table 5. For mission critical operations, it is recommended
that Full backups be performed to provide complete restoration of all data in case of corruption
or catastrophic system failure. The utility should also provide the ability to automatically
overwrite older video data files when necessary, in the interest of freeing up hard drive space.
This means that any video that is intended to be kept for planning or historical purposes must be
manually copied and archived on separate media, preferably Blu-Ray optical disk.
Table 5. BACKUP OPERATIONS.
BACKUP
TYPE
OPERATION CHARACTERISTICS
Full Provides full backup of all data. Time intensive, but provides all data for complete
restoration.
Incremental Only copies files that have changed since last
full backup.
Assuming weekly backups, subsequent backups
may be larger.
Differential Only copies data that has changed since last
differential backup.
Shortens backup time, but may increase
restoration time.
Snapshot Point-in-time copy of the data, also called
pointer-based backup. Copies pointers and file
metadata.
Very fast backup - copies only metadata and
pointers that point to the data blocks. Does not
protect against disk loss.
Schulz, Greg, Resilient Storage Networks: Designing Flexible Scalable Data Infrastructures (Massachusetts:
Elsevier Digital Press, 2004), 324.
40
A suggested backup scheme is shown in Figure 7. It is recommended that a Master
System disk be produced on Blu-Ray optical media following final system setup. This should be
maintained on site and will provide a backup in case the magnetic hard disk drive needs to be
rebuilt due to file corruption or catastrophic system events.
Additionally, a Full system backup that is stored in the external RAID storage assembly
should be scheduled as required to intermittently capture the present system configuration. This
will allow the user to recover system operation quickly by using a previously known working
system configuration. This is also useful in the event of system file corruption or other events
that may affect the system operation.
Recorded video should also be captured in the external RAID storage area to keep the
internal, primary storage medium free of unnecessary files that may adversely affect the overall
system performance. The RAID storage capacity should be in the terabyte range to
accommodate the large amount of data that will be generated by the video collection system. At
a minimum, RAID-1 architecture providing mirroring and full redundancy is recommended.
41
Figure 7. BACKUP SCHEME.
Given the nature of real-time, tactical video collection systems, large amounts of data
are produced within a short period of time. In the interest of freeing up storage space for the
recording of newer video, it is recommended that any video that is desired to be retained for
historical reference should be archived on optical media and stored on-site to facilitate its later
usage. A master system copy stored on optical disk should also be maintained for recovery
operations due to system corruption and/or catastrophic system failure. This should also be
maintained on-site and be updated as required.
VIDEO RETENTION AND DELETION SCHEMES
Tactical operations are defined as “military operations conducted on the battlefield,
generally in direct contact with the enemy.”114
Tactical data collection is indigenous to the area
of operations in which the tactical collection asset is fielded. Given the nature of tactical
114
Tactical Operations. Answers.com. The Oxford Essential Dictionary of the U.S. Military,
Oxford University Press, 2001, 2002. (accessed March 07, 2010).
42
intelligence data such as real-time video, the usefulness of the data may diminish quickly within
a very short period of time.
However, this data is still retained in storage although it may not provide any useful
intelligence value. This is generally true for most video that is collected over time and the
burden on storage assets can be significant. This section outlines a selection process that may be
used to determine automatic retention and/or deletion of tactically collected video.
Figure 8 is an example of a simple, software driven selection process, which could use
criteria such as the number of times a file has been accessed within a given period of time to
determine retention or deletion. It might also be based on guidelines established by the
command to determine the long-term or short-term tactical intelligence value of specific video.
RAW VIDEO
VIDEO
PROCESSING
VIDEO
DATABASE
VIDEO FILE
VIDEO
SCREENING
PROCESS
RETENSION
DELETION
NO
TACTICAL
VALUE
TACTICAL
VALUE
Figure 8. A SIMPLE VIDEO RETENTION/DELETION SCHEME.
A more in depth selection process is shown in Figure 9. The displayed retention times
shown are arbitrary examples. The automatic deletion of stored video imagery would rely on
specific, established criteria to determine whether a file is a candidate for deletion. At a
minimum, criteria might include the date/time stamp (from metadata) and the number of times
the specific files have been accessed, which denotes level of importance. However, user
43
intervention might be required to determine the retention of specific video that contains
significant events such as:
1. Reconnaissance operations.
2. Surveillance operations.
3. Intelligence gathering operations.
4. Mission operations support.
5. IED incidents.
6. Counter-IED operations.
7. Area/terrain observations in support of missions.
8. Areas of interest (AI).
9. Any video showing direct/indirect contact with an enemy.
10. Other significant events that are deemed to provide intelligence value of a tactical
nature, as determined by the command.
VIDEO
DATABASE
PROCESSED VIDEO VIDEO FILES
RETENTION
SCREENING
ALGORITHM
CRITICAL
VALUE
HIGH VALUE
MEDIUM
VALUE
LOW VALUE
NO VALUE
TAGGED VIDEO
RETAIN INDEFINITELY
RETAIN 6-12 MONTHS
RETAIN 3-6 MONTHS
RETAIN 1-3 MONTHS
RETAIN 30 DAYS
TAGGED
VIDEO
3-6MOS
6-12MOS
1-3MOS
AUTOMATED
DELETION
ALGORITHM
30 DAYS
DELETED VIDEO
INDEFINITE
RETENTION
Figure 9. A MORE ROBUST VIDEO RETENSION/DELETION SCHEME.
Additionally, unless it is used for planning purposes or as an historical reference in
support of future operations, most tactically captured video may have a short shelf life for
44
providing up to date intelligence information. Of course, this is dependent upon the intelligence
requirements and the value placed on specific video by the command.
FUTURE CONSIDERATIONS
Current digital video recorder software applications use a date/time stamp to sort
recorded video. This does allow for automated sorting as the video data files are sent to a
predetermined storage location and are sorted according to the date and time they were recorded.
However, this convention does not allow for the sorting of video based on its content. The
search and retrieval of multiple related records is also not supported. In normal operations, the
user must make note of the specific time and date of an event and access the file manually,
usually having to manually parse through different file locations and numerous video files to
locate the one in question. The use of metadata can alleviate this issue.
The U.S. government’s Federal Geographic Data Committee (FGDC) describes metadata
as the data about the content, quality, condition, and other characteristics of the data.115
At this
time, the FGDC is the organization that has been tasked with developing the guidelines and
standards for digital metadata that is to be used in geospatial imagery. Although metadata is
being developed primarily for use in the support of geographic information system (GIS)
mapping data, it would serve equally useful from the perspective of tactical video collection.
Some minimal considerations for items that might be included as part of a data element set that
should be included with the metadata might be as follows:
1. Time/date stamp.
2. Geographic locations of known points.
3. Location Coordinate System (Military Grid Reference System (MGRS)).
4. Distance/altitude units (feet/meters).
115
Federal Geographic Data Committee, Content Standard for Digital Geospatial Metadata,
http://www.fgdc.gov/standards/projects/FGDC-standards-projects/metadata/base-metadata/v2_0698.pdf, 64
(accessed January 20, 2010).
45
5. Altitude/elevation data in reference to location.
6. Populated areas: cities, towns, and villages.
7. Rural areas: farms, ranches, etc.
8. Infrastructure: schools, public buildings, places of worship, power stations, military
installations, police stations, telecommunications (antenna towers), water treatment
facilities, transportation (roads/highways both improved and unimproved,
intersections, traffic circles), airfields/airports/airstrips, dams, bridges, waterways
(lakes, oceans, seas, rivers, streams, man-made), etc.
Many of these items already have a place in the metadata guidelines developed by the
FGDC. Other important metadata considerations have also been outlined by the FGDC, which
include mission specific data items. These may easily be adapted in the metadata that is used in
support of tactical video collection efforts. Some of this mission specific data is as follows116
:
1. Mission data/time.
2. Mission name.
3. Mission significant events.
4. Mission Platform and Instrumentation.
At the present, the use of metadata in many video applications has been limited. Its usage
may also necessitate user intervention for the manual input of some data items. However, the
advent of the MPEG-7 and MPEG-21 video compression standards may provide a path forward
for the intelligent organization of video data files. The adoption of these compression methods
in future video collection systems will allow video files to be “tagged” and automatically sorted,
catalogued, and organized according to specific attributes that are captured as part of the
metadata architecture. This will aid in the timely search and retrieval of related video data
applicable to specific software queries.
116
Federal Geographic Data Committee, Content Standard for Digital Geospatial Metadata: Extensions for
Remote Sensing Metadata, http://www.fgdc.gov/standards/projects/FGDC-standards-
projects/csdgm_rs_ex/MetadataRemoteSensingExtens.pdf, 67-71 (accessed January 20, 2010).
46
CHAPTER 4
CONCLUSION
The purpose of this paper was to evaluate the issue of database storage for the tactical
video collection process. This is a major concern for end users of these systems given the large
amount of video that is collected on a daily basis. The previous discussion has attempted to
explain the necessary hardware that is available and the software related techniques that are used
in this process. Possible recommendations were also provided that may assist in easing the
concerns related to this issue.
First, large capacity storage media were discussed to include magnetic tape, optical disks,
magnetic disk drives, and finally solid state drives. These media types are of particular
importance since they provide the best large-scale storage capacity. Each was evaluated to
determine which of them were best suited to be used in a tactical environment. Redundant
storage in the form of the RAID architecture was also examined.
Magnetic tape media has been a mainstay for the backup of digital video and imagery as
it provides a low cost method of backing up data for archival purposes. However, the sequential
nature of magnetic tape media makes it very inefficient for random data access, and
recovery/backup operations. This is particularly important when fielding a tactical collection
system since the command cannot afford to be without the asset for long periods of time
following, for example, a catastrophic failure. The system must be recoverable very quickly.
Magnetic tape media also deteriorate over time and must be stored in moderate environments
with low humidity. The conditions in many of the current locations in which these assets are
fielded make the use of magnetic tape media problematic.
47
Optical disks have many advantages over magnetic tape to include lower cost, greater
storage capacity, data stability, environmental durability, and a long shelf life potential of over
100 years. Their constant linear velocity provides unvarying read/write bandwidth during
sequential access operations, which makes them ideal for multimedia backup and restore
operations. These reasons have made optical disk technology the primary backup media in many
remote sensing operations.
Magnetic hard disks, also known as hard disk drives, are the primary storage technology
used in most personal computer systems and many video collection systems. They provide fast,
concurrent access to data and have very large capacities. Because of this, they are still useful as
a storage medium in redundant systems such as RAID assemblies. However, as they are electro-
mechanical devices with moving parts, extreme environments together with continual use can
dramatically increase their mortality rate.
Solid state drives, on the other hand consist of high reliability flash memory technology
and have no moving parts, which makes them more durable in severe environments. They have
much faster response times, lower access times, and much higher throughput bandwidth than
magnetic hard disks. Lifespan characteristics are comparable to magnetic hard disks with on-
the-shelf data retention of up to ten years.
However, solid state drives are more expensive than magnetic hard disks and capacities
are usually much smaller. In spite of these shortcomings, solid state drives are fast becoming the
primary storage medium for applications that require quicker response time and proven
durability in extreme environments.
The discussion on RAID described its importance as a backup storage medium and some
of the methods used in the different RAID architectures. Two of the most prevalent methods
48
were examined and included RAID Level 1 (RAID-1) and RAID Level 5 (RAID-5). Both
methods provide a means of data redundancy and recovery in case of system failures and/or hard
disk drive loss.
RAID-1 architecture performs only disk mirroring, which provides full redundancy. It
also permits quicker recovery in the case of disk failure in the array, and offers increased read
performance as data is read across all disks in the array. However, it lacks in overall
performance as it is necessary to have twice the storage capacity to accommodate data storage,
which can add significantly to costs.
In comparison, RAID-5 architecture uses parity checking methods and provides more
efficient data redundancy than simple mirroring. It is also less expensive in regards to data
protection costs. However, every change in data facilitates a recalculation of parity, which
adversely affects system performance.
Following the discussion on storage media, the subject of video compression was
investigated. Video compression deals with the reduction of the amount of data required to
convey a useable facsimile of a specific data object. This is particularly important since most
present day imagery can produce extremely large files, which facilitates the need for larger
storage capacities. Both “lossless” and “lossy” compression techniques were examined and are
used to compress data according to the different standards that have been developed to include
those of the Motion Pictures Experts Group (MPEG), which are the most common.
Lossless compression techniques allow for complete reconstruction of the original data
signal and provide a slight amount of compression. Lossy compression techniques provide only
a near approximation of the original data signal since some tradeoff is required to provide
satisfactory compression. Imagery can be produced that is fundamentally and visually similar to
49
the original using ratios much greater than those of lossless compression. Lossy compression is
most commonly used for compressing multimedia data such as audio, video, and still images.
These compression algorithms are used in conjunction with different standards, with the
most prevalent being developed by the MPEG. The standards outline the basic requirements for
the compression of audio, video, and imagery data. Short discussions of the different MPEG
standards were presented with particular attention given to the MPEG-7 and MPEG-21
standards, which are still in their infancy.
These standards do provide for the necessary compression of multimedia data, which is
the primary consideration. However, it is their use of metadata techniques that is of particular
importance. Since metadata considers the peripheral, supporting information about the actual
data, its use for the indexing and cataloging of collected data is significant to the video collection
process.
Multimedia players and multimedia container formats were briefly discussed to provide a
distinction between the two. Multimedia players, like Windows Media Player®
, are the
applications that are used to replay the recorded audio-video data files. Container formats like
audio video interleave (AVI) provide the actual formatting process for the video. The AVI
container format was explained solely because of its widespread usage and to provide an
elementary understanding of this technology. No recommendation has been made for these
items since there are many different types which work equally well.
Following discussion of the different storage media and software techniques used in the
storage process, recommendations were made that might assist in alleviating storage capacity
issues in tactical video collection systems. These recommendations were divided into three
50
groups: hardware, software, and future considerations. Recommendations were provided for
each of these categories.
In developing the collection system, hardware considerations were taken into account to
include the motherboard architecture, system memory, processing power, the graphics adapter
and internal storage. As such, these suggestions were offered mainly as guidelines with the
system storage being the primary focus. In this case, it was recommended that solid state drives
be used as the primary data storage media in the computing machinery for tactical video
collection systems due to their higher performance capability and durability. Magnetic hard
disks should still be used in secondary data storage and backup operations as part of a RAID
assembly since they provide higher capacity storage than solid state drives.
Redundant storage was also discussed in the form of a RAID assembly, which provides
for system backup and large external storage capacity. Magnetic drives should be used for this
assembly as they are more efficient than solid state drives for mass storage and should suffice for
normal backup operations. The tactical video collection system is a mission-critical platform
that cannot tolerate any loss in data, if at all possible. In light of this, it was recommended that
RAID-1, providing complete data redundancy, be used to accommodate quicker system recovery
in case of system corruption or total system failure.
The archiving of data was also examined using optical disks as the storage medium.
Their large capacity, data retention characteristics, and durability made them the best choice for
this issue. Blu-Ray optical disks were recommended because they presently provide the most
storage capacity of all optical disk types.
Software considerations included the operating system, the digital video recorder (DVR)
application, and a backup and restore utility. In these cases, the Windows®
OS was
51
recommended due to its widespread usage. No DVR application was suggested as this should be
selected based on system requirements. At a minimum, it should provide the necessary
recording function as well as post-mission playback and video editing utilities. With respect to
the system backup operations, a reliable backup and restore application was also examined. Its
operation should be transparent to the host system and provide automated, scheduled backups as
deemed necessary. Full backups were recommended to provide complete repair of the system in
case of file corruption or system failure. These scheduled backups should be placed in the
secondary storage area (RAID assembly) to provide expedited system repair or rebuild as
required.
Following system setup, it was recommended that a master system disk be created on
Blu-Ray optical disk to facilitate system repair or rebuild in case of catastrophic system failure or
internal disk issues. Tactical video intended to be kept for historical purposes should also be
copied to optical media to facilitate its later usage. These items should be kept in the on-site
archive. It was also suggested that all recorded video be captured and stored on the external
RAID assembly in the interest of keeping the primary storage area free of any extraneous files
that might affect system operation.
Proposals were made for software metrics that could provide for the automated retention
or deletion of stored video in the interest of freeing up storage space. These metrics could use
items such as the time/date stamp and the number of times a file has been accessed to determine
retention or deletion of files from the video database. Other versions of this method might base
retention and deletion on the actual nature of the video and its importance to the command.
Different grades or levels of importance might be assigned to provide a clear distinction between
video that is considered of tactical value and worth retaining and video that is deemed as
52
unimportant and requiring deletion. Some user intervention may still be required for these
methods, but they could provide a basis for determining the importance of the collected video
and its subsequent tactical value.
Finally, future considerations were alluded to, in particular the use of metadata as an
automated cataloging and indexing tool. The metadata technology is used to describe the
background information attributed to the collected data. The use of this technology is
accommodated in the MPEG-7 and MPEG-21 compression standards, which are currently in
work by the Motion Pictures Experts Group (MPEG) organization.
The Federal Geographic Data Committee (FGDC) provides the oversight regarding the
use of metadata and is the current working body responsible for the standardization of the
metadata technology. The guidelines developed by the FGDC are currently used in conjunction
with the development of GIS applications.
However, these same guidelines might be easily adapted for use on tactical video
collection systems and may provide the means for the expeditious retrieval of video data sets that
correspond to specific events, times, and locations. This makes it a potentially powerful
intelligence tool and may provide a foundation for content-based video data retrieval in the
future.
53
BIBLIOGRAPHY
Bergman, Lawrence D., and Castelli, Vittorio. Image Databases: Search and Retrieval of
Digital Imagery. New York: John Wiley & Sons, Inc., 2002.
Christopher Drew. “NYT: Military deluged in intelligence from drones - Remote-controlled
planes produced about 24 years’ worth of video in 2009.” New York Times, January 10,
2010, Sunday; http://www.msnbc.msn.com/id/34798080/ns/world_news-
the_new_york_times/ (accessed January 11, 2010).
Cornell University Library. “Moving Theory Into Practice: Digital Imaging Tutorial, Chapter 5,
Metadata: Definition, Types, and Functions.”
http://www.library.cornell.edu/preservation/tutorial/metadata/metadata-01.html (accessed
January 14, 2010).
Coughlin, Tom, Ekker, Neal, and Handy, Jim. “Solid State Storage 101: An introduction to
Solid State Storage.” Storage Networking Industry Association (SNIA) Solid State
Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf
(accessed March 13, 2010).
Crothers, Brook. “Seagate Enters Solid-State Drive Market.” CNet News, December 7, 2009,
http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010).
Daintith, John. "Magneto-Optic Storage." A Dictionary of Computing. 2004. Encyclopedia.com.
http://www.encyclopedia.com/doc/1O11-magnetoopticstorage.html (accessed March 5,
2010).
Effelsberg, Wolfgang and Steinmetz, Ralf. Video Compression Techniques. Netherlands:
KoninKlijke Wöhrmann B.V., 1998.
Encyclopedia.com. "Compact Disk." The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-compactd.html (accessed March 4, 2010).
Encyclopedia.com. "Digital Versatile Disc." The Columbia Encyclopedia, Sixth Edition,
2008, http://www.encyclopedia.com/doc/1E1-DVD.html (accessed March 4, 2010).
Encyclopedia.com. "Optical Disk." The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010).
Farley, Marc. Storage Networking Fundamentals: An Introduction to Storage Devices,
Subsystems, Applications, Management, and File Systems. Indiana: Cisco Press, 2007.
Jensen, John R. Introductory Digital Image Processing: A remote Sensing Perspective. New
Jersey: Prentice-Hall, 2005.
54
Judd, Josh. Principles of SAN Design: Design Build and Manage SANS. Pennsylvania: Infinity
Publishing, 2007.
Judd, Josh and O’Conner, Michael. Introducing File Area Networks Advance Edition: Your
First Look at FAN Technology. Pennsylvania: Infinity Publishing, 2007.
Luff, John. “Technology in Transition: Video storage.” Broadcast Engineering,
http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed
January 18, 2010).
Luff, John. “Video Storage.” Broadcast Engineering, April 1, 2003,
http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed
January 17, 2010).
Luff, John. “Video Storage.” Broadcast Engineering, July 1, 2004,
http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed
January 17, 2010).
Mediatechnics Systems, Inc. "How a CD is Made." Mediatechnics Systems, Inc., FAQ,
https://www.mediatechnics.com/cdfaqs.htm (accessed March 5, 2010).
Mediatechnics Systems, Inc. "How are DVDs Made for use in Copiers?" Mediatechnics
Systems, Inc., FAQ, https://www.mediatechnics.com/dvdcopiers.htm (accessed March 5,
2010).
National Information Standards Organization. “Understanding Metadata.” (Maryland: NISO
Press, 2004), http://www.niso.org/publications/press/UnderstandingMetadata.pdf
(accessed January 20, 2010).
Nelson, Dane and Siewart, PhD, Sam. “Solid State Drive Applications in Storage and Embedded
Systems.” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf
(accessed March 2, 2010).
Park, Oskar. “What is an Optical Disk?." Self SEO (Search Engine Optimization), September
10, 2006, http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
Schulz, Greg. Resilient Storage Networks: Designing Flexible Scalable Data Infrastructures.
Massachusetts: Elsevier Digital Press, 2004.
Shrivastava, Alok, and Somasundaram, G., eds. Information Storage and Management: Storing,
Managing, and Protecting Digital Information. Indiana: Wiley Publishing Incorporated,
2009.
55
Simitci, Huseyin. Storage Network Performance Analysis. Indiana: Wiley Publishing, Inc.,
2003.
Symes, Peter. Digital Video Compression. New York: McGraw-Hill, 2004.
Thatcher, Jonathan. “NAND Flash Solid State Storage Reliability and Data Integrity: An In-
Depth Look.” Storage Networking Industry Association (SNIA) Solid State Storage
Initiative, 2009,
http://www.snia.org/forums/sssi/knowledge/education/Solid_State_Storage_Reliability_a
nd_Data_Integrity--An_In-depth_Look.pdf (accessed March 13, 2010).
The Free Dictionary. “Flash Memory.”
http://encyclopedia2.thefreedictionary.com/flash+memory (accessed March 2, 2010).
The Free On-line Dictionary of Computing. "Average Seek Time."
http://encyclopedia2.thefreedictionary.com/average+seek+time (accessed March 12
2010).
The Oxford Essential Dictionary of the U.S. Military. “Tactical Operations.” Answers.com.
The Oxford Essential Dictionary of the U.S. Military, Oxford University Press, 2001,
2002, http://www.answers.com/topic/tactical-operations (accessed March 07, 2010).
Tudor, Marius. “Are Flash Solid-State Disk Drives Ready for the Enterprise?” Bit Micro
Networks, Incorporated, 2009,
http://www.bitmicro.com/press_resources_flash_ssd_enterprise.php (accessed March 2,
2010).
Wikipedia contributors. "Audio Video Interleave." Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Audio_Video_Interleave&oldid=347331467
(accessed March 10, 2010).
Wikipedia contributors. "Blu-ray Disc.” Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc&oldid=349002833 (accessed
March 12, 2010).
Wikipedia contributors. "Blu-ray Disc recordable." Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc_recordable&oldid=346269932
(accessed March 12, 2010).
Wikipedia contributors. "Lossless data compression.” Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossless_data_compression&oldid=346709055
(accessed March 10, 2010).
Wikipedia contributors. "Lossy compression.” Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossy_compression&oldid=347940008
(accessed March 10, 2010).
56
Wikipedia contributors. "Magneto-optical drive." Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Magneto-optical_drive&oldid=338693032
(accessed March 12, 2010).

Más contenido relacionado

Destacado

Taller 4 grado 10º
Taller 4 grado 10ºTaller 4 grado 10º
Taller 4 grado 10ºMetachos
 
Publicacion en la_web
Publicacion en la_webPublicacion en la_web
Publicacion en la_webmarytxa
 
[certificate of appreciation
[certificate of appreciation[certificate of appreciation
[certificate of appreciationmuli kivanga
 
Produsen peralatan komputer
Produsen peralatan komputerProdusen peralatan komputer
Produsen peralatan komputeruteyer15
 
Csapó Endre - zsidó világkongresszus volt Budapesten
Csapó Endre -  zsidó világkongresszus volt Budapesten Csapó Endre -  zsidó világkongresszus volt Budapesten
Csapó Endre - zsidó világkongresszus volt Budapesten Arany Tibor
 
Acf422 fasa6 sudut pandangan & nilai slaid js
Acf422 fasa6 sudut pandangan & nilai slaid jsAcf422 fasa6 sudut pandangan & nilai slaid js
Acf422 fasa6 sudut pandangan & nilai slaid jsdramajalansempit
 
Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...
Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...
Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...Editor IJCATR
 
Acf422 fasa2 tema pemikiran latar rp js
Acf422 fasa2 tema pemikiran latar rp jsAcf422 fasa2 tema pemikiran latar rp js
Acf422 fasa2 tema pemikiran latar rp jsdramajalansempit
 
Future Funding for Polk's Transportation Initiative
Future Funding for Polk's Transportation InitiativeFuture Funding for Polk's Transportation Initiative
Future Funding for Polk's Transportation InitiativePolk Transit
 
Ata convenão pmdb( imprensa)
Ata convenão pmdb( imprensa)Ata convenão pmdb( imprensa)
Ata convenão pmdb( imprensa)Dinomar Miranda
 
Enhancing Data Staging as a Mechanism for Fast Data Access
Enhancing Data Staging as a Mechanism for Fast Data AccessEnhancing Data Staging as a Mechanism for Fast Data Access
Enhancing Data Staging as a Mechanism for Fast Data AccessEditor IJCATR
 
Fatty Acid Pattern and Alkaloids of Echium Rauwolfii
Fatty Acid Pattern and Alkaloids of Echium RauwolfiiFatty Acid Pattern and Alkaloids of Echium Rauwolfii
Fatty Acid Pattern and Alkaloids of Echium RauwolfiiEditor IJCATR
 
English II Module Outline FNBE 0115
English II Module Outline FNBE 0115 English II Module Outline FNBE 0115
English II Module Outline FNBE 0115 TamZhaoWei
 
Techniques for Smart Traffic Control: An In-depth Review
Techniques for Smart Traffic Control: An In-depth ReviewTechniques for Smart Traffic Control: An In-depth Review
Techniques for Smart Traffic Control: An In-depth ReviewEditor IJCATR
 

Destacado (15)

Taller 4 grado 10º
Taller 4 grado 10ºTaller 4 grado 10º
Taller 4 grado 10º
 
Publicacion en la_web
Publicacion en la_webPublicacion en la_web
Publicacion en la_web
 
[certificate of appreciation
[certificate of appreciation[certificate of appreciation
[certificate of appreciation
 
Produsen peralatan komputer
Produsen peralatan komputerProdusen peralatan komputer
Produsen peralatan komputer
 
Csapó Endre - zsidó világkongresszus volt Budapesten
Csapó Endre -  zsidó világkongresszus volt Budapesten Csapó Endre -  zsidó világkongresszus volt Budapesten
Csapó Endre - zsidó világkongresszus volt Budapesten
 
Acf422 fasa6 sudut pandangan & nilai slaid js
Acf422 fasa6 sudut pandangan & nilai slaid jsAcf422 fasa6 sudut pandangan & nilai slaid js
Acf422 fasa6 sudut pandangan & nilai slaid js
 
Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...
Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...
Enhanced Quality of Service Based Routing Protocol Using Hybrid Ant Colony Op...
 
Paris memoria
Paris memoriaParis memoria
Paris memoria
 
Acf422 fasa2 tema pemikiran latar rp js
Acf422 fasa2 tema pemikiran latar rp jsAcf422 fasa2 tema pemikiran latar rp js
Acf422 fasa2 tema pemikiran latar rp js
 
Future Funding for Polk's Transportation Initiative
Future Funding for Polk's Transportation InitiativeFuture Funding for Polk's Transportation Initiative
Future Funding for Polk's Transportation Initiative
 
Ata convenão pmdb( imprensa)
Ata convenão pmdb( imprensa)Ata convenão pmdb( imprensa)
Ata convenão pmdb( imprensa)
 
Enhancing Data Staging as a Mechanism for Fast Data Access
Enhancing Data Staging as a Mechanism for Fast Data AccessEnhancing Data Staging as a Mechanism for Fast Data Access
Enhancing Data Staging as a Mechanism for Fast Data Access
 
Fatty Acid Pattern and Alkaloids of Echium Rauwolfii
Fatty Acid Pattern and Alkaloids of Echium RauwolfiiFatty Acid Pattern and Alkaloids of Echium Rauwolfii
Fatty Acid Pattern and Alkaloids of Echium Rauwolfii
 
English II Module Outline FNBE 0115
English II Module Outline FNBE 0115 English II Module Outline FNBE 0115
English II Module Outline FNBE 0115
 
Techniques for Smart Traffic Control: An In-depth Review
Techniques for Smart Traffic Control: An In-depth ReviewTechniques for Smart Traffic Control: An In-depth Review
Techniques for Smart Traffic Control: An In-depth Review
 

Similar a Data Storage Considerations for the Tactical Field Collection of Digital Imagery - 2010

An Analyzing of different Techniques and Tools to Recover Data from Volatile ...
An Analyzing of different Techniques and Tools to Recover Data from Volatile ...An Analyzing of different Techniques and Tools to Recover Data from Volatile ...
An Analyzing of different Techniques and Tools to Recover Data from Volatile ...ijsrd.com
 
Challenges and Best Practices for Storing/ Challenges and Best Practices for ...
Challenges and Best Practices for Storing/ Challenges and Best Practices for ...Challenges and Best Practices for Storing/ Challenges and Best Practices for ...
Challenges and Best Practices for Storing/ Challenges and Best Practices for ...NetApp
 
WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN TODAY’S INFORMATIO...
WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN  TODAY’S INFORMATIO...WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN  TODAY’S INFORMATIO...
WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN TODAY’S INFORMATIO...`Shweta Bhavsar
 
Distributed parallel architecture for big data
Distributed parallel architecture for big dataDistributed parallel architecture for big data
Distributed parallel architecture for big datakamicool13
 
3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptx
3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptx3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptx
3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptxMallaAbhinaya
 
COMPACT PRESERVATION OF SCRAMBLED CD-ROM DATA
COMPACT PRESERVATION OF SCRAMBLED CD-ROM DATACOMPACT PRESERVATION OF SCRAMBLED CD-ROM DATA
COMPACT PRESERVATION OF SCRAMBLED CD-ROM DATAijcsit
 
Efficient usage of memory management in big data using “anti caching”
Efficient usage of memory management in big data using “anti caching”Efficient usage of memory management in big data using “anti caching”
Efficient usage of memory management in big data using “anti caching”eSAT Journals
 
iRODS UGM 2016 Preso Summary FINAL
iRODS UGM 2016 Preso Summary FINALiRODS UGM 2016 Preso Summary FINAL
iRODS UGM 2016 Preso Summary FINALRandy Splinter
 
secondarystoragedevices_by_sarthak_goel.pptx
secondarystoragedevices_by_sarthak_goel.pptxsecondarystoragedevices_by_sarthak_goel.pptx
secondarystoragedevices_by_sarthak_goel.pptxSarthakGoel39
 
3D Optical Data Storage
3D Optical Data Storage 3D Optical Data Storage
3D Optical Data Storage Shobha Rani
 
Holographic optical data storage jyoti-225
Holographic optical data storage jyoti-225Holographic optical data storage jyoti-225
Holographic optical data storage jyoti-225Charu Tyagi
 
Android based message encryption decryption using matrix
Android based message encryption  decryption using matrixAndroid based message encryption  decryption using matrix
Android based message encryption decryption using matrixeSAT Journals
 
Hierarchical Storage Management
Hierarchical Storage ManagementHierarchical Storage Management
Hierarchical Storage ManagementJaydeep Patel
 

Similar a Data Storage Considerations for the Tactical Field Collection of Digital Imagery - 2010 (20)

An Analyzing of different Techniques and Tools to Recover Data from Volatile ...
An Analyzing of different Techniques and Tools to Recover Data from Volatile ...An Analyzing of different Techniques and Tools to Recover Data from Volatile ...
An Analyzing of different Techniques and Tools to Recover Data from Volatile ...
 
50120130406026
5012013040602650120130406026
50120130406026
 
Data storage
Data storageData storage
Data storage
 
Challenges and Best Practices for Storing/ Challenges and Best Practices for ...
Challenges and Best Practices for Storing/ Challenges and Best Practices for ...Challenges and Best Practices for Storing/ Challenges and Best Practices for ...
Challenges and Best Practices for Storing/ Challenges and Best Practices for ...
 
WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN TODAY’S INFORMATIO...
WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN  TODAY’S INFORMATIO...WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN  TODAY’S INFORMATIO...
WHAT IS DIGITAL PRESERVATION? DISCUSS ITS SIGNIFICANCE IN TODAY’S INFORMATIO...
 
Distributed parallel architecture for big data
Distributed parallel architecture for big dataDistributed parallel architecture for big data
Distributed parallel architecture for big data
 
3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptx
3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptx3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptx
3D OPTICAL STORAGE TECHNOLOGY technical seminar 4B.pptx
 
Ex chapter7 questions
Ex chapter7   questionsEx chapter7   questions
Ex chapter7 questions
 
Holographic Memory
Holographic MemoryHolographic Memory
Holographic Memory
 
COMPACT PRESERVATION OF SCRAMBLED CD-ROM DATA
COMPACT PRESERVATION OF SCRAMBLED CD-ROM DATACOMPACT PRESERVATION OF SCRAMBLED CD-ROM DATA
COMPACT PRESERVATION OF SCRAMBLED CD-ROM DATA
 
Efficient usage of memory management in big data using “anti caching”
Efficient usage of memory management in big data using “anti caching”Efficient usage of memory management in big data using “anti caching”
Efficient usage of memory management in big data using “anti caching”
 
iRODS UGM 2016 Preso Summary FINAL
iRODS UGM 2016 Preso Summary FINALiRODS UGM 2016 Preso Summary FINAL
iRODS UGM 2016 Preso Summary FINAL
 
secondarystoragedevices_by_sarthak_goel.pptx
secondarystoragedevices_by_sarthak_goel.pptxsecondarystoragedevices_by_sarthak_goel.pptx
secondarystoragedevices_by_sarthak_goel.pptx
 
3D Optical Data Storage
3D Optical Data Storage 3D Optical Data Storage
3D Optical Data Storage
 
H123.pptx
H123.pptxH123.pptx
H123.pptx
 
Tg01
Tg01Tg01
Tg01
 
Tg01
Tg01Tg01
Tg01
 
Holographic optical data storage jyoti-225
Holographic optical data storage jyoti-225Holographic optical data storage jyoti-225
Holographic optical data storage jyoti-225
 
Android based message encryption decryption using matrix
Android based message encryption  decryption using matrixAndroid based message encryption  decryption using matrix
Android based message encryption decryption using matrix
 
Hierarchical Storage Management
Hierarchical Storage ManagementHierarchical Storage Management
Hierarchical Storage Management
 

Data Storage Considerations for the Tactical Field Collection of Digital Imagery - 2010

  • 1. Research Paper Data Storage Considerations for the Tactical Field Collection of Digital Imagery INTL625 Imagery Intelligence 20 April 2010 Robert L. Watson
  • 2. 2 Table of Contents Introduction................................................................................................................................................. 3 Chapter 1 Introduction to Storage Media ................................................................................................ 5 Magnetic Tape ........................................................................................................................................... 6 Optical Disks.............................................................................................................................................. 7 Magnetic Hard Disks ............................................................................................................................... 11 Solid State Drives .................................................................................................................................... 12 Redundant Array of Inexpensive/Independent Disks (RAID) ................................................................. 17 Chapter 2 Introduction to Video Compression...................................................................................... 25 Video Compression Standards................................................................................................................. 28 Metadata................................................................................................................................................... 30 Multimedia Players and Container Formats............................................................................................. 34 Chapter 3 Recommendations................................................................................................................... 37 Hardware Recommendations................................................................................................................... 37 Software Recommendations .................................................................................................................... 39 Video Retention and Deletion Schemes................................................................................................... 41 The Future................................................................................................................................................ 44 Chapter 4 Conclusion ............................................................................................................................... 46 Bibliography.............................................................................................................................................. 53
  • 3. 3 INTRODUCTION As a result of the Global War on Terror, the need for timely intelligence, both on a global as well as local scale has facilitated the need for more sophisticated imagery intelligence gathering systems that provide useful information from satellites, unmanned aerial vehicles and stationary surveillance systems, to name but a few. This large data gathering effort is unprecedented in history and presents new challenges to our country and its government. A recent article in the New York Times stated: "Military deluged in intelligence from drones: Remote-controlled planes produce about 24 years' worth of video in 2009."1 Other recent news has described the U.S. Intelligence community’s inability to preempt the Christmas Day 2009 terrorist attempt as a byproduct of "information overload."2 Out of necessity, the need for timely imagery intelligence that provides support to both our military and government has created the requirement for large groups of skilled analysts to evaluate the enormous amount of imagery data that is being collected to determine what is significant to the intelligence effort. In light of this, large-scale digital storage systems are necessary to provide data storage and retrieval. However, not all of the imagery that is collected is useable or of significant intelligence value. In some instances, the time or resources may not exist to have the imagery reviewed and analyzed in a timely manner, so its value as an intelligence source may diminish. 1 Christopher Drew, “NYT: Military deluged in intelligence from drones - Remote-controlled planes produced about 24 years’ worth of video in 2009,” New York Times, January 10, 2010, Sunday; http://www.msnbc.msn.com/id/34798080/ns/world_news-he_new_york_times/ (accessed January 11, 2010). 2 Dan De Luce, “US spy agencies face information overload: experts,” AFP (via Yahoo News), January 7, 2010, Thursday; http://news.yahoo.com/s/afp/20100107/pl_afp/ usattacksintelligence (accessed January 25, 2010).
  • 4. 4 The paper will explore the current storage technologies and methodologies in use, their advantages and shortcomings, and provide some insights and possible solutions to this ever- growing issue. The research will focus primarily on solutions for real-time digital video gathering systems, specifically stand-alone, tactical systems currently used in the field environment. Since a system’s storage capacity and the means of increasing this capability may be limited due to system design and mission constraints, newer storage technologies and methods will be suggested to include specific metrics, both user-initiated and software-based, that could be used as criteria in determining imagery retention and disposal.
  • 5. 5 CHAPTER 1 INTRODUCTION TO STORAGE MEDIA Digital storage devices are the building blocks of storage in disk subsystems and standalone server systems. Functioning in the microscopic realm, they perform the reading and writing functions necessary for the storage of data on nonvolatile media. Digital image processing of remote sensor data and its associated geographic information system (GIS) data requires significant storage resources, which are also necessary for the collection of large imagery databases.3 Table 1 illustrates some of the more common digital mass storage devices and their average time to physical obsolescence – the point at which the media begins to deteriorate and data loss can occur. Table 1. LONGEVITY OF DIGITAL STORAGE MEDIA. AVERAGE LONGEVITY OF DIGITAL STORAGE MEDIA MEDIA TYPE AVERAGE OBSOLESCENCE (IN YEARS) REMARKS Optical Disk >100 Very cost efficient, presently best means of long-term, digital storage Magnetic Disk (Hard Disk Drive) 20 Main system storage medium, but storage longevity normally less than 20 years. Magnetic tape 10-15 Cost efficient, but may become unreadable if not rewound and properly stored in a cool, dry place. Flash-Solid State Disk* 10 Expensive, but provides faster seek time than magnetic hard disks. Data retention for up to 10 years without power applied. Jensen, John R., Introductory Digital Image Processing: A remote Sensing Perspective, New Jersey: Prentice-Hall, 2005, 118-119. *Tudor, Marius, “Are Flash Solid-State Disk Drives Ready for the Enterprise?,” Bit Micro Networks, Inc., 2009, 1 (accessed March 2, 2010). The storage of remote sensor data can significantly impact system reliability and costs. Substantial resources are required for even the most simple of systems. In light of this, mass 3 Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 69.
  • 6. 6 storage media should permit rapid retrieval of required imagery data, provide longevity, and be cost effective.4 MAGNETIC TAPE Tapes are low cost and find frequent use as storage and backup media. They are linear access devices as all data is written to or read from them sequentially.5 They come in several types varying in size, storage capacity, density, length, thickness, number of tracks and reels, and speed. They are composed of magnetic tape with single or dual reels contained in a plastic enclosure.6 Data in older tape drive technology was written by multiple heads in a parallel track across the entire tape with some drives using a helical scanning method which wrote the data diagonally. A linear serpentine recording method is used for modern tape drives, which requires more tracks and fewer tape drive heads. Data is written using the linear method except that data is continually written with the head being adjusted and reversed once the end of the tape is arrived at.13 Tape media is not without its drawbacks. Since access to data stored on magnetic tape is linear or sequential, it is not considered efficient for random data access. The linear nature of tape also makes recovery and back up operations time consuming. This is the reason why tape is not considered for use as primary system storage, but is used mainly for offline data storage and vaulting.7, 8 4 John R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey: Prentice-Hall, 2005), 117. 5 Shrivastava. Op. Cit. 269. 6 Ibid. 33. 7 Ibid.
  • 7. 7 Additionally, magnetic tapes slowly deteriorate over time developing surface cracks, tears, and corrosion of the metal oxide coatings. Whether they are being used or stored for future use, they should be maintained in environments having moderate temperatures and low humidity.9 Improper maintenance (e.g., not rewinding the tape) and storage can cause magnetic tape media to become unreadable within ten to fifteen years.10 OPTICAL DISKS An optical disk is another recordable storage media which comes in a variety of types to include compact disks (CDs), digital video/versatile disks (DVDs), magneto-optical (MO) disks, and Blu-Ray disks (BDs).11 Optical disks comprise three broad categories, which determine their usability. These categories are as follows12 : 1. Read only optical disks, which are recorded when they are manufactured and cannot be altered or erased. They include Compact Disk (CD), Compact Disk – Read Only Memory (CD-ROM), Digital Versatile/Video Disk – Read Only Memory (DVD- ROM), Digital Versatile/Video Disk – Video (DVD-Video), and Blu-Ray Disk (BD). 2. Write Once Read Many (WORM) optical disks can be recorded once and cannot be erased. These include Compact Disk – Recordable (CD-R), Digital Versatile/Video Disk – Recordable (DVD-R), and Blu-Ray Disk – Recordable (BD-R). 3. Rewriteable/Magneto optical disks, which can be written, erased and read from any number of times. These include Compact Disk – Rewriteable (CD-RW), Digital Versatile/Video Disk – Rewriteable, Blu-Ray, and magneto-optical (MO) disks. Regardless of type, they are manufactured using similar technologies which incorporate a thin polycarbonate disk which is impressed with microscopic bumps arranged in a continuous, 8 Josh Judd, Principles of SAN Design: Design Build and Manage SANS, (Pennsylvania: Infinity Publishing, 2007), 18-19. 9 Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 83. 10 John R. Jensen, Introductory Digital Image Processing: A remote Sensing Perspective, (New Jersey: Prentice-Hall, 2005), 118. 11 Encyclopedia.com, "Optical Disk," The Columbia Encyclopedia, Sixth Edition, 2008, http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010). 12 Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006, http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
  • 8. 8 spiral track of data. Following this, the disk is coated with a thin, reflective layer of aluminum (barium ferrite for magneto-optical disks) to cover the bumps. A thin acrylic layer is then sprayed over the metallic coating for protection.13, 14 Common optical disk media and their typical storage capacities are shown in Table 2. Table 2. COMMON OPTICAL STORAGE MEDIA. OPTICAL MEDIA TYPICAL STORAGE CAPACITY REMARKS CD/CD-ROM 700Mb Recorded at time of manufacture and cannot be altered/erased CD-R 650Mb Write Once Read Many (WORM) media. Once written, can only be read from. CD-RW 650Mb Data can be written/erased/read from the disk any number of times. DVD/DVD-ROM 9.4Gb (4.7Gb per side) Recorded at time of manufacture and cannot be altered/erased DVD-R 9.4Gb (4.7Gb per side) Write Once Read Many (WORM) media. Once written, can only be read from. DVD-RW 9.4Gb (4.7Gb per side) Data can be written/erased/read from the disk any number of times. BD* 25Gb (single layer) 50Gb (double layer) Recorded at time of manufacture and cannot be altered/erased BD-R* 25Gb (single layer) 50Gb (double layer) Write Once Read Many (WORM) media. Once written, can only be read from. BD-RE* 25Gb (single layer) 50Gb (double layer) Data can be written/erased/read from the disk any number of times. MO** 128Mb to 9.2Gb depending on disk size Data can be written/erased/read from the disk any number of times. Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006, http://www.selfseo.com/story-18894.php (accessed March 11, 2010). *Wikipedia contributors, "Blu-ray Disc recordable," Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc_recordable&oldid=346269932 (accessed March 12, 2010). **Wikipedia contributors, "Magneto-optical drive," Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Magneto-optical_drive&oldid=338693032 (accessed March 12, 2010). Optical disk systems write the data to the media using a low-power laser that etches binary bits into the reflective layer. In this technique, the bits are heated to 150 degrees centigrade, which are then realigned when a magnetic field is applied, creating a binary bit one. 13 Encyclopedia.com, "Optical Disk," The Columbia Encyclopedia, Sixth Edition, 2008, http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010). 14 Mediatechnics Systems, Inc, "How a CD is Made," Mediatechnics Systems, Inc., FAQ, https://www.mediatechnics.com/cdfaqs.htm (accessed March 5, 2010).
  • 9. 9 The recording of new data requires that existing bits be reset to a binary bit zero.15 To read information from the disk, polarized light from a low-power laser is rotated according to the direction of the magnetic field and the original binary signal is reproduced.16 Together with an optical disk drive, optical disks function much the same way as hard disk drives. Access time components (of the optical disk drives laser seeking a target track and acquiring the target sector ) are similar to the seek and rotational latencies present in hard disk drives.17 The use of optical disks over magnetic storage media offers many advantages to include higher storage capacity, lower cost, high data stability, environmental tolerance, and long shelf life.18 The long-term storage potential exceeds 100 years and provides the means of storing large amounts of data on a relatively small media footprint.19 Optical disks have a constant linear velocity which provides a constant read/write bandwidth when access is sequential. This makes the media appropriate for multimedia backup and restore operations, which are generally sequential in nature.20 As such, rewriteable CD-RWs 15 R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey: Prentice- Hall, 2005), 118. 16 Daintith, John, "Magneto-optic Storage," A Dictionary of Computing. 2004, Encyclopedia.com, http://www.encyclopedia.com/doc/1O11-magnetoopticstorage.html (accessed March 5, 2010). 17 Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 123. 18 Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006, http://www.selfseo.com/story-18894.php (accessed March 11, 2010). 19 R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey: Prentice-Hall, 2005), 118. 20 Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003).
  • 10. 10 and DVD-RWs have replaced tapes as the primary backup system in most remote sensing laboratories.21 However, the recent advent of Blu-Ray, deemed as a future replacement for the DVD format, offers even greater storage capacity. Having the same physical dimensions as CDs and DVDs, a standard, double layer Blu-Ray disk can store up to 50GB of data. The Blu-Ray standard is open-ended with theoretical storage limits left unspecified. Larger disk capacities of 100 and 200 GB are currently available.22 The use of optical disks is not without its drawbacks. In relation to hard disk drives, the data access latencies are much higher in optical drives. The typical seek time for hard disk drives is 10msec compared to 100-300msec for optical drives.23, 24 Constant improvements in hard disk drive technology (i.e., price, capacity, and speed) make the decision to use optical disks less inviting. The development of DVDs did provide some improvement in the storage capability, but applications are limited with many organizations still relying on tapes and hard disk drives for business and scientific data storage. However, the Blu-Ray disk may provide the impetus necessary to facilitate a greater move towards the use of optical media for large database storage.25 21 Jensen. Op Cit. 22 Wikipedia contributors, "Blu-ray Disc," Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc&oldid=349002833 (accessed March 12, 2010). 23 Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 123. 24 The Free On-line Dictionary of Computing, "Average Seek Time," http://encyclopedia2.thefreedictionary.com/average+seek+time (accessed March 12 2010). 25 Simitci. Op Cit.
  • 11. 11 MAGNETIC HARD DISKS The Magnetic Hard Disk, also referred to as a Hard Disk Drive (HDD), is an electromechanical device which controls the performance of the storage system environment, providing the primary function of reading and writing the data that is stored on the media. They are the primary storage medium used on computers for the storage and access of data and software applications.26, 27 A magnetic hard disk consists of round, magnetic platters which encode digital data with magnetically charged media. Each platter is arranged in cylinders/tracks with sectors on which respective data is stored. The platters spin at a high rate of speed, up to 15,000 revolutions per minute (RPM). Several platters, a read/write head, and a controller constitute the main components of a HDD. The servo-controlled read/write head is attached to a rapidly moving arm and is positioned over specific sectors to provide access to data. Since the HDD is a read/write mechanism, data may be consistently written to and removed from the media.28, 29 HDDs provide quick and simultaneous access to arbitrary data locations and have large capacities. Multiple disks may be configured into storage arrays providing increased capacity and improved performance.30 In relation to other storage media like optical disks and tape, 26 Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing, Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 33. 27 Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 69. 28 Dane Nelson and Sam Siewart, PhD, “Solid State Drive Applications in Storage and Embedded Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, 30, http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf (accessed March 2, 2010). 29 Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing, Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 27. 30 Ibid.
  • 12. 12 HDDs provide high data throughput and low latency data retrieval.31 However, as degradation may occur in less than twenty years, HDDs have a comparatively short storage life. The continual use of a HDD will reduce its life expectancy significantly.32 SOLID STATE DRIVES Solid state drives (SSDs) use semiconductor flash memory chips for data storage and retrieval. Unlike mechanical HDDs, they have no moving parts. All processes are operated via a controller assembly that is also composed of semiconductor materials. SSDs have improved in recent years and are used when high performance is required for mission-critical applications.33,34 They use nonvolatile memory which supports persistent data storage and are constructed in either single-level cell (SLC) or multi-level cell (MLC) configurations, which are used to store data bits on respective memory cells. SLC storage is used in high performance memory and stores one bit per cell. MLC memory stores multiple bits per cell providing slower data transfer rates, but is cheaper to manufacture than SLC memory.35 SSDs using SLC technology and high reliability controllers imitate HDDs via a traditional storage interface and provide ultra-fast read/write performance, high reliability, high 31 Lawrence D. Bergman, and Vittorio Castelli, Image Databases: Search and Retrieval of Digital Imagery (New York: John Wiley & Sons, Inc., 2002), 144. 32 Jensen, John R. Introductory Digital Image Processing: A remote Sensing Perspective. (New Jersey: Prentice-Hall, 2005), 118. 33 Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing, Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 80. 34 Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, 30, http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf (accessed 03-02-10). 35 Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing, Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 80.
  • 13. 13 data integrity, and minimized power requirements. This makes them ideal for supporting applications that require the rapid processing of large amounts of information such as real-time data feeds.36 HHDs, on the other hand, use rotating media and servo-actuated read/write heads, which introduce seek and rotation latencies. The device is also prone to mechanical failure. Their ever- increasing capacity due to advances in the areal density of the recording surfaces and their low cost are their main advantages, but fast random access limitations have always been an issue.37 As a rule of thumb, central processing units (CPUs) are able to process data much faster than mechanical HDDs can transfer it. HDDs have been the mainstay of storage for decades with a storage capacity that has increased over 200,000 times that of the first HDDs developed in the 1950’s. During that same time period, decreases in price have also made the media more affordable.38 However, HDD performance has not managed to keep up with processor development. Within the last 30 years, processors have seen exponential increases in speed with only marginal increases in the read/write response time of HHDs. This has resulted in a major break between CPU access capabilities and those of HDDs.39 The performance differences between HDDs and SSDs concern the number of random reads/writes per second that the device can perform and the time necessary to resume full 36 Ibid. 37 Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf , 30 (accessed 03-02-10). 38 Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009, http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf , 1 (accessed March 13, 2010). 39 Ibid, 2.
  • 14. 14 operational power from a low power state.40 The number of read/write operations that can be completed in one second is called IOPs, or inputs/outputs per second. This metric is used to measure the random read/write performance of different storage media. High performance HDDs (15K RPM) normally perform 300 IOPs of random 4-kilobyte (KB) data.41 On the other hand, SSDs are presently capable of processing 25,000+ 4KB IOPs, which is a tremendous increase.42 Performance in HDDs is also affected adversely by the resumption of operation following inactivity. When a HDD is inactive for a certain period of time, it will assume a low power state, moving the read/write head off to the side and stopping the spinning platters. Upon the next read/write request, the read/write head must be moved back in place and the platters must be reactivated. This normally takes on the order of a few seconds to occur. However, when an SSD is inactive and goes into a low power state, reactivation takes only a few milliseconds.43 The following characteristics are indicative of solid state storage devices44 : 1. Lowest possible access times: 100-1000 times faster than mechanical drives. 2. High bandwidth: provides multiple gigabytes (GB) per second of random data throughput. 3. High IOPS: provides extremely high random input/output (I/O) performance due to low access times and high bandwidth. 40 Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 30, (accessed 03-02-10). 41 Ibid. 42 Brook Crothers, “Seagate Enters Solid-State Drive Market,” CNet News, December 7, 2009, http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010). 43 Nelson. Op. Cit. 44 Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009, http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf, 10 (accessed March 13, 2010).
  • 15. 15 4. Low price for performance: provides the best possible price/performance of all storage devices. 5. High reliability: provides the same levels of data integrity/endurance as other semiconductor devices. 6. Provides more consistent I/O response times. 7. Provides predictable wear and lifespan characteristics. These characteristics certainly favor solid state drives over mechanical hard disk drives. However, barriers to the large-scale adoption of SSDs do exist. First, the cost of SSDs is more than HDDs. These costs are decreasing with the increased demand by organizations that require the performance, reliability, lower power, and/or resistance to shock and vibration that they provide.45 Second, the capacity of SDDs is usually much smaller than that of mechanical HDDs. However, flash density is improving rapidly to meet user demand for increased capacity with 200GB SSDs already commercially available.46,47 Finally, SSDs have a limited number of write cycles per storage cell. After many program/erase operations, flash memory loses the capability to retain data. However, increased density and improved write wear-leveling algorithms have greatly improved the longevity of SSDs, with enterprise-grade SSDs providing as many as 1 million program/erase cycles.48,49 45 Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 31(accessed 03-02-10). 46 Ibid. 47 Brook Crothers, “Seagate Enters Solid-State Drive Market,” CNet News, December 7, 2009, http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010). 48 Nelson. Op. Cit. 49 Jonathan Thatcher, “NAND Flash Solid State Storage Reliability and Data Integrity: An In-Depth Look,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, 2009, http://www.snia.org/forums/sssi/knowledge/education/Solid_State_Storage_Reliability_and_Data_Integrity--An_In- depth_Look.pdf, 26 (accessed March 13, 2010).
  • 16. 16 Notwithstanding the shortcomings in SSDs, they are finding increased usage in environments requiring enhanced response times to include high performance computing, military applications, and data storage.50 The improving performance of SSD technology permits them to complement or replace HDDs in computer, communications, and consumer electronics. They have become the storage media of choice in avionics, industrial, medical, and military equipment, which requires higher reliability under adverse mechanical and environmental conditions.51 Direct attached storage (DAS) is typically used in aerospace, industrial, government, and military applications, which require small form factor, durability, reliability, performance, and the ability to endure extreme field environments. Currently, SSDs are the only practical solution for these requirements, finding use in aircraft, weather balloons, spacecraft, missiles, ships, submarines, trains, tanks and other armored vehicles, and portable military computers to name but a few.52 The increasing requirements for reliability and performance in military, industrial, and business applications is driving standards for data storage to extraordinary levels with faster access, mobility, and extreme reliability becoming more imperative. As the performance limitations of HDDs are reached and the costs of SSDs continue to decrease, SSDs will play a more critical role in an environment that demands faster, more reliable data storage.53 50 Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009, http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf , 10 (accessed March 13, 2010). 51 Marius Tudor, “Are Flash Solid-State Disk Drives Ready for the Enterprise?,” BitMicro, 2009, http://www.bitmicro.com/press_resources_flash_ssd_enterprise.php, 1 (accessed March 2, 2010). 52 Ibid. 3. 53 Ibid.
  • 17. 17 At the present, the mechanical HDD is the main obstacle to storage performance today, due to rotational and seek actuation latencies when accessing data. This problem is complicated when data access is random and of small, distributed I/O. Most hard drives can deliver from a few hundred to around 100 MB/sec of IOPs when accessing sequential large blocks of data.54 However, this issue can be alleviated via two methods. The first method involves the direct replacement of HDDs with SSDs, which can provide a tenfold increase in performance.55 The second method involves the use of a Redundant Array of Inexpensive/Independent Disks (RAID), which can improve HDD performance by allowing concurrent access to data over an entire array of disks, thereby increasing I/O throughput.56 As such, it is expected that HDDs will see continued usage where high capacity storage is necessary and SSDs will be used where higher performance capability is required.57 REDUNDANT ARRAY OF INEXPENSIVE/INDEPENDENT DISKS (RAID) The term “RAID” stands for “Redundant Array of Inexpensive (or Independent) Disks,” and it is one of the most important technologies in storage networking. RAID is often associated with hardware such as disk subsystems and RAID adapters, but it is actually a set of software algorithms that combine storage input/output (I/O) operations across multiple storage address locations. RAID is usually applied with disk drives in disk subsystems, but it can also be applied 54 Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 38 (accessed 03-02-10). 55 Ibid. 56 Ibid. 57 Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009, http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf, 10 (accessed March 13, 2010).
  • 18. 18 across multiple disk subsystems in a storage network. It should be noted that these algorithms are autonomous and apply to any storage media as long as the storage capacities are the same.58 Original research defined five levels of RAID with varying characteristics of performance, capacity, and redundancy. Of these, levels one and five are the most common. RAID Level 1 (RAID-1) encompasses simple disk mirroring without the use of parity or striping. This level does not provide the benefits of performance and scalability related to other RAID levels. RAID Level 5 (RAID-5) calculates parity values for stored data resulting in more efficient data redundancy than mirroring.59 Present day has seen the development of other RAID levels. Table 3 describes the current RAID levels and their main characteristics.60 Table 3. RAID LEVELS. RAID Type Characteristics RAID 0 Striping, no redundancy RAID 1 Mirroring, full redundancy RAID 10 Mirrored stripes, full redundancy RAID 2 ECC protection, not used RAID 3 Byte-interleaved parity, single parity disk RAID 4 Block-interleaved parity, single parity disk RAID 5 Block-interleaved parity, rotated parity blocks RAID 5DP RAID 5 with double parity blocks per stripe Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 117. RAID provides four main benefits for data storage. These are as follows61 : 1. Data redundancy. 2. Large capacity storage. 3. Management consolidation of devices and subsystems. 4. Parallel processing for enhanced performance. 58 Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 191. 59 Ibid. 196. 60 Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 117. 61 Farley. Op. Cit. 193.
  • 19. 19 The main building block of the RAID system is the array. RAID architecture allows combining multiple storage entities, called array members, into a single array that functions as a single, virtual storage device. A RAID can have two or more arrays assembled from member disk partitions as shown in Figure 1.62 3 U RAID CONTROLLER MEMBER MEMBER MEMBER MEMBER MEMBER ARRAY MEMBER DISK PARTITIONS Figure 1. RAID ARRAY WITH DISK PARTITION MEMBERS. Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 192. The use of RAID in image and video servers provides fault tolerance, which is the primary concern for imagery/video databases. These systems attain fault tolerance by either disk mirroring or parity encoding. Disk mirroring provides fault tolerance by the duplication of data 62 Ibid. 192.
  • 20. 20 onto separate disks. Parity encoding techniques use error-correcting codes to reduce storage overhead.63 In disk mirroring, data is stored on two separate disks creating two copies of the data – a mirrored pair, as shown in Figure 2. If one disk fails, the data remains intact on the other (redundant) disk and the controller continues to provide requested data from the surviving disk. Replacement of the failed disk automatically instructs the controller to copy data from the remaining disk to the new one – an operation transparent to the host system.64 4 2 2 1 4 3 1 3 DATA 1 DATA 2 DATA 3 DATA 4 MIRRORING Figure 2. MIRRORING. Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing, Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 55. Mirroring provides complete data redundancy and enables faster recovery time following disk failure. On the other hand, it only provides data protection and is not to be confused as a 63 Lawrence D. Bergman, and Vittorio Castelli, eds., Image Databases: Search and Retrieval of Digital Imagery (New York: John Wiley & Sons, Inc., 2002), 145. 64 Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing, Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 55.
  • 21. 21 substitute for data backup. In this instance, mirroring continuously captures data changes, as opposed to backups, which capture point-in-time images of the stored data.65 The amount of required storage is twice the amount of data needing to be stored, so mirroring is considered expensive, being used on mission-critical applications which cannot suffer data loss. The technique does provide improved read performance since read requests can be read from both disks. However, there is an obvious deterioration in write performance since each write request manifests itself as two writes on the disks.66 To understand the parity method of fault tolerance, the concept of “striping” must be explained. In a RAID array, each disk has a predefined number of contiguously addressable blocks called “strips.” The series of aligned strips spanning across all of the disks in the array is called a “stripe.”67 This division and distribution of data across multiple disks is called striping.68 RAID using this technique does not provide data protection unless mirroring or parity is used in the process. However, use of striping can considerably increase I/O performance if the RAID controller is configured to simultaneously access data across multiple disks.69 Parity is a mathematical technique used to re-create missing data. Parity RAID protects striped data from disk failure through the addition of an extra disk. This disk is added to the 65 Ibid. 55. 66 Ibid. 56. 67 Ibid. 54. 68 Ibid. 433. 69 Ibid. 54.
  • 22. 22 stripe width and is used to hold parity, which provides a redundancy check that ensures protection of data without having to maintain a full set of duplicate data.70 As opposed to mirroring, the use of parity significantly reduces data protection costs. For example, in a RAID configuration using five disks (Figure 3), one disk retains the parity information while the other four retain data. In this case, parity would require only 25 percent extra disk usage compared to mirroring, which would need 100 percent extra disk usage.71 However, the use of parity is not without its disadvantages. Since parity information is created by the data currently residing in the disk, it is recalculated for every change in data. This process is time consuming and negatively affects the performance of the RAID controller.72 3 13 1 3 1 11 2 2 3 1 2 1 1 2 7 5 7 9 PARITY DISK DATA DISKS Figure 3. PARITY RAID. Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing, Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 56. Additionally, in the case of a disk failure, the load increase on the surviving disks results in deadline violations during video stream playback operations. To prevent this occurrence using 70 Ibid. 56. 71 Ibid. 56-57. 72 Ibid.
  • 23. 23 conventional fault-tolerance methods, disk use must be decreased during the fault-free state. This reduction may be accomplished prior to imagery compression by using partitioning techniques that take advantage of the characteristics of imagery through the use of two general methods.73 First, the sequential nature of video access may be used to reduce the system overhead of on-line recovery in a RAID. This may be accomplished by the computation of parity over a video stream block sequence, ensuring that retrieved data used to recover a block that was stored on the failed disk would be requested by the system in the future. The blocks are temporarily stored in and serviced from the buffer, minimizing the time necessary for the failure recovery process.74 Second, since human perception is tolerant to minor image distortion, the inherent redundancies in imagery may be used to reconstruct lost imagery data to a fairly accurate degree. This method uses error-correcting codes to accomplish this. The imagery is divided into sub- images that are stored across the array members. The failure of a single disk will result in fractional losses for several images.75 However, if the sub-images are generated using pixels, and none of the neighboring pixels belong to the same sub-image, all elements of the lost pixels will be accessible in the event of a single disk failure. The relationship between the neighboring pixels will allow for an 73 Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital Imagery (New York: John Wiley & Sons, Inc., 2002), 146. 74 Ibid. 75 Ibid.
  • 24. 24 approximate reconstruction of the original imagery without the need for additional information from any of the other surviving disks members.76 It should be noted that use of the aforementioned techniques will adversely affect the compression of the imagery due to the reduction of the correlation between pixels allocated to the same sub-images. This will increase the bit rate, imposing greater load requirements on all disks in the RAID and reduce the number of video streams that may be retrieved simultaneously.77 Post-compression partitioning algorithms have been developed to address this limitation and are discussed in the following section. 76 Ibid. 77 Ibid.
  • 25. 25 CHAPTER 2 INTRODUCTION TO VIDEO COMPRESSION Present day imagery technology has provided an endless amount of photographic and video data, which presents a major challenge in its capture, processing and storage. As this technology has evolved, the need to improve the quality and to reduce the scale of the imagery has become of primary importance. This is particularly true since the development of high resolution photography and video has created more complex files, which can be of extremely large size, requiring more resources to use. This ever-growing database has required the development of techniques to control the size and complexity of the imagery and to allow ease of use. The science of digital compression technology was born out of this need. Compression is the science of reducing the amount of data used to convey information about an object. Because information has order and patterns, these can be extracted and reconstructed to provide the fundamental nature of the original information using less data for transmission and reception.78 Certain restrictions are required for the compression of multimedia. The encoded/decoded data should provide the best possible quality. Complexity should be minimal to provide limited data link delay and cost-effective implementation. Modern compression techniques must compromise between these requirements.79 The goal of compression technology is the elimination of redundant data while retaining only the information that is essential for the effective reproduction of the original data. In other words, the number of bits represented by a signal is reduced to provide a smaller data file that 78 Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 2. 79 Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke Wöhrmann B.V., 1998), 45.
  • 26. 26 still represents the fundamental nature of the original data. This technique is accomplished using either “lossless” compression, “lossy” compression, or a combination of both methods.80 Lossless data compression algorithms permit exact reconstruction of original data following compression and allow for compression ratios of about four to one.81, 82 It is used when it is critical that the original data be identical following decompression. This is important for executable files, source code, and some image file formats like portable network graphics (PNG) and graphics interchange format (GIF). It has many applications to include ZIP utilities, and is often used in conjunction with lossy compression methods.83 As opposed to lossless compression, lossy compression does not allow for complete recovery of the original signal. Only a near approximation is available due to the tradeoff required to provide reasonable compression of the signal.84 Compression ratios ranging from eight to one and twenty to one can provide images that are visually comparable to the original. It is possible to produce higher ratios for lossy compression, but a noticeable difference will exist between the original and the compressed image.85 The most common use of lossy compression 80 Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital Imagery (New York: John Wiley & Sons, Inc., 2002), 211. 81 Ibid. 82 Wikipedia contributors, "Lossless data compression," Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Lossless_data_compression&oldid=346709055 (accessed March 10, 2010). 83 Ibid. 84 Ibid. 85 Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital Imagery (New York: John Wiley & Sons, Inc., 2002), 211.
  • 27. 27 is for the compression of multimedia data like audio, video, and still imagery. So, it finds greater use in streaming media and internet telephony applications.86 A standard image compression method consists of three components: pixel-level redundancy reduction, data discarding, and bit-level redundancy reduction. The normal image compression process is shown in Figure 4. Note that lossless image compression systems do not use the data discarding process as compression is achieved solely from the redundancy reduction processes. Lossy image compression generally uses all three processes, but may omit bit-level redundancy.87 Figure 4. IMAGE COMPRESSION COMPONENTS. Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital Imagery (New York: John Wiley & Sons, Inc., 2002), 212. The pixel-level redundancy process reverses the mapping of the image, disassociating the original pixels from the output. This process enumerates the image pixilation (converts it to data bits) in a way that more corresponds to the human visual system frequency response. This data is then transmitted to the data discarding process.88 The data discarding process removes insignificant data from the original data stream that has been divided into separate data bits during the pixel-level redundancy process. It then discards the data bits that are deemed unnecessary, and retains those that are minimally required 86 Wikipedia contributors, "Lossy compression," Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Lossy_compression&oldid=347940008 (accessed March 10, 2010). 87 Op Cit. 212. 88 Ibid.
  • 28. 28 to produce a near-perfect rendition of the original, without loss of image fidelity. Statistical image properties and human visual characteristics are considered in this approximation. This data is then sent for bit-level redundancy reduction for final processing.89 The bit-level redundancy reduction process is considered a lossless process. It is used to remove and/or reduce any unnecessary dependencies left over from the data discarding process.90 Following compression of the imagery by one or more compression standards such as those developed by the Motion Pictures Experts Group (MPEG), the imagery is then sent to the applicable storage location. VIDEO COMPRESSION STANDARDS Several video compression techniques exist that use different techniques to reduce data size.91 The most commonly used family of digital video compression techniques is the Motion Pictures Experts Group (MPEG) standards, which are briefly described in Table 4. Table 4. MPEG STANDARDS. STANDARD OBJECTIVE CHARACTERISTICS APPLICATIONS MPEG-1 To provide a standard for encoding motion video at bit rates transportable over T1 data circuits and for replay on CD-ROM. Provided audio and video recording at the same data rate. Broadcasting in any form and large-distribution CD-ROMs. MPEG-2 To provide a standard that included broadcast quality video. Provides video interlacing, scalable syntax, a system layer to handle multiple program streams. Most popular standard presently in use. Used for HDTV and for compression of other video/audio data. MPEG-3 To provide a compression system suitable for high- definition television (HDTV) broadcasting. Abandoned when determined that MPEG-2 was able to accommodate the need for HDTV. N/A MPEG-4 To provide for encoding of video and audio at very low bit rates. Provides for the combining of multiple video streams. Increased efficiency and error forgiveness over MPEG-2. Provides end-user interaction with specific applications like games, interactive TV, and educational systems. MPEG-7 To provide a standard for applying metadata technology to recorded video and audio. Uses metadata technology to catalog/ index video/audio files to accommodate search engine retrieval. In work, but will provide ease of retrieval of data in large-scale databases. MPEG-21 To provide a complete structure for the management and use of digital assets. Provides all infrastructure support for commercial transactions and rights management. In work, but will provide intellectual property management and protection. Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 152,172, 196, 268, 276. 89 Ibid. 90 Ibid. 91 Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 2.
  • 29. 29 MPEG is a committee that was formed in 1988 under the Joint Technical Committee of the International Standards Organization (ISO) and the International Electro-technical Commission (IEC). Their formation was facilitated to derive a standard for encoding full motion video at rates suitable for transport over T1 data circuits and for replay from CD-ROM at about 1.5Mbits/sec. This was the birth of the MPEG-1 standard, which ultimate goal was to record sampled audio at 48 kHz together with video, at the same data rate.92 Following the development of MPEG-1, a standard was required that allowed for the compression, storage, and digital transmission of television broadcast quality video. This project became MPEG-2, which extended and improved MPEG-1.93 Among other things, MPEG-2 included the provision of video interlacing, a scalable syntax, and the addition of a system layer to handle multiple program streams. In light of this, MPEG-2 has become the worldwide standard for both standard- and high-definition transmissions.94 After development of MPEG-2, a standard that would support compression suitable for high-definition television (HDTV) was deemed necessary. This effort became MPEG-3. However, it was discarded very early into the program when it became clear that the new compression algorithms and schemes used in MPEG-2 were capable of handling the HDTV requirement.95 With the demise of MPEG-3, MPEG-4 was started to accommodate the encoding of video/audio at very low data rates. Later amendments also provided applications that allowed 92 Ibid. 152. 93 Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke Wöhrmann B.V., 1998), 45. 94 Symes. Op. Cit. 172-173. 95 Effelsberg. Op. Cit. 45.
  • 30. 30 end-user interaction with the video. This was particularly useful for gaming, interactive television, educational systems, and high-end data retrieval systems. However, the standard has not seen wide acceptance due to the popularity of MPEG-2.96 The current amount of digital video and audio information presently available in the public domain is enormous and continues to grow. To sort, maintain, and access required information in a timely manner, MPEG-7 has been proposed and may provide the future means by which digital video is cataloged, stored, and retrieved. The MPEG-7 standard uses metadata technology to both catalog and index audio-video files using methods that complement data retrieval.97 The latest effort in video compression by the MPEG organization is MPEG-21, which also accommodates metadata technology. The standard was undertaken to provide a complete framework for the use and management of digital resources. Infrastructure support for commercial transactions and rights management is also an important part of the standard, which focus is “to enable transparent and augmented use of multimedia resources across a wide range of networks and devices.”98 METADATA To better understand what the MPEG-7 and MPEG-21 standards can provide to the organization and retrieval of multimedia data, the concept of metadata requires further explanation. Metadata is the “data about the data” or the “information about the information.” It 96 Op. Cit. 196, 222. 97 Ibid. 268-269. 98 Ibid. 276.
  • 31. 31 describes the various attributes of a given piece of information by providing meaning and context. It also aids in the location, retrieval, use and management of the information.99, 100 Metadata provides many benefits to include101 : 1. Aids in the search for required information. 2. Aids in organization of digital resources. 3. Facilitates interoperability and legacy resource integration. 4. Provides digital identification. 5. Supports data archiving and preservation. Typical remote sensing systems have enormous amounts of imagery that must be maintained. To keep track of this data, information about the images is necessary. This metadata can be stored with the imagery or in a separate database using pointers to link the applicable imagery with its respective data set. Various metadata may be used in search and retrieval operations while other metadata may provide user information.102 Metadata, as used in remote sensing systems, consists of several characteristics with some elements associated with a collection of images and others that specify smaller subsets called “granules,” which can be a single image or a small set of images. Examples of these characteristics are as follows103 : 1. General Description, which includes the name, topic, collection version, etc. 2. Data Origin, which describes the platform, instrument, sensor used, etc 3. Spatial Coverage, which specifies the geographic area covered by the imagery in the collection (e.g., location coordinates of the corners of a bounding rectangle). 99 Cornell University Library, “Moving Theory into Practice: Digital Imaging Tutorial, Chapter 5, Metadata: Definition, Types, and Functions,” http://www.library.cornell.edu/preservation/tutorial/metadata/metadata-01.html (accessed January 14, 2010). 100 National Information Standards Organization, Understanding Metadata (Maryland: NISO Press, 2004), http://www.niso.org/publications/press/UnderstandingMetadata.pdf (accessed January 20, 2010), 1. 101 Ibid. 102 Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital Imagery (New York: John Wiley & Sons, Inc., 2002), 62. 103 Ibid.
  • 32. 32 4. Temporal Coverage, which specifies the time range during which the imagery was obtained (e.g., start/end date/time). Standards for the collection of geographic data have been established by the U.S. Federal Geographic Data Committee (FGDC). All geographic data that is managed by federal government agencies must comply with this standard. The standards serve as a guideline for the use of metadata in remote sensing applications.104 The ISO has also established a working group on metadata for the purpose of developing standards for the recording of metadata on a separate track than the audio-video. This would provide a powerful means of retrieval for later access to the digitized data and illustrates the importance of the MPEG-7 and MPEG-21 standards.105 The rest of this section will discuss metadata in regards to the MPEG-7 standard. The scope of the MPEG-7 standard is illustrated in Figure 5. From the figure, it is clear that only the syntax (Standard Description) is standardized. Devices that might generate metadata and how that data is represented are specified by the standard. Generation of metadata (i.e., Feature Extraction) and the applications that might use it (i.e., Search Engines) are unspecified and left to commercial developers to provide.106 Several groups are presently developing video content analysis algorithms to automatically extract semantic information from video data with the hopes of partially automating the metadata creation process.107 104 Ibid. 63. 105 Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke Wöhrmann B.V., 1998), 58. 106 Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 270. 107 Effelsberg. Op. Cit.
  • 33. 33 FEATURE EXTRACTION STANDARD DESCRIPTION SEARCH ENGINE MPEG-7 SCOPE Figure 5. SCOPE OF THE MPEG-7 STANDARD. Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke Wöhrmann B.V., 1998), 58. Assuming metadata is recorded simultaneously with the audio-visual data and timing information, search engines can be developed that can access specific audio or video data from large digital archives or databases.108 This timely access to required material can provide immediate use – an important consideration especially for mission critical applications. The generation of metadata poses two main issues. The first is the amount of data that is generated. Human interaction in the indexing process will result in only a fraction of the total created content being indexed. In this case, the automated analysis of digital records is critical and presents many technical challenges. The second issue regards the standardization of metadata descriptions. The situation has improved with the widespread availability of information.109 Conversely, practical application of metadata has been made problematic by the creation of different standardization methods. The MPEG-7 standard calls for extensible markup language (XML) schemas as opposed to the Society of Motion Picture and Television Engineers (SMPTE) which uses key-length-value (KLV) coding. Other organizations have created their 108 Ibid. 109 Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 270.
  • 34. 34 own metadata sets that are based on their own needs. This proliferation of different metadata methodologies creates interoperability issues, which was never the intent of MPEG-7. Efforts are being led by MPEG and SMPTE to synchronize the different methodologies to allow portability between them.110 MULTIMEDIA PLAYERS AND CONTAINER FORMATS This discussion is not complete without providing a clear distinction between multimedia players and multimedia container formats. Multimedia players are the software applications that are used to playback video and audio data files. There are many of these applications available to include Windows Media Player® , Real Player® , and QuickTime® , to name but a few. Multimedia container formats provide for the actual formatting of the recorded video and audio. Different methods exist and are signified by the actual data file extensions. These include, but are not limited to the following format types: AVI, ASF, and MOV. Many of the currently fielded video collection systems use digital video recorder utilities that apply the audio video interleave (AVI) container format for playback operations. Playback of applicable video is then performed using one of the many available multimedia players like Windows Media Player® . The AVI format is proven over some years of usage and has continued to be supported by many multimedia players. For purposes of this paper, only the AVI container format will be discussed. The AVI format has its origins in the Resource Interchange File Format (RIFF), which divides the data into “blocks” that are each identified by a tag. The AVI file is a single block, RIFF formatted file that is further divided into two mandatory sub-blocks and one optional sub- block. The first sub-block is the file header, which contains video metadata such as frame width, 110 Ibid. 275-276.
  • 35. 35 height and rate. The second sub-block contains the actual audio-visual data that is the AVI movie file. The third, optional, sub-block is used to index data block offsets within the file.111 Via the RIFF format, the sub-block containing the actual audio-visual data is encoded and/or decoded by a software algorithm called a “codec,” an abbreviation for (en) coder/decoder. Upon file creation, the codec interprets between raw data and the compressed data format used within the sub-block. AVI files can hold audio-visual information inside the data blocks in almost any compression scheme to include full frame (uncompressed), Motion JPEG, editable MPEG, and MPEG-4 to name a few.112 AVI was not originally intended to contain compressed video, which requires access to video frame data beyond the currently used frame. Methods do exist that support modern video compression use within the AVI framework (MPEG-4, et. al.), even though this is outside the scope of the original specification. Problems can occur during playback with utilities that do not anticipate any issues like not having the correct codec required to play the video file.113 However, most audio-video files can be reviewed for playback provided the necessary codec files are installed on the computing machinery that is being utilized for this purpose. As such, the AVI format continues to see major use in both commercial as well as military applications. The previous sections have described some of the concepts related to data storage. In particular, storage media, data redundancy, and video compression technologies were discussed 111 Wikipedia contributors, "Audio Video Interleave," Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Audio_Video_Interleave&oldid=347331467 (accessed March 10, 2010). 112 Ibid. 113 Ibid.
  • 36. 36 in relation to the large scale storage of recorded audio-video data files. The discussion will now turn to possible solutions to the storage of this data in relation to tactical video acquisition.
  • 37. 37 CHAPTER 3 RECOMMENDATIONS HARDWARE CONSIDERATIONS Although somewhat outside the original scope of this paper, it is important that the system developer/user consider the computing machinery to be used in the tactical video collection system. The personal computer (PC) has advanced to the point that powerful, robust systems are available commercially at a fraction of the cost of high performance systems from years past. Most PCs available for commercial-off-the-shelf (COTS) purchase are quite suitable for digital video recording functions. Notwithstanding the obvious need for robust computing machinery, given the critical nature of tactically fielded video collection systems, some minimal hardware recommendations are as follows: 1. Robust PC-based system using a dual-processor, 64-bit bus motherboard with large random access memory (RAM) capacity. 2. Best commercially available video graphics adapter using the most video RAM. 3. Main internal storage consisting of solid state hard drives (SSDs). 4. Redundant backup using an external (or internal) RAID-based system. 5. CD-RW/DVD-RW/Blu-Ray-RW for archiving purposes. Following these deliberations, the primary, internal system storage should consist of solid state drives (SSDs) of sufficient size to accommodate the operating system (OS), applicable system and recorder applications, and onboard storage. The use of SSDs will allow for enhanced system performance during mission critical operations. External (or internal) RAID storage is recommended using standard hard disk drives, which provide more efficient mass storage properties and should be sufficient for normal backup operations. Regarding archival storage, the use of Blu-Ray optical disks is recommended given both the large capacity and the storage longevity of optical disks. Compared to other optical storage mediums as well as magnetic tape, Blu-Ray best accommodates the requirements necessary for
  • 38. 38 the storage of large amounts of captured video data, and it will also reduce hard storage requirements. A simple hierarchy depicting these guidelines is shown in Figure 6 followed by brief explanations for each level. LEVEL 3 ANCILLARY STORAGE (STORED ON-SITE) LEVEL 2 SECONDARY STORAGE (EXTERNAL/INTERNAL RAID) LEVEL 1 PRIMARY STORAGE (INTERNAL) Figure 6. SIMPLE STORAGE HIERARCHY. LEVEL 1: Primary storage, internal to the computing machinery and using flash Solid State Drives (SSDs) for their enhanced performance and data retrieval. LEVEL 2: Secondary storage for database back up (automated or manual) using RAID-based architecture and consisting of mechanical Hard Disk Drives (HDDs) for their mass storage benefits. LEVEL 3: Ancillary storage using Blu-Ray optical disks for their large storage capacity, environmental durability, and long-term storage properties.
  • 39. 39 SOFTWARE CONSIDERATIONS Once the computing machinery is in place, the system developer/user must next consider the software operating system (OS) and the required application software to be used with the system. The Windows® OS is recommended for its ready availability and widespread use. The digital video recorder (DVR) application may be selected from open or proprietary sources. It should be robust, and its selection based on system considerations. In addition to its recording function it must also provide post-mission playback and video editing utilities. For backup operations, an automated backup and restore system utility is recommended. It should have a built in scheduling capability and operate in the background, transparent to the host system. It should provide the capability for different types of backup operations, the most common of which are explained in Table 5. For mission critical operations, it is recommended that Full backups be performed to provide complete restoration of all data in case of corruption or catastrophic system failure. The utility should also provide the ability to automatically overwrite older video data files when necessary, in the interest of freeing up hard drive space. This means that any video that is intended to be kept for planning or historical purposes must be manually copied and archived on separate media, preferably Blu-Ray optical disk. Table 5. BACKUP OPERATIONS. BACKUP TYPE OPERATION CHARACTERISTICS Full Provides full backup of all data. Time intensive, but provides all data for complete restoration. Incremental Only copies files that have changed since last full backup. Assuming weekly backups, subsequent backups may be larger. Differential Only copies data that has changed since last differential backup. Shortens backup time, but may increase restoration time. Snapshot Point-in-time copy of the data, also called pointer-based backup. Copies pointers and file metadata. Very fast backup - copies only metadata and pointers that point to the data blocks. Does not protect against disk loss. Schulz, Greg, Resilient Storage Networks: Designing Flexible Scalable Data Infrastructures (Massachusetts: Elsevier Digital Press, 2004), 324.
  • 40. 40 A suggested backup scheme is shown in Figure 7. It is recommended that a Master System disk be produced on Blu-Ray optical media following final system setup. This should be maintained on site and will provide a backup in case the magnetic hard disk drive needs to be rebuilt due to file corruption or catastrophic system events. Additionally, a Full system backup that is stored in the external RAID storage assembly should be scheduled as required to intermittently capture the present system configuration. This will allow the user to recover system operation quickly by using a previously known working system configuration. This is also useful in the event of system file corruption or other events that may affect the system operation. Recorded video should also be captured in the external RAID storage area to keep the internal, primary storage medium free of unnecessary files that may adversely affect the overall system performance. The RAID storage capacity should be in the terabyte range to accommodate the large amount of data that will be generated by the video collection system. At a minimum, RAID-1 architecture providing mirroring and full redundancy is recommended.
  • 41. 41 Figure 7. BACKUP SCHEME. Given the nature of real-time, tactical video collection systems, large amounts of data are produced within a short period of time. In the interest of freeing up storage space for the recording of newer video, it is recommended that any video that is desired to be retained for historical reference should be archived on optical media and stored on-site to facilitate its later usage. A master system copy stored on optical disk should also be maintained for recovery operations due to system corruption and/or catastrophic system failure. This should also be maintained on-site and be updated as required. VIDEO RETENTION AND DELETION SCHEMES Tactical operations are defined as “military operations conducted on the battlefield, generally in direct contact with the enemy.”114 Tactical data collection is indigenous to the area of operations in which the tactical collection asset is fielded. Given the nature of tactical 114 Tactical Operations. Answers.com. The Oxford Essential Dictionary of the U.S. Military, Oxford University Press, 2001, 2002. (accessed March 07, 2010).
  • 42. 42 intelligence data such as real-time video, the usefulness of the data may diminish quickly within a very short period of time. However, this data is still retained in storage although it may not provide any useful intelligence value. This is generally true for most video that is collected over time and the burden on storage assets can be significant. This section outlines a selection process that may be used to determine automatic retention and/or deletion of tactically collected video. Figure 8 is an example of a simple, software driven selection process, which could use criteria such as the number of times a file has been accessed within a given period of time to determine retention or deletion. It might also be based on guidelines established by the command to determine the long-term or short-term tactical intelligence value of specific video. RAW VIDEO VIDEO PROCESSING VIDEO DATABASE VIDEO FILE VIDEO SCREENING PROCESS RETENSION DELETION NO TACTICAL VALUE TACTICAL VALUE Figure 8. A SIMPLE VIDEO RETENTION/DELETION SCHEME. A more in depth selection process is shown in Figure 9. The displayed retention times shown are arbitrary examples. The automatic deletion of stored video imagery would rely on specific, established criteria to determine whether a file is a candidate for deletion. At a minimum, criteria might include the date/time stamp (from metadata) and the number of times the specific files have been accessed, which denotes level of importance. However, user
  • 43. 43 intervention might be required to determine the retention of specific video that contains significant events such as: 1. Reconnaissance operations. 2. Surveillance operations. 3. Intelligence gathering operations. 4. Mission operations support. 5. IED incidents. 6. Counter-IED operations. 7. Area/terrain observations in support of missions. 8. Areas of interest (AI). 9. Any video showing direct/indirect contact with an enemy. 10. Other significant events that are deemed to provide intelligence value of a tactical nature, as determined by the command. VIDEO DATABASE PROCESSED VIDEO VIDEO FILES RETENTION SCREENING ALGORITHM CRITICAL VALUE HIGH VALUE MEDIUM VALUE LOW VALUE NO VALUE TAGGED VIDEO RETAIN INDEFINITELY RETAIN 6-12 MONTHS RETAIN 3-6 MONTHS RETAIN 1-3 MONTHS RETAIN 30 DAYS TAGGED VIDEO 3-6MOS 6-12MOS 1-3MOS AUTOMATED DELETION ALGORITHM 30 DAYS DELETED VIDEO INDEFINITE RETENTION Figure 9. A MORE ROBUST VIDEO RETENSION/DELETION SCHEME. Additionally, unless it is used for planning purposes or as an historical reference in support of future operations, most tactically captured video may have a short shelf life for
  • 44. 44 providing up to date intelligence information. Of course, this is dependent upon the intelligence requirements and the value placed on specific video by the command. FUTURE CONSIDERATIONS Current digital video recorder software applications use a date/time stamp to sort recorded video. This does allow for automated sorting as the video data files are sent to a predetermined storage location and are sorted according to the date and time they were recorded. However, this convention does not allow for the sorting of video based on its content. The search and retrieval of multiple related records is also not supported. In normal operations, the user must make note of the specific time and date of an event and access the file manually, usually having to manually parse through different file locations and numerous video files to locate the one in question. The use of metadata can alleviate this issue. The U.S. government’s Federal Geographic Data Committee (FGDC) describes metadata as the data about the content, quality, condition, and other characteristics of the data.115 At this time, the FGDC is the organization that has been tasked with developing the guidelines and standards for digital metadata that is to be used in geospatial imagery. Although metadata is being developed primarily for use in the support of geographic information system (GIS) mapping data, it would serve equally useful from the perspective of tactical video collection. Some minimal considerations for items that might be included as part of a data element set that should be included with the metadata might be as follows: 1. Time/date stamp. 2. Geographic locations of known points. 3. Location Coordinate System (Military Grid Reference System (MGRS)). 4. Distance/altitude units (feet/meters). 115 Federal Geographic Data Committee, Content Standard for Digital Geospatial Metadata, http://www.fgdc.gov/standards/projects/FGDC-standards-projects/metadata/base-metadata/v2_0698.pdf, 64 (accessed January 20, 2010).
  • 45. 45 5. Altitude/elevation data in reference to location. 6. Populated areas: cities, towns, and villages. 7. Rural areas: farms, ranches, etc. 8. Infrastructure: schools, public buildings, places of worship, power stations, military installations, police stations, telecommunications (antenna towers), water treatment facilities, transportation (roads/highways both improved and unimproved, intersections, traffic circles), airfields/airports/airstrips, dams, bridges, waterways (lakes, oceans, seas, rivers, streams, man-made), etc. Many of these items already have a place in the metadata guidelines developed by the FGDC. Other important metadata considerations have also been outlined by the FGDC, which include mission specific data items. These may easily be adapted in the metadata that is used in support of tactical video collection efforts. Some of this mission specific data is as follows116 : 1. Mission data/time. 2. Mission name. 3. Mission significant events. 4. Mission Platform and Instrumentation. At the present, the use of metadata in many video applications has been limited. Its usage may also necessitate user intervention for the manual input of some data items. However, the advent of the MPEG-7 and MPEG-21 video compression standards may provide a path forward for the intelligent organization of video data files. The adoption of these compression methods in future video collection systems will allow video files to be “tagged” and automatically sorted, catalogued, and organized according to specific attributes that are captured as part of the metadata architecture. This will aid in the timely search and retrieval of related video data applicable to specific software queries. 116 Federal Geographic Data Committee, Content Standard for Digital Geospatial Metadata: Extensions for Remote Sensing Metadata, http://www.fgdc.gov/standards/projects/FGDC-standards- projects/csdgm_rs_ex/MetadataRemoteSensingExtens.pdf, 67-71 (accessed January 20, 2010).
  • 46. 46 CHAPTER 4 CONCLUSION The purpose of this paper was to evaluate the issue of database storage for the tactical video collection process. This is a major concern for end users of these systems given the large amount of video that is collected on a daily basis. The previous discussion has attempted to explain the necessary hardware that is available and the software related techniques that are used in this process. Possible recommendations were also provided that may assist in easing the concerns related to this issue. First, large capacity storage media were discussed to include magnetic tape, optical disks, magnetic disk drives, and finally solid state drives. These media types are of particular importance since they provide the best large-scale storage capacity. Each was evaluated to determine which of them were best suited to be used in a tactical environment. Redundant storage in the form of the RAID architecture was also examined. Magnetic tape media has been a mainstay for the backup of digital video and imagery as it provides a low cost method of backing up data for archival purposes. However, the sequential nature of magnetic tape media makes it very inefficient for random data access, and recovery/backup operations. This is particularly important when fielding a tactical collection system since the command cannot afford to be without the asset for long periods of time following, for example, a catastrophic failure. The system must be recoverable very quickly. Magnetic tape media also deteriorate over time and must be stored in moderate environments with low humidity. The conditions in many of the current locations in which these assets are fielded make the use of magnetic tape media problematic.
  • 47. 47 Optical disks have many advantages over magnetic tape to include lower cost, greater storage capacity, data stability, environmental durability, and a long shelf life potential of over 100 years. Their constant linear velocity provides unvarying read/write bandwidth during sequential access operations, which makes them ideal for multimedia backup and restore operations. These reasons have made optical disk technology the primary backup media in many remote sensing operations. Magnetic hard disks, also known as hard disk drives, are the primary storage technology used in most personal computer systems and many video collection systems. They provide fast, concurrent access to data and have very large capacities. Because of this, they are still useful as a storage medium in redundant systems such as RAID assemblies. However, as they are electro- mechanical devices with moving parts, extreme environments together with continual use can dramatically increase their mortality rate. Solid state drives, on the other hand consist of high reliability flash memory technology and have no moving parts, which makes them more durable in severe environments. They have much faster response times, lower access times, and much higher throughput bandwidth than magnetic hard disks. Lifespan characteristics are comparable to magnetic hard disks with on- the-shelf data retention of up to ten years. However, solid state drives are more expensive than magnetic hard disks and capacities are usually much smaller. In spite of these shortcomings, solid state drives are fast becoming the primary storage medium for applications that require quicker response time and proven durability in extreme environments. The discussion on RAID described its importance as a backup storage medium and some of the methods used in the different RAID architectures. Two of the most prevalent methods
  • 48. 48 were examined and included RAID Level 1 (RAID-1) and RAID Level 5 (RAID-5). Both methods provide a means of data redundancy and recovery in case of system failures and/or hard disk drive loss. RAID-1 architecture performs only disk mirroring, which provides full redundancy. It also permits quicker recovery in the case of disk failure in the array, and offers increased read performance as data is read across all disks in the array. However, it lacks in overall performance as it is necessary to have twice the storage capacity to accommodate data storage, which can add significantly to costs. In comparison, RAID-5 architecture uses parity checking methods and provides more efficient data redundancy than simple mirroring. It is also less expensive in regards to data protection costs. However, every change in data facilitates a recalculation of parity, which adversely affects system performance. Following the discussion on storage media, the subject of video compression was investigated. Video compression deals with the reduction of the amount of data required to convey a useable facsimile of a specific data object. This is particularly important since most present day imagery can produce extremely large files, which facilitates the need for larger storage capacities. Both “lossless” and “lossy” compression techniques were examined and are used to compress data according to the different standards that have been developed to include those of the Motion Pictures Experts Group (MPEG), which are the most common. Lossless compression techniques allow for complete reconstruction of the original data signal and provide a slight amount of compression. Lossy compression techniques provide only a near approximation of the original data signal since some tradeoff is required to provide satisfactory compression. Imagery can be produced that is fundamentally and visually similar to
  • 49. 49 the original using ratios much greater than those of lossless compression. Lossy compression is most commonly used for compressing multimedia data such as audio, video, and still images. These compression algorithms are used in conjunction with different standards, with the most prevalent being developed by the MPEG. The standards outline the basic requirements for the compression of audio, video, and imagery data. Short discussions of the different MPEG standards were presented with particular attention given to the MPEG-7 and MPEG-21 standards, which are still in their infancy. These standards do provide for the necessary compression of multimedia data, which is the primary consideration. However, it is their use of metadata techniques that is of particular importance. Since metadata considers the peripheral, supporting information about the actual data, its use for the indexing and cataloging of collected data is significant to the video collection process. Multimedia players and multimedia container formats were briefly discussed to provide a distinction between the two. Multimedia players, like Windows Media Player® , are the applications that are used to replay the recorded audio-video data files. Container formats like audio video interleave (AVI) provide the actual formatting process for the video. The AVI container format was explained solely because of its widespread usage and to provide an elementary understanding of this technology. No recommendation has been made for these items since there are many different types which work equally well. Following discussion of the different storage media and software techniques used in the storage process, recommendations were made that might assist in alleviating storage capacity issues in tactical video collection systems. These recommendations were divided into three
  • 50. 50 groups: hardware, software, and future considerations. Recommendations were provided for each of these categories. In developing the collection system, hardware considerations were taken into account to include the motherboard architecture, system memory, processing power, the graphics adapter and internal storage. As such, these suggestions were offered mainly as guidelines with the system storage being the primary focus. In this case, it was recommended that solid state drives be used as the primary data storage media in the computing machinery for tactical video collection systems due to their higher performance capability and durability. Magnetic hard disks should still be used in secondary data storage and backup operations as part of a RAID assembly since they provide higher capacity storage than solid state drives. Redundant storage was also discussed in the form of a RAID assembly, which provides for system backup and large external storage capacity. Magnetic drives should be used for this assembly as they are more efficient than solid state drives for mass storage and should suffice for normal backup operations. The tactical video collection system is a mission-critical platform that cannot tolerate any loss in data, if at all possible. In light of this, it was recommended that RAID-1, providing complete data redundancy, be used to accommodate quicker system recovery in case of system corruption or total system failure. The archiving of data was also examined using optical disks as the storage medium. Their large capacity, data retention characteristics, and durability made them the best choice for this issue. Blu-Ray optical disks were recommended because they presently provide the most storage capacity of all optical disk types. Software considerations included the operating system, the digital video recorder (DVR) application, and a backup and restore utility. In these cases, the Windows® OS was
  • 51. 51 recommended due to its widespread usage. No DVR application was suggested as this should be selected based on system requirements. At a minimum, it should provide the necessary recording function as well as post-mission playback and video editing utilities. With respect to the system backup operations, a reliable backup and restore application was also examined. Its operation should be transparent to the host system and provide automated, scheduled backups as deemed necessary. Full backups were recommended to provide complete repair of the system in case of file corruption or system failure. These scheduled backups should be placed in the secondary storage area (RAID assembly) to provide expedited system repair or rebuild as required. Following system setup, it was recommended that a master system disk be created on Blu-Ray optical disk to facilitate system repair or rebuild in case of catastrophic system failure or internal disk issues. Tactical video intended to be kept for historical purposes should also be copied to optical media to facilitate its later usage. These items should be kept in the on-site archive. It was also suggested that all recorded video be captured and stored on the external RAID assembly in the interest of keeping the primary storage area free of any extraneous files that might affect system operation. Proposals were made for software metrics that could provide for the automated retention or deletion of stored video in the interest of freeing up storage space. These metrics could use items such as the time/date stamp and the number of times a file has been accessed to determine retention or deletion of files from the video database. Other versions of this method might base retention and deletion on the actual nature of the video and its importance to the command. Different grades or levels of importance might be assigned to provide a clear distinction between video that is considered of tactical value and worth retaining and video that is deemed as
  • 52. 52 unimportant and requiring deletion. Some user intervention may still be required for these methods, but they could provide a basis for determining the importance of the collected video and its subsequent tactical value. Finally, future considerations were alluded to, in particular the use of metadata as an automated cataloging and indexing tool. The metadata technology is used to describe the background information attributed to the collected data. The use of this technology is accommodated in the MPEG-7 and MPEG-21 compression standards, which are currently in work by the Motion Pictures Experts Group (MPEG) organization. The Federal Geographic Data Committee (FGDC) provides the oversight regarding the use of metadata and is the current working body responsible for the standardization of the metadata technology. The guidelines developed by the FGDC are currently used in conjunction with the development of GIS applications. However, these same guidelines might be easily adapted for use on tactical video collection systems and may provide the means for the expeditious retrieval of video data sets that correspond to specific events, times, and locations. This makes it a potentially powerful intelligence tool and may provide a foundation for content-based video data retrieval in the future.
  • 53. 53 BIBLIOGRAPHY Bergman, Lawrence D., and Castelli, Vittorio. Image Databases: Search and Retrieval of Digital Imagery. New York: John Wiley & Sons, Inc., 2002. Christopher Drew. “NYT: Military deluged in intelligence from drones - Remote-controlled planes produced about 24 years’ worth of video in 2009.” New York Times, January 10, 2010, Sunday; http://www.msnbc.msn.com/id/34798080/ns/world_news- the_new_york_times/ (accessed January 11, 2010). Cornell University Library. “Moving Theory Into Practice: Digital Imaging Tutorial, Chapter 5, Metadata: Definition, Types, and Functions.” http://www.library.cornell.edu/preservation/tutorial/metadata/metadata-01.html (accessed January 14, 2010). Coughlin, Tom, Ekker, Neal, and Handy, Jim. “Solid State Storage 101: An introduction to Solid State Storage.” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009, http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf (accessed March 13, 2010). Crothers, Brook. “Seagate Enters Solid-State Drive Market.” CNet News, December 7, 2009, http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010). Daintith, John. "Magneto-Optic Storage." A Dictionary of Computing. 2004. Encyclopedia.com. http://www.encyclopedia.com/doc/1O11-magnetoopticstorage.html (accessed March 5, 2010). Effelsberg, Wolfgang and Steinmetz, Ralf. Video Compression Techniques. Netherlands: KoninKlijke Wöhrmann B.V., 1998. Encyclopedia.com. "Compact Disk." The Columbia Encyclopedia, Sixth Edition, 2008, http://www.encyclopedia.com/doc/1E1-compactd.html (accessed March 4, 2010). Encyclopedia.com. "Digital Versatile Disc." The Columbia Encyclopedia, Sixth Edition, 2008, http://www.encyclopedia.com/doc/1E1-DVD.html (accessed March 4, 2010). Encyclopedia.com. "Optical Disk." The Columbia Encyclopedia, Sixth Edition, 2008, http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010). Farley, Marc. Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems. Indiana: Cisco Press, 2007. Jensen, John R. Introductory Digital Image Processing: A remote Sensing Perspective. New Jersey: Prentice-Hall, 2005.
  • 54. 54 Judd, Josh. Principles of SAN Design: Design Build and Manage SANS. Pennsylvania: Infinity Publishing, 2007. Judd, Josh and O’Conner, Michael. Introducing File Area Networks Advance Edition: Your First Look at FAN Technology. Pennsylvania: Infinity Publishing, 2007. Luff, John. “Technology in Transition: Video storage.” Broadcast Engineering, http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed January 18, 2010). Luff, John. “Video Storage.” Broadcast Engineering, April 1, 2003, http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed January 17, 2010). Luff, John. “Video Storage.” Broadcast Engineering, July 1, 2004, http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed January 17, 2010). Mediatechnics Systems, Inc. "How a CD is Made." Mediatechnics Systems, Inc., FAQ, https://www.mediatechnics.com/cdfaqs.htm (accessed March 5, 2010). Mediatechnics Systems, Inc. "How are DVDs Made for use in Copiers?" Mediatechnics Systems, Inc., FAQ, https://www.mediatechnics.com/dvdcopiers.htm (accessed March 5, 2010). National Information Standards Organization. “Understanding Metadata.” (Maryland: NISO Press, 2004), http://www.niso.org/publications/press/UnderstandingMetadata.pdf (accessed January 20, 2010). Nelson, Dane and Siewart, PhD, Sam. “Solid State Drive Applications in Storage and Embedded Systems.” Intel Technology Journal, Volume 13, Issue 1, 2009, http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf (accessed March 2, 2010). Park, Oskar. “What is an Optical Disk?." Self SEO (Search Engine Optimization), September 10, 2006, http://www.selfseo.com/story-18894.php (accessed March 11, 2010). Schulz, Greg. Resilient Storage Networks: Designing Flexible Scalable Data Infrastructures. Massachusetts: Elsevier Digital Press, 2004. Shrivastava, Alok, and Somasundaram, G., eds. Information Storage and Management: Storing, Managing, and Protecting Digital Information. Indiana: Wiley Publishing Incorporated, 2009.
  • 55. 55 Simitci, Huseyin. Storage Network Performance Analysis. Indiana: Wiley Publishing, Inc., 2003. Symes, Peter. Digital Video Compression. New York: McGraw-Hill, 2004. Thatcher, Jonathan. “NAND Flash Solid State Storage Reliability and Data Integrity: An In- Depth Look.” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, 2009, http://www.snia.org/forums/sssi/knowledge/education/Solid_State_Storage_Reliability_a nd_Data_Integrity--An_In-depth_Look.pdf (accessed March 13, 2010). The Free Dictionary. “Flash Memory.” http://encyclopedia2.thefreedictionary.com/flash+memory (accessed March 2, 2010). The Free On-line Dictionary of Computing. "Average Seek Time." http://encyclopedia2.thefreedictionary.com/average+seek+time (accessed March 12 2010). The Oxford Essential Dictionary of the U.S. Military. “Tactical Operations.” Answers.com. The Oxford Essential Dictionary of the U.S. Military, Oxford University Press, 2001, 2002, http://www.answers.com/topic/tactical-operations (accessed March 07, 2010). Tudor, Marius. “Are Flash Solid-State Disk Drives Ready for the Enterprise?” Bit Micro Networks, Incorporated, 2009, http://www.bitmicro.com/press_resources_flash_ssd_enterprise.php (accessed March 2, 2010). Wikipedia contributors. "Audio Video Interleave." Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Audio_Video_Interleave&oldid=347331467 (accessed March 10, 2010). Wikipedia contributors. "Blu-ray Disc.” Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc&oldid=349002833 (accessed March 12, 2010). Wikipedia contributors. "Blu-ray Disc recordable." Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc_recordable&oldid=346269932 (accessed March 12, 2010). Wikipedia contributors. "Lossless data compression.” Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Lossless_data_compression&oldid=346709055 (accessed March 10, 2010). Wikipedia contributors. "Lossy compression.” Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Lossy_compression&oldid=347940008 (accessed March 10, 2010).
  • 56. 56 Wikipedia contributors. "Magneto-optical drive." Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Magneto-optical_drive&oldid=338693032 (accessed March 12, 2010).