Petabytes are the New Terabytes

Digital storage plays an enormous role in the creation, distribution and use of rich media content.  In addition, higher resolution content places increased demands on content archiving.  The 2013 Storage Visions Conference (SV 2013, www.storagevisions.com), a partner event to the 2013 CES in Las Vegas, taking place January 6 & 7, 2013 will explore the implications to digital storage of all elements of the content value chain.  The conference theme, based upon the growth in storage demand to meet rich media requirements, is “Petabytes are the New Terabytes.”  This article will explore trends driving the storage requirements for rich media that will be among the topics discussed at SV 2013.  Much of the content here is extracted from the 2012 Digital Storage for Media and Entertainment Report from Coughlin Associates (http://www.tomcoughlin.com/techpapers.htm).

A Never Ending Cycle

The increase in the availability of digital entertainment content and the rise of more types of distribution networks increases the value of digital storage for the consumer.  Altogether, the creation, distribution and reception of digital entertainment comprise a positive feedback loop as represented in Figure 1.  This positive feedback loop is amplified by the effect of increasing units of each element of this value chain and maturity of product design in reducing the cost of these products and thus lowering the barriers to entry for digital content creation. 

Figure 1.  Digital Entertainment Content Value Chain (An Accelerating Positive Feedback Loop)

Storage and bandwidth for raw video content are increasing with video resolution as shown in Table 1.

With higher frame rate and multiple video cameras capture becoming more common the total requirements for storage capacity and system bandwidth can be very great. 

Making Content/Storing Bytes

Once content is captured it must be processed and edited to make a finished video project.  One of the biggest elements in this post production work is non-linear editing.  Non-linear editing (NLE) is generally done from uncompressed or at most slightly compressed source content since heavy compression can lead to loss of resolution and can cause timing problems.  However the actual editing may use lower resolution proxies with the editing changes incorporated into the uncompressed content after editing is done.  Figure 2 is a schematic of a non-linear editing station showing optional connection to shared on-line storage via a SAN or NAS and a host bus adapter (HBA) and indicates that content may also be accessed from remote storage through the Internet (the cloud).

Figure 2.  Professional Non-Linear Editing Model System

For a large enough facility with several editing chairs, shared storage allows the local storage (known as Direct Attached Storage or DAS) to be kept at roughly 30 minutes per station.  Storage networks are used in all the NLE market segments but are more likely for the higher end market segments.  Video SANs today may still use Fibre Channel storage networks due to their higher performance and lower overhead.  However, with higher performance of IP-based networks, more SANS in post-production environments are starting to use these less expensive options.  An extension of IP networks is to use some resources through the Internet (the cloud) for some post-production processes or to enable collaborative workflows.  IP or Ethernet networks also enable direct file access to content stored in a (NAS) system.

Direct attached storage can be a faster option than some storage networks since modern DAS interfaces such as Thunderbolt, eSATA and USB 3.0 (or native Fibre Channel and Serial Attached SCSI) allow very fast data access (5 or 6 Gbps for USB and eSATA, 12 Gbps for SAS and up to 16 Gbps for Fibre Channel).   Intel’s Thunderbolt (formerly known as Light Peak) connections on Mac Pro notebook computers and in 2012 on many other computers provides DAS speeds up to 10 Gbps.  Thunderbolt allows storage and peripheral access much closer to the computer processor through the high speed PCIe bus.  This is the main reason for the faster connection speeds using Thunderbolt.  The current generation of Thunderbolt connectors uses copper wires as the physical transport while future versions may use optical fibers and allow data transfer speeds up to 100 Gbps.  Note that Thunderbolt allows full speed two way communication and can daisy chain up to 7 devices, including a display port display.

Modern Ethernet and Fibre Channel storage networks (single cable) can deliver data rates as high as DAS devices with these technologies but the costs have been considerably higher and thus these costs are best spread among multiple users.  However, these connectivity costs may be about to plummet, especially for 10 GE networks.  At the 2012 NAB Show Arista Networks showcased a 10 GbE router using copper Category 6 cables that supported 10GbE speed for up to 100 m (optical cable allows these speed up to 300 m).  Using Cat 6 cables reduces per port routing and cabling costs to about a quarter of the optical cable equivalents.  100 m is far enough for many data center connections and thus these new Cat 6-based 10 GE IP networks could make network storage much more affordable and further increase the growth of shared storage. NetApp and EMC/Isilon as well as many other companies specializing in this industry provide network storage for post production work.

Figure 3 plots the annual demand in storage capacity for post-production, including NLE, breaking out direct attached and network attached post-production storage capacity.  In this edition of the report are including for the first time a separate breakout of remote storage for collaborative workflows.  We assume that this “cloud storage” is a part of the projected near-line storage.

Note that with increasing use of network storage in NLE (especially for multi-seat facilities as shown above) new DAS storage requirements will experience slower growth.  Total storage capacity shown in the tables and figures includes previously acquired storage capacity that is still in service.

Figure 3.  Post Production Storage Capacity Annual Demand (TB)

Summary and Conclusions

As the depth and immersive quality of rich media increases the storage capacity required to complete there projects and the bandwidth required for moving the raw content around increases.  An increase in resolution in one aspect of the content value chain drives resolution (and thus storage capacity and bandwidth) in all other areas.  Today 2K and some 4K content is common but in the future common content may be at 8K X 4K pixel resolution or higher.

Editing and preparing this raw content is requiring an increasingly complex hierarchy of storage technologies.  Modern non-linear editing may use local buffers and HDD storage but they may also use network storage, particularly in larger facilities.  The need to create tools for more flexible collaboration on video projects that may span multiple locations and time zones has given rise to cloud-based content distribution services that supplement NLE facilities.

The growth in content complexity will drive higher storage demand for rich media projects including direct attached storage but increasingly network attached storage and even Internet-accessed cloud-based storage as the speed of these solutions increases and the costs go down.  A project that used to require terabytes of storage may now require petabytes at a higher resolution and frame rate.  Truly petabytes are the new terabytes.