5 THINGS: on Archive

1. What is Archive?

Ahh yes, Archive, the thing we rarely think about.

Archive is a lot like that high school yearbook you’ve got tucked way. You only look at it when someone from your past crawls out of the woodwork, and many memories of that time have faded.

The good part is that when you need it, it’s THERE. And that’s exactly what you want from an archive.  Archiving, for the media realm, is based on reliability.  It’s the bank safe and final resting place of your media, so nothing bad can ever, ever happen to it.  It’s not, I repeat, it is not on your SAN.  Or your NAS.  Or that locally attached storage. All of those are volatile and have fairly limited shelf lives.  Studies have shown that after 4 years, 20% of hard drives out in the wild are doorstop fodder, so we need to come up with an alternative solution.  Enter Archive.

Hard Drive Life Expectancy

Hard Drive Life Expectancy

2. Where does Archive fit in?

When we look at various storage theories that involve archive, we come across the concept of tiered storage. Tiered storage speaks to the relative importance of media combined with how fast you can access it.

While the lines in the sand of storage waver depending on your industry, we can draw some basic guidelines for folks working with video based media. Let’s discuss the various tiers of storage and how archive relates to the rest of your storage.

Tier I is your fast storage. This is your storage that must deliver the most throughput and speed. This is where availability is paramount over capacity. For VFX people this may be flash storage, for video people this is usually your DAS – Direct Attached Storage, or, on your SAN. Some companies will try to push NAS solutions for Tier I video solutions, which I believe to be a way to get cheaper solutions to unsuspecting folks, but that’s my tech cross to bear.

Tier I storage options

Tier I storage options

Next, we move to TIER II. Tier II is where you need capacity over availability. This is normally a holding pen for rarely used media, or after projects are done… but you have a decent chance of using the projects and media in the immediate future.

Tier II storage options

Tier II storage options

Often, film and TV projects use Tier II for the high res masters, while they edit with low res proxies off of TIER I; they then relink to the high res on TIER II for output. This is where NAS solutions usually fit in.

Traditional offline/online workflow for film and TV projects.

Traditional offline/online workflow for film and TV projects.

I’m also seeing facilities beginning to use LTO for TIER II, especially in reality television. Capacity is paramount in these scenarios with severely reduced daily usage.

An emerging offline/online workflow for film and TV projects using LTO.

An emerging offline/online workflow for film and TV projects using LTO.

Archive is commonly referred to as TIER III in most scenarios. In enterprise-based deployments, archive can be TIER IV or even TIER V. In either case, it’s the last stop for your media. This is where reliability is valued over availability or capacity.

Tier III storage options

Tier III storage options

Often, Archive takes the form of LTO, cloud storage, or optical storage. These formats need to have massive amounts of redundancy, and be less sensitive to the effects of father time. As you will most likely never access most of the content again, you need it somewhere where you don’t have to worry about it.

 

3. What Archive options are there?

As mentioned, Archive usually takes the form of LTO, Cloud, or Optical storage.

Cloud is a fancy-schmancy way of saying “someone else deals with it”. This usually means a 3rd party with a data center with many machines and a while lotta storage. So many of both, in fact, that if any machine or storage pool goes down, they have redundant systems to safeguard against data loss. Often, they call this the five 9’s, that is, 99.999% uptime.

This could be as simple as FTPing your media to a 3rd party webhost, or as involved as a managed archival platform running on Amazon S3 or another managed cloud provider.

Normally, the cost for storing the data is cheap, but the retrieval of that same data is the expensive part. Also, the paltry bandwidth you may have to your facility will greatly reduce the availability of your media. In many cases, that’s not a huge deal – after all, this is archival, where immediate availability is the lowest priority…but it is something to provision for if you’re looking at consolidating the tiered storage methodology.

Cloud storage considerations

Cloud storage considerations

Next we have LTO tape. If you’re unfamiliar, check out our last episode on LTO, tons of great info. In a nutshell, each LTO-7 tape can hold to 6TB – and over 2x that if the data is compressed. With read times from the tape approaching 300MB/s, and a shelf life of 15-30 years, LTO is a fantastic option. Single LTO-7 readers are under $5000, and autoloaders, multislot libraries, and robots are great workhorses if you’re serious about having an archive strategy.

LTO storage considerations

LTO storage considerations

Lastly, we have Optical. This is a way of using older blu-ray disc technology to store data. It’s kinda like that old CD changer you had in your car. It’s a magazine of BluRay disks that data can be housed on.

Sony ODA.

Sony ODA.

Sony, one of leaders in optical technology, have ODA – Optical Disc Archive. Gen 1 of ODA is shipping with 1.6TB cartridges, and read speeds approaching 140MB/s, with write speeds about half that.

Optical storage considerations.

Optical storage considerations.

Gen 2 of ODA, which is due to be released later this year is approaching double of every Gen 1 spec…at about the same price.

ODA Gen 1 vs Gen 2.

ODA Gen 1 vs Gen 2.

The bonus is that Optical solutions are random access –which means you can instantly start pulling the media back once the cartridge is loaded. This is different than LTO tape, where the drive needs to fast forward and rewind to find the data on the tape.

Data retrieval time of LTO vs Optical.

Data retrieval time of LTO vs Optical.

Lastly, as Sony claims, ODA discs last upwards of 50 years, and can actually function after being submerged in saltwater for several weeks.

ODA submerged in saltwater.

ODA submerged in saltwater.

ODA still works!

ODA still works!

 

4. How do I plan for Archive?

You may think Archive is a set it and forget it scenario. Copy it off, and move on. But oh no, there are things you need to do to ensure than just because you’ve got it, you can actually find it…if you ever need to.

Time isn’t kind to our memories. Out of sight, out of mind. Thus, having a way to catalog the media so you can find you put it is paramount. This can take the form of an archive format agnostic asset management, or, software specific to the medium you plan to backup to. Meaning, if you use Sony ODA for optical, you use the software that comes with ODA.

I’m a big fan of a agnostic asset management system, not only can it track and reconcile data on any of your Tiers of storage, but it can also automate the writing and retrieval of your archival material. Also, most asset management systems have pretty industry standard databases, so you still have the record of the archival data, even years down the road if any of your tech partners goes out of business.

Next, if you’re not using the cloud, where is your media physically being kept? On a shelf next to your TIER II, or housed offsite? Here in California, we never know when the big one will hit, so many folks with ship LTO backups out off state. Flooding and fires also don’t play well with storage. This is where safes become an option, or, again, contemplate moving your stuff offsite.

Burnt.

Burnt.

Now this can get expensive. This is where you may want to consider charging clients for archival. Offset your cost by making it a service. You can offer backing up not just the final cut and elements, but perhaps all of the originals as well.

5. How much does Archive cost?

Buying your own archive storage is priced differently than a cloud archive model, which is more a rental model or per month cost. So, I’ve averaged prices out over 4 years. After 4 years, 80% of hard drives out in the wild still function, so it’s a good ballpark number to compare from.

Cost comparison of various storage options.

Cost comparison of various storage options.

What this doesn’t take into consideration are the costs for the infrastructure to use these storage mediums. LTO drives can be under $5000, while the tapes are under $200. Optical systems are still newer, and start around $6500.

And while the initial outlay of thousands of dollars may seem steep, consider the cost of data retrieval from a cloud archive on top of the reoccurring monthly cost of a rental model, which will only rise as you archive more and more data.

Have more archive concerns other than just these 5 questions? Ask me in the Comments section. Also, please subscribe and share this tech goodness with the rest of your techy friends. Archive is a great party starter.

Until the next episode: learn more, do more – thanks for watching.

About The Author
- New (and useful) tech is über exciting. The logic behind the creativity. It’s why, by night, I do things like this; 5 of them in fact.  And by day, I'm the Director of Technology for Key Code Media, preaching the gospel on the manipulation of 1's and 0's. Oh, and I used to cut sound, too. Sometimes I get back in the chair.My heart and stomach are in Chicago, but the rest of me is in Los Angeles with my wife and dog. And a very marked up whiteboard.

9 Comments

  • Cooper Griggs
    Reply

    Are there any ODA providers that you would recommend in the Los Angeles area? I want to use that medium, but do not have the budget for a hardware solution at this time. Thanks!

  • Tim Jones
    Reply

    Your assumption on LTO recovery times is only accurate if the managing software does not use proper block addressing. Using BRU allows you to locate any file on tape in a couple of minutes at worst and then the retrieval time is on par. While ODA/ODM is random, it’s less cost effective, more expensive to get into, and far more limited on availability. A current LTO-6 tape (2.5TB native) is around $30.

    • Michael Kammes
      Reply

      I appreciate the input, even if it is a bit bias and advertise-y.

      Yes, ODA is much newer (e.g. limited availability), and the jury is still out if it will catch on – or if it’s a way to simply prolong an already dying physical format.

      LTO6 is old(er) technology. As you know, the backwards compatibility of an LTO format is limited, so what you save on an LTO tape or drive by buying old tech now may hose you years down the road. There is a reason I didn’t put LTO6 tape cost up against, lets say, a 1TB HDD…it’s not the current model.

      Thanks, Tim.

  • Chris Potter
    Reply

    Awesome article! I’m a little surprised at the relatively low cost for cloud archiving in your post. What cloud services were you looking at in your monthly price comparison? I thought AWS Glacier was $0.007 per GB, which is about $7.00 per TB / month.

  • Joyce Hart
    Reply

    Yes, the backwards compatibility of ODA is also what I worry about. As soon as the price starts coming down, a newer technology will replace it–or vice versa.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>