April, 2009

 
Backup Strategies Part 1.
A discussion of The CalDigit VR

By Brian Gary

The whole issue of data and media storage has become a minor obsession of mine and the need to deal with mass amounts of data is no longer relegated to corporate IT departments. Most of us working in post-production are dealing, if not on a daily basis, then on a weekly basis, with terabytes of data--most of it media content. Working with this data safely, defining clear backup strategies and ultimately archiving it, is quickly becoming a paramount issue for even the single editor with the single edit suite not just the post-production houses and studios. More and more we have to provide IT services as part of the editorial process.

When you add file-based workflows into the mix (meaning those projects that were not shot on tape or film and so therefore reside only on solid-state or platter-based media) you've added an entire level of urgency with regard to protecting that data. I've experienced this on more than one occasion. For example I was hired on as the editor for the independent feature film Frankie D when the first editor lost a day's worth of shot media. The movie was acquired on P2 and so the production footage came to the editor on hard drives. One of the hard drives failed and the producers soon found out that was the only copy of that particular day's production. A call to Drive Savers and $8,000 in recovery costs later, only a few clips were salvageable. Now, in the original editor's defense, P2 was a relatively new workflow, a hard drive was provided for editorial and no clear file-based post production process was ever fully established. In fact no one was managing the media because the workflow for a non-film, non-tape project wasn't planned out and it wasn't clear that anyone in particular was responsible for wrangling the media from the set to post production. When I came on board I enacted both an IT protocol and post-production strategy. Now that sounds a lot more involved than it was in practice because in reality all that meant was I ensured that every clip of production media existed in at least three places.

Which leads me to axiom number one, the prime directive of backup strategies:

You do not have data until you have data twice...preferable thrice.

In short, that means you have to have at the very minimum a mirror copy of your data and better yet another copy of it in another location offsite.

The original Frankie D workflow had the editor working on the only copy of the production media and all that precious data was housed on a LaCie Big Disk which is basically two hard drives striped together, which is the most precarious and unsafe of hard drive setups. So when the catastrophic mechanical failure occurred, most the data was gone for good. Which even if it's not your fault directly, does not bode well for job security: especially if you're the one who has to break the news to the producers.

As I said in the beginning I'll be talking about many different types of strategies during the course of these articles, but I've decided to start the discussion with a simple, accessible and affordable system: the CalDigit VR (MSRP $699 for the 2TB unit)

This is a two-drive hardware RAID that can be configured as either RAID 0 or 1. The drives are hot-swappable, but do require mounting in a tray versus some of the trayless models from WiebeTech. The entire VR line ships with drives pre-installed, but you can exchange drives at a later date if, for example, you want to increase the unit's storage capacity. The form factor and footprint are quite nice; in fact the unit is much smaller than I expected and as you can see from the picture it fits right in with Apple's design of the MacPro case.

Before I get into the unit and its usage, let's have a brief discussion of RAID (Redundant Array of Independent Disks), specifically RAID 0 and RAID 1:

RAID 0 (also referred to as a stripe) is taking two or more drives and running them in aggregate in terms of both speed and capacity, meaning that all the drives are treated as one big drive. The more drives in a RAID 0 the faster it will run. For example, if you have two 500GB drives and you place them in a RAID 0 set, the drive that will mount on your desktop will be a 1-terabyte drive (formatted capacity will be less). This drive set will run faster than a single 1-terabyte drive because you are harnessing the mechanical elements of two drives. The downside is that because the RAID controller is writing data across the two (or more) disks, if one of the drives fails you will lose ALL of the data in the RAID set. RAID 0 sets can be managed by both hardware controllers and software like Apple's Disk Utility or SoftRAID.

RAID 1 (also referred to as a mirror) is taking two or more drives and duplicating the media across all the drives in the set. For example, if you have two 500GB drives in a RAID 1 set, a 500GB drive will appear on your desktop (formatted capacity will be less) and all data that you write to the RAID 1 will be duplicated on the second 500GB drive in the RAID set. So, if you have ten 500GB drives in total in your RAID 1, you'll still only see a single 500GB drive on your desktop, but you'll have nine duplicate copies of the data on the other drives in the RAID 1 set. Therefore, data loss in a RAID 1 occurs only if every drive in the set fails simultaneously. The tradeoff to full redundancy is loss of aggregate speed and capacity. A RAID 1 will only have the total capacity of the smallest drive in the set.

As a point of clarification, all of this assumes data integrity, meaning that data corruption is different than data loss. In both RAID 1 and RAID 0 configurations, data can still exist, but it can be bad data because of errors during the write process.

CalDigit's VR is a hardware RAID that you can choose to set as either RAID 0 or RAID 1. The VR also has the ability to operate as a JBOD (Just a Bunch Of Disks) whereby the unit writes data to the first drive until it's full then starts writing to the second drive until that one is full. The VR ships with two drives pre-installed (the unit I received had two 1TB Hitachi Brand drives) and it is configured as a RAID 0. CalDigit's website lists other capacity and format configurations for the VR, so choose based on your storage needs.

The unit is easy to setup:

1. Take it out of the box.
2. Plug in the power.
3. Choose your connection.
4. Turn it on.

The VR has multiple hardware connection options and it also works with Windows and Linux systems.

As you can see the VR comes standard with eSATA, FW400 & 800 and USB 2.0 connections and it ships with eSATA and FW 800 cables. It also comes with an "eSATA Extender" which basically allows you to connect cables to the latent SATA ports on the MacPro's motherboard. These cables are then attached to a PCI slot adapter that has two eSATA connectors. I've done this surgery before and it is not a trivial task to install this option, so proceed with caution. The Extender PDF that ships with the VR describes the process for installing into a PC and that appears easier than the Mac install, but it's far simpler to add a regular eSATA PCI card; the one I used for testing the VR is the e2P from Sonnet.

Since RAID 0 is not a good backup or redundant workflow strategy I won't spend much time on it, but if your goal is raw speed here are some speed tests with the unit empty and configured as a RAID 0 (I used AJA's System Test Utility for benchmarking):

eSATA connection

FW800 connection

FW400 connection

USB 2.0 connection

My focus for using the CalDigit VR was not about speed (I would choose a RAID-5 setup for that), but more so about its redundant storage capabilities. My goal is to have an instant copy of everything I write to the unit; I want to have the data twice.

The first order of business is to switch the unit from a RAID 0 to a RAID 1 configuration. Now, it's important to note that any change in RAID setup with this unit will result in total loss of all data on the drives. That's because the unit has to reformat the drives into the new RAID configuration. So take great care when changing the unit's setup.

With that admonition firmly in place, the VR's hardware RAID controller is quite easy to use. You can either use the front panel display:

Or the bundled software:

The unit ships with the software and PDF manual on CDROM and when I checked the application installer using Pacifist it showed that only the program itself installed, no other drivers etc. CalDigit's website also has tutorial movies explaining the use of both the front panel menus and the bundled software. Either way you go, it's quite simple.

For example here are the steps to change the RAID set via software:

1. Launch the CalDigit RAID Tool

2, Select the VR:

3. Choose the RAID Configuration Button:

4. Click RAID 1 and heed the warnings:

5. After a minute or so of internal processing the VR will come back online. CalDigit recommends that you restart your computer during that time.

6. The OS will alert you that the new drive is unreadable and so you'll have to format it using disk utility.

If you use the front panel to configure the VR you'll go through a series of menus and options to chose the new RAID configuration. There are multiple steps that prompt you to be sure you want to erase the box and reconfigure it as a different RAID set.

The first thing you'll notice after formatting is that the drive is no longer 1.8TB (formatted and empty) it's approximately 930GB formatted and empty. Everything you now copy onto this unit will be simultaneously written to two disks, you just don't see the second disk mounted. As I wrote earlier, you are sacrificing both capacity and speed with a RAID 1 configuration. Fortunately, the hardware RAID controller does a good job managing the data flow, especially in the read department. Here are some speed results with a FW 800 connection.

The VR was completely empty during these tests and will inevitably slow (as all RAIDs do) as the unit reaches full capacity.

The CalDigit RAID Tool also provides information regarding the installed drives:

And the status of the VR's fans and heat sensors:

You can then use the drive as normal and in the event of a hardware based drive failure, you just have to swap out the bad drive and the VR will rebuild the Mirror. The unit comes with two "keys" that you use to open the back drive doors:

When removing a bad drive make sure to replace it with a drive of the same capacity and preferably the same manufacturer.

In conclusion, the CalDigit VR is a very good RAID 1 (mirror) solution in both form and function. It offers ease of use with both its hardware and software interfaces and I really like how the front panel displays the unit's current state and status. The price point of the VR is also quite compelling considering it has a hardware RAID controller and ships with two drives (price varies with capacity configuration).

With that said, is the CalDigit VR (or any RAID 1 system for that matter) the perfect backup solution. The answer is no, because RAID 1 technology can only partially protect against data loss in a redundant workflow. Although unlikely, both drives in the VR could fail and result in data loss. There could be a flood or fire that destroys the box. The unit could be stolen: all those possible force majeure eventualities, however uncommon or unlikely, could possibly happen.

The only realistic safeguard against total data loss is an offsite copy along with local redundant storage (like CalDigit's VR) that's configured as a RAID 1 set...thereby ensuring you have the data thrice.