Post-Production: A Direct Network Storage Solution for Editing & Color Grading (Part 2)

By David Weldon

Recently, we introduced the tools you are going to need to build out your Direct Network Storage Solution. As Shane would say, now we are going to “dive right into the fire, head first!” Haha. We’re going to walk through how we built our system here at Hurlbut Visuals.

We had some much-needed help in this process. I want to make sure that I make you aware that this is not a “plug-and-play” type of setup. Shane called an old friend, Mike McCarthy of Bandito Brothers here in Los Angeles. Mike is a Technical Engineer, Guru, Savant. You name it, this man can do it. He’s brilliant. Mike also set up a similar system for David Fincher building out Fincher’s editing team with HP Z840 workstations and a Direct Network setup.

I would highly recommend enlisting someone like Mike McCarthy in your area, someone you know or know of who is extremely proficient in this area of technology. Mike has a great blog himself on the technical work of computers, networking, and a lot more. Check it out: HD4PC.

Our Objectives of Creating this System

Many of you may be familiar with the terms NAS (Network Attached Storage) or SAN (Shared Attached Network). These are two setups that are relatively similar to each other, and they essentially allow for multiple editors or colorists to connect to the same area where the footage they need is located. Thus it is referred to as the connection being “shared.”

With a NAS or SAN, you would use a network switch like this one to connect multiple edit bays or color bays to work off the same system. Your NAS or SAN would connect into the switch, and then the individual computers would connect to the switch, completing the connection.

Depending on what type of system you get and what type of network cards you put in your computers, this can get costly. The switch we are showing here is already putting an extra $3,000 into your budget, but you may not even need it depending on the size of your operation.

Our main goal is to create a system that is cost effective to our needs while still delivering power and performance across the network to each edit/color bay.

Building via 10GbE Direct Attached Storage

If we were to build our system via the NAS or SAN route, we would end up spending additional money and resources on:

• Free standing storage controllers and storage arrays
• Fiber switches and sards
• Block level SAN sharing software

With a 10GbE Direct Attached system, we are going to accomplish the same goal but using the standard sharing technology that is already available to us within Windows or Mac OSX.

Here is what has to be done:

Step 1: Install your 10GbE Network Cards

You’ve purchased your 10GbE Network Cards off of eBay and you’re ready to install them.

Pop open the side of your HP Z840 Workstation by opening the handle towards the top.

Take out the ventilation system to reveal the PCI slots.

Locate one of the x16 lanes, which will be a preferred lane to use. A “lane” is built out of two different signal pairs, one pair for receiving data and the other pair for transmitting data. Each lane transports data packets in eight-bit “byte” formats at the same time. Imagine a highway with multiple lanes going in different directions, only the term “lane” would mean “highway.” You could also imagine your internet connection. The downstream and upstream use the same internet connection but two different sides of the connection.

Go ahead and install the card.

Because you are using 10GbE cards, this is where the Cat7 cable comes into play. You are going to connect the first computer to the second computer via the 10GbE. If you add more than two computers in the chain, you will have to make a decision on how to link them, which we will get to later.

Step 2: Setting up your Storage Arrays – G-Speed Studio XL

If you are building a system around Windows and using HP Z840 Workstations like we are, but you want to keep the option open to integrate a few Mac computers, it’s not going to be a problem. You will format your G-Speed Studio XL to Windows NTFS so that the Windows computer that will be the direct connection (via thunderbolt) will see the array. Once you share the array across the network, each computer will just dial in via a network connection and it will not matter how the drive is formatted.

NTFS is going to be the most stable and secure for Windows. Mac OSX Journaled will be the most stable and secure for a Mac only setup. We oftentimes format our G-Raids in ExFat so that both Windows and Mac computers can see the drive, but that isn’t necessary here.

G-Technology Step-by-Step Breakdown of Reformatting for Windows

Once you have formatted the G-Speed Studio XL into your HP Workstation Z840, you will install and launch the G-Speed Studio Software (You may have to download the software). Within the software, you will have the ability to check on the status of your array. You’re going to want to monitor the drives for heat issues and just overall performance as well.

Step 3: Networking Your Array

Each direct connection is a completely different network. Each needs its own subnet for the system to know which traffic to send to which physical port (the 10GbE cards in each computer).

Normally your main network for internet is set up at 192.168.0.xxx or 192.168.1.xxx. To keep it simple, Mike set it up at 192.168.10.xxx for the 10GbE links. But if you are going to add a second array (like we did) you will need to add to the chain, so it will need to be 192.168.20.xxx or .30.xxx or whatever you choose.

Now each 10GbE port on the cards you installed needs to have a unique static IP address assigned to it. This means that this address will never change. It will be a unique link to that individual system.

Mike chose to keep the last number of the address consistent for all of the IPs on a single system to keep things simpler to remember.

Maintain Consistency

When you are mapping across the network on Windows, it is helpful to use a consistent letter across the system for any given array volume. Mike chose to start us with Z: and then Y: for our second system, working backwards, so that there would never be a conflict. Normally when you plug in a drive, Windows will automatically generate a drive letter and it will start from A and work to B, etc. on an automatic basis. Unless you plan on plugging in 25 other drives at the same time, Z and Y are a good starting point. You do this by going into the Disk Manager on the system that has the drives physically attached to them (via the thunderbolt connection).

By mapping the same drive letter across the network, allows programs like DaVinci Resolve or Adobe Premiere to open projects on any system and the media files will always be found on the same network path.

You will map the shared storage using the IP address instead of the system name to force traffic to go over the 10GbE link. For example, “192.168.10.101BigArray” to see the drive on the second edit/color system. If you choose to have a third system in the setup, your path will be something like this – “192.168.20.101BigArray” due to the subnet of the 10GbE link.

Dialing in the Network Settings

Now you are all physically connected, which is the easy part. Now we have to dive into setting up the system as far as IP configurations and making sure that all of the labeling is correct. Let’s go through this step-by-step.

1: Go to the Control Panel

2: Click on Network and Sharing Center

3: Select Change Adapter settings on the left

4: Right Click and Select Properties on the adapter you want to set.

This will be either the 10GbE port or the Gigabit port you set. Since the cards we purchased have more than one 10GbE port, make sure you are selecting the correct port.

5: Select Internet Protocol Version4 (TCP/IP V4) and hit Properties

6: Set the IP address in the window that pops up.

In this case, it is 192.168.10.102. This is the second computer on our system, running through the 10GbE card.

Now you can follow the same steps if you plan to use a Gigabit network setup. We ran both the 10GbE and a Gigabit network so that we could run the Gigabit through our current Wireless Router, and this way we could have other computers connect via Cat5 Ethernet, allowing access to the server and being able to see all of the files.

The Gigabit connection is recommended for anything 1080p and below.

The 10GbE connection is recommended for anything 5K resolution and below. We are optimizing our 5K playback of RED footage using RED Rocket-X cards as well. Otherwise, we would have to lower the resolution to the lowest possible and even then, it may be difficult to playback.

I want to stress that while this seems simple by following these directions, you may run into little bugs and issues with networking connections here and there. We had a little trouble in the beginning because we also used our Gigabit connection to send to our router that connects to all of the other Gigabit ports in our office. This way we could plug in laptops or other computers, as I mentioned previously, but because this connection is going through the internet, your Windows network may give you a little trouble. It is really based on a case-by-case basis. This is why I highly recommend having someone help you setup and build this out. For us, that was Mike McCarthy of Bandito Brothers.

There are many ways to share your storage and connect within your office. For a small business setup looking to move efficiently and effectively, this is a solid way to go. Obviously, when you get bigger, you can expand your storage and add a larger 10GbE card on the main system to connect to other systems. Once you get beyond eight to ten users, you may want to look into the SAN solution process.

One Last Thing

There is one last configuration idea that may help speed up your process if you choose to. If you build your system with two G-Speed Studio XL Arrays, you can increase performance by splitting them up. What I mean is:

Your Edit/Color Bay #1 has 1 G-Speed Studio XL connected via thunderbolt

Your Edit/Color Bay #2 has 1 G-Speed Studio XL connected via thunderbolt

You then connect those two computers via a 10GbE connection. This will allow you to export and render across the network. We opted not to go for this setup because as you add computers, it can make it a bit more cumbersome to make sure it is all connected properly. We tried to build a system that is simple to understand so that it is easy for someone else to see how it all connects. This is why we have both of our G-Speed Studio XL Arrays connected to the same computer and then we send the connections from there.

Previous articles on this subject:

Foundations for Workflow, File Management and Post Production >>

Creating a Direct Network Storage Solution for Post Production Editing and Color Correction: Part 1 >>

 

Technical Specs: