by Martin Bailey | Apr 26, 2017 | Fundamentals, Podcast, Tutorial
It’s been two years since I explained my image management strategy as a traveling photographer, and I’m finding myself explaining what’s changed a lot in email conversations, so today, I’m going to walk through this with you again, and update you on the changes I’ve made.
First of all, allow me to explain the problems that I’m overcoming with my workflow, so that this all makes sense as we work through my thinking.
Problems to Overcome
One question I get asked about a lot, and why I often send people to this post, is how do I move smoothly between computers when I get back from a trip. People tend to make the process of getting images from a trip back into their main library a very painful process.
The good news is, if you build your workflow around the premise that you will travel, you don’t have to do anything special. I’ll go into details shortly, but basically I have to click one button when I get home after a trip, to initiate a backup of my images, and I’m done. In fact, I have to click that same button whenever I go to my desktop computer, so nothing changes. I literally transition between my laptop and desktop computer with zero effort, as I’ll explain.
Another problem people often come up against, is keeping track of what is backed up to where. I’ve found that it’s very important to decide which hard disk contains your working data, and which hard disks are just a backup. If you work on images in separate locations it soon becomes a real pain to keep them synchronized, so we build this into our strategy.
Finally, I think it’s vitally important that we have a multiple backups of our precious photos at home, as well as a copy in the cloud. Having everything under one roof could be a recipe for disaster, if indeed, disaster should strike. Should something catastrophic happen to your house or business premises containing all of your local backups, having the ability to contact someone to receive a backup of all your data could be the only possible way to rebuild your image library, as you rebuild your life.
Same Strategy, Different Software
One other major change over the last few years, is that I’m now using Phase One’s Capture One Pro as my raw processing and image management software. The details regarding what I did in Lightroom are still in the original post, so you can certainly still reference that post for details, and as you’ll see, most of what we’ll cover doesn’t really change depending on the software you are using in your workflow.
Move Catalog Drive Rather Than Synching Computers
The cornerstone of our digital workflow is our image catalog, or now that I’m using Capture One Pro, catalogs, in the plural. I have more than one now unfortunately. But, I’ve found that keeping track of multiple catalogs and keeping them backed up has not been a problem.
The important thing is that I keep my catalog on an external hard drive, and this needs to be relatively fast. A USB 3.0 hard drive generally won’t cut it. I’ve actually changed my hard drive twice since my fist post. I used a Drobo Mini over Thunderbolt, and then 4TB Western Digital Thunderbolt drive a long time, but as I bought a new MacBook Pro with USB-C ports at the end of last year, I picked up a Sandisk Extreme 900 portable SSD drive, and have been very happy with it. These are expensive drives though, currently retailing at $787 on B&H.
These external SSD drives over USB-C 3.1 Gen2 are incredibly fast though, and remove any and all stress related to running your image catalog and images on an external drive. The Western Digital thunderbolt drives were fast too, but nowhere near as fast as these SSD drives. The Extreme 900 comes with both a USB-C to USB-C cable and a USB 3.0 Type-A to USB-C cable, so I can plug it straight into either my new MacBook Pro or my older iMac. The speed is actually pretty respectable over USB 3.0 on my iMac as well. The problem with other USB 3.0 drives is that the 2.5 inch hard drives are slow, but that isn’t the case with SSD.
The downside, is that the largest available volume at this point in time is 1.92TB, so I had to rethink a few things. I can just about fit my Finals and current year of images on this drive, although it will be tight. I’ll explain this in more detail shortly, but the important thing to note here, is that I have all of the work that is important to work on at the current time on this drive, and I run my Capture One Pro catalogs from this drive. When I move computers, I simply unplug the drive from one, and plug it into the other. When I reopen Capture One, I’m taken right back to the location that I left off when I closed the catalog on the other computer.
Catalog Strategy
When I moved to Capture One, I found that it couldn’t handle all of my images in a single catalog, so I split my images into multiple catalogs. Each year of images has its own catalog, except for the first six years, from 2000 to 2005, because I didn’t have that much work, and so I was able to fit this all into a single catalog.
So, I currently have one catalog called 2000-2005, and separate catalogs for each year from 2006 onwards. These year catalogs contain every image that I shot for each year. I do all of my initial editing and image rating in these catalogs, until I have finalized my selection. Once I have finalized my selection, I copy my images to another catalog, called Finals. This catalog contains a separate folder for each year. I also copy the physical images to a Finals folder with one subfolder for each year. I’ll cover this in more detail later.
Catalog List in Capture One Pro
Although I wasn’t happy about having to split up my catalog initially, in practice, it hasn’t been that bad. You can easily get to each catalog from a pull down in Capture One Pro (right) and I have all of the photos that are worth a hoot in my Finals catalog anyway, so most of the time I tend to flick between my Finals and the current year catalogs.
Master and Backup Copies
As I mentioned, I learned from experience that it’s really better to avoid having multiple copies of folders and catalogs that you work on, so it’s really important to decide where you are going to put your catalogs and folders of images, and decide which on is the master, and which ones are just for backup purposes.
If you work on a copy of your image library on one computer, and then work on a different copy of your image library on another computer, at some point you are going to wonder which copy is the most recent, and you’ll have forgotten. Even when using software such as ChronoSynch that we’ll look at later, which has the ability to synchronize the latest files between locations, there will come a point when you have two copies of the same file that have both been worked on, and when you select one copy, you throw away what you did to the earlier copy, or you keep both copies and that’s avoidable, so I prefer not to.
My Master and Backup Copy Strategy
For me, I’ve found it best to have my current years worth of images and all of my Finals on my Sandisk Extreme 900 SSD drive, and I have all previous years on a Drobo 5D, which is attached to my iMac in my studio. I never need to access my original photos from previous years while traveling, and I because I do travel with all of my Final selects, I can get to those if necessary.
Let’s map this out and start to visualize my strategy, starting with my desktop computer, in the studio. When I have my Sandisk Extreme SSD drive, which I call Traveller, attached to my iMac, I can see, open and edit every image I’ve ever photographed. We’ll build this out like a presentation slide deck, so excuse the blank space on the right side of Diagram #1 (below).
Studio Workflow and Backup Strategy Diagram #1
At it’s bare minimum, my workflow starts with shooting images, and transferring them to my Traveller drive. This drive contains the master copy of all of the current year’s images and all of my Finals, which is every photo I’ve ever made that I consider good enough to use, and my current year and Finals Capture One catalogs. These things all live on my Traveller drive, so that I can easily move this to my laptop, as we’ll see shortly.
Automatic Backups
I use an application called ChronoSync from Econ Technologies to synchronize my files and catalogs around. I used Robocopy when I was on Windows, but it’s not all that intuitive. Other Windows applications that were suggested following the last post I did on this are SyncBack and GoodSync, which is multi-platform.
With ChronoSync you can create synchronization jobs and bundle them together, and schedule a batch of jobs to run whenever a specific drive is attached to a computer. Let’s walk through this
I have two sync jobs that mirror my current year and my Finals folders to my Drobo. Here is a screenshot of my current year sync job (below) and this is simply going to copy everything new in my 2017 folder on my Traveller SSD to a 2017 folder in a folder called Photo Originals on my Drobo.
ChronoSync Job
Using the Mirror option will also delete anything that I have deleted from my Traveller. This is important, because as I remove images from my main copy, I don’t want to leave them in my backup. I also create a Rule to not copy the hidden .DS_Store files to my Drobo. They are specific to each drive, so I don’t want them to be included.
I have a similar job to Mirror my entire Finals folder to my Drobo as well. I don’t Mirror just the current year of my Finals, because as I work on images, I sometimes change images from previous years, so I want to keep this all synchronized with my Drobo.
I also have two special jobs that synchronize only things that have changed inside my Finals and current year Capture One catalogs. To do this, turn on “Allow package file selection” when you are locating the drive and folder to synchronize, and then ChronoSync will treat the package files as a folder, and synchronize the contents.
Sync Package Contents
If you don’t do this, ChronoSync will synchronize the entire package, and that would cause a very large file to be copied to my Drobo every time I sync, and it would cause the entire catalog to be unnecessarily backed up to the cloud every time I sync.
Group Jobs Together in a Container
Once I have all of my sync jobs created, I wrap them up in what’s called a Container, as we can see in this screenshot (below). After adding all of the relevant jobs to a container, you can click the Add to Schedule button and schedule these jobs to run automatically.
ChronoSync Container
You can schedule sync jobs based on various actions, or simply have them run at a set time each day, but for this purpose, I select to run the job “When An Independent Volume Mounts” and this enables me to select my Traveller drive (below). I also select “Prompt user before running”. I want to be prompted, because I don’t necessarily want or need to synchronize my drive every time I plug it in to my iMac.
Schedule Sync Jobs
Once I have this set up, whenever I plug my Traveller drive into my iMac, I see a little popup like this (below) that asks me if I want to synchronize my Traveller with my iMac.
ChronoSync Popup
So, although it takes a little bit of time to set up, I can now with one click automatically backup my images and catalogs to my computer. Let’s continue to build out the slides to check where we are in our backup strategy (below).
Studio Workflow and Backup Strategy Diagram #2
We now see that as soon as I attach my Traveller drive to my iMac, my catalogs are automatically backed up to my iMac. I keep a backup of my catalogs on my iMac hard drive for a number of reasons. The first is because if I put them on the Drobo, it would take me longer to backup my Drobo, because I’d need to create separate ChronoSync sync jobs to avoid copy entire catalogs, as I mentioned earlier. The second reason for doing this is that I also set up Time Machine to backup my iMac drive, so that would be an easy way to get back to a working copy if anything went wrong.
Cloud Backup
We can also see from this diagram that as soon as any new images are copied to my Drobo, they are automatically backed up to my Backblaze account. Backblaze has been great, and for just $50 per year, you can get unlimited storage in the cloud. If I ever had some kind of catastrophic disaster that took out all of my local copies of my images, I could have Backblaze send me hard drives with my 12TB of data on them, and I’d be back up and running in no time.
Location of Files
Let’s also recap on where everything is now. The master copy of my current year’s work and my Finals library of images, and the working catalog for this work is all on my Traveller SSD. The master copy of all of my previous years work is sitting on my Drobo, and whenever I reference these images, I launch the catalogs from my iMac hard drive.
An easy way to look at this, is everything from previous years is based in my studio, on my desktop workstation. Everything that travels with me, is on my Traveller drive.
Local Fault Tolerance
Let’s move on to look at one last slide from my studio setup (below). Although my Drobo has fault tolerance built in, and I can have one drive fail without losing my data, there is always a slim but real chance that more than one drive dies at the same time, or that the entire device could die on me. Because I don’t want to rely on Backblaze sending me my work on hard drives just for a drive failure, I actually have a second Drobo 5D, which is a straight mirror of my first.
Studio Workflow and Backup Strategy Diagram #3
I have a reminder scheduled on my computer to remind me to turn on my second Drobo once a week, and mirror my first Drobo to it. I use ChronoSync for this as well. When I set up my first Drobo 5D I used drives that turned out to be very noisy, so I demoted that Drobo to the backup, and have bought quieter Western Digital Red drives for the main Drobo, as that’s turned on most of the time. They also use quite a lot of power running five 3.5 inch hard drives, so I only turn on the second one when necessary.
I know that some people have had bad experiences with Drobo drives, but I have been very happy with mine, and have never had any problems. But, technology does fail, so I just don’t want to have my main copy of all of my work to exist in just one place locally.
Mirroring Entire Drives with ChronoSync
To close the loop on the last diagram before we move on, please note that to mirror the contents of my first Drobo 5D to my second with ChronoSync Task, because we will mirror the root of the drive, I set up a few Rules to prevent ChronoSync from copying and overwriting some important system files, as we can see in this screenshot (below).
Sync Drobo #1 to Drobo #2
Let’s Get Mobile
OK, so now let’s move on and look at what happens when I’m traveling, or simply working away from my studio. As you can see, I just plug in my Traveller SSD drive, and continue working. If I had actually closed down Capture One Pro on my iMac with that Snow Monkey photo displayed, it would have opened at the same location when I move to my MacBook Pro.
Travel Workflow and Backup Strategy
I also travel with three USB 3.0 hard drives. These are too slow to run my catalogs and images from, but as backups they work fine. One is just a Time Machine backup, that I plug in at the hotel every few days, usually over night, if I can leave my MacBook Pro plugged in to the electricity.
Carry Spare Backup Drives
I then have two backups, which I once again automatically synchronize with ChronoSync. I have a schedule set up to detect each drive as it is attached to my computer, and it asks me if I want to mirror my images and catalogs to the hard drive. I make two backups, simply because one drive could fail. Actually, all of my drives could fail, but having spent many months on the road, over the years, I’ve actually had just one drive fail on me. That was in Antarctica though, and believe me, when there are no shops around, having a spare is very important.
It’s a little nerve racking to only have one master copy and one backup of my images while traveling, but if my second backup drive was to fail, I’d kill my Time Machine backup, and continue to make a backup of my images. Thinking of it this way, carrying these three drives is more to give me backup drives, than an actual backup of my images, but as it’s all automatic once I’ve plugged the drive in, I just keep them up to date at the end of each day.
Keep Your Copies Separated
One other important aspect of having these backups, is that I feel it’s very important to keep these separate as you travel. I always travel with a photographer vest, and keep my master copy in my vest, on my person, at all times. Even when I go to the bath when traveling domestically here in Japan, I take my Traveller drive with me and put it in a locker.
Having three backups of your images isn’t going to help you at all if they are all in the same bag, and you lose that bag. I generally keep one backup in my bag, and the second in my suit case. These means something would have to happen to all three copies in separate locations for me to lose my entire library of images while traveling.
Turn Off Cloud Backups While Traveling, Please!!
I don’t do cloud backups while traveling, partly because I only pay for one computer on Backblaze, and it’s better to make that my iMac as it’s always on and connected to the Internet. Also, hotel Wifi is usually not good enough to bear up to uploading large numbers of raw files. Many people have automatic backups turned on now, and you can literally watch the network go down as a bus load of photographers get to their rooms after a days shooting. I wish more people would turn this off while on the road.
Synching Settings Files
One other thing that I need to mention before we move on, is that to make moving between computers totally seamless, I also synchronize my Capture One Pro settings folder by moving it to my Dropbox, and creating a symbolic link in the original location. This isn’t officially supported by Phase One, but I’ve been working this way for 10 months now, and haven’t found any problems.
Here is the code I use with my name replaced by USER_NAME. This assumes that you’ve moved the “Capture One” preferences folder under the “Application Support” folder to a folder called “Capture One Prefs” inside a folder called “Capture One” in your Dropbox. This code only works on a Mac, and please do this at your own risk.
ln -s "/Users/USER_NAME/Dropbox/Capture One/Capture One Prefs" "/Users/USER_NAME/Library/Application Support/Capture One"
You of course have to do this on all computers that you will work on, to ensure that your preferences are copied between each via your Dropbox. If you don’t know how to create a symbolic link in Windows, this tutorial will probably help.
Set a Hard Drive Letter for Windows
If you use this strategy in a Windows environment, you’ll probably also need to ensure that the drive letter of your Traveller drive doesn’t change as you move it from computer to computer. Here’s another tutorial on how to do that. Just ensure that you select the same letter on all computers you work on. Give yourself some room too, so that you can still have lots of dynamically lettered drives on your desktop. T for Traveller would be a good option.
Not Really a Cross Plastform Solution
I should also mention that this solution may not ideal if you switch between Windows and Mac regularly. The catalog can be taken from one operating system to the other and will open, but Windows and the Mac OS reference drives differently, so you’d need to tell the other OS where your files live each time you open the catalog on the other system.
Voilla!
Once you have all of this in place, you will literally be able to move your Traveller hard drive from computer to computer, and continue working as though you were on the same computer. Because you have your Capture One Pro settings syncing too, even all of your presets are available on both computers. They essentially become identical.
Exporting Original Format Images
Export Original Images to Finals
Let’s start to wrap up now, with a few other pieces of advice based on my own workflow.
To get my final select images from my original photo folders to my Finals folder I select the images that I want to export and right click one of the thumbnails, and from the shortcut menu, select Export > Originals. You can also get to this option from the File menu.
I don’t change the image name on export, because I change it on import. After checking the destination, I ensure that Include Adjustments is turned on, then click the Export button, as you see in this screenshot (right).
I don’t package my images as EIP or Enhanced Image Package format files, because the thought of wrapping my images in something non-standard scares me. I just want my raw images in a new location, that’s all.
Synchronize Finals Folder
Once the export process has completed, I switch to my Finals catalog, locate the folder for the year I exported my images to, then right click that, and select Synchronize. Capture One will then go and look for anything new in my current year folder, and import them into my Finals catalog. As long as you turn on the Include Adjustments checkbox on export, any changes made to your images will also be applied to your new copy.
Starting a New Year
At the start of each new year, I have a little bit of cleaning up to do, to prepare for starting to photograph the new year. First of all, I ensure that I have completed all edits that I want to do on my previous year’s images, and ensure that I have run my backup to mirror these images to my Drobo.
Then, I close the catalog in Capture One Pro, and delete the folder from my Traveller drive. After that, when I reopen the catalog in Capture One, my folders all show up as missing, as you can see in this screenshot (below).
Locate Missing Folder
To fix that, and complete the process, right click the top level drive or folder, and select Locate from the shortcut menu. You’ll then be able to navigate to the copy of your year folder that was your backup copy until a few minutes ago. After spending some time locating all of your images in the new location, you are ready to continue to use your catalog. From this point on, this becomes your master copy, along with all of the other previous years.
Create a New Year Folder
And of course, you also need to create a new year folder to ingest all of the new work that you’ll make. Remember that this will live on your Traveller drive for the current year, along with your Finals, if that’s how you work, and you’ll just proceed as you did in the previous year. It’s all quite easy once you have gotten your head around it.
Wrap-up
Having spent many years tweaking and developing a smooth workflow, I’m very happy with how I work, so I hope this helps you to smooth out any possible kinks that you might have in your own workflow. As I mentioned earlier too, if you don’t use Capture One Pro, the techniques and strategy that I covered should be pretty transferable to whatever program you use to manage and edit your photographs.
Show Notes
Sandisk Extreme 900 SSD: https://mbp.ac/sd192TB
ChronoSync: https://www.econtechnologies.com
GoodSync: https://www.goodsync.com
SyncBack: https://www.2brightsparks.com/syncback/compare.html
Western Digital Red Drives on B&H: https://mbp.ac/wdr8tb
Music by Martin Bailey
Audio
Subscribe in iTunes for Enhanced Podcasts delivered automatically to your computer.
Download this Podcast in MP3 format (Audio Only).
Download this Podcast in Enhanced Podcast M4A format. This requires Apple iTunes or Quicktime to view/listen.
by Martin Bailey | Apr 7, 2015 | Gear, Podcast, Tutorial
Over the years I’ve developed and evolved a pretty sound file management workflow for working with Lightroom on multiple computers, both in the office and when I’m traveling. I’ve talked about various aspects of this in previous episodes, but I thought I’d report on my current image management workflow for the mobile photographer.
I’m going to explain how I currently manage my Lightroom catalog, settings and presets, and my photographs and video archives, including how I now move from one computer to another quite seamlessly, but first a little background.
Until now, I’ve kept my Lightroom catalog on the internal hard drive of my desktop and laptop computers, and synched between the two before I made any major changes to the library. This works and if you have a fast enough network, it’s not too much of a pain to sync your catalogs, but because the previews that Lightroom creates can often become quite a hefty chunk of data, I used to leave them out of the synchronisation, which means that I’d have to rebuild previews on the other computer before I could quickly view images. If I synchronised the image previews as well, it took quite a bit longer, and that can really slow you down when you need to move computers, which I sometimes do multiple times each day.
Preparation
My entire digital workflow revolves around Lightroom, so rather than synching the Lightroom catalog and my most recent work from computer to computer, I figured that it would be easier to just put it all on an external hard drive, and move that around. I’ve been doing this a while now, and never been happier with my workflow, which is why I decided to share this today. Let’s first look at what you need to put on that external hard drive to make this all work smoothly.
Lightroom Settings folder contents
Firstly, I recommend that you set up Lightroom so that it saves all its presets with the catalog. This means when you move the Lightroom catalog all of your settings will go with it. If you don’t do this, you’ll still have to synch the settings around separately, which we want to avoid.
To make this change go to Lightroom’s Preferences, then under the Presets tab, turn on the “Store presets with this catalog” checkbox. You’ll now see a “Lightroom Settings” folder in the same location as your Lightroom catalog (right).
If you don’t know where your Lightroom catalog is, go to the Catalog Settings and you’ll see the path to your catalog under the General tab’s Information section. There is a “Show” button there. Click that, and check that your Lightroom settings are now with your catalog. This is also of course where you’ll need to go to copy your catalog to your external hard drive. (Just copy the entire Lightroom directory, including your Lightroom Catalog.lrcat file and your Lightroom Catalog Previews.lrdata file/folder to your external hard drive when you’re ready.)
One Lightroom Catalog
Note that apart from a catalog with one image in that I use in an automated process to keep my printer from running unnecessary head cleaning processes, I have all of my images in a single Lightroom catalog. I currently have almost 300,000 images including some videos in my catalog, and it runs fine, so I like to keep them all in a single catalog.
This makes it easy to search across my entire library for images and build collections from absolutely anything I’ve shot. If you use multiple catalogs, you’ll need to decide which ones to use with this workflow, or just ensure that you move all of them to your external hard drive.
Drobo Mini
Drobo Mini
To ensure that Lightroom works as fast as it can with this portable workflow, I recently bought a Drobo Mini with 4 x 1TB 7200 rpm hard drives, and a Crucial 250GB mSATA Internal SSD which I put into the bottom of the Drobo Mini as an accelerator disk. This speeds up Drobos so much that as long as you are using Thunderbolt to connect them to your computer, you really just don’t have to worry about the hard drive speed. It’s not as fast as an internal SSD drive, but it’s fast enough to run Lightroom stress-free directly from the external hard drive.
You can also run Lightroom from slower portable hard drives, but I suggest that you use at least USB3.0 connected drives, such as the WD My Passport Ultra drives that I use in my ultra-light portable workflow when I simply cannot carry the weight of the Drobo Mini in addition to my MacBook Pro. This may be necessary especially when traveling overseas, as the one downside of the Drobo Mini is that it isn’t very, well, mini. It’s quite a hefty piece of kit to carry around in addition to a laptop.
[UPDATE Aug 1, 2015: Note that I’ve pretty much stopped using the Drobo Mini. Having to plug it into the power every time I wanted to use it became quite tiresome after a while. It’s also just too big for any kind of air travel. I have now started to use Western Digital My Passport Pro 4TB drives. These are Thunderbolt only, so won’t work on Windows at this point, but they are powered by the laptop, and they are slightly faster than the Drobo Mini, so I’m now using this as my main catalog hard drive, and they are small enough for air travel as well. I have two, with the second a straight backup of the first.]
Recent Work and Final Selects on External Drive
Lightroom Catalog Contents (click to view details)
In addition to my Lightroom catalog and settings, I also keep my main archive of all of my best work to date, which I call my “Finals” or “Final Selects”, on my Drobo Mini, as well as all of the photographs and video that I’ve take during the current year. So basically most of what I need to access regularly is in one place and always available when I travel.
Main Archive on Drobo 5D
My main archive of all images and video that I’ve ever shot and not deleted is almost 7TB of data, so it’s not practical to keep all of this on my portable hard drive, and because I have every image that I thought was good enough to sell or show people in my Finals folder, it’s not even necessary.
I can still get to my raw images and any TIFF or PSD files that I might have also created from them, right there on my portable drive, so what I call my “Photo Originals” folder lives on my Drobo 5D attached to a desktop computer in my office studio. This is literally everything from every shoot I’ve done that didn’t get deleted.
Decide and Stick with Your Strategy
One thing that will cause you to get frustrated with a strategy like this, when you’re synchronising folders around and have photos in multiple places, is if you lose track of which copy is your main copy. As we can see in this screenshot (right) I have my Finals folder on both my Drobo Mini and my Drobo #1 drive (a Drobo 5D). The main reason I do this is so that it gets backed up into the cloud via Backblaze, and we’ll talk about that shortly, but it’s important to try to keep this as a backup copy, and not a working folder of images.
I do sometimes just need to reference images or grab something quickly over the network, and because my iMac stays on all the time, from anywhere in my house I can connect to the Drobo and access my Final Selects. This is also why I keep this linked to Lightroom, but I don’t do any editing or create collections from the Drobo #1 drive, because it not only causes you to lose track of changes and break your Lightroom collections while you’re traveling, but you also have to sync your changes back to your main copy. This is doable quite easily, but I find it much better to not get into that, and my portable hard drive solution that we’re looking at today helps us to avoid this too.
Diagram #1 – Base Computer
OK, so I know that this will be heavy going without some form of graphical representation of what I’m talking about, so I’ve created a few diagrams for us to reference today as I explain this further. Let’s look first at my main computer. We all use at least one computer to work on our images, so this should be useful even if you don’t use a laptop in addition to your “base computer”.
Diagram #1 iMac with Drobo Mini
Take a look at the first diagram (above) and see on the left that my workflow starts with transferring images from the camera to my portable hard drive, which is connected to my iMac. This could just as easily be a Windows machine. It’s not important what system you use. What’s important to note here is that my images go into a folder for my current year on my external hard drive, along with the Lightroom catalog and my settings and presets.
Diagram #2 – Local and Cloud Backup
As I mentioned, I keep my main photo and video archive, my “Photo Originals” folder on a Drobo 5D, which is always attached to my base computer. As soon as I’ve finished transferring images from my camera and have them renamed, and if time allows gone through and done my first quick edit of my images, then I copy the folder for that shoot to my Drobo 5D, here called Drobo #1.
Diagram #2 iMac with Drobos and Cloud Backup
As you can also see from the diagram, because I have Backblaze set up on my iMac, as soon as I copy any new images to my Drobo 5D, they start to backup into the cloud. I will continue to synchronise changes to this Drobo 5D as I edit the images from my shoot, but I want to start to get my cloud backup started as quickly as possible. Any later changes will also sync into the cloud, so there’s little reason to wait on this, unless you are paying for data upload.
Diagram #3 – Second Backup for Paranoia’s Sake
There’s one last element of this base computer setup that I’d like to talk about before we move on, and that’s my second Drobo 5D which is purely for local backup purposes. I know this is a little paranoid, but bear with me. The Drobo 5D can have one hard disk fail without losing any data. If a hard disk fails, you simply pull it out and put a new hard disk in, and the Drobo automatically writes the necessary data back to the new hard drive, and you are safe against hard disk failures again.
In my paranoid mind though, that’s not enough to feel safe. I could have a second hard disk fail before my data is fully secured after replacing the first one, and the entire unit could fail too, leaving me with nothing local to fall back on. Assuming my Backblaze backup had already completed, I could of course download or have them send me my cloud backup on hard drives, but that takes time and I’d be panicking for days until my data was restored, so I just prefer to have a second local backup, as we see in this third diagram (below).
Diagram #3 – Second Drobo Mirrored Backup
ChronoSync for File Synchronisation
For all of my file synchronisation I use ChronoSync from Econ Technologies. This is the only operating system specific part of my workflow that we’ll touch on today. ChronoSync is only for the Mac OS. When I used Windows, I used to use a command line tool called Robocopy, but I haven’t used that for years, so I won’t go into the Windows alternative today. If you have a great tool that you’d like to recommend for Windows, please drop a note in the comments section below.
ChronoSync is an incredibly powerful file synchronisation tool. It’s important that you actually read the help to avoid deleting files unintentionally, but once you have a good understanding of how it works, it can make life a lot easier. One of the reasons for this, is because you can save your synchronisation tasks and open them again later to rerun them. For example, after I’ve transferred my images from my camera to my Drobo Mini, to copy them to my Drobo 5D and start my Backblaze backup, I simply launch a saved Sync task that will look for anything that has been changed or deleted from my 2015 folder on my Drobo Mini (see below) and copy or delete it from my Drobo 5D as necessary.
ChronoSync Screenshot
As I work on my files from a new shoot, or make any changes to my earlier 2015 files on my Drobo Mini, I just run this task again. For the whole of 2015, the current year, I will use my Drobo Mini as my main archive, and the Drobo 5D 2015 folder will be my backup, so I generally just Mirror the changes across. If necessary, you can do a synchronisation and copy any changes that you make to the target drive back to your main copy, simply by changing the Operation that you see in the middle of the screenshot.
A couple of important things to note here are that I usually run the Trial Sync with the button in the toolbar before I actually execute the sync task. This is like a dummy run, and you get a dialog to see what will be copied or deleted, so you can check that you haven’t made any stupid mistakes before you actually make them. The other thing is that you can select wether to delete files immediately, move them to trash, or move them to an archive folder instead of deleting them. I don’t like the Move to Archive option because you end up with archive folders everywhere, but I do like to turn on to just move the files to the trash, rather than delete them immediately. This is just another safety net.
Lightroom Synchronize Folder
Synchronize Folders in Lightroom
Because I also have a 2015 (current year) folder in my “Photo Originals” directory on my Drobo 5D, once I’ve synched any images, I right click the folder in Lightroom, and select “Synchronize Folder…” This tells Lightroom to check the contents of the folder for anything new or removed, and you can also have it check for metadata changes as well.
Lightroom Export to Copy “Finals”
Once I’ve completed my editing of a shoot, and have my “Finals” or “Final Selects”, I copy these to the appropriate year in my Finals archive folder. Everything from the current year goes into a single folder. If I created a TIFF or PSD copy of my raw file, say to create a black and white version in Silver Efex Pro, or did some work in Photoshop, then I will keep both the raw file and the new format files together. If no copies were made, I just copy the raw files to the Finals folder.
Because I star rate my images to help with filtering, when I’m ready to copy my files, I just filter out anything with two stars or above. In my rating system, 2 stars means an original raw file. 3 stars is anything that I will present to Offset for consideration for inclusion in my stock library. 4 stars are images that I consider good enough to show people or use in a blog post etc. 5 stars are what I consider portfolio quality images.
Lightroom Export Original Files
So, when I’m ready to copy my final selects to my Finals folder, I simply filter anything 2 stars or above from my original shoot folder, and use a Lightroom Export preset to copy these images to my Finals folder on my Drobo Mini and my Drobo 5D.
At this point, I copy to both locations because I can add the images to the Lightroom Catalog at this point, and that saves me from synchronising the Finals folder after copying files across manually or using ChronoSync.
The important thing to note here is that although this is an Export, I’m not creating a JPEG or any other new format. I select “Original” as the format, under both the Video and File Settings sections. This ensures that the files are simply copied to the new locations, whether they are a raw file, or a TIFF or PSD etc.
Once I’ve setup something like what we see in the screenshot here (right) I just save this as a Preset, then when I want to copy my Final selects to my Finals folder, I just have to select them and right click them, then select “Copy Original to Drobo Mini 2015 Finals” which is what I called the Preset, and I have a second preset to copy to my Drobo 5D.
Mirroring Entire Drives with ChronoSync
To close the loop on the last diagram before we move on, I guess I should just mention that to mirror the contents of my first Drobo 5D to my second, I also use a ChronoSync Task, but because we will mirror the root of the drive, I set up a few Rules to prevent ChronoSync from copying and overwriting some important system files, as we can see in this screenshot (below).
Sync Drobo #1 to Drobo #2
OK, so now you’ll see that we have a pretty sound process in place for managing images based on a Lightroom catalog and a few ChronoSync tasks that we can launch and run when changes have been made. It’s a little more complicated than simply transferring images to the hard drive inside your base computer, but remember, there’s one key advantage to having everything that you need to use regularly on that external hard drive.
Diagram #4 – Image Library Portability
With your workflow set up this way, all you have to do to access your images on another computer, is to eject your portable hard drive from the base computer and plug it into another computer. Whether you are in another part of your house or office, or on the other side of the planet, if you plugin your portable hard drive, you have access to everything necessary to start Lightroom and continue working as you would on your base computer.
Diagram #4 – Image Library Portability
Because Lightroom remembers the last catalog that you opened, it automatically goes to the external hard drive, even if you open Lightroom with the application icon. Of course, to cause this to happen, when you first move your Lightroom catalog to the external hard drive, you’ll need to double click on the catalog in its new location to force it to open from there, but as long as you have Lightroom set up to open the last catalog, that’s the only time you’ll have to do this. You can also select File > Open Catalog… and navigate to your new catalog location too, but again, you’ll only have to do this once.
Of course, because the main archive of all of your images, what I call my “Photo Originals” lives on a hard drive on your base computer, so that won’t be accessible, but when Lightroom can’t see anything, it just marks the folder with a question mark, to let you know that it’s offline. You can still click on the folders, and if you have previews created, you can even see the images. If you need to be able to edit photos that are essentially offline, you can enable this by going to Library > Previews, and selecting Build Smart Previews, but without that you can’t edit images in the Develop module etc. until you get back to your base computer. The point is though, Lightroom handles this gracefully.
Backups While On The Road
WD My Passport Ultra 2TB
The other items that you’ll notice in Diagram #4 (above) is my mobile backup drives. I use WD My Passport Ultra USB3.0 drives, because I think they provide great cost performance at just $99 for the 2TB drives. These are a little fatter than the 1TB drives, but I like to be able to backup my entire “Final” selects library on to these drives as well as my current year’s “Photo Originals” folder.
Now, as you know, I’m paranoid, so when I’m traveling, I actually make two backups of my images. This means that as I shoot, I backup all of my current year folder to two backup hard drives. Backup #1 and Backup #2 in diagram #4. Again, I use ChronoSync for this, and just save a task for each backup, and run it as necessary. Because I only have two USB ports on my MacBook Pro, I actually have to eject and plugin new drives when I want to run my Time Machine backup, but because my Drobo Mini connect with Thunderbolt, I can have both Backup drives attached at the same time as well.
You can even create Containers in ChronoSync, which can contain multiple sync tasks, so if you want to backup your images to both backup drives without intervention, you can do that quite easily. This is useful if you want to for example start off your double backup before taking a shower etc.
I know that some of you will consider it overkill to have a total of three backups of your images while traveling, but depending on where you’re going, I think it’s necessary, and generally do this whenever I’m on the road. I actually had one of my three external hard drives fail near the start of 7 weeks in Antarctica, and that was scary enough. If that had been my only backup drive, I’d have been climbing the walls.
Ultra-Portable Alternatives
As I mentioned earlier, the Drobo Mini is a hefty drive to lug around, especially if you’ll be jumping on international flights etc. so here are a few ultra-portable alternatives that work seamlessly with this workflow.
1) The first and most obvious alternative, is to simply synchronise your Lightroom Catalog to the hard drive of your laptop, but of course this requires that you have a large enough internal hard drive or SSD to hold your Lightroom Catalog, your Preview images and also maybe the images you’ll be shooting as you travel. This is great if you have an internal SSD, because they’re lightening fast to work from, but big SSD drives are expensive, and if you’ll be traveling for a long time, it will likely fill up.
2) The second lighter alternative is to use a lighter but still external hard drive, like my WD My Passport Ultra drives as the main archive and for your Lightroom catalog etc. This isn’t as smooth and stress free an experience as working with the Drobo Mini, because these drives are much slower, but it works, and is a nice affordable second choice if you are going to be shooting a lot. Lightroom is pretty good at finding your images etc. on the new drive as well. At least on a Mac system.
If Lightroom can’t find your images when you open the Catalog on a different drive, signified by the folders having a question mark against them, just right click the top level folder and select “Find Missing Folder” in the shortcut menu, then navigate to the folder on your new hard drive. This will remap everything, including your previews, and in my experience will not corrupt your catalog or anything.
Set a Hard Drive Letter for Windows
If you use this method of using a portable hard drive in a Window environment, you’ll probably need to ensure that the drive letter doesn’t change as you move the external hard drive around. I don’t remember exactly where you do this right now, but you can assign a drive letter to your hard drives, so it’s a good idea to assign something well away from the start of the alphabet, like M for mobile. That way other drives that you might attach that will be lettered D, E, F etc. won’t displace your external drive’s letter.
Not Really a Cross Plastform Solution
I should also mention that this solution is not ideal if you switch between Windows and Mac regularly. The catalog can be taken from one operating system to the other and will open, but Windows and the Mac OS reference drives differently, so you’d need to tell the other OS where your files live each time you open the catalog on the other system.
Also the location of your presets and settings is not recognized, so I personally think it’s more trouble than it’s worth if you are switching between operating systems. It makes it easy to move from one system to the other, but not really great if you want to switching back and forth.
Wrap-up
OK, so I hope that has been useful for you. Having synched my Lightroom catalog around for the last few years, I’m finding it much easier now to just move my external hard drive around. It might not be for everyone, but I am really enjoying this workflow. As good workflows should, it just works, and that’s important to me.
Show Notes
Drobo Mini + 4 x 1TB 7200 rpm 2.5″ HDDs: https://mbp.ac/drobomini4tb
Crucial 250GB Internal SSD: https://mbp.ac/msata250gb
WD My Passport Ultra portable hard drive: https://mbp.ac/wd2tb
Music by Martin Bailey
Audio
Subscribe in iTunes for Enhanced Podcasts delivered automatically to your computer.
Download this Podcast in MP3 format (Audio Only).
Download this Podcast in Enhanced Podcast M4A format. This requires Apple iTunes or Quicktime to view/listen.
by Martin Bailey | Sep 13, 2013 | Fundamentals, Gear, Podcast
Today I’m going to walk you through my data backup strategies at home and in the field. This is in response to a listener question from someone that heard me talking about this briefly on This Week in Photo. I should precede this with the disclaimer that I’m perhaps a little bit paranoid with my backups, but I should also add that I’ve never lost an image in 18 years of digital imaging, and that includes scans of slide film from way back when.
I’ve been running behind a little this week too, hoping to release an Iceland update a little earlier, but I’ve been struggling with some new software that I had hoped to use to show you my new Iceland portfolio images with. It’s almost ready now, and you can see my selected 50 shots on my site already, although I need a little more time to finesse the presentation. If you want a sneak preview, go to https://mbp.ac/iceland2013, and click the little arrow at the bottom right of the gallery to view the images full-screen.
Synchronization Software
So, before we jump into my backup strategies, I’d like to talk about how I synchronize my files between the various hard drives that I’ll discuss. When I first switched to the Mac OS, I initially used a product called Sync, Sync, Sync, but a couple of serious bugs introduced via upgrades and a few corrupted files that were possibly caused by this application got me looking for a new solution about six months ago, and what I decided on was an application that I’ve been very happy with, called ChronoSync.
ChronoSync is very powerful, and can be used to create all kinds of synchronization jobs. I’m not going to go into details today, but you can specify various types of Synchronization jobs, some of which could result in your deleting files by mistake if you don’t understand how a specific synchronization type works, so you have to be careful with applications like this, but as long as you read the help and set things up correctly, ChronoSync will serve you well. Another great thing about ChronoSync though is that you can click the Trial Sync button, and it will throw up a dialog that will tell you exactly what will be copied, what will be deleted and everything like that, so you can check your sync jobs before you actually run them.
Here’s a screenshot of one of my Sync jobs, which I use to synchronize my September 2013 raw files from my MacBook Pro drive to my Drobo 5D. I’ve selected Mirror Left-to-Right as the synchronization type, so that anything I delete from my local hard drive as I edit, is also deleted from my backup when I sync the two. You can also select to move deleted files to an archive folder if you don’t want to risk automatically deleting something by mistake.
ChronoSync Screenshot
The other good thing about ChronoSync is that you can create Containers that are basically batches of Sync jobs, so for example, I have one Container that holds all of the Sync jobs that I run between my two Drobos, so when I’m ready to sync, I can run individual jobs, or just run the entire batch, syncing everything that I’ve built individual sync jobs for.
On Windows, I used to use Robocopy, which is included in the operating system to sync between files. I created text based command files that I would just double click to run the sync jobs, and the result was very much what ChronoSync is doing, but without the interface and ease of configuration. You basically have to read the help and figure out all the commands you need, and write the scripts yourself with Robocopy. It’s not difficult, but it’s too complicated to try and cover here, especially as I no longer use it.
So, when I talk about running a sync job, or synchronizing between drives today, just understand that I’m talking about running a ChronoSync task, but this could be whatever you chose to use to synchronize between your own storage solutions. Another thing to note though, is that I do not recommend manually moving files around.
Manually copying files is OK for an initial backup, like say when you get home from a trip, and just copy an entire directory structure to your main hard drive, but once you have to start doing incremental backups to save changed images and deletions, any new files you create for black and white versions or other edited images to your backup drive, it quickly becomes a pain to do this manually, and it’s error prone. Do find and use a good synchronization solution for this part of your workflow.
At Home/Office
Let’s first touch on how I organize and access my images at home, and then how I back up my images. Initially, every photograph I shoot is first copied to the solid state drive in my MacBook Pro. Because of this, I always get the largest internal drive I can afford when I buy a new computer, so my MacBook Pro Retinal has a 750GB solid state drive installed. This generally enables me to save all of each months images locally, before I clean that out as I move into the following month.
It’s totally up to you how you organize your images, but for me, having everything in a three layer year, month and day structure works well, especially when backing up. I’ve seen people that use location based folder names, but that relies on you remembering what you’ve already backed up and what you have not, or manually comparing your backups etc. which is error prone, not to mention a pain.
If everything for this year is under 2013, and everything for September say is in a directory named 09, each in their own day numbered folders, it’s really easy to check that you are backed up. With the keywording and collections that we now have in Lightroom and other media management software, it really isn’t necessary to include the location or shoot name in the folder structure, unless you have some sort of process imposed on you at a workplace or something.
Anyway, I shoot away for the current month, and create a Sync job for that month that I can run after each shoot, and that automatically copies all of my images, including any changes or deletions to my main storage, which is now a Drobo 5D. This is connected to my MacBook Pro, which is currently my main computer, via Thunderbolt. The Drobo 5D with an mSATA SSD Accelerator drive installed is lightening fast. It’s almost as fast as the internal SSD drive in my MacBook Pro, so backups are very fast, even with 100+ Gigabytes of data.
Drobo 5D Screenshot
Now, I know that Drobos provide a certain amount of redundancy/fault tolerance, in that with my current configuration, if one hard drive fails, I can pull it out, and replace it, and my data will be safe, but I’ve had a drive fail, and with the amount of data I have stored, which is currenly 5.12TB, it takes two to three days to rebuild the data once you put the new hard drive in. That means there is a two to three day window in which a second failed drive would cost me all of my data, and I don’t want to take that risk.
Cloud Storage
So, I have two Drobos, basically one is a copy of the other, but there is a second very important reason for having that second Drobo, which is Cloud Storage. My second Drobo, a 2nd Generation USB connected four bay Drobo, is connected to my old MacBook Pro, on which I have Backblaze installed. Backblaze currently costs just $50 per year, or $95 for two years, and that is for unlimited storage. As I say, I have over 5 terabytes of data and every byte of that is uploaded to Backblaze, so if I lose any files, I can download them from Backblaze at any time, and that has happened in the past.
Drobos
I did a portrait shoot for a client and when I came to work on some prints for them, I found one of the images was corrupted. I believe, though I can’t prove, that this was caused by my last synchronization application during a few synchronization back and forth, so all of my backups were corrupted. Of course, the Backblaze copy was also corrupt, but Backblaze keeps up to four weeks of versions of files, and because I found this issue within a few weeks, I was able to roll-back to an uncorrupted version of the image file.
Had I gone past the four weeks, I’d have lost the file and had to deal with the embarrassing task of telling my client that I lost one of their precious portrait images, so it’s important to ensure that things don’t get corrupted, and that is why I switched synchronization software and touch-wood, nothing has been corrupted since.
So, to recap, my main workflow at home is to shoot for a month, keeping everything on my MacBook Pro, and then after each shoot everything gets backed up first to a Drobo 5D connected via Thunderbolt, and then a second backup is done over the network to a second Drobo connected via USB to my old MacBook Pro. As long as my Backblaze backup is up to date, I can usually backup around 20GB of data per day, so unless I have done a really big shoot, I’m usually backed up in the cloud too within about 24 hours.
Once I’ve finished processing each month’s images usually within the first week or so of the following month, I run one last synchronization from my local MacBook Pro drive to the Drobos, then I delete the images from the local drive. At that point, Lightroom sees that the local images are missing, and I point it to the new month directory on the Drobo 5D, and continue to access my images as normal.
I also catalog the images on my old Drobo over the network, so if I need anything while I’m not at my desk with the Drobo 5D plugged in, I can still access it over the network from anywhere in the house. This is important as if you recall, my office and studio are on the 3rd floor of our apartment, and our living space is on the second floor, and so I don’t spend too much time in the studio, I work from our living room or dining table for a while after breakfast, and then in the evenings, and it’s nice to be able to get to stuff over the network if I need to.
In the Field
That’s my basic home/office workflow, but now let’s look at what I do when I’m traveling. Right now I use four portable hard drives in the following way. I have two 2TB Western Digital My Passport Studio drives that are my main backups in the field. These are the two drives stacked together in this image (below).
Every day when I get to the hotel, I transfer all of my images to my local hard drive first. Then, I synchronize that to my first 2TB hard disk. I usually try to do at least this much before dinner, and I put the 2TB drive in my pocket before I leave the room. If I don’t have time for that, I still put the drive in my pocket because it contains all the previous days backups, but I also put the compact flash cards from that day in my pocket too, rather than leaving them in the hotel room.
Portable Backups
Once I get back to the room after dinner, if I have shot any video on my GoPros that day, I back them up to a third 1TB hard drive, that I connect to the computer via Thunderbolt. This is the white Buffalo drive that you can see to the left in this photo. This isn’t much faster than the Firewire Drives though, because it’s a slow 2.5in hard drive. Thunderbolt is kind of wasted on standard 2.5in hard drives, which I guessed would be the case, but I bought this to try it anyway. I won’t buy any more unless they boast very fast hard drive speeds to keep up with Thunderbolt. In fact, I think if I buy anything else for portable backups, it will probably be a Drobo Mini, because they have the same SSD acceleration that the Drobo 5D uses, and that screams along.
Once I have my video backed up to my 1TB hard drive, I run a synchronization between that drive and my first 2TB hard drive, and then, I synchronize my first 2TB hard drive with the second 2TB hard drive. That gives me my two backups in the field, so if I need to, I can delete the images from the local hard drive, although I try to avoid this if at all possible. As I mentioned earlier, I have a 750GB internal solid state drive so I can usually shoot for around three to four weeks before I have to start deleting stuff. This is also why I backup my video straight to an external drive, as I’d fill up my local drive too quickly otherwise.
Finally, usually before I go to sleep, I plugin my fourth portable hard drive to my computer, that you can see at the back in this photo (above), which is my portable Time Machine backup. This means that until I delete anything from the local hard drive, I actually have four copies of everything while traveling. This is enough to keep me happy. 🙂
Note that for the last few versions of the Mac OS, you can now have multiple time machines. In the photo here (above) you can also see a Belkin Thunderbolt hub, into which I basically plug everything, including my Drobo 5D which chains to my external monitor via Thunderbolt, and all of my other USB3.0 and USB2.0 devices, as well as my Firewire card reader, Wacom tablet and speakers etc. all plug into this, so when I sit at my desk, I actually only have to plug in the power to the MacBook Pro, and one thunderbolt cable, and everything just connects. The reason I mention this is because I also have a USB3.0 external hard drive attached to this hub, which continuously updates a Time Machine backup of my computer when I’m at my desk. The portable Time Machine copy is only used when traveling.
The main thing to note about these hard disks now, is that I always carry my main 2TB backup disk with me everywhere when I’m traveling. It not only goes to dinner with me, but it stays in my photographer’s vest all day long. When possible, I also carry my 1TB hard drive that you can see at the back of the photo. This disk is very tough, even withstanding a bit of a dunk in water if necessary, so as long as I haven’t deleted my local copy of my images, they are all in there, inside my Time Machine backup, so if I lost my computer, I could get all my information back, including mail and other personal data to the point of the last backup.
Note too that my 2TB drives are large enough for me to keep a backup of all of what I call my Final images. These are images that I have selected for my portfolios, or stuff that I feel is good enough to show people. If I have done a black and white conversion in Silver Efex Pro for example, I will have the original RAW file, and the converted TIFF or PSD file in my Finals folders too. These are organized by year, so I basically end each year with a new folder, with all of my best shots and original RAW files for that year. This means if I’m traveling and someone needs a few images from me, the chances are I can get them to them from on the road. I can also access all of my RAW files for my best work to give demonstrations of software etc.
I also keep all of my RAW files from every shoot that I do during any given year on this 2TB drive, because when I get home from a big trip, it will take a while for the backups to upload to Backblaze, especially if I have video to upload too. This means that I can carry my hard drive around with me for a while after I get home, and if anything should happen to my house while I’m out, I don’t lose all my recent work.
Summary
So, one last summary here, I have all of my images, and all of my documents, email, music and everything that I value, all on my Drobo 5D, which is my main storage. That is backed up to a second Drobo and that gets backed up to the cloud using Backblaze. This is three copies of all of my data, which is currently 5.12TB and counting. When I travel, I have at least two external backups of my work, as well as a Time Machine backup, in case I lose my computer.
Off Site Backup
Now that I have this much redundancy in my backups, including the cloud backup, I don’t do off-site backups as much as I used to. When I was still in my old day job, I would keep a backup of all my data on a few 3.5 inch hard disks that I would load into an external bay occasionally, and sync from my main data, then take that copy back to the office and just leave it in a drawer. This was still Tokyo though, so every year or so, I would also copy my entire library to a series of old hard drives, and send them to my brother in the UK, and would just store the hard disks somewhere for me.
This is less important to me now that all of my data is in two places at home and the cloud, but when I can, I still like to do this. It’s just one more backup that could save my ass if something really nasty happened here in Japan, at the same time as Backblaze turning pear-shaped, although I can never see that happening. Realistically though, if anything did happen to my local backups, I’d probably request a my data to be sent to me on hard disks from Backblaze rather than my brother, as the copies he has area never going to include my latest work.
As I said, I might be a little bit paranoid about my backups, but if even a part of what I do gives you a hint on how you might improve your own backup strategy, that’s great. The most important thing to remember is that all hard drives fail at some point, so you should never trust your images in just one place. The minimum you should do is backup to an external hard drive, and if possible, make a backup of that to keep away from your home, or sign up for a Backblaze account or a similar service, and ensure that your precious images are also backed up in the cloud.
Show Notes
ChronoSync: http://www.econtechnologies.com/pages/cs/chrono_overview.html
Backblase: http://www.backblaze.com/
Music by UniqueTracks
Audio
Subscribe in iTunes for Enhanced Podcasts delivered automatically to your computer.
Download this Podcast in MP3 format (Audio Only).
Download this Podcast in Enhanced Podcast M4A format. This requires Apple iTunes or Quicktime to view/listen.
by Martin Bailey | Aug 13, 2011 | Digital Products, Gear, Podcast
I’ve briefly touched on backing up in various episodes over the last six years, but following a conversation this week with Matt Marshall, a great photographer from the UK, I decided to wrap a few ideas, including Matt’s, into this week’s episode. Thanks for all the thought provoking conversation this week Matt, and good luck getting your backup and access solution set up in the coming weeks.
Let’s look at both backing up and accessing our images at home, and backing up in the field today. Matt’s queries and feedback were mainly around backing up and accessing his photos at home, so we’ll start there.
Laptops as Main Computers
As laptop computers become more and more powerful, I’m noticing more people making a Notebook PC or MacBook Pro their main computer. Even if you sometimes need more power and have a beefier desktop style computer to work on as well, laptops can now deal with just about anything we throw at them, including running the latest versions of Adobe Photoshop and Lightroom for both Mac and Windows, and of course Aperture for Mac users.
Slow Innovation of Hard Disks
Despite the incredible increase in power and speed though, and the innovation that we see in things like Compact Flash cards, as well as computers etc., one thing that just doesn’t seem to make quite the same leaps forward is the Hard Disk. Sure, we’re seeing larger drives each year, and I now use 3TB drives to store and backup my photos. The 3.5″ drives are also usually less money for double the size that were available a year earlier, but still, progress is slow, especially with the 2.5″ drives that we see in laptop style computers, and the basic design of the hard drive in general hasn’t changed for years.
As of August 2011, I’m looking at Apple’s online store and see that you can now order a MacBook Pro with a 750GB hard drive, which is a reasonable size, but this is a slower 5400 RPM hard disk. With no change in price you can select a 500GB for a faster 7200 RPM hard disk, but just six months ago I paid a premium to have one of these installed in my MacBook Pro, because I wanted faster hard disk access.
I didn’t go for a faster Solid State drive, as they are still way too expensive for a decent sized drive. Even today, if you want a 512GB SSD, it costs an extra $1,100 or 100,695 in the Japan Store. Here’s the problem though. Most people once you’ve been photographing for more than a few months end up with a library of images that can no longer practically be stored on the local hard drive, and this is where Matt was coming from in his communication with me this week. We have to figure out a way to backup our images outside of the computer and access them when necessary.
Wireless Access to Hard Disks
One of Matt’s ideas was to buy an Apple AirPort Extreme Base Station. This is basically a Wifi base station, with a USB port, and Matt is thinking of attaching multiple hard drives to this to backup and access his images, as well as doing Time Machine backups of his MacBook Pro. As Matt has a reasonably small library of images at the moment, this would probably be fine. If you have a large library of images, it can take a long time to get them on there, but once you are set up, it can be an easy and convenient way to access your images. I also access some of my images over Wifi too, and it works well.
By the way, the Apple Airport Extreme is fully compatible with Windows, but if you simply can’t allow yourself to buy something from Apple, don’t worry, there are plenty of other routers and base stations that act very much the same as the Airport Extreme. I have a Buffalo Air Station that supports 802.11n, which is currently the fastest Wifi available, and this also has a USB port for attaching hard disks. I have tried using hard disks shared via my Air Station, but I decided to change my setup.
Sharing Hard Disks with a MacMini
After switching to a MacBook Pro as my main machine, one other thing that I needed to think of was my Cloud backup. I have used Mozy and now use Backblaze for my Cloud backup of images, and these require that the hard drives be connected to a computer. To make this possible, I decided to repurpose the MacMini that I used to use to create this Podcast as a backup machine. MacMini’s are small, not much larger than the Airport Extreme, and I can attach USB hard drives, and access them over the network much the same. Of course, I can also install the Backblaze software, and set it off uploading my image library into the cloud, fulfilling both requirements.
As I have rather a large library of images, it takes forever to upload them all to Backblaze, so what I did was started off with just my final select images. I store my images in two different folder structures. My original RAW files, which are basically anything that I didn’t delete after a shoot are stored in Year, Month and Day folder structure. So my image library hard drive basically has year folders starting with everything up to 2000, then one year folder for each year since. Then in each year folder, I have 01, 02, 03, up to 12, for the months of the year, and each of these folders contains a day number folder for any day in that month on which I shot photos. This library is huge, at 2.4TB.
Whenever I’ve completed a shoot though, I of course go through the images and select the best shots, which will be what I upload to my Web site for personal work, or provide to my client for assignment work. All of these go into a folder called Finals, and this also contains year folders, up to 2011 at the moment. I used to just through them all into one Finals folder, but separated them out into years recently to make incremental backups easier. Basically, even my Finals folder is too big to keep on my internal hard drive on my Mac, so I currently only have all my final selects from 2011 stored locally.
Multiple Backups
Of course, making that my only backup is way too risk, and I also need to start off my backup to the cloud, so here’s what I do.
The first thing I do after any shoot is rate my images. Everything that I decide will be one of my final selects is given 5 stars. I do this while still in the RAW folder of course, which might be say 2011/08/12 on my hard disk. If I do any black and white conversions or stitching of panoramas etc. I save a copy of these images right there in my RAW folder too.
I then add any titles and captions before I make a copy in my Finals folder, because I want to be able to go back and locate the best shots in my RAW library too. If I’m in a hurry, the next thing I do at the very least, is run a backup script in a piece of software that I’ve started using called Sync! Sync! Sync! [link no longer valid]. You can try and use Sync!3 as Limited Edition freeware and save up to three backup jobs, but if you want to save more jobs, you have to pay $15. It’s well worth the $15 in my opinion, though the guy was a little slow sending me my full license.
You can basically though sync folders, or just copy new files. I move between modes depending on what folder I’m syncing, but at the moment, when I backup my finals, I am syncing a folder called 2011_Finals on my Mac, to Finals/2011 on an External hard drive on my MacMini. I also then do a copy backup of my Local 2011 RAW folder on my Mac. I do a copy backup here and not a sync, because I delete earlier RAW files from my Mac as the hard disk gets full. If I was to sync and not copy new files, after I delete old RAW files off my local hard drive, they would also be deleted from the hard drive on my MacMini, and I don’t want that.
Sync! Sync! Sync!
So, these are the first two backups I do, because once these are done Backblaze takes over and starts to upload these images to the cloud. Having run just two backup scripts without plugging in any external USB drives, I now have a copy on my Mac’s local hard drive, the hard drive attached to my MacMini in my studio, and then a third copy in the cloud, after a few hours that is.
3TB Drive in Nudist Stand
Now, that’s a good start, but I still need to backup my master library which is on a 3TB 3.5″ hard drive. Although the hard drive that I have attached to my Mac Mini is an external hard drive, I use standard internal 3.5″ hard drives for most other backups, and I have what are called Razoku no Otachidai, which translates to Nudist Stand to connect these hard disks to my MacMini or MacBook Pro to use or backup files. The nudist reference basically comes from the fact that you put naked hard drives into the stand. Being a little bit inhibited though, I buy clothes for my naked hard drives, as you can see in this photo of my drives in their Nudist Stands.
I use Sync! Sync! Sync! to copy images to these hard disks too, so this becomes my second local copy, once I’ve deleted the original RAW files from my hard drive on my Mac, and a third local copy of my Final images. Now, you’ll probably think I’m paranoid for doing what I do next, but I actually always buy external hard drives in twos, and once I have my master library disk up to date, I actually put a second identical hard drive into another Nudist Stand, and sync those too. This is to guard against disk failure of the large disks at home.
Way Offsite Backups
In addition to this, once a year or so, usually when my hard disk starts to get too small for my library, and I have to buy larger hard disks, having copied my library to the new hard disks, I will send my old hard disks, still containing my entire library to my brother in the UK. He just stashes them away for me in case something catastrophic happens here in Japan. This in addition to my cloud copy are my offsite backups.
iCloud
One other thing that Matt and I discussed was iCloud, which is the new cloud data synchronization solution that will soon be released by Apple. This is built in to the Mac OSX Lion operating system, and will basically allow you to store files in the cloud, and sync between all of your Apple devices, like iPhones and iPads in addition to your computers. I have searched and cannot find any information on what the upper limit for the amount of data that you will be able to store in the iCloud will be, but if it’s unlimited, then this would be bad news for companies like Mozy and Backblaze, who also offer unlimited storage. These companies charge a very small yearly fee for their services, but at the moment, it’s looking like iCloud will probably be free. That and the tight integration with the Mac OS will be very appealing, if there is no limit on the amount of data that can be uploaded. Something to look out for here.
NOTE: Since releasing this Podcast, a friend pointed me to a page about iCloud pricing and limits here:
http://www.macworld.com/article/161480/2011/08/icloud_storage_pricing.html#lsrc.rss_main
Backup in the Field
OK, so that’s pretty much what I do for backups at home, but what about when I’m out doing photography tours or workshops? Backing up to the cloud is pretty much impossible in the locations I go, so I have to make sure that I can create multiple backups while on the road.
I used to use the Epson portable media storage and viewers, like the P-5000 and P-7000, but since I started shooting with 64GB compact flash cards, I don’t really have a need to backup my images while I’m physically still in the field. It all usually happens back at the hotel now. This is no problem as well, because I pretty much always travel with a computer now, so I can just plug in hard drives.
2.5″ Portable Hard Drives
To keep weight down, I don’t travel with my external 3.5″ hard drives. I generally buy small 2.5″ portable USB hard drives. Again, I have two, so that even if I shoot so much that I have to delete the originals from my Mac’s internal hard drive, I can still keep two copies of my files to guard against hard disk failure. I keep about 200GB free on my Mac’s hard drive, and I was able to shoot for a month down in Antarctica earlier this year without deleting any originals from the internal drive.
Still though, I plugged in my two Western Digital 1TB Portable Hard Drives every day, and backed up my RAW images. With this much space I also keep a copy of all of my final select images, so I can show people the RAW files while on the road. This is especially important as I’m usually in a teaching role too, and like to be able to work on my RAW files with the group.
Two 1TB Western Digital HDDs
Again I use Sync! Sync! Sync! for these backups, as it allows you to create and save multiple backup scripts and just rerun them later.
Portable Hard Drive for Double Backup
If you do want to travel very light, I’m still a fan of portable storage, and after I became frustrated with the size of the hard drive in my old Epson storage units, I also used the Hyperdrive Colorspace, with a 500GB hard drive installed. These aren’t as slick to use at the Epson’s, but you can’t beat having 500GB of backup space in a unit not much bigger than a mobile phone. The great thing about both the Epson and Hyperdrives though, is that they both allow you to plug in a second hard drive and do a second backup of your data, without using a computer. I simply don’t feel comfortable reformatting my compact flash cards until I’ve got two separate copies of my images, so this is really a must for me.
Show Notes
Airport Extreme on B&H: https://mbp.ac/ape
1TB WD Hard Disks on B&H: https://mbp.ac/wd
Sync! Sync! Sync!: https://mbp.ac/s3
Music from Music Alley: http://www.musicalley.com/
After releasing this Podcast, a friend pointed me to a page about iCloud pricing and limits here:
http://www.macworld.com/article/161480/2011/08/icloud_storage_pricing.html#lsrc.rss_main
Audio
Subscribe in iTunes for Enhanced Podcasts delivered automatically to your computer.
Download this Podcast in MP3 format (Audio Only).
Download this Podcast in Enhanced Podcast M4A format. This requires Apple iTunes or Quicktime to view/listen.
