In my last blog
entry on this topic, I thought through the different types of files that
I have on my system and the correspondingly different levels of protection they
each need. Now I want to think about the different types of problems I might
experience and how I might recover from each. First however, just three days
ago, on 21 June, I received the following email from an author:

I have experienced a system crash on my computer — thus, I don’t know how
much writing I will be doing over the next weekend and (possibly week) as
I
recover my system. I don’t think I have lost any data regarding [book title
omitted], but I need to rebuild my system ASAP.

Wow! What a timely reminder of how vulnerable I am. This is exactly what I
don’t ever want to have happen to me. Notice the author believes he may be out
of commission for up to a week. A week! That’s much too long for me to be unable
to work. I personally want to be able to resume working in a matter of an hour
or so should anything go awry with any of my equipment. And he only "thinks"
he hasn’t lost any data pertaining to the book he’s currently writing. I want
to "know" that I haven’t lost anything, at least anything important.
With this mind, I want to think through the various disaster scenarios that
I might experience.

Bear in mind while reading these scenarios that I use two computers in my day-to-day
work: one running Windows XP and one running Suse Linux 8.0. The Windows XP
box is where most work happens, and it’s the most critical. My Linux box exists
only to run Oracle. I can usually tolerate more downtime on the Linux box than
on the XP box.

I might accidentally delete an important file — I can’t remember
the last time this has been an issue for me. Book and article files get traded
around by email so much that if I do delete one by mistake, I can almost always
count on having a copy in my email archive. And there’s always the Recycle Bin.
I’m not going to take any specific action to protect myself from this scenario.

My hard-drive could fail — If my XP hard-drive fails, I want to
be able to plug another hard-drive in, restore the files, and go. If I buy into
RAID, which I’ll almost certainly do in order to prevent the loss of even an
hour’s hard writing work, recovery from the loss of a hard-drive will be trivial.
I should probably keep a spare hard-drive on hand though, because even with
RAID I’ll be exposed until I replace the drive that failed.

My Linux box has two drives. I can mirror all the critical Oracle files: control
files, redo logs, and archive logs across those two drives. So long as I periodically
backup my database, I’ll be able to recover it in the event of a drive failure.
But such a recovery probably won’t be a one-hour affair. I’ll have to install
a new drive, possibly reinstall Linux, and only then deal with recovering Oracle.
The spare drive I keep on hand for my XP box can come in handy for this scenario
as well. I could use Suse’s software RAID on my Linux box. Maybe someday I will.
For now though, I’m going to use the two drives as separate disks in order to
give me more disk space to play around with. I have a spare Oracle instance
on my XP box that I can use in the event the Linux box is down, so recovering
quickly from the failure of a hard-drive on my Linux box is not a priority.

Some notes here: I have absolutely no clue how to replace just one hard-drive
on my Linux box with a new one. How would I partition the new drive so that
it matched the drive I was replacing? How would I do the low-level format? The
only way I know of to do these things is to reinstall Linux. I should probably
practice this scenario. Another issue: some article topics that I choose do,
in fact, make me very dependent on my Linux box, albeit only for a short period
of time. When I wrote about enterprise
users for Oracle Magazine, I ran the database on Linux and my LDAP directory
server on XP. While writing that article I needed both boxes to continue functioning.

Some other component could fail, leaving the hard drive intact but the computer
unusable — If my XP box goes down and the hard-drive is left intact,
my plan is to take the hard-drive out, move it to my Linux box, and make that
my new XP box. In essence, my Linux box serves as a spare computer. Both boxes
are the same model of Dell Dimension; the only difference being the NIC and
video cards. XP should auto-detect those cards as new hardware, reconfigure
itself, and I should be in business. Hmmm… I wonder whether XP’s product activation
feature would give me grief in this scenario. I’ll try and find time to do a
test-run this week.

In the event that my XP box fails and I also need my Linux box up and running,
I do have a company-issued notebook computer that sits on the shelf when I’m
not travelling, and that I can press into service as a spare. The notebook runs
Windows 2000, and has a version of Microsoft Office on it, and also a copy of
my email program. Other apps that I use are not on the notebook, but are less
critical to my day-to-day work (I hope). The problem would be in getting my
current work files off the hard-drive from the XP box that failed and onto the
notebook’s hard-drive. At CompUSA last week I saw an external hard-drive enclosure
that provided a USB 2.0 (or was it Firewire) interface. If I put my desktop
hard-drive into such an enclosure, I could access that drive from my USB-enabled
notebook. Maybe. I’ve had mixed-luck with USB under Windows 2000. My USB scanner
works; my USB camera doesn’t.

If my Linux box goes down leaving its hard-drives intact, I’ll just order new
parts via overnight deliverey and let it stay down until I can be bothered to
install the new parts.

A peripheral, such as a monitor or a printer, could fail — I have
spare monitors and two printers. Nothing to worry about here that I can think
of.

A housefire, theft, or some other major event could cause me to lose all
my computers, their hard-drives, and any backup media I might have —
This is a scenario I hope I never need to deal with. My current thinking is
that I’ll make archival copies of completed book and article projects and store
them in my mother’s apartment, which is only four blocks away. I’ll do the same
for my photos and web sites; I can tolerate some loss of photo and web site
files in the event of a major disaster such as a fire. Writing files for current
projects are another matter. I’m thinking I can write some sort of script to
copy those to my ISP on a daily basis.

I could get hit by the proverbial truck — This is different. Rather
than lose data, I lose me! This is my least favorite scenario <grin>.
The issue here is that various people depend on my knowledge of my systems:
my wife depends on me to know where our budgeting and money files are, my family
depends on me to know where the photo files are kept, my authors have sent me
their chapters, and my company wishes to ultimately get those chapters. It would
be a nice gesture on my part were I to leave some written documentation of how
I organize book files and such so that in the event of my demise the people
who need to get at those files can.

*   *   *

I don’t think I’ve covered every base yet with respect to the above scenarios.
There are obviously some things I haven’t figured out, and lack of money will
prevent me from covering every possible scenario to the degree that I might
like. For example, I can’t afford, or at least don’t want to afford, RAID on
both my XP and Linux boxes. And if an expensive component gives out on my Linux
box, I probably won’t want to part with the cash to order a new part right away.

Much of my work is deadline-driven. One of my goals in the event a recovery
scenario occurs, is to be able to get on with work quickly. It’s ok if the ultimate
resulotion is a week or more in coming, so long as I can quickly and easily
find some other way to get work done in the meantime. For example, if my Linux
box goes down I usually will be able to get by for some time using the spare
database on my XP box.

This is the last of my "thinking it through" blog entries. I’m beginning
the actual implementation of a recovery plan. I took advantage of my recent
trip to Detroit for MacHack to pick up a CD writer. I’ll install that soon and
use it to make those offsite copies, and to backup files that don’t change often
(photos, old book projects, web sites). And then I need to find even more money
for a RAID card and spare drive for my XP box. I’ll probably test the scenario
of my XP box failing sometime in the next week or two, so that I can see what
happens when I install the intact hard-drive into the box that is now running
Linux. I also need to find some software, or write some scripts, to help me
backup files on a regular basis. I need to automate as much of this work as
I can. The more I have to consciously think about making backups, the less likely
they are to get done.

Have I thought through these scenarios well-enough? Let me know. You can leave a talk-back message in this article, or you can email me at jgennick@oreilly.com.

I’m going to stop yammering about backup and recovery for awhile and talk instead about the Machack conference in Dearborn, since that’s where I’m at today.

After driving down from Munising, I arrived at the conference hotel Wednesday evening around 9:00 PM. When I entered the lobby, I was immediately struck by the number of Mac users. Dozens of users were huddled around various tables in lobby, and also in the various easy-chairs; all seemed to have a Mac, and all were obviously having lots of just plain fun hacking away and talking with others about what they were doing. This was cool, and it’s a kind interaction I don’t often see at the typical corporate conference (such as Oracle OpenWorld) that I usually attend.

Lots of people have their kids with them. At Tim’s keynote I sat behind a father and two sons. All three had Mac OS X notebooks, and the older son was reading our Jython Essentials book and typing in code while Tim talked. Cool! More conferences should allow and encourage young people like MacHack does. I’d have given my eye-teeth when I was young to attend a conference like this with my dad. What a great way for a family to spend time together.

Tim O’Reilly gave the keynote. He started shortly after midnight, after we’d all had our fill of the pizza that Real Software so graciously supplied for us, and between talking and answering questions he spoke until about 2:30 AM. Tim was “piped in” for his talk by a bagpiper who was fully dressed in a kilt and all the accouterments that go with it. MacHack is really fun.

This show is my first time ever to lay hands on a Mac OS X computer. All I can say is that I want one. You’ll laugh, but I found myself constantly right-clicking to bring up a context menu, only there is no right mouse button. Funny how habits like that become so ingrained as to be
second-nature.

Any thoughts about MacHack? Want me to report on anything in particular? Let me know?

I can’t believe a week has gone by since my
first installment in this series. You know, that’s probably why I don’t
have a disaster recovery strategy in place to begin with. When everything’s
working fine, it’s too easy to get caught up in the day-to-day grind and put
off planning for those inevitable problems that will occur at some undetermined
time in the future. If I had a hard drive crash actually scheduled, I’d be sure
to plan for the event, but no one schedules disasters.

Last week I identified the types of files on my systems. Today I want to go
into each of those files types in a bit more depth. I’ll share my thoughts on
how important my different files are, and I’ll think out loud about possible
backup approaches.

Following are my thoughts on protecting the different types of files on my
systems:

Digital photos: I’ll suffer no economic loss from losing my photos,
only emotional pain. I’d prefer not to lose all my photos, but I can
certainly afford to lose some of them. I add new photos on an irregular
basis, usually not more than once per week, and often less frequently than that.

Master copies of my website files:
My live website serves as my backup. If I lose my master copies, I can just
download the files from my ISP and be whole again. Well, almost whole. I’d prefer
not to lose the originals of photos that I’ve cropped, optimized, or otherwise
doctored for my website. My website is relatively stable. I don’t add content
nearly as often as I’d like.

Files for various books and magazine articles that I’ve written in the past:
If the book or article is already published, it’s no great loss if I lose my
electronic copy. These files never get changed. Rarely do I even need to look
at them.

Files for books and articles that I’m currently writing: Now these are
really critical as they represent future income that I depend on. I don’t want
to lose these at all, ever, under any circumstances. I don’t even want to risk
losing a day’s work, because if I have to rewrite something I may not be as
inspired as I was the first time. These files change frequently throughout the
day.

Files for books that I’m editing for O’Reilly: The really critical files
here are those that I’ve edited, or am in the process of editing, which I’ve
not yet sent back to the authors. I don’t want to lose any part of those files
for the same reason I don’t want to lose any part of what I’ve written: it’s
not fun to have to edit a chapter again because you lost the first copy. These
files, ones that I’m actively editing, also change frequently throughout the
day.

Email files, archived emails, email address book: Losing my address
book would be a bit of a blow, because some important (to me) email addresses
only exist in my address book. Each time in the past that I’ve switched email
programs, and thus address books, I’ve managed to lose contact with a few acquaintances.
As for the email itself, it wouldn’t be the end of the world if I lost it all.
Losing some of it would be preferable to losing all of it. Emails change very
frequently. However, I often go weeks at a time without adding to my address
book.

Various personal files such as my resume, address labels, and so forth:
Files such as these can be recreated with a bit of effort. These files tend
to be very stable.

My Oracle database: Believe it or not, I actually have a licensed copy
of Personal Oracle8i. At the moment, I have no data in my database that I can’t
easily recreate. That may change, but for now I’m not going to worry about this.
(Maybe I shouldn’t duck this issue.)

Except for books and articles that I’m actively writing or editing, most of
my collection of important files changes, or gets added to, somewhat infrequently.
In addition, some loss in these areas is acceptable. A reasonable solution might
be to periodically archive these files to some sort of removable media such
as a CD-ROM. If I archived photos, websites, and other such files weekly, I’d
be willing to live with any loss that might result if a hardware failure or
some other disaster occurred during the week.

Files for books and articles that I’m actively editing or writing are another
matter altogether. No loss is acceptable. Well, that’s my ideal. In the real
world, I may have to accept some risk. RAID is suggested here. In the event
of a hard-drive failure, RAID would protect me from losing data that had been
changed since my most recent backup. What RAID won’t protect me from is a site
disaster such as a housefire. RAID isn’t helpful when both mirrors are lost.
Frankly, I’m not sure what to do about the housefire scenario. I’ll likely have
to accept some loss of data. I need to think on this some more.

It looks like my backup solution will involve some sort of removable media
such as CD-ROM along with IDE RAID. I’d prefer DVD to CD-ROM, but when I look
at the market for DVD writers all I see is confusion. DVD is preferable, because
a single DVD holds much more than a CD-ROM (no switching disks in and out),
but which DVD standard should I buy into?

So is this it? Is this my solution? Backup files periodically to removable
media and use RAID to protect myself from losing any changes that get made between
backups. Is that all I need to do? Certainly not. Tomorrow, after I drive down
to Dearborn for MacHack, I’ll take a close
look at the various types of disasters identified in my previous
installment. After all, if a hard disk crashes, I not only need to have
a backup of the files, I need to have somewhere to restore that backup so I
can continue working. I need to think through the details of how I want to recover,
and how fast I want to be able to recover, from each of the scenarios I’ve identified.

Let me know what you think. Is CD-ROM a good media choice for periodic backups? What other choices do I have? And what about that housefire scenario?

I need to develop a good disaster recovery plan for my home office. Recently
I
consolidated two of my office computers so that I now do all my work on just
one PC. I’m also beginning to purchase various office upgrades, such as a Firewire
card, copier, and fax machine.
All this work on infrastructure has gotten me to thinking about how best to
protect myself against loss of data.

One technique I’ve used in the past is to periodically copy important files
to
a ZIP disk. However, that technique has long since become impractical due to
the collective size of my important files. Most recently, I’ve backed up
critical files by copying them back and forth between my two office computers.
But that’s become rather inconvenient since I’ve consolidated
my work onto one PC. Now I often find myself going for days, even weeks, using
the "Hail Mary, please don’t let anything break!" approach to disaster
planning.

Clearly I need a better solution to protecting my data. And I want to take
a
more wholistic approach than just backing up my files. I work from a home
office, and the important thing, really, is not to protect just my files but
rather to protect my ability to do income-producing work. With that in mind,
I’m going to begin by thinking about all the things that might go wrong:

• A major disaster such as a house-fire could cause me to loose everything,
  computers, disks, peripherals, paper files, books, even my office itself.
• My hard-drive could fail, causing me to lose all my data files.
• Some other component in my computer could fail, leaving my data intact on
  the hard-drive, but with no working computer to access that data.
• One of my peripherals, a printer for example, or a monitor, could fail.
• I might accidentally delete a critical file.
• I could get hit by the proverbial truck.

Have I missed anything? I hope not. In a future weblog entry, I’ll be thinking
out loud about how best to protect myself from each of these possible scenarios.
Actually, I shouldn’t say how "best" to protect myself. The issue
is really how do "want" to protect myself. How much disruption do
I want to tolerate and does that change given the circumstances?

Next, it’s worth taking inventory of the different types of electronic files
that I need to protect:

• Digital photos, family pictures and such that I’ve taken with my digital
  camera
• Master copies of my website files
• Files for the various books and magazine articles that I’ve written in the
  past
• Files for whatever book(s) and magazine articles I’m currently writing
• Files for books that I’m editing as part of my day-job
• Email files: archived emails and my address book
• Various personal files such as my resume, address labels, my Microsoft Money
  file, and so forth.

Obviously some files are more important than others, especially with respect
to earning a living. I’ll be thinking outloud about this issue too. The criticality
of certain files, or the lack thereof, certainly must be factored into whatever
disaster plan I ultimately develop.

It’s not just my data that needs protecting either. My daughter has an old
notebook computer on which she writes her journal and various short-stories.
There’s a computer in the dining room that my wife uses for various things.
And I have a Linux box in my office runing a small database that I’d like to
protect. I need a plan or solution that takes all of these disparate computers
into account.

Lastly, I really want an automated, set-and-forget, solution. My daughter hasn’t
backed up her files in months. My wife never does. Even I don’t make backups
often enough. And I certainly don’t want the overhead of having to run around
physically to four different computers in order to copy files to backup media.
The less I need to run around and do things, the more successful my plan will
be.

That’s it for today. In a day or two, I’ll begin drilling down more deeply
into some of the issues I’ve identified here.

What do you think? Have I missed anything? Am I on the right track? If you have a home office, what does your recovery plan look like?