Backups?

[ciphergoth]

babysimon reacted with horror when I told him at the weekend that we still hadn't sorted backups out, and pressed a spare 250GB USB external drive into my hands.

What's everyone's favourite way of doing backups? We have a Windows and a Linux laptop to back up, both are switched on at unpredictable times. I just had a go at making brad's "brackup" go, but despite the great attraction of GPG-encrypted backups, it doesn't seem to be ready for prime time yet - the documentation isn't there yet, and it creates zillions of tiny encrypted files. I am resisting the temptation to write my own Python-based alternative. What do you use?

Update: I since found this rather nice solution.

Comments

[mskala.livejournal.com]

My home machine's main partitions are mirrored to two drives with software RAID-1. That protects against the most typical kind of hardware failure, where one drive dies. If that happens I can stick in a new drive and be up and running in a day with no loss. Important files (e.g. my thesis) get rsynced to another machine by a daily cron job. That protects against a lot of the scenarios the RAID doesn't. In the case of my thesis, it's going from my desktop machine at school to a fileserver which is itself backed up by the systems people here. The basic design goal: the backups have to happen without my intervention because if I were required to do something regularly, it wouldn't happen.

Then on a much slower schedule, when I feel like doing it, I burn either my home directory or my entire file tree to write-once optical media (CDs until recently, DVDs now). That's both for backup and as an archive, so I can go back and look at what I was doing in the past.

These measures have seen me through a number of hardware failures, software failures, and human errors over the years. They're certainly not perfect, but the big thing is that I can really keep them up on an ongoing basis, unlike some of the other backup measures I've seen people attempt, which end up not really being sustained.

[ciphergoth.livejournal.com]

Anything *really* important - eg any recent code that I've more than fiddled with - will be in Mercurial, and there will be copies at home and at work. When the laptops were stolen, the BiCon accounts were saved by being in Monotone. But it's stuff like photos and music which take up a lot of space and aren't a good match for DVCS.

[juliet]

I *think* you can make Bacula do backups for things like laptops that aren't always-on (I use this at work; might be overkill at home though).

I used to use rsync (+ anacron would do the trick) on Mac. No idea at all about Windows. (I use Time Machine now on the Mac, which rocks, but that's no use to you!).

[alexmc.livejournal.com]

I've been thinking of using Bacula for ages but have not yet found the time to learn how to use it properly.

[werenerd.livejournal.com]

I keep everything backed up on gmail. Most files are things I send anyway. The rare file that I'm not sending anywhere, I'll send it to myself if it's important.

[autopope.livejournal.com]

Time Machine on the Macs, rsync to keep everything in step across platforms and on the Linux boxen.

IIRC somone was trying to implement a Linux equivalent of TM -- ah, here we are: FlyBack. Caveat: I haven't used this (yet).

[topbit.livejournal.com]

Looks a lot like rsnapshot, but with a prettier front end as opposed to a .conf file and crontab entries.

[figg.livejournal.com]

I've lost all my email before, but I guess using gmail means it will be unlikely.

Most of the code I have written I have uploaded elsewhere, so I'm not too attached to it either.

I've already lost my entire music collection twice, so that isn't too bad.

[lovelybug.livejournal.com]

I've already lost my entire music collection twice

Me too. Very annoying.

[topbit.livejournal.com]

My home server has software mirroring, and I also use http://www.rsnapshot.org/ - Rsync/ssh & hardlink based backups. I've got it to run 6 times per day, which are then promoted daily/weekly & monthly. The hardlinks mean that only new/updated files change from backup to backup. It also means I could pull from the backup files that were changed at the end of October. (and I have recovered a file I'd thought I'd lost from it before now as well - just a matter of find it in the hierarchy, and copy).

It's not so useful for very large files that change often (like video files), so I exclude those directories, but for smaller text files, and long-term static files that aren't intrinsically huge, it's a great option. I've got my /etc/ and the websites I'm developing in it.

Another option as well is an Amazon S3 account, but for that there is a Firefox plugin to upload & set permissions.

[ciphergoth.livejournal.com]

With rsnapshot the server initiates the backups, which is less useful when the clients aren't always-on.

[zz]

rsync, internally inside my server, and using cygwin/cwrsync from windows machines from a batch file using windows' scheduling thingy.

[http://users.livejournal.com/_lj_sucks_/]

rdiff-backup and Time Machine.

[kimble.livejournal.com]

Everything of importance on RAID-1 in the server (the RAID's more about preventing downtime than anything else), nightly rsync scriptage roughly equivilent to rsnapshot doing incremental backups to a dedicated disk, and monthly manually-invoked (a cron job nags daily if more than 28 days since the last run) rsync from there to an external disk that lives on a shelf.

In combination with a lack of (mobile) laptops and sufficient paranoia to ensure that everything of importance is actually on the server.

[reddragdiva]

*cough* still nothing. I should really run rsync to the house server from crontab on each of the laptops.

[reddragdiva]

Right, set up:

0 0,6,12,18 * * * /usr/bin/rsync -rl --password-file=****** /home/fun/ fun@192.168.1.3:/trollop/David/

Ubuntu includes anacron, so it will run fine.

(Kubuntu 8.04 comes with rsync 2.6.9 and I hand-installed rsync 3.0 on the server from source, but they seem to talk OK.)

[bohemiancoast.livejournal.com]

Time Machine. It's amazingly reassuring.

[hairyears.livejournal.com]

Low tech. I do a manual backup onto an 80-Gig USB drive in the weekly 'maintenance hour' - a time I set aside for updates and the scheduled virus scan.

Just about any USB drive you buy will have rudimentary software for an advanced file copy operation from MyDocuments and any other folders you specify, eliminating duplicates and version-marking (or overwriting) updated files.

[daneel-olivaw.livejournal.com]

Cronned rsync. You can get an version of rsyncd that runs on Win boxes. I have a dumb linux box that grabs a copy of all the relevant data once per hour, and then syncs the lot overnight to somewhere offsite and trustworthy.

[henry_the_cow]

Personally I'm completely crap at backups - I copy photos onto CD-Rs, mail onto a USB stick, and music onto my MP3 player. But that's not why I'm posting here.

Several of my better organised friends use the <A HREF="http://www.cis.upenn.edu/~bcpierce/home.html>Unison</A> file synchroniser to sync between different machines, e.g. work and home machines. Updates on either machine are propagated to the other.

[hughe.livejournal.com]

i nightly cron an rsync from live disk to backup disks then unmount backup disk.

found it more useful than raid for when I decide to delete things i don't mean to.

I've always meant to do a incremental backup system, but never got round to it.

[babysimon]

Although BackupPC has the annoying property that the clients have to trust the server, in other respects it's pretty spot on. It does all the same-file-unification, has a nice web GUI, works with both OSs, copes well with intermittently disconnected clients and you can ask me if you have any problems ;)

It also was obviously developed by someone who'd been bitten; it emails you automatically if it detects it's not working. Useful :)

[ciphergoth.livejournal.com]

BackupPC seems like the way to go - will sort it out. Thanks.

But looking at the answers here, it seems everyone has their own not-entirely-satisfactory string-and-sealing-wax solution. Maybe I really should think about writing an alternative.

[emarkienna.livejournal.com]

Mainly it's just manually/when-I-feel-like-it onto DVDs, but the set of things which are important and need regular backups (code, emails, photos etc) all fits onto a DVD each time (although I sometimes do photos separately). It's low tech, but it works :)

For one off things like mp3s, videos and software, I backup to 1 or preferably 2 CD/DVDs (my mp3s are also backed up onto 2nd hard disk, and my laptop).

Oh, and I always keep a recent backup DVD in my bag in case my house blows up or something.

More frequent backups I sometimes do onto my second hard disk or my laptop (either using the clunky Windows Backup - I'm sure there's better software - or just a manual copy).

[adq.livejournal.com]

For stuff that we just want a copy of with no history: selected rsync of important dirs (centralised video, mp3 archives and so on) to another machine in the flat (most machines backup to the server, the server backs itself up to the media PC).

For stuff that we want a revision history of (laptop /home, /etc and so on dirs): rdiff-backup to another machine in the flat.

For encrypted hard disk volumes: truecrypt as it is opensource and cross platform. Was nearly burned by the proprietary PGPDisk a few years back.

As for offsite: not sorted that out yet, but I tend to just burn loads of single layer DVDs out every so often, and ask a trusted someone to keep a copy in another flat.

[adq.livejournal.com]

Oh - if you need to do incremental backups of a lot of data, look into DAR (equivalent of TAR for disk oriented backups)

[phantas.livejournal.com]

Time machine and my own hack for secure remote daily backups of critical directories (Linux and Mac): link to the tar.gz

It's still a hack and is missing an easy and friendly way to recover the files as well as a database for the metadata. But it does what needs to be done: blocks are already encrypted when rsynced into the remote location and it's sufficiently broken down for close to minimum updating without leakage.

It's an ongoing project but it's very much on standby for the moment.:(

[phantas.livejournal.com]

explanation behind this: I don't have an home server. Instead I have a dreamhost account which I intend to swap for a slicehost account at some point. I prefer that a machine that is not under my sight and to which I'm not the sole root (I can't remember how VPM works in that sense) does not contain *at any stage* my data unprotected.

I use laptops so I can't rely on RAIDS or anything that needs my intervention for backups of frequently changing and critical data/directories.

[alexmc.livejournal.com]

Personally I rsync everything to another box - usually over ssh if it is going over public wire.

I use zrm for MySQL backups (zmanda)

[drdoug.livejournal.com]

If I told babysimon that I haven't sorted backups out, would he give me a spare 250Gb USB external drive too? Cos that would be cool.

Anything very important at home gets sent to the four winds (various online places and my work system of mirrors and centrally backed-up shares). For the rest, I rely on burning DVDs from time to time - the vast majority of stuff is photographs and videos. The really good photos are all online, and the ones that would make me cry if they were lost have been printed in multiple copies and circulated widely to friends and family in hard and electronic form.

My current music backup system is the original CDs. (Plus burned DVDs of the relatively small amount of stuff I've downloaded.) This made good sense originally but is now looking creaky. (Ye ghods it'd take forever to rip that lot again and get the metadata in to usable form.) Which is why I'm thinking an external drive (or something networky) would make sense. But I'm also waiting for Freesat PVRs to become available and would like a network backup store to talk to that too, which is an excuse for doing nothing yet ... but not a good one.

[babysimon]

If I told babysimon that I haven't sorted backups out, would he give me a spare 250Gb USB external drive too? Cos that would be cool.

No, I only had one spare. Also I'm sober now.

[lovingboth]

1. Email - all domains are set up to send incoming email to the relevant inbox and one of several Gmail accounts. This has the useful side-effect of being able to read email easily when away.

2. Ian generated content - still fits on a blank DVD-5 and it's easy to do that. What I want is a good offsite backup for this.

3. Disk images, especially of Windows OS / programs on USB discs. Too many backup 'solutions' leave you with the files there, but not with the right permissions / in the right place / whatever for the results to actually work.

4. 'Media files' - the vast majority of my disc space. In many cases, it would be re-torrentable, but the proportion of files this is true for became somewhat less likely with the end of Demonoid. USB discs seems to be the way to do this, but I'm still not happy.

5. For the past few months, I've started doing at least two copies of any DVDs/CDs burnt in case one is damaged/lost. This was prompted by the discovery that Post-it notes take the metal layer off one particularly bad set of CD-Rs I used a (thankfully small) batch of at one point around 2000. Ghod knows what they used to protect it. Another, even older batch of CD-Rs has developed data read problems bad enough to make CD audio audibly bad.
All copies are still stored locally, but in case of catastrophe, having lost a batch of copied DVDs is not going to be my main problem.


What I liked about the concept of brackup was that it a) used offsite storage that I trust will still be there when needed (Amazon - Google would be an alternative, but I don't think I've seen a commercial specialist offsite backup company that I would bet my data on) and b) did it in such a way that it didn't matter whether or not they were being co-operative to anyone wanting to see the contents. It's been some months since I had a look, but didn't it also keep older versions of files?

What I didn't like is that it didn't seem to work in Windows and, alas, I'm still using that enough of the time to want a Windows client.

I also want something that doesn't need a third of the local disc to be free - the less free space is needed, the better. And, unlike rsync, it has to be smart enough not to delete the backups of files that I've just deleted.

[skibbley.livejournal.com]

Still thinking about it for laptops and some desktops (considering moving a virtualised image onto an always-on backed up machine then back to the desktop). At work we use tivoli.

[jhg.livejournal.com]

I haven't either. Can I have a spare 250GB USB drive please?

[damerell.livejournal.com]

3 250Gb externals, rsynced onto as and when (that's the bad bit). Most recently written one stays with me; the previous one goes to Sarah's as an offsite.

[damerell.livejournal.com]

PS I still owe you a wodge of money.