The Linux Page

Backing up a directory with really large using rsync

I had a really bad SSD drive that had many sectors go bust. Although overall the drive was still quite functional, the number of sectors going bad was growing pretty quickly (from like 100 it quickly jumped to 1,000).

So I decided to replace the drives and as I was doing so, I also wanted to have a full backup of my /var directory. That includes databases and some other really large files. So to do the copy, I wanted to use rsync for two reasons, it is capable of restarting a copy in case it fails in the middle (imagine a copy that takes 12 hours... what are the chances that something breaks in between?)

So I used the following command:

sudo rsync --partial --info=progress2 -a /var /var2

The /var2 is a mount point on a different HDD than the /var HDD.

The command uses the following options:

  • --partial — this says that whenever a large file exists on both sides, try to continue the copy instead of starting from scratch
  • --info=progress2 — show progress, as I mentioned above, 12h copy takes a very long time and it's really nice to see whether you've copied 1% or 99%; for small copies, this is probably less useful
  • -a — archive mode; this includes -rlptgoD
    • -r — recursive mode; copy folders and they content
    • -l — copy symlinks as symlinks (so do not follow symlinks, as a side effect)
    • -p — preserve permissions; be careful with that one if you have multiple OSes accessing the same file system
    • -t — preserve modification times (i.e. do not timestamp the files at the time they are being copied)
    • -g — preserve group
    • -o — preserve owner
    • -D — same as --devices and --specials; a.k.a. preseve all special files

This has proven to be very useful and quite sufficient.

If you want to make constent backups, especially if it includes /var/log. you may want to consider deleting files that were removed in the source. This is done with the --delete option. There are a few variations of that option to delete before, after, during the transfer. If your copy is done at regular interval, the variations probably do not matter as much.