Fixing rsync on MacOS X 10.4 (Tiger)

Starting with MacOS X 10.4 the included version of rsync has been made HFS+ aware. This means that when files are copied from a HFS+ file system to a volume of some other type is it able to preserve the Extended Attributes (including the resource fork) that have been applied to the file.

At least that is the theory. Unfortunately the version of rsync that ships with 10.4 (at least up to 10.4.3) is far from ready for production use. Apple's "enhancements" to rsync have introduced several bugs when using extended attribute support, some simply cause rsync to crash, while others are more subtle, and may not be obvious to the user until they need to restore from their backup and find that things aren't quite as they should be.

Rather than go into great detail about the bugs and problems that I encountered and my attempts to fix them, here is a short list of the things that should be addressed when using this version of rsync with the --extended-attributes option flag:

* Note: While this works for me, use it at your own risk. The same could be said for the version included with tiger.

Downloading

Just the rsync universal binary file: rsync-2.6.6-bin-ea-20060818.tar.gz
(extract into /usr/local/bin)

The prepatched & precompiled source: rsync-2.6.6-ea-20060818.tar.gz
(run 'sudo make install' to install into /usr/local/bin)

To make a custom build from the baseline rsync code use this patch: rsync-2.6.6-ea-20060818.diff

Installing

If you download the binary file only, just extract it and copy the 'rsync' file to somewhere in your path like '/usr/local/bin'

For the precompiled source run 'sudo make install' and it will install the necessary files into '/usr/local/bin'

If you need to build it from scratch, or you wish to patch another system so that you can use the '--extended-attributes' option flag you will need to use the '--enable-ea-support' when running ./configure.

If you are compiling from scratch on a Darwin / MacOS X system you may also need to download and install into your include path the file 'copyfile.h' that is part of the darwin Libc source bundle. This is available from the apple developer site.

Using rsync

This version of rsync aims to be a direct replacement for the one included with Tiger, which means that the '-E' or '--extended-attributes' flag is supported. Using this option still requires that any remote host be running a patched version of rsync for this feature to work. The reason for this is somewhat trivial, but unfortunately unavoidable.

Note that while it is not necessary to patch remote systems running tiger, mixing the two versions is not recommended as there are bugs fixed in this version that will not be resolved by patching only one end.

If you are connecting to a non OSX system, also included are two new options '-X' or '--hfs-export' and '--hfs-import'. The option '--hfs-export' will allow rsync to export the hfs extended attributes in the same way the '--extended-attributes' option does, but without needing to patch the destination host. The files are sent as they would be when copying to any non HFS+ volume on the local system, creating a separate '._' prefixed file for each file with metadata. To successfully restore the files back to a HFS+ volume you need to use the '--hfs-import' option, which will rejoin the data and metadata forks into a single file.

If you have copied files from an exported volume without using the --hfs-import flag and you end up with a bunch of ._<blah> files on your HFS+ volume, don't worry. Simply run the command /System/Library/CoreServices/FixupResourceForks <blah> on the target file or directory in question to merge the file data and resources back together.

It order to avoid the unnecessary recopying of metadata it is also possible to enable checksums for metadata only using the '--ea-checksum' option flag. This will attempt to determine if the metadata has changed by exporting the extended attributes to an appledouble encoded file and performing a CRC32 checksum on the file, the same operations is performed on the existing target file and compared. For various reasons this isn't always able to determine 100% if the attributes are unchanged, so some files will still have their metadata recopied. There is an overhead involved with doing this checksum, but for the most part it shouldn't be any more expensive than recopying all metadata everytime.

Caveats

The use of the '--inplace' option flag is fundamentally incompatible with '--extended-attributes' when targeting non HFS+ volumes so using the two options together has been disabled.

When resyncing an existing backup it is necessary to use the --delete option flag to remove any extended attributes from files that are unchanged but no longer have these extended attributes.

Extended attributes for source files on HFS+ volumes will be recopied for any file that has a last change time more recent than the files modification time (this means most of them). For various reasons I won't go into, there is no easy way for rsync, using unix system calls, to identify metadata changes between two instances of a file without physically comparing the metadata, or a checksum of it. I do not believe there is a way around this problem without resorting to using Carbon function calls.

Todo

Copyright (c) 2005, 2006 - Quinton Dolan q@onthenet.com.au