What is rsync

rsync is a tool that copies files from one place to another, much like ftp or scp.

But rsync does more than just copy files blind, it checks the source and destination directories and copies only the files that are new or updated.

The fact that rsync only copies new or updated files make it an excellent solution for projects where new files and directories are being created or updated regularly; or when files and directories can become large—like a static blog site that's being actively maintained.

This realization came to me when I tried using Forge (from the same folks who make the fantastic Hammer for Mac)—dead simple to use; uploading the site is a one click affair; built-in versioning so you can just rollback to any versions of your site at the click of a button. Perfect for landing pages type sites or portfolio sites or product and services sites.

But I hit the snag very early on while sprint churning one post after another. The fundamental problem is that I have to upload the entire site every single time! Never mind that I've optimized my images and switched on Hammer's optimize option when building my static site (in preparation for uploading to Forge), I can't just overlook the fact that the size of the zipped file bundle gets incrementally larger and larger every time. It's that same zipped file bundle that gets uploaded as a version of my entire site.

So now that I've switched back to VPS, I can now do updates on a piecemeal basis. Because of the way rsync copies files to the server, if I add a new post, only a handful of files get transferred. The same is true if I add a class in my CSS file, for example. Only that single CSS file will be copied. So even if my static site will grow to be gigabytes large, new post updates will only be a few kilobytes or, at most, a megabyte or so transfer.

Keep in mind, though, that every first time synching will always be the largest file transfer you will get with rsync—it makes sense, since all files will be new from the perspective of the server—but it's smooth sailing immediately after that.

How I use rsync

The whole point is to sync the files and directories from my local development machine (on the left) to the server (on the right), so the rsync command is local-centric in my case.

Command breakdown

-a Archive mode ensures that symbolic links, devices, attributes, permissions, ownerships, modification times, etc. are preserved in the transfer.

-v Verbose mode displays additional information in the terminal when rsync is running.

-z Compressed mode compresses data during the transfer, making it even conserve more bandwidth.

-r Recursive mode tells rsync to copy directories recursively.

--delete tells rsync to delete files in the destination directory that do not exist in the source directory anymore. This is what makes it a proper sync.

. means current directory. In my case, it's an alias for /Users/islandjoe/Sites/compendiumxyz/Build. This is why I made sure I'm inside Build directory first before I issue the command because if I was in some other directory, the wrong files will be copied to the server. If this is too hairy for you, you can always use the proper path (i.e., /Users/yourusername/Sites/awesomeblog/Build) instead of ..

Command breakdown of the literal kind

Alas, it's not as frictionless as I expected it to be. It turns out that *nix interoperation is always gnarly (my local machine is a Mac and my server droplet is an Ubuntu Linux). The rsync command above gave me two error messages, both of which are permission problems:

  1. rsync: failed to set times on "/srv/www/.": Operation not permitted (1)
  2. rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(992) [sender=2.6.9]

Both errors are a result of using -a (archive mode). Archive mode preserves the original files' and folders' modification time and the file permissions when copying them over. It so happens that some servers block the changing of folder modification time and file permissions, and this is exactly what's happening here. As long as they do not affect the operation of the website in any way—and here it does not— it's safe to suppress modification time and file permission when remote synching.

So below is the final rsync command that I use that suppresses those two errors:

-O omit directories when preserving modification time on the server side. This solves the first error. You can also use --omit-dir-times.

--no-perms omit file permissions when preserving file permissions on the server side. This solves the second error.

So this is now what I see when I issue the command:

Update Exclude .DS_Store files from being uploaded

As it turns out, OS X creates .DS_Store files everywhere, and that includes the Build folder. The problem with this is that they just appear in your folders, and they being hidden files you wouldn't notice until you see in your terminal output that .DS_Store files have also been synched. I do not want them files in my server!

Fortunately, rsync has a switch for just that in the form of --exclude

So here is the command in its entirety:

rsync -avzrO --no-perms --delete . compendium.xyz:/srv/www --exclude=".DS_Store"