Replace rsync with lftp for remote file copies on Linux

rsync is an awesome tool that I have used extensively. However, I think it is time to explore other tools that can replace some of what rsync has been doing for so many years. I’m not saying you should never use rsync. What I am saying is there are other options, and in some circumstances, rsync isn’t the default choice anymore (at least for me).

I went looking for an rsync replacement because I was seeing extremely slow transfer times when moving large data files. rsync didn’t seem to take full advantage of a 1 Gbps link, and transfers were taking longer than expected. It also doesn’t run parallel syncs natively, which can be frustrating when dealing with many files or very large ones. Writing scripts with workarounds becomes cumbersome.

Here’s one example workaround from Stack Overflow:
Speed up rsync with simultaneous/concurrent file transfers

While exploring alternatives, I discovered lftp, and it’s what I eventually settled on. As a bonus, lftp works as a drop-in FTP client too. If you have legacy applications that expect /usr/bin/ftp, you can just symlink it to /usr/bin/lftp.

While researching, I realized lftp has tons of features—so many that it can get confusing. Read the man page carefully.

From the man page:

lftp has shell-like command syntax allowing you to launch several commands in parallel in background (&). It is also possible to group commands within () and execute them in back‐ ground. All background jobs are executed in the same single process. You can bring a fore‐ ground job to background with ^Z (c-z) and back with command wait' (or fg’ which is alias to wait'). To list running jobs, use command jobs’. Some commands allow redirecting their output (cat, ls, …) to file or via pipe to external command. Commands can be executed con- ditionally based on termination status of previous command (&&, ||).

An example lftp command that will use sftp to mirror a remote directory using shared keys

lftp -c "open sftp://username@server.domain.tld:/path/to/data/ && mirror --parallel=5 --verbose --delete && exit"

There are plenty of examples online, so I won’t go deep into command variations. But if you’ve never used lftp, it’s definitely worth checking out.