syq vs rclone

Preliminary benchmarks · syq 0.5.2 and rclone 1.75.1

If you know how to tune rclone for better performance in these tests, please open an issue with your suggestions.

File copies over a long-distance connection, between nearby servers, and on local and NFS filesystems.

WAN: one large file WAN: multiple files Measurement data (JSON)

Bars show dataset bytes / mean elapsed time; thin lines show the run range. Dashed lines mark nominal link ceilings. Run counts appear beside each speed. Commands use placeholder paths and servers.

LAN

LAN: 600,000 small files

600,000 × 4 KiB files in 256 directories (2.29 GiB total). RAID → RAID, approximately 0.2 ms RTT; both servers use 2 × 10 Gb/s bonds.

Port checks confirmed that rclone SFTP and WebDAV both use the bond’s two ports.

A. syqAutomatic settings

109.1 MB/s22.517 s mean · 3 measured copies

B. rclone / SFTP--transfers 64 --timeout 60s

28.0 MB/s87.787 s mean · 3 measured copies

C. rclone / HTTPS WebDAV--transfers 64 --webdav-pacer-min-sleep 0s --timeout 60s

56.7 MB/s43.320 s mean · 3 measured copies

syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_SCRATCH/syq -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATION

rclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --timeout 60s --retries 1 --stats 5s --stats-one-line --stats-log-level NOTICE --no-check-dest --sftp-host SERVER --sftp-port SSH_PORT --sftp-user USER --sftp-key-file KEY_FILE --sftp-known-hosts-file KNOWN_HOSTS --sftp-md5sum-command md5sum --sftp-sha1sum-command sha1sum --transfers 64 :local:SOURCE :sftp:DESTINATION

rclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --timeout 60s --retries 1 --stats 5s --stats-one-line --stats-log-level NOTICE --no-check-dest --webdav-url https://SERVER:HTTPS_PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 64 --webdav-pacer-min-sleep 0s :local:SOURCE :webdav:DESTINATION

Individual runs and verification

A. syq

Run 1: 105.6 MB/s · 23.267 s. Contents and stated metadata checked.
Run 2: 111.9 MB/s · 21.967 s. Contents and stated metadata checked.
Run 3: 110.1 MB/s · 22.317 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds; file and directory membership. Requested cache preparation: source files read before every timed copy. Final durability flush: no.

B. rclone / SFTP

Run 1: 29.7 MB/s · 82.875 s. Contents and stated metadata checked.
Run 2: 25.5 MB/s · 96.475 s. Contents and stated metadata checked.
Run 3: 29.3 MB/s · 84.010 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds; file and directory membership. Requested cache preparation: source files read before every timed copy. Final durability flush: no.

C. rclone / HTTPS WebDAV

Run 1: 53.6 MB/s · 45.870 s. Contents and stated metadata checked.
Run 2: 58.7 MB/s · 41.870 s. Contents and stated metadata checked.
Run 3: 58.2 MB/s · 42.219 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds; file and directory membership. Requested cache preparation: source files read before every timed copy. Final durability flush: no.

WAN

WAN: one 8 GiB file

One 8 GiB regular file. Flat directory, warm tmpfs source → remote XFS RAID; approximately 250–265 ms RTT. Sender NIC: 1 Gb/s (125 MB/s nominal ceiling).

Three verified runs per setting. HTTPS WebDAV was the faster single-file rclone backend tested.

Nominal sender-link ceiling: 125.0 MB/s (dashed line; before protocol overhead).

A. syqAutomatic settings

98.5 MB/s87.207 s mean · 3 measured copies

B. rclone / HTTPS WebDAV--transfers 16 --webdav-pacer-min-sleep 2ms

105.0 MB/s81.824 s mean · 3 measured copies

env SYQ_DEBUG=1 syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_SCRATCH/syq -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATION

rclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --retries 1 --stats 10s --stats-one-line -vv --webdav-url https://SERVER:HTTPS_PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 16 --webdav-pacer-min-sleep 2ms --no-check-dest :local:SOURCE :webdav:DESTINATION

Individual runs and verification

A. syq

Run 1: 99.0 MB/s · 86.724 s. Contents and stated metadata checked.
Run 2: 99.2 MB/s · 86.624 s. Contents and stated metadata checked.
Run 3: 97.3 MB/s · 88.274 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.

B. rclone / HTTPS WebDAV

Run 1: 105.7 MB/s · 81.274 s. Contents and stated metadata checked.
Run 2: 104.6 MB/s · 82.123 s. Contents and stated metadata checked.
Run 3: 104.7 MB/s · 82.074 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.

WAN: 32 × 1 GiB files

32 × 1 GiB regular files. Flat directory, warm tmpfs source → remote XFS RAID; approximately 250–265 ms RTT. Sender NIC: 1 Gb/s (125 MB/s nominal ceiling).

Three verified runs per setting. syq’s automatic tuning cache changed worker counts between runs.

Nominal sender-link ceiling: 125.0 MB/s (dashed line; before protocol overhead).

A. syqAutomatic settings

105.2 MB/s326.516 s mean · 3 measured copies

B. rclone / SFTP--transfers 32 --sftp-chunk-size 255Ki --sftp-concurrency 256

88.4 MB/s388.873 s mean · 3 measured copies

C. rclone / HTTPS WebDAV--transfers 16 --webdav-pacer-min-sleep 2ms

110.6 MB/s310.548 s mean · 3 measured copies

env SYQ_DEBUG=1 syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_SCRATCH/syq -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATION

rclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --retries 1 --stats 10s --stats-one-line -vv --sftp-host SERVER --sftp-port SSH_PORT --sftp-user USER --sftp-key-use-agent --sftp-known-hosts-file KNOWN_HOSTS --sftp-md5sum-command md5sum --sftp-sha1sum-command sha1sum --sftp-chunk-size 255Ki --sftp-concurrency 256 --no-check-dest --sftp-key-use-agent=false --sftp-key-file KEY_FILE --transfers 32 :local:SOURCE :sftp:DESTINATION

rclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --retries 1 --stats 10s --stats-one-line -vv --webdav-url https://SERVER:HTTPS_PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 16 --webdav-pacer-min-sleep 2ms --no-check-dest :local:SOURCE :webdav:DESTINATION

Individual runs and verification

A. syq

Run 1: 108.8 MB/s · 315.846 s. Contents and stated metadata checked.
Run 2: 106.6 MB/s · 322.350 s. Contents and stated metadata checked.
Run 3: 100.7 MB/s · 341.352 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.

B. rclone / SFTP

Run 1: 87.7 MB/s · 391.856 s. Contents and stated metadata checked.
Run 2: 90.3 MB/s · 380.654 s. Contents and stated metadata checked.
Run 3: 87.2 MB/s · 394.110 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.

C. rclone / HTTPS WebDAV

Run 1: 111.7 MB/s · 307.594 s. Contents and stated metadata checked.
Run 2: 109.9 MB/s · 312.600 s. Contents and stated metadata checked.
Run 3: 110.3 MB/s · 311.450 s. Contents and stated metadata checked.

Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.

Mounted NFS

Small files → mounted NFS

40,000 × 4 KiB files in 256 directories. Local /tmp source → mounted NFSv4.2 destination.

Three runs per setting. NFS server caches and other clients were uncontrolled.

A. syqAutomatic settings

22.0 MB/s7.441 s mean · 3 measured copies

B. rclone / local filesystemDefault performance settings

2.4 MB/s68.326 s mean · 3 measured copies

C. rclone / local filesystem--transfers 64

7.9 MB/s20.730 s mean · 3 measured copies

D. rclone / local filesystem--transfers 256

6.7 MB/s24.465 s mean · 3 measured copies

syq rsync -a --no-progress SOURCE/ DESTINATION

rclone copy --config /dev/null --metadata --create-empty-src-dirs -- :local:SOURCE :local:DESTINATION

rclone copy --config /dev/null --metadata --create-empty-src-dirs --transfers 64 -- :local:SOURCE :local:DESTINATION

rclone copy --config /dev/null --metadata --create-empty-src-dirs --transfers 256 -- :local:SOURCE :local:DESTINATION

Individual runs and verification

A. syq

Run 1: 21.2 MB/s · 7.726 s. Contents and stated metadata checked.
Run 2: 22.8 MB/s · 7.175 s. Contents and stated metadata checked.
Run 3: 22.1 MB/s · 7.424 s. Contents and stated metadata checked.

Metadata checked: regular files/directories: type, mode, uid, gid, mtime_ns. Requested cache preparation: evict. Final durability flush: no.

B. rclone / local filesystem

Run 1: 4.5 MB/s · 36.775 s. Contents and stated metadata checked.
Run 2: 2.6 MB/s · 62.819 s. Contents and stated metadata checked.
Run 3: 1.6 MB/s · 105.384 s. Contents and stated metadata checked.

Metadata checked: regular files/directories: type, mode, uid, gid, mtime_ns. Requested cache preparation: evict. Final durability flush: no.

C. rclone / local filesystem

Run 1: 11.6 MB/s · 14.084 s. Contents and stated metadata checked.
Run 2: 11.8 MB/s · 13.934 s. Contents and stated metadata checked.
Run 3: 4.8 MB/s · 34.170 s. Contents and stated metadata checked.

Metadata checked: regular files/directories: type, mode, uid, gid, mtime_ns. Requested cache preparation: evict. Final durability flush: no.

D. rclone / local filesystem

Run 1: 7.6 MB/s · 21.595 s. Contents and stated metadata checked.
Run 2: 6.6 MB/s · 24.698 s. Contents and stated metadata checked.
Run 3: 6.0 MB/s · 27.102 s. Contents and stated metadata checked.

Metadata checked: regular files/directories: type, mode, uid, gid, mtime_ns. Requested cache preparation: evict. Final durability flush: no.

Small files ← mounted NFS

400,000 × 4 KiB files in 256 directories (1.53 GiB total). Mounted NFSv4.2 source → local XFS RAID destination.

Source file contents and metadata were read before every timed copy; each destination started empty.

A. syqAutomatic settings

70.3 MB/s23.297 s mean · 3 measured copies

B. rclone / local filesystem--transfers 64

37.6 MB/s43.528 s mean · 3 measured copies

syq rsync -a --no-progress SOURCE/ DESTINATION

rclone copy --config /dev/null --metadata --create-empty-src-dirs --transfers 64 --stats 5s --stats-one-line --stats-log-level NOTICE :local:SOURCE :local:DESTINATION

Individual runs and verification

A. syq

Run 1: 78.2 MB/s · 20.943 s. Contents and stated metadata checked.
Run 2: 61.0 MB/s · 26.851 s. Contents and stated metadata checked.
Run 3: 74.1 MB/s · 22.096 s. Contents and stated metadata checked.

Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; file contents SHA256; root excluded. Requested cache preparation: source file contents and metadata read before every timed copy. Final durability flush: no.

B. rclone / local filesystem

Run 1: 52.8 MB/s · 31.012 s. Contents and stated metadata checked.
Run 2: 31.4 MB/s · 52.238 s. Contents and stated metadata checked.
Run 3: 34.6 MB/s · 47.333 s. Contents and stated metadata checked.

Filesystem and network capabilities

Local filesystem

Large file on the same filesystem

One 2 GiB file. Source and destination on the same XFS RAID filesystem.

Kernel-assisted copying may share extents; these rates are not physical disk bandwidth.

A. syqAutomatic settings

33.45 GB/s0.064 s mean · 1 measured copy

B. rclone / local filesystemDefault performance settings

593.3 MB/s3.620 s mean · 1 measured copy

C. rclone / local filesystem--multi-thread-streams 8

569.6 MB/s3.770 s mean · 1 measured copy

syq rsync -a --no-progress SOURCE/ DESTINATION

rclone copy --config /dev/null --metadata --create-empty-src-dirs -- :local:SOURCE :local:DESTINATION

rclone copy --config /dev/null --metadata --create-empty-src-dirs --multi-thread-streams 8 -- :local:SOURCE :local:DESTINATION

Individual runs and verification

A. syq

Run 1: 33.45 GB/s · 0.064 s · under 2 s. Contents and stated metadata checked.

Metadata checked: regular files/directories: type, mode, uid, gid, mtime_ns. Requested cache preparation: evict. Final durability flush: no.

B. rclone / local filesystem

Run 1: 593.3 MB/s · 3.620 s. Contents and stated metadata checked.

Metadata checked: regular files/directories: type, mode, uid, gid, mtime_ns. Requested cache preparation: evict. Final durability flush: no.

C. rclone / local filesystem

Run 1: 569.6 MB/s · 3.770 s. Contents and stated metadata checked.

Metadata checked: regular files/directories: type, mode, uid, gid, mtime_ns. Requested cache preparation: evict. Final durability flush: no.

Multi-rail LAN

syq uses all eight rails of this multi-rail LAN; rclone uses only one, even with multiple files and increased transfer concurrency.

Fast fabric: one large file

One 8 GiB file, tmpfs → tmpfs between servers in the same data center.

Timing includes an empty fixture probe.

A. syqAutomatic settings

1.63 GB/s5.273 s mean · 1 measured copy

B. rclone / SFTPDefault performance settings

276.1 MB/s31.116 s mean · 1 measured copy

C. rclone / SFTP--sftp-chunk-size 128Ki --sftp-concurrency 128

279.2 MB/s30.765 s mean · 1 measured copy

D. rclone / HTTPS WebDAV--transfers 16 --webdav-pacer-min-sleep 0s

279.7 MB/s30.715 s mean · 1 measured copy

env SYQ_DEBUG=1 syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_SYQ -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATION

rclone copy --config /dev/null --sftp-host SERVER --sftp-known-hosts-file KNOWN_HOSTS --sftp-user USER --sftp-host SERVER --sftp-port PORT --sftp-user USER --sftp-key-file KEY_FILE --sftp-known-hosts-file KNOWN_HOSTS --create-empty-src-dirs --contimeout 5s --retries 1 -- :local:SOURCE :sftp:DESTINATION

rclone copy --config /dev/null --sftp-host SERVER --sftp-known-hosts-file KNOWN_HOSTS --sftp-user USER --sftp-host SERVER --sftp-port PORT --sftp-user USER --sftp-key-file KEY_FILE --sftp-known-hosts-file KNOWN_HOSTS --create-empty-src-dirs --contimeout 5s --retries 1 --sftp-chunk-size 128Ki --sftp-concurrency 128 -- :local:SOURCE :sftp:DESTINATION

rclone copy --config /dev/null --webdav-url https://SERVER:PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 16 --webdav-pacer-min-sleep 0s --create-empty-src-dirs --contimeout 5s --retries 1 -- :local:SOURCE :webdav:DESTINATION

Individual runs and verification

A. syq

Run 1: 1.63 GB/s · 5.273 s. Contents and stated metadata checked.

Metadata checked: file mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.

B. rclone / SFTP

Run 1: 276.1 MB/s · 31.116 s. Contents and stated metadata checked.

Metadata checked: file mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.

C. rclone / SFTP

Run 1: 279.2 MB/s · 30.765 s. Contents and stated metadata checked.

Metadata checked: file mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.

D. rclone / HTTPS WebDAV

Run 1: 279.7 MB/s · 30.715 s. Contents and stated metadata checked.

Metadata checked: file mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.

Method

Generated pseudorandom files were copied into empty destinations. Timing includes connection setup and native checksumming; preparation, independent verification and cleanup are excluded. No final durability flush was requested. Mounted NFS uses rclone’s local-filesystem backend, with no SFTP or WebDAV server.

Contents and stated metadata were checked after each copy; verification scope appears under each chart. Tool order rotated. One WebDAV run overlapping unrelated work was replaced. Before each timed LAN copy, we read the source files; each tool copies into a new empty destination. WAN sources remain in memory (tmpfs).

Raw data includes timings, commands and tool hashes. The optional dataset generator reproduces the file layout; it is separate from the unreleased measurement harness.

Transfer settings

--transfers controls concurrent files; --multi-thread-streams controls streams within a file for supported destinations. We tested transfer counts, SFTP packet sizes and concurrency, SSH connection sharing, and WebDAV cache and pacing settings.

SFTP used verified SSH host keys; WebDAV used authenticated HTTPS with a verified certificate. Services were started before timing. Remote copies preserved file mtimes, but did not request Unix ownership, permissions or ACLs.

Server command and SSH wrapper

# LAN server below; WAN used --dir-cache-time 0s.
rclone serve webdav "$REMOTE_SCRATCH" --config /dev/null --addr SERVER:PORT --dir-cache-time 5m --cert CERT_FILE --key TLS_KEY_FILE

# SSH_WRAPPER used by syq:
#!/bin/sh
exec ssh -F /dev/null -p SSH_PORT -o BatchMode=yes -o IdentitiesOnly=yes -i KEY_FILE -o StrictHostKeyChecking=yes -o UserKnownHostsFile=KNOWN_HOSTS "$@"

The WebDAV server received RCLONE_USER and RCLONE_PASS through its environment; the client used RCLONE_WEBDAV_USER and rclone-obscured RCLONE_WEBDAV_PASS. The copy commands above include the remaining client settings.

Rclone documentation: transfers, multithread streams, SFTP and WebDAV.