syq vs rclone
Preliminary benchmarks · syq 0.6.0 and rclone 1.75.1
If you know how to tune rclone for better performance in these tests, please open an issue with your suggestions.
File copies over a long-distance connection, between nearby servers, and on local and NFS filesystems.
WAN: one large file WAN: multiple files Measurement data (JSON)
Bars show dataset bytes / mean elapsed time; thin lines show the run range. Dashed lines mark nominal link ceilings. Run counts appear beside each speed. Commands use placeholder paths and servers.
LAN
LAN: 600,000 small files
600,000 × 4 KiB files in 256 directories (2.29 GiB total). RAID → RAID, approximately 0.2 ms RTT; both servers use 2 × 10 Gb/s bonds.
Three verified repetitions per current-release setting. A fresh 0.5.2 control followed the first syq run. Shared host and server-cache conditions are recorded in the data. Both hosts use two-port 10 Gb/s bonds; this refresh did not repeat the earlier per-port traffic diagnostic.
syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_RESOURCE_HELPER -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATIONrclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --timeout 60s --retries 1 --stats 5s --stats-one-line --stats-log-level NOTICE --no-check-dest --sftp-host SERVER --sftp-port PORT --sftp-user USER --sftp-key-file KEY_FILE --sftp-known-hosts-file KNOWN_HOSTS --sftp-md5sum-command md5sum --sftp-sha1sum-command sha1sum --transfers 64 :local:SOURCE :sftp:DESTINATIONrclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --timeout 60s --retries 1 --stats 5s --stats-one-line --stats-log-level NOTICE --no-check-dest --webdav-url https://SERVER:PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 64 --webdav-pacer-min-sleep 0s :local:SOURCE :webdav:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 101.1 MB/s · 24.317 s. Contents and stated metadata checked.
- Run 2: 100.4 MB/s · 24.467 s. Contents and stated metadata checked.
- Run 3: 107.5 MB/s · 22.868 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds; file and directory membership. Requested cache preparation: source files read before every timed copy. Final durability flush: no.
B. rclone / SFTP
- Run 1: 24.2 MB/s · 101.377 s. Contents and stated metadata checked.
- Run 2: 17.4 MB/s · 140.932 s. Contents and stated metadata checked.
- Run 3: 27.7 MB/s · 88.876 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds; file and directory membership. Requested cache preparation: source files read before every timed copy. Final durability flush: no.
C. rclone / HTTPS WebDAV
- Run 1: 54.4 MB/s · 45.170 s. Contents and stated metadata checked.
- Run 2: 49.9 MB/s · 49.220 s. Contents and stated metadata checked.
- Run 3: 51.8 MB/s · 47.420 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds; file and directory membership. Requested cache preparation: source files read before every timed copy. Final durability flush: no.
WAN
WAN: one 8 GiB file
One 8 GiB regular file. Flat directory, warm tmpfs source → remote XFS RAID; approximately 262.3 ms RTT. Sender NIC: 1 Gb/s (125 MB/s nominal ceiling).
Three verified repetitions per current-release setting. A fresh 0.5.2 control followed the first syq run. Shared host and server-cache conditions are recorded in the data.
Nominal sender-link ceiling: 125.0 MB/s (dashed line; before protocol overhead).
env SYQ_DEBUG=1 syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_RESOURCE_HELPER -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATIONrclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --retries 1 --stats 10s --stats-one-line -vv --webdav-url https://SERVER:PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 16 --webdav-pacer-min-sleep 2ms --no-check-dest :local:SOURCE :webdav:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 99.2 MB/s · 86.574 s. Contents and stated metadata checked.
- Run 2: 99.3 MB/s · 86.474 s. Contents and stated metadata checked.
- Run 3: 98.0 MB/s · 87.624 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.
B. rclone / HTTPS WebDAV
- Run 1: 104.8 MB/s · 81.975 s. Contents and stated metadata checked.
- Run 2: 104.8 MB/s · 81.975 s. Contents and stated metadata checked.
- Run 3: 105.3 MB/s · 81.574 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.
WAN: 32 × 1 GiB files
32 × 1 GiB regular files. Flat directory, warm tmpfs source → remote XFS RAID; approximately 262.3 ms RTT. Sender NIC: 1 Gb/s (125 MB/s nominal ceiling).
Three verified repetitions per current-release setting. A fresh 0.5.2 control followed the first syq run. Shared host and server-cache conditions are recorded in the data.
Nominal sender-link ceiling: 125.0 MB/s (dashed line; before protocol overhead).
env SYQ_DEBUG=1 syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_RESOURCE_HELPER -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATIONrclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --retries 1 --stats 10s --stats-one-line -vv --sftp-host SERVER --sftp-port PORT --sftp-user USER --sftp-key-use-agent --sftp-known-hosts-file KNOWN_HOSTS --sftp-md5sum-command md5sum --sftp-sha1sum-command sha1sum --sftp-chunk-size 255Ki --sftp-concurrency 256 --no-check-dest --sftp-key-use-agent=false --sftp-key-file KEY_FILE --transfers 32 :local:SOURCE :sftp:DESTINATIONrclone copy --config /dev/null --create-empty-src-dirs --contimeout 10s --retries 1 --stats 10s --stats-one-line -vv --webdav-url https://SERVER:PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 16 --webdav-pacer-min-sleep 2ms --no-check-dest :local:SOURCE :webdav:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 101.4 MB/s · 338.751 s. Contents and stated metadata checked.
- Run 2: 100.4 MB/s · 342.151 s. Contents and stated metadata checked.
- Run 3: 105.3 MB/s · 326.399 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.
B. rclone / SFTP
- Run 1: 88.6 MB/s · 387.957 s. Contents and stated metadata checked.
- Run 2: 87.6 MB/s · 392.404 s. Contents and stated metadata checked.
- Run 3: 89.2 MB/s · 385.040 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.
C. rclone / HTTPS WebDAV
- Run 1: 110.2 MB/s · 311.699 s. Contents and stated metadata checked.
- Run 2: 111.2 MB/s · 309.096 s. Contents and stated metadata checked.
- Run 3: 110.4 MB/s · 311.349 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm tmpfs source; no eviction. Final durability flush: no.
Hetzner WAN: one 4 GiB file
One 4 GiB file, warm tmpfs source to a cloud VM root filesystem. Germany to US East Coast; approximately 108 ms RTT. Two CCX23 guests with four dedicated vCPUs and 16 GiB RAM each.
Three verified copies per setting. SFTP ran in a later block after a host-key correction; additional paired release controls matched. No final durability flush.
syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_RESOURCE_HELPER -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATIONrclone copy --config /dev/null --webdav-url https://SERVER:PORT/ --create-empty-src-dirs --no-check-dest --contimeout 10s --timeout 60s --retries 1 --webdav-vendor rclone --ca-cert CA_CERT --transfers 16 --webdav-pacer-min-sleep 2ms -- :local:SOURCE :webdav:DESTINATIONrclone copy --config /dev/null --sftp-host SERVER --sftp-known-hosts-file KNOWN_HOSTS --create-empty-src-dirs --no-check-dest --contimeout 10s --timeout 60s --retries 1 --sftp-host SERVER --sftp-user USER --sftp-key-file KEY_FILE --sftp-known-hosts-file KNOWN_HOSTS --sftp-md5sum-command md5sum --sftp-sha1sum-command sha1sum --transfers 32 --sftp-chunk-size 255Ki --sftp-concurrency 256 -- :local:SOURCE :sftp:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 324.7 MB/s · 13.228 s. Contents and stated metadata checked.
- Run 2: 296.5 MB/s · 14.485 s. Contents and stated metadata checked.
- Run 3: 294.3 MB/s · 14.594 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm. Final durability flush: no.
B. rclone / HTTPS WebDAV
- Run 1: 30.3 MB/s · 141.791 s. Contents and stated metadata checked.
- Run 2: 19.5 MB/s · 220.756 s. Contents and stated metadata checked.
- Run 3: 19.7 MB/s · 218.444 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm. Final durability flush: no.
C. rclone / SFTP
- Run 1: 18.3 MB/s · 235.258 s. Contents and stated metadata checked.
- Run 2: 18.2 MB/s · 235.613 s. Contents and stated metadata checked.
- Run 3: 18.3 MB/s · 234.551 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: warm. Final durability flush: no.
Mounted NFS
Small files → mounted NFS
40,000 × 4 KiB files in 256 directories. Local /tmp source → mounted NFSv4.2 destination.
Three verified repetitions per current-release setting. A fresh 0.5.2 control followed the first syq run. Shared host and server-cache conditions are recorded in the data.
syq rsync -a --no-progress SOURCE/ DESTINATIONrclone copy --config /dev/null --metadata --create-empty-src-dirs -- :local:SOURCE :local:DESTINATIONrclone copy --config /dev/null --metadata --create-empty-src-dirs --transfers 64 -- :local:SOURCE :local:DESTINATIONrclone copy --config /dev/null --metadata --create-empty-src-dirs --transfers 256 -- :local:SOURCE :local:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 23.0 MB/s · 7.123 s. Contents and stated metadata checked.
- Run 2: 19.4 MB/s · 8.425 s. Contents and stated metadata checked.
- Run 3: 22.7 MB/s · 7.224 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; root excluded. Requested cache preparation: evict. Final durability flush: no.
B. rclone / local filesystem
- Run 1: 4.7 MB/s · 35.122 s. Contents and stated metadata checked.
- Run 2: 4.4 MB/s · 37.426 s. Contents and stated metadata checked.
- Run 3: 4.5 MB/s · 36.023 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; root excluded. Requested cache preparation: evict. Final durability flush: no.
C. rclone / local filesystem
- Run 1: 7.0 MB/s · 23.395 s. Contents and stated metadata checked.
- Run 2: 6.1 MB/s · 26.804 s. Contents and stated metadata checked.
- Run 3: 10.9 MB/s · 14.986 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; root excluded. Requested cache preparation: evict. Final durability flush: no.
D. rclone / local filesystem
- Run 1: 12.6 MB/s · 13.032 s. Contents and stated metadata checked.
- Run 2: 12.4 MB/s · 13.234 s. Contents and stated metadata checked.
- Run 3: 12.1 MB/s · 13.585 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; root excluded. Requested cache preparation: evict. Final durability flush: no.
Small files ← mounted NFS
400,000 × 4 KiB files in 256 directories (1.53 GiB total). Mounted NFSv4.2 source → local XFS RAID destination.
Three verified repetitions per current-release setting. A fresh 0.5.2 control followed the first syq run. Shared host and server-cache conditions are recorded in the data. Source contents and metadata were read before every timed copy.
syq rsync -a --no-progress SOURCE/ DESTINATIONrclone copy --config /dev/null --metadata --create-empty-src-dirs --transfers 64 --stats 5s --stats-one-line --stats-log-level NOTICE :local:SOURCE :local:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 68.8 MB/s · 23.819 s. Contents and stated metadata checked.
- Run 2: 86.3 MB/s · 18.976 s. Contents and stated metadata checked.
- Run 3: 88.2 MB/s · 18.576 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; file contents SHA256; root excluded. Requested cache preparation: source file contents and metadata read before every timed copy. Final durability flush: no.
B. rclone / local filesystem
- Run 1: 46.7 MB/s · 35.071 s. Contents and stated metadata checked.
- Run 2: 54.0 MB/s · 30.365 s. Contents and stated metadata checked.
- Run 3: 54.3 MB/s · 30.164 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; file contents SHA256; root excluded. Requested cache preparation: source file contents and metadata read before every timed copy. Final durability flush: no.
Filesystem and network capabilities
Local filesystem
Large file on the same filesystem
One 2 GiB file. Source and destination on the same XFS RAID filesystem.
Three verified repetitions per current-release setting. A fresh 0.5.2 control followed the first syq run. Kernel-assisted copying may share extents; these rates are not physical disk bandwidth.
syq rsync -a --no-progress SOURCE/ DESTINATIONrclone copy --config /dev/null --metadata --create-empty-src-dirs -- :local:SOURCE :local:DESTINATIONrclone copy --config /dev/null --metadata --create-empty-src-dirs --multi-thread-streams 8 -- :local:SOURCE :local:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 18.71 GB/s · 0.115 s · under 2 s. Contents and stated metadata checked.
- Run 2: 33.28 GB/s · 0.065 s · under 2 s. Contents and stated metadata checked.
- Run 3: 33.43 GB/s · 0.064 s · under 2 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; root excluded. Requested cache preparation: evict. Final durability flush: no.
B. rclone / local filesystem
- Run 1: 540.9 MB/s · 3.970 s. Contents and stated metadata checked.
- Run 2: 547.7 MB/s · 3.921 s. Contents and stated metadata checked.
- Run 3: 601.6 MB/s · 3.569 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; root excluded. Requested cache preparation: evict. Final durability flush: no.
C. rclone / local filesystem
- Run 1: 534.1 MB/s · 4.020 s. Contents and stated metadata checked.
- Run 2: 534.1 MB/s · 4.021 s. Contents and stated metadata checked.
- Run 3: 555.1 MB/s · 3.869 s. Contents and stated metadata checked.
Metadata checked: regular files and directories: membership, type, mode, uid, gid, mtime_ns; root excluded. Requested cache preparation: evict. Final durability flush: no.
Multi-rail LAN
syq uses all eight rails of this multi-rail LAN; rclone uses only one, even with multiple files and increased transfer concurrency.
Fast fabric: one large file
One 8 GiB file, tmpfs → tmpfs between servers in the same data center.
Three verified repetitions per current-release setting. A fresh 0.5.2 control followed the first syq run. Shared host and server-cache conditions are recorded in the data. Timing includes an empty fixture probe; syq used eight rails while rclone used one.
env SYQ_DEBUG=1 syq rsync -rt --no-progress --syq-no-bootstrap --rsync-path REMOTE_RESOURCE_HELPER -e SSH_WRAPPER SOURCE/ USER@SERVER:DESTINATIONrclone copy --config /dev/null --sftp-host SERVER --sftp-known-hosts-file KNOWN_HOSTS --sftp-user USER --sftp-host SERVER --sftp-port PORT --sftp-user USER --sftp-key-file KEY_FILE --sftp-known-hosts-file KNOWN_HOSTS --create-empty-src-dirs --contimeout 5s --retries 1 -- :local:SOURCE :sftp:DESTINATIONrclone copy --config /dev/null --sftp-host SERVER --sftp-known-hosts-file KNOWN_HOSTS --sftp-user USER --sftp-host SERVER --sftp-port PORT --sftp-user USER --sftp-key-file KEY_FILE --sftp-known-hosts-file KNOWN_HOSTS --create-empty-src-dirs --contimeout 5s --retries 1 --sftp-chunk-size 128Ki --sftp-concurrency 128 -- :local:SOURCE :sftp:DESTINATIONrclone copy --config /dev/null --webdav-url https://SERVER:PORT/ --webdav-vendor rclone --ca-cert CA_CERT --transfers 16 --webdav-pacer-min-sleep 0s --create-empty-src-dirs --contimeout 5s --retries 1 -- :local:SOURCE :webdav:DESTINATIONIndividual runs and verification
A. syq
- Run 1: 1.38 GB/s · 6.224 s. Contents and stated metadata checked.
- Run 2: 1.45 GB/s · 5.923 s. Contents and stated metadata checked.
- Run 3: 1.41 GB/s · 6.073 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.
B. rclone / SFTP
- Run 1: 262.1 MB/s · 32.768 s. Contents and stated metadata checked.
- Run 2: 255.9 MB/s · 33.569 s. Contents and stated metadata checked.
- Run 3: 254.7 MB/s · 33.721 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.
C. rclone / SFTP
- Run 1: 259.0 MB/s · 33.169 s. Contents and stated metadata checked.
- Run 2: 254.4 MB/s · 33.768 s. Contents and stated metadata checked.
- Run 3: 251.8 MB/s · 34.119 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.
D. rclone / HTTPS WebDAV
- Run 1: 287.2 MB/s · 29.911 s. Contents and stated metadata checked.
- Run 2: 277.9 MB/s · 30.916 s. Contents and stated metadata checked.
- Run 3: 279.2 MB/s · 30.765 s. Contents and stated metadata checked.
Metadata checked: regular-file sizes and mtimes, whole seconds. Requested cache preparation: evict. Final durability flush: no.
Method
Generated pseudorandom files were copied into empty destinations. Timing includes connection setup and native checksumming; preparation, independent verification and cleanup are excluded. No final durability flush was requested. Mounted NFS uses rclone’s local-filesystem backend, with no SFTP or WebDAV server.
Contents and stated metadata were checked after each copy; verification scope appears under each chart. Tool order rotated. Previous-release controls followed the first syq run; they are kept separately from the displayed release. Before each timed LAN copy, we read the source files; each tool copies into a new empty destination. WAN sources remain in memory (tmpfs).
Raw data includes timings, commands and tool hashes. The optional dataset generator reproduces the file layout; the capture records the actual preparation procedure and matching public harness revision.
Transfer settings
--transfers controls concurrent files; --multi-thread-streams
controls streams within a file for supported destinations. The refresh uses the transfer counts, SFTP packet sizes and concurrency, and WebDAV cache and pacing settings selected in the earlier campaign.
SFTP used verified SSH host keys; WebDAV used authenticated HTTPS with a verified certificate. Services were started before timing. Remote copies preserved file mtimes, but did not request Unix ownership, permissions or ACLs.
Server command and SSH wrapper
# LAN server below; WAN used --dir-cache-time 0s.
rclone serve webdav "$REMOTE_SCRATCH" --config /dev/null --addr SERVER:PORT --dir-cache-time 5m --cert CERT_FILE --key TLS_KEY_FILE
# SSH_WRAPPER used by syq:
#!/bin/sh
exec ssh -F /dev/null -p SSH_PORT -o BatchMode=yes -o IdentitiesOnly=yes -i KEY_FILE -o StrictHostKeyChecking=yes -o UserKnownHostsFile=KNOWN_HOSTS "$@"
The WebDAV server received RCLONE_USER and RCLONE_PASS through its environment;
the client used RCLONE_WEBDAV_USER and rclone-obscured RCLONE_WEBDAV_PASS.
The copy commands above include the remaining client settings.
Rclone documentation: transfers, multithread streams, SFTP and WebDAV.