syq Fast, programmable file transfer.
Copy, reorganize, and remove files across local filesystems, remote machines, and S3-compatible storage. Choose which files to copy and where they should land, preview the changes, and rerun interrupted copies to resume.
Try a copy
Start with a local directory named project. The same copy can go to a local
folder, an SSH server, or an S3 bucket:
syq cp project --into backup
syq cp project --to server --into backup
syq cp project --to s3://backups --into backup
Each command puts project inside backup. Replace server with an SSH
hostname or alias you normally use, or backups with an existing bucket.
A relative SSH destination starts in your home directory on that server.
See S3 setup for credentials and other providers.
--from and --to choose the source and destination machines or buckets.
Without them, paths are local. --into puts selected names inside a directory;
--as chooses an exact destination name.
To copy the contents of project directly into backup, use --srcs-in:
syq cp --srcs-in project --to server --into backup
Use --dry-run to preview a summary without copying, or --dry-run -v
to list the planned changes by path.
Existing destination files are updated when needed. Unrelated files stay
unless you request --prune.
Start with a tool you know
rsync
Copy a file
rsync
rsync -a report.csv server:/backup/syq
syq cp report.csv --to server --into /backupCopies report.csv to /backup/report.csv on server.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync -t --mkpath --modify-window=-1 report.csv server:/backup/-t preserves modification times. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
Send a folder to a server
rsync
rsync -a project server:/backup/syq
syq cp project --to server --into /backupCopies project to /backup/project. Other entries in /backup stay.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync -rlt --mkpath --modify-window=-1 project server:/backup/-t preserves modification times; -r recurses into directories and -l preserves symlinks. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
Rsync can replace an empty destination directory with a source file. Syq refuses file/directory type conflicts, even for empty directories.
Copy the files inside a folder
rsync
rsync -a project/ server:/backup/syq
syq cp --srcs-in project --to server --into /backupCopies project/index.html to /backup/index.html. Destination-only files stay.
In rsync, project/ copies the contents; project copies the folder. In syq, a trailing slash does not change placement: use --srcs-in for contents.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync -rlt --mkpath --modify-window=-1 project/ server:/backup/-t preserves modification times; -r recurses into directories and -l preserves symlinks. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
Rsync can replace an empty destination directory with a source file. Syq refuses file/directory type conflicts, even for empty directories.
Match a folder, deleting extra files
rsync
rsync --delete -a project/ server:/backup/syq
syq cp --prune --srcs-in project --to server --into /backupUpdates /backup with the contents of project and removes destination entries absent from the source.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync --delete-after -rlt --mkpath --modify-window=-1 project/ server:/backup/-t preserves modification times; -r recurses into directories and -l preserves symlinks. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
--delete-after postpones deletion until after the transfer phase, closer to syq’s copy-then-prune ordering.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
Rsync can replace an empty destination directory with a source file. Syq refuses file/directory type conflicts, even for empty directories.
Download a file from a server
rsync
rsync -a server:/data/report.csv .syq
syq cp --from server /data/report.csv --into .Fetches report.csv into your current directory.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync -t --mkpath --modify-window=-1 server:/data/report.csv .-t preserves modification times. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
Preview a copy
rsync
rsync --dry-run --itemize-changes -a project/ server:/backup/syq
syq cp --dry-run -v --srcs-in project --to server --into /backupLists planned changes without copying file data.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync --dry-run --itemize-changes -rlt --mkpath --modify-window=-1 project/ server:/backup/-t preserves modification times; -r recurses into directories and -l preserves symlinks. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
Rsync can replace an empty destination directory with a source file. Syq refuses file/directory type conflicts, even for empty directories.
Rename a file while copying
rsync
rsync -a report.csv server:/backup/latest.csvsyq
syq cp report.csv --to server --as /backup/latest.csvCopies report.csv to /backup/latest.csv; the source stays.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync -t --mkpath --modify-window=-1 report.csv server:/backup/latest.csv-t preserves modification times. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
If a requested destination filename is already a directory, rsync copies inside it. Syq refuses to replace that directory with a file.
Choose destination names with a script
A script chooses beach.jpg for a.jpg and summit.jpg for b.jpg.
rsync
rsync -a photos/a.jpg server:/archive/beach.jpg
rsync -a photos/b.jpg server:/archive/summit.jpgsyq
files.ndjson
{"src":{"encoding":"utf-8","value":"a.jpg"},"dst":{"encoding":"utf-8","value":"beach.jpg"}}
{"src":{"encoding":"utf-8","value":"b.jpg"},"dst":{"encoding":"utf-8","value":"summit.jpg"}}syq cp --mapping files.ndjson -C photos --to server --into /archiveGenerate a mapping with a script.
Differences
The command above uses archive mode (-a), which preserves permissions, ownership and special files as well as timestamps and symlinks, where permitted. Syq normally preserves timestamps and links; it keeps existing destination permissions and limits new-file permissions by the destination umask. It does not preserve ownership or copy special files by default.
For a closer match:
rsync -t --mkpath --modify-window=-1 photos/a.jpg server:/archive/beach.jpg
rsync -t --mkpath --modify-window=-1 photos/b.jpg server:/archive/summit.jpg-t preserves modification times. --mkpath (rsync 3.2.3+) creates missing destination parents. --modify-window=-1 compares fractional seconds too; the simpler command compares only whole seconds.
Syq’s size-and-time comparison ignores trailing fractional digits where the destination timestamp has zeros; rsync with --modify-window=-1 requires exact nanosecond equality. Use content hashing when matching timestamps are not enough.
If a requested destination filename is already a directory, rsync copies inside it. Syq refuses to replace that directory with a file.
These rsync commands run separately. Syq validates the complete mapping for duplicate destinations and file/ancestor conflicts before copying.
rclone
Copy a file
rclone
rclone copyto report.csv s3:backups/reports/report.csvsyq
syq cp report.csv --to s3://backups --as reports/report.csvUploads report.csv to reports/report.csv in bucket backups.
Here s3 is an rclone S3 remote. Syq uses AWS credentials and its own endpoint settings.
Differences
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Rclone stores source modification times in S3 metadata named mtime; syq uses syq-mtime and syq-mtime-nsec. Each tool restores its own timestamp format; one tool’s upload does not give the other the same source-time information.
Copy the files inside a folder
rclone
rclone copy project backupsyq
syq cp --srcs-in project --into backupCopies project/index.html to backup/index.html. Destination-only files stay.
Differences
The command above does not request empty-directory or symlink preservation. For a closer match:
rclone copy --links --create-empty-src-dirs project backup--create-empty-src-dirs includes empty directories. --links preserves symlinks through rclone’s link representation, restoring real links on a local destination.
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
Upload a folder’s contents to S3
rclone
rclone copy project s3:backups/projectsyq
syq cp --srcs-in project --to s3://backups --into projectUploads project/index.html to the key project/index.html in bucket backups.
s3 is an rclone S3 remote. Syq uses AWS credentials and endpoint settings.
Differences
The command above does not request empty-directory or symlink preservation. For a closer match:
rclone copy --links --create-empty-src-dirs --s3-directory-markers project s3:backups/project--create-empty-src-dirs includes empty directories. --links preserves symlinks through rclone’s link representation, restoring real links on a local destination.
--s3-directory-markers stores empty folders as objects whose names end in /.
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Rclone stores source modification times in S3 metadata named mtime; syq uses syq-mtime and syq-mtime-nsec. Each tool restores its own timestamp format; one tool’s upload does not give the other the same source-time information.
Rclone --links uploads a symlink as a text file with a .rclonelink suffix. Syq keeps the original name: as a real link on an SSH server, or an object marked as a symlink in S3.
Match a folder, deleting extra files
rclone
rclone sync project backupsyq
syq cp --prune --srcs-in project --into backupUpdates backup with the contents of project and removes destination entries absent from the source.
Differences
The command above does not request empty-directory or symlink preservation. For a closer match:
rclone sync --links --create-empty-src-dirs project backup--create-empty-src-dirs includes empty directories. --links preserves symlinks through rclone’s link representation, restoring real links on a local destination.
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
Download a folder from S3
rclone
rclone copy s3:backups/reports reportssyq
syq cp --from s3://backups --srcs-in reports --into reportsDownloads reports/january.csv from S3 to reports/january.csv locally, keeping subfolders.
Here s3 is an rclone S3 remote. Unrelated local files stay.
Differences
The command above does not request empty-directory or symlink preservation. For a closer match:
rclone copy --links --create-empty-src-dirs s3:backups/reports reports--create-empty-src-dirs includes empty directories. --links preserves symlinks through rclone’s link representation, restoring real links on a local destination.
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Rclone stores source modification times in S3 metadata named mtime; syq uses syq-mtime and syq-mtime-nsec. Each tool restores its own timestamp format; one tool’s upload does not give the other the same source-time information.
Rclone --links restores its .rclonelink objects as local symlinks. Syq restores objects carrying syq’s symlink metadata. Neither decodes the other format.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
Copy a file between S3 buckets
rclone
rclone copyto s3:backups/report.csv s3:archive/report.csvsyq
syq cp --from s3://backups report.csv --to s3://archive --as report.csvCopies report.csv from bucket backups to bucket archive.
Here s3 is an rclone S3 remote. Both buckets must exist on the same S3 service for this syq example.
Differences
Rclone’s default skip decision uses size and modification time, with a hash comparison when times differ. Syq checks object type, size and copyable metadata, then compares full-object checksums when available; otherwise, it accepts matching ETags or a source carrying syq metadata. These tests can make different decisions about whether to copy an existing object.
Preview a copy
rclone
rclone copy --dry-run project backupsyq
syq cp --dry-run -v --srcs-in project --into backupPreviews copying the contents of project into backup.
Differences
The command above does not request empty-directory or symlink preservation. For a closer match:
rclone copy --dry-run --links --create-empty-src-dirs project backup--create-empty-src-dirs includes empty directories. --links preserves symlinks through rclone’s link representation, restoring real links on a local destination.
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
Rename a file while copying
rclone
rclone copyto report.csv latest.csvsyq
syq cp report.csv --as latest.csvCopies report.csv to latest.csv in your current directory; the source stays.
Differences
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
Remove a file
rclone
rclone deletefile s3:backups/old.csvsyq
syq rm --on s3://backups old.csvRemoves the object old.csv from bucket backups.
Here s3 is an rclone S3 remote.
Send a folder to a server
rclone
rclone copy project server:/backup/projectsyq
syq cp project --to server --into /backupCopies project to /backup/project on the server.
Configure server as an rclone SFTP remote.
Differences
The command above does not request empty-directory or symlink preservation. For a closer match:
rclone copy --links --create-empty-src-dirs project server:/backup/project--create-empty-src-dirs includes empty directories. --links preserves symlinks through rclone’s link representation, restoring real links on a local destination.
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Rclone --links uploads a symlink as a text file with a .rclonelink suffix. Syq keeps the original name: as a real link on an SSH server, or an object marked as a symlink in S3.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
Download a file from a server
rclone
rclone copyto server:/data/report.csv report.csvsyq
syq cp --from server /data/report.csv --into .Downloads report.csv into your current folder.
Here server is an rclone SFTP remote.
Differences
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
Choose destination names with a script
A script chooses beach.jpg for a.jpg and summit.jpg for b.jpg.
rclone
rclone copyto photos/a.jpg archive/beach.jpg
rclone copyto photos/b.jpg archive/summit.jpgsyq
files.ndjson
{"src":{"encoding":"utf-8","value":"a.jpg"},"dst":{"encoding":"utf-8","value":"beach.jpg"}}
{"src":{"encoding":"utf-8","value":"b.jpg"},"dst":{"encoding":"utf-8","value":"summit.jpg"}}syq cp --mapping files.ndjson -C photos --into archiveGenerate a mapping with a script.
Differences
When sizes match but modification times differ, rclone can compare available hashes and update only the timestamp if the contents match. Syq’s default unchanged-file check uses size and modification time, without that hash fallback.
Without --metadata, rclone does not restore source Unix permission bits. Syq gives new files the source read/write/execute bits limited by the destination umask and keeps existing destination permissions. Rclone’s --metadata would additionally restore ownership and other metadata, so it does not match syq’s default either.
These rclone commands run separately. Syq checks the complete mapping for duplicate destinations and file/ancestor conflicts before copying.
s5cmd
Copy a file
s5cmd
s5cmd cp report.csv s3://backups/reports/report.csvsyq
syq cp report.csv --to s3://backups --as reports/report.csvUploads report.csv to the exact key reports/report.csv in bucket backups.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
S5cmd uploads regular file contents without automatically recording source Unix permissions or modification times. Syq records those in its S3 metadata and restores modification times on download.
Download a folder from S3
s5cmd
s5cmd cp 's3://backups/reports/*' reports/syq
syq cp --from s3://backups --srcs-in reports --into reportsDownloads reports/january.csv from S3 to reports/january.csv locally, keeping subfolders.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
S5cmd downloads object bodies as regular files; it does not restore syq’s symlink or timestamp metadata. Syq recognizes that metadata and restores the original object type and modification time.
Copy a file between S3 buckets
s5cmd
s5cmd cp s3://backups/report.csv s3://archive/report.csvsyq
syq cp --from s3://backups report.csv --to s3://archive --as report.csvCopies the key report.csv from bucket backups to bucket archive.
Both buckets must exist on the same S3 service for this example.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
Preview a copy
s5cmd
s5cmd --dry-run cp report.csv s3://backups/report.csvsyq
syq cp --dry-run -v report.csv --to s3://backups --as report.csvPreviews uploading report.csv without writing the object.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
S5cmd uploads regular file contents without automatically recording source Unix permissions or modification times. Syq records those in its S3 metadata and restores modification times on download.
Rename a file while copying
s5cmd
s5cmd cp report.csv s3://backups/latest.csvsyq
syq cp report.csv --to s3://backups --as latest.csvUploads report.csv as the key latest.csv; the local source stays.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
S5cmd uploads regular file contents without automatically recording source Unix permissions or modification times. Syq records those in its S3 metadata and restores modification times on download.
Remove a file
s5cmd
s5cmd rm s3://backups/old.csvsyq
syq rm --on s3://backups old.csvRemoves the object old.csv from bucket backups.
Upload a folder’s contents to S3
s5cmd
s5cmd cp project/ s3://backups/project/syq
syq cp --srcs-in project --to s3://backups --into projectUploads project/index.html to project/index.html in bucket backups.
Unrelated objects stay.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
S5cmd uploads regular file contents without automatically recording source Unix permissions or modification times. Syq records those in its S3 metadata and restores modification times on download.
S5cmd follows source symlinks and omits empty directories. Its --no-follow-symlinks skips links rather than preserving them. Syq stores symlinks and empty directories for restoration by syq.
Copy the files inside a folder
s5cmd
s5cmd cp project/ s3://backups/project/syq
syq cp --srcs-in project --to s3://backups --into projectCopies the files inside project to the project/ folder in S3.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
S5cmd uploads regular file contents without automatically recording source Unix permissions or modification times. Syq records those in its S3 metadata and restores modification times on download.
S5cmd follows source symlinks and omits empty directories. Its --no-follow-symlinks skips links rather than preserving them. Syq stores symlinks and empty directories for restoration by syq.
Match a folder, deleting extra files
s5cmd
s5cmd sync --delete project/ s3://backups/project/syq
syq cp --prune --srcs-in project --to s3://backups --into projectUpdates the project/ folder in S3 and removes objects absent from the local project folder.
Differences
S5cmd sync copies when sizes differ or the source modification time is newer. Syq can also update a same-size destination whose modification time is newer than the source. For S3, s5cmd uses the object’s service timestamp; syq uses its stored source timestamp when present.
S5cmd runs its generated copy and removal commands concurrently. Syq waits for copying to finish and suppresses pruning after scan or copy errors.
S5cmd uploads regular file contents without automatically recording source Unix permissions or modification times. Syq records those in its S3 metadata and restores modification times on download.
S5cmd follows source symlinks and omits empty directories. Its --no-follow-symlinks skips links rather than preserving them. Syq stores symlinks and empty directories for restoration by syq.
Choose destination names with a script
A script chooses beach.jpg for a.jpg and summit.jpg for b.jpg.
s5cmd
copies.txt
cp photos/a.jpg s3://backups/archive/beach.jpg
cp photos/b.jpg s3://backups/archive/summit.jpgs5cmd run copies.txtsyq
files.ndjson
{"src":{"encoding":"utf-8","value":"a.jpg"},"dst":{"encoding":"utf-8","value":"beach.jpg"}}
{"src":{"encoding":"utf-8","value":"b.jpg"},"dst":{"encoding":"utf-8","value":"summit.jpg"}}syq cp --mapping files.ndjson -C photos --to s3://backups --into archiveGenerate a mapping with a script.
Differences
S5cmd cp copies selected files again even when the destination already matches. Syq can skip unchanged entries. S5cmd’s size-only and source-newer conditions do not reproduce syq’s default comparison.
S5cmd uploads regular file contents without automatically recording source Unix permissions or modification times. Syq records those in its S3 metadata and restores modification times on download.
S5cmd executes independent commands from copies.txt. Syq checks the entire mapping for duplicate destinations and file/ancestor conflicts before copying.
scp
Copy a file
scp
scp -p report.csv server:/backup/syq
syq cp report.csv --to server --into /backupCopies report.csv to /backup/report.csv on server.
Differences
Scp -p preserves source permission bits and access times as well as modification times. Syq preserves modification times, leaves existing destination permissions alone, and creates files with source read/write/execute bits limited by the destination umask; it does not preserve source access times.
Scp transfers each file again and writes directly to the destination. Syq can skip unchanged files and normally stages changed files before replacing them, keeping the old file available until replacement.
The scp destination directory must already exist for the placement shown. Syq creates missing destination directories.
Download a file from a server
scp
scp -p server:/data/report.csv .syq
syq cp --from server /data/report.csv --into .Copies the server’s /data/report.csv to report.csv in your current directory.
Differences
Scp -p preserves source permission bits and access times as well as modification times. Syq preserves modification times, leaves existing destination permissions alone, and creates files with source read/write/execute bits limited by the destination umask; it does not preserve source access times.
Scp transfers each file again and writes directly to the destination. Syq can skip unchanged files and normally stages changed files before replacing them, keeping the old file available until replacement.
Send a folder to a server
scp
scp -pr project server:/backup/syq
syq cp project --to server --into /backupCopies project to /backup/project.
Differences
Scp -p preserves source permission bits and access times as well as modification times. Syq preserves modification times, leaves existing destination permissions alone, and creates files with source read/write/execute bits limited by the destination umask; it does not preserve source access times.
Scp transfers each file again and writes directly to the destination. Syq can skip unchanged files and normally stages changed files before replacing them, keeping the old file available until replacement.
Scp follows symlinks inside the copied tree and copies their targets. Syq copies those links themselves.
The scp destination directory must already exist for the placement shown. Syq creates missing destination directories.
Rename a file while copying
scp
scp -p report.csv server:/backup/latest.csvsyq
syq cp report.csv --to server --as /backup/latest.csvCopies report.csv to /backup/latest.csv; the source stays.
Differences
Scp -p preserves source permission bits and access times as well as modification times. Syq preserves modification times, leaves existing destination permissions alone, and creates files with source read/write/execute bits limited by the destination umask; it does not preserve source access times.
Scp transfers each file again and writes directly to the destination. Syq can skip unchanged files and normally stages changed files before replacing them, keeping the old file available until replacement.
The scp destination directory must already exist for the placement shown. Syq creates missing destination directories.
If a requested destination filename is already a directory, scp copies inside it. Syq refuses to replace that directory with a file.
Copy the files inside a folder
scp
scp -pr project/. server:/backup/syq
syq cp --srcs-in project --to server --into /backupCopies project’s contents, including hidden files and subfolders, into /backup.
Differences
Scp -p preserves source permission bits and access times as well as modification times. Syq preserves modification times, leaves existing destination permissions alone, and creates files with source read/write/execute bits limited by the destination umask; it does not preserve source access times.
Scp transfers each file again and writes directly to the destination. Syq can skip unchanged files and normally stages changed files before replacing them, keeping the old file available until replacement.
Scp follows symlinks inside the copied tree and copies their targets. Syq copies those links themselves.
The scp destination directory must already exist for the placement shown. Syq creates missing destination directories.
Choose destination names with a script
A script chooses beach.jpg for a.jpg and summit.jpg for b.jpg.
scp
scp -p photos/a.jpg server:/archive/beach.jpg
scp -p photos/b.jpg server:/archive/summit.jpgsyq
files.ndjson
{"src":{"encoding":"utf-8","value":"a.jpg"},"dst":{"encoding":"utf-8","value":"beach.jpg"}}
{"src":{"encoding":"utf-8","value":"b.jpg"},"dst":{"encoding":"utf-8","value":"summit.jpg"}}syq cp --mapping files.ndjson -C photos --to server --into /archiveGenerate a mapping with a script.
Differences
Scp -p preserves source permission bits and access times as well as modification times. Syq preserves modification times, leaves existing destination permissions alone, and creates files with source read/write/execute bits limited by the destination umask; it does not preserve source access times.
Scp transfers each file again and writes directly to the destination. Syq can skip unchanged files and normally stages changed files before replacing them, keeping the old file available until replacement.
The scp destination directory must already exist for the placement shown. Syq creates missing destination directories.
If a requested destination filename is already a directory, scp copies inside it. Syq refuses to replace that directory with a file.
These scp commands run separately. Syq checks the complete mapping for duplicate destinations and file/ancestor conflicts before copying.
syq
Copy a file
syq
syq cp report.csv --to server --into /backupCopies report.csv to /backup/report.csv on server.
Replace server with your SSH host or alias.
Send a folder to a server
syq
syq cp project --to server --into /backupCopies project to /backup/project. Other entries in /backup stay.
Copy the files inside a folder
syq
syq cp --srcs-in project --to server --into /backupCopies project/index.html to /backup/index.html. Destination-only files stay.
Match a folder, deleting extra files
syq
syq cp --prune --srcs-in project --to server --into /backupUpdates /backup with the contents of project and removes destination entries absent from the source.
Download a file from a server
syq
syq cp --from server /data/report.csv --into .Fetches report.csv into your current directory.
Run this on your laptop. The source path is on server.
Preview a copy
syq
syq cp --dry-run -v --srcs-in project --to server --into /backupLists planned changes without copying file data.
Rename a file while copying
syq
syq cp report.csv --to server --as /backup/latest.csvCopies report.csv to /backup/latest.csv; the source stays.
Choose destination names with a script
A script chooses beach.jpg for a.jpg and summit.jpg for b.jpg.
syq
files.ndjson
{"src":{"encoding":"utf-8","value":"a.jpg"},"dst":{"encoding":"utf-8","value":"beach.jpg"}}
{"src":{"encoding":"utf-8","value":"b.jpg"},"dst":{"encoding":"utf-8","value":"summit.jpg"}}syq cp --mapping files.ndjson -C photos --into archiveUpload a folder’s contents to S3
syq
syq cp --srcs-in project --to s3://backups --into projectUploads project/index.html to project/index.html in bucket backups.
The backups bucket must already exist. Configure AWS credentials and any custom S3 endpoint.
Download a folder from S3
syq
syq cp --from s3://backups --srcs-in reports --into reportsDownloads reports/january.csv from S3 to reports/january.csv locally, keeping subfolders.
Copy a file between S3 buckets
syq
syq cp --from s3://backups report.csv --to s3://archive --as report.csvCopies report.csv from bucket backups to bucket archive.
Both buckets must exist on the same S3 service for this example.
Remove a file
syq
syq rm --on s3://backups old.csvRemoves the object old.csv from bucket backups.
Send from a server shell to a laptop with no SSH server
syq
# On the server:
syq cp results --to @laptopSends results home from the server shell. Your laptop needs no SSH server or incoming port.
Set up the receiving name laptop, then approve the copy on your laptop.
Alternatives that change the task
Rsync, scp, and rclone can pull from the laptop instead. S5cmd can stage files in S3.
Copy directly between servers without agent forwarding
syq
# On your laptop:
syq cp --from hostA --srcs-in results --to hostB --into resultsUses your laptop’s SSH logins. Files travel directly between servers; neither server gets your keys or access to your SSH agent.
The servers need a connection to each other. Setup.
Alternatives that change the task
Scp -3 and rclone relay through your laptop. Rsync on hostA or scp -R requires hostA to have its own access to hostB. S5cmd can stage files in S3.
Common tasks
| I want to… | Start here |
|---|---|
| Look up commands and options | Command reference |
| Choose exactly where files land | Copy and placement |
| Preview changes | Dry runs |
Skip build files or use .gitignore | Ignoring paths |
| Mirror a directory | Mirroring |
| Remove files in parallel | Removal |
| Send files to my laptop from a server’s shell | Receiving files |
| Run a build or open an artifact on my desktop from a server | Commands on your receiving machine |
| Copy between two servers | Remote-to-remote transfers |
| Rename or reorganize files during a copy | Mappings |
| Use syq from Python | Python SDK |
| Read results from a script | Automation results |
| Make copies faster | Speed |
Install
Syq runs on Linux and macOS, on x86-64 and ARM64.
Standalone installer
curl --proto '=https' --tlsv1.2 -LsSf https://dl.syq.christmas/latest/install.sh | sh
Installs into ~/.local/bin without sudo. Make sure that directory is on your
PATH. To choose another directory, download the script and run
sh install.sh --bin-dir DIR.
Homebrew
brew install greaber/tap/syq
Build from source
See source builds for Cargo builds, custom compilation options, and choosing between your own executable and compatible official SSH helpers.
Automatic installation on SSH servers
Syq installs its SSH helper on the server when needed. Official releases also
try to make syq available at ~/.local/bin/syq for commands you run there.
Existing commands and shell startup files are left alone.
Use syq --self-update on the server to update that command, or the standalone
installer above if it is missing. See SSH helper installation
for custom helpers and installation exceptions.
Try a benchmark
Compare syq with rsync on your own machines, or with rsync and cp locally. See Quick comparison for the script and how to read its results.
Updates
Use syq --self-update for a standalone installation, or brew upgrade syq
for Homebrew.
Standalone and Homebrew installs may print an update reminder in a terminal,
at most once a day after a successful command, naming the upgrade command for
that install. Nothing updates automatically. Set SYQ_NO_UPDATE_CHECK=1 or
DO_NOT_TRACK=1 to disable reminders.
Update-check data
Downloads and the daily reminder check go through dl.syq.christmas, a host
run by the maintainer that serves the GitHub release files from a cache. It
records each request’s time, syq version, platform, the connection’s IP
address, and the country, region, and city derived from that address, so the
project can see how many installs exist and which versions are in use.
Nothing identifies an install, and the check sends nothing else.
Non-interactive use never makes the reminder check. An installed official syq
verifies self-updates and helper downloads against a signed release manifest.
See Downloaded executables for how that
verification works and how trust is established during the first installation.
Shell completion
Add the line for your shell to its startup file:
# Bash (~/.bashrc)
eval "$(syq completion bash)"
# Zsh (~/.zshrc), after autoload -Uz compinit && compinit
source <(syq completion zsh)
# fish (~/.config/fish/config.fish)
syq completion fish | source
Completion suggests options, hosts, and paths, with file details beside path
matches. In Bash, press Tab again to list matches. Remote paths use your usual
SSH login. Open a new shell after adding the setup line or upgrading syq.
See syq completion for all options.
Keep connections open
Use persistence to reuse SSH connections across syq commands and make your laptop available to connected servers.
Copy files
syq cp project --into backup
This copies project to backup/project. Existing files are updated when
needed; unrelated files stay.
See syq cp for every option, or run syq cp --help.
Choose the files, then choose their destination. --from and --to select
machines or buckets; omit them for local paths. Placement options decide
whether to keep the source name or give it a new one.
See where files go
A named directory brings its name along. --srcs-in copies its contents;
--as chooses an exact destination name.
syq cp project --to server --into /backupCopies project into /backup/project on server. Existing files outside that directory stay.
Each example starts from the same files shown under Before.
Your machine
project/
├── index.html
└── images/
└── logo.svg
Server · Before
/backup/ ├── index.html (old) └── notes.txt
Server · After
/backup/
├── index.html (kept)
├── notes.txt (kept)
└── project/ (new)
├── index.html
└── images/
└── logo.svg
Note to rsync users: trailing slashes have no special significance in syq’s
native commands. syq rsync keeps rsync’s slash rules.
Copy over the network
Use --to to send files, or --from to fetch them:
# Copy project to /backup/project on server.
syq cp project --to server --into /backup
# Fetch /data/a and /data/b from server into ./data/a and ./data/b here.
syq cp --from server -C /data --src a --src b --into ./data
-C DIR is shorthand for --cwd DIR: look for the source files in DIR.
In the second example, /data is on the server and ./data is on your machine.
To use the default destination:
syq cp project --to server # put project in your home directory on server
syq cp --from server project # fetch project into your current directory
SSH endpoints use [USER@]HOST[:PORT], for example alice@server:2222.
Enclose IPv6 addresses in brackets: alice@[2001:db8::1]:2222.
A colon in a native path is simply part of the path.
For TCP data connections, syq can try other addresses advertised by the same receiver if the initial address is unreachable. See TCP access if copies fall back to SSH.
Use --to @NAME to send local source files to a registered receiving machine.
The @ is required: --to laptop selects an SSH destination, while
--to @laptop selects your connected receiving machine.
See Use your laptop from a server for setup and destination paths.
Use --from s3://BUCKET or --to s3://BUCKET for object storage. See
Use S3 storage for credentials, examples, and filesystem
differences.
For two SSH endpoints, see Copy between servers.
Transport compression
Remote filesystem copies compress data in transit by default, adjusting
compression to the observed transfer speed. Blocks that do not shrink enough
are sent uncompressed. Use --no-compress to disable transport compression,
for example when comparing its effect on CPU use and copy speed.
Progress
Syq shows a progress bar in a terminal, with elapsed time, speed, and an
estimated finish time. The final summary reports copied and skipped files
and any errors. Add -v to list copied paths.
Use --progress to show the bar when output is redirected, or --no-progress
to hide it. For connection details and ways to investigate performance, see
Speed.
Choose a destination
| Option | Meaning |
|---|---|
--into DIR | Put the selected names inside DIR, creating it if needed |
--as PATH | Give one source this exact destination name |
--into-new DIR, --as-new PATH | Also require the destination not to exist |
--into-existing DIR, --as-existing PATH | Also require it to exist |
# Rename the copy and refuse to overwrite an existing entry.
syq cp report.txt --as-new reports/final.txt
--as can rename a directory too. If you select several sources, each needs
a different destination name. Use mappings to rename them
individually.
Syq refuses to replace a directory with a file, or a file with a directory. See filename and type conflicts when copying between filesystems with different naming rules.
Choose which existing files to update
By default, syq adds missing files and updates files that differ. Use
--only-new to add missing entries and leave existing ones alone.
# Import new files without replacing existing files.
syq cp --only-new --srcs-in incoming --into archive
See Update policies for supported combinations.
Preview changes
Add --dry-run -v to list planned changes without copying or deleting files.
Include --hash to compare file contents during the preview:
syq cp --dry-run -v --srcs-in project --into backup
A dry run can still install syq on the server; see Automatic installation on SSH servers.
Mirror a directory
Use --prune to remove destination entries that are absent from the source:
syq cp --prune --max-delete 100 --srcs-in build --into-existing deploy
This makes the contents of deploy match build: it copies changes, then
removes extras. Preview with --dry-run -v first. If more than 100 removals
are planned, syq refuses all deletions. Scan or copy errors also prevent deletion.
Ignored paths are kept.
Placement determines where pruning happens. Compare:
# Mirror build inside backup/build; leave the rest of backup alone.
syq cp --prune build --into backup
# Mirror build directly inside backup; remove extras throughout backup.
syq cp --prune --srcs-in build --into backup
See Pruning for restrictions and files kept for recovery.
Ignoring paths
# Use the project's existing ignore rules.
syq cp --ignore-from .gitignore --srcs-in project --into backup
# Skip node_modules and object files.
syq cp --ignore node_modules --ignore '*.o' --srcs-in project --into backup
Patterns use gitignore syntax. Rules run in command-line order; the last
match wins. ! re-includes a path.
| Pattern | Matches |
|---|---|
foo | A file or directory named foo at any depth |
/foo | foo at the source root |
foo/ | Directories named foo |
* | Within one path component |
** | Across path components |
Ignored directories are not scanned. To keep part of one, include the parent:
# Skip other logs, but copy logs/keep and its contents.
syq cp --ignore 'logs/*' --ignore '!logs/keep/' --srcs-in project --into backup
Ignored paths are also protected from pruning.
Resume an interrupted copy
Rerun the same command. Syq skips completed files and can reuse matching parts of interrupted files. It assembles each updated file beside the destination and replaces the destination when complete.
Partial files may remain after a successful retry. To remove them:
syq clean-partials --dry-run -v backup
syq clean-partials backup
See syq clean-partials for remote cleanup and
recovery entries that need inspection before removal.
Check file contents
Syq normally skips files whose size and modification time match. Use --hash
to compare their contents instead:
syq cp --hash --srcs-in project --into backup
Use per-file expected hashes in mappings to require known contents, including when reusing destination bytes.
Use --dry-run --hash to compare without copying.
The Integrity checking reference covers timestamp precision, every comparison and payload-check algorithm, expected hashes, and transfer checks. For consistent source data, stop concurrent writers or copy a snapshot.
In-place writes
By default, syq builds an updated file beside the old one and replaces it when
complete. --inplace writes directly into the destination file instead:
syq cp --inplace large-file --to server --into /backup
This avoids the disk space for a second full copy and can reduce disk I/O. However, readers can see a mixture of old and new contents during the copy or after an interruption. Writes through a hard link also affect its other names. See Update policies before combining in-place writes with other copy policies.
Preserve metadata
Syq preserves modification times and copies symlinks as links, like rsync with
-t -l. Existing files keep their destination permissions. New files use the
source read, write, and execute permissions, limited by the destination umask.
For example, a new script with mode 755 stays executable with umask 022.
To preserve source permissions and ownership as well:
syq cp --preserve=permissions,ownership project --into backup
| Syq option | Corresponding rsync option |
|---|---|
--preserve=permissions | -p |
--preserve=ownership | -o -g --numeric-ids |
--preserve=specials | -D (devices and special files) |
Setting ownership requires suitable destination permissions. Syq does not preserve hard links, ACLs, or extended attributes. See the rsync option definitions and metadata details.
Symlinks
To copy through a symlink in a path you supply, use --follow-src for source
paths or --follow-dst for destination paths. --follow enables both and
also follows paths supplied by options such as --ignore-from.
# Copy the directory current-project points to as backup/current-project.
syq cp --follow-src current-project --into backup
# Put project inside the directory that backup points to.
syq cp project --follow-dst --into backup
--as PATH names the entry to replace, so --follow-dst only applies to its
parent directories. For example, syq cp report.txt --as latest --follow-dst
replaces a symlink named latest, leaving its target unchanged.
Links discovered inside a copied directory remain links; these options do not follow them. See Filesystem attacks for the security details.
Choose a source directory
Use -C DIR (or --cwd DIR) to resolve source paths from another directory:
syq cp -C /srv/data reports photos --into backup
This copies /srv/data/reports and /srv/data/photos into backup. The source
base belongs to the machine selected by --from; it does not change where
relative destination paths start.
Use --root DIR instead of -C DIR when selections must stay inside that
directory. Sources must then be relative: ../private is refused, and even
with --follow-src, symlinks cannot lead outside the root. -C permits both.
Shell pipelines and file descriptors
Use --src-fd 0 to read bytes from standard input, or --as-fd 1
to send a file’s contents to another program:
# Save generated data directly on the server.
python make_report.py | syq cp --src-fd 0 --to server --as report.csv
# Feed a remote file to a local program.
syq cp --from server report.csv --as-fd 1 | python analyze_report.py
Replace the Python scripts with your own producer and consumer commands. The same options work with local files and S3 objects. Other descriptor numbers let scripts pass files they have already opened.
Check the whole pipeline’s status: a failed producer can leave syq with an incomplete stream that ends normally. See file descriptors for process substitution, named pipes, and failure handling.
For environment overrides and cache locations, see Environment and local files.
Use S3 storage
Use buckets with cp and rm:
--from s3://BUCKET, --to s3://BUCKET, or --on s3://BUCKET.
Selectors and placement options name keys within the bucket.
The bucket must already exist.
S3 options
Syq uses your AWS credentials and detects AWS bucket regions automatically.
| Option | Use |
|---|---|
--s3-profile NAME | Select an AWS profile |
--s3-endpoint URL | Use an S3-compatible service; also accepts AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL |
--s3-region REGION | Set the signing region explicitly |
--s3-header 'NAME: VALUE' | Add a provider header to every request; repeatable |
See S3 copies for concurrency, part sizes, and retries.
Upload and download
syq cp photos --to s3://backups --into laptop
syq cp --from s3://backups laptop/photos --into restored
The first command copies photos under laptop/photos in the bucket; the
second downloads it into restored/photos. Add --dry-run to preview either
copy. Use --srcs-in when you want a directory or prefix’s contents instead
of its name. See Copy files for placement and filtering.
To copy between buckets in the same service:
syq cp --from s3://backups --srcs-in laptop --to s3://archive --into laptop
For removal examples, see Remove files. Read Versions and deletion before deleting versioned objects.
Filesystem differences
- Buckets and prefixes: keys must be relative UTF-8 paths. A named source selects an exact object if present, otherwise its
NAME/prefix. Use--srcs-infor prefix contents. A prefix exists when it contains objects, including an empty directory marker. - Metadata: syq stores timestamps, permissions, ownership, and symlinks in
object metadata or contents. Downloads restore timestamps and symlinks;
--preserverestores permissions or ownership. Special files are unsupported. - Updates:
--only-new,--into-new, and--as-newprotect individual objects against concurrent creation. Prefix checks are not transactional.--inplaceand SSH/S3 combinations are unsupported. - Recovery: rerun an interrupted copy to reuse multipart work. If you abandon an upload, remove its unfinished parts with provider tools or a lifecycle rule.
Copies between S3 buckets
Bucket-to-bucket copies run within one service, using the same endpoint, region, and credentials. They preserve metadata and tags. Changes to tags, encryption, or storage class alone do not trigger a copy. Content hashing and expected hashes are unsupported on this route.
Authorize from your laptop
Add --auth-from @NAME to an S3 cp or rm command to use credentials on a
connected receiving machine.
--s3-profile selects a profile there:
syq cp results --to s3://my-bucket --into runs \
--auth-from @laptop --s3-profile storage
Approve on your laptop, then wait for storage authorization ready before disconnecting. Data travels directly between the server and storage. Authorization lasts up to seven days, subject to credentials and provider policies; restarting requires fresh approval. Both machines need the same syq build. See Storage authorization for the security implications.
Descriptor copies
With --src-fd or --as-fd, cp transfers one exact UTF-8 key’s raw contents
instead of selecting a prefix tree. Keys are literal unless a download uses
--cwd or --root; those options resolve the source relative to a prefix using
the usual S3 path rules. Regular-file uploads store syq’s file metadata. Output
descriptors receive raw bytes; use --preserve to apply attributes to a regular
output file. Pipes carry only bytes. No local temporary file is created. See
File descriptors for command examples.
S3 descriptor copies default to four parallel 16 MiB parts, with about one
part of payload buffering per worker plus one input/output part. Use s3-part-size and s3-retries to change the part size and retry budget.
The usual S3 request and part concurrency controls apply; object concurrency
is always one. Unknown-length uploads stop at 10,000 parts: 156.25 GiB at the
default part size. Select a larger part size before starting a larger upload.
Buffered parts can be retried, but there is no restart recovery. Uploads replace the object only on completion. On failure, syq attempts to abort the multipart upload; unconfirmed cleanup may need provider tools. A lost completion response can mean an upload was published despite a reported failure.
Versions and deletion
Ordinary rm and cp --prune retain historical versions in versioned buckets.
Use rm --s3-all-versions to permanently remove selected versions and delete
markers, or --s3-version-id ID for one version. Deleting a marker can reveal
an older version. Preview version deletions with --dry-run -v.
Named removal selectors choose exact keys; --src-dir and --srcs-in choose
prefix trees and accept a trailing /. When deleting an exact directory-marker
version, keep the trailing / in its key. An empty S3 source prefix is rejected
by cp, so it cannot prune an entire local destination.
Remove files
Remove a file or symlink:
syq rm old-file
Named paths, --src, and --srcs refuse directories. Select a tree explicitly
to remove it recursively:
syq rm --src-dir old-output
Remove a directory’s contents recursively, leaving the directory itself:
syq rm --srcs-in cache
Add --dry-run -v to preview what would be removed first.
A missing path is not an error. Filesystem removal is permanent; completed
deletions cannot be rolled back.
On another machine
syq rm --on server --src-dir /scratch/old-output
Keep the connection open until removal finishes. A dry run can still install syq on the server; see Automatic installation on SSH servers.
For object storage, use --on s3://BUCKET:
syq rm --on s3://backups --src-dir old-backup --dry-run -v
Ordinary removal respects bucket
versioning; explicit version deletion is available with --s3-all-versions or
--s3-version-id. See Versions and deletion.
Limit the selection
syq rm --root /srv --src-dir cache --src-dir old-output
This removes /srv/cache and /srv/old-output, with selection confined to
/srv. Syq refuses paths or symlinks that lead outside that directory.
See Selection rules for the supported selectors.
Symlinks
Removing a symlink leaves its target alone. This applies to selected links and links found inside a directory being removed.
Use --follow-src to reach a file through a symlink in its parent path:
# If current points to releases/v1, remove releases/v1/log.txt.
syq rm --follow-src current/log.txt
syq rm --follow-src current still removes only the link. To remove a linked
directory or its contents, select the actual directory. With --root, paths
must stay inside that root.
Results
If some entries cannot be removed, syq reports the errors and continues with independent entries. Completed deletions remain in effect. Use automation results for per-path outcomes in scripts.
See syq rm for all options.
Copy between servers
Copy directly between servers without putting private keys on either server or giving one server unrestricted access to your SSH agent. Your machine authorizes the copy and shows the results; the file data bypasses it.
Run the copy from your laptop
# Copy the contents of hostA's big directory into hostB's big directory.
syq cp --from hostA --srcs-in big --to hostB --into big
HostA gets permission for this transfer only. HostB checks that permission and reports what it changed. See A compromised source server for what this protects against.
What you need
- SSH access from your machine to both servers, with their host keys already trusted. Connect with ordinary SSH once if either server is new to you.
- An SSH agent on your machine, and OpenSSH 8.9 or newer on your machine, hostA’s SSH client, and hostB’s SSH server.
- SSH connectivity from hostA to hostB. A reachable TCP data port
on hostB, normally in
47600–47699, enables encrypted TCP workers; see Make TCP reachable. Otherwise the copy uses SSH workers on the same hostA-to-hostB route.--no-tcpselects SSH directly. - An existing parent directory for the destination.
Keep your laptop command running until the copy finishes.
First copy and access management
The first copy sets up a restricted receiver on hostB automatically. It adds
a restricted key to authorized_keys; the private key stays on your machine.
Later copies reuse this setup.
To prepare /archive on hostB ahead of time, including before a dry run:
syq receiver enroll hostB:/archive
syq cp --dry-run -v --from hostA --srcs-in data --to hostB --into /archive
Inspect or remove this access with:
syq receiver list
syq receiver revoke ID
Use the ID from list. Revocation stops active copies for that enrollment
and removes its access; completed writes remain. If cleanup fails, retry
receiver revoke. See Enrollment
for upgrades and sharing between installations.
If your machine reaches hostB through hostA, add --via hostA to enroll or
revoke.
Run the copy from a server
You can also start a copy in the source server’s shell and use your laptop’s SSH credentials to authorize it. This uses a receiving connection from your laptop; see Authorize copies between servers for setup, approval, and connectivity requirements.
Mirror a directory
From your laptop, include a deletion limit when pruning:
syq cp --prune --max-delete 100 --from hostA --srcs-in data --to hostB --into-existing /archive
This updates /archive and removes extras. If more than 100 removals are
planned, none are performed and the command exits 25. Preview first with
--dry-run -v.
Other routes and authentication
Copies started from your laptop can switch between encrypted TCP and SSH on the selected route. Syq never silently relays file data through your machine when a direct connection fails. If the servers cannot connect directly, explicitly relay through your machine:
syq cp --coordinate-at local --from hostA --srcs-in data --to hostB --into /archive
This uses your machine’s bandwidth and ordinary SSH access to each endpoint.
Other authentication modes can use server-held credentials, a destination- restricted SSH agent, or full agent forwarding. They grant different authority; see the Remote copy reference before choosing one. That page also covers detached copies, signed results, and direct-mode limits.
Keep connections open
Persistence keeps SSH connections ready for repeated copies and other syq commands. It is off by default. Enable it and connect to a server without copying files:
syq persist connect server
You can close the terminal afterward. To enable persistence for connections
opened by later syq commands instead, run syq persist on.
Persistence also speeds up remote path completion: completion reuses the open connection, avoiding a new SSH login for each lookup.
Receiving starts automatically with persistent connections unless you have turned it off. It lets connected servers request file copies to your machine, commands on it, and authorization for copies between servers, with approval on your machine. See Use your laptop from a server for setup and approval controls.
To keep SSH connections open only for commands you start locally, turn
receiving off with syq persist receive off. The persistent SSH login remains
usable by processes running as your local user even if your SSH key or agent
is no longer available. See Persistent connections
for the security implications.
Inspect connections with syq persist status. Close them, including receiving
connections, with syq persist off.
Receiving reconnects after a network interruption or laptop sleep; other SSH
connections reopen on their next use. An interrupted copy still needs to be
rerun to resume. After rebooting, run syq persist connect server again.
See Persistence details for troubleshooting,
upgrading, and isolated connections for scripts, or syq persist
for all commands and options.
Use your laptop from a server
Receiving lets you use your laptop from a connected server:
- Copy files to your laptop.
- Run commands on your laptop.
- Authorize copies between servers using your laptop’s SSH credentials.
- Authorize object-storage transfers using your laptop’s storage credentials.
These use a persistent connection opened by your laptop. It needs no SSH server, public address, or incoming network port.
Set up receiving
Receiving starts automatically when syq opens a persistent SSH connection, unless you have turned it off or restricted its profiles to other servers. To enable persistence and connect to a server now, run this on your laptop with syq installed on both machines:
syq persist connect server
This opens the connection and waits until receiving is ready. An ordinary
ssh server session does not enable receiving. If you already have a persistent
connection to this server, you do not need to connect again. For example, after
syq persist on, a syq copy or remote path completion can open that connection.
If you previously turned receiving off, run syq persist receive on first.
By default, your receiving name is your laptop’s short hostname, and downloads
and commands start in your home directory. connect prints the receiving name.
The examples below use @laptop; replace it with your own name.
Once connect finishes, you can close that terminal and make requests from any
shell on the server, including an existing tmux session.
Optional name and directory
To create a receiving profile named laptop with a different starting
directory, run these commands on your laptop:
mkdir -p ~/Downloads/server
syq persist receive on --name laptop --cwd ~/Downloads/server
Here, receive on configures the profile; it is not required to use the default
settings. --cwd sets the starting directory for downloads and commands.
Copy files to your laptop
On the server, send files to your receiving directory:
syq cp results --to @laptop
syq cp report.pdf --to @laptop --as reports/latest.pdf
File data uses encrypted TCP when your laptop can reach a data port on the
server, with SSH as a fallback. Your laptop opens both connections; it still
needs no incoming port. Use --no-tcp to send all data through SSH.
See Make TCP reachable for server setup.
Approving copies
By default, each incoming copy waits for approval on your laptop. Review the destination
and permissions, then choose Allow once or Deny. To see the complete
request, use Details on macOS or syq persist receive pending in a local
terminal. You can also approve or deny there:
syq persist receive pending
syq persist receive approve REQUEST_ID
syq persist receive deny REQUEST_ID
Approving a copy trusts the server to supply its contents. Requests expire after
five minutes. If a prompt is missing or dismissed, the request stays pending;
it is never approved automatically. To use only terminal approval, run syq persist receive on --name laptop --notify off.
Names and paths
Use --to @laptop for your connected receiving machine. Without @,
--to laptop names an SSH destination instead.
Without --into or --as, files go into the profile’s starting directory,
shown by syq persist receive status. Use either option to choose a path
relative to that directory, or an absolute path to copy elsewhere.
To contain copies within a directory instead:
syq persist receive on --name laptop --root ~/Downloads/server
With --root, incoming paths must be relative and stay inside that directory,
even when following symlinks. Use --no-root to remove this restriction;
changing --cwd does not remove it. Unless you set --cwd, the starting
directory follows the hard root, then the automatic approval root, then your
home directory. See Directories for
changing or resetting these settings.
Changing a profile’s settings stops its active copies, commands, and pending requests. Other profiles keep working.
Skip approval for downloads
Choose a directory for downloads that do not need approval:
syq persist receive on --name laptop --auto-approve-root ~/Downloads/server
Downloads confined to that directory need no approval; downloads elsewhere ask.
--root, if configured, remains a hard boundary even with approval.
Automatic approval trusts all processes running as the connected server accounts,
including for overwrites inside that directory. You can
limit a profile to particular servers.
Commands on your laptop, authorization for copies between servers, and storage authorization require approval every time. See Receivers for the trust boundary.
To require approval for every download again:
syq persist receive on --name laptop --no-auto-approve-root
Copy permissions and limits
By default, each copy is limited to 100 GiB and one million entries. Pruning
requires a positive deletion limit on both machines. Change limits with
syq persist receive on --name laptop --max-bytes SIZE --max-entries N --max-delete N.
Most copy options work here; ownership preservation, special-file preservation,
and --inplace are unsupported. See
copy limits for details.
Run commands on your laptop
From the server, request a command in a project directory on your laptop:
syq exec --on @laptop --cwd /path/to/project -- make
syq exec --on @laptop --cwd /path/to/project -- open report.html
The second command uses macOS’s open program to display a report. Replace
it with any program installed on your laptop.
Every command requires approval on your laptop and runs with your local user’s permissions, including access to files and credentials. The download root and copy limits do not restrict commands. Build tools and scripts can execute code from their input files, so consider those files when approving a command.
Output streams back to the server terminal, and syq returns the command’s exit
code. Interrupting the request or stopping receiving stops the command;
completed changes are not rolled back. See syq exec
for arguments, working directories, and cancellation details.
Authorize copies between servers
Use your laptop’s SSH access to copy directly between servers, while running the command in your source server’s shell:
# Run on hostA, including in an existing tmux shell.
syq cp results --to hostB --into /archive --auth-from @laptop
Your laptop asks for approval for each copy, then uses its SSH access to hostB to authorize it without giving the source server your private SSH keys. Trust hostB’s SSH host key on the laptop beforehand. Relative destination paths start in the hostB account’s home directory; your laptop’s receiving root does not contain this copy, but its transfer limits still apply.
Files go directly from hostA to hostB over encrypted TCP. HostB needs a reachable data port; see Make TCP reachable. This route cannot use SSH for file data. Keep the laptop connection and source command running until completion. See Authorization selection for automatic selection, other authorizers, and supported options.
See Multiple profiles and server-specific settings.
Rename and reorganize during a copy
List the files, change their destination names, then copy them. syq map
produces the list, a script changes it, and syq cp --mapping makes the copy.
For Python scripts, see the Python SDK and its mapping examples.
Lowercase destination names
set -o pipefail
syq map --srcs-in src \
| jq -c '.dst.value |= ascii_downcase' \
| syq cp --mapping - -C src --to nas --into /pub
src/ nas:/pub/
Berlin/IMG_1234.JPG berlin/img_1234.jpg
Notes.TXT notes.txt
If another entry also claims notes.txt, the copy is refused before files
are transferred. Symlink target text is not rewritten: renaming its target
can leave a link dangling on a case-sensitive destination.
Emitting a mapping
syq map --srcs-in photos # contents; paths relative to photos
syq map photos # named directory; paths include photos/
syq map photo.jpg --as albums/cover.jpg
map lists local files without copying them. Use the same source directory
when making the copy: a list produced with --srcs-in photos needs -C photos
on cp. Each listed directory includes separate entries for its contents.
Group photos by modification month
set -o pipefail
syq map --srcs-in photos \
| jq -c 'select(.kind == "file")
| .dst.value = (.mtime | gmtime | strftime("%Y/%m")) + "/" + .dst.value' \
| syq cp --mapping - -C photos --to nas --into /archive
A July 2024 file IMG_1234.JPG lands at /archive/2024/07/IMG_1234.JPG.
The filter keeps regular files only; missing parent directories are created.
Dates use file modification time in UTC, not photo EXIF dates.
Check the producer before copying
A pipeline’s consumer sees only the bytes it receives. If a generator fails
after emitting valid entries, those entries can still be copied.
set -o pipefail makes the pipeline report failure, but does not undo writes.
To require successful generation before copying, save the manifest first:
set -o pipefail
syq map --srcs-in src | jq -c '.dst.value |= ascii_downcase' > m.ndjson \
&& syq cp --mapping m.ndjson -C src --to nas --into /pub
Add --dry-run -v to cp to preview placement.
The format
A manifest contains one JSON object per line (NDJSON):
{"src":{"encoding":"utf-8","value":"IMG_1234.JPG"},"dst":{"encoding":"utf-8","value":"2024/07/photo.jpg"},"kind":"file","size":4194304,"mtime":1721900000}
Each line names a source and destination relative to the directories you give
cp. You can generate this list with any program:
syq cp --mapping pairs.ndjson -C photos --to nas --into /archive
See Mapping format for fields, encodings, and validation rules.
Use --mapping in place of source selectors, with an --into placement.
For supported combinations and path rules, see
Mapping restrictions.
Copy between servers
The manifest is read on the machine where you run the command. Its source paths resolve on the source server, and its destination paths resolve beneath the destination container:
syq cp --from hostA -C /data --mapping pairs.ndjson --to hostB --into /archive
File contents travel directly between the servers. See Copy between servers for setup.
Machine-readable results
Add --results r.ndjson to record outcomes in a fresh file outside the copy
trees. After fixing a failure, rerun the original mapping to finish the copy.
For scripts that select only failed entries to retry, see
Retry failed mapping entries.
Use rsync-style commands
Use syq rsync for familiar rsync flags and source/destination syntax with
syq’s transfer engine. It supports local copies, SSH pushes, and SSH pulls:
syq rsync -av project/ backup/project/
syq rsync -av project/ server:backup/project/
syq rsync -av server:project/ backup/project/
-a selects archive mode; -v lists files. A trailing slash on the source
copies the directory’s contents. Without it, the directory itself is copied.
Add -n to preview a copy without changing source or destination files:
syq rsync -avn project/ server:backup/project/
Remote copies use syq’s protocol and helper, installed automatically over SSH;
see Automatic installation on SSH servers.
They cannot communicate with an rsync server or daemon. To copy between two
servers, use native syq cp.
Before adapting an existing command or script, check
Rsync compatibility: some options are unsupported, and
filter rules and deletion limits differ. See syq rsync
for all accepted options.
Speed
Syq copies files in parallel and adjusts its connection count automatically. Start with the defaults. To avoid logging in for every remote copy, see Keep connections open.
Quick comparison
Use the quick script to get a sense of performance on your own machines:
curl --proto '=https' --tlsv1.2 -fLsS -o try-benchmark.sh \
https://raw.githubusercontent.com/greaber/syq/master/scripts/try-benchmark.sh
bash try-benchmark.sh
By default, the script copies 1,024 files of 8 KiB to an SSH host you choose,
comparing syq with rsync over three rounds. It creates disposable test data,
checks every copy, and cleans up afterward. Use --mode pull for downloads or
--mode local to include cp.
Network tests first warm up syq’s connection tuning. This targets about a minute
of copying, but setup and slow transfers can take longer. It uses up to four
copies of datasets no larger than 1 GiB each, subject to free space. Use
--warmup off for a shorter comparison.
Compare tools using the main table: it includes connection startup in total command time. Network results show average MB/s; local results show seconds, since filesystem cloning may avoid moving the bytes. The separate syq timing table helps diagnose setup costs; its copying-only rate is not comparable with another tool’s total-time rate.
Run bash try-benchmark.sh --help for workload sizes, dependencies, scratch
directories, and custom syq options. If trials are too short to measure sustained
speed, choose a larger workload.
What the comparison measures
Each timed copy uses the same data and an empty destination, preserving permissions and modification times. Generation, helper preparation, warm-up, and content checks are untimed. Failed copies or content checks stop the test.
Network tests disable SSH connection reuse for both tools. Syq’s learned connection counts remain active unless you override tuning; warm-up does not guarantee that tuning has settled. Caches are not flushed, and copies do not wait for durable storage. Local cloning times therefore do not measure disk bandwidth.
Benchmarks
The separate syq-bench project runs more extensive experiments across workloads, storage systems, and network routes. Browse its results, or run its experiments for a more detailed comparison. The quick script above does not use syq-bench.
| Tool | Average speed |
|---|---|
| syq | 159.9 MB/s |
| syq over SSH | 88.3 MB/s |
| rsync | 18.3 MB/s |
Diagnose a slow copy
syq cp -vv --stats data --to server --into /backup
-vv shows the chosen transport and connections; --stats adds totals and
available TCP statistics, worker wait fractions, endpoint operations and bytes,
and process CPU. These wait fractions
help locate delays; they are not proof of their cause. Add --results run.ndjson
to inspect how worker waits, endpoint operations, CPU and TCP backpressure change
over time. Remote evidence includes its age. See the
performance measurements reference for interpretation and limitations.
| Symptom | Try |
|---|---|
| Data falls back to SSH | Check Make TCP reachable |
| Many short commands spend time logging in | syq persist on |
| CPU is saturated on a fast link | Compare with --no-compress |
| A long-distance path suffers loss | Investigate Test congestion control |
Compare the same workload and direction using empty test destinations. A second copy into the same destination may just measure skipping existing files.
TCP data connections
SSH authenticates remote copies. When reachable, encrypted TCP carries file
data on a port in 47600–47699; otherwise copies use SSH on the same route.
See Make TCP reachable for firewall settings.
To Run the copy from a server
with your laptop’s approval, direct encrypted TCP must be reachable.
Use --no-tcp to select SSH data transport or --tcp-ports LO-HI to choose a
port range. On Linux, --tcp-congestion ALGO selects an available congestion
control algorithm. --tcp-plain disables data encryption and authentication;
use it only on a trusted network. Restricted receivers refuse it.
Local copies and NFS
For local copies, syq uses the filesystem’s copy optimizations automatically when it can. On filesystems that support cloning, this can avoid physically copying every byte. You can also copy to or from a mounted NFS directory using its local path. See Check local storage placement for how the source and destination filesystems affect performance.
Reported bytes count the file’s size even when cloning avoids physical I/O, so the displayed rate can exceed disk throughput.
Limit bandwidth
Use --resource-limits bandwidth=RATE to leave bandwidth for other work:
syq cp data --to server --into /backup --resource-limits bandwidth=10M
This limits file data to 10 MiB/s across the copy’s workers. It controls the average copy rate; buffering and protocol overhead can cause network bursts. See Resource limits for all units and supported routes.
Compression and in-place writes
--no-compress can help when compression costs more CPU time than it saves
in network traffic; see Transport compression. --inplace saves temporary
disk space, but exposes incomplete updates to readers. Read
in-place writes before using it.
Investigate tuning
Manual connection counts and other tuning options are for troubleshooting and development. Normal copies should not need them.
Server setup
Start by checking TCP access and the source and destination filesystems. Use test data when comparing settings.
Make TCP reachable
Syq listens on one available port in 47600–47699 during a copy. Choose another
range with --tcp-ports LO-HI. For a server using ufw, an administrator can allow
a trusted client:
sudo ufw allow from <trusted-client-address> to any port 47600:47699 proto tcp
Allow the range in any cloud firewall too. Check the transport with
syq cp -vv --stats. Ordinary copies fall back to SSH on the same route when
TCP is blocked. To Run the copy from a server
with your laptop’s approval, direct encrypted TCP must be reachable.
On Linux, syq can also discover TCP addresses on IP over InfiniBand (IPoIB) interfaces.
Tailscale
Tailscale can make servers reachable
across NAT and firewalls. Allow syq’s data ports through the host firewall and
tailnet rules; syq can discover Tailscale addresses. Use tailscale status to
check whether the connection is direct or relayed. See
Tailscale’s performance guide.
Test congestion control
On Linux, check which TCP congestion-control algorithms are available and allowed on both endpoints:
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_allowed_congestion_control
If bbr is listed in both, compare it with cubic on your route:
syq cp --tcp-congestion bbr --stats data --to server --into /backup
Use the same data, a fresh destination, and both transfer directions. The option applies to syq’s TCP sockets. If BBR is missing, see the administrator setup guidance.
Let SSH connections start promptly
OpenSSH’s MaxStartups limit can slow parallel logins. For servers handling
parallel transfers, an administrator can consider:
MaxStartups 100:30:200
This allows 100 unauthenticated connections before random rejection begins,
and rejects all new ones at 200. It also admits larger bursts from unrelated
clients. MaxSessions controls channels sharing a connection; very low values
can force extra logins.
Validate changes with sshd -t, then reload SSH using your system’s procedure.
Keep an administrative session open. See OpenSSH’s settings.
Check local storage placement
On Linux, inspect the source and destination filesystems:
findmnt -T /path/to/source -o TARGET,SOURCE,FSTYPE,OPTIONS
findmnt -T /path/to/destination-parent -o TARGET,SOURCE,FSTYPE,OPTIONS
Use an existing destination parent and run remote-path checks on the machine that owns the path. Copies within a filesystem supporting cloning can share storage while remaining independently writable. See local copies and NFS for filesystem and mount considerations.
Measure and track improvements
Use syq-bench for repeatable comparisons. Record the commands, versions, mounts, cache state, and other load. Keep reporting and flush settings consistent across runs. See performance tuning to compare worker counts and request sizes.
Security
Syq often acts with more authority than the files or machines involved in a copy. A backup account may read another user’s directory, or your laptop may authorize a copy between two servers. The protections below limit how those files and servers can use that authority.
Report vulnerabilities through SECURITY.md.
Filesystem safety
Filesystem attacks
Suppose a backup account is copying an upload directory that another user can change. That user could replace a subdirectory with a symlink to private files elsewhere. A copy program that follows the link would read those files using the backup account’s permissions. At a destination, the same trick could redirect writes or deletions into an unrelated directory.
Checking a path before starting is not enough: someone who can write its parent directory may replace it while the copy is running. Syq therefore keeps directories open and accesses their entries through those open handles. Renaming an opened directory does not change which directory the handle refers to, and replacing its old name with a symlink does not redirect subsequent operations through that handle. These checks assume the attacker can change directory entries but does not control the account running syq.
The same principle applies to names supplied by a remote machine and to removal:
- A remote source tries to escape the destination. If you are copying into
backup, a peer cannot make an entry named../private/keyor/private/keywrite outside it. Syq rejects those names. - A directory becomes a symlink during copying. If
backup/photosis replaced with a link to another directory, syq either continues through a handle it already opened or refuses to traverse the link. It does not start writing into the link’s target. - An entry changes during removal. If a file selected for removal becomes a symlink, removing the link leaves its target alone. If it becomes a directory, syq does not start recursively deleting that directory’s contents.
Which symlinks are followed?
Native syq distinguishes paths you explicitly select from entries discovered inside a tree:
| Where the symlink appears | Treatment |
|---|---|
| In a source path you supply | --follow-src permits following; a named link otherwise copies as a link |
| In a destination path you supply | --follow-dst permits following; --as PATH always replaces the final entry itself |
| Inside a source directory being scanned | Copied as a link; its target is not scanned or read |
| Inside a destination directory being scanned | Inspected as a link; its target is not scanned or written through. A copied file can replace the link, but a copied directory cannot use it as a directory |
--follow enables both source and destination following. These options affect
supplied paths only; they do not change either scan rule. See Symlinks for examples.
Relationship to rsync 3.5.0
Syq uses the same basic approach as rsync 3.5.0’s security design: keep directories open and resolve later operations relative to them. The policy for following links in supplied paths differs:
| Command | When a symlink in a supplied path may be followed |
|---|---|
Rsync 3.5.0 or syq rsync | The link belongs to root or the process’s effective user, or --insecure-links is set (local paths only in syq rsync) |
| Native syq | You explicitly request following with the applicable follow option above |
Limits of path protection
These protections apply when running as root too. They prevent paths from redirecting an operation; they do not make all changes within the selected directory safe. The following risks require separate attention, especially when syq has more permissions than the people who can modify that directory.
Hard links and --inplace. Suppose backup/report and a file outside
backup are hard links to the same file. Copying over backup/report with
--inplace changes the contents visible through both names. Avoid --inplace
when you cannot trust the destination’s existing files. Metadata changes,
such as permissions or timestamps, also affect every hard link to that file,
even without --inplace.
Permissions supplied by the source. Path protection does not decide
whether copied permissions are appropriate. For example,
--preserve=permissions can make a copied file writable by everyone or
preserve its set-user-ID bit; --preserve=ownership can give it to the account
identified by the source’s user ID. When running as root, those choices can
grant other users access or privileges. Enable these options only when you
trust the source’s ownership and permission settings.
Files left by an interrupted copy. Someone could plant a symlink or hard link where an old partial file should be, hoping that resuming will overwrite another file. Syq creates its own partial for the new copy. It only reuses bytes from old partials that are regular files owned by its user with no extra hard links, checks those bytes against the source, and leaves the old files unchanged. It does not resume by writing through the planted link.
Concurrent replacement of directory entries. Checking an entry and removing it are separate operations. Someone able to rename files in that directory can replace the entry in between, causing syq to remove the new entry instead. Similarly, publishing a copied file can replace another writer’s new file. These races do not cause syq to follow a replacement symlink, but they can lose a concurrent writer’s changes. Use a destination other processes cannot modify if those changes must be protected.
Named pipes. If someone replaces an input file with a named pipe just before syq opens it, the open can briefly wake the pipe’s writer before syq detects the change and fails. That writer may send data that is then discarded. On macOS, rapidly replacing one pipe with another can also escape syq’s identity check if the filesystem reuses the same inode number. These limitations matter when another process can replace the pipe or change a file’s type during the copy. Keep pipes used by sensitive producers in directories untrusted users cannot modify.
Network security
Remote copies encrypt and authenticate file data, whether it travels through SSH or syq’s direct TCP connections.
--tcp-plain disables this protection for TCP: someone on the network can
read or alter the traffic, including its authentication token. Use it only
on a network you trust.
Downloaded executables
Official releases provide executables for each supported operating system and
CPU architecture. They are published as GitHub release assets and served
through dl.syq.christmas. An installed syq downloads them for explicit
self-updates and, when needed, to install a matching helper on an SSH server.
Remote helpers use the same release as the client, even when the server needs
a different platform’s executable.
A signed release manifest identifies the release and lists the size and SHA-256 hash of each archive and executable. The installed client carries the release public key. It verifies the manifest’s Ed25519 signature with that key, then checks downloaded files against the signed sizes and hashes before using them. The verification key comes from the installed executable, not from the server supplying the download.
This also applies when the SSH server downloads its own helper: your client verifies the manifest and checks the reported archive hash before authorizing installation. If that download cannot be verified, syq discards it and uploads a locally verified helper over SSH instead. A download host cannot substitute arbitrary executable code without a valid release signature. This relies on the release signing key, your installed client, and the SSH account remaining trusted.
The first installation establishes that trust. The standalone installer checks the archive against a size and SHA-256 hash embedded in the script, but it does not independently verify the release signature. You trust the script obtained over HTTPS. Homebrew installations instead start with trust in Homebrew and the tap. When using your own source build, you trust that build; by default syq uploads the running executable as its remote helper. See Install for the installation methods.
Copies between servers
Suppose your laptop can log into hostA and hostB, and you want hostA to send files directly to hostB. Giving hostA a private key or unrestricted access to your SSH agent would let a compromised hostA do more than the intended copy. Syq’s default direct-copy mode lets your laptop authorize the transfer while limiting the access hostA receives.
Destination permissions
The protection has several parts:
- Set up a restricted entry point on hostB. Your laptop uses its normal
SSH access to install a receiver and a dedicated public key. That key’s
authorized_keysentry permits only the receiver command, with SSH forwarding disabled. The private key stays on your laptop. Later copies reuse this setup. - Authenticate hostA’s connection without handing it the key. A small signing service on your laptop answers hostA’s SSH authentication requests. Before signing, it checks OpenSSH’s cryptographic proof of which server the connection reaches, along with the requested login account. HostA cannot use it to authenticate to a different host or account, sign arbitrary messages, or access your other agent keys.
- Authorize the particular copy separately. Your laptop signs a grant stating the permitted destination paths, write and deletion permissions, limits, and expiry. HostB’s receiver verifies the signature, records that the grant has been used, and enforces it throughout the transfer. HostA cannot broaden the grant or redeem it again for a later copy.
- Check the outcome with hostB. The receiver signs a receipt of its changes. Your laptop verifies it using hostB’s receipt key, saved during setup. HostA can relay the receipt, but cannot forge it.
File data travels directly from hostA to hostB, over encrypted TCP or SSH. Your laptop provides authorization and verifies the result without carrying the file data. See Copy between servers for setup and revoking access.
Alternative authentication modes grant more authority. --peer-auth broker
limits authentication to the chosen host and user but allows that account’s
full authority. --peer-auth full-agent exposes your SSH agent as ssh -A
would. See Other authentication modes.
A compromised source server
Suppose the grant permits hostA to copy into hostB’s /archive. A compromised
hostA can misuse the access that copy requires, but cannot grant itself more:
HostA can:
- Supply false contents, names, sizes, or timestamps.
- Overwrite files where the grant permits overwriting.
- Omit a source file and, if pruning is authorized, cause its destination copy to be deleted.
- Inspect destination entries for copy planning and consume the allowed space.
- Stop the transfer or withhold its receipt.
HostA cannot:
- Change the signed destination scope to write into
/etcor an SSH configuration directory. - Overwrite existing files when the grant permits only creating new ones, or delete files without permission.
- Exceed the signed byte, entry, or deletion limits.
- Reuse the grant for another copy or use the restricted key to run a shell.
- Forge hostB’s receipt.
The filesystem limitations above still apply, including effects through hard links. HostB’s account and receiver remain trusted to enforce the grant and report accurately. A verified receipt describes their work; it cannot prove that hostA supplied the right files. For example, a successful receipt can accurately report that hostB wrote false contents provided by hostA.
For checking contents against a trusted hash, see Expected hashes. General corruption, consistency, and durability considerations are covered in Integrity checking.
Persistent connections
A persistent SSH login lets processes running as your local user access the
server without another key touch or agent approval. This access remains
available even with receiving turned off. syq persist off closes the
persistent connections, including receiving.
Receivers
With receiving enabled, servers you have persistent connections to can request
copies to or commands on your machine, or authorization for copies to another
server or object storage. Receiving is configured separately and defaults to enabled. syq persist receive off
disables these requests while keeping SSH reuse.
Requests from a server are subject to local approval:
| Request | Approval on your machine |
|---|---|
| Send files to your machine | Required by default; --auto-approve-root permits unattended downloads confined to that directory |
| Use your SSH access for a copy to another server | Always required |
| Run a command on your machine | Always required |
| Use your storage credentials for a transfer | Always required |
The prompt identifies the server account and requested operation. It cannot
prove who typed the command there. Approving a copy does not approve a later
command. By default, every connected server can use every enabled profile.
A profile’s optional --server list limits which locally selected SSH
connections may use it. Within each allowed server account, all processes
share this authority; choosing a different profile name does not isolate them.
Receiving files on your laptop
Copy approval permits the shown destination, overwrite policy, and limits;
syq enforces them on every filesystem operation. The server can supply false
contents, inspect destination entries during planning, and use disk space
within those limits. The default starting directory is your home, without
containment; syq persist receive on --root DIRECTORY confines copies to that
directory. See Use your laptop from a server for setup and
approval controls. Automatic approval trusts those accounts to overwrite files
inside its root. Choose an inbox whose downloaded contents are not automatically
executed or loaded as trusted configuration.
A receiving name is tied to a public key; reconnecting requires proof of the matching private key. This prevents another client from claiming your name, but the server account can replace its stored name assignments. The receiver’s private key stays on your machine and grants no SSH login access.
Copying between servers
A connected server can ask your laptop to authorize a copy to another server. The copy uses your laptop’s SSH access and the restricted receiver protections described in Copies between servers.
Running commands on your laptop
With syq exec, a connected server can request a command on your
laptop. An approved command runs with your local user’s full permissions;
it is not sandboxed or confined to a copy destination directory.
Storage authorization
Storage authorization gives the server signed URLs for approved paths and operations. The secret access key stays on your laptop. Approval trusts the server to choose uploaded contents. Anyone with the URLs can reuse them until expiry; stopping receiving does not revoke them. Filesystem receiver roots, aggregate limits, one-use grants, and signed receipts do not apply.
Command reference
Look up commands and options below. For copy examples, start with Copy files.
The advanced option groups have separate references:
- Performance tuning: workers, request sizes, and copy methods.
- Resource limits: bandwidth and concurrency ceilings.
- Integrity checking: comparisons and checksums.
Angle brackets mark values you supply; square brackets mark optional arguments,
and ... allows repetition.
syq
syq <COMMAND> [OPTIONS]
syq --self-update
| Command | Purpose |
|---|---|
cp | Copy files and directories, optionally removing destination-only files |
exec | Run a command on a named receiving machine after local approval |
rm | Remove selected files and directory trees |
clean-partials | Delete syq partial files in directory trees |
map | Print source-to-destination mappings as NDJSON |
rsync | Copy using rsync-compatible syntax |
persist | Manage persistent connections, receiving, and return destinations |
completion | Generate shell completion and manage cached endpoint suggestions |
tuning-cache | Inspect and clear local transfer tuning history |
receiver | Manage manual receiver enrollment and recovery |
help | Show help for any command or nested command |
Options
| Argument / option | Meaning |
|---|---|
--self-update | Install the newest signed release (standalone installs); Homebrew: brew upgrade syq |
Help and version
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
--help-all | Show all options and details |
-V, --version | Print version |
syq –self-update
Update a standalone installation. For Homebrew and source installations, see installation updates.
syq --self-update
Options
| Argument / option | Meaning |
|---|---|
--self-update | Update the executable registered by the standalone installer |
Help and version
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
--help-all | Show all options and details |
syq help
syq help [COMMAND...] [--help-all]
Show common help for a command or nested command, such as syq help persist receive on. Add --help-all for its complete help. With no command, show the
root help. A trailing --help or -h selects common help.
Build identity
syq --build-identity prints the executable’s build identity. See development builds.
syq cp
Copy files, directories, and symlinks locally, over SSH, or with S3. See Copy files for examples and remote and S3 copy details.
syq cp --srcs-in project --to server --into backup --dry-run -v
Put source selectors and --mapping before --to or a placement option.
For scripting, see environment variables
and results.
syq cp [OPTIONS] SOURCE... [PLACEMENT]
syq cp [OPTIONS] --src-fd FD --as PATH
syq cp [OPTIONS] SOURCE --as-fd FD
Sources and filtering
| Argument / option | Meaning |
|---|---|
--src-fd <FD> | Read raw bytes from an inherited local descriptor (0 for stdin), instead of a source path |
--from <ENDPOINT> | Source endpoint ([USER@]HOST[:PORT] or s3://BUCKET); omitted means local |
-C, --cwd <DIR> | Resolve relative source selectors from DIR |
--root <DIR> | Resolve source selectors beneath DIR and refuse any escape |
--src <PATH> | Select a named source object; attach =PATH when it begins with - (repeatable) |
--srcs-in <DIR> | Select a directory’s contents; attach =DIR when it begins with - (repeatable) |
--src-non-dir <PATH> | Select a named non-directory source object; attach =PATH when it begins with - (repeatable) |
--src-dir <DIR> | Select a named source directory; attach =DIR when it begins with - (repeatable) |
--src-non-dirs <PATH>... | Select several named non-directory source objects |
--src-dirs <DIR>... | Select several named source directories |
--srcs <PATH>... | Select several named source objects |
--ignore <PATTERN> | Skip paths matching a gitignore-style pattern (repeatable) |
--ignore-from <FILE> | Securely open and read gitignore-style patterns from raw-byte FILE (repeatable; stacks in command-line order) |
[PATH]... | Named source objects (shorthand for –src) |
Destination and mapping
| Argument / option | Meaning |
|---|---|
--as-fd <FD> | Write raw bytes to an inherited local descriptor (1 for stdout), instead of a destination path |
--to <ENDPOINT> | Destination SSH endpoint, @NAME, or s3://BUCKET; placement defaults to –into |
--into <DIR> | Put selected names inside DIR, creating it if necessary |
--into-new <DIR> | Put selected names inside DIR, which must not exist |
--into-existing <DIR> | Put selected names inside an existing directory |
--as <PATH> | Map one named source exactly to PATH; never follow its final entry |
--as-new <PATH> | Map one named source exactly to PATH; its final entry must not exist and is never followed |
--as-existing <PATH> | Map one named source exactly to PATH; its final entry must exist and is never followed |
--mapping <FILE> | Copy the entries of a local NDJSON mapping manifest (- reads stdin), acquired before destination changes, instead of selecting sources; entry src paths are relative to -C and dst paths are relative to the –into container |
Updates and deletion
| Argument / option | Meaning |
|---|---|
--only-new | Copy entries found missing; keep metadata of entries found present; adding children requires write access |
--inplace | Update destination files directly, using no full-sized staging file; interruption can leave them incomplete |
--prune | After copying, remove target-only objects in mapped directory scopes; ignored source paths remain protected |
--max-delete <N> | With –prune, refuse all removals if more than N are planned |
Metadata and symlinks
| Argument / option | Meaning |
|---|---|
--follow | Follow symlinks in all directly supplied filesystem paths |
--follow-src | Follow symlinks in directly supplied source paths |
--follow-dst | Follow symlinks in directly supplied destination paths |
--preserve <FEATURE> | Preserve times, permissions or ownership, or copy special files (repeatable/comma-separated) Possible values: - times: Preserve modification times (already the default for named destinations) - permissions: Preserve permission bits - ownership: Preserve owner and group IDs - specials: Copy device nodes and special files |
Verification
| Argument / option | Meaning |
|---|---|
--hash | Hash existing source and destination files instead of trusting size and modification time |
--integrity-checking <KEY=VALUE,...> | Payload checks |
Connections and remote execution
| Argument / option | Meaning |
|---|---|
--no-compress | Disable transport compression |
--receiver-max-entries <N> | Command-restricted receiver ceiling: refuse to touch more than N destination entries |
--receiver-max-bytes <SIZE> | Command-restricted receiver ceiling: refuse to write more than SIZE bytes of file data in total |
--receiver-receipt <DETAIL> | Command-restricted receiver receipt detail: final sizes (default) or also final BLAKE3 file hashes Possible values: - sizes: Final type and size of every path the transfer could have changed - hashes: Sizes plus a closure-time BLAKE3 hash of every regular file |
--auth-from <auto|ssh|@NAME> | Authorize through @NAME (also S3 copies), or use local SSH access (default: auto) |
--coordinate-at <COORDINATE_AT> | Choose the endpoint that runs the coordinator Possible values: - auto: Run locally unless both endpoints are remote, then run at the source - src: Run the coordinator at the source endpoint - dst: Run the coordinator at the destination endpoint - local: Keep the coordinator on the invoking machine and relay the data there [default: auto] |
--rsh <COMMAND> | Remote shell command (default: ssh); the command owns SSH and agent policy when set |
--syq-path <PATH> | Use this remote syq executable instead of installing a helper |
--no-bootstrap | Use syq on the remote PATH instead of installing a helper |
--tcp-plain | Use TCP data connections without encryption (trusted networks only) |
--no-tcp | Send file data through SSH rather than separate TCP data connections |
--tcp-ports <LO-HI> | Port range remote listeners use for TCP data connections [default: 47600-47699] |
--tcp-congestion <ALGO> | Use this congestion-control algorithm for TCP data sockets (Linux only) |
--detach | Run at the remote coordinator and return after launch; requires –peer-auth own-credentials or –rsh |
--peer-auth <MODE> | How the coordinator authenticates to the peer (see the values below); –rsh takes over this policy entirely Possible values: - restricted: Constrained agent broker plus the command-restricted receiver on the peer - broker: Constrained agent broker only; the peer runs no command-restricted receiver - own-credentials: Forward nothing; the coordinator must hold its own credentials for the peer - full-agent: Expose the complete local SSH agent to the coordinator, as ssh -A would[default: restricted] |
--pscope <PATH> | Use an ephemeral SSH persistence scope created by syq persist on --ephemeral |
S3 connection settings
| Argument / option | Meaning |
|---|---|
--s3-endpoint <URL> | S3 API endpoint URL (also AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL) |
--s3-region <REGION> | S3 signing region, used as given (otherwise syq asks AWS where the bucket is) |
--s3-profile <NAME> | AWS shared configuration/credentials profile |
--s3-header <NAME: VALUE> | Add a header before signing every S3 request (repeatable; S3-to-S3 metadata/tag overrides are refused) |
Performance and resource limits
| Argument / option | Meaning |
|---|---|
--performance-tuning <KEY=VALUE,...> | Workers, request sizes, and copy methods |
--resource-limits <KEY=VALUE,...> | Bandwidth and concurrency ceilings |
Preview, progress, and results
| Argument / option | Meaning |
|---|---|
--results <FILE> | Write the machine-readable NDJSON result stream to FILE (created fresh; an existing file is refused) |
--results-fd <FD> | Write the result stream to an inherited file descriptor the caller opened (e.g. --results-fd 3 3>run.ndjson); must be above 2 |
-n, --dry-run | Preview without changing copy/removal data; remote setup may still cache the helper or install syq; requested results files are still written |
-v, --verbose... | List files with -v; explain helpers and transport with -vv |
-q, --quiet | Suppress non-error messages |
--progress | Show progress even when stderr is not a terminal |
--no-progress | Never show the human progress display |
--stats | Print transfer statistics, worker waits, endpoint operations and CPU at the end |
Help and version
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
-V, --version | Print version |
--help-all | Show all options and details |
Update policies
--into-existing requires the destination directory to exist but allows new
files inside it. --only-new can add children to an existing directory, but
does not change that directory’s permissions to make it writable.
--only-new cannot combine with --inplace: an interrupted write could leave
a file that a retry skips. Restricted receivers also reject --as-new --inplace,
because direct writes do not enforce that destination condition. S3 and named
receiving destinations do not support --inplace; see
Copy limits.
These policies do not disable requested pruning. Descriptor-specific restrictions are listed under file descriptors.
Pruning
--prune deletes only within the copied directories, after copying succeeds.
Ignored paths, syq partial files, and recovery entries are
kept. This also keeps their parent directories; extra hard links to copied
files may be kept too. An interruption during deletion can leave some extras
already removed.
Keep the source outside the destination. Syq detects overlap locally and for remote paths using the same host name, user, and port. Different SSH aliases or shared storage can hide overlap from this check. Wait for other copies into the destination to finish before pruning, so their new files are not deleted as extras.
Filename and type conflicts
If the source has both Report.txt and report.txt, a case-insensitive
destination cannot store both. Syq does not check for that before copying,
and one file can replace the other. The same problem applies to distinct
Unicode spellings that the destination treats as one name. Rename the source
entries or use a destination that can distinguish them. Unsupported names
are reported as copy errors.
Syq refuses replacements between directories and non-directories, including empty directories. Move or remove the conflicting destination before retrying. Other replacements can fail if the filesystem lacks the operation needed to replace an entry safely; the old entry is kept.
Capacity checks
For missing or empty filesystem destinations, syq checks available bytes and capacity for new files when the filesystem reports them. A clear shortage fails before copying. Updates to populated destinations do not use this whole-copy estimate: existing data may be reused or replaced. Allocation errors still fail the affected copy.
Metadata details
Source setuid, setgid, and sticky bits are not copied without
--preserve=permissions. Ownership uses numeric IDs. On macOS, an existing
destination directory must be readable before syq can temporarily repair
missing write or search permission.
Modification times are preserved for named file destinations. Output
descriptors require explicit --preserve=times; see below.
File descriptors
--src-fd 0 reads stdin; --as-fd 1 writes stdout. Use them to connect local,
SSH, or S3 copies to other programs; see
Shell pipelines and file descriptors.
Bash process substitution also works with ordinary source syntax:
syq cp --src <(gzip -c data) --to server --as data.gz
Paths such as /dev/fd/63 refer to descriptors in the local syq process.
Each stream copy takes one source. Stdin and process substitution have no
filename, so use --as to choose one, or --as-fd to write to a descriptor.
A named pipe can use --into: syq cp incoming.fifo --into saved waits for a
writer, then saves its bytes as the regular file saved/incoming.fifo.
A symlink to a pipe requires --follow-src. Selecting a pipe alongside other
sources, including through a shell glob, is an error. Directory copies never
read pipes; --preserve=specials copies the pipe itself.
Completion and failures
A named destination is replaced only after the transfer succeeds. The -new
and -existing placement conditions apply as usual, before a named pipe is
opened. S3 new-object writes also refuse replacement if an object appears
during the upload.
If the producer fails halfway through, syq can still successfully save the
bytes it received: EOF does not tell it whether the producer succeeded.
Bash’s set -o pipefail detects failures in a pipeline but cannot undo a file
already saved. Process substitution needs a separate check of the producer’s
status. In Python, managed streams let you commit
only after your producer succeeds.
An output descriptor may contain incomplete data after a failure. Check syq’s
exit status before using the result. A consumer closing early makes syq fail.
Forced termination can leave a .syq-stream-* temporary file beside the
destination.
Options and file behavior
Progress, --stats, and -v go to stderr, leaving stdout for payload.
Statistics report bytes, elapsed time, and average rate; pipe lengths are
unknown until EOF. Use --resource-limits bandwidth=RATE to limit throughput.
--only-new skips a destination that exists. Existing directories, S3 key
prefixes, and dangling symlinks also count as existing for --only-new. Skips succeed without reading input or
opening a named FIFO. A shell producer can therefore receive SIGPIPE; in Python, check the writer’s
skipped property before producing bytes. An output FD already exists, so
--only-new --as-fd N always skips after validating the source.
Use --dry-run to check source and destination placement without reading input,
opening a named pipe, or changing the destination. It cannot check a payload
hash or predict the length of a pipe. --results and --results-fd report
stream_result records separately from payload;
results and payload/completion descriptors must differ.
Transport and limits
Filesystem streams use parallel data workers over SSH or encrypted TCP, with
automatic worker tuning as in regular-file copies. They accept workers,
request-size, pipeline-depth, and bw-pacing tuning; workers=N fixes the
worker count, and --no-tcp keeps data on SSH. S3 transfers one object using
multipart controls; see Descriptor copies.
Restart recovery, named receiving destinations, detached execution, directory selection, and content comparison are unsupported.
Descriptor offsets and metadata
Other inherited descriptors work too, except 2, which is reserved for
diagnostics. Dedicate each descriptor to the copy. Syq advances its offset,
respects append mode, and leaves its blocking mode alone;
it does not truncate it. A literal - is a filename.
When a regular file is copied to a named destination, syq preserves its modification time. New named files use the source permissions limited by the destination umask; existing files keep their permissions. S3 uploads store file attributes in object metadata.
Output descriptors use the timestamps from normal writes, including when
appending to an existing file. Add --preserve=times to copy the source
modification time instead; this changes the whole destination file’s timestamp
even for a partial write. --preserve=permissions,ownership copies those
attributes without changing timestamps. S3 downloads interpret object metadata
when attributes are requested; time preservation uses S3’s modification time if
no syq attributes are stored.
Input pipes, sockets, and devices have no payload metadata, so they reject
--preserve. Their new named destinations use 0666 limited by the umask
and the time of the write; existing files keep their permissions.
Output pipes likewise cannot preserve times, permissions, or ownership. Parent
directories are created as needed. The source --cwd / --root options
apply to pathname sources. --root with --src-fd is rejected because it cannot
confine a descriptor that is already open. Symlink handling is unchanged; see Symlinks. Named remote
sources must be regular files.
syq rm
Remove local files, remote filesystem entries, or S3 objects. See Remove files for selection rules and Versions and deletion for versioned objects.
syq rm --src-dir old-output --dry-run -v
For scripting, see environment variables and Removal records.
syq rm [OPTIONS] PATH...
syq rm [OPTIONS] --srcs-in DIR
SSH and transport
| Argument / option | Meaning |
|---|---|
--auth-from <@NAME> | Request storage authorization from a connected receiving machine |
--syq-path <PATH> | Use this exact syq executable on the remote removal endpoint |
--no-bootstrap | Use syq on the remote PATH instead of installing a helper |
--pscope <PATH> | Use an ephemeral SSH persistence scope created by syq persist on --ephemeral |
Copy policy and filtering
| Argument / option | Meaning |
|---|---|
--s3-endpoint <URL> | S3 API endpoint URL (also AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL) |
--s3-region <REGION> | S3 signing region, used as given (otherwise syq asks AWS where the bucket is) |
--s3-profile <NAME> | AWS shared configuration/credentials profile |
--s3-header <NAME: VALUE> | Add a header before signing every S3 request (repeatable; S3-to-S3 metadata/tag overrides are refused) |
--s3-all-versions | Permanently remove all selected S3 object versions and delete markers |
--s3-version-id <ID> | Permanently remove one version or delete marker of one exact S3 key |
Sources and selection
| Argument / option | Meaning |
|---|---|
--on <ENDPOINT> | Removal endpoint ([USER@]HOST[:PORT] or s3://BUCKET); omitted means local |
-C, --cwd <DIR> | Resolve relative selectors from DIR at the removal endpoint |
--root <DIR> | Confine resolution and removal beneath DIR |
--follow | Like –follow-src; also follow symlinks in the –results path |
--follow-src | Follow symlinks in –cwd, –root, and selector parent directories; always unlink a final selected symlink |
--src <PATH> | Select a non-directory object; attach =PATH when it begins with - (repeatable) |
--srcs-in <DIR> | Recursively select a directory’s contents, keeping the directory; attach =DIR when it begins with - (repeatable) |
--src-non-dir <PATH> | Select a non-directory object; attach =PATH when it begins with - (repeatable) |
--src-dir <DIR> | Select a directory tree; attach =DIR when it begins with - (repeatable) |
--src-non-dirs <PATH>... | Select several non-directory objects |
--src-dirs <DIR>... | Select several directory trees |
--srcs <PATH>... | Select several non-directory objects |
[PATH]... | Selected objects (shorthand for –src) |
Preview and output
| Argument / option | Meaning |
|---|---|
-n, --dry-run | Preview without changing copy/removal data; remote setup may still cache the helper or install syq; requested results files are still written |
-v, --verbose... | List removed paths |
-q, --quiet | Suppress non-error messages |
Performance tuning
| Argument / option | Meaning |
|---|---|
--performance-tuning <KEY=VALUE,...> | Filesystem removal workers: workers=N |
Progress and results
| Argument / option | Meaning |
|---|---|
--progress | Show progress even when stderr is not a terminal |
--no-progress | Never show the human progress display |
--results <FILE> | Write the machine-readable NDJSON result stream to FILE (created fresh; an existing file is refused) |
--results-fd <FD> | Write the result stream to an inherited file descriptor the caller opened (e.g. --results-fd 3 3>run.ndjson); must be above 2 |
Help and version
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
-V, --version | Print version |
--help-all | Show all options and details |
Selection rules
Named paths, --src, and --src-non-dir require non-directories. Use
--src-dir to require a directory and remove it recursively, or --srcs-in
to remove its contents while keeping the directory. These directory selectors
reject a final symlink even when following is enabled. All selections are
checked before deletion begins. Filters are not supported.
syq clean-partials
Remove syq partial files below local or SSH directories. Wait for active copies into those directories to finish before running cleanup:
syq clean-partials --dry-run -v backup
syq clean-partials backup
Add --on server to clean a remote tree. For results output, see
Removal records.
Which files are removed
clean-partials removes regular files named .FILENAME.syq-tmp.RANDOM, with
16 random characters at the end. The filename portion may be shortened or
omitted. It does not follow symlinks or remove old partial-name formats.
A regular file deliberately named like a partial is also selected, so preview
before deleting.
Interrupted replacements and macOS clones can leave .syq-swap-... entries
containing displaced originals or temporary clone data. Neither this command
nor pruning removes them. Stop copies using the destination, inspect these
entries, and recover anything you need before removing them manually.
syq clean-partials [OPTIONS] <TREE>...
Sources and selection
| Argument / option | Meaning |
|---|---|
--on <ENDPOINT> | Removal endpoint ([USER@]HOST[:PORT]); omitted means local |
-C, --cwd <DIR> | Resolve relative trees from DIR at the removal endpoint |
--root <DIR> | Confine traversal beneath DIR |
<TREE>... | Directory trees to search |
Preview and output
| Argument / option | Meaning |
|---|---|
-n, --dry-run | Preview without changing copy/removal data; remote setup may still cache the helper or install syq; requested results files are still written |
-v, --verbose... | List removed paths |
-q, --quiet | Suppress non-error messages |
Performance tuning
| Argument / option | Meaning |
|---|---|
--performance-tuning <KEY=VALUE,...> | Filesystem removal workers: workers=N |
Progress and results
| Argument / option | Meaning |
|---|---|
--progress | Show progress even when stderr is not a terminal |
--no-progress | Never show the human progress display |
--results <FILE> | Write the machine-readable NDJSON result stream to FILE (created fresh; an existing file is refused) |
--results-fd <FD> | Write the result stream to an inherited file descriptor the caller opened (e.g. --results-fd 3 3>run.ndjson); must be above 2 |
SSH and transport
| Argument / option | Meaning |
|---|---|
--syq-path <PATH> | Use this exact syq executable on the remote removal endpoint |
--no-bootstrap | Use syq on the remote PATH instead of installing a helper |
Help and version
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
-V, --version | Print version |
--help-all | Show all options and details |
syq map
Print local source/destination pairs as NDJSON for syq cp --mapping:
syq map --srcs-in photos > photos.ndjson
syq cp --mapping photos.ndjson -C photos --into archive
Named selectors must be relative to the source base; use -C DIR or --root DIR
to choose that base. --srcs-in must be the only selector when used. --as
requires one named object and cannot combine with --srcs-in.
See Rename and reorganize during a copy for worked examples.
syq map [OPTIONS] PATH...
syq map [OPTIONS] --srcs-in DIR
Sources and selection
| Argument / option | Meaning |
|---|---|
-C, --cwd <DIR> | Resolve relative source selectors from DIR |
--root <DIR> | Resolve source selectors beneath DIR and refuse any escape |
--follow | Follow symlinks in all directly supplied filesystem paths |
--follow-src | Follow symlinks in directly supplied source paths |
--src <PATH> | Select a named source object; attach =PATH when it begins with - (repeatable) |
--srcs-in <DIR> | Select a directory’s contents; attach =DIR when it begins with - (repeatable) |
--src-non-dir <PATH> | Select a named non-directory source object; attach =PATH when it begins with - (repeatable) |
--src-dir <DIR> | Select a named source directory; attach =DIR when it begins with - (repeatable) |
--src-non-dirs <PATH>... | Select several named non-directory source objects |
--src-dirs <DIR>... | Select several named source directories |
--srcs <PATH>... | Select several named source objects |
[PATH]... | Named source objects (shorthand for –src) |
Destination placement
| Argument / option | Meaning |
|---|---|
--as <PATH> | Emit the single selected root at PATH, relative to the future destination container; PATH may be nested |
Help and version
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
-V, --version | Print version |
--help-all | Show all options and details |
Mapping format
cp --mapping reads one JSON object per line. map emits this format from local
sources; other programs can generate it too.
| Field | Meaning |
|---|---|
src | Required path relative to the copy’s source base (-C or --root) |
dst | Required path relative to the destination container (--into) |
kind | Optional file, dir, symlink, or special precondition |
size, mtime | Optional information for transforms; ignored during execution |
metadata | Optional destination attributes; see below |
expected_hash | Optional whole-file expectation: {"algorithm":"md5","value":"900150983cd24fb0d6963f7d28e17f72"} |
Paths use encoding: "utf-8", or "base64" with standard base64 of raw
filename bytes. Absolute or empty paths, and any . or .. component, are
refused. Unknown fields are refused too.
expected_hash checks a regular file’s complete contents, including reused
bytes. A mismatch fails the entry. See expected hashes
for algorithms and behavior with in-place writes.
To set destination attributes without changing the source, add a metadata
object, for example "metadata": {"mode": 416, "mtime": 1700000000}. This sets
permissions to 0640 and the modification time to the given Unix second.
Optional fields are mode (permission bits, 0–4095), numeric uid and gid,
mtime (Unix seconds), and mtime_nsec (0–999999999, requires mtime).
A supplied mtime defaults to zero fractional seconds. Omitted attributes
follow normal copy behavior; the top-level mtime remains informational.
Explicit attributes apply without --preserve. Ownership requests fail if the
filesystem refuses them; symlinks cannot have a requested mode.
S3 uploads store the attributes in syq object metadata; downloads apply them
to the filesystem. S3-to-S3 copies keep other object metadata and stay server-side.
Restricted receivers also require matching --preserve permissions in the signed
grant. Selection rules such as --only-new still take precedence. Supplied
timestamps disable the size/time shortcut for those entries. Use --hash to
compare contents when repeating a copy with a fixed destination timestamp.
Older binaries that do not support metadata reject the manifest.
Each entry copies one object. A directory entry is not recursive.
syq map emits its descendants as separate entries. A missing source or
wrong kind fails that entry while independent entries continue.
Mapping restrictions
Use --mapping instead of source selectors, with --into, --into-new, or
--into-existing. It cannot combine with --as, --prune, or --detach.
Use the same source base for map and cp. --root confines source selection.
Follow options apply to supplied paths, never to links traversed by a manifest
entry. map refuses non-UTF-8 names; hand-written manifests may use base64.
The whole manifest is validated before copying. Malformed input, duplicate destination names, and declared file/ancestor conflicts refuse the run. The destination container may already have been created. Memory use grows with manifest size. Conflicts discovered while copying fail the affected entries.
For restricted copies between servers, mapped destinations and their parent directories count against the receiver’s entry limit. Each line can be up to 1 MiB, and each destination path up to 4096 bytes. There is no separate total manifest-size limit. Missing parent directories are created; a file or symlink blocking a parent must be moved or removed before retrying.
kind: "special" checks the source type; add --preserve=specials to copy those
entries. Use --mapping - for a pipeline; a named FIFO manifest requires Linux
with procfs.
Filter a mapping
These jq filters keep or remove entries before cp --mapping reads them:
# Keep files of at least 1 MiB, plus directory and link entries:
jq -c 'select(.kind != "file" or .size >= 1048576)'
# Drop device, FIFO, and socket entries:
jq -c 'select(.kind != "special")'
syq exec
Run a command on a connected receiving machine, with approval on that machine:
syq exec --on @laptop --cwd work/project -- cargo test
See Run commands on your laptop for setup and examples.
syq exec [OPTIONS] --on <@NAME> -- <PROGRAM>...
Arguments
| Argument / option | Meaning |
|---|---|
<PROGRAM>... | Program and literal arguments; use sh -c explicitly for shell syntax |
Options
| Argument / option | Meaning |
|---|---|
--on <@NAME> | Receiving name; requires a live return connection (never falls back to SSH) |
-C, --cwd <DIR> | Working directory on that machine, relative to its receiving directory [default: .] |
Help and version
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
--help-all | Show all options and details |
Approve each command locally
Every command requires approval on the receiving machine, even if copies are approved automatically. Review the program, arguments, requesting server, and working directory. If the desktop prompt is missing or truncated, inspect and answer the request from a local terminal:
syq persist receive pending
syq persist receive approve REQUEST_ID
syq persist receive deny REQUEST_ID
Requests expire after five minutes. Commands run with your local user’s
permissions; the receiving --root and copy limits do not contain them.
See Commands on your laptop for the
trust boundary.
Arguments and working directories
Put -- before the program. Everything after it is passed as a literal
argument, including strings starting with --. There is no implicit shell
expansion on the receiving machine. To use shell syntax, request a shell:
syq exec --on @laptop --cwd work/project -- sh -c 'cargo build && ./target/debug/demo'
--cwd DIR (or -C DIR) is relative to the profile’s effective starting directory
(shown by persist receive status). Its default is that directory. Absolute paths and
.. can select elsewhere. The directory must exist. Syq does not expand ~
on the receiving machine; use a relative path or an absolute path instead.
The command inherits the receiving service’s local environment, including
PATH and its desktop session. It does not inherit the server’s environment.
Restart receiving from a terminal in the desired desktop session when those
values change. Stdin is closed and there is no interactive terminal.
Output, completion and cancellation
Stdout and stderr stream back to your terminal, and syq returns the command’s
exit code, or 128 + signal if it was killed by a signal. Setup and connection
failures return nonzero; losing the exit status is an error. Commands are not
retried automatically, and completed file changes are not rolled back.
Interrupting the request, stopping or changing receiving, or losing the
connection forcibly stops the command’s process group. Cleanup handlers do not
run. Remaining children in that group are also stopped when the foreground
program exits. Detached processes and applications launched through macOS
open can survive this cleanup. Commands do not emit copy receipts or copy
automation records.
Execution details
Requests allow up to 256 arguments (16 KiB total) and a 4096-byte working-directory path. Each server connection permits one pending approval and eight active commands.
syq rsync
Copy with rsync-style arguments:
syq rsync -av project/ server:backup/project/
Check Rsync compatibility before replacing a script.
Trailing slashes follow rsync’s rules. Here -h means human-readable sizes;
use --help for help. SYQ_RSYNC_OPTIONS supplies
extra arguments.
syq rsync [OPTIONS] SRC... DEST
syq rsync [OPTIONS] [USER@]HOST:SRC... DEST
syq rsync [OPTIONS] SRC... [USER@]HOST:DEST
Copy policy and filtering
| Argument / option | Meaning |
|---|---|
-a, --archive | Archive mode; same as -rlptgoD |
-r, --recursive | Recurse into directories |
-l, --links | Copy symlinks as symlinks |
--insecure-links | Follow symlinks in this machine’s rsync operator paths regardless of ownership (local only, as in rsync) |
-p, --perms | Preserve permissions |
-t, --times | Preserve modification times |
-g, --group | Preserve group |
-o, --owner | Preserve owner (root only) |
-D | Preserve device and special files |
-h, --human-readable | No-op accepted for rsync compatibility (sizes are always human-readable) |
--numeric-ids | No-op accepted for rsync compatibility (syq always uses numeric uid/gid) |
--bwlimit <RATE> | Limit the aggregate file-data rate across all workers (default unit: KiB/s; 0 disables) |
-P | Same as –progress –partial |
--partial | No-op accepted for rsync compatibility (syq always keeps partial files) |
-c, --checksum | Skip quick check; compare file contents block by block and repair differences |
--inplace | Update files in place instead of writing a partial and renaming. Use this to modify a large existing file without copying it first (saves time and disk space when only part of it changes). Cannot be combined with -u or –ignore-existing: an interrupted in-place write leaves a newer-looking final file those filters would then skip forever |
--syq-ignore <PATTERN> | Syq extension: skip paths matching PATTERN (gitignore syntax: foo matches at any depth, /foo only at the source root, foo/ only directories, !pat re-includes). Repeatable; together with –syq-ignore-from the patterns act like the lines of one .gitignore file, in command-line order, anchored at each source root. Skipping a directory skips its whole subtree, so to copy only .jpg use: –syq-ignore ‘’ –syq-ignore ‘!/’ –syq-ignore ‘!.jpg’ |
--syq-ignore-from <FILE> | Syq extension: securely open and read ignore patterns from raw-byte FILE (one per line, # comments); repeatable |
--delete | Delete extraneous files from the destination directories (paths the source does not have). Deletion happens after the transfer and is skipped entirely if the source scan reported any error. Ignored paths (–syq-ignore) are protected on both sides. rsync’s –delete-after and –delete-delay mean the same thing and are accepted. Cannot be combined with –files-from |
--delete-excluded | With –delete, also remove destination paths that the –syq-ignore patterns exclude |
--max-delete <N> | With –delete, refuse all deletions if more than N are planned (exit 25). Unlike positive rsync limits, this is atomic; 0 and -1 both prohibit deletion |
-u, --update | Skip regular files that are newer on the destination (directories, symlinks and specials are unaffected) |
--ignore-existing | Skip updating files that already exist on the destination |
--existing | Never create anything that doesn’t exist yet on the destination — files, symlinks, specials, directories, or the destination root itself; existing files are still updated |
--max-size <SIZE> | Don’t transfer regular files larger than SIZE (e.g. 100M). With –delete the destination copy of such a file is left alone |
--min-size <SIZE> | Don’t transfer regular files smaller than SIZE |
--files-from <FILE> | Copy only the paths listed in raw-byte FILE, securely opened before transfer (one per line, relative to the single source directory; - reads stdin). Listed directories are copied without their contents unless -r is given explicitly; missing parent directories are created |
--from0 | –files-from entries are NUL-separated instead of one per line |
Preview and output
| Argument / option | Meaning |
|---|---|
-v, --verbose... | List files with -v; explain helpers and transport with -vv |
-q, --quiet | Suppress non-error messages |
-n, --dry-run | Resolve mappings and transport, then estimate transfers, exclusions, and deletions; leave source and destination data unchanged (remote setup may still cache the helper or install syq) |
SSH and transport
| Argument / option | Meaning |
|---|---|
-z, --compress | Compress remote data in transit automatically (default) |
--no-compress | Disable transport compression |
-e, --rsh <COMMAND> | Remote shell command (default: ssh); controls agent forwarding when set |
--rsync-path <PATH> | Use this exact syq executable on the remote instead of the managed helper |
--syq-no-bootstrap | Syq extension: require syq on the remote PATH instead of installing a versioned helper |
--syq-tcp-plain | Syq extension: use TCP data connections without encryption (trusted networks only) |
--syq-no-tcp | Syq extension: send all data over ssh instead of separate TCP data connections |
--syq-tcp-ports <LO-HI> | Syq extension: port range the remote listens on for TCP data connections [default: 47600-47699] |
--syq-tcp-congestion <ALGO> | Syq extension: use this congestion-control algorithm for TCP data sockets (Linux only) |
--syq-pscope <PATH> | Syq extension: use an isolated SSH persistence scope created by syq persist on --ephemeral |
Performance tuning
| Argument / option | Meaning |
|---|---|
-B, --block-size <SIZE> | Comparison and reuse block size (64K through 64M) [default: 4M] |
--performance-tuning <KEY=VALUE,...> | Workers, request sizes, and copy methods |
Resource limits
| Argument / option | Meaning |
|---|---|
--resource-limits <KEY=VALUE,...> | Bandwidth and concurrency ceilings |
Integrity checking
| Argument / option | Meaning |
|---|---|
--integrity-checking <KEY=VALUE,...> | Payload checks |
Progress and results
| Argument / option | Meaning |
|---|---|
--progress | Show progress (default when stderr is a terminal) |
--no-progress | Never show progress |
--stats | Print transfer statistics, worker waits, endpoint operations and CPU at the end |
Sources and selection
| Argument / option | Meaning |
|---|---|
[PATH]... | Source(s) and destination |
Help and version
| Argument / option | Meaning |
|---|---|
-V, --version | Print version |
--help-all | Show all options and details |
--help | Show common usage and options |
syq persist
Manage reusable SSH connections and receiving profiles. Start with Use your laptop from a server for setup, or see Persistence details for profiles and limits.
syq persist <COMMAND>
| Command | Purpose |
|---|---|
persist receive | Configure receiving and decide incoming copy or command requests |
persist destinations | Inspect or recover named return destinations |
persist connect | Connect to an SSH server and wait until enabled receiving is ready |
persist on | Enable persistent connections for later syq commands |
persist off | Disable persistence and close its live SSH control connections |
persist status | Show connection readiness and any receiving problem |
Help (also available on subcommands)
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
--help-all | Show all options and details |
syq persist receive
Configure receiving and approve its requests. See the three uses of receiving for file copies to your machine, commands on it, and authorizing copies between servers.
syq persist receive <COMMAND>
| Command | Purpose |
|---|---|
persist receive pending | Show incoming copy and command requests awaiting approval on this machine |
persist receive approve | Allow one pending request using the ID from persist receive pending |
persist receive deny | Deny one pending request using the ID from persist receive pending |
persist receive on | Enable or configure a receiving profile (without –name, use the first profile) |
persist receive off | Disable receiving and stop its background connections; keep ordinary persistence |
persist receive remove | Remove a saved receiving profile and stop its connections |
persist receive status | Show receiving settings and background connection state |
persist receive wait | Wait for a connection with a deadline |
syq persist receive pending
--json prints structured requests. --wait waits for a request, up to --timeout seconds (default 30). Use the returned request ID with approve or deny.
syq persist receive pending [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--json | See the command description above. |
--wait | Wait for an incoming request, with a deadline |
--timeout <TIMEOUT> | [default: 30] |
syq persist receive approve
ID is a pending request ID from persist receive pending on this machine. Approval applies to that request only.
syq persist receive approve <ID>
Arguments
| Argument / option | Meaning |
|---|---|
<ID> | See the command description above. |
syq persist receive deny
ID is a pending request ID from persist receive pending on this machine.
syq persist receive deny <ID>
Arguments
| Argument / option | Meaning |
|---|---|
<ID> | See the command description above. |
syq persist receive on
Omitted settings keep saved values. New profiles ask for approval with desktop
prompts and use the home directory. Set --root to confine copies to a directory.
Changing settings cancels the profile’s active requests. See
profile settings and
copy limits.
syq persist receive on [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--auto-approve-root <AUTO_APPROVE_ROOT> | Automatically approve downloads confined to this directory |
--no-auto-approve-root | Require approval for every download again |
--server <SERVERS> | Limit this profile to these SSH destinations (repeat to allow several) |
--all-servers | Make this profile available through every connected server |
--notify <NOTIFICATIONS> | Show desktop prompts, or use only local pending/approve/deny commands [possible values: desktop, off] |
--name <NAME> | Create or update this named profile; omitted means the first profile |
-C, --cwd <CWD> | Default destination directory; absolute paths and .. may select elsewhere |
--auto-cwd | Choose cwd from root, auto-approve-root, then HOME |
--root <ROOT> | Confinement boundary for every download, even with approval |
--no-root | Remove the hard download boundary |
--max-bytes <MAX_BYTES> | Maximum bytes one transfer may reserve/write (default: 100G) |
--max-entries <MAX_ENTRIES> | Maximum entries one transfer may touch (default: 1000000) |
--max-delete <MAX_DELETE> | Permit pruning up to N entries per transfer (default: 0) |
syq persist receive off
Disable the named profile, or all profiles when --name is omitted. This stops incoming requests while leaving forward SSH persistence available.
syq persist receive off [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--name <NAME> | Stop only this profile; without –name, stop all profiles |
syq persist receive remove
NAME identifies the saved profile to remove. Its connections and requests stop. The last profile can be disabled but cannot be removed; server-side name assignments remain until explicitly forgotten.
syq persist receive remove <NAME>
Arguments
| Argument / option | Meaning |
|---|---|
<NAME> | See the command description above. |
syq persist receive status
Show all profiles, or select one with --name. Use --json for structured output.
syq persist receive status [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--json | See the command description above. |
--name <NAME> | See the command description above. |
syq persist receive wait
HOST is the SSH server endpoint. Wait for the selected profile, or every enabled profile when --name is omitted. --timeout is in seconds; failure or timeout exits nonzero. This command does not start a connection.
syq persist receive wait [OPTIONS] <HOST>
Arguments
| Argument / option | Meaning |
|---|---|
<HOST> | See the command description above. |
Options
| Argument / option | Meaning |
|---|---|
--name <NAME> | Wait for this profile; otherwise wait for every enabled profile |
--timeout <TIMEOUT> | [default: 30] |
syq persist destinations
syq persist destinations <COMMAND>
| Command | Purpose |
|---|---|
persist destinations list | Print registrations and whether their receiving laptop responds |
persist destinations forget | Remove an offline destination name so another laptop can register it |
persist destinations wait | Wait for a connection with a deadline |
syq persist destinations list
syq persist destinations list
syq persist destinations forget
NAME is an offline receiving name to release before replacing its machine. Forgetting a live connection is refused.
syq persist destinations forget <NAME>
Arguments
| Argument / option | Meaning |
|---|---|
<NAME> | See the command description above. |
syq persist destinations wait
NAME is a receiving name without @. Wait up to --timeout seconds for it to respond; timeout exits nonzero.
syq persist destinations wait [OPTIONS] <NAME>
Arguments
| Argument / option | Meaning |
|---|---|
<NAME> | See the command description above. |
Options
| Argument / option | Meaning |
|---|---|
--timeout <TIMEOUT> | [default: 30] |
syq persist connect
Connect to HOST and wait for receiving to be ready. --timeout applies to
the receiving wait after SSH and helper setup. --syq-path and --no-bootstrap
cannot combine. Use --pscope for an existing ephemeral scope.
syq persist connect [OPTIONS] <HOST>
Arguments
| Argument / option | Meaning |
|---|---|
<HOST> | SSH endpoint ([USER@]HOST[:PORT]); receiving names are not accepted |
Options
| Argument / option | Meaning |
|---|---|
--syq-path <PATH> | Use this remote syq executable instead of installing a matching helper |
--no-bootstrap | Use syq on the remote PATH instead of installing a matching helper |
--timeout <TIMEOUT> | Wait this many seconds for receiving after SSH/helper setup [default: 30] |
--pscope <PATH> | Reuse forward SSH in an existing ephemeral scope, without enabling receiving |
syq persist on
syq persist on [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--ephemeral | Create an ephemeral scope and print its path instead of changing the user setting |
syq persist off
syq persist off [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--pscope <PATH> | Operate on this ephemeral persistence scope instead of the user setting |
syq persist status
Inspect connections without starting them. --json uses the connection-status contract. --pscope selects an ephemeral scope.
syq persist status [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--json | Print structured connection state |
--pscope <PATH> | Inspect this ephemeral persistence scope instead of the user setting |
syq receiver
Manage restricted destinations for direct server-to-server copies. See access management and enrollment details for setup and upgrades.
syq receiver <COMMAND>
| Command | Purpose |
|---|---|
receiver enroll | Manually enroll or refresh a receiver |
receiver list | List local active and pending enrollments |
receiver revoke | Stop active receivers and remove their enrollment from both machines |
Help (also available on subcommands)
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
--help-all | Show all options and details |
syq receiver enroll
The destination’s parent must exist. Repeating enrollment updates the receiver; stop active copies before updating it.
syq receiver enroll [OPTIONS] <[USER@]HOST:DESTINATION>
Arguments
| Argument / option | Meaning |
|---|---|
<[USER@]HOST:DESTINATION> | Remote destination, e.g. alice@nas:/backup/photos |
Options
| Argument / option | Meaning |
|---|---|
--via <ENDPOINT> | Retry through this SSH jump host if the direct management connection fails |
syq receiver list
syq receiver list
syq receiver revoke
syq receiver revoke [OPTIONS] <ENROLLMENT-ID>
Arguments
| Argument / option | Meaning |
|---|---|
<ENROLLMENT-ID> | Enrollment ID printed by syq receiver list |
Options
| Argument / option | Meaning |
|---|---|
--via <ENDPOINT> | Retry through this SSH jump host if the direct management connection fails |
syq completion
Generate shell completion and manage cached endpoint suggestions. See shell setup to enable completion.
syq completion <COMMAND>
| Command | Purpose |
|---|---|
completion bash | Print the Bash completion adapter |
completion zsh | Print the Zsh completion adapter |
completion fish | Print the fish completion adapter |
completion cache | Inspect or clear cached endpoint suggestions |
Help (also available on subcommands)
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
--help-all | Show all options and details |
syq completion bash
syq completion bash
syq completion zsh
syq completion zsh
syq completion fish
syq completion fish
syq completion cache
syq completion cache <COMMAND>
| Command | Purpose |
|---|---|
completion cache list | List learned endpoint suggestions, most recently used first |
completion cache forget | Forget one exact native endpoint spelling |
completion cache clear | Remove all learned endpoint suggestions |
syq completion cache list
syq completion cache list
syq completion cache forget
syq completion cache forget <ENDPOINT>
Arguments
| Argument / option | Meaning |
|---|---|
<ENDPOINT> | Endpoint to forget: [USER@]HOST[:PORT], e.g. alice@nas:2222 |
syq completion cache clear
syq completion cache clear
syq tuning-cache
Inspect the measurements and decisions recorded by filesystem transfers. See tuning history for privacy, retention, and startup behavior.
syq tuning-cache <COMMAND>
| Command | Purpose |
|---|---|
tuning-cache list | List recent transfers |
tuning-cache show | Show a transfer’s measurements and decisions |
tuning-cache export | Export history as NDJSON; omit ID for every transfer |
tuning-cache clear | Delete recorded history and filesystem startup hints |
Help (also available on subcommands)
| Argument / option | Meaning |
|---|---|
-h, --help | Show common usage and options |
--help-all | Show all options and details |
syq tuning-cache list
List recent transfers, newest first. The worker column shows the count saved
as a startup hint; ? means the transfer did not supply a recommendation.
syq tuning-cache list [OPTIONS]
Options
| Argument / option | Meaning |
|---|---|
--limit <limit> | [default: 50] |
syq tuning-cache show
Print one transfer’s context and timestamped events. With --html, redirect
stdout to a file and open it in a browser for an interactive timeline.
syq tuning-cache show [OPTIONS] <id>
Arguments
| Argument / option | Meaning |
|---|---|
<id> | See the command description above. |
Options
| Argument / option | Meaning |
|---|---|
--html | Write a standalone interactive timeline to stdout |
syq tuning-cache export
Export one transfer, or all retained transfers when the ID is omitted, as NDJSON. Each transfer record is followed by its event records.
syq tuning-cache export [id]
Arguments
| Argument / option | Meaning |
|---|---|
[id] | See the command description above. |
syq tuning-cache clear
Delete the local history and its filesystem startup hints. The older connection-count cache remains unchanged.
syq tuning-cache clear
Performance tuning
--performance-tuning overrides syq’s automatic choices. To keep automatic
choices within a ceiling, use resource limits instead.
Leave performance tuning unset for everyday copies. These experimental controls
are available in syq cp and syq rsync; syq rm and syq clean-partials accept
only workers for filesystem removal.
Performance-tuning keys, accepted values, and behavior may change or be removed between releases without deprecation. Pin the syq version when a script depends on these overrides.
Transfer controls
syq cp and syq rsync accept --performance-tuning. Supply
comma-separated KEY=VALUE pairs:
syq cp large-file --to server --as /scratch/benchmark-copy \
--performance-tuning workers=1 -v \
--performance-tuning copy-path=ranges,request-size=1M,pipeline-depth=8
| Key | Default | Accepted values |
|---|---|---|
workers | Automatic | 1 through 65536 filesystem workers; route-specific receiver limits also apply |
comparison-block-size | 4 MiB | 64 KiB through 64 MiB; filesystem copies only |
request-size | Hash block size (normally 4 MiB) for ordinary requests; at most 2 MiB for streaming | 512 bytes through 64 MiB |
pipeline-depth | 4 | 1 through 64 outstanding range requests per endpoint per worker |
copy-path | auto | auto, ranges, or experimental streaming / auto-streaming |
batch-files | Up to 2048, sharing queued files across active workers | 1 through 4096 files per worker batch |
batch-bytes | 16 MiB | 512 bytes through 64 MiB per worker batch, including the first file |
split-min-size | 32 MiB, at least two hash blocks | 1 byte through 1 GiB, raised to at least two hash blocks |
bw-pacing | 125ms when capped | average, or an integer interval from 1ms through 10s; requires a nonzero --resource-limits bandwidth=RATE |
Sizes accept K, M, and G, using powers of 1024. Unknown keys, repeated
keys, and out-of-range values fail the command. Overrides apply to the remote
coordinator too and are not saved.
S3 copies
Use these keys for syq cp with S3 endpoints. workers applies only to
filesystem copies; use s3-requests for the shared S3 data-request count.
| Key | Default | Accepted values / meaning |
|---|---|---|
s3-requests | Automatic | 1–65536 simultaneous data requests across objects; excludes metadata requests and idle sockets |
s3-objects | Automatic | 1–65536 objects in progress, including preparation and finalization |
s3-parts-per-object | Automatic | 1–1024 simultaneous parts or ranges per object |
s3-part-size | Automatic | 5 MiB–5 GiB per upload part or download range |
s3-retries | 10 | 0–100 retries for transient failures and throttling; 0 disables retries |
The concurrency limits are nested. For example:
syq cp data --to s3://backups --into archive \
--performance-tuning s3-objects=4,s3-parts-per-object=8,s3-requests=16
This allows four objects in progress and up to eight parts per object, with at
most sixteen simultaneous data requests across them. These performance-tuning
values fix the available slots and disable automatic adjustment of each
specified count; unused slots can remain idle. To let syq
choose counts within these ceilings instead, pass the same keys through
--resource-limits. A count cannot be specified in both groups.
Part size grows when needed to stay within 10,000 upload parts. For server-side copies, an explicit part size also selects the multipart threshold, capped at 5 GiB. Without an explicit part limit, server-side copies can use the shared request budget’s full tuning range.
For S3 streams, see Descriptor copies for buffering and upload-size limits. S3 tuning is not saved between runs.
Large unfiltered prefix copies, current-object removals, and destination scans
for pruning can list subtrees concurrently. Discovery may use extra LIST
requests; selectors still name literal keys and prefixes. For copies,
--performance-tuning s3-parts-per-object=N also caps unfiltered
discovery; setting N=1 keeps flat pagination. If a policy denies discovery, these operations retry with flat
pagination at the original prefix.
Remembered connection counts
Syq remembers useful starting worker counts from successful filesystem copies and continues adjusting as the next copy runs. History is matched to the route, filesystems, transport, and copy settings when that information is available. A remembered count is a starting point, not a fixed limit or a promise of the best speed for a different workload.
On macOS and Linux, remote-copy hints also distinguish local networks using available default-router hardware addresses, without requesting Wi-Fi location permission. These are hints about the network around the machine running syq; they cannot detect changes upstream of a phone hotspot. Linux currently reads IPv4 routers; macOS also reads IPv6 routers. If the network cannot be identified, syq uses its existing route and filesystem matches. Hints without network context remain available for that fallback; a known network starts its own history. Local-copy hints are unchanged. A change of transport or observed network context during a copy prevents saving a new hint for its initial path.
--performance-tuning bypasses remembered counts and does not save a new
recommendation. --resource-limits workers=N caps the starting count while
leaving recommendations unchanged.
The older cache remains at ~/.cache/syq/tuning.json, in its existing format.
SYQ_TUNING_CACHE names another file; an empty value disables both this cache
and the history below. XDG_CACHE_HOME changes their parent directory.
Syq can keep idle connections ready for later tuning changes. These connections and their helper processes still use resources, so the active worker count is not a count of open connections.
Inspect tuning history
Filesystem copies record a local timeline by default, including short and failed copies. The history contains opaque endpoint and filesystem identifiers, the UTC date, copy settings, numerical progress, worker readiness, and tuning decisions. It does not contain filenames, command lines, file contents, or Wi-Fi names, and nothing is uploaded. Timing and sizes still reveal activity; this is performance history, not anonymous data or a complete audit trail.
TCP preflight events record opaque candidate-address identifiers, reported link
speeds (null when unknown), reachability (null when the probe had not finished
at selection time), and selection. Selected addresses are eligible for data
connections; they may not all carry data. Reported speeds are interface hints,
not measured throughput. These observations do not change startup-hint reuse.
syq tuning-cache list
syq tuning-cache show 42
syq tuning-cache show 42 --html > tuning-42.html
syq tuning-cache export 42 > tuning-42.ndjson
syq tuning-cache clear
Open the HTML file in a browser to plot worker counts and byte progress, then
select a decision to inspect its evidence. It is a standalone file with no
external scripts. export without an ID exports all retained transfers as
NDJSON. These diagnostic records are versioned, but their event details may
evolve; they are separate from the automation results contract.
The timeline distinguishes requested workers from workers ready to copy, and shows which measurements informed each decision. Copies completed by a single control request record that copy path without a tuning timeline.
Descriptor and pipe copies also record their tuning decisions, but do not use filesystem history to choose their starting count. S3 copies do not currently record these timelines. For remote-coordinated copies, history belongs to the machine running the coordinator; run the inspection commands there.
The default file is ~/.cache/syq/tuning.history-v1.sqlite. When
SYQ_TUNING_CACHE selects another file, the history uses that name with its
extension replaced by .history-v1.sqlite. SYQ_TUNING_HISTORY selects an
independent history file; an empty value disables history and its startup hints
while leaving the older cache available. New database files are private to the
user. SQLite may create adjacent -wal and -shm files while in use.
SYQ_TUNING_HISTORY_SIZE sets how much history to keep, default 128M, minimum
16M. History may be removed when this size target is exceeded.
Recording is best effort: an interrupted transfer or a storage error can leave gaps in the history. Copies still proceed when history cannot be saved. Clearing history also removes its startup hints; it leaves the older connection-count cache intact.
Filesystem tuning examples
Use disposable destinations when comparing settings. Larger requests and deeper
pipelines can increase memory use. copy-path=ranges disables small-file batches
and whole-file shortcuts, including local kernel copying and APFS cloning.
Scattered edits in existing files
Smaller comparison blocks can reduce the data sent for scattered edits, at the cost of more hashes and requests:
syq cp --srcs-in source --to host --into destination \
--performance-tuning comparison-block-size=64K,request-size=4M
Set request-size too: it otherwise follows the comparison block size.
At 64 KiB, files must be smaller than 130 GiB or comparison fails. Increase
comparison-block-size for larger files; doubling it doubles that limit.
Both endpoints still read the full file to compare it.
In syq rsync, -B / --block-size selects the comparison block size.
Do not combine it with comparison-block-size.
Streaming and request windows
Syq normally streams remote ranges larger than four ordinary requests
(16 MiB with default settings), with stream blocks of at most 2 MiB. The
threshold follows the effective request size, including bandwidth limits.
copy-path=streaming forces streaming and disables whole-file and small-file
shortcuts. copy-path=auto-streaming keeps those shortcuts and streams the
remaining ranges.
An explicit request-size also sets the streaming block size; bandwidth and
receiver limits may reduce it. Setting pipeline-depth disables automatic
streaming and cannot combine with either forced streaming mode.
To compare pipeline depths, hold the worker count and request size fixed:
syq cp data.bin --to host --as /scratch/pipeline.bin --performance-tuning workers=1 -v \
--performance-tuning copy-path=ranges,request-size=1M,pipeline-depth=4
Repeat with pipeline-depth=8 and 16, using a fresh destination each time.
These allow up to 4, 8, and 16 MiB of outstanding requests per endpoint per worker.
Compare with the defaults too; larger windows may add memory use without improving speed.
Batch size and splitting
Tune small-file batches with batch-files and batch-bytes:
syq cp --srcs-in small-files --to server --into /scratch/benchmark-small \
--performance-tuning workers=1,batch-files=256,batch-bytes=8M -v
Explicit batch settings replace the small-copy shortcut with worker batches.
With a bandwidth cap, each batch contains at most one file. Batch controls
cannot combine with copy-path=ranges or copy-path=streaming;
auto-streaming accepts them.
Remote new-file copies can batch files up to the smaller of request-size and
batch-bytes. Larger limits allow more data to be held in memory; interrupted
whole-file copies restart from the beginning. On macOS, files above the batching
limit can use APFS cloning. That limit is the smallest of the comparison block
size, batch-bytes, and request-size.
split-min-size controls how small a region an idle worker can take from another
worker. Lower values allow finer sharing; higher values reduce assignments.
Splits align to comparison blocks and need twice the minimum remaining size.
Average rate and burst patterns
With a bandwidth cap, bw-pacing controls when data is sent:
125ms(default): send smaller requests at regular intervals. The first request starts immediately, so short copies can exceed the average rate.average: wait for each request’s byte budget before sending it. A 2 MiB request at 1 MiB/s waits about two seconds, then can arrive in a burst.
Streaming and transport buffering can also produce bursts. Restricted receivers apply their authorized rate and request-size limits.
Recording a comparison
Use the same reporting options and fresh destinations for each run.
Prefer -v; --stats can change which copy optimizations run.
With overrides, -v reports effective settings and a final
syq: tuning observed: diagnostic. Check elapsed time, exit status, and copied
contents. See Quick comparison for the benchmark script.
Resource limits
--resource-limits sets ceilings while syq chooses how to run the copy.
--performance-tuning fixes individual settings instead. Both are available
in syq cp and syq rsync; S3 copies use syq cp.
Supply comma-separated KEY=VALUE pairs:
| Key | Default | Meaning |
|---|---|---|
bandwidth | 0 (unlimited) | Aggregate logical file-data bytes per second across the copy’s workers |
workers | Automatic | Ceiling of 1–65536 filesystem copy-worker slots |
s3-requests | Automatic | Ceiling of 1–65536 simultaneous S3 data requests across objects; excludes metadata requests and idle sockets |
s3-objects | Automatic | Ceiling of 1–65536 S3 objects in progress, including preparation and finalization |
s3-parts-per-object | Automatic | Ceiling of 1–1024 simultaneous parts or ranges per S3 object |
syq cp data --to server --into backup --resource-limits bandwidth=10M
This limits the average copy rate to 10 MiB/s. It counts bytes before compression, encryption, and protocol overhead; it does not cap every network burst.
Concurrency ceilings
syq cp data --to server --into backup --resource-limits workers=4,bandwidth=10M
Syq can adjust the filesystem copy-worker count up to four. By contrast,
--performance-tuning workers=4 fixes four worker slots. Unused slots can
remain idle in either case. Worker counts do not include directory scanning,
metadata processing, or control connections, and do not cap total threads,
sockets, CPU use, or memory.
Each concurrency key conflicts with the same key in --performance-tuning,
even when the values match. Controls for different quantities can combine:
for example, a fixed request size with a worker ceiling. Unknown keys,
duplicate keys, zero counts, and out-of-range values are rejected.
A ceiling does not force syq to use that many slots. Filesystem copies have no default tuning ceiling; syq adjusts the count from measured throughput. Restricted receiver limits still apply. S3 ceilings constrain the route’s normal automatic range without raising it. They are nested: object and per-object part counts also share the aggregate request ceiling. A fixed setting for one count still operates within ceilings on the other counts.
Use workers only for filesystem copies and the S3 keys only for S3 copies.
Limits apply to the remote coordinator too and are not saved.
Bandwidth units
Rates accept decimals and case-insensitive suffixes:
| Spelling | Unit |
|---|---|
No suffix, K, KiB | 1,024 bytes per second |
M, MiB; G, GiB; T, TiB; P, PiB | Successive powers of 1,024 bytes per second |
KB, MB, GB, TB, PB | Successive powers of 1,000 bytes per second |
B | Bytes per second |
A final +1 or -1 adjusts the scaled value by one byte before rounding.
Rates are rounded to the nearest KiB/s. Zero disables the cap; nonzero values
below 512 bytes/s are rejected. For example, 1024 and 1M both select 1 MiB/s.
Where the limit applies
For filesystem copies, workers share one aggregate limit. Buffering, streaming,
and SSH/TCP overhead can produce bursts.
bw-pacing controls pacing.
A capped copy uses normal copying instead of local filesystem cloning.
For local/S3 copies, the limit applies to scheduled data, with upload bursts up to a part. Server-side S3 copies do not pass object bodies through this machine and are not paced by this setting.
A worker ceiling also limits the starting connection count and leaves the remembered count unchanged. A bandwidth limit alone still allows syq to update it.
In syq rsync, --bwlimit RATE selects the same rate limit. Do not combine it
with --resource-limits bandwidth=RATE.
Integrity checking
--integrity-checking controls extra payload checks:
| Key | Default | Accepted values |
|---|---|---|
transfer | off | off, blake3, sha256, md5, xxh3-128 |
BLAKE3 and SHA-256 are cryptographic hashes. MD5 supports existing manifests; XXH3-128 is a noncryptographic checksum. Use BLAKE3 or SHA-256 with an expected hash from a trusted source to check authenticity; see Expected hashes.
Comparison
Syq normally skips files whose size and modification time match. For filesystem
copies, syq cp compares whole seconds exactly and ignores as many trailing
fractional digits as are zero in the destination timestamp. For example, destination .120000000
seconds matches source .123456789; a whole-second destination timestamp
ignores the source fraction entirely. This accommodates destinations that
truncate fractional seconds, but it also ignores coincidental trailing zeros;
the rule uses the timestamp, not a measurement of filesystem precision.
Directory metadata previews use the same fractional precision rule. syq rsync
compares whole seconds only when checking file contents.
Local/S3 copies compare timestamps exactly, including stored nanoseconds; they do not use the filesystem rule above. Uploads need syq metadata on the existing object for this shortcut. Downloads use the stored source timestamp, or S3’s modification time in whole seconds when syq metadata is absent.
A timestamp difference outside that precision triggers checking even when the
source is older. Use --hash to check contents even when size and timestamp
match:
syq cp --hash --srcs-in project --into backup
--hash uses BLAKE3. In rsync syntax, use -c or --checksum for the same
comparison.
Payload checks
Extra payload checks default to transfer=off. Enable them with, for example,
--integrity-checking transfer=blake3.
SSH and encrypted TCP retain their transport authentication independently.
--tcp-plain does not enable payload checks automatically, and checksums do
not authenticate plaintext traffic: an attacker can replace both the data and
its checksum.
For a complete check of a local copy, use an expected hash.
For local/S3 copies, provider request checksums remain enabled. The transfer
setting does not disable them. Single-part uploads with a SHA-256 request
checksum also reuse that digest for payload signing, without another hashing
pass. The transfer algorithm records a whole-file hash on upload and checks
stored hashes on download when present. Use an expected hash for objects without
a stored hash. See Filesystem differences.
Server-side S3 copies preserve stored hashes without reading or verifying object bodies. They do not support content-hash comparison, extra transfer hashing, or expected hashes.
Filesystem descriptor copies accept transfer=ALGORITHM for optional payload
checks. S3 descriptor copies reject extra transfer hashing: they keep provider
checksums but do not store syq hash metadata. Descriptor copies do not reread the result by default.
Expected hashes
Supply expected_hash on individual mapping entries
to require known whole-file contents. The expectation includes its algorithm
and hexadecimal value, independently of the comparison and transfer settings.
Syq checks the complete result, including reused bytes. A mismatch fails the
file. With normal staging, checking happens before replacing the destination;
with --inplace, the destination has already been modified. Matching size and
time allow syq to check the existing destination first and skip copying if its
hash matches. Selection filters still apply. Dry runs do not check expectations.
Compare without copying
Use --dry-run --hash to compare without copying:
syq cp --dry-run --hash --srcs-in project --into backup
Differences appear as planned changes. For machine-readable output, add
--results; for two servers, see
remote comparisons.
S3 comparisons may use a stored object hash without reading the object body.
Consistency and durability
A copy reads files over time. If another program changes them while syq is
reading, the result may combine data from different moments. Syq does not
create a snapshot; use a filesystem snapshot or stop the writer when you need
a consistent view. --inplace also lets destination readers see partial
updates, including after an interrupted copy.
Successful completion does not guarantee that the copy will survive an
immediate power loss. Normal file copies do not force transferred data onto
durable storage with fsync.
Remote copy reference
For setup and examples, start with Copy between servers.
This page covers copies from hostA (the source) to hostB (the destination).
See syq cp and syq receiver for
the option lists.
SSH configurations
Host-certificate-only trust, KnownHostsCommand, and RevokedHostKeys are
unsupported by the broker. Custom known-hosts paths must be unambiguous:
one absolute, whitespace-free filename per configured user/global directive.
The default known-hosts file list works. Syq reports unsupported configurations
instead of relaxing host verification.
Your local SSH configuration selects hostB’s login, address, port, and trusted host keys. HostA’s SSH configuration does not override those choices.
Enrollment
Enrollment needs normal command authority on the destination during setup.
Use a real directory, not a symlink. Transfers cannot overwrite the receiver’s
SSH configuration, programs, or enrollment state. Manage that state with
syq receiver commands; it is not a disposable cache.
Repeating enroll updates the receiver to match your local build. A pending
enrollment can be retried or revoked. Revoke and enroll again to rotate its
receipt key. Revocation stops active receivers before removing their state;
see access management
for interruption and retry behavior.
Stop active copies before upgrading: replacing the executable does not update
running receivers. Repeat syq receiver enroll hostB:/destination afterward
to refresh the installed receiver. Compatible enrollment keys and replay records
are preserved. Different client builds cannot share one installed receiver
concurrently; each needs a matching receiver.
An incompatible enrollment requires fresh setup, which needs ordinary SSH
access. Eligible copies install it automatically, or you can use enroll.
Incompatible old enrollments are not listed or removed by the new build;
they require manual cleanup on both machines.
Enrollment detects the destination platform and installs a matching executable.
Source builds require compatible platforms; --syq-path does not select the
restricted receiver. See Developing syq
for testing source builds.
Limits and unsupported options
Default limits are 100 million entries and 8 TiB of file data. Override them
for one transfer with --receiver-max-entries N and
--receiver-max-bytes SIZE. Values can raise or lower the defaults within the
allowed ranges. The transfer must start within 24 hours of authorization and
finish within seven days of authorization. New authorizations allow up to
128 worker connections, including for named destinations; a smaller worker
setting or resource limit lowers that allowance. Existing authorizations keep
their original limits.
| Option or combination | Restricted receiver |
|---|---|
--no-tcp | Use SSH workers directly from source to destination |
--tcp-congestion | The receiver enforces the algorithm authorized for TCP |
--tcp-plain | Unsupported; data connections must be encrypted |
--mapping | Listed destinations and necessary parent creation are authorized |
Fixed workers above 128 | Unsupported |
--inplace with --as-new | Unsupported |
--detach | Unsupported; the local broker must remain attached |
Native rm | Unsupported; use a normal SSH login |
Restricted copies use encrypted TCP when reachable and otherwise use SSH
workers from the source to the destination. Both transports share the same
copy authorization, limits, revocation, and signed receipt. A failed direct
connection never selects a relay through your machine; choose
--coordinate-at local explicitly to send data through it.
Signed results
The destination signs a receipt and your machine verifies it before reporting
results. Use -v for totals or --results FILE for
automation. These results arrive after verification, rather
than as live per-file progress. For --dry-run --results, use
--coordinate-at local to get the preview stream.
--receiver-receipt hashes adds BLAKE3 hashes of affected regular files.
Receipts allow up to four million records and 512 MiB of plaintext. Reaching
a cap stops further changes and reports an incomplete outcome.
A receipt does not prove that the source supplied every intended file or the right contents. See A compromised source server.
Other authentication modes
| Option | Authority available to the coordinating server |
|---|---|
--peer-auth own-credentials | Credentials already on that server |
--peer-auth broker | Your full destination-account authority, limited to that host and user |
--peer-auth full-agent | Ordinary, unrestricted agent forwarding |
--rsh COMMAND | Whatever your supplied SSH command permits |
The authentication broker allows 129 simultaneous clients by default. An explicit worker count or ceiling changes this to that count plus one control connection; restricted copies remain capped at 129 clients.
To make hostB pull from hostA using credentials already on hostB:
syq cp --coordinate-at dst --peer-auth own-credentials --from hostA data --to hostB --into /archive
There is no restricted source receiver, so direct pulls require one of these alternatives to the default authentication.
Detached copies
--detach leaves the copy running after your command exits. It requires
--peer-auth own-credentials or an explicit --rsh policy, so the coordinating
server can authenticate independently. It cannot use the restricted receiver
or --results.
Save the reported remote log location. The log is not a signed receipt.
If printing the location fails, the command reports an error but the job may
still be running. The coordinating server needs /bin/kill and either
setsid or perl.
Authorization selection
For syq cp with local sources and an SSH destination, --auth-from auto
tries live receiving machines in alphabetical order, allowing up to two seconds
for each reply. Offline or unsupported connections are skipped. With none
available, or with unsupported options, it uses the source machine’s SSH access.
Once approval is requested, refusal or failure ends the attempt.
For object-storage copies and removal, explicit --auth-from @NAME uses
storage authorization.
For SSH copies, --auth-from @NAME requires that receiving machine
to authorize the copy. --auth-from ssh uses the source machine’s SSH access.
These options choose authorization, not the destination: --to host names an
SSH destination, while --to @NAME sends files to a receiving machine.
SSH authorization through a receiving machine does not support --detach, custom
--rsh or --syq-path, --no-bootstrap, --pscope, alternative --peer-auth
or --coordinate-at, --no-tcp, or --tcp-plain. It requires direct encrypted
TCP from source to destination. Destination completion does not request
permission through a receiving machine; use --auth-from ssh for completion
through the source’s own SSH access.
On this route, quoted ~ and ~/archive select the destination account’s home
directory. Use ./~/archive for a literal directory called ~. Avoid
~//archive: explicit receiving authorization keeps it under the home directory,
but automatic selection uses ordinary SSH, where it resolves to /archive.
Verification
For comparisons between two servers, use --dry-run --hash --coordinate-at local.
This uses ordinary SSH access and supports --results FILE without restricted
receiver enrollment. Files are hashed on the servers; your machine receives
hashes, listings, and results. Remote setup may still cache the helper or
install syq.
Persistence details
See syq persist for the option list.
For everyday setup, start with Keep connections open or Use your laptop from a server.
Names and profiles
Give a project its own receiving name and directory:
mkdir -p ~/work/project
syq persist receive on --name laptop --cwd ~
syq persist receive on --name project --root ~/work/project
syq persist connect server
Both profiles work from the same server account: syq cp results --to @project
and syq cp report.pdf --to @laptop. Each profile has its own directory, copy
root, limits, automatic approval root, and background connection to each
allowed server.
New profiles start with the usual defaults, including asking for approval; they
do not inherit another profile’s trust or confinement settings. Up to 32 profiles
can be saved.
Change or stop a profile
receive on --name NAME creates a profile or updates that name’s settings;
omitted options keep their saved values.
Without --name, receive on updates the first saved profile, shown first by
receive status. The initial hostname profile becomes a saved profile when
persistence first connects; adding a new name then keeps that original profile.
If no preferences have been saved yet, the first explicitly chosen name replaces
the implicit hostname default.
syq persist receive status
syq persist receive status --name project
syq persist receive off --name project
syq persist receive on --name project
syq persist receive remove project
syq persist receive wait server --name laptop --timeout 30
Updating, stopping, or removing a profile cancels only that profile’s copies,
commands, and pending approvals. Other profiles keep working. receive off
without a name stops all profiles; receive on enables the first profile, and
receive on --name NAME enables another. Removing the first profile makes the
next saved profile the default. The last profile can be disabled but cannot be
removed. pending, approve, and deny work across all profiles; prompts name
the receiving profile. Without --name, receive wait waits for every enabled
profile on that server.
Choose allowed servers
To give one server an inbox for downloads that do not need approval:
mkdir -p ~/Downloads/work
syq persist receive on --name work-inbox --server work \
--auto-approve-root ~/Downloads/work
syq persist connect work
On work, use syq cp results --to @work-inbox. Other connections cannot
use that profile. Your general profile can still ask for approval on every
download.
Profiles default to all connected servers. receive on --name NAME --server HOST
restricts a profile to the exact SSH destination used on the receiving machine.
Use the endpoint shown by persist status: for example work, alice@work,
or alice@work:2222. Matching includes an explicitly selected user and port;
it does not expand SSH aliases or equate omitted ports with explicit ports.
An alias uses the account and host configured for it in your local SSH settings.
The server cannot select its own identity for this check. Changing your local
SSH configuration can change which account an allowed alias reaches.
Repeat --server to supply several destinations. Each supplied list replaces
the saved list; --all-servers clears the restriction. Changes apply to existing
persistent connections too. receive wait HOST waits only for profiles allowed
on HOST.
Move a name to another machine
Names belong to a server account and stay assigned to their original receiving machine when it disconnects. Another laptop cannot claim the same name, even while the original is offline. Stopping receiving or removing a local profile does not release its server-side name.
To replace a laptop, stop its receiving connection, then run
syq persist destinations forget laptop on the server. Run
syq persist connect server on the replacement laptop to claim the released
name. A rejected connection needs this explicit retry. Forgetting a live
connection is refused.
Back up the receiving identity
Syq generates one receiver key per local account, shared by receiving profiles
and syq versions. It lives in ~/.syq-receiver-identity/identity_ed25519 and is
independent of your SSH login keys and profile settings. Keep that directory
across upgrades; include it in private backups if you want to restore the same
identity. Losing it requires releasing the old names on each server. Copying it
to another machine gives that machine the same receiver identity. If the old
connection is still responsive, the replacement is rejected; use different
profile names to receive on both machines at once. An unresponsive connection
must time out before its name can reconnect.
Directories
The three directory settings are independent:
| Setting | Purpose | Clear it with |
|---|---|---|
--cwd DIR | Starting directory for relative paths | --auto-cwd |
--root DIR | Hard boundary for downloads, even with approval | --no-root |
--auto-approve-root DIR | Downloads confined here skip approval | --no-auto-approve-root |
Each requires an existing directory with a UTF-8 path; neither root can be /.
An explicit cwd must be inside the hard root, if set. Otherwise the starting
directory is the hard root, the automatic approval root, or your home directory,
in that order. Status reports the effective cwd and whether it is explicit.
Downloads outside the automatic approval root ask for approval, provided they stay within the hard root if one is set. Symlinks cannot grant automatic writes outside the approval root. Replacing the automatic root directory itself requires approval, and is prohibited when it is also the hard root. These roots do not restrict approved commands or the destinations of approved copies to other servers.
Invalid settings leave the saved configuration unchanged. If a saved directory disappears, you can still inspect, disable, or reconfigure the profile. Filenames inside it may use normal Unix filename bytes.
Copy limits
Named copies use encrypted TCP workers initiated by the receiving machine,
falling back to SSH when TCP is unreachable. --no-tcp forces SSH;
--tcp-ports selects the listening port range on the sending server.
The receiving connection must stay open throughout the copy.
Copies support directories, symlinks, modification times, filters, hashing,
resume, mappings, --preserve=permissions, and the
overwrite policies.
Ownership preservation, special-file preservation, and --inplace are
unsupported. Timestamp comparisons trust the source’s reported modification
times.
Each copy is limited to 100 GiB and one million touched entries by default.
Change these ceilings with syq persist receive on --max-bytes 20G --max-entries 100000.
Lower limits requested by the sender also apply. Limits are per copy; repeated
copies can fill the disk. Copies support at most 128 workers each.
Pruning is disabled unless the laptop sets a positive --max-delete.
A sending --prune command must also supply its own --max-delete ceiling,
no higher than the laptop’s. Validation failures leave the copy unstarted.
Errors during copying fail visibly and may leave partial files for retry.
The sender verifies a signed receipt before reporting success.
Connection lifetime and waits
persist on enables persistence for subsequent syq SSH connections.
persist connect server enables it and connects immediately. With receiving
enabled, it waits until receiving is ready. --timeout 30 limits that wait
after SSH and helper setup, not authentication or installation. Failure leaves
persistence enabled; a healthy connection is reused without cancelling requests.
Connections have no idle expiry. Receiving reconnects after interruptions; other SSH logins reopen on their next use. Syq installs no login service, so connect again after reboot. Copies are not queued or retried automatically.
Use syq persist receive wait server --timeout 30 to wait without starting or
restarting a connection. On the server:
syq persist destinations list
syq persist destinations wait laptop --timeout 30
syq persist destinations forget laptop
forget releases the name assignment while its connection is stopped. For structured
status and approval requests, see connection status.
Isolated script scopes
syq persist on --ephemeral prints a scope path. Pass it as --pscope PATH
to syq persist connect server and subsequent copy commands, then close it
with syq persist off --pscope PATH. This does not change your user setting.
These scopes reuse SSH logins only. They do not enable receiving, authorization
through your machine, or commands on it. Idle connections close within ten
minutes, including helper sessions. Explicitly closing the scope ends reuse
immediately. For return copies or commands, use persistence without --pscope.
Setup and recovery
Linux desktop prompts need libnotify 0.7.10 or later and a running notification
service. macOS uses a native dialog. Start receiving from a terminal in your
desktop session. After changing sessions, run syq persist receive off, then
enable the profiles you need from a terminal in the new session.
syq persist receive pending shows complete requests and prompt errors.
Overwrite warnings are advisory; destination entries can change before copying.
--notify off selects terminal approval; --notify desktop restores prompts.
Start the connection from your laptop with syq persist connect server.
Opening a plain SSH session does not enable receiving, and connecting onward
from that server to another host does not carry your laptop’s receiving name
with you.
For automatic reconnection to work, your SSH key or agent must be available and the server’s host key must already be trusted. The background service cannot ask for a password. The server must allow remote Unix socket forwarding; OpenSSH 9.2 also requires permission for remote TCP forwarding.
The server command uses the matching helper installed by the receiving connection. If that helper is missing or does not recognize an option, update syq on both machines and reconnect from your laptop.
Updating connections
Before upgrading, run syq persist off on the receiving machine to stop its
background services. Replacing the executable alone does not update running
services. Close script scopes with syq persist off --pscope PATH too.
After upgrading both machines, run syq persist connect server for each server.
Saved names, directories, and limits carry over. Existing working directories
remain explicitly selected. Settings using the former --approve always now
require approval; choose --auto-approve-root explicitly to enable unattended
downloads. Older binaries reject the updated preferences; use the newer binary
to manage receiving.
Connections created before persistent receiver identities remain discoverable. Their names become assigned when an updated receiving machine reconnects. Updated commands verify assigned names even when their connection uses an older helper; a receiver without a matching identity is rejected. Older binaries do not enforce these assignments, so use updated syq commands on the server and stop older receiving services before switching versions. The receiver key and assignments are independent of the helper build and survive later upgrades.
Automation results
Native cp, rm, and clean-partials can write structured outcomes alongside
human output:
syq cp --srcs-in project --into backup --results copy.ndjson
syq rm old-output --results removal.ndjson
Each line is a JSON record. Require a final result record before trusting
completion. EOF without one means the outcome is unknown, even if no error
record appeared.
This page describes stream semantics. The JSON Schema lists precise field shapes; example streams show complete runs. For mapping manifests, see Mapping format.
The channel
--results FILE creates a fresh regular file and refuses an existing one.
Use a new name for each run and keep it outside trees being copied or removed.
--results-fd N uses an already opened writable descriptor above 2:
syq cp project --into backup --results-fd 3 3>copy.ndjson
Use a descriptor for pipes or other non-regular sinks. Named results paths
follow the normal native symlink rules; --follow permits traversal.
Human stdout and stderr are not part of this contract.
Results are written on the invoking machine, including for remote removal.
For direct restricted remote-to-remote copies, they are derived from the
verified destination receipt and marked provenance: "receiver_attested".
Those records arrive after receipt verification. Otherwise remote-to-remote
results require --coordinate-at local, which relays data through your
machine. Remote-to-remote --dry-run --results
also requires that local route.
--results supports native cp (including pruning), rm, and clean-partials.
Cleanup uses the removal records described below. Results cannot combine with
--detach. The restricted receiver supports copy changes,
including capped pruning, but not native removal.
Records arrive in seq order; error and terminal records are flushed
immediately. If writing the stream fails, syq warns where possible and stops
writing it. Filesystem work can continue, leaving an incomplete stream.
Argument errors exit 2 without a stream. Failure to open the results sink exits 1 without a stream. Once opened, completed runs and fatal setup failures emit a terminal record if the sink remains writable. Crashes and interruptions can leave it missing.
Consumer rules
- Check the terminal record and process exit code; a mismatch is a protocol error. Use terminal totals, not progress or a count of operation records.
- Retry individual mapping entries only after terminal
successorpartial; see Retry failed mapping entries. Other statuses can leave entries unresolved. - Ignore unknown record types and optional fields within a supported schema version.
- Reject unknown
schema,schema_version, terminalstatus, or pathencodingvalues. - Treat human
messagetext as display only; parse structured fields.
--results includes both progress samples and final outcomes; use it for new
integrations, including live progress displays.
Exit codes
| Code | Terminal status | Meaning |
|---|---|---|
0 | success | Requested operation succeeded |
1 | failed / aborted | Fatal failure or abort |
2 | No stream | Invalid arguments |
23 | partial | Per-entry failures; independent work finished |
25 | refused | A safety cap refused deletions |
Removal terminals use only success, partial, or failed. A results-sink
startup error also exits 1 without a stream.
Record envelope
Every record carries:
| Field | Value |
|---|---|
schema | "syq.automation" |
schema_version | 2; Stream mappings use 3 |
seq | Integer starting at 0, strictly increasing |
type | Record type |
Paths are tagged: {"encoding":"utf-8","value":"docs/a.txt"}, or
encoding: "base64" with standard base64 of raw filename bytes. Byte and
count fields are non-negative integers within the u64 range.
Record types
run
Always first. Identifies the invocation with run_id, started_at (Unix
seconds), syq_version, mode (cp or rm), dry_run, and endpoints.
Endpoints identify role and kind (local, ssh, or s3). SSH endpoints carry
host/user; S3 endpoints carry host: "s3://BUCKET" without a user. They omit
credentials, headers, ports, and raw command arguments. Clients that predate S3
must be upgraded to decode S3 endpoint kinds; local and SSH streams retain
their existing representation.
Copy runs also carry prune and mapping. Removal has one source endpoint
regardless of selector count and omits those copy fields.
For S3 downloads using --only-new, unchanged-file totals
include skipped symlinks selected through a prefix, but exclude symlinks named
directly or through a mapping.
progress
Sampled telemetry, approximately once per second, for displays rather than
accounting. It includes bytes, files, exclusions, scan state, elapsed time, and
optional rate_bytes_per_second and eta_ms estimates.
Removal has zero byte and unchanged/excluded counts; its file counts reflect
outcomes received so far. Copies that reach transfer completion emit a final
progress sample, even for short copies. The terminal record owns final totals.
For performance measurements collected with --stats, see
Performance measurements.
trace
One intended copy change in a dry run. Includes action, destination, source
for mapping entries, kind and bytes where applicable, plus a reason:
destination_missing, type_differs, content_differs, metadata_differs,
or destination_only.
For filesystem copies of regular files, content_differs means sizes or compared
hashes differ;
metadata_differs also covers copies selected by the metadata quick check.
A transfer_file trace omits bytes when only metadata would change.
A trace cannot be matched by identity to a later live operation: the filesystem may change between runs.
operation_result
An outcome for a completed copy change or a failed mapping entry.
| Field | Meaning |
|---|---|
action | transfer_file, create_directory, create_symlink, create_special, or delete; attested streams also use set_metadata and observe_hash |
dst | Path relative to the destination container |
src | Mapping source path, where available; absent for ordinary copies and deletions |
kind | file, dir, symlink, or special, when known |
disposition | succeeded, failed, blocked; attested streams also use incomplete and observed |
bytes, attempts | Optional transfer information |
expected_hash | Expected whole-file digest, when supplied: an object with algorithm and hexadecimal value; preserve it in retry mappings |
metadata | Explicit destination attributes from the mapping, when supplied; preserve them in retry mappings |
retryable | On failures: yes, no, or unknown |
class, os_kind, message | Error details where available |
provenance, scope, code | Attested origin, signed destination-scope index, and receiver outcome code |
In attested records, dst is relative to the destination area identified by
scope. An attested set_metadata omits kind.
Unchanged and excluded entries have totals only. Ordinary live streams do not
emit per-operation records for metadata-only updates, though dry runs emit
metadata_differs traces. Failed implicit parent creation can lack src and
is non-retryable. Do not construct a retry source from its destination name.
stream_ready
A producer may start supplying payload. Uploads without a skip policy emit this
before destination setup finishes, allowing several writers to connect concurrently.
With --only-new or metadata preservation requested, source selection and destination checks come first.
Skipped copies and dry runs finish without this record. Setup, transfer, or
publication can still fail; require the terminal result for completion.
stream_result
Descriptor copies emit one stream_result instead of pathname-based
operation_result or trace records. source and destination each contain
one of path (a tagged path) or fd (the caller’s descriptor number). Paths
are source/destination operands, with a source basename appended for --into;
the run’s endpoints identify where they belong. An FD is a handle for this
invocation, not a reusable source or retry identity.
disposition is succeeded, failed, skipped, or planned; dry_run identifies a
preview. bytes counts transferred bytes, including partial work on failure,
or planned bytes in a dry run. It is absent for a preview of unknown-length
input. A failed upload may have transferred bytes without publishing them.
Skipped copies consume no payload, report zero bytes, and count as one excluded
file. The normal terminal result still establishes completion. Its optional
bytes_total_known field is false when the source length is unknown; preview
byte totals then count only known bytes, rather than asserting an empty input.
Stream mappings use version 3 and emit one stream_result for each
callback entry, with an additional zero-based entry index. Each endpoint is
{"path": <tagged path>} or {"entry": N, "callback": true}. Failed entries
include message. These records identify callbacks within this invocation;
they cannot be converted to a pathname retry manifest. Ordinary entries in a
mixed mapping still emit their usual records. The
Python callback reference describes
publication and application retries. Callback admission uses the SDK’s private
channel, so these runs do not emit stream_ready.
Removal records
For S3 version removal, removal_trace and removal_result include optional
s3_version_id (string) and s3_delete_marker (boolean) fields. Each record and
entry count refers to one version or delete marker. Ordinary S3 removal omits
these fields.
| Type | Meaning |
|---|---|
selection_result | One explicit selector resolved or found missing |
removal_trace | One entry that a dry run would remove |
removal_result | One finished removal or inspection failure |
selection_result uses a zero-based selector index, the original path,
status (resolved or missing), and kind when resolved. Missing selectors
succeed. Overlapping or duplicate selectors keep separate indexes. If a later
selector cannot resolve, earlier selection records may precede the fatal
error and terminal record.
removal_trace includes selector, path, kind, and
disposition: "would_remove". Directories follow their descendants.
removal_result includes selector, path, kind when known, attempts, and
disposition: removed, already_absent, or failed. Already absent is
success. Failures include error details and retryability where available,
and also produce a counted error record. Dry-run inspection failures can
produce failed removal results, but never successful removal results.
error
One per counted error. message is display text; class and os_kind are
provided where known. Classes are io, transport, conflict, integrity,
safety_limit, usage, and internal.
OS kinds are not_found, permission_denied, already_exists,
invalid_input, no_space, quota_exceeded, read_only, and other.
They preserve OS error meaning across hosts without requiring matching errno
numbers. Receiver refusals use class: "safety_limit" with provenance and
the receiver’s code.
final_state
Attested streams only: the destination’s final observation of a path the
transfer could have changed. Includes scope, dst, and an object:
absent, an observation failure, or present with kind, size, applicable
metadata, and symlink target. With --receiver-receipt hashes, regular
files also have a BLAKE3 hash.
Object kinds distinguish directories, files, symlinks, FIFOs, sockets,
character/block devices, and other objects. Metadata fields are mode, uid,
gid, mtime, mtime_nsec, and rdev. Consult the schema for exact shapes.
This observes destination state; it does not attest source completeness.
result
Exactly one terminal record, always last when the stream completes. Common
fields are status, exit_code, dry_run, errors, and elapsed_ms.
Copy terminals may also include copying_elapsed_ms: the wall-clock span from
first file work to last completed file work across workers, including per-file
checks, finalization and gaps. Initial setup before file work is excluded;
planning and connections can overlap this interval. It is not a sum of worker
times or pure network time. The field is absent when no bytes moved or the
coordinator does not supply it, including older releases and attested terminals.
Use elapsed_ms for end-to-end throughput comparisons.
Copy totals include transferred/unchanged/excluded files, created directories,
symlinks and specials, transferred/unchanged bytes, and on pruning runs
deletions_planned, deletions_completed, and deletions_blocked. A fatal
failure reports what finished before it stopped. Dry-run totals describe
planned work, not committed changes.
Attested terminals add receipt_status (clean, failed, or incomplete),
provenance, and receipt counts. They can attest only what the receiver saw:
- Unchanged and excluded totals are zero, not a claim that every source entry changed.
- Only
deletions_completedappears; planned and blocked deletion totals are omitted. errorscounts attested error records for failed/incomplete operations, refusals, and failed or partial final-state observations.
Removal terminals have mode: "rm" and totals for selectors, resolved and
missing selectors, planned/removed/already-absent entries, and failed entries.
Live removal leaves entries_planned zero. Dry removal leaves
entries_removed and entries_already_absent zero, but inspection failures
may increase entries_failed.
The human summary uses the same terminal totals.
Performance measurements
See Performance measurements for worker time,
endpoint operations, CPU, and TCP counters collected with --stats.
Compatibility
Within a schema version, required fields keep their types and meanings; existing types, actions, dispositions, statuses, classes, and reasons are not renamed or reused. New record types and optional fields may be added. Human messages may change at any time.
Connection status
syq persist status --json reports the persistence setting, scope, and endpoints
without starting connections. Endpoint states are starting, connecting,
ready, reconnecting, failed, or inactive. Each entry also reports whether
SSH is connected and the receiving state and errors.
With receiving enabled for an endpoint, ready means its allowed receiving
profiles are online; SSH can reconnect on its next use. If receiving preferences
cannot be read, SSH entries are still listed, receiving_error explains the
failure, and each entry’s
receiving_enabled is null.
persist receive status --json reports each profile’s cwd, cwd_explicit,
root, auto_approve_root, and servers. An empty server list means all
connections. A null automatic approval root means every download asks.
Connection entries list only profiles allowed on that endpoint.
Command approvals in syq persist receive pending --json use kind: "command"
and include argv, cwd, and permission. Argument and directory strings in
this summary are escaped for display. Use an up-to-date syq binary to inspect
and approve commands; clients that only support copy requests omit them.
Retry failed mapping entries
Add --results r.ndjson to record outcomes in a fresh file outside the copy
trees.
Failed mapping entries contain src, dst, and kind, so they can form a
retry manifest. Preserve expected_hash and metadata too when present. First require a terminal result with success or partial:
a missing terminal or an early stop means some entries may have no results.
In those cases, rerun the original copy instead.
set -o pipefail
syq cp --mapping big.ndjson -C src --to nas --into /data --results r.ndjson
jq -cs 'if any(.[]; .schema != "syq.automation" or .schema_version != 2)
then "unsupported results schema for pathname retry" | halt_error
elif (.[-1].type? // "") != "result"
then "incomplete results stream (no terminal record)" | halt_error
elif (.[-1].status != "success" and .[-1].status != "partial")
then "run stopped early (status \(.[-1].status)); rerun it instead of retrying" | halt_error
else .[] | select(.type == "operation_result"
and .disposition == "failed"
and .retryable != "no")
| {src, dst, kind}
+ (if has("expected_hash") then {expected_hash} else {} end)
+ (if has("metadata") then {metadata} else {} end)
end' r.ndjson \
| syq cp --mapping - -C src --to nas --into /data
The filter skips non-retryable entries, including failed implicit parent creation without a source path. It does not guarantee that retrying will succeed; fix the underlying error first. Unchanged and excluded files appear only in summary totals, not as individual results.
For command-restricted copies between servers, --results contains verified
receiver receipts; see Signed results. These describe
destination changes rather than source entry failures; retry the original
mapping instead of applying the operation_result filter above.
Performance measurements
Use these measurements to investigate a slow copy. For a starting command and common symptoms, see Diagnose a slow copy. The measurements appear in automation progress records; use the terminal result for the copy’s outcome and totals.
With --stats (or debug logging), progress records can include an activity
object, exposed as ProgressEvent.activity in Python. --results alone does
not collect it. Accept missing activity data: collection can be unavailable
or lose its remote connection while the copy continues.
| Activity field | Meaning |
|---|---|
elapsed_ms | Time since the preceding coordinator sample |
workers | Counts at sampling time and fractions of observed worker time |
endpoints | Local operations or the latest reports from remote connections |
processes | User and system CPU deltas in nanoseconds, once per process |
summary | The same cumulative worker, endpoint and CPU summary printed by --stats, with observed seconds beside the fractions; idle actors are omitted |
Worker time
Worker fractions divide each state’s accumulated duration by observed_ns,
including waits still in progress. They sum to one when activity was observed;
an empty object means none was observed. cumulative_fractions uses all observed
worker time since collection began. Both exclude tuner parking; parked_ns and
cumulative_parked_ns report that time separately in nanoseconds. These are
fractions of worker time, not command elapsed time. observed includes retired
workers; active excludes retired workers and workers parked by connection tuning. awaiting_work
counts workers waiting for a scheduler job. Compare this with scan_done to
investigate insufficient work supply.
source_request and source_response cover sending a request and awaiting its
response. destination_send and destination_ack cover sending writes and waiting
for replies; synchronous local work can occur inside these calls. pacing covers
bandwidth-limit waits. other_work is remaining worker activity. A source-response
wait alone cannot distinguish storage, CPU, transport or downstream backpressure.
Endpoint operations
Endpoint actors separate filesystem operations, server communication and Linux
read-ahead helpers. cumulative_actors reports the same measurements since the
connection subscribed, including when its latest sample has not advanced. Labels
include worker IDs to distinguish connections. Their fractions use each actor’s
own observed time; do not add fractions across actors or to worker fractions.
source_read measures demand-read syscalls, including reads used to hash existing destination data; hashing measures
content hashing; destination_write measures write syscalls. filesystem_copy
keeps a combined measurement where filesystem copy operations do not separate reads
and writes. handling covers remaining filesystem request work. Server
request_wait covers its request queue and response_send covers serialization
and writing replies. Server idleness does not by itself prove a transport limit.
prefetch_advice measures Linux read-ahead advice calls; its bytes are requested
bytes, not bytes physically read. prefetch_fence measures waiting for outstanding
advice before releasing or shrinking a range. Read and write byte counters count
completed operations; filesystem-copy bytes are logical bytes and may include
cloning or offload.
Process CPU
cpu is the interval delta; cumulative_cpu is the delta since collection began
for that process. Process identities apply only to this run. Helper helper_cpu
is part of its process CPU, so do not count it twice. Live Linux helper CPU uses
kernel clock ticks; short intervals can report zero.
Sample timing
Remote reports arrive at response boundaries and on connection retirement.
sample_age_ms measures time since receipt, so network delivery delay is additional
and clocks on different hosts need not agree. state_at_sample is the state at that
report. When no newer report arrives, endpoint elapsed_ms is null and actors is
empty; process CPU deltas are null too. A blocked remote call can therefore leave
stale evidence. Endpoint intervals can span several coordinator intervals. The
first remote CPU sample establishes a baseline and has no delta.
TCP counters
tcp and tcp_delta describe the coordinator end of a TCP socket; peer_tcp and
peer_tcp_delta describe the remote end at its reported sample. The raw counters
include RTT and retransmissions; deltas include receive-window-limited and
send-buffer-limited durations. Local counters are sampled by the progress ticker;
the final reading is retained after retirement. Null means unavailable, whereas
zero is a measurement. See the schema for the
complete field definitions.
Stream-mapping protocol
A program can supply or consume several byte streams in one syq cp process.
Entries share transfer connections and resource limits. This is the subprocess
interface used by Python callback mappings.
Ordinary shell pipelines use --src-fd or --as-fd.
Start a session
Create a connected Unix stream socket pair. Pass one end to the child, keeping it open across exec, and close that end in the parent after spawning. Start:
syq cp --mapping entries.jsonl --stream-mapping-fd FD \
--stream-concurrency 8 --results-fd RESULTS_FD \
--to server --into archives
FD must be greater than 2 and distinct from the results descriptor. These
programmatic options are hidden from help. --stream-concurrency defaults to
4 and accepts 1–256 active entries. Each entry can have a producer, a consumer,
or both. Payload bytes use separate descriptors; they never enter JSON messages.
Drain the control connection and results while producers and consumers run.
The manifest uses the existing mapping format, extended with a stream endpoint in place of either tagged pathname:
{"src":{"stream":0},"dst":{"encoding":"utf-8","value":"first.tar"}}
{"src":{"stream":1,"size":1024},"dst":{"encoding":"utf-8","value":"second.bin"}}
stream is the zero-based physical line index, not an OS descriptor number.
Blank lines still count toward indices. An optional source size promises the
exact byte count; a mismatch fails the entry before publication. kind can be
omitted or file. expected_hash checks the transferred bytes. Explicit
metadata applies to a named destination without requiring --preserve.
Syq reads and validates the whole manifest before copying. This catches malformed entries, duplicate destinations, and conflicts with declared file kinds. As with pathname mappings, actual source kinds are checked during execution. Ordinary pathname entries run first, followed by stream entries; mixed batches therefore have a separate endpoint setup for each phase. Already successful entries are not rolled back when another entry fails.
Control framing and version
Every frame contains:
- The single byte
S, received withrecvmsgso attachedSCM_RIGHTSdescriptors are collected with this byte. - A four-byte unsigned big-endian JSON byte length, between 1 and 65,536.
- That many UTF-8 JSON bytes, containing one object.
Read exactly one byte with recvmsg, then read the length and body normally.
Do not read ahead across frame boundaries: that could lose an attached descriptor.
Reject truncated ancillary data and close every descriptor on rejected frames.
Received descriptors belong to the client; mark them close-on-exec promptly.
Linux clients can use MSG_CMSG_CLOEXEC. On macOS, avoid launching subprocesses
concurrently with receiving and marking descriptors.
Syq first sends {"type":"hello","version":1} without descriptors. The client
replies {"version":1} using the same framing, also without descriptors. Reject
unsupported versions before producing bytes. Version 1 is the contract for this
interface; incompatible changes require a different version. Clients should
ignore unknown optional JSON fields, but reject unknown message types or
unexpected attached descriptors. A failed handshake may end without an end
message; the process exit and results still need checking.
Entry lifecycle
| Message from syq | Attached descriptors | Meaning |
|---|---|---|
{"type":"start","entry":N,"direction":"produce"} | payload, completion | Write this entry’s bytes |
{"type":"start","entry":N,"direction":"consume"} | payload, completion | Read this entry’s bytes |
{"type":"transferred","entry":N,"error":null} | none | Byte transfer and requested validation succeeded |
{"type":"transferred","entry":N,"error":"..."} | none | Transfer failed |
{"type":"end"} | none | No further control messages |
Each selected direction starts at most once. A skipped or dry-run entry has no
start message. Its outcome appears in the results stream. Descriptors are
independent streams and may be processed concurrently. Their numeric values are
chosen by the receiving OS, unrelated to the manifest index.
A producer writes and closes its payload descriptor. Only after its work has
succeeded does it write the single byte C to its completion descriptor and
close it. EOF without C fails the entry. Syq publishes a named destination
only after producer success and transfer validation; upload transferred
arrives after publication. Closing a buffering wrapper alone must not authorize
publication.
A consumer reads through payload EOF and checks transferred before reporting
success. On success, close the payload, write C to completion, and close
completion. A consumer that returns early must drain the remaining payload to
allow validation to finish. A consumer may acknowledge its own processing before
transferred if it postpones publishing any output until the complete session
succeeds. A stream-to-stream entry reports transferred after producer success
and before waiting for consumer acknowledgement. Syq cannot undo a consumer’s
side effects; extract archives into a staging directory.
A client-side failure closes completion without C. To cancel the entire
session, send SIGTERM to the coordinator, allowing it to abort staged files and
multipart uploads. Continue draining outputs while it exits; a caller’s deadline
may require killing its subprocess group. Neither lost control connections nor
payload EOF establishes successful completion. Non-replayable streams cannot be
automatically retried: the caller must regenerate the entry.
Results and applicable options
Use automation schema version 3. Each stream entry has one
stream_result with its index and outcome. Successful completion requires the
terminal result and process status to agree, and all expected entry results to
be present. Entry errors can yield a partial result and exit 23. Setup errors
may terminate before all entries have results. Application-level failures must
also fail the caller even if it has already consumed a terminal record.
Placement, preview, existence policies, compression, reporting, transfer tuning,
and resource limits apply. Stream entries share limits across endpoint sessions;
--stream-concurrency limits archive jobs separately from network workers.
Streams run on the invoking machine. Remote coordinator placement, detached
execution, and named receiver grants are unavailable for these entries: they
cannot inherit the caller’s live stream, and named grants authorize pathnames.
--inplace cannot provide publication after producer success. Pruning and tree
filters require a source tree; select entries in the calling program. Source
metadata policies require actual source attributes; generated streams have none.
Content comparison and size filters are unavailable in a stream-mapping batch;
use a known expected hash or select promised sizes in the caller. These
restrictions also apply to pathname entries in a mixed batch; use separate calls
when their options differ.
Environment and local files
Syq has no general configuration file. It uses your SSH configuration for SSH connections and AWS configuration for S3; see S3 options.
Add options through the environment
SYQ_CP_OPTIONS, SYQ_RSYNC_OPTIONS, and SYQ_RM_OPTIONS supply extra
arguments to their respective commands. They are useful when a script does
not let you change the syq command it runs:
SYQ_CP_OPTIONS='--resource-limits bandwidth=10M' ./nightly-backup.sh
The value is parsed with shell-style quoting and inserted before the command’s other arguments. Repeating an option follows the same rules as repeating it on the command line. Syq removes these three variables before launching SSH, helpers, or other programs; the rest of the environment is passed through.
Other variables
| Variable | Purpose |
|---|---|
SYQ_NO_UPDATE_CHECK, DO_NOT_TRACK | Disable update reminders; see Updates |
SYQ_TUNING_CACHE | Select the connection-count cache; an empty value disables it. See Remembered connection counts |
SYQ_TUNING_HISTORY | Select the local tuning history; empty disables it. See Tuning history |
SYQ_TUNING_HISTORY_SIZE | History retention target, default 128M |
SYQ_DEBUG | Add internal diagnostics to stderr |
SYQ_S3_DIAGNOSTICS=1 | Add S3 request diagnostics to stderr |
XDG_CACHE_HOME, XDG_CONFIG_HOME, XDG_RUNTIME_DIR | Relocate cache, preference, and runtime files |
Diagnostic formats can change between versions. For scripts, use
automation results. Usual system variables such as HOME,
TMPDIR, and SSH_AUTH_SOCK also apply.
Local state
Default locations include:
| Location | Contents |
|---|---|
~/.cache/syq/tuning.json | Learned connection counts in the legacy format |
~/.cache/syq/tuning.history-v1.sqlite | Local tuning timelines and filesystem startup hints |
~/.cache/syq/completion-endpoints.json | Hosts offered by completion |
~/.cache/syq/helpers/ | Downloaded SSH helpers |
~/.config/syq/persistence.json | Whether persistence is enabled |
~/.config/syq/receive.json | Receiving profiles |
~/.config/syq/install.json, last-update-check | Install receipt and update-check timing |
$XDG_RUNTIME_DIR/syq-persist-UID/ | Persistent connection sockets |
~/.syq-destinations-v3/ | Registered receiving names on a server |
~/.local/share/syq/restricted/ | Receiver enrollment state |
Copies can proceed when optional caches cannot be written. Persistent connections need a writable runtime directory. See Enrollment for receiver state and Names and profiles for backing up or replacing a receiving identity.
SSH helper installation
SSH helpers are installed under ~/.cache/syq/helpers/ on the server too.
If that directory cannot be created, select an installed helper with
--syq-path, or use --no-bootstrap when a matching syq is on the server’s
PATH.
When an official release installs a helper, it also tries to install the command
at ~/.local/bin/syq. Completion and background connections can trigger this
step. Failure to install the command does not stop the copy. Reusing a cached
helper does not repeat the command installation; use the standalone installer
if the command is missing. Development builds and connections using
--syq-path or --no-bootstrap do not install the command.
Rsync compatibility
For examples, start with Use rsync-style commands.
See syq rsync for all accepted options.
Differences to check before switching
| Area | Syq behavior |
|---|---|
| Filters | Gitignore syntax via --syq-ignore / --syq-ignore-from; rsync filters are unsupported |
| Deletion timing | Extras are deleted after copying; no delete-before or delete-during mode |
Positive --max-delete N | Deletes nothing if the plan exceeds N; rsync deletes up to N |
| Destination collisions | Distinct sources claiming the same destination fail before copying |
--ignore-existing | Keeps an existing non-directory even where the source would create a directory |
| Type changes | Refuses replacement between a directory and any non-directory, even when the directory is empty |
--update | Checks mtimes only for regular files; replacements between non-directory types still occur |
| Resume | Always keeps syq partial files; cannot reuse rsync partials |
| Delta transfer | Reuses matching blocks at the same offsets; does not find shifted blocks |
--rsync-path PATH | Exact syq executable path, not a shell fragment |
| Remote-to-remote | Refused; use native syq cp |
Scan and copy errors prevent deletion. A destination that contains its source
on the same host cannot be pruned. Preview deletion scope with --dry-run -v;
see deletion rules.
The compatibility command uses rsync’s default size-and-whole-second timestamp
quick check. Native syq cp also compares fractional seconds at the precision
suggested by the destination timestamp; see timestamp matching.
Use -c to compare contents when size and timestamp match; source timestamps
are preserved, so ordinary clock skew does not require the source timestamp
to be newer.
Syq uses numeric IDs and always keeps partial files, so --numeric-ids and
--partial are accepted no-ops. -P enables progress. Compression is on by
default; -z does not enable anything extra. -B / --block-size changes
syq’s comparison and reuse block size; see the tuning table
for its default and allowed range. Values outside that range are rejected.
Unsupported features
| Feature | Options or syntax |
|---|---|
| Rsync filter rules | --exclude, --include, --filter |
| Hard links, ACLs, xattrs, sparse files | -H, -A, -X, -S |
| Backup and alternate destination trees | --backup, --backup-dir, --suffix, --link-dest, --compare-dest, --copy-dest |
| Following descendant links | -L, --copy-links, --copy-unsafe-links, -k, --copy-dirlinks, -K, --keep-dirlinks |
| Link filtering or rewriting | --safe-links, --munge-links |
| Other placement and filesystem controls | -R / --relative, --partial-dir, -x / --one-file-system |
| Early deletion | --delete-before, --delete-during, --force |
| Other comparison and output controls | --size-only, -I / --ignore-times, --modify-window, --chmod, --log-file, -i / --itemize-changes |
| Daemon connections | rsync://, host::module |
Unsupported common flags are rejected with an explanation. Selected symlink
targets are copied unchanged. --insecure-links does not enable any of the
unsupported descendant-link options.
File-list parsing and help
--files-from cannot combine with syq ignore rules or deletion. A listed
source whose parent is a symlink fails that entry with exit 23, without
creating its implied destination parent. --insecure-links only relaxes the
ownership check on symlinks in typed local paths; it does not allow symlink
traversal beneath a selected source, including entries from --files-from.
Other parsing differences:
..components are rejected rather than clamped at the source root.dir/anddirselect the same entry. Neither selects contents unless-ris explicitly supplied;-aalone does not enable recursion here..and/are rejected as root entries.- Use
--from0for NUL separators;-0is unsupported.
Blank entries and comment-looking names starting with # or ; are ignored
in both separator modes. Use ./#name for a literal name. The paths are relative to one source directory; destination parents are
created as needed.
-h alone does not show help; use --help or --help-all.
Syq extensions
Most syq-specific options carry a --syq- prefix. Common ones are
--syq-ignore and --syq-ignore-from.
Use --integrity-checking to choose comparison and
transfer hashes in rsync mode. -c selects content comparison.
Filters are last-match-wins with ! re-inclusion, unlike rsync’s first-match
rules. Check the gitignore examples when
converting a filtered command.
The compatibility tests record comparison evidence and version-specific details.
Compare without copying
syq rsync -ancv project/ backup/
Compares contents without copying and lists planned changes.
Performance controls use --performance-tuning; resource ceilings use
--resource-limits. The standard rsync spelling --bwlimit remains available
in syq rsync, as does --checksum for BLAKE3 content comparison.
Python
Copy, remove, and reorganize files with typed results and streaming events. Synchronous and asyncio clients are available.
Install
The syq package on PyPI supports Python 3.13.4+ on Linux and macOS:
python -m pip install syq
Prebuilt wheels include the matching syq executable, so no Rust compiler or
separate syq installation is needed. The syq command is also available in
the Python environment.
Copy files
Copy a directory named data into backup, producing backup/data:
import syq
result = syq.cp("data", into="backup")
print(result.files_transferred, result.bytes_transferred)
To copy just its contents, use srcs_in:
result = syq.cp(srcs_in="data", into="backup")
Arguments follow the command-line names: replace hyphens with underscores,
and add a trailing underscore for Python keywords, such as from_ and as_.
The copy guide explains placement,
filtering, and verification options.
Copy over SSH
Use to for an SSH destination and from_ for an SSH source:
syq.cp("data", to="user@server", into="/backup")
syq.cp("report.csv", from_="user@server", cwd="/exports", into="downloads")
SSH hosts and paths are separate arguments. See
Copy between servers
for copies with two remote endpoints. A typed remote-to-remote preview needs
coordinate_at="local"; a live copy can also use an enrolled receiver.
Preview changes
Pass dry_run=True to preview a copy or removal. The result has the same type
as a live call, with counters describing the planned changes:
preview = syq.cp("data", into="backup", dry_run=True)
print(preview.files_transferred, preview.bytes_transferred)
Remove dry_run=True to apply the changes. syq checks the filesystem again
when that call runs.
Mirror a directory
Use prune=True to also remove destination entries absent from the source.
Here, staging must already exist, and max_delete limits deletions:
result = syq.cp(
srcs_in="build",
into_existing="staging",
prune=True,
max_delete=100,
)
Remove files
result = syq.rm(src_dir="old-output", root="/srv/jobs")
print(result.entries_removed, result.selectors_missing)
Positional paths and src refuse directories. Use src_dir to remove
a tree, or srcs_in to recursively empty it while keeping its root.
root confines removal to that directory. Add on="server" to remove files
over ordinary SSH. Command-restricted receivers do not support rm. See
Remove files for selector behavior.
Handle failures
A failed copy or removal raises SyqOperationError with its typed result:
try:
result = syq.cp("data", into="backup")
except syq.SyqOperationError as error:
print(error.result.status, error.result.errors)
print(error.stderr.decode(errors="replace"))
Use check=False to receive unsuccessful results without that exception.
Invalid arguments, installation failures, and incomplete or invalid results
still raise exceptions. Catch syq.SyqError to handle any SDK-defined exception,
or catch a specific subclass as above. Completed filesystem changes are not
rolled back.
Watch events and save results
on_event receives records as the operation runs. For example, show each copied
entry or planned change:
def observe(event: syq.AutomationEvent) -> None:
if isinstance(event, (syq.TraceEvent, syq.OperationResult)):
print(event.action, event.dst)
result = syq.cp("data", into="backup", on_event=observe)
Events are not collected in the returned result. To save the validated NDJSON records, pass an open binary stream:
with open("run.ndjson", "wb") as records:
result = syq.cp("data", into="backup", results=records)
Rename while copying
Create a mapping, change its destination paths, then copy:
from dataclasses import replace
with syq.map(srcs_in="photos") as mapping:
renamed = mapping.transform(
lambda entry: replace(entry, dst=syq.RelativePath("archive") / entry.dst)
)
result = syq.cp(mapping=renamed, into="published")
This places the contents of photos under published/archive. The mapping
carries its source base and symlink-following policy through the transform.
Return None from the transform to skip an entry. The entire transform must
finish successfully before copying starts; a failed transform leaves the
destination untouched. See
Rename and reorganize for mapping rules.
For a complete program built on mappings, see Pull and push DVC data.
Generated data and binary streams
Use open_writer to generate one file or object without a temporary payload
file. For example, the standard library can write an archive directly to S3:
import tarfile
import syq
with syq.open_writer(to="s3://backups", as_="dataset.tar") as output:
with tarfile.open(fileobj=output, mode="w|") as archive:
archive.add("dataset", arcname="dataset")
open_reader supplies the contents of one local file, SSH file, or S3 object:
with syq.open_reader("dataset.tar", from_="s3://backups") as source:
while chunk := source.read(65536):
consume(chunk)
The writer commits when its with block exits successfully; an exception
aborts it. The reader checks transfer success when its with block exits,
reading any bytes your code left unread. To cancel instead, call abort().
If you extract or publish files from a stream, wait for both decoding and the
reader context to finish successfully before making those files available.
See Byte streams
for wrappers, explicit completion, and async behavior.
For several generated files, put producers in a mapping. syq starts them as transfer capacity becomes available and reuses its connections:
from functools import partial
def pack(directory, output):
with tarfile.open(fileobj=output, mode="w|") as archive:
archive.add(directory, arcname=directory)
syq.cp(mapping=[syq.MappingEntry(syq.StreamSource(partial(pack, name)), f"{name}.tar")
for name in ("images", "labels")],
to="s3://backups", into="dataset", stream_concurrency=2)
A StreamDestination(consumer) receives downloaded bytes in the same way.
Archive formats belong to your callback.
See Callback mappings
for completion, retries, and async callbacks.
Use asyncio
Await operations on AsyncClient. Its arguments and results match Client:
import asyncio
import syq
async def main():
client = syq.AsyncClient()
result = await client.cp("data", into="backup")
print(result.files_transferred)
asyncio.run(main())
Async event callbacks are awaited in record order. Mapping streams use
async with and async for; there is no await before client.map():
async def copy_photos():
client = syq.AsyncClient()
async with client.map(srcs_in="photos") as mapping:
return await client.cp(mapping=mapping, into="published")
Configure a client
Share a local working directory and timeout across calls:
client = syq.Client(process_cwd="/srv/jobs", timeout=3600)
result = client.cp("data", into="backup")
process_cwd sets the local subprocess directory; typed cwd sets the source
base, which may be on a remote host. Omit timeout on a call to use the client
default; pass timeout=None to disable it for that call:
result = client.cp("data", into="backup", timeout=None)
To use an existing executable, pass Client(executable="/opt/bin/syq").
This bypasses the bundled version; see
Compatibility.
Run other commands
run accepts arguments after the executable name and returns captured bytes:
result = syq.run(["--help"])
print(result.stdout.decode())
Use it for commands without a typed method, including rsync and receiver
administration. See the
operations reference for
process options and Results and events
for exceptions.
To request a command on your receiving machine, use the syq executable you use for receiving:
import syq
result = syq.run(
["exec", "--on", "@laptop", "--cwd", "work/project", "--", "cargo", "test"],
executable="/path/to/syq",
timeout=600,
)
print(result.stdout.decode())
This captures output and raises on a nonzero exit. The receiving machine asks for approval before running the command. See Run commands on your receiving machine.
Building from source
Installing from a source distribution builds the executable and requires Rust and a C compiler. Source builds upload themselves to compatible SSH hosts by default. See source builds for compiler requirements and SSH helper selection.
Python operations
Module functions syq.cp, syq.rm, syq.map, syq.open_reader, and
syq.open_writer use a default Client.
AsyncClient has the same arguments and result types; await its operations
except map, open_reader, and open_writer, which return async context managers.
For generated data and entry transformations, see Streams and mappings. For returned objects, callbacks, and errors, see Results and events.
Client and executable selection
Client(*, executable=None, cache_dir=None, process_cwd=None, env=None, timeout=None)
and AsyncClient(...) accept:
| Argument | Meaning |
|---|---|
executable | Custom executable path, or name to find on PATH; default: bundled syq |
cache_dir | Opt into a separately downloaded executable at this cache root |
process_cwd | Local subprocess working directory; default: inherit |
env | Subprocess environment mapping; default: inherit |
timeout | Operation timeout in seconds; default: no limit |
client.version() returns the executable version as text.
syq.version(executable=None) does the same without a client.
syq.managed_executable(cache_dir=None) returns the verified executable path,
downloading it if needed. See Compatibility for version selection and caching.
Arguments and validation
Options use CLI names with hyphens replaced by underscores. Python keywords
get a trailing underscore: from_, as_. Paths accept str, bytes, or
os.PathLike; selector keywords accept one path or an iterable of paths.
Use src=["a", "b"] for CLI --srcs a b, and likewise src_non_dir and src_dir.
Shared by cp, rm, and map | Meaning |
|---|---|
*sources, src | Select named objects |
srcs_in | Select a directory’s contents |
src_non_dir, src_dir | Require non-directory objects or directories |
cwd | Source resolution base; may be remote for cp and rm |
root | Confine source resolution beneath this directory; requires relative selectors; conflicts with cwd |
follow, follow_src | Follow source symlinks; follow also enables destination following for cp |
timeout | Omitted: use client default; None: no timeout; number: timeout in seconds |
Boolean flags default to False; other optional arguments default to None,
except check=True and timeout, whose omitted value inherits the client default.
Invalid argument combinations raise SyqInvocationError;
filesystem and remote checks happen in syq.
cp
cp(*sources, **options) → CpResult copies files. Choose one placement option unless every destination is a callback.
In addition to the shared arguments above, it accepts:
| Options | Values / purpose |
|---|---|
from_, to | SSH endpoint strings or s3://BUCKET; omitted endpoints are local |
into, into_new, into_existing | Destination directory paths |
as_, as_new, as_existing | Exact destination paths |
mapping | Mapping, MapStream, manifest path, or iterable of MappingEntry; replaces selectors; conflicts with as_* and prune. Async clients also accept AsyncMapping and async iterables |
stream_concurrency | Maximum callback entries active at once; default 4, range 1..256; transport worker and request limits are shared across entries |
follow_dst | Boolean: follow destination symlinks |
prune, dry_run, hash | Boolean: mirror, preview, or compare content |
integrity_checking | Comma-separated string, e.g. "transfer=sha256"; defaults to size/mtime comparison and no extra payload checks |
only_new | Boolean: copy missing entries without replacing existing ones |
ignore | Pattern string, IgnoreFrom(path), or ordered iterable of either |
ignore_from | Rule file path or iterable of paths; applied after ignore |
preserve | Preservation string or iterable: times, permissions, ownership, specials |
inplace, no_compress | Boolean: update destination files in place or disable compression |
max_delete | Nonnegative integer deletion limit; requires prune=True |
resource_limits | Comma-separated ceilings that keep automatic tuning, e.g. "bandwidth=10M,workers=4"; a concurrency key conflicts with the same key in performance_tuning |
performance_tuning | Comma-separated overrides, e.g. "workers=4" or "s3-objects=4,s3-parts-per-object=8,s3-requests=16"; omitted means automatic |
s3_endpoint, s3_region, s3_profile | Endpoint URL, signing region, and AWS profile strings |
s3_header | Iterable of "NAME: VALUE" strings; applied before signing every request |
auth_from | Credential source string |
coordinate_at, rsh, peer_auth | Coordinator, SSH command, and peer authentication strings |
pscope | Existing ephemeral scope path for forward SSH connection reuse |
syq_path | Remote executable path |
no_bootstrap, tcp_plain, no_tcp | Boolean remote/transport controls |
tcp_ports, tcp_congestion | Port range and congestion-control strings |
receiver_max_entries, receiver_max_bytes | Receiver ceilings: integer entries, native size string or integer bytes |
receiver_receipt | "sizes" or "hashes" |
on_event, results, check | See events and failures |
Option behavior is covered in Copy files, Remote copy details, and Object storage.
For example, client.cp("data", to="s3://bucket", into="backup", s3_header=["X-Tigris-Consistent: true"]) uploads local data using credentials
from the subprocess environment or AWS configuration. Copies between two S3
endpoints use server-side copying within the same service; content verification
options that require reading object bodies are rejected. SSH/S3 combinations
are not supported. S3 results use EndpointKind.S3.
pscope selects an isolated scope for reusing SSH connections. For return
copies or commands, use syq persist connect server and omit pscope. See
persistence in scripts
for setup and cleanup, and
Compatibility
for executable selection.
Typed SSH-to-SSH copies require an enrolled receiver or
coordinate_at="local". With dry_run=True, they require
coordinate_at="local". Use run for detached commands and human output options.
IgnoreFrom(path) is a frozen dataclass holding a rule-file path (str,
bytes, or os.PathLike). To interleave rule files and inline patterns:
ignore=[syq.IgnoreFrom("rules"), "!keep.tmp"]. The last matching rule wins.
rm
Positional sources and src refuse directories. src_dir removes a
tree recursively; srcs_in removes its contents recursively and keeps the root.
on="server" selects the removal endpoint. A final selected symlink is
always unlinked. Both follow_src=True and follow=True permit symlinks in
cwd, root, and selector parent directories.
Directory and contents selectors reject a final symlink even with following
enabled.
rm(*sources, **options) → RmResult removes selected entries. Besides the shared
arguments, it accepts on, dry_run, performance_tuning, syq_path,
no_bootstrap, pscope, on_event, results, and check with the types above.
It supports local, ordinary SSH, and S3 endpoints. Command-restricted receivers
reject removal. See Remove files.
S3 removal uses rm(..., on="s3://bucket") with optional s3_endpoint,
s3_region, s3_profile, s3_header, and auth_from="@NAME". s3_all_versions=True permanently
removes all selected versions and delete markers; s3_version_id="ID" selects
one version of one exact key. These options are mutually exclusive.
RemovalTrace and RemovalResult expose optional s3_version_id and
s3_delete_marker fields. The same arguments work with AsyncClient.rm.
run
client.run(args, *, check=True, cwd=None, env=None, timeout=CLIENT_DEFAULT, input=None)
returns Result. input accepts bytes. args is a sequence of arguments
after the executable name, passed without a shell.
Here, cwd is the local subprocess directory. cwd and env use client
defaults when omitted or None. timeout uses the client default only when
omitted; explicit None disables it. Module-level syq.run takes the same
arguments plus executable=None and defaults to no timeout.
Wrappers can forward syq.CLIENT_DEFAULT to preserve client inheritance.
syq.Timeout is the type alias for a number, None, or that sentinel:
def copy_data(client: syq.Client, *, timeout: syq.Timeout = syq.CLIENT_DEFAULT):
return client.cp("data", into="backup", timeout=timeout)
CLIENT_DEFAULT applies to client methods (including module-level cp, rm,
and map); client constructors and module-level run have no client default
to inherit.
Timeouts cover subprocess execution. Managed installation and mapping-input materialization happen before the copy process starts and are not covered by its timeout. Async cancellation still stops mapping-input preparation.
For syq exec, pass --cwd in the argument list to select the receiving
working directory. The SDK’s cwd= parameter selects the local working
directory of the requesting process. See the
command example.
Result
Frozen dataclass returned by run() and held in SyqProcessError.result:
| Attribute | Type | Meaning |
|---|---|---|
argv | tuple[str | bytes, ...] | Executed argument list, including the executable |
returncode | int | Process exit status |
stdout | bytes | Complete captured standard output |
stderr | bytes | Complete captured standard error |
Compatibility
Python 3.13.4+ on Linux and macOS; no runtime Python dependencies. Each Python
package uses the matching syq release. syq.__version__ and
syq.PINNED_SYQ_VERSION report those versions. Pin the package in your dependency
file to keep the pairing.
Bundled executable
By default, the SDK runs the executable installed with its Python wheel. It
locates that executable through the package’s installation record, without
searching PATH, downloading files, or creating an executable cache. Each
Python environment has its own installation. Removing the package also removes
its executable.
Managed executable
For callers using a separate cache, Client(cache_dir=...) downloads the matching executable on first use and verifies
it against the package’s embedded release manifest. It checks the cached binary
before every use and replaces missing or corrupt entries. It does not search
PATH. syq.managed_executable() uses the same download and verification checks.
The default cache is $XDG_CACHE_HOME/syq/sdk/python/v<version>/ when
XDG_CACHE_HOME is absolute, or ~/.cache/syq/sdk/python/v<version>/ otherwise.
Use Client(cache_dir=...) to change the cache root, or
syq.managed_executable() to download ahead of time and get the path.
Custom executable
Client(executable="/opt/bin/syq") uses that binary;
Client(executable="syq") searches PATH. Overrides bypass bundled selection and managed download
and verification, so you are responsible for compatibility and origin. Typed
calls still validate automation output. A failed executable selection does not
fall back to another binary.
Python streams and mappings
Use byte streams for generated data or incremental reads, and mappings to choose each entry’s source and destination. See the Python guide for examples and the operations reference for client configuration and shared arguments. Results and exceptions apply across these interfaces.
Byte streams
client.open_writer(*, as_=None, as_new=None, as_existing=None, to=None, follow_dst=False, ...) returns a StreamWriter. Choose exactly one of
as_, as_new, or as_existing; the latter two require the destination to
be absent or present, following cp placement semantics. Writers have no
source basename, so they require an exact destination path.
client.open_reader(src, *, from_=None, cwd=None, root=None, follow_src=False, ...)
returns a StreamReader. cwd resolves relative sources; root also confines
them. Choose at most one, as with cp. These bases belong to the source
endpoint, independently of the client’s local process_cwd. Both accept rsh,
syq_path, pscope, no_bootstrap, no_compress, no_tcp, tcp_plain,
tcp_ports, tcp_congestion, auth_from (S3), s3_endpoint, s3_region, s3_profile, s3_header,
performance_tuning, resource_limits, integrity_checking, only_new,
dry_run, stats, verbose, quiet, progress, no_progress,
and timeout with the same meanings as cp.
See the CLI stream reference for the applicable tuning and integrity controls.
The client supplies the executable, process working directory, environment,
and default timeout. Stream calls always check transfer failures.
These methods transfer raw bytes, using the CLI’s stream semantics for destination permissions, metadata, and endpoint restrictions. They are sequential, non-seekable interfaces; use one operation at a time on each stream. No whole-object retry or restart recovery is attempted. S3 can retry buffered multipart parts.
| Object | Operations |
|---|---|
StreamWriter | write(bytes) writes the complete buffer and returns its length; flush() has no Python buffer to flush; close() ends payload input; commit() publishes and checks completion; abort() cancels |
StreamReader | read(size=-1), readinto(buffer); close() drains remaining bytes in bounded chunks and checks completion; abort() cancels |
Streams use bounded transport buffers. read() without a size collects all
remaining bytes in Python memory and checks completion before returning.
Writer completion
Use a with block to commit on successful exit or abort on an exception.
An abort before commit does not replace the destination. close() only ends
payload input; it does not publish the file. Successful context exit commits
even if you already called close(), unless you also called abort().
Writers support io.BufferedWriter and io.TextIOWrapper. Close the wrapper
before committing the underlying writer so its buffered data is flushed.
Outside a context, finish with commit() or abort(); close() alone leaves
the transfer pending. commit() can follow payload closure, is repeatable
after success, and fails after an abort. Explicit commit publishes immediately,
so later application errors cannot undo it.
A timeout or connection loss during commit can leave the outcome uncertain;
the method reports failure rather than claiming rollback. Whole datasets
need their own final publication step after all object transfers succeed.
Skips and previews
Writers normally return while destination setup continues, so opening several
writers lets their connections start concurrently. Setup errors can surface at
write() or commit(). With only_new=True or dry_run=True,
opening waits for the destination decision (also for options inherited from the
environment). Check output.skipped before producing data:
with client.open_writer(to="server", as_="archive.tar", only_new=True) as output:
if not output.skipped:
produce_archive(output)
A skipped writer rejects write() and exits its context successfully without
committing anything. Skipped readers return no payload; their skipped property
is settled at completion. Both count the skipped object in files_excluded.
With dry_run=True, a context checks placement without transferring bytes.
Enter and exit a writer context without calling write(); preview writers
reject payload writes. Preview readers return no payload. The result contains
planned totals; bytes_total_known=False distinguishes an unknown pipe length
from an empty source. A dry run does not check an expected payload hash.
Results and failures
After completion, stream.result holds a CpResult, including byte counts and
elapsed time. It remains None if no terminal result arrived, such as after
forced termination. Missing or invalid completion records raise
SyqProtocolError even if the process exits successfully.
A bounded reader call can yield partial data before a later transfer error.
An unbounded read() checks transfer completion before returning its bytes.
An exception inside either context cancels the transfer
and preserves the original exception. Writers that have not committed are
aborted during garbage collection, including after payload closure; cleanup
can block, so use explicit contexts for timely cleanup.
timeout covers the stream’s lifetime, including blocked reads/writes. Timeout
raises subprocess.TimeoutExpired unless the transfer has completed successfully;
transfer failures raise SyqProcessError,
whose result contains the exit status and the last 8 KiB of diagnostics, without
capturing payload bytes. The stream’s stderr property exposes those same
last 8 KiB as bytes, including on success. For example, request stats=True
and read output.stderr.decode() after the writer context exits.
Async streams
Async streams use async with directly and await I/O and explicit closure:
client = syq.AsyncClient()
async with client.open_writer(to="server", as_="generated.bin") as output:
await output.write(b"generated bytes")
async with client.open_reader("generated.bin", from_="server") as source:
while chunk := await source.read(65536):
await consume(chunk)
Async streams start on context entry. Cancellation terminates the owned
transfer process and releases blocked I/O. AsyncStreamReader and
AsyncStreamWriter expose async read/write, close, and abort methods.
AsyncStreamWriter.commit() explicitly publishes, with the same semantics
as its synchronous counterpart; successful async context exit commits automatically.
map
map(*sources, **options) → MapStream lists local mapping entries without
copying. Besides the shared arguments, it accepts as_ to rename a selected
object. srcs_in must be the sole selector when used.
MapStream is a Mapping and an iterable context manager; use with.
AsyncMapStream is an AsyncMapping and an async context manager; use
async with. Streams must be consumed inside their context and cannot be reused
after completion or closure. Async streams capture the producer directory,
environment, and timeout when created, even though execution starts later.
Mapping and AsyncMapping
A mapping combines entries with their local source context. Pass it directly
to cp(mapping=...), even on a client with a different process_cwd.
Mapping(entries, *, cwd=None, root=None, follow_src=False) accepts an iterable
of MappingEntry. AsyncMapping(...) accepts an async iterable. If both cwd
and root are omitted, the source base is the current directory at construction.
Otherwise exactly one may be supplied. Relative cwd and root paths resolve
against the Python process directory at construction, not a client’s
process_cwd. root confines source resolution.
| Member | Type | Meaning |
|---|---|---|
cwd | pathlib.Path | Absolute source-base spelling, preserving symlinks and .. |
root | pathlib.Path or None | Confinement base, when created with root |
follow_src | bool | Whether the copy should follow source symlinks |
transform(function) | Mapping or AsyncMapping | Lazy transformation that keeps the source context |
The context properties are read-only. A transform receives each MappingEntry
and returns a replacement entry, or None to omit it. Transformations can be
chained. Async transforms also accept awaitable callbacks and await them in
entry order. Neither form caches entries; reuse depends on the supplied iterable.
A context-carrying mapping rejects from_, cwd, and root overrides on cp.
It automatically enables the source-following policy used by map; enabling
follow or follow_src on the copy is also permitted. The destination options
remain independent. map(root=..., srcs_in=...) carries the selected directory
as the consuming copy’s root. The copy resolves that root again; it does not
inherit an open directory handle or a snapshot of the source tree.
Normal end of stream iteration checks the mapping process status. Leaving its context early stops the process. An exhausted or closed stream cannot be copied as an empty mapping.
For cp(mapping=iterable), the entire iterable is saved to a temporary manifest
before copying starts. An iteration, transformation, or serialization failure
starts no copy. Plain iterables and manifest paths have no source context; pass
cwd, root, or from_ explicitly as needed. See
mapping rules.
Callback mappings
StreamSource(produce, *, size=None) supplies bytes to one mapping entry.
StreamDestination(consume) receives them. Pass either or both as the src
and dst of a MappingEntry; callback entries require kind=None or "file".
These objects work with live cp(mapping=...) calls, not saved pathname manifests.
The complete iterable is validated before any callback or destination change.
Callbacks start lazily, with at most stream_concurrency entries active; one
entry can have both a producer and consumer. Pathname entries run first with
their usual comparison and recovery behavior, then callback entries share
transfer sessions. A mixed mapping therefore has one endpoint setup per phase.
A failed entry does not undo entries that already succeeded.
Producers receive a writable binary file object. Returning normally authorizes
publication; closing the object only ends the bytes. An exception, including
one raised after a wrapper closes the writer, cancels unpublished work. Consumers
receive a readable binary object. Reading through EOF checks transfer success;
a normal early return drains and checks the rest. Consumer exceptions cancel
remaining work. Write extracted files into a staging directory and publish it
after cp succeeds. syq cannot undo a consumer’s own side effects.
Client runs callbacks on worker threads. AsyncClient accepts the same
synchronous callbacks, or async def callbacks that await read, write, and
close; async callbacks run on the caller’s event loop. Cancellation releases
blocked payload I/O. Application code that keeps computing without I/O must
cooperate with cancellation. Do not retain the supplied file object after return.
StreamSource.size promises the exact byte count; a mismatch fails before
publication. MappingEntry.size remains informational. expected_hash checks
the bytes during transfer on every backend; it adds no second download. Without
an expectation or requested integrity check, no extra whole-stream hash is made.
metadata sets attributes on named destinations without preserve. Source
callbacks have no file metadata to preserve. S3 stores explicit attributes in
syq’s existing object metadata format; omitted attributes use new-file defaults.
Skip policies and dry runs do not invoke excluded callbacks. Callbacks always
transfer when selected; they have no saved content identity for comparison or
restart recovery. They cannot use hash, content comparison, inplace, pruning,
or source-tree ignore rules. Filter entries in your script. Named receiver
grants authorize paths and do not accept callbacks. Callbacks run locally;
mixed pathname transfers between SSH hosts require coordinate_at="local".
Callback runs emit MappingStreamResult events with the original zero-based
entry index, source and destination (PathValue, or None for a callback),
disposition, dry_run, optional bytes, and optional error message.
After a completed run, use failed entry indices to construct an application
retry. Callbacks are never replayed automatically. Their automation records use
schema version 3; ordinary pathname and descriptor calls retain version 2.
Hash and HashAlgorithm
Hash(algorithm, value) describes the expected hash of all bytes in one
regular file. algorithm accepts a HashAlgorithm value or its string:
"blake3", "sha256", "md5", or "xxh3-128". value is hexadecimal:
64 digits for BLAKE3 and SHA-256, 32 for MD5 and XXH3-128. The immutable object
validates the length and characters and stores lowercase hex.
The expectation covers the complete resulting file, including reused bytes.
A mismatch fails the file rather than reporting a successful copy. Files excluded
by selection rules are not hash-verified. hash=True still controls whether
existing contents are compared instead of trusting size and modification time.
Dry runs preview changes without validating the expectation. An expected
whole-file hash is independent of the algorithm used for block comparison or
transport checks. MD5 and XXH3-128 are useful for compatibility
and accidental-error detection, but do not provide cryptographic collision
resistance.
MappingEntry
Frozen dataclass describing one source-to-destination mapping. Pass an iterable
of these to cp(mapping=...); use dataclasses.replace to change an entry.
| Attribute | Type | Meaning |
|---|---|---|
src | RelativePath or StreamSource | Path relative to the source base, or a producer callback |
dst | RelativePath or StreamDestination | Path relative to the destination container, or a consumer callback |
kind | EntryKind or None | Object kind, when known; default None |
size | int or None | Informational size in bytes; default None |
mtime | int or None | Informational modification time in Unix seconds; default None |
expected_hash | Hash or None | Expected whole-file hash; requires a regular file; default None |
metadata | DestinationMetadata or None | Explicit destination attributes; default None |
MappingEntry(src, dst, kind=None, size=None, mtime=None, expected_hash=None, metadata=None) also accepts text or
byte paths for src and dst and converts them to RelativePath. size and
mtime do not impose preconditions on the copy. expected_hash does: a file
cannot succeed unless its contents match. For example, an adapter can supply
an MD5 from a DVC manifest without changing syq’s ordinary comparison algorithm:
entry = syq.MappingEntry(
"cache/object", "data.bin", kind="file",
expected_hash=syq.Hash("md5", "900150983cd24fb0d6963f7d28e17f72"),
)
client.cp(mapping=[entry], cwd="source", into="download")
Mapping files encode it as "expected_hash": {"algorithm": "md5", "value": "..."}.
Older syq versions that do not support this field reject the mapping.
DestinationMetadata
Set destination attributes without modifying the source:
entry = syq.MappingEntry(
"source.bin", "payload.bin",
metadata=syq.DestinationMetadata(mode=0o640, mtime=1700000000),
)
client.cp(mapping=[entry], cwd="source", into="output")
The keyword-only fields are mode, uid, gid, mtime, and mtime_nsec,
all optional integers. mode contains permission bits only; uid and gid
are numeric IDs. Times use Unix seconds plus optional nanoseconds. Supplying
mtime without mtime_nsec uses zero nanoseconds. Omitted attributes follow
normal copy behavior. Explicit attributes do not require preserve, except
where a restricted receiver’s signed grant needs the corresponding permission.
The mapping reference describes backend behavior.
RelativePath and PathValue
RelativePath(value) accepts text, bytes, or os.PathLike. It rejects empty
paths, absolute paths, NUL bytes, and empty, . or .. components. Join paths
with /, for example syq.RelativePath("archive") / entry.dst.
PathValue(raw: bytes) holds a path received in an event, which may be absolute.
Both types are immutable and provide:
| Member | Type | Meaning |
|---|---|---|
raw | bytes | Original filename bytes; also returned by bytes(path) |
text | str | UTF-8 decoding; raises UnicodeDecodeError for invalid UTF-8 |
str(path) | str | Filesystem decoding with os.fsdecode |
PathValue.display | str | Same as str(path) |
RelativePath also implements os.PathLike, returning bytes.
Python results and events
Typed operations return results and can deliver events as work progresses. Use this reference to interpret those objects and handle failures. See the operations reference for calling conventions, or streams and mappings for their completion rules.
Events and terminal results
cp and rm return frozen dataclasses after validating the complete results
stream and process exit status. A truncated stream raises even if the process
exits successfully. Dry runs return the same types, with planned totals.
CpResult
Returned by cp(), or available as SyqOperationError.result after an
unsuccessful copy. Read attributes directly, for example
result.files_transferred. It includes the common result fields below plus:
| Attribute | Type | Meaning |
|---|---|---|
files_transferred | int | Regular files transferred |
files_unchanged | int | Regular files skipped as unchanged |
files_excluded | int | Files excluded from copying |
directories_created | int | Directories created |
symlinks_created | int | Symbolic links created |
specials_created | int | Special filesystem objects created |
bytes_transferred | int | File-content bytes transferred, not compressed network traffic |
bytes_total_known | bool | None | For descriptor copies, whether the source length is known; otherwise absent (None) |
bytes_unchanged | int | Bytes in unchanged files |
deletions_planned | int or None | Entries selected for pruning |
deletions_completed | int or None | Entries pruned |
deletions_blocked | int or None | Pruning deletions blocked by a safety limit |
receipt | ReceiptSummary or None | Verified receiver receipt details; None for ordinary copies |
With dry_run=True, mutation totals describe planned changes.
A failed call reports work completed before it stopped.
Ordinary copies have all three deletion fields only with prune=True;
otherwise they are None. Receiver-attested results have only
deletions_completed, and their unchanged/excluded totals are always zero
because the receiver cannot observe source-side skips. receipt is
None for ordinary copies.
RmResult
Returned by rm(), or available as SyqOperationError.result after an
unsuccessful removal. It includes the common result fields below plus:
| Attribute | Type | Meaning |
|---|---|---|
selectors_total | int | Explicit source selectors supplied |
selectors_resolved | int | Selectors that resolved to an object |
selectors_missing | int | Selectors already missing; this is not an error |
entries_planned | int | Entries a dry run would remove; zero in live runs |
entries_removed | int | Entries removed; zero in dry runs |
entries_already_absent | int | Entries gone by removal time; zero in dry runs |
entries_failed | int | Removal or inspection failures, including during dry runs |
mode | str | Always "rm" |
Selectors identify requests; entries count individual filesystem objects. One directory selector can account for many entries. Duplicate and overlapping selectors have separate indexes.
Common result fields
Both CpResult and RmResult include:
| Attribute | Type | Meaning |
|---|---|---|
status | OperationStatus | Outcome from the table below |
exit_code | int | syq process exit code |
dry_run | bool | Whether this was a preview |
errors | int | Counted errors |
elapsed_ms | int | Run duration in milliseconds |
protocol | ProtocolMetadata | Automation envelope; also present on every event |
ProtocolMetadata
Frozen dataclass accessed through result.protocol or event.protocol.
The SDK checks these fields; applications normally only need them when recording
or diagnosing a stream.
| Attribute | Type | Meaning |
|---|---|---|
schema | str | "syq.automation" |
schema_version | int | 2 |
seq | int | Record sequence number, starting at zero |
type | str | Wire record type; "result" for terminal totals |
ReceiptSummary
Frozen dataclass accessed through CpResult.receipt for receiver-attested copies.
| Attribute | Type | Meaning |
|---|---|---|
status | ReceiptStatus | Receiver receipt outcome |
operations | int | Attested operation record count |
final_states | int | Attested final-state record count |
records | int | Total receipt record count |
provenance | str | "receiver_attested" |
OperationStatus
String enum: compare with syq.OperationStatus.SUCCESS, or use .value for
"success".
| Member | Value | Exit code | Meaning |
|---|---|---|---|
SUCCESS | "success" | 0 | Requested operation succeeded |
PARTIAL | "partial" | 23 | Per-entry failures; independent work finished |
REFUSED | "refused" | 25 | A safety cap refused deletions; copy only |
ABORTED | "aborted" | 1 | Operation aborted; copy only |
FAILED | "failed" | 1 | Fatal failure |
Event callbacks
on_event(event) receives an AutomationEvent in stream order. Events are not
stored in the result. AsyncClient accepts synchronous or awaitable callbacks;
awaitable callbacks count toward the timeout.
AutomationEvent is the union of the event classes below and CpResult and
RmResult. Each event is a frozen dataclass. Its fields are listed below;
all events also carry protocol: ProtocolMetadata, just like results.
event.protocol.type identifies the wire record type. Optional fields use None when unavailable.
See Automation results for stream
semantics. Python exposes class as class_ and paths as PathValue.
results= accepts a binary file-like object with write(bytes) returning a
positive byte count for nonempty writes, and optional flush(). The client
copies validated NDJSON records, flushes after completion, and leaves it open.
It withholds the terminal record if stream validation, process completion, or
a callback fails. Sink failures raise and abort the operation.
OperationResult.is_retryable identifies retryable failures; retry_entry()
preserves expected_hash and metadata and returns a MappingEntry when a complete mapping
identity is available, otherwise None. Only use collected entries after the call returns a validated success
or partial result. A terminal callback alone does not establish completion.
The client does not retry automatically.
Event types
RunEvent
Invocation details. started_at is Unix seconds; mode is "cp" or "rm".
prune and mapping are None for removal.
protocol.type = "run". Fields in addition to the common envelope:
run_id: str
started_at: int
syq_version: str
mode: str
prune: bool | None
mapping: bool | None
dry_run: bool
endpoints: tuple[Endpoint, ...]
ProgressEvent
Sampled progress for displays; use the terminal result for final totals.
Byte fields measure file content,
scanned counts scanned entries, and elapsed_ms is milliseconds. Optional
activity contains diagnostic measurements
when the producer collects them; otherwise it is None.
Optional rate_bytes_per_second and eta_ms provide rate and remaining-time
estimates; use final byte counts and elapsed time for completed-run measurements.
protocol.type = "progress". Fields in addition to the common envelope:
bytes_done: int
bytes_total: int
bytes_unchanged: int
files_done: int
files_total: int
files_unchanged: int
files_excluded: int
scanned: int
scan_done: bool
elapsed_ms: int
activity: dict[str, Any] | None
rate_bytes_per_second: int | None
eta_ms: int | None
TraceEvent
One planned copy change. dst is relative to the destination container;
src is the mapping source when available. bytes is the planned file size,
and reason describes why the change is needed.
protocol.type = "trace". Fields in addition to the common envelope:
action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind
bytes: int | None
reason: TraceReason
OperationResult
One copy outcome. dst is destination-relative, or relative to the signed
destination scope for a receiver receipt. src is present for mapping entries
when available. bytes and attempts give transfer information. Error details
are present when known; message is display text. provenance, scope, and
code apply to receiver-attested outcomes.
protocol.type = "operation_result". Fields in addition to the common envelope:
action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind | None
disposition: Disposition
bytes: int | None
attempts: int | None
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None
expected_hash: Hash | None
metadata: DestinationMetadata | None
provenance: str | None
scope: int | None
code: ReceiptCode | None
SelectionResult
One removal selector, indexed from zero. path is the original selector;
status says whether it resolved. kind is None for a missing selector.
protocol.type = "selection_result". Fields in addition to the common envelope:
selector: int
path: PathValue
status: SelectionStatus
kind: EntryKind | None
RemovalTrace
One entry a preview would remove. selector identifies its source selector;
path identifies the entry. disposition is always WOULD_REMOVE.
protocol.type = "removal_trace". Fields in addition to the common envelope:
selector: int
path: PathValue
kind: EntryKind
disposition: RemovalDisposition
s3_version_id: str | None
s3_delete_marker: bool | None
RemovalResult
One removal outcome or preview inspection failure. selector identifies its
source selector; path identifies the entry. attempts counts attempts;
retryable, class_, os_kind, and message describe failures when available.
protocol.type = "removal_result". Fields in addition to the common envelope:
selector: int
path: PathValue
kind: EntryKind | None
disposition: RemovalDisposition
attempts: int
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None
s3_version_id: str | None
s3_delete_marker: bool | None
ErrorEvent
One counted error. message is display text; class_ and os_kind classify
it when known. Receiver errors can also carry provenance and code.
protocol.type = "error". Fields in addition to the common envelope:
message: str
class_: ErrorClass | None
os_kind: OsKind | None
provenance: str | None
code: ReceiptCode | None
FinalStateEvent
Receiver-attested destination state. dst is relative to the signed scope.
provenance is "receiver_attested". Present objects have kind and byte
size; metadata, hash, and symlink_target are present where available.
observation_error describes a partial observation. Absent objects have no
object details; failed observations have code and optional message.
protocol.type = "final_state". Fields in addition to the common envelope:
provenance: str
scope: int
dst: PathValue
state: FinalObjectState
kind: FinalObjectKind | None
size: int | None
metadata: ObjectMetadata | None
hash: AttestedHash | None
symlink_target: PathValue | None
observation_error: str | None
code: ReceiptCode | None
message: str | None
Endpoint, ObjectMetadata, and AttestedHash
Nested frozen dataclasses used by events:
| Type | Fields | Meaning |
|---|---|---|
Endpoint | role: EndpointRole, kind: EndpointKind, host: str | None, user: str | None | Source/destination and local/SSH/S3 identity; host and user are optional |
ObjectMetadata | mode: int, uid: int, gid: int, mtime: int, mtime_nsec: int, rdev: int | Unix mode, owner/group IDs, modification time (seconds plus nanoseconds), and device ID |
AttestedHash | algorithm: str, value: str | "blake3" and its 64 lowercase hexadecimal hash characters |
Enums
All enums are string enums exported by syq. Members use uppercase names;
values use lowercase, for example EntryKind.FILE.value == "file".
OperationStatus is defined with the result types above.
| Type | Members |
|---|---|
EntryKind | FILE, DIR, SYMLINK, SPECIAL |
EndpointKind | LOCAL, SSH, S3 |
EndpointRole | SOURCE, DESTINATION |
OperationAction | TRANSFER_FILE, CREATE_DIRECTORY, CREATE_SYMLINK, CREATE_SPECIAL, DELETE, SET_METADATA, OBSERVE_HASH |
Disposition | SUCCEEDED, FAILED, BLOCKED, INCOMPLETE, OBSERVED |
ReceiptCode | NONE, EXECUTION_FAILED, AUTHORIZATION_REFUSED, FILE_LIFECYCLE_INCOMPLETE, OBSERVATION_FAILED |
FinalObjectState | PRESENT, ABSENT, OBSERVATION_FAILED |
FinalObjectKind | DIR, FILE, SYMLINK, FIFO, SOCKET, CHARACTER_DEVICE, BLOCK_DEVICE, OTHER |
ReceiptStatus | CLEAN, FAILED, INCOMPLETE |
Retryability | YES, NO, UNKNOWN |
ErrorClass | IO, TRANSPORT, CONFLICT, INTEGRITY, SAFETY_LIMIT, USAGE, INTERNAL |
OsKind | NOT_FOUND, PERMISSION_DENIED, ALREADY_EXISTS, INVALID_INPUT, NO_SPACE, QUOTA_EXCEEDED, READ_ONLY, OTHER |
TraceReason | DESTINATION_MISSING, TYPE_DIFFERS, CONTENT_DIFFERS, METADATA_DIFFERS, DESTINATION_ONLY |
SelectionStatus | RESOLVED, MISSING |
RemovalDisposition | WOULD_REMOVE, REMOVED, ALREADY_ABSENT, FAILED |
ReceiptStatus.CLEAN means a clean receipt, FAILED reports receiver failures,
and INCOMPLETE reports an incomplete lifecycle. Retryability.YES, NO, and
UNKNOWN state whether a failed entry can be retried.
Failure model
SyqError is the base class for every SDK-defined exception below. Specific
subclasses retain their details so callers can distinguish an unsuccessful
operation from a broken results stream.
| Exception | Meaning / useful attributes |
|---|---|
SyqInstallError | Bundled executable is missing, or managed installation or verification failed |
SyqInvocationError | Invalid Python arguments |
SyqOperationError | Typed operation was unsuccessful; .result and .stderr (last 8 KiB) |
SyqProtocolError | Invalid, unsupported, inconsistent, or incomplete results; .returncode, .stderr |
SyqProcessError | Raw run exited nonzero; .result contains complete output |
SyqOutputError | A helper such as version() received unexpected output |
For cp and rm, check=False returns unsuccessful typed results; for run,
it returns nonzero process results. It does not suppress other errors.
Spawn failures, timeouts, and ordinary Python type/value errors use standard
Python exceptions. Exceptions from application callbacks or mapping iterators
are re-raised unchanged. Async cancellation remains asyncio.CancelledError.
These exceptions are not wrapped in SyqError.
Timeout, cancellation, early mapping exit, and streaming failures stop the local process group, including SSH children. Filesystem changes already completed are not rolled back.
Pull and push DVC data
dvc_syq.py
pulls and pushes DVC data with syq. It plans the copies and passes them to syq
as a batch. Use it to try syq’s transfers with your existing DVC repository.
It is a short program written with the Python SDK, meant to be
used, read, and adapted. DVC keeps working alongside it: both use the same
cache and the same remote, so dvc status, dvc push, and dvc pull treat
the script’s results as their own.
Run it
Install it as a dvc-syq command with uv:
uv tool install 'git+https://github.com/greaber/syq#subdirectory=examples/dvc-syq'
Then, inside a DVC repository:
# Pull one tracked path, or the .dvc file that describes it.
dvc-syq pull models/speech.dvc
# Pull everything tracked under a directory; "-R ." covers the repository.
dvc-syq pull -R datasets
# Download into DVC's cache without touching the workspace.
dvc-syq fetch -R datasets
# Upload what the remote does not have yet.
dvc-syq push models/speech.dvc
uv tool upgrade dvc-syq fetches the current version. The script is also a
single file that declares its own dependencies, so you can download it and
run it without installing anything: uv run dvc_syq.py pull -R datasets.
These options have the same meaning as in DVC:
| Option | Meaning |
|---|---|
-R, --recursive | Include every .dvc file under a directory target |
-r NAME, --remote NAME | Use this DVC remote instead of the default one |
-f, --force | Let pull replace files you have changed and remove untracked files from tracked directories. Without it, pull stops and lists them |
And two are the script’s own:
| Option | Meaning |
|---|---|
--verify | Check downloaded files against their DVC MD5 and reject mismatches |
--dry-run | Show what would be copied |
How it works
DVC’s current cache layout names objects after their MD5. The same relative path identifies an object in the cache and remote; a separate checkout step gives workspace files their original names.
def object_path(md5):
return f"files/md5/{md5[:2]}/{md5[2:]}"
# Remote to cache: the same path on both sides.
downloads = [MappingEntry(src=object_path(md5), dst=object_path(md5)) for md5 in missing_from_cache]
client.cp(mapping=downloads, from_="s3://my-bucket", into=".dvc/cache", only_new=True)
# Cache to workspace: each object gets the name recorded in its .dvc file.
checkout = [MappingEntry(src=object_path(md5), dst=path) for path, md5 in tracked_files]
client.cp(mapping=checkout, cwd=".dvc/cache", into=".")
Each list is a mapping: pairs of source and destination paths
that syq copies in one run. A push is the first copy in reverse, and
only_new makes it skip objects the remote already has. With --verify,
download mappings carry the expected MD5 so syq can check the completed file
before adding it to the cache. Older DVC text objects can use a newline-normalized
MD5; the script checks those after download and removes mismatches. Files already
in the cache are skipped.
Differences from DVC
- A target is required. DVC pulls or pushes the whole repository when you
name none; here that is
-R .. - Remotes can be local directories,
ssh://host/pathwithout an explicit port, ors3://bucket/prefix. For S3 the script readsprofileandendpointurlfrom the DVC remote and otherwise uses your usual AWS credentials. Remotes that use DVC’s cloud versioning are refused. - Data brought in with
dvc importis skipped with a message, because it lives in another repository’s remote. Usedvc pullfor it. - DVC options not listed above, such as
--all-branches, are not available.