These docs describe unreleased changes on master. Read the latest stable documentation.

syq Fast, programmable file transfer.

Copy, reorganize, and remove files across local filesystems, remote machines, and S3-compatible storage. Resume interrupted transfers, send files to your laptop or run commands there from a server’s remote shell, and script syq using the JSON API or Python SDK. Your laptop needs no SSH server or incoming network port.

Try a copy

Replace server with an SSH hostname or alias you normally connect to, and project with a local directory:

syq cp project --to server --into backup

This creates or updates backup/project in your home directory on the server. To copy the contents of project directly into backup, use --srcs-in:

syq cp --srcs-in project --to server --into backup

Use --dry-run to preview a summary without copying, or --dry-run -v to list the planned changes by path. Existing destination files are updated when needed. Unrelated files stay unless you request --prune.

Already use rsync?

Start with your usual command, prefixed by syq:

syq rsync -av project/ server:backup/project/
syq rsync -av server:data/ ./data/

Syq supports common rsync options, but uses its own protocol. Rsync filter rules, hard links, ACLs, xattrs, sparse files, and rolling-checksum deltas are not supported. Check rsync compatibility before substituting it in an existing script.

Common tasks

Install

Syq runs on Linux and macOS, on x86-64 and ARM64.

Standalone installer

curl --proto '=https' --tlsv1.2 -LsSf https://dl.syq.christmas/latest/install.sh | sh

Installs into ~/.local/bin without sudo. Make sure that directory is on your PATH. To choose another directory, download the script and run sh install.sh --bin-dir DIR.

Automatic installation on SSH servers

When an official syq release installs its helper on an SSH server, it also tries to install the same version at ~/.local/bin/syq for use on that server. Existing files and symlinks there are left alone; shell startup files are never edited. Syq reports installation or failure unless --quiet is set. Shell completion and background connections can also trigger installation, without printing a notice. Failure to install this command does not stop the transfer.

Use syq --self-update on the server to update this command. To reinstall a removed command, run the standalone installer above on the server.

Reusing a cached helper does not repeat this installation step. Development builds and connections using --syq-path or --no-bootstrap do not install the command.

Homebrew

brew install greaber/tap/syq

Build from source

See source builds for Cargo builds, custom compilation options, and choosing between your own executable and compatible official SSH helpers.

Try a benchmark

Compare syq with rsync on your own machines, or with rsync and cp locally:

curl --proto '=https' --tlsv1.2 -fLsS https://raw.githubusercontent.com/greaber/syq/master/scripts/try-benchmark.sh | bash

Choose an SSH host to compare syq with rsync, or a local copy to include cp. The script creates test data, checks the copied contents, and cleans up afterward. If syq is missing, it offers to install it. See quick comparison for workload sizes, warm-up time, and command-line options.

Updates

Use syq --self-update for a standalone installation, or brew upgrade syq for Homebrew.

Standalone and Homebrew installs may print an update reminder in a terminal, at most once a day after a successful command, naming the upgrade command for that install. Nothing updates automatically. Set SYQ_NO_UPDATE_CHECK=1 or DO_NOT_TRACK=1 to disable reminders.

Downloads and the daily reminder check go through dl.syq.christmas, a host run by the maintainer that serves the GitHub release files from a cache. It records each request’s time, syq version, platform, the connection’s IP address, and the country, region, and city derived from that address, so the project can see how many installs exist and which versions are in use. Nothing identifies an install, and the check sends nothing else. Non-interactive use never makes the reminder check. Every download is verified against the signed release manifest, so the host cannot substitute files.

Shell completion

See the completion command reference for every command and cache-management option.

Add the line for your shell to its startup file:

# Bash (~/.bashrc)
eval "$(syq completion bash)"

# Zsh (~/.zshrc), after autoload -Uz compinit && compinit
source <(syq completion zsh)

# fish (~/.config/fish/config.fish)
syq completion fish | source

Completion suggests options, hosts, and paths, with file details beside path matches. In Bash, press Tab again to list matches. Remote paths use your usual SSH login. Open a new shell after adding the setup line or upgrading syq.

Keep connections open

Keep an SSH connection ready for repeated copies:

syq persist connect server

This enables persistence and connects without copying files. It also lets you send files back from the server, with approval on your machine. Connections stay open until you close them with syq persist off. Use syq persist status to see them.

See background connections for reconnecting, turning receiving off, and using persistence in scripts.

Copy files

syq cp project --into backup

This copies project to backup/project. Existing files are updated when needed; unrelated files stay. See syq cp for the option list.

Local copies use filesystem copy optimizations when available. On the same APFS volume, eligible files can share disk blocks while remaining independently writable. See local copies and NFS for what this means for storage use and reported speed.

The final summary shows what was copied or skipped, how long it took, and any errors. Add -v to list copied paths. For connection and performance details, see diagnosing a slow copy.

File transfers have no fixed duration or stall deadline. They can continue through slowdowns and pauses; cancel the command if you no longer want to wait. Connection setup, SSH keepalives, and return-connection heartbeats still have time limits, and restricted server-to-server copies must finish before their signed authorization expires. SDK callers can also set their own deadlines.

Shell pipelines and file descriptors

Use --src-fd 0 for stdin or --as-fd 1 for stdout. The file can be local, on an SSH host, or in S3:

gzip -c data | syq cp --src-fd 0 --to server --as data.gz
syq cp --from s3://backups data.gz --as-fd 1 | gzip -dc > data

These copies transfer raw bytes without source metadata or restart recovery. EOF ends input even if the producer failed; output can be partial after a failure. See file descriptors for the full contract and restrictions.

See where files go

A named directory brings its name along. --srcs-in copies its contents; --as chooses an exact destination name.

syq cp project --to server --into /backup

project/
├── index.html
└── images/
    └── logo.svg

/backup/
├── index.html (old)
└── notes.txt

/backup/
├── index.html (kept)
├── notes.txt (kept)
└── project/ (new)
    ├── index.html
    └── images/
        └── logo.svg

Note to rsync users: trailing slashes have no special significance in syq’s native commands. syq rsync keeps rsync’s slash rules.

Copy over the network

Use --to to send files, or --from to fetch them:

# Copy project to /backup/project on server.
syq cp project --to server --into /backup

# Fetch /data/a and /data/b from server into ./data/a and ./data/b here.
syq cp --from server -C /data --src a --src b --into ./data

-C DIR is shorthand for --cwd DIR: look for the source files in DIR. In the second example, /data is on the server and ./data is on your machine.

To use the default destination:

syq cp project --to server       # put project in your home directory on server
syq cp --from server project     # fetch project into your current directory

SSH endpoints use [USER@]HOST[:PORT], for example alice@server:2222. Host names cannot start with a dash, including when using an --rsh wrapper. Enclose IPv6 addresses in brackets: alice@[2001:db8::1]:2222. A colon in a native path is simply part of the path.

Use --to @NAME to send local source files to a registered receiving machine. The @ is required: --to laptop selects an SSH destination, while --to @laptop requires that receiver to be connected and pass its identity check. See Send files home from a server for setup and destination paths.

Use S3 buckets with the same selectors and placement options:

syq cp photos --to s3://backups --into laptop
syq cp --from s3://backups laptop/photos --into restored
syq cp --from s3://backups --srcs-in laptop --to s3://archive --into laptop

See S3 options and behavior for credentials and filesystem differences.

For two SSH endpoints, see Copy between servers.

Progress

Syq shows a progress bar when running in a terminal. It tracks bytes processed; while files are still being discovered, the percentage is unknown. Wider terminals also show elapsed time, speed, ETA, and file counts.

Use --progress to force the display or --no-progress to hide it. --quiet hides it too. After five seconds without a byte update, no update shows how long it has been; this does not by itself mean the connection has failed. Wait for the final summary to confirm success, even if the byte bar looks full.

The bar also covers syq rm, counting entries instead of bytes. --progress-json provides JSON progress for displays; use results records to track completion in scripts.

Choose a destination

# Copy report.txt under a new name; refuse to overwrite an existing entry.
syq cp report.txt --as-new reports/final.txt

--into uses or creates a directory. --as can rename a directory too. Sources that would collide at one destination are refused before copying.

For a missing or empty destination, syq checks available space and, where the filesystem reports it, capacity for new files. It refuses a clear shortage, but these estimates do not guarantee that the copy will fit.

Choose which existing files to update

By default, selected destination entries are updated when needed. These options apply to individual entries inside the copy:

# Import new files without replacing existing files.
syq cp --only-new --srcs-in incoming --into archive

# Refresh only files already in the destination.
syq cp --only-existing --srcs-in project --into deployed

--only-new keeps existing entries and their metadata, while adding missing children to existing directories. Those directories must be writable; syq does not change their permissions to add files. Adding children can still change directory timestamps. A dry run does not test write access.

For S3 downloads with these flags, the summary’s unchanged-file count includes skipped symlinks selected through a prefix, but excludes symlinks named directly or through --mapping.

If a source directory meets an existing non-directory, --only-new skips that subtree. --only-existing skips a subtree when its destination is missing or is not a directory. It cannot combine with --into-new or --as-new. The placement options --into-existing and --as-existing check only the placement path, rather than every copied entry.

--skip-newer compares timestamps, not the age of the contents. It affects only regular-file pairs; replacements between non-directory entry types still occur, but replacing a directory with a non-directory or the reverse is refused. Combine it with --only-existing to avoid creating missing entries too.

--only-new cannot combine with either policy. Neither --only-new nor --skip-newer can combine with --inplace: an interrupted write could leave an incomplete file that the next run skips. Restricted receivers also refuse --only-existing --inplace.

These options do not disable --prune; requested pruning still removes extras.

Preview changes

--dry-run shows planned changes without carrying out the copy or deletions. Add -v to list the changes by path:

syq cp --dry-run -v --srcs-in project --into backup

The summary shows where files would land, what would change, and how much data may move. A requested results file is still written, and remote setup may cache the helper or install syq. The filesystem can change between preview and execution.

Mirror a directory

--prune removes destination files that have no counterpart in the source, including copies to and from S3:

syq cp --prune --max-delete 100 --srcs-in build --into-existing deploy

This updates deploy from build, then removes extras. Preview with --dry-run -v first. If more than 100 removals are planned, none are performed and the command exits 25.

Pruning stays inside the copied directories. Copying named directories a and b into backup prunes backup/a and backup/b, leaving backup/c alone. Ignored paths and files skipped by size limits are protected.

Scan or copy errors prevent deletion. An interruption after deletion starts can leave some extras removed. Do not prune while another copy is writing into the same tree: its completed files can be treated as extras.

Keep the source outside the destination you are pruning. Syq checks this for local copies and remote paths with the same host name, user, and port, but cannot detect overlap through different SSH aliases, local-to-SSH connections, or shared storage across hosts.

Pruning keeps syq’s partial files and recovery entries, including their contents and parent directories. Use -v to see files kept because their names match the partial-file format. If different filename spellings resolve to a copied file, syq protects it; this can also keep extra hard links to it.

Ignoring paths

# Use the project's existing ignore rules.
syq cp --ignore-from .gitignore --srcs-in project --into backup

# Skip node_modules and object files.
syq cp --ignore node_modules --ignore '*.o' --srcs-in project --into backup

Patterns use gitignore syntax. Rules run in command-line order; the last match wins. ! re-includes a path.

Ignored directories are not scanned. To keep part of one, include the parent:

# Skip other logs, but copy logs/keep and its contents.
syq cp --ignore 'logs/*' --ignore '!logs/keep/' --srcs-in project --into backup

Ignored paths are also protected from pruning.

Resume an interrupted copy

Rerun the command. Completed files are skipped, and syq can reuse matching parts of an interrupted file. It writes a new temporary file beside the destination and replaces the final file only when complete. Previous partials stay unchanged. Reuse is not guaranteed; local copies may use the filesystem’s faster copy operations instead.

When resuming, syq reuses the interrupted copy rather than the old destination. If the source changes between attempts, this can resend bytes that still match the old destination.

Resuming requires space for the new output as well as the previous partial. This can require enough free space for another complete file, even when only a small amount remains to transfer.

Concurrent copies use separate temporary files. With unchanged sources, each completed file comes from one copy, but the whole tree is not a snapshot. In-place writes expose unfinished updates, and pruning can delete another copy’s completed files.

Partials are named .FILENAME.syq-tmp.RANDOM, with 16 random characters at the end. The filename portion is shortened or omitted when space is tight. Syq removes its own partial when it publishes the completed file. Interrupted runs can leave partials behind, including after a later successful retry. Partials with shortened or omitted filenames may not be reused.

To remove leftover partials, stop copies writing into the tree, then preview and run:

syq clean-partials --dry-run -v backup
syq clean-partials backup
# Search several remote trees with the parallel removal workers.
syq clean-partials --on server --cwd /data --performance-tuning workers=8 backup archive

This command removes regular files with the current partial-name format. It keeps directories, other filenames, and symlinks, and does not follow symlinks. Use --root DIR to confine traversal and --results FILE for the same removal records as syq rm (mode: "rm"). A regular file deliberately named like a partial is also selected. Old partial formats are neither reused nor selected by this command; remove those manually.

Interrupted replacements and macOS clones can also leave .syq-swap-... entries beside the destination. These may contain displaced originals or temporary clone data. Stop all copies using the destination, inspect each entry, and recover anything you want to keep before removing it. Neither clean-partials nor pruning removes these recovery entries.

Conflicting names and file types

Some filesystems treat names that differ only in case or Unicode spelling as the same name. Syq does not detect these collisions before copying, so one source can overwrite another. Rename conflicting sources first to keep both. Names the destination cannot create are reported as copy errors.

Both syq cp and syq rsync refuse to replace a directory with a file, symlink, or special file, or the reverse, even when the directory is empty. The copy reports an error and skips that directory’s contents. Move or remove the conflicting destination before retrying.

Other replacements can fail if the filesystem lacks the operation needed to replace the old entry safely; the old entry is kept. See interrupted-copy recovery for entries left beside the destination.

Check file contents

Syq normally skips files whose size and modification time match. Matching metadata does not prove that contents match; use --hash to compare contents:

syq cp --hash --srcs-in project --into backup

To require a known whole-file digest, use --expected-hash ALGORITHM:HEX with one named regular file. Syq checks the complete result, including reused bytes. A mismatch fails the file; with normal staging it does not replace the destination. Selection filters still exclude files from checking. For batch copies, use expected digests in mappings.

To compare without copying:

syq cp --verify-only --srcs-in project --into backup

Missing or different entries make the command fail. This compares contents, symlink targets, and entry types, without comparing metadata or looking for extra destination files.

The Integrity checking reference covers timestamp precision, every comparison and payload-check algorithm, expected digests, and verification restrictions. For consistent source data, stop concurrent writers or copy a snapshot.

In-place writes

By default, syq builds an updated file beside the old one and replaces it when complete. --inplace writes directly into the destination file instead:

syq cp --inplace large-file --to server --into /backup

This saves temporary disk space and can avoid copying unchanged data into a new file. Readers can see a mixture of old and new contents during the copy. If interrupted, the incomplete file stays at its final name until you finish the copy. Writes through a hard link also affect its other names.

Use the default when other programs need to read a complete file throughout an update. Copies sent back to your laptop do not support --inplace.

Preserve metadata

Copy keeps modification times and copies symlinks as symlinks. New files use the source read, write, and execute permissions limited by the destination umask; existing files keep their destination permissions. For example, a new script with mode 755 stays executable with umask 022. Source setuid, setgid, and sticky bits are not copied by default. On macOS, an existing destination directory must be readable before syq can temporarily repair missing write or search permission.

To copy source permissions exactly, including onto existing files, or request ownership too:

syq cp --preserve=permissions,ownership project --into backup

permissions preserves modes; ownership requests numeric owner and group; specials enables device, FIFO, and socket nodes. Ownership needs suitable permissions on the destination. Hard links, ACLs, and xattrs are not preserved.

Symlinks

A named symlink is copied as a link. Syq refuses to follow links in paths you supply unless you ask it to:

# Copy the directory current-project points to as backup/current-project.
syq cp --follow-src current-project --into backup

With --as link, the final link itself is replaced, even with --follow-dst. Links found inside a directory are never followed. See the security explanation.

Keep sources inside a directory

--root DIR both sets the source directory and prevents selections from escaping it:

# Copy /srv/data/reports and /srv/data/photos into backup.
syq cp --root /srv/data reports photos --into backup

Sources must be relative to that root. A selection such as ../private is refused; even with --follow-src, symlinks cannot lead outside the root. Unlike -C, this is a boundary, not just a starting directory. It does not constrain the destination.

Output and diagnostics

Human output, including persist status and persist receive status, escapes terminal control characters, Unicode line separators, and directional marks in names and peer diagnostics. JSON status output keeps the original values.

Environment variables and local files

Syq reads no configuration file. Besides the usual system variables such as HOME, TMPDIR, and SSH_AUTH_SOCK, and the AWS credential, region, and endpoint variables described under object storage, syq honors:

• SYQ_CP_OPTIONS, SYQ_RSYNC_OPTIONS, and SYQ_RM_OPTIONS hold extra arguments for syq cp, syq rsync, and syq rm, respectively. Use them to adjust a command inside a script or program that does not let you change its syq options. The value is split like a shell command line and inserted right after the command name, before the arguments the script supplies, so SYQ_CP_OPTIONS='--resource-limits bandwidth=10M' ./nightly-backup.sh limits every syq cp the script runs. An option that the script also gives is reported the way any repeated option is. Syq removes these variables from its own environment before starting any other program, so ssh, an --rsh command, and syq’s own helper processes never see them. The rest of the environment reaches ssh unchanged.
• SYQ_NO_UPDATE_CHECK and DO_NOT_TRACK turn off update reminders.
• SYQ_TUNING_CACHE names the remembered connection count file; an empty value turns that cache off.
• SYQ_DEBUG adds internal diagnostics to stderr, and SYQ_S3_DIAGNOSTICS=1 does the same for object storage requests. Their content changes between versions.
• XDG_CACHE_HOME, XDG_CONFIG_HOME, and XDG_RUNTIME_DIR relocate the files below; HOME supplies the defaults.

Syq keeps these files on the machine where you run it:

None of the caches or the update stamp are required. When they cannot be written, for example from a read-only home directory, syq skips them and the copy proceeds; the remembered connection counts are still read if the file exists. Persistent connections do need a writable runtime directory, and a server that receives files must be able to keep its enrollment state.

On an SSH server, the helper syq installs lives under ~/.cache/syq/helpers/ in the server account. When that directory cannot be created, the copy fails with a message saying so; point --syq-path at an installed helper or pass --no-bootstrap when one is already on the server’s PATH.

More options

--src-non-dir and --src-dir require a non-directory or directory respectively. Use --min-size and --max-size to select regular files by size.

For parallelism and bandwidth controls, see Speed. For scripts, see Automation results.

The cp command reference lists every option, including tuning, scripting, and manual setup. In a terminal, use --help (or -h) for everyday options and --help-all for the full list. syq help COMMAND shows the same help without running the command. In syq rsync, -h means human-readable sizes; use --help for help.

Remove files

See syq rm for the option list.

Remove a file or symlink:

syq rm old-file

Named paths, --src, and --srcs refuse directories. Select a tree explicitly to remove it recursively:

syq rm --src-dir old-output

Remove a directory’s contents recursively, leaving the directory itself:

syq rm --srcs-in cache

Add --dry-run -v to either command to see what would be removed first. Missing paths succeed. Filesystem removal is permanent; completed deletions cannot be rolled back.

On another machine

syq rm --on server --src-dir /scratch/old-output

This removes /scratch/old-output on server. Remote removal runs while your connection stays open; there is no detached mode. With --dry-run, nothing is removed, but remote setup may still cache the helper or install syq.

For object storage, use --on s3://BUCKET:

syq rm --on s3://backups --src-dir old-backup --dry-run -v

Ordinary removal respects bucket versioning; explicit version deletion is available with --s3-all-versions or --s3-version-id. See S3 removal.

Limit the selection

syq rm --root /srv --src-dir cache --src-dir old-output

This removes /srv/cache and /srv/old-output, with selection confined to /srv. Use --src-non-dir PATH or --src-dir DIR when the path must be a non-directory or directory respectively. All selections are checked before deletion begins. Filters are not supported.

Symlinks

A selected symlink is removed as a link, leaving its target alone. Symlinks inside a selected directory are also only unlinked.

--follow-src permits traversal through symlinks in --cwd, --root, and selector parent directories. --follow also permits symlinks in the --results path. The final selected symlink is always removed as a link, even with --follow-src or --follow. For example, if current points to releases/v1, syq rm --follow-src current/log.txt removes releases/v1/log.txt, while syq rm --follow-src current removes only the link.

--src-dir and --srcs-in reject a final selected symlink, including when following is enabled. Select the actual directory to remove it or its contents. With --root, traversal must still stay inside that root.

Results

Removal continues with independent entries after per-entry failures and exits 23. S3 removal uses concurrent batches and reports each key or version separately. During permanent removal, a data-version failure preserves all selected delete markers; see S3 removal. Fatal setup or connection failures exit 1. Use --results for per-path outcomes in scripts.

For the full option list, run syq rm --help-all.

Send files home from a server

Inspect files on your server, then copy them to your laptop from the same shell. The laptop opens and maintains the connection. It needs no SSH server, public address, or incoming network port.

With syq installed on both machines, connect from your laptop:

syq persist connect server

This keeps a connection open so the server can send files back to your laptop. Once the command finishes, you can close the terminal and continue working on the server. By default, your laptop is available under its short hostname, and received files go into your home directory.

To give it the name laptop and choose a different starting directory:

mkdir -p ~/Downloads/server
syq persist receive on --name laptop --cwd ~/Downloads/server
syq persist connect server

On the server, use the name from any shell, including an existing tmux session:

ls -lh results
syq cp results --to @laptop
syq cp report.pdf --to @laptop --as reports/latest.pdf

Each incoming copy waits for approval on your laptop before it can inspect or change destination entries. Review the destination and permissions, then choose Allow once or Deny. Use Details on macOS or syq persist receive pending in a local terminal to see the complete request. Approving a copy trusts the server to supply its contents.

You can also run commands on the receiving machine, with separate local approval, to build a project there or open a copied artifact.

Multiple receiving profiles

Give a project its own receiving name and directory:

syq persist receive on --name project --root ~/work/project
syq persist connect server

Then run syq cp results --to @project on the server. The directory must already exist. Each name has its own settings and approval policy, so you can keep a project separate from your general laptop destination. Names stay assigned to their receiving machine while it is offline; see replacing a receiver when moving a name to another laptop.

Use syq persist receive status to list profiles and syq persist receive off --name project to stop one. See profile management for more options.

Approving copies

Approve or deny from the desktop prompt, or from a terminal on your laptop:

syq persist receive pending
syq persist receive approve REQUEST_ID
syq persist receive deny REQUEST_ID

Requests expire after five minutes. If a prompt is missing or dismissed, the request stays pending; it is never approved automatically. To use only terminal approval, run syq persist receive on --notify off.

For unattended copies from trusted server accounts, explicitly enable automatic approval:

syq persist receive on --approve always
syq persist receive on --approve ask   # require approval again

Automatic approval trusts all processes running as the connected server accounts, including for overwrites. Commands and copies to another server still require separate approval. See persistence security for the trust boundary.

Names and paths

Use --to @laptop to select a receiving machine. The command fails if that receiver is offline or fails its identity check. Without @, --to laptop always names an SSH destination, resolved through SSH configuration or DNS. Once a copy starts, it keeps the same destination even if the connection fails.

The directory you set with --cwd is where incoming copies start. You can choose a path relative to it with --into or --as, or use an absolute path to copy elsewhere. Without either option, files go into the starting directory.

To contain copies within a directory instead:

syq persist receive on --name laptop --root ~/Downloads/server

With --root, all incoming copies must stay inside that directory. Absolute paths and .. are rejected, and symlinks cannot lead outside it. A copy cannot replace the root itself with --as .. Switching back to --cwd removes this restriction.

Changing a profile’s settings stops its active copies so the new settings can take effect. Other profiles keep working.

Syq also protects its own receiving files, executable, and SSH authority files from incoming copies. See directory requirements if a receiving location cannot be opened.

Copy permissions and limits

By default, each copy is limited to 100 GiB and one million entries. Pruning requires a positive deletion limit on both machines. Change limits with syq persist receive on --max-bytes SIZE --max-entries N --max-delete N.

Most copy options work here; ownership and special-file preservation, --inplace, and --min-size are unsupported. See copy limits for details.

Background connections

You can inspect your connections with syq persist status. To stop receiving while keeping SSH connections open for your own copies, run syq persist receive off. Use syq persist off to close both directions.

After a network interruption or laptop sleep, syq reconnects automatically. An interrupted copy still needs to be rerun so it can resume. After rebooting your laptop, run syq persist connect server again.

If a connection fails to start, syq persist receive status shows the error. The persistence reference covers troubleshooting, upgrading, and using connections in scripts.

Run commands on your receiving machine

See syq exec for the option list.

From a server shell, ask your Mac or Linux desktop to run a command:

syq exec --on @laptop --cwd work/project -- cargo test
syq exec --on @laptop --cwd work/project -- open report.html

The second command uses macOS’s open program to display an artifact. Any program installed on the receiving machine can be requested, including a native application or a version of syq you have just built there.

If you have already set up receiving files, you can request commands through the same connection. Otherwise, run syq persist connect server on your desktop first. Your desktop needs no SSH server or incoming network port, and you can make requests from any shell on the server, including an existing tmux session.

Replace @laptop with your desktop’s receiving name. You can find it by running syq persist destinations list on the server or syq persist receive status on your desktop. Use --on @laptop; the desktop must be connected.

Approve each command locally

Each command waits for approval on the receiving machine. Review the program, arguments, requesting server, and working directory. Desktop notifications may truncate long commands; inspect the complete request from a local terminal:

syq persist receive pending
syq persist receive approve REQUEST_ID
syq persist receive deny REQUEST_ID

Command requests are available whenever receiving is enabled. Every command requires its own decision, even with syq persist receive on --approve always for copies. Approving a copy does not approve commands. A missing or dismissed desktop prompt never grants permission; use the local terminal commands. Pending requests expire after five minutes and are cancelled when the requester disconnects or receiving restarts or stops.

An approved command runs with your local user’s permissions, including access to files and credentials. The receiving --root and copy limits do not contain commands. Build tools and scripts can execute code from their input files; approving a displayed command does not establish that those files are safe.

Arguments and working directories

Put -- before the program. Everything after it is passed as a literal argument, including strings starting with --. There is no implicit shell expansion on the receiving machine. To use shell syntax, request a shell:

syq exec --on @laptop --cwd work/project -- sh -c 'cargo build && ./target/debug/demo'

--cwd DIR (or -C DIR) is relative to the directory selected by persist receive on --cwd or persist receive on --root. Its default is that directory. Absolute paths and .. can select elsewhere. The directory must exist. Syq does not expand ~ on the receiving machine; use a relative path or an absolute path instead.

The command inherits the receiving service’s local environment, including PATH and its desktop session. It does not inherit the server’s environment. Restart receiving from a terminal in the desired desktop session when those values change. Stdin is closed and there is no interactive terminal.

A request supports at most 256 arguments, including the program, with at most 16 KiB of argument bytes and a working-directory path of at most 4096 bytes. Each server connection permits one pending approval and eight active commands. Active commands do not prevent a new approval or an ordinary copy.

Output, completion and cancellation

Stdout and stderr stream back separately without text conversion. Syq returns the command’s exit code; when the command is killed by a signal it reports the signal and returns 128 + signal. A setup error or connection failure is a nonzero result. A connection that closes before delivering the exit status is an error even if some output arrived successfully.

Interrupting the request, losing the connection, changing receiving settings, or stopping receiving cancels execution. Syq forcibly stops the command’s process group, including remaining children when the foreground program exits; cleanup handlers do not run. Detached processes and applications launched through macOS open can outlive the request. Syq does not manage detached jobs.

A command may already have changed files when interrupted. It is never retried automatically after a lost connection. Inspect the outcome before requesting it again. Commands do not produce copy receipts or copy automation records.

Python callers can use the SDK’s existing raw process interface with their chosen executable:

import syq

result = syq.run(
    ["exec", "--on", "@laptop", "--cwd", "work/project", "--", "cargo", "test"],
    executable="/path/to/syq",
    timeout=600,
)
print(result.stdout.decode())

The raw SDK call captures byte output and raises on a nonzero exit by default. The async client’s run method accepts the same command arguments. For live terminal output, invoke the CLI directly or use a subprocess with inherited stdout and stderr.

Copy between servers

Copy directly between servers without putting private keys on either server or giving one server unrestricted access to your SSH agent. Your machine authorizes the copy and shows the results; the file data bypasses it.

# Copy the contents of hostA's big directory into hostB's big directory.
syq cp --from hostA --srcs-in big --to hostB --into big

HostA gets permission for this transfer only. HostB checks that permission and reports what it changed. See Security for what this protects against.

Start a copy from the source server

If your laptop has a return connection to the source server, you can inspect files in any server shell and send them to another SSH host:

# Run on hostA, including in an existing tmux shell.
ls -lh results
syq cp results --to hostB --into /archive

Syq uses the first available receiving machine in alphabetical name order. If none is available, or the requested options are unsupported on this route, it uses hostA’s SSH access. --to @laptop instead sends the files to the receiving machine itself.

Use --auth-from @laptop to choose your laptop explicitly, or --auth-from ssh to use hostA’s SSH access. The default is --auth-from auto. --via @NAME is an alias for choosing a receiving machine. See authorization selection for name rules and route restrictions.

The selected laptop asks for approval before contacting hostB. Approve with the desktop prompt or syq persist receive pending and syq persist receive approve REQUEST_ID on the laptop. These requests require a decision even when persist receive on --approve always permits automatic copies onto the laptop itself. Once an approval request is sent, a refusal, interrupted connection, setup failure, or copy failure ends that attempt; syq does not try another authorizer or SSH. --auth-from @NAME and --via @NAME fail if that receiving machine is unavailable.

The laptop uses its own SSH configuration, credentials, and trusted host keys to connect to hostB and install the matching syq helper. Connect to hostB with ordinary SSH from the laptop first if its host key is not yet trusted. The source server receives no SSH credentials or agent access. The control stream passes through the laptop; file data goes directly from hostA to hostB over encrypted TCP. HostB must expose a reachable data port to hostA. Failure to reach it fails the copy without switching to SSH data.

The prompt shows the requested SSH endpoint and destination path. Relative paths start in that account’s home directory on hostB. Your laptop’s receiving root does not contain this copy, but its byte, entry, and deletion limits still apply.

Keep the source command and your laptop’s connection alive until completion. Stopping receiving or losing that connection cancels the copy. Retry with a new approval to resume it. Supported copy options match return copies, with additional route restrictions.

What you need

• SSH access from your machine to both servers, with their host keys already trusted. Connect with ordinary SSH once if either server is new to you.
• An SSH agent on your machine, and OpenSSH 8.9 or newer on your machine, hostA’s SSH client, and hostB’s SSH server.
• SSH connectivity from hostA to hostB. A reachable TCP data port on hostB, normally in 47600–47699, enables encrypted TCP workers. Otherwise the copy uses SSH workers on the same hostA-to-hostB route. --no-tcp selects SSH directly.
• An existing parent directory for the destination.

Keep your command running until the copy finishes. Use native syq cp; syq rsync does not accept two remote endpoints.

First copy and access management

The first copy sets up a restricted receiver on hostB automatically. It adds a restricted key to authorized_keys; the private key stays on your machine. Later copies reuse this setup.

To prepare /archive on hostB ahead of time, including before a dry run:

syq receiver enroll hostB:/archive
syq cp --dry-run -v --from hostA --srcs-in data --to hostB --into /archive

Inspect or remove this access with:

syq receiver list
syq receiver revoke ID

Use the ID from list. Revocation stops active copies for that enrollment, blocks new ones, and removes its setup from both machines. Other enrollments keep working. Completed writes remain; interrupted copies can leave partial files. Enroll again before retrying those copies.

If revocation reports that receivers have not stopped, the enrollment stays revoked. Retry receiver revoke to finish cleanup. See enrollment details for upgrades and sharing between installations.

If your machine reaches hostB through hostA, add --via hostA to enroll or revoke.

Mirror a directory

Include a deletion limit when pruning:

syq cp --prune --max-delete 100 --from hostA --srcs-in data --to hostB --into-existing /archive

This updates /archive and removes extras. If more than 100 removals are planned, none are performed and the command exits 25. Preview first with --dry-run -v.

Other routes and authentication

Syq may switch between encrypted TCP and SSH on the selected route. It never silently relays file data through your machine when a direct connection fails. If the servers cannot connect directly, explicitly relay through your machine:

syq cp --coordinate-at local --from hostA --srcs-in data --to hostB --into /archive

This uses your machine’s bandwidth and ordinary SSH access to each endpoint.

Other authentication modes can use server-held credentials, a destination- restricted SSH agent, or full agent forwarding. They grant different authority; see the advanced reference before choosing one. That page also covers detached copies, signed results, and direct-mode limits.

Rename and reorganize during a copy

See syq map for the option list.

syq map lists source/destination pairs as one JSON object per line. Transform that list with a script, then give it to syq cp --mapping. The copy checks for destination collisions and supports normal resume.

For Python scripts, see the Python SDK and its mapping examples.

Lowercase destination names

set -o pipefail
syq map --srcs-in src \
  | jq -c '.dst.value |= ascii_downcase' \
  | syq cp --mapping - -C src --to nas --into /pub

src/                       nas:/pub/
  Berlin/IMG_1234.JPG         berlin/img_1234.jpg
  Notes.TXT                  notes.txt

If another entry also claims notes.txt, the copy is refused before files are transferred. Symlink target text is not rewritten: renaming its target can leave a link dangling on a case-sensitive destination.

Group photos by modification month

set -o pipefail
syq map --srcs-in photos \
  | jq -c 'select(.kind == "file")
        | .dst.value = (.mtime | gmtime | strftime("%Y/%m")) + "/" + .dst.value' \
  | syq cp --mapping - -C photos --to nas --into /archive

A July 2024 file IMG_1234.JPG lands at /archive/2024/07/IMG_1234.JPG. The filter keeps regular files only; missing parent directories are created. Dates use file modification time in UTC, not photo EXIF dates.

Other filters can replace the jq stage:

# Keep files of at least 1 MiB, plus directory and link entries:
jq -c 'select(.kind != "file" or .size >= 1048576)'
# Drop device, FIFO, and socket entries:
jq -c 'select(.kind != "special")'

Check the producer before copying

A pipeline’s consumer sees only the bytes it receives. If a generator fails after emitting valid entries, those entries can still be copied. set -o pipefail makes the pipeline report failure, but does not undo writes.

To require successful generation before copying, save the manifest first:

set -o pipefail
syq map --srcs-in src | jq -c '.dst.value |= ascii_downcase' > m.ndjson \
  && syq cp --mapping m.ndjson -C src --to nas --into /pub

Add --dry-run -v to cp to preview placement.

The format

A manifest contains one JSON object per line (NDJSON):

{"src":{"encoding":"utf-8","value":"IMG_1234.JPG"},"dst":{"encoding":"utf-8","value":"2024/07/photo.jpg"},"kind":"file","size":4194304,"mtime":1721900000}

Paths use encoding: "utf-8", or "base64" with standard base64 of raw filename bytes. Absolute or empty paths, and any . or .. component, are refused. Unknown fields are refused too.

expected_digest requires a regular file. Algorithms are blake3, sha256, md5, and xxh3-128; the hex value contains 64 digits for BLAKE3 or SHA-256, and 32 for MD5 or XXH3-128. It checks the complete resulting file, including reused bytes, and a mismatch fails the entry. See content checks for staging, in-place writes, and selection filters. Older binaries that do not support this field reject the manifest.

Each entry copies one object. A directory entry is not recursive. syq map emits its descendants as separate entries. A missing source or wrong kind fails that entry while independent entries continue. A kind mismatch is a non-retryable conflict; a missing source is an I/O failure whose retryability is unknown.

Any program can generate this format:

syq cp --mapping pairs.ndjson -C photos --to nas --into /archive

Copy between servers

The manifest is read on the machine where you run the command. Its source paths resolve on the source server, and its destination paths resolve beneath the destination container:

syq cp --from hostA -C /data --mapping pairs.ndjson --to hostB --into /archive

--mapping - reads the manifest from stdin. File contents follow the selected copy route. The restricted receiver permits writes only at the listed destinations and creates missing parent directories as needed. New parents use permissions limited by the receiver’s umask; existing parents keep theirs. If a file or symlink blocks a parent directory, move or remove it before retrying. The affected entries fail while unrelated mappings continue.

Mapped destinations and their parent directories count against the receiver’s entry limit. Each manifest line can be up to 1 MiB, and each destination path up to 4096 bytes. There is no separate limit on the total manifest size.

Emitting a mapping

syq map --srcs-in photos     # contents; paths relative to photos
syq map photos              # named directory; paths include photos/
syq map photo.jpg --as albums/cover.jpg

map is local and does not contact a destination. It takes source selectors, -C or --root, source follow options, and --as for one named selection. Copy options and filters belong to the later cp command or your transform. map refuses non-UTF-8 names; hand-written manifests may use base64.

Semantics and limits

• --mapping replaces cp source selectors. Use --into, --into-new, or --into-existing for the destination. It cannot combine with --as or --prune or --detach. Both endpoints may be remote.
• A contents selection emits paths relative to the selected directory. Use that same directory as the consuming copy’s -C base. Named map selectors must be relative and resolve inside their base; a contents selector may point outside it. --root confines either kind of selection.
• Follow options apply to command-line paths, never to manifest entries. A manifest entry that would traverse a symlink fails. Named map selectors followed with --follow-src emit the referent path relative to their base and refuse referents outside it.
• Named manifest paths follow the normal control-path rules: only --follow permits link traversal in that path. Use --mapping - for portable stream input; named FIFOs need Linux with procfs.
• The whole manifest is read and validated before copying. Malformed input, duplicate destinations (including identical duplicate lines), and declared file/ancestor conflicts refuse the run. The destination container may already have been created. Memory grows with manifest size.
• Conflicts found only when inspecting actual source or destination objects fail individual entries. A missing destination parent is created implicitly.
• Normal native preservation applies. kind: "special" checks the type but does not enable special-file copying: use --preserve=specials or those entries are visibly excluded.

Machine-readable results

Add --results r.ndjson to record outcomes in a fresh file outside the copy trees. See Automation results for the stream contract.

Failed mapping entries contain src, dst, and kind, so they can form a retry manifest. Preserve expected_digest too when present. First require a terminal result with success or partial: a missing terminal or an early stop means some entries may have no results. In those cases, rerun the original copy instead.

set -o pipefail
syq cp --mapping big.ndjson -C src --to nas --into /data --results r.ndjson
jq -cs 'if (.[-1].type? // "") != "result"
        then "incomplete results stream (no terminal record)" | halt_error
        elif (.[-1].status != "success" and .[-1].status != "partial")
        then "run stopped early (status \(.[-1].status)); rerun it instead of retrying" | halt_error
        else .[] | select(.type == "operation_result"
                          and .disposition == "failed"
                          and .retryable != "no")
             | {src, dst, kind}
               + (if has("expected_digest") then {expected_digest} else {} end)
        end' r.ndjson \
  | syq cp --mapping - -C src --to nas --into /data

The filter skips non-retryable entries, including failed implicit parent creation without a source path. It does not guarantee that retrying will succeed; fix the underlying error first. Unchanged and excluded files appear only in summary totals, not as individual results.

For command-restricted copies between servers, --results contains verified receiver receipts. These describe destination changes rather than source entry failures; retry the original mapping instead of applying the operation_result filter above.

Speed

Syq copies files in parallel and adjusts its connection count automatically. Start with the defaults. For repeated remote copies, keep the connection open to avoid logging in each time.

Quick comparison

Use the quick script to get a sense of performance on your own machines:

curl --proto '=https' --tlsv1.2 -fLsS -o try-benchmark.sh \
  https://raw.githubusercontent.com/greaber/syq/master/scripts/try-benchmark.sh
bash try-benchmark.sh

The default pushes 1,024 files of 8 KiB to an SSH host you choose, comparing syq with rsync over three rounds. Use --mode pull for downloads or --mode local to include cp. --workload large selects one 64 MiB file; --workload both runs both workloads. The script checks every copy and cleans up afterward.

Network comparisons first run an untimed syq warm-up so it can learn a useful connection count. This aims for 60 seconds of copying, using up to four copies of datasets no larger than 1 GiB each, subject to free space. Slow copies and preparation can take longer. Use --warmup off for a shorter comparison using the existing learned or default count. Local copies and manual tuning skip this warm-up.

--size quick is the fixed-size default. For longer tests, --size medium uses 1 GiB or 4,096 small files; --size large uses 8 GiB or 16,384 files. --size auto grows the test data until a syq copy takes about five seconds or scratch space limits growth, with no fixed total-data or runtime limit. All scored tools use the same dataset. Longer tests help reveal sustained throughput, but neither sizing nor warm-up guarantees tuning has settled.

To try your own syq options, put them after --:

bash try-benchmark.sh --yes --mode pull --host server --tool syq --rounds 1 \
  -- --no-tcp --performance-tuning workers=1 -v

These options also apply to setup and warm-up copies. Path, removal, and output-file options are excluded to keep the test inside its disposable directories. -v shows full commands and scratch paths. Use --tool rsync for a separate rsync comparison, omitting syq options after --. See tuning options for experiments, or run bash try-benchmark.sh --help for all script options.

The script needs Bash, rsync, OpenSSL, standard Unix utilities, and Perl’s core JSON::PP module locally. Remote tests need SSH access and rsync on the other machine; pull warm-ups also need Bash, OpenSSL, dd, and split there. Scratch parent directories must exist. --source-dir selects the local parent; --dest-dir selects the remote parent for both push and pull.

Each scored copy uses an empty destination and preserves permissions and modification times. Generation, helper preparation, warm-up, and content checks are untimed. Failed commands or content checks stop the comparison. SSH connection reuse is disabled for both tools during network tests; syq’s learned connection counts remain active unless you override tuning. Caches are not flushed, and copies do not wait for durable storage.

Compare tools using the main table, which includes connection startup in total command time. Network results show decimal MB/s averaged across trials. Local results show seconds, because filesystem cloning can avoid moving bytes; those times do not measure disk bandwidth.

The separate syq timing table helps explain slow results. It divides total time into the copying interval and time outside it, such as setup and finishing. The copying interval includes waiting and per-file work, and can overlap planning and connection setup. Use it to diagnose syq, not to compare against another tool’s total time. If the script flags a short copying interval or substantial time outside it, try a larger workload to investigate sustained throughput. Each verified syq trial also reports worker activity, endpoint operations, and CPU use; see diagnostics below.

Benchmarks

The separate syq-bench project runs more extensive experiments across workloads, storage systems, and network routes. Browse its results, or run its experiments for a more detailed comparison. The quick script above does not use syq-bench.

Diagnose a slow copy

syq cp -vv --stats data --to server --into /backup

-vv shows the chosen transport and connections; --stats adds totals and available TCP statistics, worker wait fractions, endpoint operations and bytes, and process CPU. These wait fractions help locate delays; they are not proof of their cause. Add --results run.ndjson to inspect how worker waits, endpoint operations, CPU and TCP backpressure change over time. Remote evidence includes its age. See the activity record for interpretation and limitations.

Compare the same workload and direction using empty test destinations. A second copy into the same destination may just measure skipping existing files.

TCP data connections

SSH authenticates remote copies. When reachable, encrypted TCP carries file data on a port in 47600–47699; otherwise copies use SSH on the same route. See server setup for firewall settings. Copies authorized through another machine require direct encrypted TCP.

Use --no-tcp to select SSH data transport or --tcp-ports LO-HI to choose a port range. On Linux, --tcp-congestion ALGO selects an available congestion control algorithm. --tcp-plain disables data encryption and authentication; use it only on a trusted network. Restricted receivers refuse it.

Local copies and NFS

For local copies, syq uses the filesystem’s copy optimizations automatically when it can. On filesystems that support cloning, this can avoid physically copying every byte. You can also copy to or from a mounted NFS directory using its local path. See storage placement for how the source and destination filesystems affect performance.

On macOS, eligible copies within one APFS volume share disk blocks; later writes to either file are independent. Reported bytes count the file’s size, so the displayed rate can exceed physical disk throughput. Small files travel together in batches; see batch size and splitting for the cloning threshold and how tuning changes it.

Cloning keeps the usual overwrite and metadata rules and omits source extended attributes and file flags. --inplace, checksum comparison, bandwidth limits, and explicit range transfers use normal copying. Syq also falls back to normal copying when cloning cannot preserve those rules, such as with inheritable access control entries on the destination directory.

Limit bandwidth

Use --resource-limits bandwidth=RATE to leave bandwidth for other work:

syq cp data --to server --into /backup --resource-limits bandwidth=10M

This limits file data to 10 MiB/s across the copy’s workers. It controls the average copy rate; buffering and protocol overhead can cause network bursts. See Resource limits for all units and supported routes.

Compression and in-place writes

--no-compress can help when compression costs more CPU time than it saves in network traffic. --inplace saves temporary disk space, but exposes incomplete updates to readers. Read in-place writes before using it.

Investigate tuning

Manual connection counts and other tuning options are for troubleshooting and development. Normal copies should not need them.

Server setup

Start by checking TCP access and the source and destination filesystems. Use test data when comparing settings.

Make TCP reachable

Syq listens on one available port in 47600–47699 during a copy. Choose another range with --tcp-ports LO-HI. For a server using ufw, an administrator can allow a trusted client:

sudo ufw allow from <trusted-client-address> to any port 47600:47699 proto tcp

Allow the range in any cloud firewall too. Check the transport with syq cp -vv --stats. Ordinary copies fall back to SSH on the same route when TCP is blocked. Copies authorized through another machine require direct encrypted TCP.

Tailscale

Tailscale can make servers reachable across NAT and firewalls. Allow syq’s data ports through the host firewall and tailnet rules; syq can discover Tailscale addresses. Use tailscale status to check whether the connection is direct or relayed. See Tailscale’s performance guide.

Test congestion control

On Linux, check which TCP congestion-control algorithms are available and allowed on both endpoints:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_allowed_congestion_control

If bbr is listed in both, compare it with cubic on your route:

syq cp --tcp-congestion bbr --stats data --to server --into /backup

Use the same data, a fresh destination, and both transfer directions. The option applies to syq’s TCP sockets. If BBR is missing, see the administrator setup guidance.

Let SSH connections start promptly

OpenSSH’s MaxStartups limit can slow parallel logins. For servers handling parallel transfers, an administrator can consider:

MaxStartups 100:30:200

This allows 100 unauthenticated connections before random rejection begins, and rejects all new ones at 200. It also admits larger bursts from unrelated clients. MaxSessions controls channels sharing a connection; very low values can force extra logins.

Validate changes with sshd -t, then reload SSH using your system’s procedure. Keep an administrative session open. See OpenSSH’s settings.

Check local storage placement

On Linux, inspect the source and destination filesystems:

findmnt -T /path/to/source -o TARGET,SOURCE,FSTYPE,OPTIONS
findmnt -T /path/to/destination-parent -o TARGET,SOURCE,FSTYPE,OPTIONS

Use an existing destination parent and run remote-path checks on the machine that owns the path. Copies within a filesystem supporting cloning can share storage while remaining independently writable. See local copies and NFS for filesystem and mount considerations.

Measure and track improvements

Use syq-bench for repeatable comparisons. Record the commands, versions, mounts, cache state, and other load. Keep reporting and flush settings consistent across runs. See performance tuning to compare worker counts and request sizes.

Security

Syq is designed for copies where the tool has more authority than some of the files or machines involved: a backup account reading another user’s tree, root copying an upload, or two servers transferring through your laptop.

Report vulnerabilities through SECURITY.md.

Filesystem attacks

Assume an attacker can change files and directory entries inside a selected tree, or in a parent directory they can write. They may act before the copy starts or while it is running. They do not control the account running syq.

The attacks we aim to stop include:

• Redirecting a read or write with a symlink. A directory might be replaced with a link to a private source or unrelated destination.
• Escaping through a supplied filename. A peer might send an absolute path or .. components to reach outside the selected tree.
• Turning a deletion into a walk of another directory. An entry might change type after syq has decided what to remove.

Native syq refuses symlink traversal unless you explicitly request it. It keeps selected directories open and works through those handles, so later renames cannot redirect the copy to another tree. Names received from the other endpoint are validated; deletion does not follow replacement links or recurse into unexpected directories.

These protections bound where the operation can go. They do not prevent an authorized writer from changing the contents of that tree. In a removal race, a single replacement entry can still be unlinked.

Relationship to rsync 3.5.0

Syq’s path protection is inspired by rsync 3.5.0’s security design: keep directories open, work relative to them, and distrust peer-supplied paths.

The default for paths you type is simpler in native syq:

Rsync’s ownership policy preserves familiar symlinked directory setups. Our concern is that link ownership alone does not prove who placed it there: rename permissions come from its parent directories, subject to restrictions such as the sticky bit. A relative link moved to another directory can point somewhere different without changing its owner. See the rename rules.

Native syq avoids that implicit trust decision. This is a stricter default for this case, not a claim that syq is more secure overall. syq rsync keeps the ownership-based policy for compatibility. Its local-only --insecure-links option permits foreign-owned symlinks in typed local source, destination, and control paths. After opening a source root, scans and content reads still use its directory handles and refuse descendant symlink traversal.

TCP data connections

TCP workers authenticate using credentials delivered through the control connection. The initial handshake has a ten-second deadline, but there is no general copy I/O timeout: a stalled peer can leave a copy waiting until you cancel it.

Syq bounds incoming messages, decompression, and decoded collections to limit memory use from malformed peers. These are not a total process memory cap; memory also grows with connection count and request size. Invalid replies fail the connection visibly. Ordinary copies can fall back to SSH if TCP setup fails, keeping the same endpoints.

A compromised source server

Syq checks peer-supplied paths and data ranges before using them. These checks reject malformed replies; they cannot establish that a source’s file listing or contents are truthful.

For a default direct remote-to-remote copy, the source gets permission for one transfer, not your SSH agent or a reusable destination credential. The restricted receiver independently enforces the allowed destination paths, write and deletion permissions, and limits. Selection based on source facts, such as --skip-newer timestamp comparisons, relies on the source’s reports. The source cannot enlarge or replay that permission. SSH and encrypted TCP workers share the same live receiver and copy limits. Falling back to SSH keeps file data on the source-to-destination route; relaying through your machine requires explicit --coordinate-at local.

The receiver signs what it did, and your machine verifies the receipt. A source cannot forge a clean account of destination changes. It can still omit files, invent content or metadata, or stop. A receipt does not prove the source supplied everything you intended.

The destination machine, receiver, and account remain trusted. Other authentication choices have different boundaries: broker-only authentication permits that destination account’s full authority during the session; full agent forwarding exposes your agent as ssh -A would.

Named receiving destinations

A named destination lets a server account request copies through an outbound connection maintained by your laptop. persist on enables this for syq’s SSH connections by default. Each request requires approval on the receiving machine through a desktop prompt or persist receive approve. Paths and limits are validated before prompting; the restricted filesystem executor checks every operation after approval. The server receives no SSH agent. Commands need separate approval.

Receiving names stay assigned to a persistent receiver public key. Reconnecting requires proof of the corresponding private key; knowing the public key does not let another receiver claim the name. This prevents accidental reassignment and impersonation through registration, but the server account controls the assignment files and can replace them. It is not a security boundary against someone who controls that account. The receiver private key stays on the receiving machine and does not grant SSH login access.

Approval permits that pending copy’s destination, overwrite policy, and limits. It does not authenticate what you typed on a remote server or attest to source contents. The receiving user and desktop session remain trusted. Request IDs are local, expire after five minutes, and cannot be reused. Disconnecting or stopping receiving cancels pending decisions. Desktop failure never approves a copy. syq persist receive on --approve always explicitly removes the per-copy decision and trusts connected server accounts for repeated copies.

The default starting directory is your home directory, with no containment. syq persist receive on --root DIRECTORY contains copies; syq persist receive off disables receiving while keeping ordinary persistence. A compromised connected server account can request more copies and invent their content. Once approved, it can inspect destination entries during copy planning and consume disk space within the approved limits. The laptop’s receiving account is trusted.

Receiver destinations require @name and fail if that receiver is offline. Bare names always identify SSH destinations, resolved through SSH configuration or DNS. A copy never switches routes after selecting its destination.

Limits to keep in mind

• Privileged copies need trusted destination directories. Do not copy as root into a directory writable by untrusted users. Syq writes private partial files. When reusing bytes from another partial, it only reads regular files owned by its effective user. It checks those bytes against source hashes and leaves the other partial unchanged. This does not make a shared writable directory trusted.
• Hard links share contents and metadata. In-place writes and metadata changes through a destination hard link affect every name for that file, including names outside the selected destination or a restricted grant’s path scope.
• Copies are not snapshots or transactions. Stop concurrent writers or use snapshots for consistent data. --inplace exposes incomplete updates. Syq does not fsync transfer data, so completion is not a power-loss durability guarantee.
• Preserving authority is a choice. Leave --preserve=ownership and --preserve=permissions off when copying from an untrusted source.
• Protocol assurance is still developing. Syq’s process protocol has not been fuzzed as extensively as rsync’s.

Code and transport integrity

File data is encrypted and authenticated by default. --tcp-plain sends file contents, protocol messages, and the worker authentication token in plaintext. An observer can steal the token and connect as a worker while the transfer is active; a network attacker can also alter traffic. Use it only on a network you trust. Downloaded code for remote operations and explicit self-updates is verified against a signed release manifest before use. That verification cannot protect a machine whose trusted account or programs have already been compromised.

Optional --integrity-checking transfer=blake3 checks detect accidental data corruption; they do not authenticate plaintext traffic because an attacker can replace both data and checksums. An expected whole-file digest supplied through a trusted channel checks the resulting file against that expectation. Use BLAKE3 or SHA-256 when resistance to malicious content substitution matters.

A server can also request a copy to another SSH host using a live receiving machine’s SSH access. Eligible copies discover that machine automatically; --auth-from @name chooses it explicitly. This always needs a local decision, even if automatic receiving is enabled. Refusing the request ends the attempt; syq does not try another authorization source. Approval authorizes a connection using the receiving machine’s SSH access and installation of a matching helper. The destination helper enforces one copy’s paths, permissions and limits. The server gets a restricted control stream and encrypted TCP worker access for that copy; it gets no SSH agent, private key, or command-running interface. Host trust and SSH configuration are those of the approving machine. The destination account remains trusted, including its interpretation of relative paths. A compromised source can substitute content within the approved scope, just as with other return copies.

Approved commands on receiving machines

syq exec uses an existing return connection and always asks for a local decision before starting a program. Command approval is separate from copy approval, including when copies are automatically approved. The prompt shows the server account, argument list and working directory. A server account can request commands from any of its processes; syq cannot establish what a person typed in a remote shell.

Approving execution grants the command your local user’s authority. Copy root confinement, file protection and transfer limits do not restrict that program. Scripts and build files can change what it does. Commands receive the local service environment and closed stdin. They do not expose a general SSH agent forwarding interface, but an approved program can access credentials available to the local user.

Disconnecting cancels the foreground process group; commands are never replayed automatically. Completed effects cannot be rolled back, and programs that create separate process sessions can outlive cancellation. See the command reference for execution and interruption behavior.

Human copy listings escape control characters in filenames. Diagnostics also escape terminal control sequences from peers; NDJSON keeps its JSON encoding.

Persistent connections

An open SSH login can be reused by other processes running as your local user without another key touch or agent approval. Persistence keeps that access available until the connection closes. syq persist off ends it; use syq persist receive off to stop incoming requests while keeping SSH reuse.

Persistence also enables receiving by default. Its approvals and copy limits are separate from the SSH login’s authority.

Isolated script scopes reuse SSH logins without enabling receiving. Stop background services when upgrading; replacing a binary does not change services already running it.

Command reference

Look up commands and options below. For copy examples, start with Copy files.

The advanced option groups have separate references:

• Performance tuning: workers, request sizes, and copy methods.
• Resource limits: bandwidth.
• Integrity checking: comparisons and checksums.

Angle brackets mark values you supply; square brackets mark optional arguments, and ... allows repetition.

syq

syq <COMMAND> [OPTIONS]
syq --self-update

Options

Help and version

syq –self-update

Update a standalone installation. For Homebrew and source installations, see installation updates.

syq --self-update

Options

Help and version

syq help

syq help [COMMAND...] [--help-all]

Show common help for a command or nested command, such as syq help persist receive on. Add --help-all for its complete help. With no command, show the root help. A trailing --help or -h selects common help.

Build identity

syq --build-identity prints the executable’s build identity. See development builds.

syq cp

Copy files, directories, and symlinks locally, over SSH, or with S3. See Copy files for examples and remote and S3 copy details.

syq cp --srcs-in project --to server --into backup --dry-run -v

Put source selectors and --mapping before --to or a placement option. For scripting, see environment variables and results.

File descriptors

Use a pipe or process substitution as an explicit local source. --src, --src-non-dir, and positional sources accept these inputs. Use --src-fd 0 to read stdin, or --as-fd 1 to send one file’s contents to stdout. The other endpoint can be a local file, an SSH file, or an S3 object:

gzip -c data | syq cp --src-fd 0 --to server --as data.gz
syq cp --src <(gzip -c data) --to server --as data.gz
syq cp --from server data.gz --as-fd 1 | gzip -dc > data
syq cp --src-non-dir incoming.fifo --into saved
syq cp data.bin --as-fd 3 3>received.bin

Each stream copy takes exactly one source. Reading a FIFO waits for a writer. A FIFO or descriptor path among several sources is an error before any files are copied, including when a shell glob expands to both files and a FIFO. A named FIFO has a basename: --into saved puts incoming.fifo at saved/incoming.fifo as a regular file. An inherited descriptor or process-substitution path has no usable name; choose --as PATH, --as-new PATH, --as-existing PATH, or --as-fd FD. Syq never uses a descriptor number as an output name. --src-fd replaces source paths and --from; --as-fd replaces --to and destination placement. Both together copy between descriptors. Put source arguments before destination arguments. A literal - is a filename.

Only explicitly selected local FIFOs are consumed. Recursive copies keep the normal special-file behavior, and --preserve=specials copies FIFO nodes instead of reading their contents. Ordinary source symlinks retain their usual behavior; use --follow-src to read a symlink to a FIFO. Descriptor paths such as /dev/fd/N and /proc/self/fd/N refer to the invoking process and are read locally before data is sent to a helper. Remote path sources must be regular files.

Descriptors must be inherited and open in the requested direction. Descriptor 2 is reserved for diagnostics. Dedicate each descriptor to the transfer. Reads and writes advance its current offset; append mode is respected. Syq preserves its blocking or nonblocking mode and does not truncate, rename, or apply metadata to a descriptor. Small native stream writes are forwarded without waiting for a full transfer block or EOF. No progress or summary is emitted, so stdout contains only payload when selected as the output.

Named file destinations are published after the complete input has been written and checked. Existing regular-file permission bits are kept; new files use 0666 filtered by the destination process’s umask. Ownership follows normal destination creation rules, and modification time is the time of writing. Source metadata is not copied. Parent directories are created if needed. The -new and -existing placement variants apply the same conditions as other copies: --as-new and --as-existing check the destination entry; --into-new and --into-existing check the container, not the file inside it. Placement conditions are checked before opening a named FIFO or transferring bytes. S3 new-object writes also refuse replacement if an object appears before publication. Use --cwd to resolve a relative pathname source, or --root to confine it beneath a directory (an S3 key prefix for S3 sources). An inherited descriptor already refers to an open object and cannot be confined with --root. Directly supplied symlink parents require --follow-src or --follow-dst; a destination’s final symlink is replaced, never followed.

EOF finishes input. Syq cannot distinguish a successful producer from one that exited early. In Bash, set -o pipefail reports a pipeline producer or consumer failure, but cannot undo a destination already published. Process substitutions run separately; check their exit status separately. An output descriptor can contain partial data after failure; require a successful syq exit before treating it as complete. A consumer closing early makes syq fail.

Stream copies use bounded buffers and apply backpressure. File transfers over SSH use a single SSH connection and the usual helper bootstrap and version checks. They do not support named receiving destinations, detached execution, restart recovery, directory selection, comparison policies, metadata preservation, dry runs, result records, statistics, or file-copy tuning. Unsupported options are rejected before transferring. S3 has its own part controls and limits.

A cancelled file upload removes its temporary file when cleanup completes; forced termination can leave a .syq-stream-* file in the destination directory.

syq cp [OPTIONS] SOURCE... [PLACEMENT]
syq cp [OPTIONS] --src-fd FD --as PATH
syq cp [OPTIONS] SOURCE --as-fd FD

Sources and filtering

Destination and mapping

Updates and deletion

Metadata and symlinks

Verification

Connections and remote execution

S3 connection settings

Performance and resource limits

Preview, progress, and results

Help and version

syq rm

Remove local files, remote filesystem entries, or S3 objects. See Remove files for selection rules and S3 removal for versioned objects.

syq rm --src-dir old-output --dry-run -v

For scripting, see environment variables and removal results.

syq rm [OPTIONS] PATH...
syq rm [OPTIONS] --srcs-in DIR

Copy policy and filtering

Sources and selection

Preview and output

Performance tuning

Progress and results

SSH and transport

Help and version

syq clean-partials

Remove syq partial files below local or SSH directory trees. Stop copies writing into those trees before deleting partials:

syq clean-partials --dry-run -v backup
syq clean-partials backup

See interrupted-copy recovery for which files this removes. Results use the rm record format.

syq clean-partials [OPTIONS] <TREE>...

Sources and selection

Preview and output

Performance tuning

Progress and results

SSH and transport

Help and version

syq map

Print local source/destination pairs as NDJSON for syq cp --mapping:

syq map --srcs-in photos > photos.ndjson
syq cp --mapping photos.ndjson -C photos --into archive

See Rename and reorganize for selection rules, transformations, and the record format.

syq map [OPTIONS] PATH...
syq map [OPTIONS] --srcs-in DIR

Sources and selection

Destination placement

Help and version

syq exec

Run a command on a connected receiving machine, with approval on that machine:

syq exec --on @laptop --cwd work/project -- cargo test

Put -- before the program and its arguments; use sh -c for shell syntax. The program runs with the receiving user’s permissions. See Run commands on your receiving machine for setup, output, and cancellation.

syq exec [OPTIONS] --on <@NAME> -- <PROGRAM>...

Arguments

Options

Help and version

syq rsync

Copy with rsync-style arguments:

syq rsync -av project/ server:backup/project/

Check Rsync compatibility before replacing a script. Trailing slashes follow rsync’s rules. Here -h means human-readable sizes; use --help for help. SYQ_RSYNC_OPTIONS supplies extra arguments.

syq rsync [OPTIONS] SRC... DEST
syq rsync [OPTIONS] [USER@]HOST:SRC... DEST
syq rsync [OPTIONS] SRC... [USER@]HOST:DEST

Copy policy and filtering

Preview and output

SSH and transport

Performance tuning

Resource limits

Integrity checking

Progress and results

Sources and selection

Help and version

syq persist

Manage reusable SSH connections and receiving profiles. Start with Send files home from a server for setup, or see Persistence details for profiles and limits.

syq persist <COMMAND>

Help (also available on subcommands)

syq persist receive

syq persist receive <COMMAND>

syq persist receive pending

--json prints structured requests. --wait waits for a request, up to --timeout seconds (default 30). Use the returned request ID with approve or deny.

syq persist receive pending [OPTIONS]

Options

syq persist receive approve

ID is a pending request ID from persist receive pending on this machine. Approval applies to that request only.

syq persist receive approve <ID>

Arguments

syq persist receive deny

ID is a pending request ID from persist receive pending on this machine.

syq persist receive deny <ID>

Arguments

syq persist receive on

Omitted settings keep saved values. New profiles ask for approval with desktop prompts and use the home directory. Set --root to confine copies to a directory. Changing settings cancels the profile’s active requests. See profile settings and copy limits.

syq persist receive on [OPTIONS]

Options

syq persist receive off

Disable the named profile, or all profiles when --name is omitted. This stops incoming requests while leaving forward SSH persistence available.

syq persist receive off [OPTIONS]

Options

syq persist receive remove

NAME identifies the saved profile to remove. Its connections and requests stop. The last profile can be disabled but cannot be removed; server-side name assignments remain until explicitly forgotten.

syq persist receive remove <NAME>

Arguments

syq persist receive status

Show all profiles, or select one with --name. Use --json for structured output.

syq persist receive status [OPTIONS]

Options

syq persist receive wait

HOST is the SSH server endpoint. Wait for the selected profile, or every enabled profile when --name is omitted. --timeout is in seconds; failure or timeout exits nonzero. This command does not start a connection.

syq persist receive wait [OPTIONS] <HOST>

Arguments

Options

syq persist destinations

syq persist destinations <COMMAND>

syq persist destinations list

syq persist destinations list

syq persist destinations forget

NAME is an offline receiving name to release before replacing its machine. Forgetting a live connection is refused.

syq persist destinations forget <NAME>

Arguments

syq persist destinations wait

NAME is a receiving name without @. Wait up to --timeout seconds for it to respond; timeout exits nonzero.

syq persist destinations wait [OPTIONS] <NAME>

Arguments

Options

syq persist connect

Connect to HOST and wait for receiving to be ready. --timeout applies to the receiving wait after SSH and helper setup. --syq-path and --no-bootstrap cannot combine. Use --pscope for an existing ephemeral scope.

syq persist connect [OPTIONS] <HOST>

Arguments

Options

syq persist on

syq persist on [OPTIONS]

Options

syq persist off

syq persist off [OPTIONS]

Options

syq persist status

Inspect connections without starting them. --json uses the connection-status contract. --pscope selects an ephemeral scope.

syq persist status [OPTIONS]

Options

syq receiver

Manage restricted destinations for direct server-to-server copies. See access management and enrollment details for setup and upgrades.

syq receiver <COMMAND>

Help (also available on subcommands)

syq receiver enroll

The destination’s parent must exist. Repeating enrollment updates the receiver; stop active copies before updating it.

syq receiver enroll [OPTIONS] <[USER@]HOST:DESTINATION>

Arguments

Options

syq receiver list

syq receiver list

syq receiver revoke

syq receiver revoke [OPTIONS] <ENROLLMENT-ID>

Arguments

Options

syq completion

Generate shell completion and manage cached endpoint suggestions. See shell setup to enable completion.

syq completion <COMMAND>

Help (also available on subcommands)

syq completion bash

syq completion bash

syq completion zsh

syq completion zsh

syq completion fish

syq completion fish

syq completion cache

syq completion cache <COMMAND>

syq completion cache list

syq completion cache list

syq completion cache forget

syq completion cache forget <ENDPOINT>

Arguments

syq completion cache clear

syq completion cache clear

S3 options and behavior

Use buckets with cp and rm: --from s3://BUCKET, --to s3://BUCKET, or --on s3://BUCKET. Selectors and placement options name keys within the bucket. See copy examples and removal examples.

S3 options

Syq uses your AWS credentials and detects AWS bucket regions automatically.

See S3 tuning for concurrency, part sizes, and retries.

Descriptor copies

With --src-fd or --as-fd, cp transfers one exact UTF-8 key’s raw contents, without path normalization, prefix selection, or syq file metadata. No local temporary file is created.

S3 descriptor copies default to four parallel 16 MiB parts, with about one part of payload buffering per worker plus one input/output part. The supported tuning keys are s3-part-size, s3-max-concurrent-parts-per-object, and s3-retries. Unknown-length uploads stop at 10,000 parts: 156.25 GiB at the default part size. Select a larger part size before starting a larger upload.

Buffered parts can be retried, but there is no restart recovery. Uploads replace the object only on completion. On failure, syq attempts to abort the multipart upload; unconfirmed cleanup may need provider tools. A lost completion response can mean an upload was published despite a reported failure.

Filesystem differences

• Buckets and prefixes: the bucket must already exist. Keys must be relative UTF-8 paths. A named source selects an exact object if present, otherwise its NAME/ prefix. Use --srcs-in for prefix contents. A prefix exists when it contains objects, including an empty directory marker.
• Metadata: syq stores timestamps, permissions, ownership, and symlinks in object metadata or contents. Downloads restore timestamps and symlinks; --preserve restores permissions or ownership. Special files are unsupported.
• Updates: --only-new, --into-new, and --as-new protect individual objects against concurrent creation. Prefix checks are not transactional. --inplace and SSH/S3 combinations are unsupported.
• Recovery: rerun an interrupted copy to reuse multipart work. If you abandon an upload, remove its unfinished parts with provider tools or a lifecycle rule.

Copies between S3 buckets

Bucket-to-bucket copies run within one service, using the same endpoint, region, and credentials. They preserve metadata and tags. Changes to tags, encryption, or storage class alone do not trigger a copy. Content hashing, expected digests, and --verify-only are unsupported on this route.

Versions and deletion

Ordinary rm and cp --prune retain historical versions in versioned buckets. Use rm --s3-all-versions to permanently remove selected versions and delete markers, or --s3-version-id ID for one version. Deleting a marker can reveal an older version. Preview version deletions with --dry-run -v.

Named removal selectors choose exact keys; --src-dir and --srcs-in choose prefix trees. An empty S3 source prefix is rejected by cp, so it cannot prune an entire local destination.

Performance tuning

--performance-tuning overrides syq’s automatic choices. To keep automatic choices within a ceiling, use resource limits instead. Leave performance tuning unset for everyday copies. These experimental controls are available in syq cp and syq rsync; syq rm and syq clean-partials accept only workers for filesystem removal.

Transfer controls

syq cp and syq rsync accept --performance-tuning. Supply comma-separated KEY=VALUE pairs:

syq cp large-file --to server --as /scratch/benchmark-copy \
  --performance-tuning workers=1 -v \
  --performance-tuning copy-path=ranges,request-size=1M,pipeline-depth=8

Sizes accept K, M, and G, using powers of 1024. Unknown keys, repeated keys, and out-of-range values fail the command. Overrides apply to the remote coordinator too and are not saved.

S3 copies

Use these keys for syq cp with S3 endpoints.

The concurrency limits are nested. For example:

syq cp data --to s3://backups --into archive \
  --performance-tuning s3-max-concurrent-objects=4,s3-max-concurrent-parts-per-object=8,s3-max-concurrent-requests=16

This allows four objects in progress and up to eight parts per object, with at most sixteen simultaneous data requests across them. These performance-tuning values fix the available slots and disable automatic adjustment of each specified count; unused slots can remain idle. To let syq choose counts within these ceilings instead, pass the same keys through --resource-limits. A count cannot be specified in both groups.

Part size grows when needed to stay within 10,000 upload parts. For server-side copies, an explicit part size also selects the multipart threshold, capped at 5 GiB. Without an explicit part limit, server-side copies can use the shared request budget’s full tuning range.

See S3 parallelism for memory use and buffering limits. S3 tuning is not saved between runs.

Remembered connection counts

Remote copies start from the last learned count for the same route, direction, and transport, or from 8 workers over SSH and 16 over TCP. Successful copies update the cache after comparing enough worker counts. Short copies may finish before syq learns a better count.

The cache is ~/.cache/syq/tuning.json; XDG_CACHE_HOME changes its parent. SYQ_TUNING_CACHE names another file, or disables the cache when empty. Supplying --performance-tuning bypasses the cache. With --resource-limits workers=N, syq starts from the remembered count or the ceiling, whichever is lower, and leaves the cache unchanged. A bandwidth limit alone still reads and updates it. Live tuning continues unless you fix workers. Use -vv to see the starting count.

Filesystem tuning examples

Use disposable destinations when comparing settings. Larger requests and deeper pipelines can increase memory use. copy-path=ranges disables small-file batches and whole-file shortcuts, including local kernel copying and APFS cloning.

Scattered edits in existing files

Smaller comparison blocks can reduce the data sent for scattered edits, at the cost of more hashes and requests:

syq cp --srcs-in source --to host --into destination \
  --performance-tuning comparison-block-size=64K,request-size=4M

Set request-size too: it otherwise follows the comparison block size. At 64 KiB, files must be smaller than 130 GiB or comparison fails. Increase comparison-block-size for larger files; doubling it doubles that limit. Both endpoints still read the full file to compare it.

In syq rsync, -B / --block-size selects the comparison block size. Do not combine it with comparison-block-size.

Streaming and request windows

Syq normally streams remote ranges above 16 MiB, with blocks of at most 2 MiB. copy-path=streaming forces streaming and disables whole-file and small-file shortcuts. copy-path=auto-streaming keeps those shortcuts and streams the remaining ranges.

An explicit request-size also sets the streaming block size; bandwidth and receiver limits may reduce it. Setting pipeline-depth disables automatic streaming and cannot combine with either forced streaming mode.

To compare pipeline depths, hold the worker count and request size fixed:

syq cp data.bin --to host --as /scratch/pipeline.bin --performance-tuning workers=1 -v \
  --performance-tuning copy-path=ranges,request-size=1M,pipeline-depth=4

Repeat with pipeline-depth=8 and 16, using a fresh destination each time. These allow up to 4, 8, and 16 MiB of outstanding requests per endpoint per worker. Compare with the defaults too; larger windows may add memory use without improving speed.

Batch size and splitting

Tune small-file batches with batch-files and batch-bytes:

syq cp --srcs-in small-files --to server --into /scratch/benchmark-small \
  --performance-tuning workers=1,batch-files=256,batch-bytes=8M -v

Explicit batch settings replace the small-copy shortcut with worker batches. With a bandwidth cap, each batch contains at most one file. Batch controls cannot combine with copy-path=ranges or copy-path=streaming; auto-streaming accepts them.

Remote new-file copies can batch files up to the smaller of request-size and batch-bytes. Larger limits allow more data to be held in memory; interrupted whole-file copies restart from the beginning. On macOS, files above the batching limit can use APFS cloning. That limit is the smallest of the comparison block size, batch-bytes, and request-size.

split-min-size controls how small a region an idle worker can take from another worker. Lower values allow finer sharing; higher values reduce assignments. Splits align to comparison blocks and need twice the minimum remaining size.

Average rate and burst patterns

With a bandwidth cap, bw-pacing controls when data is sent:

• 125ms (default): send smaller requests at regular intervals. The first request starts immediately, so short copies can exceed the average rate.
• average: wait for each request’s byte budget before sending it. A 2 MiB request at 1 MiB/s waits about two seconds, then can arrive in a burst.

Streaming and transport buffering can also produce bursts. Restricted receivers apply their authorized rate and request-size limits.

Recording a comparison

Use the same reporting options and fresh destinations for each run. Prefer -v; --stats can change which copy optimizations run. With overrides, -v reports effective settings and a final syq: tuning observed: diagnostic. Check elapsed time, exit status, and copied contents. See Speed for the benchmark script.

Resource limits

--resource-limits sets ceilings while syq chooses how to run the copy. --performance-tuning fixes individual settings instead. Both are available in syq cp and syq rsync; S3 copies use syq cp.

Supply comma-separated KEY=VALUE pairs:

syq cp data --to server --into backup --resource-limits bandwidth=10M

This limits the average copy rate to 10 MiB/s. It counts bytes before compression, encryption, and protocol overhead; it does not cap every network burst.

Concurrency ceilings

syq cp data --to server --into backup --resource-limits workers=4,bandwidth=10M

Syq can adjust the filesystem copy-worker count up to four. By contrast, --performance-tuning workers=4 fixes four worker slots. Unused slots can remain idle in either case. Worker counts do not include directory scanning, metadata processing, or control connections, and do not cap total threads, sockets, CPU use, or memory.

Each concurrency key conflicts with the same key in --performance-tuning, even when the values match. Controls for different quantities can combine: for example, a fixed request size with a worker ceiling. Unknown keys, duplicate keys, zero counts, and out-of-range values are rejected.

A ceiling does not force syq to use that many slots. Filesystem copies have no default tuning ceiling; syq adjusts the count from measured throughput. Restricted receiver limits still apply. S3 ceilings constrain the route’s normal automatic range without raising it. They are nested: object and per-object part counts also share the aggregate request ceiling. A fixed setting for one count still operates within ceilings on the other counts.

Use workers only for filesystem copies and the S3 keys only for S3 copies. Limits apply to the remote coordinator too and are not saved.

Bandwidth units

Rates accept decimals and case-insensitive suffixes:

A final +1 or -1 adjusts the scaled value by one byte before rounding. Rates are rounded to the nearest KiB/s. Zero disables the cap; nonzero values below 512 bytes/s are rejected. For example, 1024 and 1M both select 1 MiB/s.

Where the limit applies

For filesystem copies, workers share one aggregate limit. Buffering, streaming, and SSH/TCP overhead can produce bursts. bw-pacing controls pacing. A capped copy uses normal copying instead of local filesystem cloning.

For local/S3 copies, the limit applies to scheduled data, with upload bursts up to a part. Server-side S3 copies do not pass object bodies through this machine and are not paced by this setting.

With a worker ceiling, syq uses the remembered connection count up to that ceiling and leaves the cache unchanged. A bandwidth limit alone still reads and updates it. Live tuning runs unless you fix workers through --performance-tuning; see remembered connection counts.

In syq rsync, --bwlimit RATE selects the same rate limit. Do not combine it with --resource-limits bandwidth=RATE.

Integrity checking

--integrity-checking controls how syq cp and syq rsync compare contents and check transferred data:

Supply comma-separated KEY=VALUE pairs, or repeat the option with different keys. Comparison and transfer algorithms may differ:

syq cp data --into backup --integrity-checking compare=blake3,transfer=sha256

BLAKE3 and SHA-256 are cryptographic hashes. MD5 supports existing manifests; XXH3-128 is a noncryptographic checksum. Use BLAKE3 or SHA-256 with an expected digest from a trusted source to check authenticity; see code and transport integrity.

Comparison

Syq normally skips files whose size and modification time match. syq cp compares whole seconds exactly and ignores as many trailing fractional digits as are zero in the destination timestamp. For example, destination .120000000 seconds matches source .123456789; a whole-second destination timestamp ignores the source fraction entirely. This accommodates destinations that truncate fractional seconds. Directory metadata previews use the same fractional precision rule. syq rsync compares whole seconds only when checking file contents.

A timestamp difference outside that precision triggers checking even when the source is older, unless you request --skip-newer. Use --hash to check contents even when size and timestamp match:

syq cp --hash --srcs-in project --into backup

--hash is shorthand for compare=blake3. It conflicts with a different explicit comparison choice. In rsync syntax, use -c or --checksum for the same BLAKE3 comparison.

Payload checks

Extra payload checks default to transfer=off. Enable them with, for example, --integrity-checking transfer=blake3.

SSH and encrypted TCP retain their transport authentication independently. --tcp-plain does not enable payload checks automatically, and checksums do not authenticate plaintext traffic.

For a complete check of a local copy, use an expected digest.

For local/S3 copies, provider request checksums remain enabled. The transfer algorithm records a whole-file digest on upload and checks stored digests on download when present. Use an expected digest for objects without a stored digest. See S3 metadata and integrity.

Server-side S3 copies preserve stored digests without reading or verifying object bodies. They do not support content-hash comparison, extra transfer hashing, expected digests, or --verify-only.

Expected digests

To require a particular whole-file digest, use --expected-hash ALGORITHM:HEX with one named regular file:

syq cp data.bin --as backup.bin --expected-hash md5:900150983cd24fb0d6963f7d28e17f72

This checks all resulting bytes, including reused data, before reporting success. When size and modification time match, syq validates the existing destination and skips copying if its digest matches. Otherwise it copies and validates the result; a mismatch fails that file. With normal staging, validation happens before replacing the destination. With --inplace, the file has already been modified when validation finishes. Use per-file mapping expectations for a batch. Selection filters still apply. The expected digest’s algorithm can differ from either integrity-checking hash type. Dry runs preview changes without validating the expectation.

The algorithms are blake3, sha256, md5, and xxh3-128. Supply 64 hex digits for BLAKE3 or SHA-256, and 32 for MD5 or XXH3-128. In syq rsync, use --syq-expected-hash ALGORITHM:HEX.

Compare without copying

To compare without writing, use --verify-only:

syq cp --verify-only --srcs-in project --into backup

This compares file contents, symlink targets, and entry types without writing. Missing or different entries make the command fail. It does not compare metadata or look for extra destination files.

For two servers, add --coordinate-at local to compare through your machine using ordinary SSH access, with no restricted receiver enrollment. This also supports --results. See remote verification.

--verify-only cannot combine with --dry-run, --prune, --inplace, or an overwrite policy. Filters and size limits still select what is compared; special files require --preserve=specials. In rsync syntax, use --syq-verify-only.

For files being changed by another program, stop the writer or copy a snapshot.

Remote copy reference

See syq cp and syq receiver for the option lists.

For the usual setup and copy commands, start with Copy between servers.

SSH configurations

Host-certificate-only trust, KnownHostsCommand, and RevokedHostKeys are unsupported by the broker. Custom known-hosts paths must be unambiguous: one absolute, whitespace-free filename per configured user/global directive. The default known-hosts file list works. Syq reports unsupported configurations instead of relaxing host verification.

Your local SSH configuration selects hostB’s login, address, port, and trusted host keys. HostA’s SSH configuration does not override those choices.

The constrained authentication broker admits at most 129 simultaneous clients by default, independently of automatic copy-worker tuning. An explicit worker setting or ceiling adjusts that bound to the requested count plus one control connection. Command-restricted copies never exceed 129 broker clients.

Enrollment

Enrollment needs normal command authority on the destination during setup. Use a real directory, not a symlink. Transfers cannot overwrite the receiver’s SSH configuration, programs, or enrollment state. Manage that state with syq receiver commands; it is not a disposable cache.

Repeating enroll updates the receiver to match your local build. A pending enrollment can be retried or revoked. Revoke and enroll again to rotate its receipt key. Revocation stops active receivers before removing their state; see access management for interruption and retry behavior.

Stop active copies before upgrading: replacing the executable does not update running receivers. Repeat syq receiver enroll hostB:/destination afterward to refresh the installed receiver. Compatible enrollment keys and replay records are preserved. Different client builds cannot share one installed receiver concurrently; each needs a matching receiver.

An incompatible enrollment requires fresh setup, which needs ordinary SSH access. Eligible copies install it automatically, or you can use enroll. Incompatible old enrollments are not listed or removed by the new build; they require manual cleanup on both machines.

Enrollment detects the destination platform and installs a matching executable. Source builds require compatible platforms; --syq-path does not select the restricted receiver. See Developing syq for testing source builds.

Limits and unsupported options

Default limits are 100 million entries and 8 TiB of file data. Override them for one transfer with --receiver-max-entries N and --receiver-max-bytes SIZE. Values can raise or lower the defaults within the allowed ranges. The transfer must start within 24 hours of authorization and finish within seven days of authorization. New authorizations allow up to 128 worker connections, including for named destinations; a smaller worker setting or resource limit lowers that allowance. Existing authorizations keep their original limits.

Restricted copies use encrypted TCP when reachable and otherwise use SSH workers from the source to the destination. Both transports share the same copy authorization, limits, revocation, and signed receipt. A failed direct connection never selects a relay through your machine; choose --coordinate-at local explicitly to send data through it.

Signed results

The destination signs a receipt and your machine verifies it before reporting results. Use -v for totals or --results FILE for automation. These results arrive after verification, rather than as live per-file progress. For --dry-run --results, use --coordinate-at local to get the preview stream.

--receiver-receipt digests adds BLAKE3 hashes of affected regular files. Receipts allow up to four million records and 512 MiB of plaintext. Reaching a cap stops further changes and reports an incomplete outcome.

A receipt does not prove that the source supplied every intended file or the right contents. See the threat model.

Other authentication modes

To make hostB pull from hostA using credentials already on hostB:

syq cp --coordinate-at dst --peer-auth own-credentials --from hostA data --to hostB --into /archive

There is no restricted source receiver, so direct pulls require one of these alternatives to the default authentication.

Detached copies

--detach leaves the copy running after your command exits. It requires --peer-auth own-credentials or an explicit --rsh policy, so the coordinating server can authenticate independently. It cannot use the restricted receiver or --results.

Save the reported remote log location. The log is not a signed receipt. If printing the location fails, the command reports an error but the job may still be running. The coordinating server needs /bin/kill and either setsid or perl.

Authorization selection

For syq cp with local sources and an SSH destination, --auth-from auto tries live receiving machines in alphabetical order, allowing up to two seconds for each reply. Offline or unsupported connections are skipped. With none available, or with unsupported options, it uses the source machine’s SSH access. Once approval is requested, refusal or failure ends the attempt.

--auth-from @NAME and its alias --via @NAME require that receiving machine to authorize the copy. --auth-from ssh uses the source machine’s SSH access. These options choose authorization, not the destination: --to host names an SSH destination, while --to @NAME sends files to a receiving machine.

Authorization through a receiving machine does not support --detach, custom --rsh or --syq-path, --no-bootstrap, --pscope, alternative --peer-auth or --coordinate-at, --no-tcp, or --tcp-plain. It requires direct encrypted TCP from source to destination. Destination completion does not request permission through a receiving machine; use --auth-from ssh for completion through the source’s own SSH access.

On this route, quoted ~ and ~/archive select the destination account’s home directory. Use ./~/archive for a literal directory called ~. Avoid ~//archive: explicit receiving authorization keeps it under the home directory, but automatic selection uses ordinary SSH, where it resolves to /archive.

Verification

For comparisons between two servers, --coordinate-at local uses ordinary SSH access from your machine to both endpoints. It needs no restricted receiver enrollment and supports --verify-only --results FILE. Files are hashed on the servers; your machine compares their hashes and receives listings and results, not the full file contents.

Direct restricted verification requires an existing enrollment and cannot produce --results; a receiver receipt cannot attest to the source’s comparison. Verification never installs an enrollment.

--verify-only cannot combine with --dry-run, --prune, --inplace, or overwrite policies. Filters and size limits select the entries to compare; special files require --preserve=specials. Metadata is not compared, but device identity is. A requested results file may still be written, and remote setup may still cache the helper or install syq.

Persistence details

See syq persist for the option list.

For everyday setup, start with Send files home from a server.

Names and profiles

Give different receiving locations their own names:

syq persist receive on --name laptop --cwd ~
syq persist receive on --name project --root ~/work/project
syq persist connect server

Both names work from the same server account: syq cp results --to @project and syq cp report.pdf --to @laptop. Each profile has its own directory, copy root, limits, approval policy, and background connection to each connected server. New profiles start with the usual defaults, including asking for approval; they do not inherit another profile’s trust or confinement settings. Up to 32 profiles can be saved.

receive on --name NAME creates a profile or updates that name’s settings. Without --name, receive on updates the first saved profile, shown first by receive status. The initial hostname profile becomes a saved profile when persistence first connects; adding a new name then keeps that original profile. If no preferences have been saved yet, the first explicitly chosen name replaces the implicit hostname default.

syq persist receive status
syq persist receive status --name project
syq persist receive off --name project
syq persist receive on --name project
syq persist receive remove project
syq persist receive wait server --name laptop --timeout 30

Updating, stopping, or removing a profile cancels only that profile’s copies, commands, and pending approvals. Other profiles keep working. receive off without a name stops all profiles; receive on enables the first profile, and receive on --name NAME enables another. Removing the first profile makes the next saved profile the default. The last profile can be disabled but cannot be removed. pending, approve, and deny work across all profiles; prompts name the receiving profile. Without --name, receive wait waits for every enabled profile on that server.

Names belong to a server account and stay assigned to their original receiving machine when it disconnects. Another laptop cannot claim the same name, even while the original is offline. Stopping receiving or removing a local profile does not release its server-side name. Other profiles remain usable.

To replace a laptop, stop its receiving connection, then run syq persist destinations forget laptop on the server. Run syq persist connect server on the replacement laptop to claim the released name. A rejected connection needs this explicit retry. Forgetting a live connection is refused.

Syq generates one receiver key per local account, shared by receiving profiles and syq versions. It lives in ~/.syq-receiver-identity/identity_ed25519 and is independent of your SSH login keys and profile settings. Keep that directory across upgrades; include it in private backups if you want to restore the same identity. Losing it requires releasing the old names on each server. Copying it to another machine gives that machine the same receiver identity. If the old connection is still responsive, the replacement is rejected; use different profile names to receive on both machines at once. An unresponsive connection must time out before its name can reconnect.

Directories

--cwd chooses the starting directory; --root also contains copies within it. Both require an existing directory with a UTF-8 path. Invalid settings leave the saved configuration unchanged. If a saved directory disappears, you can still inspect, disable, or reconfigure the profile. Filenames inside it may use normal Unix filename bytes.

Copy limits

Copies support directories, symlinks, modification times, filters, hashing, resume, mappings, --preserve=permissions, --verify-only, and the overwrite policies. Ownership and special-file preservation, --inplace, and --min-size are unsupported. Timestamp comparisons trust the source’s reported modification times.

Each copy is limited to 100 GiB and one million touched entries by default. Change these ceilings with syq persist receive on --max-bytes 20G --max-entries 100000. Lower limits requested by the sender also apply. Limits are per copy; repeated copies can fill the disk. Copies support at most 128 workers each.

Pruning is disabled unless the laptop sets a positive --max-delete. A sending --prune command must also supply its own --max-delete ceiling, no higher than the laptop’s. Validation failures leave the copy unstarted. Errors during copying fail visibly and may leave partial files for retry. The sender verifies a signed receipt before reporting success.

Connection lifetime and waits

persist on enables persistence for subsequent syq SSH connections. persist connect server enables it and connects immediately. With receiving enabled, it waits until receiving is ready. --timeout 30 limits that wait after SSH and helper setup, not authentication or installation. Failure leaves persistence enabled; a healthy connection is reused without cancelling requests.

Connections have no idle expiry. Receiving reconnects after interruptions; other SSH logins reopen on their next use. Syq installs no login service, so connect again after reboot. Copies are not queued or retried automatically.

Use syq persist receive wait server --timeout 30 to wait without starting or restarting a connection. On the server:

syq persist destinations list
syq persist destinations wait laptop --timeout 30
syq persist destinations forget laptop

forget releases the name assignment while its connection is stopped. For structured status and approval requests, see connection status.

Isolated script scopes

syq persist on --ephemeral prints a scope path. Pass it as --pscope PATH to syq persist connect server and subsequent copy commands, then close it with syq persist off --pscope PATH. This does not change your user setting.

These scopes reuse SSH logins only. They do not enable receiving, authorization through your machine, or commands on it. Idle connections close within ten minutes, including helper sessions. Explicitly closing the scope ends reuse immediately. For return copies or commands, use persistence without --pscope.

Setup and recovery

Linux desktop prompts need libnotify 0.7.10 or later and a running notification service. macOS uses a native dialog. Start receiving from a terminal in your desktop session. After changing sessions, run syq persist receive off, then enable the profiles you need from a terminal in the new session.

syq persist receive pending shows complete requests and prompt errors. Overwrite warnings are advisory; destination entries can change before copying. --notify off selects terminal approval; --notify desktop restores prompts.

Start the connection from your laptop with syq persist connect server. Opening a plain SSH session does not enable receiving, and connecting onward from that server to another host does not carry your laptop’s receiving name with you.

For automatic reconnection to work, your SSH key or agent must be available and the server’s host key must already be trusted. The background service cannot ask for a password. The server must allow remote Unix socket forwarding; OpenSSH 9.2 also requires permission for remote TCP forwarding.

The server command uses the matching helper installed by the receiving connection. If that helper is missing or does not recognize an option, update syq on both machines and reconnect from your laptop.

Updating connections

Before upgrading, run syq persist off on the receiving machine to stop its background services. Replacing the executable alone does not update running services. Close script scopes with syq persist off --pscope PATH too. After upgrading both machines, run syq persist connect server for each server.

Saved names, directories, and limits carry over. Settings predating approval prompts require approval after upgrading. Older binaries may not read updated preferences; use the newer binary to manage receiving.

Connections created before persistent receiver identities remain discoverable. Their names become assigned when an updated receiving machine reconnects. Updated commands verify assigned names even when their connection uses an older helper; a receiver without a matching identity is rejected. Older binaries do not enforce these assignments, so use updated syq commands on the server and stop older receiving services before switching versions. The receiver key and assignments are independent of the helper build and survive later upgrades.

Automation results

Native cp and rm can write structured outcomes alongside human output:

syq cp --srcs-in project --into backup --results copy.ndjson
syq rm old-output --results removal.ndjson

Each line is a JSON record. Require a final result record before trusting completion. EOF without one means the outcome is unknown, even if no error record appeared.

This page describes stream semantics. The JSON Schema lists precise field shapes; example streams show complete runs. Mapping manifests are a different format.

The channel

--results FILE creates a fresh regular file and refuses an existing one. Use a new name for each run and keep it outside trees being copied or removed. --results-fd N uses an already opened writable descriptor above 2:

syq cp project --into backup --results-fd 3 3>copy.ndjson

Use a descriptor for pipes or other non-regular sinks. Named results paths follow the normal native symlink rules; --follow permits traversal. Human stdout and stderr are not part of this contract.

Results are written on the invoking machine, including for remote removal. For direct restricted remote-to-remote copies, they are derived from the verified destination receipt and marked provenance: "receiver_attested". Those records arrive after receipt verification. Otherwise remote-to-remote results require --coordinate-at local, which relays data through your machine. Remote-to-remote --dry-run --results and --verify-only --results also require that local route.

--results supports native cp (including pruning) and rm. It cannot be combined with --detach. The restricted receiver supports copy changes, including capped pruning, but not native removal.

Records arrive in seq order; error and terminal records are flushed immediately. If writing the stream fails, syq warns where possible and stops writing it. Filesystem work can continue, leaving an incomplete stream.

Argument errors exit 2 without a stream. Failure to open the results sink exits 1 without a stream. Once opened, completed runs and fatal setup failures emit a terminal record if the sink remains writable. Crashes and interruptions can leave it missing.

Consumer rules

• Check the terminal record and process exit code; a mismatch is a protocol error. Use terminal totals, not progress or a count of operation records.
• Build mapping retries only after terminal success or partial. Other statuses can leave entries unresolved.
• Ignore unknown record types and optional fields within a supported schema version.
• Reject unknown schema, schema_version, terminal status, or path encoding values.
• Treat human message text as display only; parse structured fields.

--progress-json is a separate progress display whose format may change. Use --results when you need a stable consumer contract.

Record envelope

Every record carries:

Paths are tagged: {"encoding":"utf-8","value":"docs/a.txt"}, or encoding: "base64" with standard base64 of raw filename bytes. Byte and count fields are non-negative integers within the u64 range.

Record types

run

Always first. Identifies the invocation with run_id, started_at (Unix seconds), syq_version, mode (cp or rm), dry_run, and endpoints. Endpoints identify role and kind (local, ssh, or s3). SSH endpoints carry host/user; S3 endpoints carry host: "s3://BUCKET" without a user. They omit credentials, headers, ports, and raw command arguments. Clients that predate S3 must be upgraded to decode S3 endpoint kinds; local and SSH streams retain their existing representation.

Copy runs also carry prune and mapping. Compare-only runs add optional verify_only: true; absence means false. Removal has one source endpoint regardless of selector count and omits those copy fields.

With --verify-only, differences and inspection failures produce error records and a nonzero terminal status. Matching regular files count as files_unchanged and bytes_unchanged; transfer and creation totals remain zero. Successful comparisons do not emit copy operations. Progress bytes measure comparison work, not bytes written. As with dry runs, use --coordinate-at local for JSON comparison results between two remote hosts; a receiver receipt cannot attest the source’s comparison claims.

progress

Sampled telemetry, approximately once per second, for displays rather than accounting. It includes bytes, files, exclusions, scan state, and elapsed time. Removal has zero byte and unchanged/excluded counts; its file counts reflect outcomes received so far. The terminal record owns final totals.

With --stats (or debug logging), copy records also include an optional activity object. --results alone does not collect these measurements. Copies that reach transfer completion emit a final progress sample before their terminal result, including for short copies. Older producers may omit activity; consumers should accept that. The Python SDK exposes it as ProgressEvent.activity. A rejected telemetry request warns and leaves the copy running. If the subscription loses its connection, syq reconnects without requesting further remote telemetry; local measurements remain available.

Worker fractions divide each state’s accumulated duration by observed_ns, including waits still in progress. They sum to one when activity was observed; an empty object means none was observed. cumulative_fractions uses all observed worker time since collection began. Both exclude tuner parking; parked_ns and cumulative_parked_ns report that time separately in nanoseconds. These are fractions of worker time, not fractions of command elapsed time. observed includes retired workers; active excludes retired workers and workers parked by connection tuning. awaiting_work counts workers waiting for a scheduler job. Compare this with scan_done to investigate insufficient work supply.

source_request and source_response cover sending a request and awaiting its response. destination_send and destination_ack cover sending writes and waiting for replies; synchronous local work can occur inside these calls. pacing covers bandwidth-limit waits. other_work is remaining worker activity. A source-response wait alone cannot distinguish storage, CPU, transport or downstream backpressure.

Endpoint actors separate filesystem operations, server communication and Linux read-ahead helpers. cumulative_actors reports the same measurements since the connection subscribed, including when its latest sample has not advanced. Labels include worker IDs to distinguish connections. Their fractions use each actor’s own observed time; do not add fractions across actors or to worker fractions. source_read measures demand-read syscalls, including reads used to hash existing destination data; hashing measures content hashing; destination_write measures write syscalls. filesystem_copy keeps a combined measurement where filesystem copy operations do not separate reads and writes. handling covers remaining filesystem request work. Server request_wait covers its request queue and response_send covers serialization and writing replies. Server idleness does not by itself prove a transport limit.

prefetch_advice measures Linux read-ahead advice calls; its bytes are requested bytes, not bytes physically read. prefetch_fence measures waiting for outstanding advice before releasing or shrinking a range. Read and write byte counters count completed operations; filesystem-copy bytes are logical bytes and may include cloning or offload. Helper helper_cpu is a subset of its process CPU, not additional CPU. Live Linux helper CPU uses the kernel clock-tick resolution, so short intervals can report zero. Process identities are temporary identifiers for this run. Process cpu is the interval delta; cumulative_cpu is the delta since collection began for that process.

Remote reports arrive at response boundaries and on connection retirement. sample_age_ms measures time since receipt, so network delivery delay is additional and clocks on different hosts need not agree. state_at_sample is the state at that report. When no newer report arrives, endpoint elapsed_ms is null and actors is empty; process CPU deltas are null too. A blocked remote call can therefore leave stale evidence. Endpoint intervals can span several coordinator intervals. The first remote CPU sample establishes a baseline and has no delta.

tcp and tcp_delta describe the coordinator end of a TCP socket; peer_tcp and peer_tcp_delta describe the remote end at its reported sample. The raw counters include RTT and retransmissions; deltas include receive-window-limited and send-buffer-limited durations. Local counters are sampled by the progress ticker; the final reading is retained after retirement. Null means unavailable, whereas zero is a measurement. See the schema for the complete field definitions. Writes retain the usual completion semantics and do not wait for durable storage.

trace

One intended copy change in a dry run. Includes action, destination, source for mapping entries, kind and bytes where applicable, plus a reason: destination_missing, type_differs, content_differs, metadata_differs, or destination_only.

A trace cannot be matched by identity to a later live operation: the filesystem may change between runs.

operation_result

An outcome for a completed copy change or a failed mapping entry.

In attested records, dst is relative to the destination area identified by scope. An attested set_metadata omits kind.

Unchanged and excluded entries have totals only. Ordinary live streams do not emit per-operation records for metadata-only updates, though dry runs emit metadata_differs traces. Failed implicit parent creation can lack src and is non-retryable. Do not construct a retry source from its destination name.

Removal records

For S3 version removal, removal_trace and removal_result include optional s3_version_id (string) and s3_delete_marker (boolean) fields. Each record and entry count refers to one version or delete marker. Ordinary S3 removal omits these fields.

selection_result uses a zero-based selector index, the original path, status (resolved or missing), and kind when resolved. Missing selectors succeed. Overlapping or duplicate selectors keep separate indexes. If a later selector cannot resolve, earlier selection records may precede the fatal error and terminal record.

removal_trace includes selector, path, kind, and disposition: "would_remove". Directories follow their descendants.

removal_result includes selector, path, kind when known, attempts, and disposition: removed, already_absent, or failed. Already absent is success. Failures include error details and retryability where available, and also produce a counted error record. Dry-run inspection failures can produce failed removal results, but never successful removal results.

error

One per counted error. message is display text; class and os_kind are provided where known. Classes are io, transport, conflict, integrity, safety_limit, usage, and internal.

OS kinds are not_found, permission_denied, already_exists, invalid_input, no_space, quota_exceeded, read_only, and other. They preserve OS error meaning across hosts without requiring matching errno numbers. Receiver refusals use class: "safety_limit" with provenance and the receiver’s code.

final_state

Attested streams only: the destination’s final observation of a path the transfer could have changed. Includes scope, dst, and an object: absent, an observation failure, or present with kind, size, applicable metadata, and symlink target. With --receiver-receipt digests, regular files also have a BLAKE3 digest.

Object kinds distinguish directories, files, symlinks, FIFOs, sockets, character/block devices, and other objects. Metadata fields are mode, uid, gid, mtime, mtime_nsec, and rdev. Consult the schema for exact shapes. This observes destination state; it does not attest source completeness.

result

Exactly one terminal record, always last when the stream completes. Common fields are status, exit_code, dry_run, errors, and elapsed_ms.

Copy terminals may also include copying_elapsed_ms: the wall-clock span from first file work to last completed file work across workers, including per-file checks, finalization and gaps. Initial setup before file work is excluded; planning and connections can overlap this interval. It is not a sum of worker times or pure network time. The field is absent when no bytes moved or the coordinator does not supply it, including older releases and attested terminals. Use elapsed_ms for end-to-end throughput comparisons.

Copy totals include transferred/unchanged/excluded files, created directories, symlinks and specials, transferred/unchanged bytes, and on pruning runs deletions_planned, deletions_completed, and deletions_blocked. A fatal failure reports what finished before it stopped. Dry-run totals describe planned work, not committed changes.

Attested terminals add receipt_status (clean, failed, or incomplete), provenance, and receipt counts. They can attest only what the receiver saw:

• Unchanged and excluded totals are zero, not a claim that every source entry changed.
• Only deletions_completed appears; planned and blocked deletion totals are omitted.
• errors counts attested error records for failed/incomplete operations, refusals, and failed or partial final-state observations.

Removal terminals have mode: "rm" and totals for selectors, resolved and missing selectors, planned/removed/already-absent entries, and failed entries. Live removal leaves entries_planned zero. Dry removal leaves entries_removed and entries_already_absent zero, but inspection failures may increase entries_failed.

The human summary uses the same terminal totals.

Exit codes

Removal terminals use only success, partial, or failed. A results-sink startup error also exits 1 without a stream.

Compatibility

Within a schema version, required fields keep their types and meanings; existing types, actions, dispositions, statuses, classes, and reasons are not renamed or reused. New record types and optional fields may be added. Human messages may change at any time.

Connection status

syq persist status --json reports the persistence setting, scope, and endpoints without starting connections. Endpoint states are starting, connecting, ready, reconnecting, failed, or inactive. Each entry also reports whether SSH is connected and the receiving state and errors.

With receiving enabled, ready means the return connection is online; SSH can reconnect on its next use. If receiving preferences cannot be read, SSH entries are still listed, receiving_error explains the failure, and each entry’s receiving_enabled is null.

Command approvals in syq persist receive pending --json use kind: "command" and include argv, cwd, and permission. Argument and directory strings in this summary are escaped for display. Use an up-to-date syq binary to inspect and approve commands; clients that only support copy requests omit them.

Rsync compatibility

See syq rsync for the option list.

syq rsync accepts common rsync commands for local copies, pushes, and pulls. It uses its own protocol: the remote program must be syq, not rsync. Run syq rsync --help-all for all accepted flags.

syq rsync -av project/ server:backup/project/

Differences to check before switching

Scan and copy errors prevent deletion. A destination that contains its source on the same host cannot be pruned. Preview deletion scope with --dry-run -v; see deletion rules.

The compatibility command uses rsync’s default size-and-whole-second timestamp quick check. Native syq cp also compares fractional seconds at the precision suggested by the destination timestamp; see timestamp matching. Use -c to compare contents when size and timestamp match; source timestamps are preserved, so ordinary clock skew does not require the source timestamp to be newer.

Syq uses numeric IDs and always keeps partial files, so --numeric-ids and --partial are accepted no-ops. -P enables progress. Compression is on by default; -z does not enable anything extra. -B / --block-size changes syq’s comparison and reuse block size; see the tuning table for its default and allowed range. Values outside that range are rejected.

Unsupported features

Unsupported common flags are rejected with an explanation. Selected symlink targets are copied unchanged. --insecure-links does not enable any of the unsupported descendant-link options.

File-list parsing and help

--files-from cannot combine with syq ignore rules or deletion. A listed source whose parent is a symlink fails that entry with exit 23, without creating its implied destination parent. --insecure-links only relaxes the ownership check on symlinks in typed local paths; it does not allow symlink traversal beneath a selected source, including entries from --files-from.

Other parsing differences:

• .. components are rejected rather than clamped at the source root.
• dir/ and dir select the same entry. Neither selects contents unless -r is explicitly supplied; -a alone does not enable recursion here.
• . and / are rejected as root entries.
• Use --from0 for NUL separators; -0 is unsupported.

Blank entries and comment-looking names starting with # or ; are ignored in both separator modes. Use ./#name for a literal name. The paths are relative to one source directory; destination parents are created as needed.

-h alone does not show help; use --help or --help-all.

Syq extensions

Most syq-specific options carry a --syq- prefix. Common ones are --syq-ignore, --syq-ignore-from, and --syq-verify-only. The last compares selected contents without writing; it does not produce rsync’s itemized-change format.

The hashing controls are --integrity-checking and --syq-expected-hash in rsync mode. -c still selects content comparison.

Filters are last-match-wins with ! re-inclusion, unlike rsync’s first-match rules. Check the gitignore examples when converting a filtered command.

The compatibility tests record comparison evidence and version-specific details.

Compare without copying

syq rsync -a --syq-verify-only project/ backup/

Hashes selected contents on both sides, writes nothing, and reports DIFFERS or MISSING. Differences or inspection failures produce a nonzero exit status.

Performance controls use --performance-tuning; resource ceilings use --resource-limits. The standard rsync spelling --bwlimit remains available in syq rsync, as does --checksum for BLAKE3 content comparison.

Python

Copy, remove, and reorganize files from Python, with typed results and streaming events. Synchronous and asyncio clients are available.

Install

The syq package on PyPI supports Python 3.13.4+ on Linux and macOS:

python -m pip install syq

Prebuilt wheels include the matching syq executable. Installation needs no Rust compiler, and SDK calls need no executable download or writable home directory for installation. The syq command is also available in the Python environment.

Installing from a source distribution builds the executable and requires Rust and a C compiler. Its remote behavior follows the native source version bundled in that distribution: source builds upload themselves to compatible SSH hosts by default. See source builds for helper selection and symbols when building from a checkout with those options. Older distributions retain their bundled native version’s build options.

Guide and examples

Use the syq package to call syq from Python. See installation if you have not installed it yet.

Copy files

Copy a directory named data into backup, producing backup/data:

import syq

result = syq.cp("data", into="backup")
print(result.files_transferred, result.bytes_transferred)

To copy just its contents, use srcs_in:

result = syq.cp(srcs_in="data", into="backup")

Arguments follow the command-line names: replace hyphens with underscores, and add a trailing underscore for Python keywords, such as from_ and as_. The copy guide explains placement, filtering, and verification options. Advanced groups take comma-separated strings, for example resource_limits="bandwidth=10M", performance_tuning="workers=4", or integrity_checking="compare=blake3,transfer=sha256". They are optional; ordinary copies choose performance settings automatically.

Copy over SSH

Use to for an SSH destination and from_ for an SSH source:

syq.cp("data", to="user@server", into="/backup")
syq.cp("report.csv", from_="user@server", cwd="/exports", into="downloads")

SSH hosts and paths are separate arguments. See Copy between servers for copies with two remote endpoints. A typed remote-to-remote preview needs coordinate_at="local"; a live copy can also use an enrolled receiver.

Preview changes

Pass dry_run=True to preview a copy or removal. The result has the same type as a live call, with counters describing the planned changes:

preview = syq.cp("data", into="backup", dry_run=True)
print(preview.files_transferred, preview.bytes_transferred)

Remove dry_run=True to apply the changes. syq checks the filesystem again when that call runs.

Mirror a directory

Use prune=True to also remove destination entries absent from the source. Here, staging must already exist, and max_delete limits deletions:

result = syq.cp(
    srcs_in="build",
    into_existing="staging",
    prune=True,
    max_delete=100,
)

Remove files

result = syq.rm(src_dir="old-output", root="/srv/jobs")
print(result.entries_removed, result.selectors_missing)

Positional paths and src refuse directories. Use src_dir to remove a tree, or srcs_in to recursively empty it while keeping its root.

root confines removal to that directory. Add on="server" to remove files over ordinary SSH. Command-restricted receivers do not support rm. See Remove files for selector behavior.

Handle failures

A failed copy or removal raises SyqOperationError with its typed result:

try:
    result = syq.cp("data", into="backup")
except syq.SyqOperationError as error:
    print(error.result.status, error.result.errors)
    print(error.stderr.decode(errors="replace"))

Use check=False to receive unsuccessful results without that exception. Invalid arguments, installation failures, and incomplete or invalid results still raise exceptions. Catch syq.SyqError to handle any SDK-defined exception, or catch a specific subclass as above. Completed filesystem changes are not rolled back.

Watch events and save results

on_event receives records as the operation runs. For example, show each copied entry or planned change:

def observe(event: syq.AutomationEvent) -> None:
    if isinstance(event, (syq.TraceEvent, syq.OperationResult)):
        print(event.action, event.dst)

result = syq.cp("data", into="backup", on_event=observe)

Events are not collected in the returned result. To save the validated NDJSON records, pass an open binary stream:

with open("run.ndjson", "wb") as records:
    result = syq.cp("data", into="backup", results=records)

Rename while copying

Create a mapping, change its destination paths, then copy:

from dataclasses import replace

with syq.map(srcs_in="photos") as mapping:
    renamed = mapping.transform(
        lambda entry: replace(entry, dst=syq.RelativePath("archive") / entry.dst)
    )
    result = syq.cp(mapping=renamed, into="published")

This places the contents of photos under published/archive. The mapping carries its source base and symlink-following policy through the transform. Return None from the transform to skip an entry. The entire transform must finish successfully before copying starts; a failed transform leaves the destination untouched. See Rename and reorganize for mapping rules.

For a complete program built on mappings, see Pull and push DVC data.

Generated data and binary streams

Use open_writer to generate one file or object without a temporary payload file. A successful context exit commits the upload; an exception aborts it. For example, the standard library can write an archive directly to S3:

import tarfile
import syq

with syq.open_writer(to="s3://backups", as_="dataset.tar") as output:
    with tarfile.open(fileobj=output, mode="w|") as archive:
        archive.add("dataset", arcname="dataset")

open_reader supplies the contents of one local file, SSH file, or S3 object:

with syq.open_reader("dataset.tar", from_="s3://backups") as source:
    while chunk := source.read(65536):
        consume(chunk)

Writers work with io.BufferedWriter and io.TextIOWrapper. Closing a wrapper ends payload input; the outer syq context commits only on successful exit. Outside a context, explicitly call commit() or abort() to finish the transfer.

The streams use bounded transport buffers. Reading without a size requests all remaining bytes into Python memory and checks transfer completion before returning them. Reader context exit drains unread bytes and checks transfer success, so archive readers may stop at their own end marker. Call abort() to cancel instead. If a consumer publishes files, keep them staged until both decoding and the reader context finish successfully. See byte streams for lifecycle, timeout, and async behavior.

Use asyncio

Await operations on AsyncClient. Its arguments and results match Client:

import asyncio
import syq

async def main():
    client = syq.AsyncClient()
    result = await client.cp("data", into="backup")
    print(result.files_transferred)

asyncio.run(main())

Async event callbacks are awaited in record order. Mapping streams use async with and async for; there is no await before client.map():

async def copy_photos():
    client = syq.AsyncClient()
    async with client.map(srcs_in="photos") as mapping:
        return await client.cp(mapping=mapping, into="published")

Configure a client

Share a local working directory and timeout across calls:

client = syq.Client(process_cwd="/srv/jobs", timeout=3600)
result = client.cp("data", into="backup")

process_cwd sets the local subprocess directory; typed cwd sets the source base, which may be on a remote host. Omit timeout on a call to use the client default; pass timeout=None to disable it for that call:

result = client.cp("data", into="backup", timeout=None)

To use an existing executable, pass Client(executable="/opt/bin/syq"). This bypasses the bundled version; see Compatibility.

Run other commands

run accepts arguments after the executable name and returns captured bytes:

result = syq.run(["--help"])
print(result.stdout.decode())

Use it for commands without a typed method, including rsync and receiver administration. See the API reference for process options and exceptions.

API reference

Module functions syq.cp, syq.rm, syq.map, syq.open_reader, and syq.open_writer use a default Client. AsyncClient has the same arguments and result types; await its operations except map, open_reader, and open_writer, which return async context managers.

Client and executable selection

Client(*, executable=None, cache_dir=None, process_cwd=None, env=None, timeout=None) and AsyncClient(...) accept:

client.version() returns the executable version as text. syq.version(executable=None) does the same without a client. syq.managed_executable(cache_dir=None) returns the verified executable path, downloading it if needed. See Compatibility for version selection and caching.

Arguments and validation

Options use CLI names with hyphens replaced by underscores. Python keywords get a trailing underscore: from_, as_. Paths accept str, bytes, or os.PathLike; selector keywords accept one path or an iterable of paths. Use src=["a", "b"] for CLI --srcs a b, and likewise src_non_dir and src_dir.

Boolean flags default to False; other optional arguments default to None, except check=True and timeout, whose omitted value inherits the client default. Invalid argument combinations raise SyqInvocationError; filesystem and remote checks happen in syq.

cp

cp(*sources, **options) → CpResult copies files. Choose one placement option. In addition to the shared arguments above, it accepts:

Option behavior is covered in Copy files, Remote copy details, and Object storage.

For example, client.cp("data", to="s3://bucket", into="backup", s3_header=["X-Tigris-Consistent: true"]) uploads local data using credentials from the subprocess environment or AWS configuration. Copies between two S3 endpoints use server-side copying within the same service; content verification options that require reading object bodies are rejected. SSH/S3 combinations are not supported. S3 results use EndpointKind.S3; older SDKs reject this new endpoint kind instead of interpreting it as SSH.

pscope selects an isolated scope for reusing SSH connections. For return copies or commands, use syq persist connect server and omit pscope. See persistence in scripts for setup and cleanup, and Compatibility for executable selection.

Typed SSH-to-SSH copies require an enrolled receiver or coordinate_at="local". With dry_run=True or verify_only=True, they require coordinate_at="local". Use run for detached commands and human output options.

IgnoreFrom(path) is a frozen dataclass holding a rule-file path (str, bytes, or os.PathLike). To interleave rule files and inline patterns: ignore=[syq.IgnoreFrom("rules"), "!keep.tmp"]. The last matching rule wins.

Byte streams

client.open_writer(*, as_=None, as_new=None, as_existing=None, to=None, follow_dst=False, ...) returns a StreamWriter. Choose exactly one of as_, as_new, or as_existing; the latter two require the destination to be absent or present, following cp placement semantics. Writers have no source basename, so they require an exact destination path. client.open_reader(src, *, from_=None, cwd=None, root=None, follow_src=False, ...) returns a StreamReader. cwd resolves relative sources; root also confines them. Choose at most one, as with cp. These bases belong to the source endpoint, independently of the client’s local process_cwd. Both accept rsh, syq_path, pscope, no_bootstrap, no_compress, s3_endpoint, s3_region, s3_profile, s3_header, performance_tuning, and timeout with the same meanings as cp. Only the stream-supported S3 part controls apply to performance_tuning. The client supplies the executable, process working directory, environment, and default timeout. Stream calls always check transfer failures.

These methods transfer raw bytes, using the CLI’s stream semantics for destination permissions, metadata, and endpoint restrictions. They do not emit cp result records. They are sequential, non-seekable interfaces; use one operation at a time on each stream. No whole-object retry or restart recovery is attempted. S3 can retry buffered multipart parts.

Use a with block. Successful writer exit sends a separate commit signal after closing the payload; EOF alone cannot publish a managed upload. An exception in the writer body before commit aborts without replacing the destination. close() only ends payload input, so a buffered or text wrapper can close it while unwinding an exception without publishing partial data. Successful context exit commits even if a wrapper already closed the payload. Calling abort() explicitly cancels that automatic commit.

Outside a context, finish with commit() or abort(); close() alone leaves the transfer pending. commit() can follow payload closure, is repeatable after success, and fails after an abort. Explicit commit publishes immediately, so later application errors cannot undo it. Closing an outer wrapper flushes its buffered data; do that before committing the underlying writer. A timeout or connection loss during commit can leave the outcome uncertain; the method reports failure rather than claiming rollback. Whole datasets need their own final publication step after all object transfers succeed.

A bounded reader call can yield partial data before a later transfer error. An unbounded read() checks transfer completion before returning its bytes. Successful reader exit establishes transfer completion, not that downstream application work was successful. An exception inside either context cancels the transfer and preserves the original exception. Writers that have not committed are aborted during garbage collection, including after payload closure; cleanup can block, so use explicit contexts for timely cleanup.

timeout covers the stream’s lifetime, including blocked reads/writes. Timeout raises subprocess.TimeoutExpired unless the transfer has completed successfully; transfer failures raise SyqProcessError, whose result contains the exit status and the last 8 KiB of diagnostics, without capturing payload bytes.

Async streams use async with directly and await I/O and explicit closure:

client = syq.AsyncClient()
async with client.open_writer(to="server", as_="generated.bin") as output:
    await output.write(b"generated bytes")
async with client.open_reader("generated.bin", from_="server") as source:
    while chunk := await source.read(65536):
        await consume(chunk)

Async streams start on context entry. Cancellation terminates the owned transfer process and releases blocked I/O. AsyncStreamReader and AsyncStreamWriter expose async read/write, close, and abort methods. AsyncStreamWriter.commit() explicitly publishes, with the same semantics as its synchronous counterpart; successful async context exit commits automatically.

rm

Positional sources and src refuse directories. src_dir removes a tree recursively; srcs_in removes its contents recursively and keeps the root.

on="server" selects the removal endpoint. A final selected symlink is always unlinked. Both follow_src=True and follow=True permit symlinks in cwd, root, and selector parent directories. Directory and contents selectors reject a final symlink even with following enabled.

rm(*sources, **options) → RmResult removes selected entries. Besides the shared arguments, it accepts on, dry_run, performance_tuning, syq_path, no_bootstrap, pscope, on_event, results, and check with the types above. It supports local, ordinary SSH, and S3 endpoints. Command-restricted receivers reject removal. See Remove files.

S3 removal uses rm(..., on="s3://bucket") with optional s3_endpoint, s3_region, s3_profile, and s3_header. s3_all_versions=True permanently removes all selected versions and delete markers; s3_version_id="ID" selects one version of one exact key. These options are mutually exclusive. RemovalTrace and RemovalResult expose optional s3_version_id and s3_delete_marker fields. The same arguments work with AsyncClient.rm.

map

map(*sources, **options) → MapStream lists local mapping entries without copying. Besides the shared arguments, it accepts as_ to rename a selected object. srcs_in must be the sole selector when used.

MapStream is a Mapping and an iterable context manager; use with. AsyncMapStream is an AsyncMapping and an async context manager; use async with. Streams must be consumed inside their context and cannot be reused after completion or closure. Async streams capture the producer directory, environment, and timeout when created, even though execution starts later.

Mapping and AsyncMapping

A mapping combines entries with their local source context. Pass it directly to cp(mapping=...), even on a client with a different process_cwd.

Mapping(entries, *, cwd=None, root=None, follow_src=False) accepts an iterable of MappingEntry. AsyncMapping(...) accepts an async iterable. If both cwd and root are omitted, the source base is the current directory at construction. Otherwise exactly one may be supplied. Relative cwd and root paths resolve against the Python process directory at construction, not a client’s process_cwd. root confines source resolution.

The context properties are read-only. A transform receives each MappingEntry and returns a replacement entry, or None to omit it. Transformations can be chained. Async transforms also accept awaitable callbacks and await them in entry order. Neither form caches entries; reuse depends on the supplied iterable.

A context-carrying mapping rejects from_, cwd, and root overrides on cp. It automatically enables the source-following policy used by map; enabling follow or follow_src on the copy is also permitted. The destination options remain independent. map(root=..., srcs_in=...) carries the selected directory as the consuming copy’s root. The copy resolves that root again; it does not inherit an open directory handle or a snapshot of the source tree.

Digest and HashAlgorithm

Digest(algorithm, value) describes the expected digest of all bytes in one regular file. algorithm accepts a HashAlgorithm value or its string: "blake3", "sha256", "md5", or "xxh3-128". value is hexadecimal: 64 digits for BLAKE3 and SHA-256, 32 for MD5 and XXH3-128. The immutable object validates the length and characters and stores lowercase hex.

client.cp(
    "data.bin", as_="verified.bin",
    expected_digest=syq.Digest("md5", "900150983cd24fb0d6963f7d28e17f72"),
)

The expectation covers the complete resulting file, including reused bytes. A mismatch fails the file rather than reporting a successful copy. Files excluded by selection rules are not digest-verified. hash=True still controls whether existing contents are compared instead of trusting size and modification time; integrity_checking="compare=HASH" selects and enables content comparison; hash=True is a shorthand for compare=blake3. Dry runs preview changes without validating the expectation. An expected whole-file digest is independent of the algorithm used for block comparison or transport checks. MD5 and XXH3-128 are useful for compatibility and accidental-error detection, but do not provide cryptographic collision resistance. Receiver receipt digests continue to use BLAKE3.

MappingEntry

Frozen dataclass describing one source-to-destination mapping. Pass an iterable of these to cp(mapping=...); use dataclasses.replace to change an entry.

MappingEntry(src, dst, kind=None, size=None, mtime=None, expected_digest=None) also accepts text or byte paths for src and dst and converts them to RelativePath. size and mtime do not impose preconditions on the copy. expected_digest does: a file cannot succeed unless its contents match. For example, an adapter can supply an MD5 from a DVC manifest without changing syq’s ordinary comparison algorithm:

entry = syq.MappingEntry(
    "cache/object", "data.bin", kind="file",
    expected_digest=syq.Digest("md5", "900150983cd24fb0d6963f7d28e17f72"),
)
client.cp(mapping=[entry], cwd="source", into="download")

Mapping files encode it as "expected_digest": {"algorithm": "md5", "value": "..."}. Older syq versions that do not support this field reject the mapping.

RelativePath and PathValue

RelativePath(value) accepts text, bytes, or os.PathLike. It rejects empty paths, absolute paths, NUL bytes, and empty, . or .. components. Join paths with /, for example syq.RelativePath("archive") / entry.dst.

PathValue(raw: bytes) holds a path received in an event, which may be absolute. Both types are immutable and provide:

RelativePath also implements os.PathLike, returning bytes.

Normal end of stream iteration checks the mapping process status. Leaving its context early stops the process. An exhausted or closed stream cannot be copied as an empty mapping.

For cp(mapping=iterable), the entire iterable is saved to a temporary manifest before copying starts. An iteration, transformation, or serialization failure starts no copy. Plain iterables and manifest paths have no source context; pass cwd, root, or from_ explicitly as needed. See mapping rules.

Events and terminal results

cp and rm return frozen dataclasses after validating the complete results stream and process exit status. A truncated stream raises even if the process exits successfully. Dry runs return the same types, with planned totals.

CpResult

Returned by cp(), or available as SyqOperationError.result after an unsuccessful copy. Read attributes directly, for example result.files_transferred. It includes the common result fields below plus:

With dry_run=True, mutation totals describe planned changes. With verify_only=True, matching files count as unchanged; transfer and creation totals are zero. A failed call reports work completed before it stopped.

Ordinary copies have all three deletion fields only with prune=True; otherwise they are None. Receiver-attested results have only deletions_completed, and their unchanged/excluded totals are always zero because the receiver cannot observe source-side skips. receipt is None for ordinary copies.

RmResult

Returned by rm(), or available as SyqOperationError.result after an unsuccessful removal. It includes the common result fields below plus:

Selectors identify requests; entries count individual filesystem objects. One directory selector can account for many entries. Duplicate and overlapping selectors have separate indexes.

Common result fields

Both CpResult and RmResult include:

ProtocolMetadata

Frozen dataclass accessed through result.protocol or event.protocol. The SDK checks these fields; applications normally only need them when recording or diagnosing a stream.

ReceiptSummary

Frozen dataclass accessed through CpResult.receipt for receiver-attested copies.

OperationStatus

String enum: compare with syq.OperationStatus.SUCCESS, or use .value for "success".

Event callbacks

on_event(event) receives an AutomationEvent in stream order. Events are not stored in the result. AsyncClient accepts synchronous or awaitable callbacks; awaitable callbacks count toward the timeout.

AutomationEvent is the union of the event classes below and CpResult and RmResult. Each event is a frozen dataclass. Its fields are listed below; all events also carry protocol: ProtocolMetadata, just like results. event.protocol.type identifies the wire record type. Optional fields use None when unavailable.

See Automation results for stream semantics. Python exposes class as class_ and paths as PathValue.

results= accepts a binary file-like object with write(bytes) returning a positive byte count for nonempty writes, and optional flush(). The client copies validated NDJSON records, flushes after completion, and leaves it open. It withholds the terminal record if stream validation, process completion, or a callback fails. Sink failures raise and abort the operation.

OperationResult.is_retryable identifies retryable failures; retry_entry() preserves expected_digest and returns a MappingEntry when a complete mapping identity is available, otherwise None. Only use collected entries after the call returns a validated success or partial result. A terminal callback alone does not establish completion. The client does not retry automatically.

Event types

RunEvent

Invocation details. started_at is Unix seconds; mode is "cp" or "rm". prune and mapping are None for removal; verify_only defaults to False.

protocol.type = "run". Fields in addition to the common envelope:

run_id: str
started_at: int
syq_version: str
mode: str
prune: bool | None
mapping: bool | None
dry_run: bool
endpoints: tuple[Endpoint, ...]
verify_only: bool

ProgressEvent

Sampled progress for displays; use the terminal result for final totals. Byte fields measure file content (comparison work with verify_only=True), scanned counts scanned entries, and elapsed_ms is milliseconds. Optional activity contains diagnostic measurements when the producer collects them; otherwise it is None.

protocol.type = "progress". Fields in addition to the common envelope:

bytes_done: int
bytes_total: int
bytes_unchanged: int
files_done: int
files_total: int
files_unchanged: int
files_excluded: int
scanned: int
scan_done: bool
elapsed_ms: int
activity: dict[str, Any] | None

TraceEvent

One planned copy change. dst is relative to the destination container; src is the mapping source when available. bytes is the planned file size, and reason describes why the change is needed.

protocol.type = "trace". Fields in addition to the common envelope:

action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind
bytes: int | None
reason: TraceReason

OperationResult

One copy outcome. dst is destination-relative, or relative to the signed destination scope for a receiver receipt. src is present for mapping entries when available. bytes and attempts give transfer information. Error details are present when known; message is display text. provenance, scope, and code apply to receiver-attested outcomes.

protocol.type = "operation_result". Fields in addition to the common envelope:

action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind | None
disposition: Disposition
bytes: int | None
attempts: int | None
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None
expected_digest: Digest | None
provenance: str | None
scope: int | None
code: ReceiptCode | None

SelectionResult

One removal selector, indexed from zero. path is the original selector; status says whether it resolved. kind is None for a missing selector.

protocol.type = "selection_result". Fields in addition to the common envelope:

selector: int
path: PathValue
status: SelectionStatus
kind: EntryKind | None

RemovalTrace

One entry a preview would remove. selector identifies its source selector; path identifies the entry. disposition is always WOULD_REMOVE.

protocol.type = "removal_trace". Fields in addition to the common envelope:

selector: int
path: PathValue
kind: EntryKind
disposition: RemovalDisposition
s3_version_id: str | None
s3_delete_marker: bool | None

RemovalResult

One removal outcome or preview inspection failure. selector identifies its source selector; path identifies the entry. attempts counts attempts; retryable, class_, os_kind, and message describe failures when available.

protocol.type = "removal_result". Fields in addition to the common envelope:

selector: int
path: PathValue
kind: EntryKind | None
disposition: RemovalDisposition
attempts: int
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None
s3_version_id: str | None
s3_delete_marker: bool | None

ErrorEvent

One counted error. message is display text; class_ and os_kind classify it when known. Receiver errors can also carry provenance and code.

protocol.type = "error". Fields in addition to the common envelope:

message: str
class_: ErrorClass | None
os_kind: OsKind | None
provenance: str | None
code: ReceiptCode | None

FinalStateEvent

Receiver-attested destination state. dst is relative to the signed scope. provenance is "receiver_attested". Present objects have kind and byte size; metadata, digest, and symlink_target are present where available. observation_error describes a partial observation. Absent objects have no object details; failed observations have code and optional message.

protocol.type = "final_state". Fields in addition to the common envelope:

provenance: str
scope: int
dst: PathValue
state: FinalObjectState
kind: FinalObjectKind | None
size: int | None
metadata: ObjectMetadata | None
digest: AttestedDigest | None
symlink_target: PathValue | None
observation_error: str | None
code: ReceiptCode | None
message: str | None

Endpoint, ObjectMetadata, and AttestedDigest

Nested frozen dataclasses used by events:

Enums

All enums are string enums exported by syq. Members use uppercase names; values use lowercase, for example EntryKind.FILE.value == "file". OperationStatus is defined with the result types above.

ReceiptStatus.CLEAN means a clean receipt, FAILED reports receiver failures, and INCOMPLETE reports an incomplete lifecycle. Retryability.YES, NO, and UNKNOWN state whether a failed entry can be retried.

Failure model

SyqError is the base class for every SDK-defined exception below. Specific subclasses retain their details so callers can distinguish an unsuccessful operation from a broken results stream.

For cp and rm, check=False returns unsuccessful typed results; for run, it returns nonzero process results. It does not suppress other errors. Spawn failures, timeouts, and ordinary Python type/value errors use standard Python exceptions. Exceptions from application callbacks or mapping iterators are re-raised unchanged. Async cancellation remains asyncio.CancelledError. These exceptions are not wrapped in SyqError.

Timeout, cancellation, early mapping exit, and streaming failures stop the local process group, including SSH children. Filesystem changes already completed are not rolled back.

run

client.run(args, *, check=True, cwd=None, env=None, timeout=CLIENT_DEFAULT, input=None) returns Result. input accepts bytes. args is a sequence of arguments after the executable name, passed without a shell.

Here, cwd is the local subprocess directory. cwd and env use client defaults when omitted or None. timeout uses the client default only when omitted; explicit None disables it. Module-level syq.run takes the same arguments plus executable=None and defaults to no timeout.

Wrappers can forward syq.CLIENT_DEFAULT to preserve client inheritance. syq.Timeout is the type alias for a number, None, or that sentinel:

def copy_data(client: syq.Client, *, timeout: syq.Timeout = syq.CLIENT_DEFAULT):
    return client.cp("data", into="backup", timeout=timeout)

CLIENT_DEFAULT applies to client methods (including module-level cp, rm, and map); client constructors and module-level run have no client default to inherit.

Timeouts cover subprocess execution. Managed installation and mapping-input materialization happen before the copy process starts and are not covered by its timeout. Async cancellation still stops mapping-input preparation.

syq exec is also available through run; its command output and exit status are a process result:

result = syq.run(
    ["exec", "--on", "@mac", "--cwd", "work/project", "--", "cargo", "test"],
    executable="/path/to/syq",
)

For exec, pass --cwd in the argument list to select the receiving working directory. The SDK’s cwd= parameter selects the local working directory of the requesting syq process.

Result

Frozen dataclass returned by run() and held in SyqProcessError.result:

Compatibility

Python 3.13.4+ on Linux and macOS; no runtime Python dependencies. Each Python package uses the matching syq release. syq.__version__ and syq.PINNED_SYQ_VERSION report those versions. Pin the package in your dependency file to keep the pairing.

Bundled executable

By default, the SDK runs the executable installed with its Python wheel. It locates that executable through the package’s installation record, without searching PATH, downloading files, or creating an executable cache. Each Python environment has its own installation. Removing the package also removes its executable.

Managed executable

For callers using a separate cache, Client(cache_dir=...) downloads the matching executable on first use and verifies it against the package’s embedded release manifest. It checks the cached binary before every use and replaces missing or corrupt entries. It does not search PATH. The existing syq.managed_executable() API also keeps this behavior.

The default cache is $XDG_CACHE_HOME/syq/sdk/python/v<version>/ when XDG_CACHE_HOME is absolute, or ~/.cache/syq/sdk/python/v<version>/ otherwise. Use Client(cache_dir=...) to change the cache root, or syq.managed_executable() to download ahead of time and get the path.

Custom executable

Client(executable="/opt/bin/syq") uses that binary; Client(executable="syq") searches PATH. Overrides bypass bundled selection and managed download and verification, so you are responsible for compatibility and origin. Typed calls still validate automation output. A failed executable selection does not fall back to another binary.

Pull and push DVC data

dvc_syq.py pulls and pushes DVC data with syq. It plans the copies and passes them to syq as a batch. Use it to try syq’s transfers with your existing DVC repository.

It is a short program written with the Python SDK, meant to be used, read, and adapted. DVC keeps working alongside it: both use the same cache and the same remote, so dvc status, dvc push, and dvc pull treat the script’s results as their own.

Run it

Install it as a dvc-syq command with uv:

uv tool install 'git+https://github.com/greaber/syq#subdirectory=examples/dvc-syq'

Then, inside a DVC repository:

# Pull one tracked path, or the .dvc file that describes it.
dvc-syq pull models/speech.dvc

# Pull everything tracked under a directory; "-R ." covers the repository.
dvc-syq pull -R datasets

# Download into DVC's cache without touching the workspace.
dvc-syq fetch -R datasets

# Upload what the remote does not have yet.
dvc-syq push models/speech.dvc

uv tool upgrade dvc-syq fetches the current version. The script is also a single file that declares its own dependencies, so you can download it and run it without installing anything: uv run dvc_syq.py pull -R datasets.

These options have the same meaning as in DVC:

And two are the script’s own:

How it works

DVC’s current cache layout names objects after their MD5. The same relative path identifies an object in the cache and remote; a separate checkout step gives workspace files their original names.

def object_path(md5):
    return f"files/md5/{md5[:2]}/{md5[2:]}"

# Remote to cache: the same path on both sides.
downloads = [MappingEntry(src=object_path(md5), dst=object_path(md5)) for md5 in missing_from_cache]
client.cp(mapping=downloads, from_="s3://my-bucket", into=".dvc/cache", only_new=True)

# Cache to workspace: each object gets the name recorded in its .dvc file.
checkout = [MappingEntry(src=object_path(md5), dst=path) for path, md5 in tracked_files]
client.cp(mapping=checkout, cwd=".dvc/cache", into=".")

Each list is a mapping: pairs of source and destination paths that syq copies in one run. A push is the first copy in reverse, and only_new makes it skip objects the remote already has. With --verify, download mappings carry the expected MD5 so syq can check the completed file before adding it to the cache. Older DVC text objects can use a newline-normalized MD5; the script checks those after download and removes mismatches. Files already in the cache are skipped.

Differences from DVC

• A target is required. DVC pulls or pushes the whole repository when you name none; here that is -R ..
• Remotes can be local directories, ssh://host/path without an explicit port, or s3://bucket/prefix. For S3 the script reads profile and endpointurl from the DVC remote and otherwise uses your usual AWS credentials. Remotes that use DVC’s cloud versioning are refused.
• Data brought in with dvc import is skipped with a message, because it lives in another repository’s remote. Use dvc pull for it.
• DVC options not listed above, such as --all-branches, are not available.