syq Fast, programmable file operations.

Copy, reorganize and remove files—locally or across machines. Resume interrupted transfers, send files to your laptop or run commands there from a server’s remote shell, and script syq using the JSON API or Python SDK. Your laptop needs no SSH server or incoming network port.

Try a copy

Replace server with an SSH hostname or alias you normally connect to, and project with a local directory:

syq cp project --to server --into backup

This creates or updates backup/project in your home directory on the server. To copy the contents of project directly into backup, use --srcs-in:

syq cp --srcs-in project --to server --into backup

Use --dry-run to preview a summary without copying, or --dry-run -v to list the planned changes by path. Existing destination files are updated when needed. Unrelated files stay unless you request --prune.

Already use rsync?

Start with your usual command, prefixed by syq:

syq rsync -av project/ server:backup/project/
syq rsync -av server:data/ ./data/

Syq supports common rsync options, but uses its own protocol. Rsync filter rules, hard links, ACLs, xattrs, sparse files, and rolling-checksum deltas are not supported. Check rsync compatibility before substituting it in an existing script.

Common tasks

Install

Syq runs on Linux and macOS, on x86-64 and ARM64.

Standalone installer

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/greaber/syq/releases/latest/download/install.sh | sh

Installs into ~/.local/bin without sudo. Make sure that directory is on your PATH. To choose another directory, download the script and run sh install.sh --bin-dir DIR.

Homebrew

brew install greaber/tap/syq

Try a benchmark

Compare syq with rsync on your own machines, or with rsync and cp locally:

curl --proto '=https' --tlsv1.2 -fLsS https://raw.githubusercontent.com/greaber/syq/master/scripts/try-benchmark.sh | bash

The default sends 1,024 small throwaway files to an SSH host you choose and compares syq with rsync over three rounds, following an untimed tuning warm-up (--warmup off skips it). Local copies, large files, and automatic sizing are optional. The script checks the copied contents and cleans up afterward. If syq is missing, it offers to install it. See quick comparison to download the script and run it again.

Updates

Use syq --self-update for a standalone installation, or brew upgrade syq for Homebrew.

Standalone installs print an update reminder; nothing updates automatically. Set SYQ_NO_UPDATE_CHECK=1 to disable reminders.

Shell completion

Add the line for your shell to its startup file:

# Bash (~/.bashrc)
eval "$(syq completion bash)"

# Zsh (~/.zshrc), after autoload -Uz compinit && compinit
source <(syq completion zsh)

# fish (~/.config/fish/config.fish)
syq completion fish | source

Completion suggests options, hosts, and paths, with file details beside path matches. In Bash, press Tab again to list matches. Remote paths use your usual SSH login. Open a new shell after adding the setup line or upgrading syq.

Keep connections open

Keep an SSH connection ready for repeated copies:

syq persist connect server

This enables persistence and connects without copying files. It also lets you send files back from the server, with approval on your machine. Connections stay open until you close them with syq persist off. Use syq persist status to see them.

See background connections for reconnecting, turning receiving off, and using persistence in scripts.

Copy files

syq cp project --into backup

This copies project to backup/project. Existing files are updated when needed; unrelated files stay.

The default final summary reports transferred files and bytes, unchanged files and bytes, directories created, elapsed time, rate, and any errors.

Add -v to list copied paths. -vv also explains helper selection and transport; --stats adds scan totals, excluded-file counts, connection count, and available TCP statistics. For example:

syq cp -vv --stats project --into backup

See diagnosing a slow copy for interpreting transport and performance details.

See where files go

A named directory brings its name along. --srcs-in copies its contents; --as chooses an exact destination name.

syq cp project --to server --into /backup

project/
├── index.html
└── images/
    └── logo.svg

/backup/
├── index.html (old)
└── notes.txt

/backup/
├── index.html (kept)
├── notes.txt (kept)
└── project/ (new)
    ├── index.html
    └── images/
        └── logo.svg

Note to rsync users: trailing slashes have no special significance in syq’s native commands. syq rsync keeps rsync’s slash rules.

Copy over the network

Use --to to send files, or --from to fetch them:

# Copy project to /backup/project on server.
syq cp project --to server --into /backup

# Fetch /data/a and /data/b from server into ./data/a and ./data/b here.
syq cp --from server -C /data --src a --src b --into ./data

-C DIR is shorthand for --cwd DIR: look for the source files in DIR. In the second example, /data is on the server and ./data is on your machine.

To use the default destination:

syq cp project --to server       # put project in your home directory on server
syq cp --from server project     # fetch project into your current directory

Endpoints use [USER@]HOST[:PORT], for example alice@server:2222. Host names cannot start with a dash, including when using an --rsh wrapper. Enclose IPv6 addresses in brackets: alice@[2001:db8::1]:2222. A colon in a native path is simply part of the path.

For two remote endpoints, see Copy between servers. To send files to your laptop from a server shell, see Send files home from a server.

Progress

Syq shows a progress bar when running in a terminal. It tracks bytes processed; while files are still being discovered, the percentage is unknown. Wider terminals also show elapsed time, speed, ETA, and file counts.

Use --progress to force the display or --no-progress to hide it. --quiet hides it too. After five seconds without a byte update, no update shows how long it has been; this does not by itself mean the connection has failed. Wait for the final summary to confirm success, even if the byte bar looks full.

The bar also covers syq rm, counting entries instead of bytes. --progress-json provides JSON progress for displays; use results records to track completion in scripts.

Choose a destination

# Copy report.txt under a new name; refuse to overwrite an existing entry.
syq cp report.txt --as-new reports/final.txt

--into uses or creates a directory. --as can rename a directory too. Sources that would collide at one destination are refused before copying.

Choose which existing files to update

By default, selected destination entries are updated when needed. These options apply to individual entries inside the copy:

# Import new files without replacing existing files.
syq cp --only-new --srcs-in incoming --into archive

# Refresh only files already in the destination.
syq cp --only-existing --srcs-in project --into deployed

--only-new keeps existing entries and their metadata, while adding missing children to existing directories. Those directories must be writable; syq does not change their permissions to add files. Adding children can still change directory timestamps. A dry run does not test write access.

If a source directory meets an existing non-directory, --only-new skips that subtree. --only-existing skips a subtree when its destination is missing or is not a directory. It cannot combine with --into-new or --as-new. The placement options --into-existing and --as-existing check only the placement path, rather than every copied entry.

--skip-newer compares timestamps, not the age of the contents. It affects only regular-file pairs; replacing a different entry type still occurs. Combine it with --only-existing to avoid creating missing entries too.

--only-new cannot combine with either policy. Neither --only-new nor --skip-newer can combine with --inplace: an interrupted write could leave an incomplete file that the next run skips. Restricted receivers also refuse --only-existing --inplace.

These options do not disable --prune; requested pruning still removes extras.

Preview changes

--dry-run previews a copy without changing the destination. On its own it prints a summary; combine it with -v to list the planned changes by path:

syq cp --dry-run -v --srcs-in project --into backup

The summary shows where files would land, what would change, and how much data may move. Copy data stays unchanged; a requested results file is still written. The filesystem can change between preview and execution.

Mirror a directory

--prune removes destination files that have no counterpart in the source:

syq cp --prune --max-delete 100 --srcs-in build --into-existing deploy

This updates deploy from build, then removes extras. Preview with --dry-run -v first. If more than 100 removals are planned, none are performed and the command exits 25.

Pruning stays inside the copied directories. Copying named directories a and b into backup prunes backup/a and backup/b, leaving backup/c alone. Ignored paths and files skipped by size limits are protected.

Scan errors prevent deletion. An interruption after deletion starts can leave some extras removed. Do not prune while another copy is writing into the same tree: its completed files can be treated as extras. Recognized partial files and directories containing them are protected from pruning. With -v, syq lists each extra file it keeps because its name matches the partial-file format.

Ignoring paths

# Use the project's existing ignore rules.
syq cp --ignore-from .gitignore --srcs-in project --into backup

# Skip node_modules and object files.
syq cp --ignore node_modules --ignore '*.o' --srcs-in project --into backup

Patterns use gitignore syntax. Rules run in command-line order; the last match wins. ! re-includes a path.

Ignored directories are not scanned. To keep part of one, include the parent:

# Skip other logs, but copy logs/keep and its contents.
syq cp --ignore 'logs/*' --ignore '!logs/keep/' --srcs-in project --into backup

Ignored paths are also protected from pruning.

Resume an interrupted copy

Rerun the command. Completed files are skipped; partially copied files can reuse matching blocks. Each run writes its own fresh partial beside the destination and replaces the final file only when complete. When resuming, syq can copy bytes from a previous partial or the existing destination into its own output, hash the bytes it copied, and transfer blocks that differ from the source before publishing. The previous partial stays unchanged. Reuse is best effort; local direct copies can be faster than looking for reusable blocks and take priority.

Resuming requires space for the new output as well as the previous partial. This can require enough free space for another complete file, even when only a small amount remains to transfer.

Concurrent copies use separate partials. With unchanged sources, each completed file comes from one copy; different copies may win for different files. This does not make a whole tree a snapshot. --inplace still exposes unfinished updates, and pruning can delete another copy’s completed files.

Partials are named .FILENAME.syq-tmp.RANDOM, with 16 random characters at the end. The filename portion is shortened or omitted when space is tight. Syq removes its own partial when it publishes the completed file. Interrupted runs can leave partials behind, including after a later successful retry. Partials with shortened or omitted filenames may not be reused.

To remove leftover partials, stop copies writing into the tree, then preview and run:

syq clean-partials --dry-run -v backup
syq clean-partials backup
# Search several remote trees with the parallel removal workers.
syq clean-partials --on server --cwd /data -j 8 backup archive

This command removes regular files with the current partial-name format. It keeps directories, other filenames, and symlinks, and does not follow symlinks. Use --root DIR to confine traversal and --results FILE for removal results. The results use the same mode: "rm" records as syq rm; they do not distinguish a partial sweep from other removal commands. A regular file deliberately named like a partial is also selected. Old partial formats are neither reused nor selected by this command; remove those manually.

Check file contents

Syq normally skips files whose size and modification time match. --hash checks contents even when those two attributes match:

syq cp --hash --srcs-in project --into backup

This changes how syq decides what needs copying. For larger network copies, syq still compares blocks when size or modification time differs, even without --hash, so it can reuse unchanged data. Local and small copies may use faster paths instead.

Transferred data is always checked for corruption. For files being changed by another program, stop the writer or copy a snapshot. No copy makes the whole tree transactional or guarantees durability across power loss.

To compare without writing, use --verify-only:

syq cp --verify-only --srcs-in project --into backup

This compares file contents, symlink targets, and entry types without writing. Missing or different entries make the command fail. It does not compare metadata or look for extra destination files.

For two servers, add --coordinate-at local to compare through your machine using ordinary SSH access, with no restricted receiver enrollment. This also supports --results. See remote verification.

In-place writes

By default, syq builds an updated file beside the old one and replaces it when complete. --inplace writes directly into the destination file instead:

syq cp --inplace large-file --to server --into /backup

This saves temporary disk space and can avoid copying unchanged data into a new file. Readers can see a mixture of old and new contents during the copy. If interrupted, the incomplete file stays at its final name until you finish the copy. Writes through a hard link also affect its other names.

Use the default when other programs need to read a complete file throughout an update. Copies sent back to your laptop do not support --inplace.

Preserve metadata

Copy keeps modification times and copies symlinks as symlinks. New files use the source read, write, and execute permissions limited by the destination umask; existing files keep their destination permissions. For example, a new script with mode 755 stays executable with umask 022. Source setuid, setgid, and sticky bits are not copied by default.

To copy source permissions exactly, including onto existing files, or request ownership too:

syq cp --preserve=permissions,ownership project --into backup

permissions preserves modes; ownership requests numeric owner and group; specials enables device, FIFO, and socket nodes. Ownership needs suitable permissions on the destination. Hard links, ACLs, and xattrs are not preserved.

Symlinks

A named symlink is copied as a link. Syq refuses to follow links in paths you supply unless you ask it to:

# Copy the directory current-project points to as backup/current-project.
syq cp --follow-src current-project --into backup

With --as link, the final link itself is replaced, even with --follow-dst. Links found inside a directory are never followed. See the security explanation.

Keep sources inside a directory

--root DIR both sets the source directory and prevents selections from escaping it:

# Copy /srv/data/reports and /srv/data/photos into backup.
syq cp --root /srv/data reports photos --into backup

Sources must be relative to that root. A selection such as ../private is refused; even with --follow-src, symlinks cannot lead outside the root. Unlike -C, this is a boundary, not just a starting directory. It does not constrain the destination.

Output and diagnostics

Human output, including persist status and persist receive status, escapes terminal control characters, Unicode line separators, and directional marks in names and peer diagnostics. JSON status output keeps the original values.

More options

--src-non-dir and --src-dir require a non-directory or directory respectively. Use --min-size and --max-size to select regular files by size.

For parallelism and bandwidth controls, see Speed. For scripts, see Automation results.

Use --help (or -h) for everyday options and --help-all for the full list, including tuning, scripting, and manual setup. syq help COMMAND shows the same help without running the command. In syq rsync, -h means human-readable sizes; use --help for help.

Remove files

Remove a file or a directory tree:

syq rm old-output

Remove a directory’s contents, leaving the directory itself:

syq rm --srcs-in cache

Add --dry-run -v to either command to see what would be removed first. Missing paths succeed. Removal is permanent; completed deletions cannot be rolled back.

On another machine

syq rm --on server /scratch/old-output

This removes /scratch/old-output on server. Remote removal runs while your connection stays open; there is no detached mode.

Limit the selection

syq rm --root /srv cache old-output

This removes /srv/cache and /srv/old-output, with selection confined to /srv. Use --src-non-dir PATH or --src-dir DIR when the path must be a non-directory or directory respectively. All selections are checked before deletion begins. Filters are not supported.

Symlinks

A selected symlink is removed as a link, leaving its target alone. Symlinks inside a selected directory are also only unlinked.

--follow-src permits traversal through symlinks in --cwd, --root, and selector parent directories. --follow also permits symlinks in the --results path. The final selected symlink is always removed as a link, even with --follow-src or --follow. For example, if current points to releases/v1, syq rm --follow-src current/log.txt removes releases/v1/log.txt, while syq rm --follow-src current removes only the link.

--src-dir and --srcs-in reject a final selected symlink, including when following is enabled. Select the actual directory to remove it or its contents. With --root, traversal must still stay inside that root.

Results

The command continues with independent entries after per-entry failures and exits 23. Fatal setup or connection failures exit 1. Use --results for per-path outcomes in scripts.

For the full option list, run syq rm --help-all.

Send files home from a server

Inspect files on your server, then copy them to your laptop from the same shell. The laptop opens and maintains the connection. It needs no SSH server, public address, or incoming network port.

With syq installed on both machines, connect from your laptop:

syq persist connect server

This keeps a connection open so the server can send files back to your laptop. Once the command finishes, you can close the terminal and continue working on the server. By default, your laptop is available under its short hostname, and received files go into your home directory.

To give it the name laptop and choose a different starting directory:

mkdir -p ~/Downloads/server
syq persist receive on --name laptop --cwd ~/Downloads/server
syq persist connect server

On the server, use the name from any shell, including an existing tmux session:

ls -lh results
syq cp results --to laptop
syq cp report.pdf --to laptop --as reports/latest.pdf

Each incoming copy waits for approval on your laptop before it can inspect or change destination entries. Review the destination and permissions, then choose Allow once or Deny. Use Details on macOS or syq persist receive pending in a local terminal to see the complete request. Approving a copy trusts the server to supply its contents.

You can also run commands on the receiving machine, with separate local approval, to build a project there or open a copied artifact.

Multiple receiving profiles

Give a project its own receiving name and directory:

syq persist receive on --name project --root ~/work/project
syq persist connect server

Then run syq cp results --to @project on the server. The directory must already exist. Each name has its own settings and approval policy, so you can keep a project separate from your general laptop destination.

Use syq persist receive status to list profiles and syq persist receive off --name project to stop one. See profile management for more options.

Approving copies

Approve or deny from the desktop prompt, or from a terminal on your laptop:

syq persist receive pending
syq persist receive approve REQUEST_ID
syq persist receive deny REQUEST_ID

Requests expire after five minutes. If a prompt is missing or dismissed, the request stays pending; it is never approved automatically. To use only terminal approval, run syq persist receive on --notify off.

For unattended copies from trusted server accounts, explicitly enable automatic approval:

syq persist receive on --approve always
syq persist receive on --approve ask   # require approval again

Automatic approval trusts all processes running as the connected server accounts, including for overwrites. Commands and copies to another server still require separate approval. See persistence security for the trust boundary.

Names and paths

When you use --to laptop, syq looks for a connected receiving machine with that name. If it is offline, syq tries an SSH host called laptop instead. Use --to @laptop when you want the command to fail if your laptop is offline. Once a copy starts, it keeps the same destination even if the connection fails.

The directory you set with --cwd is where incoming copies start. You can choose a path relative to it with --into or --as, or use an absolute path to copy elsewhere. Without either option, files go into the starting directory.

To contain copies within a directory instead:

syq persist receive on --name laptop --root ~/Downloads/server

With --root, all incoming copies must stay inside that directory. Absolute paths and .. are rejected, and symlinks cannot lead outside it. A copy cannot replace the root itself with --as .. Switching back to --cwd removes this restriction.

Changing a profile’s settings stops its active copies so the new settings can take effect. Other profiles keep working.

Syq also protects its own receiving files, executable, and SSH authority files from incoming copies. See directory requirements if a receiving location cannot be opened.

Copy permissions and limits

By default, each copy is limited to 100 GiB and one million entries. Pruning requires a positive deletion limit on both machines. Change limits with syq persist receive on --max-bytes SIZE --max-entries N --max-delete N.

Most copy options work here; ownership and special-file preservation, --inplace, and --min-size are unsupported. See copy limits for details.

Background connections

You can inspect your connections with syq persist status. To stop receiving while keeping SSH connections open for your own copies, run syq persist receive off. Use syq persist off to close both directions.

After a network interruption or laptop sleep, syq reconnects automatically. An interrupted copy still needs to be rerun so it can resume. After rebooting your laptop, run syq persist connect server again.

If a connection fails to start, syq persist receive status shows the error. The persistence reference covers troubleshooting, upgrading, and using connections in scripts.

Run commands on your receiving machine

From a server shell, ask your Mac or Linux desktop to run a command:

syq exec --on @laptop --cwd work/project -- cargo test
syq exec --on @laptop --cwd work/project -- open report.html

The second command uses macOS’s open program to display an artifact. Any program installed on the receiving machine can be requested, including a native application or a version of syq you have just built there.

If you have already set up receiving files, you can request commands through the same connection. Otherwise, run syq persist connect server on your desktop first. Your desktop needs no SSH server or incoming network port, and you can make requests from any shell on the server, including an existing tmux session.

Replace @laptop with your desktop’s receiving name. You can find it by running syq persist destinations list on the server or syq persist receive status on your desktop. The desktop must be connected: both --on laptop and --on @laptop fail while it is offline.

Approve each command locally

Each command waits for approval on the receiving machine. Review the program, arguments, requesting server, and working directory. Desktop notifications may truncate long commands; inspect the complete request from a local terminal:

syq persist receive pending
syq persist receive approve REQUEST_ID
syq persist receive deny REQUEST_ID

Command requests are available whenever receiving is enabled. Every command requires its own decision, even with syq persist receive on --approve always for copies. Approving a copy does not approve commands. A missing or dismissed desktop prompt never grants permission; use the local terminal commands. Pending requests expire after five minutes and are cancelled when the requester disconnects or receiving restarts or stops.

An approved command runs with your local user’s permissions, including access to files and credentials. The receiving --root and copy limits do not contain commands. Build tools and scripts can execute code from their input files; approving a displayed command does not establish that those files are safe.

Arguments and working directories

Put -- before the program. Everything after it is passed as a literal argument, including strings starting with --. There is no implicit shell expansion on the receiving machine. To use shell syntax, request a shell:

syq exec --on @laptop --cwd work/project -- sh -c 'cargo build && ./target/debug/demo'

--cwd DIR (or -C DIR) is relative to the directory selected by persist receive on --cwd or persist receive on --root. Its default is that directory. Absolute paths and .. can select elsewhere. The directory must exist. Syq does not expand ~ on the receiving machine; use a relative path or an absolute path instead.

The command inherits the receiving service’s local environment, including PATH and its desktop session. It does not inherit the server’s environment. Restart receiving from a terminal in the desired desktop session when those values change. Stdin is closed and there is no interactive terminal.

A request supports at most 256 arguments, including the program, with at most 16 KiB of argument bytes and a working-directory path of at most 4096 bytes. Each server connection permits one pending approval and eight active commands. Active commands do not prevent a new approval or an ordinary copy.

Output, completion and cancellation

Stdout and stderr stream back separately without text conversion. Syq returns the command’s exit code; when the command is killed by a signal it reports the signal and returns 128 + signal. A setup error or connection failure is a nonzero result. A connection that closes before delivering the exit status is an error even if some output arrived successfully.

Interrupting the request, losing the connection, changing receiving settings, or stopping receiving cancels execution. Syq forcibly stops the command’s process group, including remaining children when the foreground program exits; cleanup handlers do not run. Detached processes and applications launched through macOS open can outlive the request. Syq does not manage detached jobs.

A command may already have changed files when interrupted. It is never retried automatically after a lost connection. Inspect the outcome before requesting it again. Commands do not produce copy receipts or copy automation records.

Python callers can use the SDK’s existing raw process interface with their chosen executable:

import syq

result = syq.run(
    ["exec", "--on", "@laptop", "--cwd", "work/project", "--", "cargo", "test"],
    executable="/path/to/syq",
    timeout=600,
)
print(result.stdout.decode())

The raw SDK call captures byte output and raises on a nonzero exit by default. The async client’s run method accepts the same command arguments. For live terminal output, invoke the CLI directly or use a subprocess with inherited stdout and stderr.

Copy between servers

Copy directly between servers without putting private keys on either server or giving one server unrestricted access to your SSH agent. Your machine authorizes the copy and shows the results; the file data bypasses it.

# Copy the contents of hostA's big directory into hostB's big directory.
syq cp --from hostA --srcs-in big --to hostB --into big

HostA gets permission for this transfer only. HostB checks that permission and reports what it changed. See Security for what this protects against.

Start a copy from the source server

If your laptop has a return connection to the source server, you can inspect files in any server shell and send them to another SSH host:

# Run on hostA, including in an existing tmux shell.
ls -lh results
syq cp results --to hostB --into /archive

Syq uses the first available receiving machine in alphabetical name order. If none is available, or the requested options are unsupported on this route, it uses hostA’s SSH access. A copy addressed to a live receiving name goes to that machine itself.

Use --auth-from @laptop to choose your laptop explicitly, or --auth-from ssh to use hostA’s SSH access. The default is --auth-from auto. --via NAME is an alias for choosing a receiving machine. See authorization selection for name rules and route restrictions.

The selected laptop asks for approval before contacting hostB. Approve with the desktop prompt or syq persist receive pending and syq persist receive approve REQUEST_ID on the laptop. These requests require a decision even when persist receive on --approve always permits automatic copies onto the laptop itself. Once an approval request is sent, a refusal, interrupted connection, setup failure, or copy failure ends that attempt; syq does not try another authorizer or SSH. Explicit receiving names require a live return connection and never fall back to DNS lookup.

The laptop uses its own SSH configuration, credentials, and trusted host keys to connect to hostB and install the matching syq helper. Connect to hostB with ordinary SSH from the laptop first if its host key is not yet trusted. The source server receives no SSH credentials or agent access. The control stream passes through the laptop; file data goes directly from hostA to hostB over encrypted TCP. HostB must expose a reachable data port to hostA. Failure to reach it fails the copy without switching to SSH data.

The prompt shows the requested SSH endpoint and destination path. Relative paths start in that account’s home directory on hostB. Your laptop’s receiving root does not contain this copy, but its byte, entry, and deletion limits still apply.

Keep the source command and your laptop’s connection alive until completion. Stopping receiving or losing that connection cancels the copy. Retry with a new approval to resume it. Supported copy options match return copies, with additional route restrictions.

What you need

• SSH access from your machine to both servers, with their host keys already trusted. Connect with ordinary SSH once if either server is new to you.
• An SSH agent on your machine, and OpenSSH 8.9 or newer on your machine, hostA’s SSH client, and hostB’s SSH server.
• SSH connectivity from hostA to hostB. A reachable TCP data port on hostB, normally in 47600–47699, enables encrypted TCP workers. Otherwise the copy uses SSH workers on the same hostA-to-hostB route. --no-tcp selects SSH directly.
• An existing parent directory for the destination.

Keep your command running until the copy finishes. Use native syq cp; syq rsync does not accept two remote endpoints.

First copy and access management

The first copy sets up a restricted receiver on hostB automatically. It adds a restricted key to authorized_keys; the private key stays on your machine. Later copies reuse this setup.

To prepare /archive on hostB ahead of time, including before a dry run:

syq receiver enroll hostB:/archive
syq cp --dry-run -v --from hostA --srcs-in data --to hostB --into /archive

Inspect or remove this access with:

syq receiver list
syq receiver revoke ID

Use the ID from list. Revocation stops all active restricted receivers for that enrollment and blocks new sessions. It waits for their shutdown before reporting success, then removes the enrollment from both machines. Other enrollments keep running. Revocation does not undo completed writes; interrupted copies fail and can leave partial files. Enroll again to authorize a new copy that can resume them.

If receivers have not stopped within ten seconds, revocation reports failure and keeps the enrollment marked revoked. Retry receiver revoke to finish cleanup; an enrollment with unfinished revocation cannot be refreshed.

For upgrading or sharing a receiver between installations, see enrollment details.

If your machine reaches hostB through hostA, add --via hostA to enroll or revoke.

Mirror a directory

Include a deletion limit when pruning:

syq cp --prune --max-delete 100 --from hostA --srcs-in data --to hostB --into-existing /archive

This updates /archive and removes extras. If more than 100 removals are planned, none are performed and the command exits 25. Preview first with --dry-run -v.

Other routes and authentication

Syq may switch between encrypted TCP and SSH on the selected route. It never silently relays file data through your machine when a direct connection fails. If the servers cannot connect directly, explicitly relay through your machine:

syq cp --coordinate-at local --from hostA --srcs-in data --to hostB --into /archive

This uses your machine’s bandwidth and ordinary SSH access to each endpoint. Syq never switches to it silently.

Other authentication modes can use server-held credentials, a destination- restricted SSH agent, or full agent forwarding. They grant different authority; see the advanced reference before choosing one. That page also covers detached copies, signed results, and direct-mode limits.

Rename and reorganize during a copy

syq map lists source/destination pairs as one JSON object per line. Transform that list with a script, then give it to syq cp --mapping. The copy checks for destination collisions and supports normal resume.

For Python scripts, see the Python SDK and its mapping examples.

Lowercase destination names

set -o pipefail
syq map --srcs-in src \
  | jq -c '.dst.value |= ascii_downcase' \
  | syq cp --mapping - -C src --to nas --into /pub

src/                       nas:/pub/
  Berlin/IMG_1234.JPG         berlin/img_1234.jpg
  Notes.TXT                  notes.txt

If another entry also claims notes.txt, the copy is refused before files are transferred. Symlink target text is not rewritten: renaming its target can leave a link dangling on a case-sensitive destination.

Group photos by modification month

set -o pipefail
syq map --srcs-in photos \
  | jq -c 'select(.kind == "file")
        | .dst.value = (.mtime | gmtime | strftime("%Y/%m")) + "/" + .dst.value' \
  | syq cp --mapping - -C photos --to nas --into /archive

A July 2024 file IMG_1234.JPG lands at /archive/2024/07/IMG_1234.JPG. The filter keeps regular files only; missing parent directories are created. Dates use file modification time in UTC, not photo EXIF dates.

Other filters can replace the jq stage:

# Keep files of at least 1 MiB, plus directory and link entries:
jq -c 'select(.kind != "file" or .size >= 1048576)'
# Drop device, FIFO, and socket entries:
jq -c 'select(.kind != "special")'

Check the producer before copying

A pipeline’s consumer sees only the bytes it receives. If a generator fails after emitting valid entries, those entries can still be copied. set -o pipefail makes the pipeline report failure, but does not undo writes.

To require successful generation before copying, save the manifest first:

set -o pipefail
syq map --srcs-in src | jq -c '.dst.value |= ascii_downcase' > m.ndjson \
  && syq cp --mapping m.ndjson -C src --to nas --into /pub

Add --dry-run -v to cp to preview placement.

The format

A manifest contains one JSON object per line (NDJSON):

{"src":{"encoding":"utf-8","value":"IMG_1234.JPG"},"dst":{"encoding":"utf-8","value":"2024/07/photo.jpg"},"kind":"file","size":4194304,"mtime":1721900000}

Paths use encoding: "utf-8", or "base64" with standard base64 of raw filename bytes. Absolute or empty paths, and any . or .. component, are refused. Unknown fields are refused too.

Each entry copies one object. A directory entry is not recursive. syq map emits its descendants as separate entries. A missing source or wrong kind fails that entry while independent entries continue. A kind mismatch is a non-retryable conflict; a missing source is an I/O failure whose retryability is unknown.

Any program can generate this format:

syq cp --mapping pairs.ndjson -C photos --to nas --into /archive

Copy between servers

The manifest is read on the machine where you run the command. Its source paths resolve on the source server, and its destination paths resolve beneath the destination container:

syq cp --from hostA -C /data --mapping pairs.ndjson --to hostB --into /archive

--mapping - reads the manifest from stdin. File contents follow the selected copy route. The restricted receiver verifies the authorized manifest and permits writes only at its listed destinations, plus creation of necessary parent directories. New implicit parents use normal directory permissions subject to the receiver’s umask. Existing implicit parents keep their permissions, including restoration if copying temporarily requires write access. A listed directory still selects only that directory, not its unlisted children. If a file or symlink blocks an implicit parent, the affected entries fail and unrelated mappings continue. Replacing that obstruction requires a directory entry for the parent in the manifest. Restricted receivers count mapped destinations and their parent directories against the copy’s entry limit. Each manifest line can be up to 1 MiB, and each destination path up to 4096 bytes. There is no separate limit on the total manifest size.

Emitting a mapping

syq map --srcs-in photos     # contents; paths relative to photos
syq map photos              # named directory; paths include photos/
syq map photo.jpg --as albums/cover.jpg

map is local and does not contact a destination. It takes source selectors, -C or --root, source follow options, and --as for one named selection. Copy options and filters belong to the later cp command or your transform. map refuses non-UTF-8 names; hand-written manifests may use base64.

Semantics and limits

• --mapping replaces cp source selectors. Use --into, --into-new, or --into-existing for the destination. It cannot combine with --as or --prune or --detach. Both endpoints may be remote.
• A contents selection emits paths relative to the selected directory. Use that same directory as the consuming copy’s -C base. Named map selectors must be relative and resolve inside their base; a contents selector may point outside it. --root confines either kind of selection.
• Follow options apply to command-line paths, never to manifest entries. A manifest entry that would traverse a symlink fails. Named map selectors followed with --follow-src emit the referent path relative to their base and refuse referents outside it.
• Named manifest paths follow the normal control-path rules: only --follow permits link traversal in that path. Use --mapping - for portable stream input; named FIFOs need Linux with procfs.
• The whole manifest is read and validated before copying. Malformed input, duplicate destinations (including identical duplicate lines), and declared file/ancestor conflicts refuse the run. The destination container may already have been created. Memory grows with manifest size.
• Conflicts found only when inspecting actual source or destination objects fail individual entries. A missing destination parent is created implicitly.
• Normal native preservation applies. kind: "special" checks the type but does not enable special-file copying: use --preserve=specials or those entries are visibly excluded.

Machine-readable results

Add --results r.ndjson to record outcomes in a fresh file outside the copy trees. See Automation results for the stream contract.

Failed mapping entries contain src, dst, and kind, so they can form a retry manifest. First require a terminal result with success or partial: a missing terminal or an early stop means some entries may have no results. In those cases, rerun the original copy instead.

set -o pipefail
syq cp --mapping big.ndjson -C src --to nas --into /data --results r.ndjson
jq -cs 'if (.[-1].type? // "") != "result"
        then "incomplete results stream (no terminal record)" | halt_error
        elif (.[-1].status != "success" and .[-1].status != "partial")
        then "run stopped early (status \(.[-1].status)); rerun it instead of retrying" | halt_error
        else .[] | select(.type == "operation_result"
                          and .disposition == "failed"
                          and .retryable != "no")
             | {src, dst, kind} end' r.ndjson \
  | syq cp --mapping - -C src --to nas --into /data

The filter skips non-retryable entries, including failed implicit parent creation without a source path. It does not guarantee that retrying will succeed; fix the underlying error first. Unchanged and excluded files appear only in summary totals, not as individual results.

For command-restricted copies between servers, --results contains verified receiver receipts. These describe destination changes rather than source entry failures; retry the original mapping instead of applying the operation_result filter above.

Speed

Syq copies files in parallel and adjusts its connection count automatically. Start with the defaults. For repeated remote copies, keep the connection open to avoid logging in each time.

Quick comparison

Use the quick script to get a sense of performance on your own machines:

curl --proto '=https' --tlsv1.2 -fLsS -o try-benchmark.sh \
  https://raw.githubusercontent.com/greaber/syq/master/scripts/try-benchmark.sh
bash try-benchmark.sh

The default pushes 1,024 files of 8 KiB to an SSH host you choose, comparing syq with rsync over three rounds. Use --mode pull for downloads or --mode local to include cp. --workload large selects one 64 MiB file; --workload both runs both workloads. The script checks every copy and cleans up afterward.

Before scoring each network workload, an untimed syq warm-up gives automatic tuning time to refine the remembered connection count. It uses the same direction, transport options and file type as the scored copies. It starts with 64 MiB or 1,024 small files and grows toward 60 seconds of copying, stopping after at most four copies or at 1 GiB per dataset, subject to free space. A slow copy can take longer than 60 seconds; preparation and checks also add time. The script reports the measured duration and warns if it hits a limit before the target. Reaching the target does not prove tuning settled.

Use --warmup off for a short comparison using the existing cached or default count. A cached count alone does not currently skip the warm-up automatically. Warm-up is skipped for local copies, comparisons without scored syq trials, and manual --connections or --tuning-options overrides. The scored dataset size stays the same. For pull warm-ups, identical data is generated locally and remotely and checked, avoiding a large preliminary upload; the remote host needs Bash, OpenSSL, dd and split for that step.

For one scored syq copy with your own options:

bash try-benchmark.sh --yes --mode pull --host j5 --tool syq --rounds 1 \
  -- --no-tcp --connections 1 -v

An untimed tiny copy still prepares the helper. Options after -- apply to syq’s setup, warm-up, calibration and scored copies; path, removal and output-file options are excluded to keep copies inside the disposable dataset. Use --tool rsync for a separate rsync comparison, omitting syq options after --. Full commands and scratch paths appear only with -v, -vv, or --verbose after --. Syq’s extra summary appears with those flags or --stats; the benchmark always reports verified trial results and failures. See tuning options for request-window and small-file batch experiments, and bash try-benchmark.sh --help for all options.

--size quick is the fixed-size default. For longer tests, --size medium uses 1 GiB or 4,096 small files; --size large uses 8 GiB or 16,384 files. --size auto makes verified, unscored syq copies and grows the dataset until copying takes about five seconds or scratch space limits growth. There is no fixed total-data or runtime limit. All scored tools use the same dataset.

The script needs Bash, rsync, OpenSSL and standard Unix utilities locally; syq timing uses Perl’s core JSON::PP module. Remote tests need SSH access and rsync on the other machine. Scratch parents must exist: --source-dir is always local and --dest-dir is the remote scratch parent for push and pull. Generation and checks keep the launch directory visible to tmux.

Generation, helper preparation, warm-up and content checks are untimed. Each scored copy uses an empty destination, with permissions and modification times preserved. Syq persistence is disabled in private settings and rsync uses fresh SSH connections, so the total timer includes connection startup. This leaves the normal learned connection counts active unless you override tuning; trials can reuse and update them. Caches are not flushed and copies do not wait for durable storage. Failed commands or content checks stop the comparison.

Network results use decimal MB/s, averaging trial speeds. Local comparisons use seconds: filesystem cloning can avoid moving bytes, so fast copy times are not disk bandwidth measurements. Each tool may use its normal optimizations.

The main table always uses total command time, including connection startup. A separate syq table shows mean total time, its copying interval, other time (total minus copying), and copying speed in decimal MB/s. Copying speed is copied bytes divided by copying seconds and 1,000,000, averaged across trial speeds. It also appears after each syq trial. A copying interval below timer resolution makes copying speed unavailable. Other time covers work outside that interval, such as setup and finishing. Copying spans the first file work through the last completed work: it includes waiting and per-file overhead, and can overlap planning and connection setup. It is neither pure network time nor an exact separation of setup from transfer. The script does not estimate this breakdown for rsync or cp.

The script adds a note when at least 20% of syq’s total time falls outside copying, or its mean copying interval is under one second. These are diagnostic thresholds, not guarantees that longer tests saturate the link. Short jobs still measure useful completion time; use --workload large --size auto or a larger fixed size to investigate sustained throughput. Compare both tools using their total times, rather than comparing syq’s copying interval with rsync’s total.

Benchmarks

The separate syq-bench project runs more extensive experiments across workloads, storage systems, and network routes. Browse its results, or run its experiments for a more detailed comparison. The quick script above does not use syq-bench.

Diagnose a slow copy

syq cp -vv --stats data --to server --into /backup

-vv shows the chosen transport and connections; --stats adds totals and available TCP statistics.

Compare the same workload and direction using empty test destinations. A second copy into the same destination may just measure skipping existing files.

TCP data connections

SSH authenticates remote copies. When reachable, encrypted TCP carries file data on a port in 47600–47699; otherwise copies use SSH on the same route. See server setup for firewall settings. Copies authorized through another machine require direct encrypted TCP.

Use --no-tcp to select SSH data transport or --tcp-ports LO-HI to choose a port range. On Linux, --tcp-congestion ALGO selects an available congestion control algorithm. --tcp-plain disables data encryption and authentication; use it only on a trusted network. Restricted receivers refuse it.

Local copies and NFS

For local copies, syq uses the filesystem’s copy optimizations automatically when it can. On filesystems that support cloning, this can avoid physically copying every byte. You can also copy to or from a mounted NFS directory using its local path. See storage placement for how the source and destination filesystems affect performance.

Limit bandwidth

Use --bwlimit to leave bandwidth for other work:

syq cp data --to server --into /backup --bwlimit 10M

This limits file data to 10 MiB/s across the copy’s workers. It controls the average copy rate; buffering and protocol overhead can cause network bursts.

Compression and in-place writes

--no-compress can help when compression costs more CPU time than it saves in network traffic. --inplace saves temporary disk space, but exposes incomplete updates to readers. Read in-place writes before using it.

Investigate tuning

Manual connection counts and other tuning options are for troubleshooting and development. Normal copies should not need them.

Server setup

Server settings and storage placement can make a substantial difference, especially on long-distance links or when syq must carry data over SSH.

Make TCP reachable

Allowing syq’s encrypted TCP connections often gives the biggest improvement. The server listens on one available port in 47600–47699 for the duration of the copy. Choose another range with --tcp-ports LO-HI.

For a server using ufw, an administrator can allow a trusted client:

sudo ufw allow from <trusted-client-address> to any port 47600:47699 proto tcp

Allow the port range in any cloud firewall too. Check the selected route with syq cp -vv --stats. Copies, including the default direct server-to-server mode, fall back to SSH on the same route when TCP is blocked. Copies authorized through another machine still require direct encrypted TCP; they never relay file data through the authorizing machine.

Tailscale

Tailscale is a useful way to make servers reachable across NAT and firewalls without exposing syq’s ports to the public internet. Allow the data ports through the host firewall and your tailnet’s access rules; syq can discover the Tailscale addresses.

TCP over Tailscale can be faster than sending data over SSH without it. The tunnel can also limit throughput, especially if Tailscale uses a relay. Compare on your own route, and use tailscale status to check whether the connection is direct. See Tailscale’s performance guide.

Test congestion control

TCP’s congestion-control algorithm decides how quickly to send and when to slow down. On long-distance links with packet loss, BBR can be dramatically faster than CUBIC: loss does not always mean the link is full. It is worth trying even when copies finish successfully. Results depend on the route and direction; BBR does not win everywhere.

On Linux, check both endpoints:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_allowed_congestion_control

If bbr is available and permitted on both, try it for a copy:

syq cp --tcp-congestion bbr --stats data --to server --into /backup

This changes only syq’s TCP sockets, not the host default. If BBR is missing, ask the server administrator to enable it; see the BBR setup guidance. The option cannot tune SSH’s own connections.

Compare with --tcp-congestion cubic using the same data and an empty test destination each time. Try both directions. Use --bwlimit if you need to leave bandwidth for other users.

Let SSH connections start promptly

When data travels over SSH, syq opens several connections. OpenSSH’s usual MaxStartups 10:30:100 starts randomly rejecting logins once ten are still authenticating. Syq retries and reduces simultaneous logins, so the symptom can be a slow start rather than a failed copy.

For a server handling parallel transfers, an administrator can consider:

MaxStartups 100:30:200

This allows 100 simultaneous unauthenticated connections before random rejection begins, and rejects all new ones at 200. It also allows larger bursts from unrelated clients, so choose limits that suit the server.

MaxSessions is a different limit: channels sharing one SSH connection. Very low values can force extra logins when syq tries to reuse a connection. Larger copies also open independent connections, so raising MaxSessions alone does not remove login limits.

Validate configuration changes with sshd -t, then reload SSH using your system’s normal procedure. Keep an administrative session open while doing so. See OpenSSH’s settings.

Check local storage placement

For a local copy, identify the filesystems containing the actual source and destination. Two directories on one server can use different disks, filesystems, or an NFS mount. On Linux, inspect each path with:

findmnt -T /path/to/source -o TARGET,SOURCE,FSTYPE,OPTIONS
findmnt -T /path/to/destination-parent -o TARGET,SOURCE,FSTYPE,OPTIONS

Use an existing destination parent if the destination does not exist yet. For remote paths, run these checks on the machine that owns the path. An NFS mount’s type does not identify the server’s backing filesystem.

Eligible local copies use the filesystem’s copy support automatically. Within a filesystem that supports cloning, such as XFS with reflink enabled, the filesystem can share data extents between source and destination instead of copying every byte. The files remain independently writable. Reported throughput then measures logical file bytes, not physical disk traffic.

If your storage requirements allow it, keeping a local source and destination on the same filesystem can make this optimization available. Copying between filesystems cannot share those extents. This does not mean XFS is always faster than ext4: the devices, workload, and available copy operations also matter. See local copies and NFS for syq’s other copy paths and NFS mount considerations.

Measure and track improvements

Use syq-bench to compare settings on your machines and save repeatable results over time. Test the workloads and transfer directions you actually use. Record elapsed time and throughput alongside tool versions and commands.

When results differ between servers, check these conditions before changing system settings:

• Transport and parallelism: inspect the selected transport and connection count with -vv --stats. For a controlled worker-count comparison, use --connections N; see benchmark tuning. Use the same reporting options in each run, and start with defaults for everyday copies. More workers need not help once storage or an NFS service is saturated; compare repeated runs before choosing a lower count.
• Storage and cache: record both mounts, whether source data is cached, and whether timing includes a final storage flush. Buffered copy completion and completion after flushing measure different things; compare like with like.
• Other work: check CPU, local disk, and network activity at both ends. With shared NFS, other clients can compete for the same service even when your own machine is otherwise idle.
• CPU policy: record the governor and observed clock speeds where available. A governor change can affect tools differently, so a gain for rsync does not establish a gain for syq. Compare your workload before making a persistent change; performance is not a general requirement for syq.

Security

Syq is designed for copies where the tool has more authority than some of the files or machines involved: a backup account reading another user’s tree, root copying an upload, or two servers transferring through your laptop.

Report vulnerabilities through SECURITY.md.

Filesystem attacks

Assume an attacker can change files and directory entries inside a selected tree, or in a parent directory they can write. They may act before the copy starts or while it is running. They do not control the account running syq.

The attacks we aim to stop include:

• Redirecting a read or write with a symlink. A directory might be replaced with a link to a private source or unrelated destination.
• Escaping through a supplied filename. A peer might send an absolute path or .. components to reach outside the selected tree.
• Turning a deletion into a walk of another directory. An entry might change type after syq has decided what to remove.

Native syq refuses symlink traversal unless you explicitly request it. It keeps selected directories open and works through those handles, so later renames cannot redirect the copy to another tree. Names received from the other endpoint are validated; deletion does not follow replacement links or recurse into unexpected directories.

These protections bound where the operation can go. They do not prevent an authorized writer from changing the contents of that tree. In a removal race, a single replacement entry can still be unlinked.

Relationship to rsync 3.5.0

Syq’s path protection is inspired by rsync 3.5.0’s security design: keep directories open, work relative to them, and distrust peer-supplied paths.

The default for paths you type is simpler in native syq:

Rsync’s ownership policy preserves familiar symlinked directory setups. Our concern is that link ownership alone does not prove who placed it there: rename permissions come from its parent directories, subject to restrictions such as the sticky bit. A relative link moved to another directory can point somewhere different without changing its owner. See the rename rules.

Native syq avoids that implicit trust decision. This is a stricter default for this case, not a claim that syq is more secure overall. syq rsync keeps the ownership-based policy for compatibility. Its local-only --insecure-links option permits foreign-owned symlinks in typed local source, destination, and control paths. After opening a source root, scans and content reads still use its directory handles and refuse descendant symlink traversal.

TCP data connections

TCP workers authenticate using credentials delivered through the control connection. The initial handshake has a ten-second deadline, but there is no general copy I/O timeout: a stalled peer can leave a copy waiting until you cancel it.

Syq bounds incoming messages, decompression, and decoded collections to limit memory use from malformed peers. These are not a total process memory cap; memory also grows with connection count and request size. Invalid replies fail the connection visibly. Ordinary copies can fall back to SSH if TCP setup fails, keeping the same endpoints.

A compromised source server

Syq checks peer-supplied paths and data ranges before using them. These checks reject malformed replies; they cannot establish that a source’s file listing or contents are truthful.

For a default direct remote-to-remote copy, the source gets permission for one transfer, not your SSH agent or a reusable destination credential. The restricted receiver independently enforces the allowed destination paths, write and deletion permissions, and limits. Selection based on source facts, such as --skip-newer timestamp comparisons, relies on the source’s reports. The source cannot enlarge or replay that permission. SSH and encrypted TCP workers share the same live receiver and copy limits. Falling back to SSH keeps file data on the source-to-destination route; relaying through your machine requires explicit --coordinate-at local.

The receiver signs what it did, and your machine verifies the receipt. A source cannot forge a clean account of destination changes. It can still omit files, invent content or metadata, or stop. A receipt does not prove the source supplied everything you intended.

The destination machine, receiver, and account remain trusted. Other authentication choices have different boundaries: broker-only authentication permits that destination account’s full authority during the session; full agent forwarding exposes your agent as ssh -A would.

Named receiving destinations

A named destination lets a server account request copies through an outbound connection maintained by your laptop. persist on enables this for syq’s SSH connections by default. Each request requires approval on the receiving machine through a desktop prompt or persist receive approve. Paths and limits are validated before prompting; the restricted filesystem executor checks every operation after approval. The server receives no SSH agent. Commands need separate approval.

Approval permits that pending copy’s destination, overwrite policy, and limits. It does not authenticate what you typed on a remote server or attest to source contents. The receiving user and desktop session remain trusted. Request IDs are local, expire after five minutes, and cannot be reused. Disconnecting or stopping receiving cancels pending decisions. Desktop failure never approves a copy. syq persist receive on --approve always explicitly removes the per-copy decision and trusts connected server accounts for repeated copies.

The default starting directory is your home directory, with no containment. syq persist receive on --root DIRECTORY contains copies; syq persist receive off disables receiving while keeping ordinary persistence. A compromised connected server account can request more copies and invent their content. Once approved, it can inspect destination entries during copy planning and consume disk space within the approved limits. The laptop’s receiving account is trusted.

Bare destination names fall back to ordinary SSH while the laptop is offline. Use @name when you require a return connection and want failure instead of host resolution. A copy never switches routes after selecting its destination.

Limits to keep in mind

• Privileged copies need trusted destination directories. Do not copy as root into a directory writable by untrusted users. Syq writes private partial files. When reusing bytes from another partial, it only reads regular files owned by its effective user. It checks those bytes against source hashes and leaves the other partial unchanged. This does not make a shared writable directory trusted.
• Hard links share contents and metadata. In-place writes and metadata changes through a destination hard link affect every name for that file, including names outside the selected destination or a restricted grant’s path scope.
• Copies are not snapshots or transactions. Stop concurrent writers or use snapshots for consistent data. --inplace exposes incomplete updates. Syq does not fsync transfer data, so completion is not a power-loss durability guarantee.
• Preserving authority is a choice. Leave --preserve=ownership and --preserve=permissions off when copying from an untrusted source.
• Protocol assurance is still developing. Syq’s process protocol has not been fuzzed as extensively as rsync’s.

Code and transport integrity

File data is encrypted and authenticated by default. --tcp-plain sends file contents, protocol messages, and the worker authentication token in plaintext. An observer can steal the token and connect as a worker while the transfer is active; a network attacker can also alter traffic. Use it only on a network you trust. Downloaded code for remote operations and explicit self-updates is verified against a signed release manifest before use. That verification cannot protect a machine whose trusted account or programs have already been compromised.

A server can also request a copy to another SSH host using a live receiving machine’s SSH access. Eligible copies discover that machine automatically; --auth-from @name chooses it explicitly. This always needs a local decision, even if automatic receiving is enabled. Refusing the request ends the attempt; syq does not try another authorization source. Approval authorizes a connection using the receiving machine’s SSH access and installation of a matching helper. The destination helper enforces one copy’s paths, permissions and limits. The server gets a restricted control stream and encrypted TCP worker access for that copy; it gets no SSH agent, private key, or command-running interface. Host trust and SSH configuration are those of the approving machine. The destination account remains trusted, including its interpretation of relative paths. A compromised source can substitute content within the approved scope, just as with other return copies.

Approved commands on receiving machines

syq exec uses an existing return connection and always asks for a local decision before starting a program. Command approval is separate from copy approval, including when copies are automatically approved. The prompt shows the server account, argument list and working directory. A server account can request commands from any of its processes; syq cannot establish what a person typed in a remote shell.

Approving execution grants the command your local user’s authority. Copy root confinement, file protection and transfer limits do not restrict that program. Scripts and build files can change what it does. Commands receive the local service environment and closed stdin. They do not expose a general SSH agent forwarding interface, but an approved program can access credentials available to the local user.

Disconnecting cancels the foreground process group; commands are never replayed automatically. Completed effects cannot be rolled back, and programs that create separate process sessions can outlive cancellation. See the command reference for execution and interruption behavior.

Human copy listings escape control characters in filenames. Diagnostics also escape terminal control sequences from peers; NDJSON keeps its JSON encoding.

Persistent connections

An open SSH login can be reused by other processes running as your local user without another key touch or agent approval. Persistence keeps that access available until the connection closes. syq persist off ends it; use syq persist receive off to stop incoming requests while keeping SSH reuse.

Persistence also enables receiving by default. Its approvals and copy limits are separate from the SSH login’s authority.

Isolated script scopes reuse SSH logins without enabling receiving. Stop background services when upgrading; replacing a binary does not change services already running it.

Remote copy reference

For the usual setup and copy commands, start with Copy between servers.

SSH configurations

Host-certificate-only trust, KnownHostsCommand, and RevokedHostKeys are unsupported by the broker. Custom known-hosts paths must be unambiguous: one absolute, whitespace-free filename per configured user/global directive. The default known-hosts file list works. Syq reports unsupported configurations instead of relaxing host verification.

Your local SSH configuration selects hostB’s login, address, port, and trusted host keys. HostA’s SSH configuration does not override those choices.

Enrollment

Enrollment needs normal command authority on the destination during setup. Use a real directory, not a symlink. Transfers cannot overwrite the receiver’s SSH configuration, programs, or enrollment state. Manage that state with syq receiver commands; it is not a disposable cache.

Repeating enroll updates the receiver to match your local build. A pending enrollment can be retried or revoked. Revoke and enroll again to rotate its receipt key. Revocation stops active receivers before removing their state; see access management for interruption and retry behavior.

Stop active copies before upgrading: replacing the executable does not update running receivers. Repeat syq receiver enroll hostB:/destination afterward to refresh the installed receiver. Compatible enrollment keys and replay records are preserved. Different client builds cannot share one installed receiver concurrently; each needs a matching receiver.

An incompatible enrollment requires fresh setup, which needs ordinary SSH access. Eligible copies install it automatically, or you can use enroll. Incompatible old enrollments are not listed or removed by the new build; they require manual cleanup on both machines.

Enrollment detects the destination platform and installs a matching executable. Source builds require compatible platforms; --syq-path does not select the restricted receiver. See Developing syq for testing source builds.

Limits and unsupported options

Default limits are 100 million entries and 8 TiB of file data. Override them for one transfer with --receiver-max-entries N and --receiver-max-bytes SIZE. Values can raise or lower the defaults within the allowed ranges. The transfer must start within 24 hours of authorization and finish within seven days of authorization.

Restricted copies use encrypted TCP when reachable and otherwise use SSH workers from the source to the destination. Both transports share the same copy authorization, limits, revocation, and signed receipt. A failed direct connection never selects a relay through your machine; choose --coordinate-at local explicitly to send data through it.

Signed results

The destination signs a receipt and your machine verifies it before reporting results. Use -v for totals or --results FILE for automation. These results arrive after verification, rather than as live per-file progress. For --dry-run --results, use --coordinate-at local to get the preview stream.

--receiver-receipt digests adds BLAKE3 hashes of affected regular files. Receipts allow up to four million records and 512 MiB of plaintext. Reaching a cap stops further changes and reports an incomplete outcome.

A receipt does not prove that the source supplied every intended file or the right contents. See the threat model.

Other authentication modes

To make hostB pull from hostA using credentials already on hostB:

syq cp --coordinate-at dst --peer-auth own-credentials --from hostA data --to hostB --into /archive

There is no restricted source receiver, so direct pulls require one of these alternatives to the default authentication.

Detached copies

--detach leaves the copy running after your command exits. It requires --peer-auth own-credentials or an explicit --rsh policy, so the coordinating server can authenticate independently. It cannot use the restricted receiver or --results.

Save the reported remote log location. The log is not a signed receipt. If printing the location fails, the command reports an error but the job may still be running. The coordinating server needs /bin/kill and either setsid or perl.

Authorization selection

For syq cp with local sources and an SSH destination, --auth-from auto tries live receiving machines in alphabetical order, allowing up to two seconds for each reply. Offline or unsupported connections are skipped. With none available, or with unsupported options, it uses the source machine’s SSH access. Once approval is requested, refusal or failure ends the attempt.

--auth-from NAME and --auth-from @NAME require that receiving machine. Names ssh and auto need the @ prefix to distinguish them from the option values. --via NAME always means a receiving name. --auth-from ssh always treats --to as an SSH endpoint, even if it matches a receiving name.

Authorization through a receiving machine does not support --detach, custom --rsh or --syq-path, --no-bootstrap, --pscope, alternative --peer-auth or --coordinate-at, --no-tcp, or --tcp-plain. It requires direct encrypted TCP from source to destination. Destination completion does not request permission through a receiving machine; use --auth-from ssh for completion through the source’s own SSH access.

On this route, quoted ~ and ~/archive select the destination account’s home directory. Use ./~/archive for a literal directory called ~. Avoid ~//archive: explicit receiving authorization keeps it under the home directory, but automatic selection uses ordinary SSH, where it resolves to /archive.

Verification

For comparisons between two servers, --coordinate-at local uses ordinary SSH access from your machine to both endpoints. It needs no restricted receiver enrollment and supports --verify-only --results FILE. Files are hashed on the servers; your machine compares their hashes and receives listings and results, not the full file contents.

Direct restricted verification requires an existing enrollment and cannot produce --results; a receiver receipt cannot attest to the source’s comparison. Verification never installs an enrollment.

--verify-only cannot combine with --dry-run, --prune, --inplace, or overwrite policies. Filters and size limits select the entries to compare; special files require --preserve=specials. Metadata is not compared, but device identity is. A requested results file and remote helper caches may still be written.

Persistence details

For everyday setup, start with Send files home from a server.

Names and profiles

Give different receiving locations their own names:

syq persist receive on --name laptop --cwd ~
syq persist receive on --name project --root ~/work/project
syq persist connect server

Both names work from the same server account: syq cp results --to @project and syq cp report.pdf --to @laptop. Each profile has its own directory, copy root, limits, approval policy, and background connection to each connected server. New profiles start with the usual defaults, including asking for approval; they do not inherit another profile’s trust or confinement settings. Up to 32 profiles can be saved.

receive on --name NAME creates a profile or updates that name’s settings. Without --name, receive on updates the first saved profile, shown first by receive status. The initial hostname profile becomes a saved profile when persistence first connects; adding a new name then keeps that original profile. If no preferences have been saved yet, the first explicitly chosen name replaces the implicit hostname default.

syq persist receive status
syq persist receive status --name project
syq persist receive off --name project
syq persist receive on --name project
syq persist receive remove project
syq persist receive wait server --name laptop --timeout 30

Updating, stopping, or removing a profile cancels only that profile’s copies, commands, and pending approvals. Other profiles keep working. receive off without a name stops all profiles; receive on enables the first profile, and receive on --name NAME enables another. Removing the first profile makes the next saved profile the default. The last profile can be disabled but cannot be removed. pending, approve, and deny work across all profiles; prompts name the receiving profile. Without --name, receive wait waits for every enabled profile on that server.

Names belong to a server account. Different laptops can receive through the same account under different names. If a name already has a live connection, another client’s attempt is rejected without disturbing the existing connection. Other profiles remain usable. Choose a different name, or stop the original connection and run syq persist connect server on the waiting client to retry.

Directories

--cwd chooses the starting directory; --root also contains copies within it. Both require an existing directory with a UTF-8 path. Invalid settings leave the saved configuration unchanged. If a saved directory disappears, you can still inspect, disable, or reconfigure the profile. Filenames inside it may use normal Unix filename bytes.

Copy limits

Copies support directories, symlinks, modification times, filters, hashing, resume, mappings, --preserve=permissions, --verify-only, and the overwrite policies. Ownership and special-file preservation, --inplace, and --min-size are unsupported. Timestamp comparisons trust the source’s reported modification times.

Each copy is limited to 100 GiB and one million touched entries by default. Change these ceilings with syq persist receive on --max-bytes 20G --max-entries 100000. Lower limits requested by the sender also apply. Limits are per copy; repeated copies can fill the disk. Copies support at most 32 workers each.

Pruning is disabled unless the laptop sets a positive --max-delete. A sending --prune command must also supply its own --max-delete ceiling, no higher than the laptop’s. Validation failures leave the copy unstarted. Errors during copying fail visibly and may leave partial files for retry. The sender verifies a signed receipt before reporting success.

Connection lifetime and waits

persist on enables persistence for subsequent syq SSH connections. persist connect server enables it and connects immediately. With receiving enabled, it waits until receiving is ready. --timeout 30 limits that wait after SSH and helper setup, not authentication or installation. Failure leaves persistence enabled; a healthy connection is reused without cancelling requests.

Connections have no idle expiry. Receiving reconnects after interruptions; other SSH logins reopen on their next use. Syq installs no login service, so connect again after reboot. Copies are not queued or retried automatically. A return copy must finish within seven days of approval.

Use syq persist receive wait server --timeout 30 to wait without starting or restarting a connection. On the server:

syq persist destinations list
syq persist destinations wait laptop --timeout 30
syq persist destinations forget laptop

forget removes a stale entry while its connection is stopped. For structured status and approval requests, see connection status.

Isolated script scopes

syq persist on --ephemeral prints a scope path. Pass it as --pscope PATH to syq persist connect server and subsequent copy commands, then close it with syq persist off --pscope PATH. This does not change your user setting.

These scopes reuse SSH logins only. They do not enable receiving, authorization through your machine, or commands on it. Idle connections close within ten minutes, including helper sessions. Explicitly closing the scope ends reuse immediately. For return copies or commands, use persistence without --pscope.

Setup and recovery

Linux desktop prompts need libnotify 0.7.10 or later and a running notification service. macOS uses a native dialog. Start receiving from a terminal in your desktop session. After changing sessions, run syq persist receive off, then enable the profiles you need from a terminal in the new session.

syq persist receive pending shows complete requests and prompt errors. Overwrite warnings are advisory; destination entries can change before copying. --notify off selects terminal approval; --notify desktop restores prompts.

Start the connection from your laptop with syq persist connect server. Opening a plain SSH session does not enable receiving, and connecting onward from that server to another host does not carry your laptop’s receiving name with you.

For automatic reconnection to work, your SSH key or agent must be available and the server’s host key must already be trusted. The background service cannot ask for a password. The server must allow remote Unix socket forwarding; OpenSSH 9.2 also requires permission for remote TCP forwarding.

The server command uses the matching helper installed by the receiving connection. If that helper is missing or does not recognize an option, update syq on both machines and reconnect from your laptop.

Updating connections

Before upgrading, run syq persist off on the receiving machine to stop its background services. Replacing the executable alone does not update running services. Close script scopes with syq persist off --pscope PATH too. After upgrading both machines, run syq persist connect server for each server.

Saved names, directories, and limits carry over. Settings predating approval prompts require approval after upgrading. Older binaries may not read updated preferences; use the newer binary to manage receiving.

Tuning options

Syq adjusts performance automatically. These controls are for developers and for investigating copies where the defaults perform poorly. These experimental controls appear in --help-all; their keys and bounds may change between releases.

--connections N (or -j N) fixes the connection count and disables automatic adjustment. In syq rsync, use --syq-connections N. Use the same count when comparing other tuning settings. Leave it unset for everyday copies.

Remembered connection counts

Remote copies start from the last learned count for the same host route, direction and transport, or from 8 workers over SSH and 16 over TCP. The cache normally lives at ~/.cache/syq/tuning.json (XDG_CACHE_HOME can change its parent). The quick benchmark uses this cache, even though it disables SSH connection persistence. Its temporary file paths do not change the cache key.

Learning takes time: syq samples every 2.5 seconds and needs a warm-up interval plus two stable samples for one measurement. Saving a count requires a successful copy with at least two measured worker counts and an unchanged transport. Failed or aborted copies leave the previous cached count intact. If a copy finishes during another probe, syq saves the last accepted count. Short copies may only use their starting count. The benchmark’s untimed warm-up allows more time before scoring; neither its 60-second target nor the separate five-second automatic sizing target guarantees tuning has settled.

--connections N disables automatic adjustment and cache use. Supplying --tuning-options bypasses reading and updating learned counts, but live auto-tuning continues unless you also fix --connections. Use -vv to see when syq starts from a remembered count.

Transfer controls

syq cp and syq rsync accept --tuning-options. Supply comma-separated KEY=VALUE pairs:

syq cp large-file --to server --as /scratch/benchmark-copy \
  --connections 1 -v \
  --tuning-options copy-path=ranges,request-size=1M,pipeline-depth=8

Sizes accept K, M, and G, using powers of 1024. Unknown keys, repeated keys, and out-of-range values fail the command. Overrides apply to the remote coordinator too. They are not saved, and these runs neither read nor update the remembered connection count. Connection auto-tuning still runs unless you fix --connections (--syq-connections with syq rsync). These are experimental controls whose keys and bounds may change between releases.

Larger requests reduce overhead per byte; deeper pipelines allow more requests to await replies at once. Both can increase memory use. Neither changes the hash blocks used for integrity checks and resume.

copy-path=ranges disables small-file batches and whole-file shortcuts, including local kernel copying. Matching data can still be skipped or reused. auto lets syq choose normally.

Streaming and request windows

Syq automatically streams larger remote ranges (usually above 16 MiB), using blocks of at most 2 MiB by default. Streaming still checks contents and write errors and supports resume. No setting is needed for everyday copies.

For comparisons, copy-path=streaming forces streaming and disables whole-file and small-file shortcuts. copy-path=auto-streaming keeps those shortcuts and streams the remaining ranges. Both include local and short ranges.

An explicit request-size also sets the streaming block size; bandwidth and receiver limits may reduce it. An explicit pipeline-depth disables automatic streaming. Neither forced streaming mode accepts pipeline-depth.

To test the amount of outstanding large-file work over SSH, keep the worker count and copy method fixed, then vary only the request window:

bash try-benchmark.sh --yes --mode pull --host j5 --workload large \
  --tool syq --rounds 1 --size quick -- --no-tcp --connections 1 -v \
  --tuning-options copy-path=ranges,request-size=1M,pipeline-depth=4

Repeat with pipeline-depth=8 and 16. With 1 MiB requests, these allow up to 4, 8 and 16 MiB of outstanding range requests per endpoint per worker. They do not resize TCP or SSH flow-control windows. Compare separately with a run omitting --tuning-options: long remote ranges normally stream, and an explicit pipeline depth disables that behavior. The ordinary defaults already allow 4 MiB × 4 requests; a larger application request window may not help. Use a larger fixed size if the timing notes show the test is too short.

For the small-file workload, vary batch-files and batch-bytes instead, such as --tuning-options batch-files=512,batch-bytes=4M. Those are batch ceilings, and the scheduler may choose smaller batches. pipeline-depth does not multiply small-file batches. After testing these controls, vary --connections separately to assess parallelism and its startup cost.

For a direct comparison without the benchmark script, use fresh scratch destinations:

syq cp data.bin --to host --as /scratch/pipeline.bin --connections 1 -v \
  --tuning-options copy-path=ranges,request-size=1M,pipeline-depth=4
syq cp data.bin --to host --as /scratch/streaming.bin --connections 1 -v \
  --tuning-options copy-path=streaming,request-size=1M

Streaming can be slower on short or CPU-limited copies. Memory use depends on request size, worker count, compression, and transport buffering. With --bwlimit, a remote source can send ahead of paced destination writes, so the limit is an average copy rate, not a strict cap on incoming bursts.

Batch size and splitting

For example, compare small-file batches with:

syq cp --srcs-in small-files --to server --into /scratch/benchmark-small \
  --connections 1 -v --tuning-options batch-files=256,batch-bytes=8M

Explicit batch controls replace the small-copy shortcut with worker batches. File and byte limits are ceilings; syq may choose smaller batches. Files larger than the byte limit use another copy method. With --bwlimit, each batch contains at most one file. Batch controls cannot combine with copy-path=ranges or copy-path=streaming; auto-streaming accepts them.

split-min-size sets the smallest file region an idle worker can take from another worker. Lower values allow finer sharing; higher values avoid small assignments. Splits align to hash blocks and need at least twice the minimum remaining size.

Average rate and burst patterns

--bwlimit caps logical file-data bytes per second across workers, before compression, encryption, and protocol overhead.

• Timed pacing (bw-pacing=125ms, the default) sends smaller requests at regular intervals. The first request can start immediately, so a short copy can exceed the average by that initial request.
• Average pacing (bw-pacing=average) waits for each request’s byte budget before sending it, including the first. It allows larger requests, which can arrive in bursts. A 2 MiB request at 1 MiB/s waits about two seconds.

For example, to test larger requests under an average rate cap:

syq cp large-file --to server --as /scratch/benchmark-capped \
  --connections 1 --bwlimit 1M -v \
  --tuning-options copy-path=ranges,request-size=4M,bw-pacing=average

Neither mode limits the size of every network burst. Streaming, queues, and transport buffering affect when bytes cross the link. Restricted receivers also enforce their authorized rate and request-size limits. Measure both sustained throughput and short-interval traffic when comparing capped runs.

Recording a comparison

With overrides, -v reports effective request sizes and a final syq: tuning observed: diagnostic with copy-method counts and request and batch sizes. Retries can count more than once. These experimental diagnostics are separate from completion records.

Use the same reporting options and fresh disposable destinations for each comparison. Prefer -v: --stats bypasses the small-copy shortcut and can change what you measure. Use explicit defaults for the baseline, such as --tuning-options copy-path=auto.

Record the source data, transport, connections, settings, elapsed time, CPU use, and peak memory. Check exit status and copied contents. Leave --bwlimit unset when measuring unrestricted throughput.

Automation results

Native cp and rm can write structured outcomes alongside human output:

syq cp --srcs-in project --into backup --results copy.ndjson
syq rm old-output --results removal.ndjson

Each line is a JSON record. Require a final result record before trusting completion. EOF without one means the outcome is unknown, even if no error record appeared.

This page describes stream semantics. The JSON Schema lists precise field shapes; example streams show complete runs. Mapping manifests are a different format.

The channel

--results FILE creates a fresh regular file and refuses an existing one. Use a new name for each run and keep it outside trees being copied or removed. --results-fd N uses an already opened writable descriptor above 2:

syq cp project --into backup --results-fd 3 3>copy.ndjson

Use a descriptor for pipes or other non-regular sinks. Named results paths follow the normal native symlink rules; --follow permits traversal. Human stdout and stderr are not part of this contract.

Results are written on the invoking machine, including for remote removal. For direct restricted remote-to-remote copies, they are derived from the verified destination receipt and marked provenance: "receiver_attested". Those records arrive after receipt verification. Otherwise remote-to-remote results require --coordinate-at local, which relays data through your machine. Remote-to-remote --dry-run --results and --verify-only --results also require that local route.

--results supports native cp (including pruning) and rm. It cannot be combined with --detach. The restricted receiver supports copy changes, including capped pruning, but not native removal.

Records arrive in seq order; error and terminal records are flushed immediately. If writing the stream fails, syq warns where possible and stops writing it. Filesystem work can continue, leaving an incomplete stream.

Argument errors exit 2 without a stream. Failure to open the results sink exits 1 without a stream. Once opened, completed runs and fatal setup failures emit a terminal record if the sink remains writable. Crashes and interruptions can leave it missing.

Consumer rules

• Check the terminal record and process exit code; a mismatch is a protocol error. Use terminal totals, not progress or a count of operation records.
• Build mapping retries only after terminal success or partial. Other statuses can leave entries unresolved.
• Ignore unknown record types and optional fields within a supported schema version.
• Reject unknown schema, schema_version, terminal status, or path encoding values.
• Treat human message text as display only; parse structured fields.

--progress-json is a separate progress display whose format may change. Use --results when you need a stable consumer contract.

Record envelope

Every record carries:

Paths are tagged: {"encoding":"utf-8","value":"docs/a.txt"}, or encoding: "base64" with standard base64 of raw filename bytes. Byte and count fields are non-negative integers within the u64 range.

Record types

run

Always first. Identifies the invocation with run_id, started_at (Unix seconds), syq_version, mode (cp or rm), dry_run, and endpoints. Endpoints identify role, local/SSH kind, and SSH host/user; they omit credentials, ports, and raw command arguments.

Copy runs also carry prune and mapping. Compare-only runs add optional verify_only: true; absence means false. Removal has one source endpoint regardless of selector count and omits those copy fields.

With --verify-only, differences and inspection failures produce error records and a nonzero terminal status. Matching regular files count as files_unchanged and bytes_unchanged; transfer and creation totals remain zero. Successful comparisons do not emit copy operations. Progress bytes measure comparison work, not bytes written. As with dry runs, use --coordinate-at local for JSON comparison results between two remote hosts; a receiver receipt cannot attest the source’s comparison claims.

progress

Sampled telemetry, approximately once per second, for displays rather than accounting. It includes bytes, files, exclusions, scan state, and elapsed time. Removal has zero byte and unchanged/excluded counts; its file counts reflect outcomes received so far. The terminal record owns final totals.

trace

One intended copy change in a dry run. Includes action, destination, source for mapping entries, kind and bytes where applicable, plus a reason: destination_missing, type_differs, content_differs, metadata_differs, or destination_only.

A trace cannot be matched by identity to a later live operation: the filesystem may change between runs.

operation_result

An outcome for a completed copy change or a failed mapping entry.

In attested records, dst is relative to the destination area identified by scope. An attested set_metadata omits kind.

Unchanged and excluded entries have totals only. Ordinary live streams do not emit per-operation records for metadata-only updates, though dry runs emit metadata_differs traces. Failed implicit parent creation can lack src and is non-retryable. Do not construct a retry source from its destination name.

Removal records

selection_result uses a zero-based selector index, the original path, status (resolved or missing), and kind when resolved. Missing selectors succeed. Overlapping or duplicate selectors keep separate indexes. If a later selector cannot resolve, earlier selection records may precede the fatal error and terminal record.

removal_trace includes selector, path, kind, and disposition: "would_remove". Directories follow their descendants.

removal_result includes selector, path, kind when known, attempts, and disposition: removed, already_absent, or failed. Already absent is success. Failures include error details and retryability where available, and also produce a counted error record. Dry-run inspection failures can produce failed removal results, but never successful removal results.

error

One per counted error. message is display text; class and os_kind are provided where known. Classes are io, transport, conflict, integrity, safety_limit, usage, and internal.

OS kinds are not_found, permission_denied, already_exists, invalid_input, no_space, quota_exceeded, read_only, and other. They preserve OS error meaning across hosts without requiring matching errno numbers. Receiver refusals use class: "safety_limit" with provenance and the receiver’s code.

final_state

Attested streams only: the destination’s final observation of a path the transfer could have changed. Includes scope, dst, and an object: absent, an observation failure, or present with kind, size, applicable metadata, and symlink target. With --receiver-receipt digests, regular files also have a BLAKE3 digest.

Object kinds distinguish directories, files, symlinks, FIFOs, sockets, character/block devices, and other objects. Metadata fields are mode, uid, gid, mtime, mtime_nsec, and rdev. Consult the schema for exact shapes. This observes destination state; it does not attest source completeness.

result

Exactly one terminal record, always last when the stream completes. Common fields are status, exit_code, dry_run, errors, and elapsed_ms.

Copy terminals may also include copying_elapsed_ms: the wall-clock span from first file work to last completed file work across workers, including per-file checks, finalization and gaps. Initial setup before file work is excluded; planning and connections can overlap this interval. It is not a sum of worker times or pure network time. The field is absent when no bytes moved or the coordinator does not supply it, including older releases and attested terminals. Use elapsed_ms for end-to-end throughput comparisons.

Copy totals include transferred/unchanged/excluded files, created directories, symlinks and specials, transferred/unchanged bytes, and on pruning runs deletions_planned, deletions_completed, and deletions_blocked. A fatal failure reports what finished before it stopped. Dry-run totals describe planned work, not committed changes.

Attested terminals add receipt_status (clean, failed, or incomplete), provenance, and receipt counts. They can attest only what the receiver saw:

• Unchanged and excluded totals are zero, not a claim that every source entry changed.
• Only deletions_completed appears; planned and blocked deletion totals are omitted.
• errors counts attested error records for failed/incomplete operations, refusals, and failed or partial final-state observations.

Removal terminals have mode: "rm" and totals for selectors, resolved and missing selectors, planned/removed/already-absent entries, and failed entries. Live removal leaves entries_planned zero. Dry removal leaves entries_removed and entries_already_absent zero, but inspection failures may increase entries_failed.

The human summary uses the same terminal totals.

Exit codes

Removal terminals use only success, partial, or failed. A results-sink startup error also exits 1 without a stream.

Compatibility

Within a schema version, required fields keep their types and meanings; existing types, actions, dispositions, statuses, classes, and reasons are not renamed or reused. New record types and optional fields may be added. Human messages may change at any time.

Connection status

syq persist status --json reports the persistence setting, scope, and endpoints without starting connections. Endpoint states are starting, connecting, ready, reconnecting, failed, or inactive. Each entry also reports whether SSH is connected and the receiving state and errors.

With receiving enabled, ready means the return connection is online; SSH can reconnect on its next use. If receiving preferences cannot be read, SSH entries are still listed, receiving_error explains the failure, and each entry’s receiving_enabled is null.

Command approvals in syq persist receive pending --json use kind: "command" and include argv, cwd, and permission. Argument and directory strings in this summary are escaped for display. Use an up-to-date syq binary to inspect and approve commands; clients that only support copy requests omit them.

Rsync compatibility

syq rsync accepts common rsync commands for local copies, pushes, and pulls. It uses its own protocol: the remote program must be syq, not rsync. Run syq rsync --help-all for all accepted flags.

syq rsync -av project/ server:backup/project/

Differences to check before switching

A failed source or destination scan prevents deletion. A per-file read failure during transfer does not by itself prevent deletion: that source entry remains present, so its destination counterpart is not an extra. Preview deletion scope with --dry-run -v; see deletion rules.

Syq uses numeric IDs and always keeps partial files, so --numeric-ids and --partial are accepted no-ops. -P enables progress. Compression is on by default; -z does not enable anything extra. -B / --block-size changes syq’s fixed hash and resume block size.

Unsupported features

Unsupported common flags are rejected with an explanation. Selected symlink targets are copied unchanged. --insecure-links does not enable any of the unsupported descendant-link options.

File-list parsing and help

--files-from cannot combine with syq ignore rules or deletion. A listed source whose parent is a symlink fails that entry with exit 23, without creating its implied destination parent. --insecure-links only relaxes the ownership check on symlinks in typed local paths; it does not allow symlink traversal beneath a selected source, including entries from --files-from.

Other parsing differences:

• .. components are rejected rather than clamped at the source root.
• dir/ and dir select the same entry. Neither selects contents unless -r is explicitly supplied; -a alone does not enable recursion here.
• . and / are rejected as root entries.
• Use --from0 for NUL separators; -0 is unsupported.

Blank entries and comment-looking names starting with # or ; are ignored in both separator modes. Use ./#name for a literal name. The paths are relative to one source directory; destination parents are created as needed.

-h alone does not show help; use --help or --help-all.

Syq extensions

Syq-specific options carry a --syq- prefix. Common ones are --syq-connections, --syq-ignore, --syq-ignore-from, and --syq-verify-only. The last compares selected contents without writing; it does not produce rsync’s itemized-change format.

Filters are last-match-wins with ! re-inclusion, unlike rsync’s first-match rules. Check the gitignore examples when converting a filtered command.

The compatibility tests record comparison evidence and version-specific details.

Compare without copying

syq rsync -a --syq-verify-only project/ backup/

Hashes selected contents on both sides, writes nothing, and reports DIFFERS or MISSING. Differences or inspection failures produce a nonzero exit status.

Developing syq

Use a source build when changing syq itself. For everyday use, follow Install and setup.

Build and try a remote copy

Install Rust with rustup, Git, and a C compiler, then:

git clone https://github.com/greaber/syq.git
cd syq
cargo build --locked --release
./target/release/syq --build-identity
./target/release/syq cp data --to server --into /tmp/syq-dev-copy

Use test data and a disposable destination. Run ./target/release/syq explicitly so you do not accidentally test a release installed on your PATH. --release selects Cargo’s optimized build; it is still a development version. Source builds do not check for release updates or support --self-update.

For ordinary SSH copies, syq uploads its running executable automatically when the remote host needs it. You do not need to commit or push local edits first. The helper is cached for syq’s own use; it does not install a syq command on the remote PATH. Rebuild after edits and rerun the copy to use the new build.

The remote OS, CPU, and required system libraries must be compatible with your executable. A Linux build can still fail on another Linux host with older libraries. Syq reports the failure instead of substituting a released helper.

Another platform

For an ordinary SSH copy from, for example, macOS to Linux, build syq for the remote platform too. The simplest reproducible setup is a clean checkout of the same commit on both machines, built with cargo build --locked --release. If using uncommitted edits, reproduce those changes on both machines as well. Compare the identities, replacing the remote path with your actual checkout:

./target/release/syq --build-identity
ssh server /home/me/syq/target/release/syq --build-identity
./target/release/syq cp data --to server --into /tmp/syq-dev-copy \
  --syq-path /home/me/syq/target/release/syq

Both identities must match exactly; matching --version alone is insufficient. Rebuild both sides after changes. Use --no-bootstrap instead if the matching remote executable is already on the remote SSH session’s PATH. In rsync mode, the corresponding flags are --rsync-path and --syq-no-bootstrap.

Direct server-to-server copies

These use a restricted receiver on the destination, separate from the ordinary SSH helper cache. Follow the SSH-agent, host-key, and connectivity prerequisites in Copy between servers.

The first real copy can enroll the destination automatically, including with a development build. A dry run cannot create an enrollment. To make setup explicit and to refresh an existing receiver after rebuilding, run:

cargo build --locked --release
./target/release/syq receiver enroll hostB:/tmp/syq-dev-copy
./target/release/syq cp --dry-run -v --from hostA --srcs-in data \
  --to hostB --into /tmp/syq-dev-copy
./target/release/syq cp --from hostA --srcs-in data \
  --to hostB --into /tmp/syq-dev-copy

Repeat enrollment for the same host and root to install your current executable; rebuilding alone does not refresh an existing receiver. Enrollment preserves its receipt key. Use receiver list to find enrollment IDs and receiver revoke ID to remove access when finished. If setup needs a jump host, add --via hostA to receiver enroll or receiver revoke.

For a source build, enrollment uploads the running executable after checking that hostB has a matching platform. The executable must also run there. Official releases also upload the running executable when the platforms match; for a different platform, they install the verified release executable for hostB. --syq-path does not select the restricted receiver. To develop across incompatible platforms, run the coordinating command from a compatible machine or explicitly choose --coordinate-at local to relay through your machine using ordinary SSH helpers. For that relay, the manual helper selection above is available.

Before a pull request

For Rust changes, run:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --bin syq

Also run integration tests that exercise your change. SSH, remote-helper, enrollment, receiver, transport, and remote-coordinator changes need scripts/test-real-ssh.sh; see the real-SSH test setup. For documentation changes, run python3 scripts/check-doc-links.py. See the repository’s AGENTS.md for the full contribution workflow.

Machine-facing completion commands

The generated shell adapters invoke syq completion __complete SHELL INDEX -- WORDS... or the Bash-specific syq completion __complete-bash REPLACEMENT -- LINE. INDEX is the zero-based cursor-word index; WORDS are dequoted command words including syq. REPLACEMENT is Readline’s current fragment and LINE is the command line through the cursor. These entry points serve the generated adapters and are omitted from user help. For interactive use, generate an adapter with syq completion bash, zsh, or fish; use syq completion cache to inspect or clear endpoint suggestions.

Python

Copy, remove, and reorganize files from Python, with typed results and streaming events. Synchronous and asyncio clients are available.

Install

The syq package on PyPI supports Python 3.10+ on Linux and macOS:

python -m pip install syq

On first use, it downloads and verifies the matching syq executable, then caches it for later calls. You do not need to install the command-line tool separately.

Guide and examples

Use the syq package to call syq from Python. See installation if you have not installed it yet.

Copy files

Copy a directory named data into backup, producing backup/data:

import syq

result = syq.cp("data", into="backup")
print(result.files_transferred, result.bytes_transferred)

To copy just its contents, use srcs_in:

result = syq.cp(srcs_in="data", into="backup")

Arguments follow the command-line names: replace hyphens with underscores, and add a trailing underscore for Python keywords, such as from_ and as_. The copy guide explains placement, filtering, and verification options.

Copy over SSH

Use to for an SSH destination and from_ for an SSH source:

syq.cp("data", to="user@server", into="/backup")
syq.cp("report.csv", from_="user@server", cwd="/exports", into="downloads")

SSH hosts and paths are separate arguments. See Copy between servers for copies with two remote endpoints. A typed remote-to-remote preview needs coordinate_at="local"; a live copy can also use an enrolled receiver.

Preview changes

Pass dry_run=True to preview a copy or removal. The result has the same type as a live call, with counters describing the planned changes:

preview = syq.cp("data", into="backup", dry_run=True)
print(preview.files_transferred, preview.bytes_transferred)

Remove dry_run=True to apply the changes. syq checks the filesystem again when that call runs.

Mirror a directory

Use prune=True to also remove destination entries absent from the source. Here, staging must already exist, and max_delete limits deletions:

result = syq.cp(
    srcs_in="build",
    into_existing="staging",
    prune=True,
    max_delete=100,
)

Remove files

result = syq.rm(src_dir="old-output", root="/srv/jobs")
print(result.entries_removed, result.selectors_missing)

root confines removal to that directory. Add on="server" to remove files over ordinary SSH. Command-restricted receivers do not support rm. See Remove files for selector behavior.

Handle failures

A failed copy or removal raises SyqOperationError with its typed result:

try:
    result = syq.cp("data", into="backup")
except syq.SyqOperationError as error:
    print(error.result.status, error.result.errors)
    print(error.stderr.decode(errors="replace"))

Use check=False to receive unsuccessful results without that exception. Invalid arguments, installation failures, and incomplete or invalid results still raise exceptions. Catch syq.SyqError to handle any SDK-defined exception, or catch a specific subclass as above. Completed filesystem changes are not rolled back.

Watch events and save results

on_event receives records as the operation runs. For example, show each copied entry or planned change:

def observe(event: syq.AutomationEvent) -> None:
    if isinstance(event, (syq.TraceEvent, syq.OperationResult)):
        print(event.action, event.dst)

result = syq.cp("data", into="backup", on_event=observe)

Events are not collected in the returned result. To save the validated NDJSON records, pass an open binary stream:

with open("run.ndjson", "wb") as records:
    result = syq.cp("data", into="backup", results=records)

Rename while copying

Create a mapping, change its destination paths, then copy:

from dataclasses import replace

with syq.map(srcs_in="photos") as mapping:
    renamed = mapping.transform(
        lambda entry: replace(entry, dst=syq.RelativePath("archive") / entry.dst)
    )
    result = syq.cp(mapping=renamed, into="published")

This places the contents of photos under published/archive. The mapping carries its source base and symlink-following policy through the transform. Return None from the transform to skip an entry. The entire transform must finish successfully before copying starts; a failed transform leaves the destination untouched. See Rename and reorganize for mapping rules.

Use asyncio

Await operations on AsyncClient. Its arguments and results match Client:

import asyncio
import syq

async def main():
    client = syq.AsyncClient()
    result = await client.cp("data", into="backup")
    print(result.files_transferred)

asyncio.run(main())

Async event callbacks are awaited in record order. Mapping streams use async with and async for; there is no await before client.map():

async def copy_photos():
    client = syq.AsyncClient()
    async with client.map(srcs_in="photos") as mapping:
        return await client.cp(mapping=mapping, into="published")

Configure a client

Share a local working directory and timeout across calls:

client = syq.Client(process_cwd="/srv/jobs", timeout=3600)
result = client.cp("data", into="backup")

process_cwd sets the local subprocess directory; typed cwd sets the source base, which may be on a remote host. Omit timeout on a call to use the client default; pass timeout=None to disable it for that call:

result = client.cp("data", into="backup", timeout=None)

To use an existing executable, pass Client(executable="/opt/bin/syq"). This bypasses the managed version; see Compatibility.

Run other commands

run accepts arguments after the executable name and returns captured bytes:

result = syq.run(["--help"])
print(result.stdout.decode())

Use it for commands without a typed method, including rsync and receiver administration. See the API reference for process options and exceptions.

API reference

Module functions syq.cp, syq.rm, and syq.map use a default Client. AsyncClient has the same arguments and result types; await its operations except map, which returns an async context manager.

Client and executable selection

Client(*, executable=None, cache_dir=None, process_cwd=None, env=None, timeout=None) and AsyncClient(...) accept:

client.version() returns the executable version as text. syq.version(executable=None) does the same without a client. syq.managed_executable(cache_dir=None) returns the verified executable path, downloading it if needed. See Compatibility for version selection and caching.

Arguments and validation

Options use CLI names with hyphens replaced by underscores. Python keywords get a trailing underscore: from_, as_. Paths accept str, bytes, or os.PathLike; selector keywords accept one path or an iterable of paths. Use src=["a", "b"] for CLI --srcs a b, and likewise src_non_dir and src_dir.

Boolean flags default to False; other optional arguments default to None, except check=True and timeout, whose omitted value inherits the client default. Invalid argument combinations raise SyqInvocationError; filesystem and remote checks happen in syq.

cp

cp(*sources, **options) → CpResult copies files. Choose one placement option. In addition to the shared arguments above, it accepts:

Option behavior is covered in Copy files and Remote copy details.

pscope selects an isolated scope for reusing SSH connections. For return copies or commands, use syq persist connect server and omit pscope. See persistence in scripts for setup and cleanup, and Compatibility for executable selection.

Typed remote-to-remote copies require an enrolled receiver or coordinate_at="local". With dry_run=True or verify_only=True, they require coordinate_at="local". Use run for detached commands and human output options.

IgnoreFrom(path) is a frozen dataclass holding a rule-file path (str, bytes, or os.PathLike). To interleave rule files and inline patterns: ignore=[syq.IgnoreFrom("rules"), "!keep.tmp"]. The last matching rule wins.

With only_new=True, directories already present when syq first checks them keep their metadata. Missing children are still added. Adding children may naturally change directory timestamps. Directories copied as new receive normal copy metadata. When several sources supply the same new directory, the last source supplies its metadata, just as without only_new=True. Adding children to those existing directories requires write access; permissions are not temporarily widened. Permission failures are reported in the result and raise SyqOperationError unless check=False. A dry run does not test write permission.

rm

on="server" selects the removal endpoint. A final selected symlink is always unlinked. Both follow_src=True and follow=True permit symlinks in cwd, root, and selector parent directories. Directory and contents selectors reject a final symlink even with following enabled.

rm(*sources, **options) → RmResult removes selected entries. Besides the shared arguments, it accepts on, dry_run, connections, syq_path, no_bootstrap, pscope, on_event, results, and check with the types above. It supports local and ordinary SSH endpoints. Command-restricted receivers reject removal. See Remove files.

map

map(*sources, **options) → MapStream lists local mapping entries without copying. Besides the shared arguments, it accepts as_ to rename a selected object. srcs_in must be the sole selector when used.

MapStream is a Mapping and an iterable context manager; use with. AsyncMapStream is an AsyncMapping and an async context manager; use async with. Streams must be consumed inside their context and cannot be reused after completion or closure. Async streams capture the producer directory, environment, and timeout when created, even though execution starts later.

Mapping and AsyncMapping

A mapping combines entries with their local source context. Pass it directly to cp(mapping=...), even on a client with a different process_cwd.

Mapping(entries, *, cwd=None, root=None, follow_src=False) accepts an iterable of MappingEntry. AsyncMapping(...) accepts an async iterable. If both cwd and root are omitted, the source base is the current directory at construction. Otherwise exactly one may be supplied. Relative cwd and root paths resolve against the Python process directory at construction, not a client’s process_cwd. root confines source resolution.

The context properties are read-only. A transform receives each MappingEntry and returns a replacement entry, or None to omit it. Transformations can be chained. Async transforms also accept awaitable callbacks and await them in entry order. Neither form caches entries; reuse depends on the supplied iterable.

A context-carrying mapping rejects from_, cwd, and root overrides on cp. It automatically enables the source-following policy used by map; enabling follow or follow_src on the copy is also permitted. The destination options remain independent. map(root=..., srcs_in=...) carries the selected directory as the consuming copy’s root. The copy resolves that root again; it does not inherit an open directory handle or a snapshot of the source tree.

MappingEntry

Frozen dataclass describing one source-to-destination mapping. Pass an iterable of these to cp(mapping=...); use dataclasses.replace to change an entry.

MappingEntry(src, dst, kind=None, size=None, mtime=None) also accepts text or byte paths for src and dst and converts them to RelativePath. size and mtime do not impose preconditions on the copy.

RelativePath and PathValue

RelativePath(value) accepts text, bytes, or os.PathLike. It rejects empty paths, absolute paths, NUL bytes, and empty, . or .. components. Join paths with /, for example syq.RelativePath("archive") / entry.dst.

PathValue(raw: bytes) holds a path received in an event, which may be absolute. Both types are immutable and provide:

RelativePath also implements os.PathLike, returning bytes.

Normal end of stream iteration checks the mapping process status. Leaving its context early stops the process. An exhausted or closed stream cannot be copied as an empty mapping.

For cp(mapping=iterable), the entire iterable is saved to a temporary manifest before copying starts. An iteration, transformation, or serialization failure starts no copy. Plain iterables and manifest paths have no source context; pass cwd, root, or from_ explicitly as needed. See mapping rules.

Events and terminal results

cp and rm return frozen dataclasses after validating the complete results stream and process exit status. A truncated stream raises even if the process exits successfully. Dry runs return the same types, with planned totals.

CpResult

Returned by cp(), or available as SyqOperationError.result after an unsuccessful copy. Read attributes directly, for example result.files_transferred. It includes the common result fields below plus:

With dry_run=True, mutation totals describe planned changes. With verify_only=True, matching files count as unchanged; transfer and creation totals are zero. A failed call reports work completed before it stopped.

Ordinary copies have all three deletion fields only with prune=True; otherwise they are None. Receiver-attested results have only deletions_completed, and their unchanged/excluded totals are always zero because the receiver cannot observe source-side skips. receipt is None for ordinary copies.

RmResult

Returned by rm(), or available as SyqOperationError.result after an unsuccessful removal. It includes the common result fields below plus:

Selectors identify requests; entries count individual filesystem objects. One directory selector can account for many entries. Duplicate and overlapping selectors have separate indexes.

Common result fields

Both CpResult and RmResult include:

ProtocolMetadata

Frozen dataclass accessed through result.protocol or event.protocol. The SDK checks these fields; applications normally only need them when recording or diagnosing a stream.

ReceiptSummary

Frozen dataclass accessed through CpResult.receipt for receiver-attested copies.

OperationStatus

String enum: compare with syq.OperationStatus.SUCCESS, or use .value for "success".

Event callbacks

on_event(event) receives an AutomationEvent in stream order. Events are not stored in the result. AsyncClient accepts synchronous or awaitable callbacks; awaitable callbacks count toward the timeout.

AutomationEvent is the union of the event classes below and CpResult and RmResult. Each event is a frozen dataclass. Its fields are listed below; all events also carry protocol: ProtocolMetadata, just like results. event.protocol.type identifies the wire record type. Optional fields use None when unavailable.

See Automation results for stream semantics. Python exposes class as class_ and paths as PathValue.

results= accepts a binary file-like object with write(bytes) returning a positive byte count for nonempty writes, and optional flush(). The client copies validated NDJSON records, flushes after completion, and leaves it open. It withholds the terminal record if stream validation, process completion, or a callback fails. Sink failures raise and abort the operation.

OperationResult.is_retryable identifies retryable failures; retry_entry() returns a MappingEntry when a complete mapping identity is available, otherwise None. Only use collected entries after the call returns a validated success or partial result. A terminal callback alone does not establish completion. The client does not retry automatically.

Event types

RunEvent

Invocation details. started_at is Unix seconds; mode is "cp" or "rm". prune and mapping are None for removal; verify_only defaults to False.

protocol.type = "run". Fields in addition to the common envelope:

run_id: str
started_at: int
syq_version: str
mode: str
prune: bool | None
mapping: bool | None
dry_run: bool
endpoints: tuple[Endpoint, ...]
verify_only: bool

ProgressEvent

Sampled progress for displays; use the terminal result for final totals. Byte fields measure file content (comparison work with verify_only=True), scanned counts scanned entries, and elapsed_ms is milliseconds.

protocol.type = "progress". Fields in addition to the common envelope:

bytes_done: int
bytes_total: int
bytes_unchanged: int
files_done: int
files_total: int
files_unchanged: int
files_excluded: int
scanned: int
scan_done: bool
elapsed_ms: int

TraceEvent

One planned copy change. dst is relative to the destination container; src is the mapping source when available. bytes is the planned file size, and reason describes why the change is needed.

protocol.type = "trace". Fields in addition to the common envelope:

action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind
bytes: int | None
reason: TraceReason

OperationResult

One copy outcome. dst is destination-relative, or relative to the signed destination scope for a receiver receipt. src is present for mapping entries when available. bytes and attempts give transfer information. Error details are present when known; message is display text. provenance, scope, and code apply to receiver-attested outcomes.

protocol.type = "operation_result". Fields in addition to the common envelope:

action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind | None
disposition: Disposition
bytes: int | None
attempts: int | None
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None
provenance: str | None
scope: int | None
code: ReceiptCode | None

SelectionResult

One removal selector, indexed from zero. path is the original selector; status says whether it resolved. kind is None for a missing selector.

protocol.type = "selection_result". Fields in addition to the common envelope:

selector: int
path: PathValue
status: SelectionStatus
kind: EntryKind | None

RemovalTrace

One entry a preview would remove. selector identifies its source selector; path identifies the entry. disposition is always WOULD_REMOVE.

protocol.type = "removal_trace". Fields in addition to the common envelope:

selector: int
path: PathValue
kind: EntryKind
disposition: RemovalDisposition

RemovalResult

One removal outcome or preview inspection failure. selector identifies its source selector; path identifies the entry. attempts counts attempts; retryable, class_, os_kind, and message describe failures when available.

protocol.type = "removal_result". Fields in addition to the common envelope:

selector: int
path: PathValue
kind: EntryKind | None
disposition: RemovalDisposition
attempts: int
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None

ErrorEvent

One counted error. message is display text; class_ and os_kind classify it when known. Receiver errors can also carry provenance and code.

protocol.type = "error". Fields in addition to the common envelope:

message: str
class_: ErrorClass | None
os_kind: OsKind | None
provenance: str | None
code: ReceiptCode | None

FinalStateEvent

Receiver-attested destination state. dst is relative to the signed scope. provenance is "receiver_attested". Present objects have kind and byte size; metadata, digest, and symlink_target are present where available. observation_error describes a partial observation. Absent objects have no object details; failed observations have code and optional message.

protocol.type = "final_state". Fields in addition to the common envelope:

provenance: str
scope: int
dst: PathValue
state: FinalObjectState
kind: FinalObjectKind | None
size: int | None
metadata: ObjectMetadata | None
digest: AttestedDigest | None
symlink_target: PathValue | None
observation_error: str | None
code: ReceiptCode | None
message: str | None

Endpoint, ObjectMetadata, and AttestedDigest

Nested frozen dataclasses used by events:

Enums

All enums are string enums exported by syq. Members use uppercase names; values use lowercase, for example EntryKind.FILE.value == "file". OperationStatus is defined with the result types above.

ReceiptStatus.CLEAN means a clean receipt, FAILED reports receiver failures, and INCOMPLETE reports an incomplete lifecycle. Retryability.YES, NO, and UNKNOWN state whether a failed entry can be retried.

Failure model

SyqError is the base class for every SDK-defined exception below. Specific subclasses retain their details so callers can distinguish an unsuccessful operation from a broken results stream.

For cp and rm, check=False returns unsuccessful typed results; for run, it returns nonzero process results. It does not suppress other errors. Spawn failures, timeouts, and ordinary Python type/value errors use standard Python exceptions. Exceptions from application callbacks or mapping iterators are re-raised unchanged. Async cancellation remains asyncio.CancelledError. These exceptions are not wrapped in SyqError.

Timeout, cancellation, early mapping exit, and streaming failures stop the local process group, including SSH children. Filesystem changes already completed are not rolled back.

run

client.run(args, *, check=True, cwd=None, env=None, timeout=CLIENT_DEFAULT, input=None) returns Result. input accepts bytes. args is a sequence of arguments after the executable name, passed without a shell.

Here, cwd is the local subprocess directory. cwd and env use client defaults when omitted or None. timeout uses the client default only when omitted; explicit None disables it. Module-level syq.run takes the same arguments plus executable=None and defaults to no timeout.

Wrappers can forward syq.CLIENT_DEFAULT to preserve client inheritance. syq.Timeout is the type alias for a number, None, or that sentinel:

def copy_data(client: syq.Client, *, timeout: syq.Timeout = syq.CLIENT_DEFAULT):
    return client.cp("data", into="backup", timeout=timeout)

CLIENT_DEFAULT applies to client methods (including module-level cp, rm, and map); client constructors and module-level run have no client default to inherit.

Timeouts cover subprocess execution. Managed installation and mapping-input materialization happen before the copy process starts and are not covered by its timeout. Async cancellation still stops mapping-input preparation.

syq exec is also available through run; its command output and exit status are a process result:

result = syq.run(
    ["exec", "--on", "@mac", "--cwd", "work/project", "--", "cargo", "test"],
    executable="/path/to/syq",
)

For exec, pass --cwd in the argument list to select the receiving working directory. The SDK’s cwd= parameter selects the local working directory of the requesting syq process.

Result

Frozen dataclass returned by run() and held in SyqProcessError.result:

Compatibility

Python 3.10+ on Linux and macOS; no runtime Python dependencies. Each Python package uses the matching syq release. syq.__version__ and syq.PINNED_SYQ_VERSION report those versions. Pin the package in your dependency file to keep the pairing.

Managed executable

The default client downloads the matching executable on first use and verifies it against the package’s embedded release manifest. It checks the cached binary before every use and replaces missing or corrupt entries. It does not search PATH.

The default cache is $XDG_CACHE_HOME/syq/sdk/python/v<version>/ when XDG_CACHE_HOME is absolute, or ~/.cache/syq/sdk/python/v<version>/ otherwise. Use Client(cache_dir=...) to change the cache root, or syq.managed_executable() to download ahead of time and get the path.

Custom executable

Client(executable="/opt/bin/syq") uses that binary; Client(executable="syq") searches PATH. Overrides bypass managed download and verification, so you are responsible for compatibility and origin. Typed calls still validate automation output. A failed executable selection does not fall back to another binary.