These docs describe unreleased changes on master. Read the latest stable documentation.

API reference

Module functions syq.cp, syq.rm, syq.map, syq.open_reader, and syq.open_writer use a default Client. AsyncClient has the same arguments and result types; await its operations except map, open_reader, and open_writer, which return async context managers.

Client and executable selection

Client(*, executable=None, cache_dir=None, process_cwd=None, env=None, timeout=None) and AsyncClient(...) accept:

client.version() returns the executable version as text. syq.version(executable=None) does the same without a client. syq.managed_executable(cache_dir=None) returns the verified executable path, downloading it if needed. See Compatibility for version selection and caching.

Arguments and validation

Options use CLI names with hyphens replaced by underscores. Python keywords get a trailing underscore: from_, as_. Paths accept str, bytes, or os.PathLike; selector keywords accept one path or an iterable of paths. Use src=["a", "b"] for CLI --srcs a b, and likewise src_non_dir and src_dir.

Boolean flags default to False; other optional arguments default to None, except check=True and timeout, whose omitted value inherits the client default. Invalid argument combinations raise SyqInvocationError; filesystem and remote checks happen in syq.

cp

cp(*sources, **options) → CpResult copies files. Choose one placement option. In addition to the shared arguments above, it accepts:

Option behavior is covered in Copy files, Remote copy details, and Object storage.

For example, client.cp("data", to="s3://bucket", into="backup", s3_header=["X-Tigris-Consistent: true"]) uploads local data using credentials from the subprocess environment or AWS configuration. Copies between two S3 endpoints use server-side copying within the same service; content verification options that require reading object bodies are rejected. SSH/S3 combinations are not supported. S3 results use EndpointKind.S3; older SDKs reject this new endpoint kind instead of interpreting it as SSH.

pscope selects an isolated scope for reusing SSH connections. For return copies or commands, use syq persist connect server and omit pscope. See persistence in scripts for setup and cleanup, and Compatibility for executable selection.

Typed SSH-to-SSH copies require an enrolled receiver or coordinate_at="local". With dry_run=True or verify_only=True, they require coordinate_at="local". Use run for detached commands and human output options.

IgnoreFrom(path) is a frozen dataclass holding a rule-file path (str, bytes, or os.PathLike). To interleave rule files and inline patterns: ignore=[syq.IgnoreFrom("rules"), "!keep.tmp"]. The last matching rule wins.

Byte streams

client.open_writer(*, as_=None, as_new=None, as_existing=None, to=None, follow_dst=False, ...) returns a StreamWriter. Choose exactly one of as_, as_new, or as_existing; the latter two require the destination to be absent or present, following cp placement semantics. Writers have no source basename, so they require an exact destination path. client.open_reader(src, *, from_=None, cwd=None, root=None, follow_src=False, ...) returns a StreamReader. cwd resolves relative sources; root also confines them. Choose at most one, as with cp. These bases belong to the source endpoint, independently of the client’s local process_cwd. Both accept rsh, syq_path, pscope, no_bootstrap, no_compress, s3_endpoint, s3_region, s3_profile, s3_header, performance_tuning, and timeout with the same meanings as cp. Only the stream-supported S3 part controls apply to performance_tuning. The client supplies the executable, process working directory, environment, and default timeout. Stream calls always check transfer failures.

These methods transfer raw bytes, using the CLI’s stream semantics for destination permissions, metadata, and endpoint restrictions. They do not emit cp result records. They are sequential, non-seekable interfaces; use one operation at a time on each stream. No whole-object retry or restart recovery is attempted. S3 can retry buffered multipart parts.

Use a with block. Successful writer exit sends a separate commit signal after closing the payload; EOF alone cannot publish a managed upload. An exception in the writer body before commit aborts without replacing the destination. close() only ends payload input, so a buffered or text wrapper can close it while unwinding an exception without publishing partial data. Successful context exit commits even if a wrapper already closed the payload. Calling abort() explicitly cancels that automatic commit.

Outside a context, finish with commit() or abort(); close() alone leaves the transfer pending. commit() can follow payload closure, is repeatable after success, and fails after an abort. Explicit commit publishes immediately, so later application errors cannot undo it. Closing an outer wrapper flushes its buffered data; do that before committing the underlying writer. A timeout or connection loss during commit can leave the outcome uncertain; the method reports failure rather than claiming rollback. Whole datasets need their own final publication step after all object transfers succeed.

A bounded reader call can yield partial data before a later transfer error. An unbounded read() checks transfer completion before returning its bytes. Successful reader exit establishes transfer completion, not that downstream application work was successful. An exception inside either context cancels the transfer and preserves the original exception. Writers that have not committed are aborted during garbage collection, including after payload closure; cleanup can block, so use explicit contexts for timely cleanup.

timeout covers the stream’s lifetime, including blocked reads/writes. Timeout raises subprocess.TimeoutExpired unless the transfer has completed successfully; transfer failures raise SyqProcessError, whose result contains the exit status and the last 8 KiB of diagnostics, without capturing payload bytes.

Async streams use async with directly and await I/O and explicit closure:

client = syq.AsyncClient()
async with client.open_writer(to="server", as_="generated.bin") as output:
    await output.write(b"generated bytes")
async with client.open_reader("generated.bin", from_="server") as source:
    while chunk := await source.read(65536):
        await consume(chunk)

Async streams start on context entry. Cancellation terminates the owned transfer process and releases blocked I/O. AsyncStreamReader and AsyncStreamWriter expose async read/write, close, and abort methods. AsyncStreamWriter.commit() explicitly publishes, with the same semantics as its synchronous counterpart; successful async context exit commits automatically.

rm

Positional sources and src refuse directories. src_dir removes a tree recursively; srcs_in removes its contents recursively and keeps the root.

on="server" selects the removal endpoint. A final selected symlink is always unlinked. Both follow_src=True and follow=True permit symlinks in cwd, root, and selector parent directories. Directory and contents selectors reject a final symlink even with following enabled.

rm(*sources, **options) → RmResult removes selected entries. Besides the shared arguments, it accepts on, dry_run, performance_tuning, syq_path, no_bootstrap, pscope, on_event, results, and check with the types above. It supports local, ordinary SSH, and S3 endpoints. Command-restricted receivers reject removal. See Remove files.

S3 removal uses rm(..., on="s3://bucket") with optional s3_endpoint, s3_region, s3_profile, and s3_header. s3_all_versions=True permanently removes all selected versions and delete markers; s3_version_id="ID" selects one version of one exact key. These options are mutually exclusive. RemovalTrace and RemovalResult expose optional s3_version_id and s3_delete_marker fields. The same arguments work with AsyncClient.rm.

map

map(*sources, **options) → MapStream lists local mapping entries without copying. Besides the shared arguments, it accepts as_ to rename a selected object. srcs_in must be the sole selector when used.

MapStream is a Mapping and an iterable context manager; use with. AsyncMapStream is an AsyncMapping and an async context manager; use async with. Streams must be consumed inside their context and cannot be reused after completion or closure. Async streams capture the producer directory, environment, and timeout when created, even though execution starts later.

Mapping and AsyncMapping

A mapping combines entries with their local source context. Pass it directly to cp(mapping=...), even on a client with a different process_cwd.

Mapping(entries, *, cwd=None, root=None, follow_src=False) accepts an iterable of MappingEntry. AsyncMapping(...) accepts an async iterable. If both cwd and root are omitted, the source base is the current directory at construction. Otherwise exactly one may be supplied. Relative cwd and root paths resolve against the Python process directory at construction, not a client’s process_cwd. root confines source resolution.

The context properties are read-only. A transform receives each MappingEntry and returns a replacement entry, or None to omit it. Transformations can be chained. Async transforms also accept awaitable callbacks and await them in entry order. Neither form caches entries; reuse depends on the supplied iterable.

A context-carrying mapping rejects from_, cwd, and root overrides on cp. It automatically enables the source-following policy used by map; enabling follow or follow_src on the copy is also permitted. The destination options remain independent. map(root=..., srcs_in=...) carries the selected directory as the consuming copy’s root. The copy resolves that root again; it does not inherit an open directory handle or a snapshot of the source tree.

Digest and HashAlgorithm

Digest(algorithm, value) describes the expected digest of all bytes in one regular file. algorithm accepts a HashAlgorithm value or its string: "blake3", "sha256", "md5", or "xxh3-128". value is hexadecimal: 64 digits for BLAKE3 and SHA-256, 32 for MD5 and XXH3-128. The immutable object validates the length and characters and stores lowercase hex.

client.cp(
    "data.bin", as_="verified.bin",
    expected_digest=syq.Digest("md5", "900150983cd24fb0d6963f7d28e17f72"),
)

The expectation covers the complete resulting file, including reused bytes. A mismatch fails the file rather than reporting a successful copy. Files excluded by selection rules are not digest-verified. hash=True still controls whether existing contents are compared instead of trusting size and modification time; integrity_checking="compare=HASH" selects and enables content comparison; hash=True is a shorthand for compare=blake3. Dry runs preview changes without validating the expectation. An expected whole-file digest is independent of the algorithm used for block comparison or transport checks. MD5 and XXH3-128 are useful for compatibility and accidental-error detection, but do not provide cryptographic collision resistance. Receiver receipt digests continue to use BLAKE3.

MappingEntry

Frozen dataclass describing one source-to-destination mapping. Pass an iterable of these to cp(mapping=...); use dataclasses.replace to change an entry.

MappingEntry(src, dst, kind=None, size=None, mtime=None, expected_digest=None) also accepts text or byte paths for src and dst and converts them to RelativePath. size and mtime do not impose preconditions on the copy. expected_digest does: a file cannot succeed unless its contents match. For example, an adapter can supply an MD5 from a DVC manifest without changing syq’s ordinary comparison algorithm:

entry = syq.MappingEntry(
    "cache/object", "data.bin", kind="file",
    expected_digest=syq.Digest("md5", "900150983cd24fb0d6963f7d28e17f72"),
)
client.cp(mapping=[entry], cwd="source", into="download")

Mapping files encode it as "expected_digest": {"algorithm": "md5", "value": "..."}. Older syq versions that do not support this field reject the mapping.

RelativePath and PathValue

RelativePath(value) accepts text, bytes, or os.PathLike. It rejects empty paths, absolute paths, NUL bytes, and empty, . or .. components. Join paths with /, for example syq.RelativePath("archive") / entry.dst.

PathValue(raw: bytes) holds a path received in an event, which may be absolute. Both types are immutable and provide:

RelativePath also implements os.PathLike, returning bytes.

Normal end of stream iteration checks the mapping process status. Leaving its context early stops the process. An exhausted or closed stream cannot be copied as an empty mapping.

For cp(mapping=iterable), the entire iterable is saved to a temporary manifest before copying starts. An iteration, transformation, or serialization failure starts no copy. Plain iterables and manifest paths have no source context; pass cwd, root, or from_ explicitly as needed. See mapping rules.

Events and terminal results

cp and rm return frozen dataclasses after validating the complete results stream and process exit status. A truncated stream raises even if the process exits successfully. Dry runs return the same types, with planned totals.

CpResult

Returned by cp(), or available as SyqOperationError.result after an unsuccessful copy. Read attributes directly, for example result.files_transferred. It includes the common result fields below plus:

With dry_run=True, mutation totals describe planned changes. With verify_only=True, matching files count as unchanged; transfer and creation totals are zero. A failed call reports work completed before it stopped.

Ordinary copies have all three deletion fields only with prune=True; otherwise they are None. Receiver-attested results have only deletions_completed, and their unchanged/excluded totals are always zero because the receiver cannot observe source-side skips. receipt is None for ordinary copies.

RmResult

Returned by rm(), or available as SyqOperationError.result after an unsuccessful removal. It includes the common result fields below plus:

Selectors identify requests; entries count individual filesystem objects. One directory selector can account for many entries. Duplicate and overlapping selectors have separate indexes.

Common result fields

Both CpResult and RmResult include:

ProtocolMetadata

Frozen dataclass accessed through result.protocol or event.protocol. The SDK checks these fields; applications normally only need them when recording or diagnosing a stream.

ReceiptSummary

Frozen dataclass accessed through CpResult.receipt for receiver-attested copies.

OperationStatus

String enum: compare with syq.OperationStatus.SUCCESS, or use .value for "success".

Event callbacks

on_event(event) receives an AutomationEvent in stream order. Events are not stored in the result. AsyncClient accepts synchronous or awaitable callbacks; awaitable callbacks count toward the timeout.

AutomationEvent is the union of the event classes below and CpResult and RmResult. Each event is a frozen dataclass. Its fields are listed below; all events also carry protocol: ProtocolMetadata, just like results. event.protocol.type identifies the wire record type. Optional fields use None when unavailable.

See Automation results for stream semantics. Python exposes class as class_ and paths as PathValue.

results= accepts a binary file-like object with write(bytes) returning a positive byte count for nonempty writes, and optional flush(). The client copies validated NDJSON records, flushes after completion, and leaves it open. It withholds the terminal record if stream validation, process completion, or a callback fails. Sink failures raise and abort the operation.

OperationResult.is_retryable identifies retryable failures; retry_entry() preserves expected_digest and returns a MappingEntry when a complete mapping identity is available, otherwise None. Only use collected entries after the call returns a validated success or partial result. A terminal callback alone does not establish completion. The client does not retry automatically.

Event types

RunEvent

Invocation details. started_at is Unix seconds; mode is "cp" or "rm". prune and mapping are None for removal; verify_only defaults to False.

protocol.type = "run". Fields in addition to the common envelope:

run_id: str
started_at: int
syq_version: str
mode: str
prune: bool | None
mapping: bool | None
dry_run: bool
endpoints: tuple[Endpoint, ...]
verify_only: bool

ProgressEvent

Sampled progress for displays; use the terminal result for final totals. Byte fields measure file content (comparison work with verify_only=True), scanned counts scanned entries, and elapsed_ms is milliseconds. Optional activity contains diagnostic measurements when the producer collects them; otherwise it is None.

protocol.type = "progress". Fields in addition to the common envelope:

bytes_done: int
bytes_total: int
bytes_unchanged: int
files_done: int
files_total: int
files_unchanged: int
files_excluded: int
scanned: int
scan_done: bool
elapsed_ms: int
activity: dict[str, Any] | None

TraceEvent

One planned copy change. dst is relative to the destination container; src is the mapping source when available. bytes is the planned file size, and reason describes why the change is needed.

protocol.type = "trace". Fields in addition to the common envelope:

action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind
bytes: int | None
reason: TraceReason

OperationResult

One copy outcome. dst is destination-relative, or relative to the signed destination scope for a receiver receipt. src is present for mapping entries when available. bytes and attempts give transfer information. Error details are present when known; message is display text. provenance, scope, and code apply to receiver-attested outcomes.

protocol.type = "operation_result". Fields in addition to the common envelope:

action: OperationAction
dst: PathValue
src: PathValue | None
kind: EntryKind | None
disposition: Disposition
bytes: int | None
attempts: int | None
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None
expected_digest: Digest | None
provenance: str | None
scope: int | None
code: ReceiptCode | None

SelectionResult

One removal selector, indexed from zero. path is the original selector; status says whether it resolved. kind is None for a missing selector.

protocol.type = "selection_result". Fields in addition to the common envelope:

selector: int
path: PathValue
status: SelectionStatus
kind: EntryKind | None

RemovalTrace

One entry a preview would remove. selector identifies its source selector; path identifies the entry. disposition is always WOULD_REMOVE.

protocol.type = "removal_trace". Fields in addition to the common envelope:

selector: int
path: PathValue
kind: EntryKind
disposition: RemovalDisposition
s3_version_id: str | None
s3_delete_marker: bool | None

RemovalResult

One removal outcome or preview inspection failure. selector identifies its source selector; path identifies the entry. attempts counts attempts; retryable, class_, os_kind, and message describe failures when available.

protocol.type = "removal_result". Fields in addition to the common envelope:

selector: int
path: PathValue
kind: EntryKind | None
disposition: RemovalDisposition
attempts: int
retryable: Retryability | None
class_: ErrorClass | None
os_kind: OsKind | None
message: str | None
s3_version_id: str | None
s3_delete_marker: bool | None

ErrorEvent

One counted error. message is display text; class_ and os_kind classify it when known. Receiver errors can also carry provenance and code.

protocol.type = "error". Fields in addition to the common envelope:

message: str
class_: ErrorClass | None
os_kind: OsKind | None
provenance: str | None
code: ReceiptCode | None

FinalStateEvent

Receiver-attested destination state. dst is relative to the signed scope. provenance is "receiver_attested". Present objects have kind and byte size; metadata, digest, and symlink_target are present where available. observation_error describes a partial observation. Absent objects have no object details; failed observations have code and optional message.

protocol.type = "final_state". Fields in addition to the common envelope:

provenance: str
scope: int
dst: PathValue
state: FinalObjectState
kind: FinalObjectKind | None
size: int | None
metadata: ObjectMetadata | None
digest: AttestedDigest | None
symlink_target: PathValue | None
observation_error: str | None
code: ReceiptCode | None
message: str | None

Endpoint, ObjectMetadata, and AttestedDigest

Nested frozen dataclasses used by events:

Enums

All enums are string enums exported by syq. Members use uppercase names; values use lowercase, for example EntryKind.FILE.value == "file". OperationStatus is defined with the result types above.

ReceiptStatus.CLEAN means a clean receipt, FAILED reports receiver failures, and INCOMPLETE reports an incomplete lifecycle. Retryability.YES, NO, and UNKNOWN state whether a failed entry can be retried.

Failure model

SyqError is the base class for every SDK-defined exception below. Specific subclasses retain their details so callers can distinguish an unsuccessful operation from a broken results stream.

For cp and rm, check=False returns unsuccessful typed results; for run, it returns nonzero process results. It does not suppress other errors. Spawn failures, timeouts, and ordinary Python type/value errors use standard Python exceptions. Exceptions from application callbacks or mapping iterators are re-raised unchanged. Async cancellation remains asyncio.CancelledError. These exceptions are not wrapped in SyqError.

Timeout, cancellation, early mapping exit, and streaming failures stop the local process group, including SSH children. Filesystem changes already completed are not rolled back.

run

client.run(args, *, check=True, cwd=None, env=None, timeout=CLIENT_DEFAULT, input=None) returns Result. input accepts bytes. args is a sequence of arguments after the executable name, passed without a shell.

Here, cwd is the local subprocess directory. cwd and env use client defaults when omitted or None. timeout uses the client default only when omitted; explicit None disables it. Module-level syq.run takes the same arguments plus executable=None and defaults to no timeout.

Wrappers can forward syq.CLIENT_DEFAULT to preserve client inheritance. syq.Timeout is the type alias for a number, None, or that sentinel:

def copy_data(client: syq.Client, *, timeout: syq.Timeout = syq.CLIENT_DEFAULT):
    return client.cp("data", into="backup", timeout=timeout)

CLIENT_DEFAULT applies to client methods (including module-level cp, rm, and map); client constructors and module-level run have no client default to inherit.

Timeouts cover subprocess execution. Managed installation and mapping-input materialization happen before the copy process starts and are not covered by its timeout. Async cancellation still stops mapping-input preparation.

syq exec is also available through run; its command output and exit status are a process result:

result = syq.run(
    ["exec", "--on", "@mac", "--cwd", "work/project", "--", "cargo", "test"],
    executable="/path/to/syq",
)

For exec, pass --cwd in the argument list to select the receiving working directory. The SDK’s cwd= parameter selects the local working directory of the requesting syq process.

Result

Frozen dataclass returned by run() and held in SyqProcessError.result:

Compatibility

Python 3.13.4+ on Linux and macOS; no runtime Python dependencies. Each Python package uses the matching syq release. syq.__version__ and syq.PINNED_SYQ_VERSION report those versions. Pin the package in your dependency file to keep the pairing.

Bundled executable

By default, the SDK runs the executable installed with its Python wheel. It locates that executable through the package’s installation record, without searching PATH, downloading files, or creating an executable cache. Each Python environment has its own installation. Removing the package also removes its executable.

Managed executable

For callers using a separate cache, Client(cache_dir=...) downloads the matching executable on first use and verifies it against the package’s embedded release manifest. It checks the cached binary before every use and replaces missing or corrupt entries. It does not search PATH. The existing syq.managed_executable() API also keeps this behavior.

The default cache is $XDG_CACHE_HOME/syq/sdk/python/v<version>/ when XDG_CACHE_HOME is absolute, or ~/.cache/syq/sdk/python/v<version>/ otherwise. Use Client(cache_dir=...) to change the cache root, or syq.managed_executable() to download ahead of time and get the path.

Custom executable

Client(executable="/opt/bin/syq") uses that binary; Client(executable="syq") searches PATH. Overrides bypass bundled selection and managed download and verification, so you are responsible for compatibility and origin. Typed calls still validate automation output. A failed executable selection does not fall back to another binary.