Automation results

Native cp, rm, and clean-partials can write structured outcomes alongside human output:

syq cp --srcs-in project --into backup --results copy.ndjson
syq rm old-output --results removal.ndjson

Each line is a JSON record. Require a final result record before trusting completion. EOF without one means the outcome is unknown, even if no error record appeared.

This page describes stream semantics. The JSON Schema lists precise field shapes; example streams show complete runs. For mapping manifests, see Mapping format.

The channel

--results FILE creates a fresh regular file and refuses an existing one. Use a new name for each run and keep it outside trees being copied or removed. --results-fd N uses an already opened writable descriptor above 2:

syq cp project --into backup --results-fd 3 3>copy.ndjson

Use a descriptor for pipes or other non-regular sinks. Named results paths follow the normal native symlink rules; --follow permits traversal. Human stdout and stderr are not part of this contract.

Results are written on the invoking machine, including for remote removal. For direct restricted remote-to-remote copies, they are derived from the verified destination receipt and marked provenance: "receiver_attested". Those records arrive after receipt verification. Otherwise remote-to-remote results require --coordinate-at local, which relays data through your machine. Remote-to-remote --dry-run --results also requires that local route.

--results supports native cp (including pruning), rm, and clean-partials. Cleanup uses the removal records described below. Results cannot combine with --detach. The restricted receiver supports copy changes, including capped pruning, but not native removal.

Records arrive in seq order; error and terminal records are flushed immediately. If writing the stream fails, syq warns where possible and stops writing it. Filesystem work can continue, leaving an incomplete stream.

Argument errors exit 2 without a stream. Failure to open the results sink exits 1 without a stream. Once opened, completed runs and fatal setup failures emit a terminal record if the sink remains writable. Crashes and interruptions can leave it missing.

Consumer rules

• Check the terminal record and process exit code; a mismatch is a protocol error. Use terminal totals, not progress or a count of operation records.
• Retry individual mapping entries only after terminal success or partial; see Retry failed mapping entries. Other statuses can leave entries unresolved.
• Ignore unknown record types and optional fields within a supported schema version.
• Reject unknown schema, schema_version, terminal status, or path encoding values.
• Treat human message text as display only; parse structured fields.

--results includes both progress samples and final outcomes; use it for new integrations, including live progress displays.

Exit codes

Removal terminals use only success, partial, or failed. A results-sink startup error also exits 1 without a stream.

Record envelope

Every record carries:

Paths are tagged: {"encoding":"utf-8","value":"docs/a.txt"}, or encoding: "base64" with standard base64 of raw filename bytes. Byte and count fields are non-negative integers within the u64 range.

Record types

run

Always first. Identifies the invocation with run_id, started_at (Unix seconds), syq_version, mode (cp or rm), dry_run, and endpoints. Endpoints identify role and kind (local, ssh, or s3). SSH endpoints carry host/user; S3 endpoints carry host: "s3://BUCKET" without a user. They omit credentials, headers, ports, and raw command arguments. Clients that predate S3 must be upgraded to decode S3 endpoint kinds; local and SSH streams retain their existing representation.

Copy runs also carry prune and mapping. Removal has one source endpoint regardless of selector count and omits those copy fields.

For S3 downloads using --only-new, unchanged-file totals include skipped symlinks selected through a prefix, but exclude symlinks named directly or through a mapping.

progress

Sampled telemetry, approximately once per second, for displays rather than accounting. It includes bytes, files, exclusions, scan state, elapsed time, and optional rate_bytes_per_second and eta_ms estimates. Removal has zero byte and unchanged/excluded counts; its file counts reflect outcomes received so far. Copies that reach transfer completion emit a final progress sample, even for short copies. The terminal record owns final totals.

For performance measurements collected with --stats, see Performance measurements.

trace

One intended copy change in a dry run. Includes action, destination, source for mapping entries, kind and bytes where applicable, plus a reason: destination_missing, type_differs, content_differs, metadata_differs, or destination_only.

For filesystem copies of regular files, content_differs means sizes or compared hashes differ; metadata_differs also covers copies selected by the metadata quick check. A transfer_file trace omits bytes when only metadata would change.

A trace cannot be matched by identity to a later live operation: the filesystem may change between runs.

operation_result

An outcome for a completed copy change or a failed mapping entry.

In attested records, dst is relative to the destination area identified by scope. An attested set_metadata omits kind.

Unchanged and excluded entries have totals only. Ordinary live streams do not emit per-operation records for metadata-only updates, though dry runs emit metadata_differs traces. Failed implicit parent creation can lack src and is non-retryable. Do not construct a retry source from its destination name.

stream_ready

A producer may start supplying payload. Uploads without a skip policy emit this before destination setup finishes, allowing several writers to connect concurrently. With --only-new or metadata preservation requested, source selection and destination checks come first. Skipped copies and dry runs finish without this record. Setup, transfer, or publication can still fail; require the terminal result for completion.

stream_result

Descriptor copies emit one stream_result instead of pathname-based operation_result or trace records. source and destination each contain one of path (a tagged path) or fd (the caller’s descriptor number). Paths are source/destination operands, with a source basename appended for --into; the run’s endpoints identify where they belong. An FD is a handle for this invocation, not a reusable source or retry identity.

disposition is succeeded, failed, skipped, or planned; dry_run identifies a preview. bytes counts transferred bytes, including partial work on failure, or planned bytes in a dry run. It is absent for a preview of unknown-length input. A failed upload may have transferred bytes without publishing them. Skipped copies consume no payload, report zero bytes, and count as one excluded file. The normal terminal result still establishes completion. Its optional bytes_total_known field is false when the source length is unknown; preview byte totals then count only known bytes, rather than asserting an empty input.

Stream mappings use version 3 and emit one stream_result for each callback entry, with an additional zero-based entry index. Each endpoint is {"path": <tagged path>} or {"entry": N, "callback": true}. Failed entries include message. These records identify callbacks within this invocation; they cannot be converted to a pathname retry manifest. Ordinary entries in a mixed mapping still emit their usual records. The Python callback reference describes publication and application retries. Callback admission uses the SDK’s private channel, so these runs do not emit stream_ready.

Removal records

For S3 version removal, removal_trace and removal_result include optional s3_version_id (string) and s3_delete_marker (boolean) fields. Each record and entry count refers to one version or delete marker. Ordinary S3 removal omits these fields.

selection_result uses a zero-based selector index, the original path, status (resolved or missing), and kind when resolved. Missing selectors succeed. Overlapping or duplicate selectors keep separate indexes. If a later selector cannot resolve, earlier selection records may precede the fatal error and terminal record.

removal_trace includes selector, path, kind, and disposition: "would_remove". Directories follow their descendants.

removal_result includes selector, path, kind when known, attempts, and disposition: removed, already_absent, or failed. Already absent is success. Failures include error details and retryability where available, and also produce a counted error record. Dry-run inspection failures can produce failed removal results, but never successful removal results.

error

One per counted error. message is display text; class and os_kind are provided where known. Classes are io, transport, conflict, integrity, safety_limit, usage, and internal.

OS kinds are not_found, permission_denied, already_exists, invalid_input, no_space, quota_exceeded, read_only, and other. They preserve OS error meaning across hosts without requiring matching errno numbers. Receiver refusals use class: "safety_limit" with provenance and the receiver’s code.

final_state

Attested streams only: the destination’s final observation of a path the transfer could have changed. Includes scope, dst, and an object: absent, an observation failure, or present with kind, size, applicable metadata, and symlink target. With --receiver-receipt hashes, regular files also have a BLAKE3 hash.

Object kinds distinguish directories, files, symlinks, FIFOs, sockets, character/block devices, and other objects. Metadata fields are mode, uid, gid, mtime, mtime_nsec, and rdev. Consult the schema for exact shapes. This observes destination state; it does not attest source completeness.

result

Exactly one terminal record, always last when the stream completes. Common fields are status, exit_code, dry_run, errors, and elapsed_ms.

Copy terminals may also include copying_elapsed_ms: the wall-clock span from first file work to last completed file work across workers, including per-file checks, finalization and gaps. Initial setup before file work is excluded; planning and connections can overlap this interval. It is not a sum of worker times or pure network time. The field is absent when no bytes moved or the coordinator does not supply it, including older releases and attested terminals. Use elapsed_ms for end-to-end throughput comparisons.

Copy totals include transferred/unchanged/excluded files, created directories, symlinks and specials, transferred/unchanged bytes, and on pruning runs deletions_planned, deletions_completed, and deletions_blocked. A fatal failure reports what finished before it stopped. Dry-run totals describe planned work, not committed changes.

Attested terminals add receipt_status (clean, failed, or incomplete), provenance, and receipt counts. They can attest only what the receiver saw:

• Unchanged and excluded totals are zero, not a claim that every source entry changed.
• Only deletions_completed appears; planned and blocked deletion totals are omitted.
• errors counts attested error records for failed/incomplete operations, refusals, and failed or partial final-state observations.

Removal terminals have mode: "rm" and totals for selectors, resolved and missing selectors, planned/removed/already-absent entries, and failed entries. Live removal leaves entries_planned zero. Dry removal leaves entries_removed and entries_already_absent zero, but inspection failures may increase entries_failed.

The human summary uses the same terminal totals.

Performance measurements

See Performance measurements for worker time, endpoint operations, CPU, and TCP counters collected with --stats.

Compatibility

Within a schema version, required fields keep their types and meanings; existing types, actions, dispositions, statuses, classes, and reasons are not renamed or reused. New record types and optional fields may be added. Human messages may change at any time.

Connection status

syq persist status --json reports the persistence setting, scope, and endpoints without starting connections. Endpoint states are starting, connecting, ready, reconnecting, failed, or inactive. Each entry also reports whether SSH is connected and the receiving state and errors.

With receiving enabled for an endpoint, ready means its allowed receiving profiles are online; SSH can reconnect on its next use. If receiving preferences cannot be read, SSH entries are still listed, receiving_error explains the failure, and each entry’s receiving_enabled is null.

persist receive status --json reports each profile’s cwd, cwd_explicit, root, auto_approve_root, and servers. An empty server list means all connections. A null automatic approval root means every download asks. Connection entries list only profiles allowed on that endpoint.

Command approvals in syq persist receive pending --json use kind: "command" and include argv, cwd, and permission. Argument and directory strings in this summary are escaped for display. Use an up-to-date syq binary to inspect and approve commands; clients that only support copy requests omit them.

Retry failed mapping entries

Add --results r.ndjson to record outcomes in a fresh file outside the copy trees.

Failed mapping entries contain src, dst, and kind, so they can form a retry manifest. Preserve expected_hash and metadata too when present. First require a terminal result with success or partial: a missing terminal or an early stop means some entries may have no results. In those cases, rerun the original copy instead.

set -o pipefail
syq cp --mapping big.ndjson -C src --to nas --into /data --results r.ndjson
jq -cs 'if any(.[]; .schema != "syq.automation" or .schema_version != 2)
        then "unsupported results schema for pathname retry" | halt_error
        elif (.[-1].type? // "") != "result"
        then "incomplete results stream (no terminal record)" | halt_error
        elif (.[-1].status != "success" and .[-1].status != "partial")
        then "run stopped early (status \(.[-1].status)); rerun it instead of retrying" | halt_error
        else .[] | select(.type == "operation_result"
                          and .disposition == "failed"
                          and .retryable != "no")
             | {src, dst, kind}
               + (if has("expected_hash") then {expected_hash} else {} end)
               + (if has("metadata") then {metadata} else {} end)
        end' r.ndjson \
  | syq cp --mapping - -C src --to nas --into /data

The filter skips non-retryable entries, including failed implicit parent creation without a source path. It does not guarantee that retrying will succeed; fix the underlying error first. Unchanged and excluded files appear only in summary totals, not as individual results.

For command-restricted copies between servers, --results contains verified receiver receipts; see Signed results. These describe destination changes rather than source entry failures; retry the original mapping instead of applying the operation_result filter above.