syq cp

Copy files, directories, and symlinks locally, over SSH, or with S3. See Copy files for examples and remote and S3 copy details.

syq cp --srcs-in project --to server --into backup --dry-run -v

Put source selectors and --mapping before --to or a placement option. For scripting, see environment variables and results.

syq cp [OPTIONS] SOURCE... [PLACEMENT]
syq cp [OPTIONS] --src-fd FD --as PATH
syq cp [OPTIONS] SOURCE --as-fd FD

Sources and filtering

Destination and mapping

Updates and deletion

Metadata and symlinks

Verification

Connections and remote execution

S3 connection settings

Performance and resource limits

Preview, progress, and results

Help and version

Update policies

--into-existing requires the destination directory to exist but allows new files inside it. --only-new can add children to an existing directory, but does not change that directory’s permissions to make it writable.

--only-new cannot combine with --inplace: an interrupted write could leave a file that a retry skips. Restricted receivers also reject --as-new --inplace, because direct writes do not enforce that destination condition. S3 and named receiving destinations do not support --inplace; see Copy limits.

These policies do not disable requested pruning. Descriptor-specific restrictions are listed under file descriptors.

Pruning

--prune deletes only within the copied directories, after copying succeeds. Ignored paths, syq partial files, and recovery entries are kept. This also keeps their parent directories; extra hard links to copied files may be kept too. An interruption during deletion can leave some extras already removed.

Keep the source outside the destination. Syq detects overlap locally and for remote paths using the same host name, user, and port. Different SSH aliases or shared storage can hide overlap from this check. Wait for other copies into the destination to finish before pruning, so their new files are not deleted as extras.

Filename and type conflicts

If the source has both Report.txt and report.txt, a case-insensitive destination cannot store both. Syq does not check for that before copying, and one file can replace the other. The same problem applies to distinct Unicode spellings that the destination treats as one name. Rename the source entries or use a destination that can distinguish them. Unsupported names are reported as copy errors.

Syq refuses replacements between directories and non-directories, including empty directories. Move or remove the conflicting destination before retrying. Other replacements can fail if the filesystem lacks the operation needed to replace an entry safely; the old entry is kept.

Capacity checks

For missing or empty filesystem destinations, syq checks available bytes and capacity for new files when the filesystem reports them. A clear shortage fails before copying. Updates to populated destinations do not use this whole-copy estimate: existing data may be reused or replaced. Allocation errors still fail the affected copy.

Metadata details

Source setuid, setgid, and sticky bits are not copied without --preserve=permissions. Ownership uses numeric IDs. On macOS, an existing destination directory must be readable before syq can temporarily repair missing write or search permission.

Modification times are preserved for named file destinations. Output descriptors require explicit --preserve=times; see below.

File descriptors

--src-fd 0 reads stdin; --as-fd 1 writes stdout. Use them to connect local, SSH, or S3 copies to other programs; see Shell pipelines and file descriptors. Bash process substitution also works with ordinary source syntax:

syq cp --src <(gzip -c data) --to server --as data.gz

Paths such as /dev/fd/63 refer to descriptors in the local syq process.

Each stream copy takes one source. Stdin and process substitution have no filename, so use --as to choose one, or --as-fd to write to a descriptor. A named pipe can use --into: syq cp incoming.fifo --into saved waits for a writer, then saves its bytes as the regular file saved/incoming.fifo. A symlink to a pipe requires --follow-src. Selecting a pipe alongside other sources, including through a shell glob, is an error. Directory copies never read pipes; --preserve=specials copies the pipe itself.

Completion and failures

A named destination is replaced only after the transfer succeeds. The -new and -existing placement conditions apply as usual, before a named pipe is opened. S3 new-object writes also refuse replacement if an object appears during the upload.

If the producer fails halfway through, syq can still successfully save the bytes it received: EOF does not tell it whether the producer succeeded. Bash’s set -o pipefail detects failures in a pipeline but cannot undo a file already saved. Process substitution needs a separate check of the producer’s status. In Python, managed streams let you commit only after your producer succeeds.

An output descriptor may contain incomplete data after a failure. Check syq’s exit status before using the result. A consumer closing early makes syq fail. Forced termination can leave a .syq-stream-* temporary file beside the destination.

Options and file behavior

Progress, --stats, and -v go to stderr, leaving stdout for payload. Statistics report bytes, elapsed time, and average rate; pipe lengths are unknown until EOF. Use --resource-limits bandwidth=RATE to limit throughput.

--only-new skips a destination that exists. Existing directories, S3 key prefixes, and dangling symlinks also count as existing for --only-new. Skips succeed without reading input or opening a named FIFO. A shell producer can therefore receive SIGPIPE; in Python, check the writer’s skipped property before producing bytes. An output FD already exists, so --only-new --as-fd N always skips after validating the source.

Use --dry-run to check source and destination placement without reading input, opening a named pipe, or changing the destination. It cannot check a payload hash or predict the length of a pipe. --results and --results-fd report stream_result records separately from payload; results and payload/completion descriptors must differ.

Transport and limits

Filesystem streams use parallel data workers over SSH or encrypted TCP, with automatic worker tuning as in regular-file copies. They accept workers, request-size, pipeline-depth, and bw-pacing tuning; workers=N fixes the worker count, and --no-tcp keeps data on SSH. S3 transfers one object using multipart controls; see Descriptor copies.

Restart recovery, named receiving destinations, detached execution, directory selection, and content comparison are unsupported.

Descriptor offsets and metadata

Other inherited descriptors work too, except 2, which is reserved for diagnostics. Dedicate each descriptor to the copy. Syq advances its offset, respects append mode, and leaves its blocking mode alone; it does not truncate it. A literal - is a filename.

When a regular file is copied to a named destination, syq preserves its modification time. New named files use the source permissions limited by the destination umask; existing files keep their permissions. S3 uploads store file attributes in object metadata.

Output descriptors use the timestamps from normal writes, including when appending to an existing file. Add --preserve=times to copy the source modification time instead; this changes the whole destination file’s timestamp even for a partial write. --preserve=permissions,ownership copies those attributes without changing timestamps. S3 downloads interpret object metadata when attributes are requested; time preservation uses S3’s modification time if no syq attributes are stored.

Input pipes, sockets, and devices have no payload metadata, so they reject --preserve. Their new named destinations use 0666 limited by the umask and the time of the write; existing files keep their permissions. Output pipes likewise cannot preserve times, permissions, or ownership. Parent directories are created as needed. The source --cwd / --root options apply to pathname sources. --root with --src-fd is rejected because it cannot confine a descriptor that is already open. Symlink handling is unchanged; see Symlinks. Named remote sources must be regular files.