Force Sequential Output with SCP?

November 9th, 2024
tech
In my bioinformatics work I often stream files between linux hosts and Amazon S3. This could look like:

$ scp host:/path/to/file /dev/stdout | \
    aws s3 cp - s3://bucket/path/to/file

This recently stopped working after upgrading:

ftruncate "/dev/stdout": Invalid argument
Couldn't write to "/dev/stdout": Illegal seek

I think I figured out why this is happening:

With scp I can give the -O flag:

Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a '~' prefix for older SFTP servers.

This does work, but it doesn't seem ideal: probably servers will drop support for the SCP protocol at some point? I've filed a bug with OpenSSH.


[1] "man scp" gives me: "Since OpenSSH 8.8 (8.7 in Red Hat/Fedora builds), scp has used the SFTP protocol for transfers by default."

Comment via: facebook, lesswrong, mastodon, substack

Recent posts on blogs I like:

AI incompetence often comes from misalignment

Sometimes I see people say “I’m not worried about AI risk because AIs are really bad at things.” I think this is a misunderstanding.

via Thing of Things April 27, 2026

You should try contra dancing

a story of middle school Ben • a not-very-illuminating description of the mechanics • flow, joy, and community • the antidote to the rest of life • how to try contra

via benkuhn.net April 24, 2026

On AI writing in 2026

I use AI to write a little bit: I ask it for high level feedback on blog post drafts, make mechanical edits, and sometimes use it to brainstorm options for wording at a paragraph level. It’s unusual that I accept its wording or changes without modificatio…

via Home April 16, 2026

more     (via openring)