Force Sequential Output with SCP?

November 9th, 2024
tech
In my bioinformatics work I often stream files between linux hosts and Amazon S3. This could look like:

$ scp host:/path/to/file /dev/stdout | \
    aws s3 cp - s3://bucket/path/to/file

This recently stopped working after upgrading:

ftruncate "/dev/stdout": Invalid argument
Couldn't write to "/dev/stdout": Illegal seek

I think I figured out why this is happening:

With scp I can give the -O flag:

Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a '~' prefix for older SFTP servers.

This does work, but it doesn't seem ideal: probably servers will drop support for the SCP protocol at some point? I've filed a bug with OpenSSH.


[1] "man scp" gives me: "Since OpenSSH 8.8 (8.7 in Red Hat/Fedora builds), scp has used the SFTP protocol for transfers by default."

Comment via: facebook, lesswrong, mastodon

Recent posts on blogs I like:

Facts I Learned From A City On Mars

Space bastardry!

via Thing of Things April 14, 2025

Which Came First, the Chicken or the Egg?

When I thought about this question it was really hard to figure out because the way it's phrased it's essentially either a chicken just pops into existence, or an egg just pops into existence, without any parent animals involved. I thought about t…

via Lily Wise's Blog Posts April 13, 2025

Advice for time management as a manager

have accurate expectations of yourself • prioritize ruthlessly • unemploy your future self • a five-step “help, I’m overwhelmed” checklist • carve out focused time

via benkuhn.net April 1, 2025

more     (via openring)