scp is ... outdated?

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Have a few more examples!

Tony0945 · Posted: Sat Apr 20, 2019 6:58 pm Post subject:

Tony0945 · Posted: Sat Apr 20, 2019 7:10 pm Post subject:

tld · Veteran Joined: 09 Dec 2003 Posts: 1845

tld · Veteran Joined: 09 Dec 2003 Posts: 1845

To clarify a little more, without using -r rsync will never send or receive anything but the files expressly specified. In addition, just like scp, for it so send anything to anywhere other than the users home you'd have to specify a full path, like user@host:/some/path.

This is why I've been stressing that, especially without -r and --delete, rsync is almost identical to scp. One thing I actually prefer is the syntax for specifying multiple remote files. With scp the syntax would be, for example:

Tony0945 · Posted: Sat Apr 20, 2019 8:40 pm Post subject:

Hu · Administrator Joined: 06 Mar 2007 Posts: 22658

When Portage calls rsync, the passed arguments include --delete to clear out old ebuilds.

If you are concerned what rsync will do, use -ni (dry-run, itemize) to prevent changes and get a detailed listing of what would have been done.

Anon-E-moose · Posted: Sat Apr 20, 2019 10:22 pm Post subject:

From my nightly rsync runs...
Options on root are "-aPxS --delete" because I want to keep it as much a mirror as possible.
Options on two other partitions are "-aPxO --delete" with some --excludes, because I generally want mirror copies of what's there but don't care about dir times.
Options on distfiles and packages are "-aPx" because I want to keep the packages, even if removed from source dirs.

Other than an occasional "-n" if I've changed something and want to make sure it does what I think I've told it to do I don't use the other flags.

Having said that, if I need to grab a file on the desktop when I'm on the laptop, I still use scp, but that's on a home network, not public.

Tony: as Hu said use the n flag for a dry-run but if the modification times are different it will try and copy those files again, even if they are the same size.
If I were going to change to rsync, and the mod times weren't in sync, I would write a script to make the backup times match the source times.
I used to do my backups with "cp -a" to keep the times the same, so it wasn't a real problem to convert to rsync.

Edit to add: the "x" option I use so I don't cross mount points.
_________________
UM780, 6.1 zen kernel, gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

Tony0945 · Posted: Sat Apr 20, 2019 10:41 pm Post subject:

tld · Veteran Joined: 09 Dec 2003 Posts: 1845

Interesting. That example is a lot more complex than anything I've needed to use. I backup a development directory under my home directory to a few places, but in that case everything owned by my user and there are no links or anything special. The options I use are just:

Akkara · Posted: Sun Apr 21, 2019 5:52 am Post subject:

If you're just starting to use rsync and the file times hadn't been preserved previously, you can use the --checksum (-c is the short synonym) option to force a comparison based on checksums of the actual file contents. This will read thru the entire source (and destination, if you're copying recursively) computing checksums, so if you have a lot of data be prepared for a wait. Or do it piecemeal on smaller subdirectories at a time. Or try the --size-only option to get started, but beware that that one will miss changes that result in the same-sized file.

I'll sometimes use the -c option even when I know little has changed, as a way of periodically checking my backups for bitrot (often in conjunction with the -n dry-run option for that purpose).

Unfortunately it isn't particularly intelligent in how it schedules the operations and it will read from the source device a long time checksumming it before turning attention to the destination and starting there, when both could have been proceeding in parallel and cut the time needed nearly in half.

Also note that (if I recall correctly), rsync uses md5 checksums. Collision attacks on md5 are known to exist, and thus rsync may be unreliable if operating in an adversarial environment. For normal everyday use without an active agent trying to mess with you, md5 is fine, the odds of collision are miniscule.
_________________
Many think that Dilbert is a comic. Unfortunately it is a documentary.

krinn · Watchman Joined: 02 May 2003 Posts: 7470

Like i said, i would myself not sync directories with scp but with rsync
But i would also just never use rsync to do that "scp /etc/hosts faramir:/etc" ; it scare me too much doing that with rsync
With scp it's an easy task, with rsync, i only use it for specific directories i like keep in sync, it's ability to delete files is so dangerous, and because no way i'll use an rsync server over /etc

The right tool for the right task

blubbi · Posted: Sun Apr 21, 2019 10:54 am Post subject:

One big + for rsync is the ability to resume a transfer - but since I prefer scp because of its simplicity:

Tony0945 · Posted: Sun Apr 21, 2019 2:36 pm Post subject:

The files I back up are mostly video and databases so checksum might be worthwhile but will take a long time checking terabytes of data moving between HDD's over USB.

When I moved root from a 10,000 RPM HDD to an SSD, I just booted sysrescuecd, partition the SSD, mounted both drives (/mnt/new and /mnt/old) then

mike155 · Posted: Sun Apr 21, 2019 4:27 pm Post subject:

I am surprised that rsync is so much faster than scp:

krinn · Watchman Joined: 02 May 2003 Posts: 7470

tld · Veteran Joined: 09 Dec 2003 Posts: 1845

tld · Veteran Joined: 09 Dec 2003 Posts: 1845

Anon-E-moose · Posted: Sun Apr 21, 2019 6:35 pm Post subject:

According to this from techmint

bobbymalone · n00b Joined: 21 Apr 2019 Posts: 1

Hiya,

They are referring specifically to CVE-2019-6111 and the less than ideal mitigation due to inherited design flaw from rcp.
https://nvd.nist.gov/vuln/detail/CVE-2019-6111
https://github.com/openssh/openssh-portable/commit/391ffc4b9d31fa1f4ad566499fef9176ff8a07dc

miket · Posted: Mon Apr 22, 2019 3:09 am Post subject:

I'm a big fan of using tar streams when copying multiple files between hosts. Unlike scp, this technique deals happliy with multiple files and/or directory trees. It really well for me in moving files to and from my web host, which allows ssh but not the Secure Copy Protocol. If both computers are local, I use the technique without using encryption (netcat instead of ssh).

Example of transport over ssh:

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

It's interesting that most of the discussion here is around:

sshfs (mount an ssh share, then perform your copy, then unmount -- 3x the things to type) but STILL ssh protocol
rsync (but it seems to use ssh by default, so STILL ssh protocol)

My most common use case is to transfer a single file or a recursive directory to a remote host, which is very often across the Internet. Very often that data contains customer information, which means I am bound by contract to keep the data private. So any non-encrypted transfer option is out of the question.

My earlier observation was that I find scp command structure very convenient. I do, although rsync can use the same exact structure and do the job. I also said that if this is just a client-side security hole, then the client should be able to be rewritten from scratch and remove all the badness.

The thing is, every solution I see here as a viable option uses ssh protocol for the transfer. So either all those solutions are flawed with the same vulnerability, or the scp client could be rewritten from scratch and get every bit as much security as any of these other solutions have.

Based on my understanding of the issue at https://seclists.org/fulldisclosure/2019/Jan/43, this is just a buggy client based on an antique implementation. Nothing wrong with the ssh protocol or server. I don't see why they can't fix it or replace it.