This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.
More supplementary Bash tips
Pathname expansion; part 2 of 2
Expansion
As we saw in the last episode 2278 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:
Brace expansion (we looked at this subject in episode 1884)
Comment #1 posted on 2017-05-21 16:42:38 by Jonathan Kulp
What about with SCP?
Hi Dave,
This was a really excellent show! Just the kind of stuff that I wish I had known for about the last 10 years. I tried something after listening to this that worked wonderfully as long I was long as I was just doing a list command, but when I tried the same thing using secure copy to get the same list of arguments, it didn't work.
What I wanted to do was to push all MP3 and OGG files in a given directory over to my server in a single command, excluding the HTML and markdown files in the same directory.
The following command worked perfectly to **list** all of the MP3s and OGGs: "ls *(*.mp3|*.ogg)", but when I tried the same arguments with SCP it failed. Have you tried doing these kinds of expansions with secure copy?
Comment #2 posted on 2017-05-21 18:08:45 by Dave Morriss
SCP is a bit weird
Hi Jon,
Glad you enjoyed the show.
To look into this I did the usual thing of creating a directory 'scptest' and making files in it:
$ touch scptest/{a,b}{00..10}{a..z}.{dat,txt}
I could copy selected files TO a remote directory:
This works. The quotes prevent there being two scp invocations with associated password prompts.
Comment #4 posted on 2017-05-22 11:12:48 by Jonathan Kulp
Details, details...
Aha! As always the devil is in the details. This last comment you left gave me the hint I needed. I was putting the asterisk in the wrong place and also using parentheses instead of curly braces. The following command works just like I want:
scp *.{mp3,ogg}
jonserver:~/destination/dir/
Thanks, Dave!
Comment #5 posted on 2017-05-22 11:29:36 by Dave Morriss
TMTOWTDI
"There's more than one way to do it" - Larry Wall (actually he was talking about Perl, but it works here)
I'm glad I helped you to get where you wanted to be (even though I realise I wasn't quite answering your original question).
Having researched this and thought about it a bit I started putting together a brief(ish) HPR show about it.
Comment #6 posted on 2017-05-22 12:26:21 by Jonathan Kulp
Ken is smiling
Dave, somehow I suspected when I asked whether this worked with secure copy that it would end up becoming another show from you. You're the best!
Comment #7 posted on 2017-05-24 04:52:46 by clacke
scp brace expansion??!
Dave, I had no idea scp would do brace expansion on the server side. That's quite unexpected, and quite the discovery!
I tried it with rsync, and rsync *also* supports it. That made me suspicious.
As I feared, it seems to mean it runs the server side of rsync (and of scp) through the shell. Testing confirms it. If I want to (explicitly) copy files with spaces in the names, the quoting nightmare starts. :-(
So what started out as a happy discovery just turned into another disappointment in how broken our software is.
All this time, I had assumed that rsync started a server with no specific arguments, and then communicated over the rsync protocol which files to get. I guess I've been lucky all these years and avoided explicitly naming weird file names.
Comment #8 posted on 2017-05-24 10:09:52 by Dave Morriss
scp is a bit of a hack!!
Hi clacke.
I had been messing with scp using the -v option which generates a lot of information. You can see it connecting via ssh then, if the original command was:
scp -v dave@rpi4:'scptest/*.{mp3,ogg}' .
it sends across:
scp -v -f scptest/*.{mp3,ogg}
where the quotes prevent local expansion and the (undocumented) -f option in the command marks it as running on the remote end.
I don't know how this mechanism deals with names with spaces and so forth, but I imagine it's nasty. It needs some experiments. My ideas for a "brief" show about this subject look doomed to be "l o n g" :-)
I think rsync will talk to a remote server (never used it) but I'd expect it would need to exist before the transfer.
Leave Comment
Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.
Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).