Site Map - skip to main content - dyslexic font - mobile - text - print

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.


hpr2293 :: More supplementary Bash tips

Finishing off the subject of expansion in Bash (part 2)

<< First, < Previous, Latest >>

Host Image
Hosted by Dave Morriss on 2017-05-17 is flagged as Explicit and is released under a CC-BY-SA license.
Listen in ogg, spx, or mp3 format. | Comments (8)

Part of the series: Bash Scripting

This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.

More supplementary Bash tips

Pathname expansion; part 2 of 2

Expansion

As we saw in the last episode 2278 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion (seen in episode 1951)
  • Process substitution (seen in episode 2045)
  • Word splitting (seen in episode 2045)
  • Pathname expansion (the previous episode 2278 and this one)

This is the last topic in the (sub-) series about expansion in Bash.

In this episode we will look at extended pattern matching as also defined in the “Manual Page Extracts” section at the end of the long notes.

Long Show Notes

I have written out a moderately long set of notes about this subject and these are available here.


Comments

Subscribe to the comments RSS feed.

Comment #1 posted on 2017-05-21T16:42:38Z by Jonathan Kulp

What about with SCP?

Hi Dave,

This was a really excellent show! Just the kind of stuff that I wish I had known for about the last 10 years. I tried something after listening to this that worked wonderfully as long I was long as I was just doing a list command, but when I tried the same thing using secure copy to get the same list of arguments, it didn't work.

What I wanted to do was to push all MP3 and OGG files in a given directory over to my server in a single command, excluding the HTML and markdown files in the same directory.

The following command worked perfectly to **list** all of the MP3s and OGGs: "ls *(*.mp3|*.ogg)", but when I tried the same arguments with SCP it failed. Have you tried doing these kinds of expansions with secure copy?

Comment #3 posted on 2017-05-22T08:44:32Z by Dave Morriss

SCP without extended globs

It didn't occur to me at the time to try this (on rpi5, pulling files off a remote machine):

$ scp dave@desktop:'scptest/a??w.{txt,dat}' .
dave@desktop's password: ...

This works. The quotes prevent there being two scp invocations with associated password prompts.

Comment #4 posted on 2017-05-22T11:12:48Z by Jonathan Kulp

Details, details...

Aha! As always the devil is in the details. This last comment you left gave me the hint I needed. I was putting the asterisk in the wrong place and also using parentheses instead of curly braces. The following command works just like I want:

scp *.{mp3,ogg}
jonserver:~/destination/dir/

Thanks, Dave!

Comment #5 posted on 2017-05-22T11:29:36Z by Dave Morriss

TMTOWTDI

"There's more than one way to do it" - Larry Wall (actually he was talking about Perl, but it works here)

I'm glad I helped you to get where you wanted to be (even though I realise I wasn't quite answering your original question).

Having researched this and thought about it a bit I started putting together a brief(ish) HPR show about it.

Comment #6 posted on 2017-05-22T12:26:21Z by Jonathan Kulp

Ken is smiling

Dave, somehow I suspected when I asked whether this worked with secure copy that it would end up becoming another show from you. You're the best!

Comment #7 posted on 2017-05-24T04:52:46Z by clacke

scp brace expansion??!

Dave, I had no idea scp would do brace expansion on the server side. That's quite unexpected, and quite the discovery!

I tried it with rsync, and rsync *also* supports it. That made me suspicious.

As I feared, it seems to mean it runs the server side of rsync (and of scp) through the shell. Testing confirms it. If I want to (explicitly) copy files with spaces in the names, the quoting nightmare starts. :-(

So what started out as a happy discovery just turned into another disappointment in how broken our software is.

All this time, I had assumed that rsync started a server with no specific arguments, and then communicated over the rsync protocol which files to get. I guess I've been lucky all these years and avoided explicitly naming weird file names.

Comment #8 posted on 2017-05-24T10:09:52Z by Dave Morriss

scp is a bit of a hack!!

Hi clacke.

I had been messing with scp using the -v option which generates a lot of information. You can see it connecting via ssh then, if the original command was:

scp -v dave@rpi4:'scptest/*.{mp3,ogg}' .

it sends across:

scp -v -f scptest/*.{mp3,ogg}

where the quotes prevent local expansion and the (undocumented) -f option in the command marks it as running on the remote end.

I don't know how this mechanism deals with names with spaces and so forth, but I imagine it's nasty. It needs some experiments. My ideas for a "brief" show about this subject look doomed to be "l o n g" :-)

I think rsync will talk to a remote server (never used it) but I'd expect it would need to exist before the transfer.

Leave Comment

Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.

Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to
record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).

Provide feedback
Your Name/Handle:
Title:
Comment:
Anti Spam Question: What does the P in HPR stand for ?
Are you a spammer →
Who hosted this show →
What does HPR mean to you ?