Site Map - skip to main content

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes every weekday Monday through Friday.
This page was generated by The HPR Robot at


hpr2293 :: More supplementary Bash tips

Finishing off the subject of expansion in Bash (part 2)

<< First, < Previous, , Latest >>

Thumbnail of Dave Morriss
Hosted by Dave Morriss on 2017-05-17 is flagged as Explicit and is released under a CC-BY-SA license.
Bash, expansion, filename expansion, extglob, extended pattern matching. 8.
The show is available on the Internet Archive at: https://archive.org/details/hpr2293

Listen in ogg, spx, or mp3 format. Play now:

Duration: 00:38:25

Bash Scripting.

This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.

More supplementary Bash tips

Pathname expansion; part 2 of 2

Expansion

As we saw in the last episode 2278 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion (seen in episode 1951)
  • Process substitution (seen in episode 2045)
  • Word splitting (seen in episode 2045)
  • Pathname expansion (the previous episode 2278 and this one)

This is the last topic in the (sub-) series about expansion in Bash.

In this episode we will look at extended pattern matching as also defined in the “Manual Page Extracts” section at the end of the long notes.

Long Show Notes

I have written out a moderately long set of notes about this subject and these are available here.


Comments

Subscribe to the comments RSS feed.

Comment #1 posted on 2017-05-21 16:42:38 by Jonathan Kulp

What about with SCP?

Hi Dave,

This was a really excellent show! Just the kind of stuff that I wish I had known for about the last 10 years. I tried something after listening to this that worked wonderfully as long I was long as I was just doing a list command, but when I tried the same thing using secure copy to get the same list of arguments, it didn't work.

What I wanted to do was to push all MP3 and OGG files in a given directory over to my server in a single command, excluding the HTML and markdown files in the same directory.

The following command worked perfectly to **list** all of the MP3s and OGGs: "ls *(*.mp3|*.ogg)", but when I tried the same arguments with SCP it failed. Have you tried doing these kinds of expansions with secure copy?

Comment #2 posted on 2017-05-21 18:08:45 by Dave Morriss

SCP is a bit weird

Hi Jon,

Glad you enjoyed the show.

To look into this I did the usual thing of creating a directory 'scptest' and making files in it:

$ touch scptest/{a,b}{00..10}{a..z}.{dat,txt}

I could copy selected files TO a remote directory:

$ scp scptest/*(???y.txt|???y.dat) dave@rpi5:test/

However, on rpi5 I couldn't copy FROM the other machine.

I did find a solution, but it's quite long for a comment, and I'm not 100% sure I understand it. I tried it and it did copy the files I specified.

Ken would suggest a show on the subject, but perhaps if I pointed you to the link I found it might do the job :-)

https://unix.stackexchange.com/questions/103058/exclude-characters-for-scp-filepattern

See what you think.

Actually, I shunt files around a lot between systems, but I often tend to use 'rsync'. However, that's a whole other subject.

Comment #3 posted on 2017-05-22 08:44:32 by Dave Morriss

SCP without extended globs

It didn't occur to me at the time to try this (on rpi5, pulling files off a remote machine):

$ scp dave@desktop:'scptest/a??w.{txt,dat}' . dave@desktop's password: ...

This works. The quotes prevent there being two scp invocations with associated password prompts.

Comment #4 posted on 2017-05-22 11:12:48 by Jonathan Kulp

Details, details...

Aha! As always the devil is in the details. This last comment you left gave me the hint I needed. I was putting the asterisk in the wrong place and also using parentheses instead of curly braces. The following command works just like I want:

scp *.{mp3,ogg} jonserver:~/destination/dir/

Thanks, Dave!

Comment #5 posted on 2017-05-22 11:29:36 by Dave Morriss

TMTOWTDI

"There's more than one way to do it" - Larry Wall (actually he was talking about Perl, but it works here)

I'm glad I helped you to get where you wanted to be (even though I realise I wasn't quite answering your original question).

Having researched this and thought about it a bit I started putting together a brief(ish) HPR show about it.

Comment #6 posted on 2017-05-22 12:26:21 by Jonathan Kulp

Ken is smiling

Dave, somehow I suspected when I asked whether this worked with secure copy that it would end up becoming another show from you. You're the best!

Comment #7 posted on 2017-05-24 04:52:46 by clacke

scp brace expansion??!

Dave, I had no idea scp would do brace expansion on the server side. That's quite unexpected, and quite the discovery!

I tried it with rsync, and rsync *also* supports it. That made me suspicious.

As I feared, it seems to mean it runs the server side of rsync (and of scp) through the shell. Testing confirms it. If I want to (explicitly) copy files with spaces in the names, the quoting nightmare starts. :-(

So what started out as a happy discovery just turned into another disappointment in how broken our software is.

All this time, I had assumed that rsync started a server with no specific arguments, and then communicated over the rsync protocol which files to get. I guess I've been lucky all these years and avoided explicitly naming weird file names.

Comment #8 posted on 2017-05-24 10:09:52 by Dave Morriss

scp is a bit of a hack!!

Hi clacke.

I had been messing with scp using the -v option which generates a lot of information. You can see it connecting via ssh then, if the original command was:

scp -v dave@rpi4:'scptest/*.{mp3,ogg}' .

it sends across:

scp -v -f scptest/*.{mp3,ogg}

where the quotes prevent local expansion and the (undocumented) -f option in the command marks it as running on the remote end.

I don't know how this mechanism deals with names with spaces and so forth, but I imagine it's nasty. It needs some experiments. My ideas for a "brief" show about this subject look doomed to be "l o n g" :-)

I think rsync will talk to a remote server (never used it) but I'd expect it would need to exist before the transfer.

Leave Comment

Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.

Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).

Provide feedback
Your Name/Handle:
Title:
Comment:
Anti Spam Question: What does the letter P in HPR stand for?
Are you a spammer?
What is the HOST_ID for the host of this show?
What does HPR mean to you?