May 2013
M T W T F S S
« Apr    
 12345
6789101112
13141516171819
20212223242526
2728293031  
99

Refs

Categories

Archives

9,331slm
●5 ●38 ●132
 

How to rsync certain files, exclude the rest, all while ignoring .svn directories?

I came across this question on the Stack Exchange site Unix & Linux. The question interested me so I answered it but thought I’d cross post it on my blog as well, given I took a pretty significant amount of time to put together a test case and write-up of how the solution ultimately worked.

Problem

I’m using rsync to copy some files from a share to another.

Recursively, I need to:

- delete files at the destination that are deleted in the origin
- Only sync php and js files
- exclude de rest of file types
- Don’t delete .svn/ directory in the destination

If I use this:

rsync -zavC --delete --include='*.php' --include='*.js' --exclude="*" /media/datacod/Test/ /home/lucas/Desktop/rsync/

Then rsync is not recursive because exclude=”*” excludes all files but also folders

If I add --include="*/" then the .svn/ directory gets deleted (it also gets included)

How can I solve this mind blasting dilemma?

Solution

The solution I ultimately came up with made use of a little known feature, at least to me, called filters. Filters allow you to play games with the includes/excludes by protecting portions based on regular expressions. Read on, I’ll discuss them further down.

1
2
rsync -avzC --filter='-rs_*/.svn*' --include="*/" --include='*.js' --include='*.php' \
     --exclude="*" --delete dir1/ dir2/

test data

To help determine if my solution was going to work or not I created some sample data so that I could test it out. For starters I wrote a script that would generate the data. Here’s that script, setup_svn_sample.bash:


…. Continue reading → How to rsync certain files, exclude the rest, all while ignoring .svn directories? »»

[one-liner]: Debugging Bash Scripts

Background

From time to time it’s useful if you can turn up the debugging messages that come from Bash, when working out either interactive or shell script problems. Here are 2 methods that can help in getting down to the details.

Solution

There are essentially 2 methods.

Method #1: -x method

When writing a shell script you’ll sometimes want to turn on line by line debugging. There’s basically 2 ways to to this.

Before we get started, suppose we have this sample script, myscript.bash:

1
2
3
4
#!/bin/bash
 
echo "hi"
echo "bye"

First you can run your script like so:

1
2
3
4
5
% bash -x myscript.bash
+ echo hi
hi
+ echo bye
bye

As an alternative you can add the following line, set -x to the top of our shell script to enable debugging as well:

1
2
3
4
5
#!/bin/bash
 
set -x
echo "hi"
echo "bye"
1
2
3
4
5
% ./myscript.bash
+ echo hi
hi
+ echo bye
bye
Method #2: env SHELLOPTS=xtrace …

This approach sets the env. variable SHELLOPTS=xtrace which has the same effect as using bash -x.

For example:

1
2
3
4
5
% env SHELLOPTS=xtrace ./myscript.bash 
+ echo hi
hi
+ echo bye
bye

You can also use this technique to debug your bash environment (think .bashrc and .bash_profile) like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
% env SHELLOPTS=xtrace bash
...
...
+++++ line='complete -f -X
'\''!*.@(zip|[ejw]ar|exe|pk3|wsz|zargo|xpi|sxw|o[tx]t|od[fgpst]|epub|apk)'\''
unzip zipinfo'
+++++ line=' unzip zipinfo'
+++++ list=("${list[@]}" $line)
+++++ read line
+++++ '[' 'complete -f -X '\''*.Z'\'' compress znew' '!=' 'complete -f
-X '\''*.Z'\'' compress znew' ']'
+++++ line='complete -f -X '\''*.Z'\'' compress znew'
+++++ line='complete -f -X '\''*.Z'\'' compress znew'
+++++ line=' compress znew'
+++++ list=("${list[@]}" $line)
+++++ read line
+++++ '[' ' zcmp, zdiff, z*grep, zless, zmore intentionally not here,
see Debian: #455510' '!=' '# zcmp, zdiff, z*grep, zless, zmore
intentionally not here, see Debian: #455510' ']'
...
...

Here you can see every command getting executed from the system and user’s .bashrc and .bash_profile as bash starts up.

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

[one-liner]: How do you capture the status of a command ($?) in Bash, when run through a pipe?

Background

While answering questions on the stackexchage website Unix & Linux I saw the following question which was about something I’d encountered, but until today never knew how to accomplish, so I’m posting it here for my own reference in the future.

The question?

How do you get the exit status ( $? ) from the command haconf -makerw in place of grep? i.e. what need to add in to my syntax in order to understand if haconf -makerw succeeded?

1
2
    haconf -makerw | grep -iq "Cluster already writable"
        # echo $? ( will print the exe status from haconf -makerw  )

Solution

There are 3 ways of doing this. However your current setup should work. The reason here being that the grep won’t match anything if the command fails, so grep will return with status 1 (unless the program always shows that text no matter what).

Pipefail

The first way is to set the pipefail option. This is the simplest and what it does is basically set the exit status $? to the exit code of the last program to exit non-zero (or zero if all exited successfully).

1
2
3
4
5
# false | true; echo $?
0
# set -o pipefail
# false | true; echo $?
1
$PIPESTATUS

Bash also has a variable called $PIPESTATUS which contains the exit status of all the programs in the last command.


…. Continue reading → [one-liner]: How do you capture the status of a command ($?) in Bash, when run through a pipe? »»

[one-liner]: Overview of Bash I/O Redirection

Background

I recently came across this question on the Stackexchange site Unix & Linux. This question was interesting in the sense that it covered much of the I/O redirection facilities that are available in the Bash Shell, so for posterity sake I’m adding my answer to this question here on my blog.

Solution

  • a number 1 = standard out (i.e. STDOUT)
  • a number 2 = standard error (i.e. STDERR)
  • if a number isn’t explicitly given, then number 1 is assumed by the shell (bash)

First let’s tackle the function of these. For reference see the Advanced Bash-Scripting Guide.

Functions
  • 2>&-
- The general form of this one is M>&-, where “M” is a file descriptor number. This will close output for whichever file descriptor is referenced, i.e. “M”.
  • 2>/dev/null
- The general form of this one is M>/dev/null, where “M” is a file descriptor number. This will redirect the file descriptor, “M”, to /dev/null.
  • 2>&1
- The general form of this one is M>&N, where “M” & “N” are file descriptor numbers. It combines the output of file descriptors “M” and “N” into a single stream.
  • |&
- This is just an abbreviation for 2>&1. It was added in Bash 4.
  • &>/dev/null
- This is just an abbreviation for 2>&1 >/dev/null. It too was added in Bash 4. It redirects file descriptor 2 (STDERR) and descriptor 1 (STDOUT) to /dev/null.
  • >/dev/null
- This is just an abbreviation for 1>/dev/null. It redirects file descriptor 1 (STDOUT) to /dev/null.
Portability to non-bash, tcsh, mksh, etc.

I’ve not dealt much with other shells outside of csh and tcsh. My experience with those 2 compared to bash’s redirection operators, is that bash is superior in that regard. See the tcsh man page for more details.

Of the commands asked about in the Unix & Linux question, none are directly supported by csh or tcsh. You’d have to use different syntaxes to construct similar functions.

References

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

[one-liner]: How can I Trigger a Notification When a Command Line Job/Process Ends?

Background

Occasionally I need to notify myself when a long running command has finished. Here are a couple of tricks that I use to give me either a audible or visual notification.

Solution

The trick with this is to construct your command like so:

1
% command; command-after &

Then in the command-after position you can do things like this:

1
2
3
4
5
6
7
8
# ring the terminal bell
% tput bel
 
# use espeak to say stuff
% espeak "job's done!"
 
# notify-send (show's a message in upper right)
% notify-send "job's done!"
notify-send ss

notify-send ss

1
2
# zenity (show's a dialog box with OK button)
% zenity --info --text="Job's done"
zenity ss

zenity ss

Additionally if you want to only display a message if the command succeeds or fails you can use these patterns:

1
2
3
4
5
# command succeeded
% command && command-after &
 
# command failed
% command || command-after &

References

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

[one-liner]: Using scp to Copy Files that Contain Spaces

Background

Have you ever needed to copy either a single file or a directory that contains spaces using scp? Here are a couple of techniques for how to do this.

Solution

Here are several ways to accomplish the same thing. Copying a entire directory with spaces and copying a file with spaces using scp.

1
2
3
4
5
6
7
# Ways to copy entire directories
% scp -r user@myserver.com:'"/path/with/a/Space In It"' .
% scp -r user@myserver.com:"'/path/with/a/Space In It'" .
% scp -r user@myserver.com:'/path/with/a/Space\ In\ It' .
% scp -r user@myserver.com:"/path/with/a/Space\ In\ It" .
% scp -r user@myserver.com:/path/with/a/Space\\\ In\\\ It .
% scp -r user@myserver.com:"\"/path/with/a/Space In It\"" .
1
2
3
4
5
6
7
# Ways to copy a single file
% scp -r user@myserver.com:'"/path/with/a/Space In It/file with spaces.txt"' .
% scp -r user@myserver.com:"'/path/with/a/Space In It/file with spaces.txt'" .
% scp -r user@myserver.com:'/path/with/a/Space\ In\ It/file with spaces.txt' .
% scp -r user@myserver.com:"/path/with/a/Space\ In\ It/file\ with\ spaces.txt" .
% scp -r user@myserver.com:/path/with/a/Space\\\ In\\\ It/file\\\ with\\\ spaces.txt .
% scp user@myserver.com:"\"/path/with/a/Space In It/file with spaces.txt\"" .
links

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

Page 1 of 3123