A question came up on the Stack Exchange site Unix & Linux in which I wrote up a pretty good answer, that describes some of the mechanics of how a process deals with its user credentials, so I’m adding my writeup to the blog.

It really comes down to what makes up a process in Unix. A process can come into existence in one of 2 ways. Either via the fork() function or through one of the exec() functions in C.

fork()

fork() basically just makes a copy of the current process, but assigns it a new process ID (PID). It’s a child of the original process. You can see this relationship in the output of @ps@:

1
2
3
4
5
6
7

$ ps axjf
 PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
    1  5255  1964  1964 ?           -1 Sl     500   0:39 gnome-terminal
 5255  5259  1964  1964 ?           -1 S      500   0:00  \_ gnome-pty-helper
 5255 18422 18422 18422 pts/1    18422 Ss+    500   0:01  \_ bash
 5255 30473 30473 30473 pts/4    30473 Ss+    500   0:00  \_ bash
30473   782   782 30473 pts/4    30473 Sl     500   1:14  |   \_ evince s.pdf

Here you can see that gnome-terminal is the parent process (PID = 5255) and that bash is it’s child (PID = 18422, PPID = 5255).

When a process forks from its parent, it “inherits” certain things, such as copies of all the file descriptors that the parent currently has for open files and the parent’s user and group IDs.

NOTE1: PPID = Parent Process ID.
NOTE2: The last 2 are what identify what file and group permissions this process will have when accessing the file system.

So if a process just inherits its user and group ID from its parent, then why isn’t everything just owned by root or a single user? This is where exec() comes in.

exec() Part #1

The exec() family of functions, specifically execve(), “replace” a current process image with a new process image. The terminology “process image” is really just a file, i.e. an executable on disk. So this is how a bash script can execute a program such as /usr/bin/time.

So what about the user ID and group ID? Well to understand that let’s first discuss the concept of “Persona”.

Persona

At any time, each process has an effective user ID, an effective group ID, and a set of supplementary group IDs. These IDs determine the privileges of the process. They are collectively called the [persona of the process]¹, because they determine “who it is” for purposes of access control.

exec() Part #2

So in addition to being able to swap out the “process image”, exec() can also change the user & group IDs from the original “real” ones to “effective” ones.

An example

For this demonstration I’m going to show you what happens when we start out in a shell as our default UID/GID, and then spawn a child shell using one of my supplementary GIDs, making it the child shell’s effective GID.

To perform this I’m going to make use of the unix command newgrp. newgrp allows you to spawn a new shell passing it the supplementary group that I’d like to make my effective GID.

For starters:

1
2

$ id -a
uid=500(saml) gid=501(saml) groups=501(saml),502(vboxusers),503(jupiter)

We can see that this shell is currently configured with my default UID/GID of saml & saml. Touching some files shows that this is the case as well:

1
2
3
4
5
6

$ touch afile1
$ touch afile2
$ ls -l
total 0
-rw-rw-r-- 1 saml saml 0 May 21 23:47 afile1
-rw-rw-r-- 1 saml saml 0 May 21 23:47 afile2

Now we make our supplementary group jupiter the effective GID:

1
2
3

$ newgrp jupiter
$ id -a
uid=500(saml) gid=503(jupiter) groups=501(saml),502(vboxusers),503(jupiter)

Now if we touch some files:

1
2
3
4
5
6
7
8

$ touch afile3
$ touch afile4
$ ls -l
total 0
-rw-rw-r-- 1 saml saml    0 May 21 23:47 afile1
-rw-rw-r-- 1 saml saml    0 May 21 23:47 afile2
-rw-r--r-- 1 saml jupiter 0 May 21 23:49 afile3
-rw-r--r-- 1 saml jupiter 0 May 21 23:49 afile4

We see that the shell’s effective GID is jupiter, so any interactions with the disk result in files being created with jupiter rather than my normal default group of saml.

References

• fork() man page
• exec() man page
• execve() man page
• credentials man page
• newgrp man page
• The GNU C Library
• The GNU C Library – 26.4 Creating a Process
• The GNU C Library – 26.5 Executing a File
• GID, current, primary, supplementary, effective and real group IDs?

People that use Linux on a daily basis probably are completely oblivious to the actual mechanisms being used to store their passwords safely and securely on a given Linux system. Oh they might guess that their password is stored in the /etc/passwd file (they’d be wrong by the way) but most probably never even gave it a passing thought. So I thought I’d take the opportunity to shed some light on how Linux systems “stash” your precious password away.

Solution

So if your password isn’t actually stored in the /etc/passwd file then where does it get stored?

Answer: the /etc/shadow file.

This file is where all the keys to each user’s account are kept for safe keeping. Obviously only the root user can peer inside this file so all the commands we’ll be dealing with in this post, it should be assumed that you’ll need to either be root, or use sudo to run.

/etc/shadow

A typical /etc/shadow entry:

1

root:$6$bbmDJwcZHy5bgEDz$kFO.W/T7nUqcszZWl5RglxoDDAcDxevWpHVfN3v3f.Cx2ZeMcn5PX23VvnnkgtNWZf8hYtqsL0pPkZqyj50NY/:14362:0:33333:7:::

NOTE1: Don’t get too excited, the above isn’t really my entry, I made this one up.
NOTE2: Each field is separated by a colon (:) & we’re only concerned with the first two columns!

dissecting the hash

The key pieces to notice in that line of what looks like gibberish is the following:

• The first column, root is the user whom this entry belongs to from the /etc/passwd file.
• The second column, $6$..... is essentially the user’s hashed password.

Taking the second column apart further you should start to notice that’s it’s not complete gibberish after all.

For example:

• the first couple of characters, $6$, is a mark that tells the system what type of hashing was used to hash the password.
• The text between the next set of dollar signs, $bbmDJwcZHy5bgEDz$, is the actual salt that was used to hash your password.
• Everything else after, is your password + salt hashed using whatever hash function was specified at the beginning, $6$, in our example here.

Specifically if you look at the man page for the crypt command, man 3 crypt there is a section that discusses what the $6$ notation means:

So $5$salt$encrypted is an SHA-256 encoded password and $6$salt$encrypted is an SHA-512 encoded one.

NOTE: So in our case the password + salt is being hashed using the SHA-512 scheme.

design details

For reference purposes here’s the rest of that excerpt from the crypt man page:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

If salt is a character string starting with the characters "$id$" followed by a
string terminated by "$":
 
       $id$salt$encrypted
 
then instead of using the DES machine, id identifies the encryption method used
and this then determines how the rest of the password string is interpreted.
The following values of id are supported:
 
       ID  | Method
       ─────────────────────────────────────────────────────────
       1   | MD5
       2a  | Blowfish (not in mainline glibc; added in some
           | Linux distributions)
       5   | SHA-256 (since glibc 2.7)
       6   | SHA-512 (since glibc 2.7)
 
So $5$salt$encrypted is an SHA-256 encoded password and $6$salt$encrypted is an
SHA-512 encoded one.
 
"salt" stands for the up to 16 characters following "$id$" in the salt. The
encrypted part of the password string is  the actual computed password. The
size of this string is fixed:
 
MD5     | 22 characters
SHA-256 | 43 characters
SHA-512 | 86 characters
 
The characters in "salt" and "encrypted" are drawn from the set [a–zA–Z0–9./].
In the MD5 and SHA implementations the entire key is significant (instead of
only the first 8 bytes in DES).

Now what?

So by now you’re probably saying to yourself. OK, big deal, my password is hashed with some salt and stored in /etc/shadow. What else?

generating the hash manually using mkpasswd

For starters you can generate the $6$... string yourself manually using the mkpasswd command:

1
2

$ mkpasswd -m sha-512 password saltsalt
$6$saltsalt$qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/

In the above command we’re specifying that we want to use the SHA-512 hash, our password is the string password and our salt string is saltsalt. As before we can see in our resulting string the following components:

• $6$ – which hash function was used
• saltsalt – the string “saltsalt” was used
• qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/ – password + salt hashed using SHA-512

generating the hash manually using Python

I came across the following nice Python one-liner that effectively does the same thing as the mkpasswd command discussed above.

1
2
3

$ python -c "import crypt, getpass, pwd; \
             print crypt.crypt('password', '\$6\$saltsalt\$')"
$6$saltsalt$qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/

generating the hash manually using Perl

1
2

$ perl -e 'print crypt("password","\$6\$saltsalt\$") . "\n"'
$6$saltsalt$qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/

authconfig

Before I wrap up I thought I’d mention one final tool authconfig that’s included on Red Hat distros such as Fedora, CentOS, and RHEL. This tool allows you to change what hash algorithm is being used on a particular system. The command to change a system to use SHA-512 would be as follows:

1

authconfig –passalgo sha512 –update

See the man page for authconfig for more details.

conclusions

And with that you are now a little more in the know as to how Linux systems take your password and store them in the /etc/shadow file.

References

links

• crypt man page
• Thread: generate hashed password for /etc/shadow – ubuntuforums.org
• How to produce test hashes for various formats
• Shadow Fu
• Using ruby to generate SHA512 crypt-style hashes formatted for /etc/shadow?
• Calculate SHA Hash of Linux Password in Shadow File

Problem

I’m using rsync to copy some files from a share to another.

Recursively, I need to:

- delete files at the destination that are deleted in the origin
- Only sync php and js files
- exclude de rest of file types
- Don’t delete .svn/ directory in the destination

If I use this:

rsync -zavC --delete --include='*.php' --include='*.js' --exclude="*" /media/datacod/Test/ /home/lucas/Desktop/rsync/

Then rsync is not recursive because exclude=”*” excludes all files but also folders

If I add --include="*/" then the .svn/ directory gets deleted (it also gets included)

How can I solve this mind blasting dilemma?

Solution

The solution I ultimately came up with made use of a little known feature, at least to me, called filters. Filters allow you to play games with the includes/excludes by protecting portions based on regular expressions. Read on, I’ll discuss them further down.

1
2

rsync -avzC --filter='-rs_*/.svn*' --include="*/" --include='*.js' --include='*.php' \
     --exclude="*" --delete dir1/ dir2/

test data

To help determine if my solution was going to work or not I created some sample data so that I could test it out. For starters I wrote a script that would generate the data. Here’s that script, setup_svn_sample.bash:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/bin/bash
 
# setup .svn dirs
mkdir -p dir{1,2}/dir{1,2,3,4}/.svn
 
# fake data under .svn
mkdir -p dir1/dir{1,2,3,4}/.svn/origdir
mkdir -p dir2/dir{1,2,3,4}/.svn/keepdir
 
# files to not sync
touch dir1/dir{1,2,3,4}/file{1,2}
 
# files to sync
touch dir1/dir{1,2,3,4}/file1.js
touch dir1/dir{1,2,3,4}/file1.php

Running the above script produces the following directories (dir1 & dir2):

source dir

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

$ tree -a dir1
dir1
|-- dir1
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
|-- dir2
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
|-- dir3
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
`-- dir4
    |-- file1
    |-- file1.js
    |-- file1.php
    |-- file2
    `-- .svn
        `-- origdir

destination dir

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ tree -a dir2
dir2
|-- dir1
|   `-- .svn
|       `-- keepdir
|-- dir2
|   `-- .svn
|       `-- keepdir
|-- dir3
|   `-- .svn
|       `-- keepdir
`-- dir4
    `-- .svn
        `-- keepdir

Running the above rsync command which includes the --filter below we can see that it’s only syncing the files that match the --include patterns:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

rsync -avzC --filter='-rs_*/.svn*' --include="*/" --include='*.js' --include='*.php' \
     --exclude="*" --delete dir1/ dir2/
sending incremental file list
dir1/file1.js
dir1/file1.php
dir2/file1.js
dir2/file1.php
dir3/file1.js
dir3/file1.php
dir4/file1.js
dir4/file1.php
 
sent 480 bytes  received 168 bytes  1296.00 bytes/sec
total size is 0  speedup is 0.00

Resulting dir2 afterwards:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

$ tree -a dir2
dir2
|-- dir1
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
|-- dir2
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
|-- dir3
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
`-- dir4
    |-- file1.js
    |-- file1.php
    `-- .svn
        `-- keepdir

Why does it work?

The key piece to this script is to make use of the filters capability of rsync. Filters allow you to remove files from the matched set at various points in the command. So in our case we’re filtering any files that match the pattern */.svn*. The modifiers -rs_ tell the filter that we want to filter on both the source side as well as the target side.

excerpt from the FILTER NOTES section of rsync’s man page

- An s is used to indicate that the rule applies to the sending side. When a rule affects the sending side, it prevents files from being
transferred. The default is for a rule to affect both sides unless --delete-excluded was specified, in which case default rules become sender-side only. See also the hide (H) and show (S) rules, which are an alternate way to specify sending-side includes/excludes.

- An r is used to indicate that the rule applies to the receiving side. When a rule affects the receiving side, it prevents files from being deleted. See the s modifier for more info. See also the protect (P) and risk ® rules, which are an alternate way to specify receiver-side includes/excludes.

See man rsync for more details.

Tips for figuring this out (hint using --dry-run)

While describing how to do this I thought I’d mention the --dry-run switch to rsync. It’ extremely useful in seeing what will happen without having the rsync actually take place.

For Example

Using the following command will do a test run and show us the decision logic behind rsync:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

rsync --dry-run -avvzC --filter='-rs_*/.svn*' --include="*/" \
     --include='*.js' --include='*.php' --exclude="*" --delete dir1/ dir2/
sending incremental file list
[sender] showing directory dir3 because of pattern */
[sender] showing directory dir2 because of pattern */
[sender] showing directory dir4 because of pattern */
[sender] showing directory dir1 because of pattern */
[sender] hiding file dir1/file1 because of pattern *
[sender] showing file dir1/file1.js because of pattern *.js
[sender] hiding file dir1/file2 because of pattern *
[sender] showing file dir1/file1.php because of pattern *.php
[sender] hiding directory dir1/.svn because of pattern */.svn*
[sender] hiding file dir2/file1 because of pattern *
[sender] showing file dir2/file1.js because of pattern *.js
[sender] hiding file dir2/file2 because of pattern *
[sender] showing file dir2/file1.php because of pattern *.php
[sender] hiding directory dir2/.svn because of pattern */.svn*
[sender] hiding file dir3/file1 because of pattern *
[sender] showing file dir3/file1.js because of pattern *.js
[sender] hiding file dir3/file2 because of pattern *
[sender] showing file dir3/file1.php because of pattern *.php
[sender] hiding directory dir3/.svn because of pattern */.svn*
[sender] hiding file dir4/file1 because of pattern *
[sender] showing file dir4/file1.js because of pattern *.js
[sender] hiding file dir4/file2 because of pattern *
[sender] showing file dir4/file1.php because of pattern *.php
[sender] hiding directory dir4/.svn because of pattern */.svn*
delta-transmission disabled for local transfer or --whole-file
[generator] risking directory dir3 because of pattern */
[generator] risking directory dir2 because of pattern */
[generator] risking directory dir4 because of pattern */
[generator] risking directory dir1 because of pattern */
[generator] protecting directory dir1/.svn because of pattern */.svn*
dir1/file1.js
dir1/file1.php
[generator] protecting directory dir2/.svn because of pattern */.svn*
dir2/file1.js
dir2/file1.php
[generator] protecting directory dir3/.svn because of pattern */.svn*
dir3/file1.js
dir3/file1.php
[generator] protecting directory dir4/.svn because of pattern */.svn*
dir4/file1.js
dir4/file1.php
total: matches=0  hash_hits=0  false_alarms=0 data=0
 
sent 231 bytes  received 55 bytes  572.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

In the above output you can see that the ./svn directories are being protected by our filter rule. Valuable insight for debugging the rsync.

References

- Delete extraneous files from dest dir via rsync?
- Above scripts in a tarball

UEFI looks to be a major pain in the @$$, but like it or hate it everyone in the Linux community will need to learn to navigate it. Here’s a list of useful UEFI resources that I’ve come across as I’ve started to get smarter about how to deal with this beast.

Solution

Wikipedia

• Unified Extensible Firmware Interface

Ubuntu Docs

• UEFI
• UEFIBooting – Booting Linux natively on a UEFI system (without BIOS CSM) using GRUB2

AskUbuntu

• Install Ubuntu next to Windows 8
• Installing Ubuntu on a Pre-Installed UEFI Supported Windows 8 system

Rodsbooks.com

• Managing EFI Boot Loaders for Linux
• Managing EFI Boot Loaders for Linux: Using GRUB 2
• Managing EFI Boot Loaders for Linux: Dealing with Secure Boot

Misc.

• Running RHEL 6 on EFI BIOS-based system

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

From time to time it’s useful if you can turn up the debugging messages that come from Bash, when working out either interactive or shell script problems. Here are 2 methods that can help in getting down to the details.

Solution

There are essentially 2 methods.

Method #1: -x method

When writing a shell script you’ll sometimes want to turn on line by line debugging. There’s basically 2 ways to to this.

Before we get started, suppose we have this sample script, myscript.bash:

1
2
3
4

#!/bin/bash
 
echo "hi"
echo "bye"

First you can run your script like so:

1
2
3
4
5

% bash -x myscript.bash
+ echo hi
hi
+ echo bye
bye

As an alternative you can add the following line, set -x to the top of our shell script to enable debugging as well:

1
2
3
4
5

#!/bin/bash
 
set -x
echo "hi"
echo "bye"

1
2
3
4
5

% ./myscript.bash
+ echo hi
hi
+ echo bye
bye

Method #2: env SHELLOPTS=xtrace …

This approach sets the env. variable SHELLOPTS=xtrace which has the same effect as using bash -x.

For example:

1
2
3
4
5

% env SHELLOPTS=xtrace ./myscript.bash 
+ echo hi
hi
+ echo bye
bye

You can also use this technique to debug your bash environment (think .bashrc and .bash_profile) like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

% env SHELLOPTS=xtrace bash
...
...
+++++ line='complete -f -X
'\''!*.@(zip|[ejw]ar|exe|pk3|wsz|zargo|xpi|sxw|o[tx]t|od[fgpst]|epub|apk)'\''
unzip zipinfo'
+++++ line=' unzip zipinfo'
+++++ list=("${list[@]}" $line)
+++++ read line
+++++ '[' 'complete -f -X '\''*.Z'\'' compress znew' '!=' 'complete -f
-X '\''*.Z'\'' compress znew' ']'
+++++ line='complete -f -X '\''*.Z'\'' compress znew'
+++++ line='complete -f -X '\''*.Z'\'' compress znew'
+++++ line=' compress znew'
+++++ list=("${list[@]}" $line)
+++++ read line
+++++ '[' ' zcmp, zdiff, z*grep, zless, zmore intentionally not here,
see Debian: #455510' '!=' '# zcmp, zdiff, z*grep, zless, zmore
intentionally not here, see Debian: #455510' ']'
...
...

Here you can see every command getting executed from the system and user’s .bashrc and .bash_profile as bash starts up.

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

If you’ve every dealt with Conky you may have gotten a little confused when you’re trying to get it to display your CPU frequency like so:

1

${freq_g cpu0} Ghz

… and Conky is reporting your CPU frequency as 1.12GHz when in fact it’s actually much higher than that, say 2.67GHz. Most likely this is being caused by the CPU governing features that are present in most modern hardware. Here’s a behind the scenes 5 second tour of seeing what’s going on within your Linux Kernel.

Solution

First, from the conky man page.

cpu (cpuN)

CPU usage in percents. For SMP machines, the CPU number can
be provided as an argument. ${cpu cpu0} is the total usage, and ${cpu
cpuX} (X >= 1) are individual CPUs.

freq_g (n)

Returns CPU #n’s frequency in GHz. CPUs are counted from 1.
If omitted, the parameter defaults to 1.

You most likely have something like SpeedStep enabled which is acting like a governor on a car, regulating the speed of the cores inside your CPU. You can confirm that this is going on by looking at the output of this command:

1
2
3
4
5
6
7
8
9

% less /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 37
model name      : Intel(R) Core(TM) i5 CPU       M 560  @ 2.67GHz
stepping        : 5
cpu MHz         : 1199.000
...

The 2 numbers that matter are the 2.67GHz, that the GHz that my CPU is rated to operate at followed by the number 1199.00, this is what my CPU is allowed to run at by the governor setup on my Linux laptop.

You can see what governor is currently configured like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# available governors
% sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors 
powersave ondemand userspace performance 
# which one am I using?
% sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 
powersave
 
# what's my current frequency scaling?
% sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 
1199000
 
# what maximum is available?
% sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq 
2667000
 
# what's the minimum?
% sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq 
1199000
 
# what scaling frequencies can my CPU support?
% sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies 
2667000 2666000 2533000 2399000 2266000 2133000 1999000 1866000 1733000 1599000 1466000 1333000 1199000

You can override your governor by doing the following, using one of the governor’s listed above:

1

% sudo sh -c "echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor"

References

links

• Conky
• Conky man page
• SpeedStep – wikipedia
• Using CPU Frequency (CPUFreq) on Linux Reference

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

While answering questions on the stackexchage website Unix & Linux I saw the following question which was about something I’d encountered, but until today never knew how to accomplish, so I’m posting it here for my own reference in the future.

The question?

How do you get the exit status ( $? ) from the command haconf -makerw in place of grep? i.e. what need to add in to my syntax in order to understand if haconf -makerw succeeded?

1
2

    haconf -makerw | grep -iq "Cluster already writable"
        # echo $? ( will print the exe status from haconf -makerw  )

Solution

There are 3 ways of doing this. However your current setup should work. The reason here being that the grep won’t match anything if the command fails, so grep will return with status 1 (unless the program always shows that text no matter what).

Pipefail

The first way is to set the pipefail option. This is the simplest and what it does is basically set the exit status $? to the exit code of the last program to exit non-zero (or zero if all exited successfully).

1
2
3
4
5

# false | true; echo $?
0
# set -o pipefail
# false | true; echo $?
1

$PIPESTATUS

Bash also has a variable called $PIPESTATUS which contains the exit status of all the programs in the last command.

1
2
3
4
5
6
7
8

# true | true; echo "${PIPESTATUS[@]}"
0 0
# false | true; echo "${PIPESTATUS[@]}"
1 0
# false | true; echo "${PIPESTATUS[0]}"
1
# true | false; echo "${PIPESTATUS[@]}"
0 1

NOTE: You can use the 3rd command example to get the specific value in the pipeline that you need.

This solution might not be available though. I think $PIPESTATUS might have been added in a fairly recent version of bash, and your OS may not have it.

Separate executions

This is the most unwieldy of the solutions. Run each command separately and capture the status.

1
2
3

# OUTPUT="$(haconf -makerw)"
# STATUS_HACONF="$?"
# printf '%s' "$OUTPUT" | grep -iq "Cluster already writable"

References

links

• how to get exit status from the command before the last – U&L
• Answer to U&L Question
• bash: tee output AND capture exit status – stackoverflow

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

Here’s a quick tip for checking out the overall health of your computer’s hard drive. It makes use of a little known tool called udisks which provides access to information about storage devices from the D-Bus interface.

Solution

Before we jump into udisks here are some resources that might prove useful when dealing with the design/architecture of udisks.

• udisks’ main website
• udisks’ reference manual

To see which drives are currently under udisks watchful eye, you can use the following command:

1
2
3
4
5
6
7
8
9
10
11
12

% sudo udisks --enumerate
/org/freedesktop/UDisks/devices/dm_2d0
/org/freedesktop/UDisks/devices/dm_2d1
/org/freedesktop/UDisks/devices/dm_2d2
/org/freedesktop/UDisks/devices/sda1
/org/freedesktop/UDisks/devices/sda2
/org/freedesktop/UDisks/devices/sda3
/org/freedesktop/UDisks/devices/sda4
/org/freedesktop/UDisks/devices/sda5
/org/freedesktop/UDisks/devices/sda6
/org/freedesktop/UDisks/devices/sr0
/org/freedesktop/UDisks/devices/sda

This output shows that I have several partitions that are currently bing monitored, the last bit in the path tells you which device, i.e. sda1, sda2, sda3, etc. These are partitions /dev/sda1, /dev/sda2, etc.

Now on to the actual health output. This next command will show you the overall health of device /dev/sda, i.e. my primary HDD on my laptop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

udisks --show-info /dev/sda
Showing information for /org/freedesktop/UDisks/devices/sda
  native-path:                 /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda
  device:                      8:0
  device-file:                 /dev/sda
    presentation:              /dev/sda
    by-id:                     /dev/disk/by-id/ata-ST9500420AS_5VJ9H3BY
    by-id:                     /dev/disk/by-id/scsi-SATA_ST9500420AS_5VJ9H3BY
    by-id:                     /dev/disk/by-id/wwn-0x5000c5002ed97512
    by-path:                   /dev/disk/by-path/pci-0000:00:1f.2-scsi-0:0:0:0
  detected at:                 Mon 15 Apr 2013 01:00:39 PM EDT
  system internal:             1
  removable:                   0
  has media:                   1 (detected at Mon 15 Apr 2013 01:00:39 PM EDT)
    detects change:            0
    detection by polling:      0
    detection inhibitable:     0
    detection inhibited:       0
  is read only:                0
  is mounted:                  0
  mount paths:             
  mounted by uid:              0
  presentation hide:           0
  presentation nopolicy:       0
  presentation name:           
  presentation icon:           
  size:                        500107862016
  block size:                  512
  job underway:                no
  usage:                       
  type:                        
  version:                     
  uuid:                        
  label:                       
  partition table:
    scheme:                    mbr
    count:                     6
  drive:
    vendor:                    ATA
    model:                     ST9500420AS
    revision:                  0003LV1M
    serial:                    5VJ9H3BY
    WWN:                       5000c5002ed97512
    detachable:                0
    can spindown:              1
    rotational media:          Yes, at 7200 RPM
    write-cache:               enabled
    ejectable:                 0
    adapter:                   /org/freedesktop/UDisks/adapters/0000_3a00_3a1f_2e2
    ports:
      /org/freedesktop/UDisks/adapters/0000_3a00_3a1f_2e2/host0
    similar devices:
    media:                     
      compat:                 
    interface:                 ata
    if speed:                  (unknown)
    ATA SMART:                 Updated at Mon 15 Apr 2013 09:26:39 PM EDT
      overall assessment:      Good
===============================================================================
 Attribute       Current|Worst|Threshold  Status   Value       Type     Updates
===============================================================================
 raw-read-error-rate         104| 99| 34   good    6520505     Pre-fail Online 
 spin-up-time                100| 99|  0    n/a    0           Pre-fail Online 
 start-stop-count             98| 98| 20   good    2785        Old-age  Online 
 reallocated-sector-count    100|100| 36   good    0 sectors   Pre-fail Online 
 seek-error-rate              72| 60| 30   good    25872919731 Pre-fail Online 
 power-on-hours               89| 89|  0    n/a    424.5 days  Old-age  Online 
 spin-retry-count            100|100| 97   good    0           Pre-fail Online 
 power-cycle-count            98| 98| 20   good    2753        Old-age  Online 
 attribute-184               100|100| 99   good    0           Old-age  Online 
 reported-uncorrect          100|100|  0    n/a    0 sectors   Old-age  Online 
 attribute-188               100| 96|  0    n/a    0           Old-age  Online 
 high-fly-writes             100|100|  0    n/a    0           Old-age  Online 
 airflow-temperature-celsius  58| 42| 45 FAIL_PAST 42C / 108F  Old-age  Online 
 g-sense-error-rate          100|100|  0    n/a    124         Old-age  Online 
 power-off-retract-count     100|100|  0    n/a    15          Old-age  Online 
 load-cycle-count              1|  1|  0    n/a    248327      Old-age  Online 
 temperature-celsius-2        42| 58|  0    n/a    42C / 108F  Old-age  Online 
 hardware-ecc-recovered       44| 38|  0    n/a    6520505     Old-age  Online 
 reallocated-event-count      89| 89| 30   good    79886391715520 Pre-fail Online 
 current-pending-sector      100|100|  0    n/a    0 sectors   Old-age  Online 
 offline-uncorrectable       100|100|  0    n/a    0 sectors   Old-age  Offline
 udma-crc-error-count        200|200|  0    n/a    0           Old-age  Online 
 attribute-254               100|100|  0    n/a    0           Old-age  Online

One thing to note, my HDD is essentially overheating at 58 degrees celcius while threshold for that value is suppose to be 45 degrees celcius. Might need to think about replacing it some time soon 8-).

Here’s another way to get just the health information using the –dump switch instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

% udisks --dump | grep -A 24 Updates
 
 Attribute       Current|Worst|Threshold  Status   Value       Type     Updates
===============================================================================
 raw-read-error-rate         103| 99| 34   good    6214810     Pre-fail Online 
 spin-up-time                100| 99|  0    n/a    0           Pre-fail Online 
 start-stop-count             98| 98| 20   good    2785        Old-age  Online 
 reallocated-sector-count    100|100| 36   good    0 sectors   Pre-fail Online 
 seek-error-rate              72| 60| 30   good    25872903767 Pre-fail Online 
 power-on-hours               89| 89|  0    n/a    424.4 days  Old-age  Online 
 spin-retry-count            100|100| 97   good    0           Pre-fail Online 
 power-cycle-count            98| 98| 20   good    2753        Old-age  Online 
 attribute-184               100|100| 99   good    0           Old-age  Online 
 reported-uncorrect          100|100|  0    n/a    0 sectors   Old-age  Online 
 attribute-188               100| 96|  0    n/a    0           Old-age  Online 
 high-fly-writes             100|100|  0    n/a    0           Old-age  Online 
 airflow-temperature-celsius  58| 42| 45 FAIL_PAST 42C / 108F  Old-age  Online 
 g-sense-error-rate          100|100|  0    n/a    124         Old-age  Online 
 power-off-retract-count     100|100|  0    n/a    15          Old-age  Online 
 load-cycle-count              1|  1|  0    n/a    248327      Old-age  Online 
 temperature-celsius-2        42| 58|  0    n/a    42C / 108F  Old-age  Online 
 hardware-ecc-recovered       44| 38|  0    n/a    6214810     Old-age  Online 
 reallocated-event-count      89| 89| 30   good    181604102186687 Pre-fail Online 
 current-pending-sector      100|100|  0    n/a    0 sectors   Old-age  Online 
 offline-uncorrectable       100|100|  0    n/a    0 sectors   Old-age  Offline
 udma-crc-error-count        200|200|  0    n/a    0           Old-age  Online 
 attribute-254               100|100|  0    n/a    0           Old-age  Online

References

links

• Introduction to Udisks – Finnbarr P. Murphy
• udisks’ main website
• udisks’ reference manual

local copies

• Introduction to Udisks – Finnbarr P. Murphy

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

This post is intended to capture various online resources for taking classes and/or getting training in anything. They’re predominately programming language and software engineering courses but their are other tracks that focus on history and chemistry, for example.

Solution

This is by no means meant to be an exhaustive list, it’s a clearinghouse of the different resources as I come across them.

General Sites

• MIT Open Course Ware
• 700 Free Online Courses from Top Universities
• MOOCs: Top 10 Sites for Free Education With Elite Universities
• Coursera

General Programming Courses

• Codecademy
• Stanford Engineering Everywhere courses
• Harvard Extension School – Intensive Introduction to Computer Science

HTML/Javascript/Web/etc.

• Computer Science E-75: Building Dynamic Websites
• Computer Science E-76: Building Mobile Applications
• Code Avengers
• Openclassroom

Ruby

• Ruby Quiz
• Ruby4Kids
• Learn Ruby with the Neo Ruby Koans

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

I recently came across this question on the Stackexchange site Unix & Linux. This question was interesting in the sense that it covered much of the I/O redirection facilities that are available in the Bash Shell, so for posterity sake I’m adding my answer to this question here on my blog.

Solution

• a number 1 = standard out (i.e. STDOUT)
• a number 2 = standard error (i.e. STDERR)
• if a number isn’t explicitly given, then number 1 is assumed by the shell (bash)

First let’s tackle the function of these. For reference see the Advanced Bash-Scripting Guide.

Functions

Portability to non-bash, tcsh, mksh, etc.

I’ve not dealt much with other shells outside of csh and tcsh. My experience with those 2 compared to bash’s redirection operators, is that bash is superior in that regard. See the tcsh man page for more details.

Of the commands asked about in the Unix & Linux question, none are directly supported by csh or tcsh. You’d have to use different syntaxes to construct similar functions.

References

• Advanced Bash-Scripting Guide
• tcsh man page
• stackexchange unix & linux question regarding shell redirections

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.