Lâmôlabs » shell http://www.lamolabs.org/blog Lame Oh Labs .... Linux & Tech! Is there anything else? Thu, 29 Jul 2010 16:00:07 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 [one-liner]: Previewing a Pretty Printed Text File using enscript & ps2pdf http://www.lamolabs.org/blog/2842/one-liner-previewing-a-pretty-printed-text-file-using-enscript-ps2pdf/ http://www.lamolabs.org/blog/2842/one-liner-previewing-a-pretty-printed-text-file-using-enscript-ps2pdf/#comments Mon, 09 Nov 2009 15:45:07 +0000 slmingol http://www.lamolabs.org/blog/?p=2842 Before my wife and I had kids we thought it would be nice to collect the United States quarters that were released during 1999 through 2008 to commemorate each of the 50 states. Seemed like something simple to do and would be a nice gift for the kids when they got older. So we bought [...]]]> Background

Before my wife and I had kids we thought it would be nice to collect the United States quarters that were released during 1999 through 2008 to commemorate each of the 50 states. Seemed like something simple to do and would be a nice gift for the kids when they got older. So we bought a couple of the blue books which you can fill up with quarters as you find them. Each book contains 100 slots, 2 for each state. One slot is for the Philadelphia minted version of the quarter, and the other slot is for the Denver mint.

Problem

Well we ended up having 3 kids so we have to collect 300 quarters. The task of finding the quarters has been more of a dad task so when I have a chance, I’ll put a $5 dollar bill in various soda machines at work and go quarter fishing. This approach has been working fairly well and we’ve collected ~130 of the 300 quarters thus far.

However I’ve started getting to the point where I’m netting a lot of duplicates and the job of having to bring them home to weed through them is starting to get old. Having a list of which quarters we already have would sure be nice, so I could quickly nix any duplicates.

Solution

Of course I wanted a low tech solution, i.e. a piece of paper in my wallet would do the job, but how to do it?

The answer? A text file that I could maintain would suffice. No need for a bloated spreadsheet or some fancy handheld app. So I created a file, quarters.txt, like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

State           Year  Cnt (P/D)
=====           ====  =========
Alabama         2003  2/1
Alaska          2008  3/1
Arizona         2008  3/0
Arkansas        2003  3/0
California      2005  3/0
Colorado        2006  3/0
Connecticut     1999  3/2
Delaware        1999  0/0
Florida         2004  2/0
Georgia         1999  2/1
Hawaii          2008  3/0
...

It’s about as simple a text file as you can get. 3 columns, State, Year, and Counts. The 3rd column shows how many P and D quarters I have for a given state. So for example, for Alabama, I’ve got 2 Philadelphia minted quarters, and 1 Denver minted.

So you’re probably wondering, “why the hell is this guy writing up this in a blog post?”

We’ll the interesting bit to this low tech solution is how I print this list out. For this task I make use of a pretty powerful UNIX command called enscript, which lets you do all kinds of nifty things to a text file to augment how it looks when it gets printed.

About the only thing enscript doesn’t do for you, is give you the ability to preview your text file prior to printing. To accomplish this bit, I made use of another powerful UNIX command called ps2pdf. This command will take a postscript file (ps) and convert it to a pdf file.

So putting all the pieces together I came up with the following command:

1

enscript --fancy-header -U 4 quarters.txt -o - | ps2pdf - quarters_sm.pdf

The first part of this command, will call enscript instructing it to convert the file quarters.txt, and print it to standard out -o -. The printout will include some fancy headers and enscript will print the text file out in what is called 4 UP. This means that 4 pages will be printed on a single piece of paper. You could also print the page out in 2 UP, 8 UP, etc. It only needs to be a power of 2. BTW, 2 and 4 are the most commonly used, 8 is pretty hard to read.

The second part of this command passes the postscript generated by enscript through a UNIX pipe which gets picked up by ps2pdf, and converts it into a PDF file, quarters_sm.pdf.

From here you can check what the page would look like using your favorite PDF viewer, such as evince or xpdf. Once you’re comfortable with the page you can actually print it out from the PDF reader, or via the command-line.

Here’s what the resulting PDF file looks like:

PDF file

PDF file

Here’s a portion of the PDF file at a 150% of it’s original size:

screenshot of quarters_sm.pdf at 150%

screenshot of quarters_sm.pdf at 150%

Useful Links

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2842/one-liner-previewing-a-pretty-printed-text-file-using-enscript-ps2pdf/feed/ 0 [one-liner]: Copying & Moving Files efficiently with xargs http://www.lamolabs.org/blog/2745/one-liner-copying-moving-files-efficiently-with-xargs/ http://www.lamolabs.org/blog/2745/one-liner-copying-moving-files-efficiently-with-xargs/#comments Thu, 15 Oct 2009 14:13:24 +0000 slmingol http://www.lamolabs.org/blog/?p=2745 From time to time I need to move and/or copy a subset of files from one directory to another. I typically would use something like one of these to do the task:

1 2 3 4 5 #-- COPY find . -type f -ctime -1 | xargs -I {} cp {} /some/other/directory   #-- MOVE find . -type f -ctime -1 | xargs -I {} [...]]]> Background

From time to time I need to move and/or copy a subset of files from one directory to another. I typically would use something like one of these to do the task:

1
2
3
4
5

#-- COPY
find . -type f -ctime -1 | xargs -I {} cp {} /some/other/directory
 
#-- MOVE
find . -type f -ctime -1 | xargs -I {} mv {} /some/other/directory

NOTE: The 1st command finds all the files in the current directory that are less than 24 hours old, and copies them to /some/other/directory. The 2nd command finds all the files in the current directory that are less than 24 hours old, and moves them to /some/other/directory.

But then I realized that by using xargs’ –I switch I was basically limiting xargs to doing a single file at a time. According to xargs’ man page, when you use the –I switch you’re implying the –x switch AND the –L 1 switch. The L switch is what tells xargs how many lines of input to process at a time, so we’re basically telling it to only handle one file at a time. This made no sense. I was unintentially limiting xargs’ ability to optimize the command-line. So I found a better way.

New Approach

By utilizing 2 little used switches (–t | ––target-directory) on cp and mv I could un-tie xargs’ hands.

copy
1
2
3
4
5

# long form
find . -type f -ctime -1 | xargs -0 cp --target-directory=/some/other/directory
 
# short form
find . -type f -ctime -1 | xargs -0 cp -t /some/other/directory
move
1
2
3
4
5

# long form
find . -type f -ctime -1 | xargs -0 mv --target-directory=/some/other/directory
 
# short form
find . -type f -ctime -1 | xargs -0 mv -t /some/other/directory

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2745/one-liner-copying-moving-files-efficiently-with-xargs/feed/ 4 [one-liner]: How to Use the Bash Shell’s export Command http://www.lamolabs.org/blog/2681/one-liner-how-to-use-the-bash-shells-export-command/ http://www.lamolabs.org/blog/2681/one-liner-how-to-use-the-bash-shells-export-command/#comments Thu, 01 Oct 2009 14:35:12 +0000 slmingol http://www.lamolabs.org/blog/?p=2681 Recently at my day job I’ve been having to go through some pretty old Bash scripts that I’ve basically inherited. As I’ve been going through them I’ve been seeing a lot of confusion as to the proper use of Bash’s export command. The major offense? Not really understand whether a particular variables needs to be [...]]]> Problem

Recently at my day job I’ve been having to go through some pretty old Bash scripts that I’ve basically inherited. As I’ve been going through them I’ve been seeing a lot of confusion as to the proper use of Bash’s export command. The major offense? Not really understand whether a particular variables needs to be exported, or not. So I thought I’d take a moment just to clarify when and when not to use export.

The export command has really only one true purpose. To mark and/or unmark variables (and functions) that you want to have automatically exported to environments of subsequently executed commands. So if you create a script that calls other commands, and you want to push variables into the environment of these commands, then you’ll want to use export.

Example #1 (without export)

For example, let’s say we have the following 2 scripts:

1
2
3
4
5
6
7

#!/bin/bash
# script #1: parent.bash
 
var1="this was set by the parent shell script"
echo "inside $0 script: $var1"
 
./child.bash
1
2
3
4

#!/bin/bash
# script #2: child.bash
 
echo "inside $0 script: $var1"

And when I run the script parent.bash I get this output:

1
2
3
4
5

# output from parent.bash & child.bash (without export)
 
% ./parent.bash
inside ./parent.bash script: this was set by the parent shell script
inside ./child.bash script:

Notice how the variable $var1, which was set in the parent.bash script, didn’t get displayed by the child.bash script? Now watch this example with the variable $var1 exported in the parent.bash script.

Example #2 (with export)

1
2
3
4
5
6
7

#!/bin/bash
# script #1: parent.bash
 
export var1="this was set by the parent shell script"
echo "inside $0 script: $var1"
 
./child.bash
1
2
3
4

#!/bin/bash
# script #2: child.bash
 
echo "inside $0 script: $var1"

And when we run parent.bash

1
2
3
4
5

# output from parent.bash & child.bash (with export)
 
% ./parent.bash
inside ./parent.bash script: this was set my the parent shell script
inside ./child.bash script: this was set my the parent shell script

Example #3 (un-exporting)

Export isn’t just a one trick pony. It can also unmark a previously exported variable.

1
2
3
4
5
6
7
8
9

#!/bin/bash
# script #1: parent.bash
 
export var1="this was set by the parent shell script"
echo "inside $0 script: $var1"
 
./child.bash
export -n var1
./child.bash
1
2
3
4

#!/bin/bash
# script #2: child.bash
 
echo "inside $0 script: $var1"
1
2
3
4
5
6

# output from parent.bash & child.bash (with a before/after export)
 
% ./parent.bash
inside ./parent.bash script: this was set by the parent shell script
inside ./child.bash script: this was set by the parent shell script
inside ./child.bash script:

Here we see the effects of the export -n on child.bash the 2nd time it’s called.

Which Variables are Flagged for Export?

You can use the command export -p to get a list of all the variables marked for export, like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

% export -p
declare -x CCACHE_DIR="/var/cache/ccache"
declare -x CCACHE_UMASK="002"
declare -x COLORTERM="gnome-terminal"
declare -x CVS_RSH="ssh"
declare -x DESKTOP_SESSION="gnome"
declare -x DISPLAY=":0.0"
...
...
declare -x SHELL="/bin/bash"
declare -x SHLVL="4"
declare -x TERM="xterm"
declare -x USER="root"
declare -x WINDOWID="77021004"
declare -x XMODIFIERS="@im=imsettings"
declare -x var1="this was set by the parent shell script"

What else?

There is one additional trick related to exporting variables, but it doesn’t make use of the export command. It uses Bash’s set command. This command allows you to automatically export ALL the variables that have been modified or created to the environment of subsequent commands.

Here’s a quick example:

1
2
3
4
5
6
7
8
9
10
11
12

#!/bin/bash
# script #1: parent.bash
 
# automatically export EVERYTHING
set -a
 
var1="this was set by the parent shell script"
var2="this was set by the parent shell too"
echo "inside $0 script: $var1"
echo "inside $0 script: $var2"
 
./child.bash
1
2
3
4
5

#!/bin/bash
# script #2: child.bash
 
echo "inside $0 script: $var1"
echo "inside $0 script: $var2"

Useful links

]]> http://www.lamolabs.org/blog/2681/one-liner-how-to-use-the-bash-shells-export-command/feed/ 0 [one-liner]: Determining a Hard Drive’s Manufaturer Under Fedora 10 & CentOS 5 http://www.lamolabs.org/blog/2602/one-liner-determining-a-hard-drives-manufaturer-under-fedora-10-centos-5/ http://www.lamolabs.org/blog/2602/one-liner-determining-a-hard-drives-manufaturer-under-fedora-10-centos-5/#comments Mon, 21 Sep 2009 14:13:51 +0000 slmingol http://www.lamolabs.org/blog/?p=2602 I recently saw a post over on Linux Journal that discussed how to glean information about a system’s hard drive, such as its serial number, without having to actually open up the case and physically check it. So I thought I’d take the opportunity to write up a blog post with the specifics of how [...]]]> Background

I recently saw a post over on Linux Journal that discussed how to glean information about a system’s hard drive, such as its serial number, without having to actually open up the case and physically check it. So I thought I’d take the opportunity to write up a blog post with the specifics of how to do this under Fedora & CentOS, just so I’d have this info handy for future use.

BTW, I was able to accomplish this task several different ways, so this post will cover all the different ways that I could get this info.

Command #1: lshw

This is probably the best tool for getting at a system’s internals. First make sure it’s installed.

1

yum install lshw

For our example you would run the command lshw -class disk:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

% lshw -class disk
  *-disk                  
       description: ATA Disk
       product: HTS726060M9AT00
       vendor: Hitachi
       physical id: 0
       bus info: scsi@0:0.0.0
       logical name: /dev/sda
       version: MH4O
       serial: MRH403M4GS551Y
       size: 55GiB (60GB)
       capabilities: partitioned partitioned:dos
       configuration: ansiversion=5 signature=cccdcccd
  *-cdrom
       description: DVD reader
       product: UJDA755yDVD/CDRW
       vendor: MATSHITA
       physical id: 1
       bus info: scsi@1:0.0.0
       logical name: /dev/cdrom
       logical name: /dev/cdrw
       logical name: /dev/dvd
       logical name: /dev/scd0
       logical name: /dev/sr0
       version: 1.71
       capabilities: removable audio cd-r cd-rw dvd
       configuration: ansiversion=5 status=nodisc

The first section that’s returned is called -disk. Here’s you’ll see the vendor: Hitachi, the product number, HTS726060M9AT00, and my serial number: MRH403M4GS551Y.

Command #2: smartctl

The next tool that would give this type of info is called smartctl. It’s a tool that’s part of the smartmontool package. You may be familiar with the acronym S.M.A.R.T.. The acronym stands for: Self-Monitoring, Analysis, and Reporting Technology. This is a standard that most modern disks have in which vital statistics about a disk drive are provided through a standard API. Here’s how to install it.

1

yum install smartmontools

…and once installed you can use the bundled in tool smartctl like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# smartctl example
 
% smartctl -i /dev/sda
smartctl version 5.38 [i386-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
 
=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Travelstar 7K60
Device Model:     HTS726060M9AT00
Serial Number:    MRH403M4GS551Y
Firmware Version: MH4OA6BA
User Capacity:    60,011,642,880 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 3a
Local Time is:    Mon Sep 21 00:03:50 2009 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Command #3: hdparm

Another way that I’ve used to get at hard drive meta data is with the command hdparm. This is probably the oldest way, at least that I’m familiar with, for getting at hard drive meta data. To install it:

1

yum install hdparm

Using it is simply:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# hdparm example
 
% hdparm -i /dev/sda
 
/dev/sda:
 
 Model=HTS726060M9AT00                         , FwRev=MH4OA6BA, SerialNo=      MRH403M4GS551Y
 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
 BuffType=DualPortCache, BuffSize=7877kB, MaxMultSect=16, MultSect=?0?
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=117210240
 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4 
 DMA modes:  mdma0 mdma1 mdma2 
 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 
 AdvancedPM=yes: mode=0xC0 (192) WriteCache=enabled
 Drive conforms to: ATA/ATAPI-6 T13 1410D revision 3a:  ATA/ATAPI-2,3,4,5,6
 
 * signifies the current active mode

Command #4: lsscsi

Here’s another tool, lsscsi, that I’ve used off and on to get at hard drive meta data. It’s probably the least known of all the tools mentioned here. Installation is the same as the others:

1

yum install lsscsi

It’s usage is pretty much in-line with the other commands too:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# lsscsi example
 
% lsscsi -lll -vvv
 sysfsroot: /sys
[0:0:0:0]    disk    ATA      HTS726060M9AT00  MH4O  /dev/sda
  device_blocked=0
  iocounterbits=32
  iodone_cnt=0x92983
  ioerr_cnt=0x4f
  iorequest_cnt=0x92983
  queue_depth=1
  queue_type=none
  scsi_level=6
  state=running
  timeout=60
  type=0
  dir: /sys/bus/scsi/devices/0:0:0:0  [/sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0]
[1:0:0:0]    cd/dvd  MATSHITA UJDA755yDVD/CDRW 1.71  /dev/sr0
  device_blocked=0
  iocounterbits=32
  iodone_cnt=0xa0f7
  ioerr_cnt=0x0
  iorequest_cnt=0x283a7
  queue_depth=1
  queue_type=none
  scsi_level=6
  state=running
  timeout=0
  type=5
  dir: /sys/bus/scsi/devices/1:0:0:0  [/sys/devices/pci0000:00/0000:00:1f.1/host1/target1:0:0/1:0:0:0]

Command #4: /dev/disk/by-id directory

Finally you can get the manufacturer’s product number & serial number can be had by going directly to the /dev directory, specifically here: /dev/disk/by-id. In the resulting output you’ll see 2 important substrings. The first, HTS726060M9AT00, is he product number while the second, MRH403M4GS551Y, is the serial number.

1
2
3
4
5
6
7
8
9

# /dev/disk/by-id example
 
% ls -1 /dev/disk/by-id
ata-HTS726060M9AT00_MRH403M4GS551Y
ata-HTS726060M9AT00_MRH403M4GS551Y-part1
ata-HTS726060M9AT00_MRH403M4GS551Y-part2
scsi-SATA_HTS726060M9AT00_MRH403M4GS551Y
scsi-SATA_HTS726060M9AT00_MRH403M4GS551Y-part1
scsi-SATA_HTS726060M9AT00_MRH403M4GS551Y-part2

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2602/one-liner-determining-a-hard-drives-manufaturer-under-fedora-10-centos-5/feed/ 0 [one-liner]: Filtering ps from ps (redux) http://www.lamolabs.org/blog/2461/one-liner-filtering-ps-from-ps-redux/ http://www.lamolabs.org/blog/2461/one-liner-filtering-ps-from-ps-redux/#comments Sat, 05 Sep 2009 05:13:57 +0000 slmingol http://www.lamolabs.org/blog/?p=2461 As it is with UNIX & Linux there is always another way. In my previous article [one-liner]: Filtering ps from ps, one reader, Christoph, mentioned an alternative method to the one I outlined. In this case, I would consider his to be a better way, so I thought I would take a second to demonstrate [...]]]> As it is with UNIX & Linux there is always another way. In my previous article [one-liner]: Filtering ps from ps, one reader, Christoph, mentioned an alternative method to the one I outlined. In this case, I would consider his to be a better way, so I thought I would take a second to demonstrate this alternative method. The alternative? Use the command pgrep.

The Original Approach

My original post offered the following one-liner:

1
2
3
4
5
6
7
8
9
10

% ps -eaf | grep "[h]ttpd"
root      2683     1  0  2008 ?        00:20:31 /usr/sbin/httpd
apache   17146  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17147  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17149  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17150  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17151  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17152  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17153  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17154  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd

This one-liner provided a list of all the httpd processes running, while filtering out the actual string from the grep httpd command.

The Alternative Approach

By using the command pgrep, the same effect can be achieved and a lot more. For starters, you can get a list of all the httpd PIDs:

1
2
3
4
5
6
7
8
9

# list of httpd PIDs
 
% pgrep httpd
1608
7645
9739
10051
27712
27859

This could be useful in a shell script, if needed, to check for any running httpd processes. For example:

1
2
3
4

# test for httpd processes
 
% [ -z "`pgrep httpd`" ] || echo "running"
running

Here are some other examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# list of PIDs with corresponding command name
 
% pgrep -l httpd
1608 httpd
7645 httpd
9739 httpd
10051 httpd
27712 httpd
27859 httpd
 
# list of PIDs with corresponding command name owned by user root
 
% pgrep -l -u root httpd
1608 httpd
 
# list of PIDs, separated with a comma delimiter
 
% pgrep -d, httpd
1608,7645,9739,14119,14162,27859
 
# detailed list of httpd PIDs via ps
# NOTE: $(...) runs the command above, returning the list of PIDs to ps
 
% ps -fp $(pgrep -d, httpd)
UID        PID  PPID  C STIME TTY          TIME CMD
root      1608     1  0 Aug03 ?        00:00:05 /usr/sbin/httpd
apache    7645  1608  0 Sep04 ?        00:00:47 /usr/sbin/httpd
apache    9739  1608  0 Sep04 ?        00:01:50 /usr/sbin/httpd
apache   14119  1608  0 Sep04 ?        00:00:13 /usr/sbin/httpd
apache   14162  1608  0 Sep04 ?        00:00:13 /usr/sbin/httpd
apache   27859  1608  0 Sep04 ?        00:07:19 /usr/sbin/httpd

Thanks again to Christoph for pointing out this alternative.

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2461/one-liner-filtering-ps-from-ps-redux/feed/ 0 [one-liner]: Filtering ps from ps http://www.lamolabs.org/blog/2425/one-liner-filtering-ps-from-ps/ http://www.lamolabs.org/blog/2425/one-liner-filtering-ps-from-ps/#comments Fri, 04 Sep 2009 17:13:26 +0000 slmingol http://www.lamolabs.org/blog/?p=2425 This is a pretty handy trick to know when you want to filter out the command you’re running, so that it’s not included in ps output. This proves handy when writing a shell script that needs to parse output from ps.

NOTE: The command ps, allows you to see all the processes being run on [...]]]> Background

This is a pretty handy trick to know when you want to filter out the command you’re running, so that it’s not included in ps output. This proves handy when writing a shell script that needs to parse output from ps.

NOTE: The command ps, allows you to see all the processes being run on a UNIX/Linux system. You typically use it with the switches “-eaf” or “-ef”.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# columns description:
# UID   = user who owns the process
# PID   = process #
# PPID  = parents' process #
# C     = 
# STIME = start time
# TTY   = terminal type (has to do with which shell command ran from)
# TIME  = system time its run
# CMD   = command (aka. program name)
 
% ps -eaf
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0  2008 ?        00:24:46 init [5]
root         2     0  0  2008 ?        00:00:00 [kthreadd]
root         3     2  0  2008 ?        00:01:58 [migration/0]
root         4     2  0  2008 ?        00:23:13 [ksoftirqd/0]
root         5     2  0  2008 ?        00:00:00 [watchdog/0]
root         6     2  0  2008 ?        00:02:35 [migration/1]
root         7     2  0  2008 ?        00:09:55 [ksoftirqd/1]
root         8     2  0  2008 ?        00:00:00 [watchdog/1]
root         9     2  0  2008 ?        00:06:36 [events/0]
root        10     2  0  2008 ?        00:06:16 [events/1]
root        11     2  0  2008 ?        00:00:10 [khelper]
root        54     2  0  2008 ?        00:19:26 [kblockd/0]

Problem

Here’s an example where we want to see if a program is running, so we grep the output of ps like so:

1
2
3
4
5
6
7
8
9
10
11

% ps -eaf | grep httpd
root      2683     1  0  2008 ?        00:20:31 /usr/sbin/httpd
user1    13188  3984  0 12:45 pts/1    00:00:00 grep httpd
apache   17146  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17147  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17149  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17150  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17151  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17152  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17153  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17154  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd

The problem? Notice that a portion of our command, “grep httpd” is polluting our ps output. How can we get rid of it?

The Trick

You can alter what you are grepping for, without actually altering the results, by using a benign regular expression.

1
2
3
4
5
6
7
8
9
10

% ps -eaf | grep "[h]ttpd"
root      2683     1  0  2008 ?        00:20:31 /usr/sbin/httpd
apache   17146  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17147  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17149  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17150  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17151  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17152  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17153  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd
apache   17154  2683  0 Aug30 ?        00:00:02 /usr/sbin/httpd

Explanation

By re-encoding what we are looking for from httpd to [h]ttpd we’ve altered it by utilizing grep’s capability to use regular expressions. Regular expressions provide us the ability to use short-handed notations to represent patterns of text. Most people should be somewhat familiar with this capability when using the command ls to list files. The command ls can be used to find all the files that start with the letter a by utilizing a wildcard regular expression, aka. the asterisk character, aka. star, aka. *.

1
2
3
4
5
6
7
8
9
10
11
12

# wildcard examples
 
% ls
afile1  afile2  afile3  bfile1  bfile2  bfile3  cfile1  cfile2  cfile3
 
# example 1
% ls a*
afile1  afile2  afile3
 
# example 2
% ls b*
bfile1  bfile2  bfile3

Now lets say we want to create a single command that provides us with the list of ALL the files that start with an a or b. The bracket notation [ ] regular expression allows you to list sets of characters. So for example:

1
2
3
4
5
6
7
8
9
10
11
12

# set examples
 
% ls
afile1  afile2  afile3  bfile1  bfile2  bfile3  cfile1  cfile2  cfile3
 
# example 1
% ls [ab]*
afile1  afile2  afile3  bfile1  bfile2  bfile3
 
# example 2
% ls [bc]*
bfile1  bfile2  bfile3  cfile1  cfile2  cfile3

So back to our ps example. What does the bracket notation regular expression provide us here? This is the trick. By telling grep that we are looking for [h]ttpd, we’ve encoded our grep query to look like this: [h]ttpd, while what we’re actually grepping for to be: httpd. Since [h]ttpd doesn’t match httpd at the literal string level, grep will effectively ignore itself as being a match in the ps output.

A pretty neat [t]rick.

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2425/one-liner-filtering-ps-from-ps/feed/ 1 [one-liner]: Shell Builtins http://www.lamolabs.org/blog/2362/one-liner-shell-builtins/ http://www.lamolabs.org/blog/2362/one-liner-shell-builtins/#comments Tue, 25 Aug 2009 05:12:29 +0000 slmingol http://www.lamolabs.org/blog/?p=2362 I recently worked a problem in a previous post where it wasn’t clear which version of the command nohup was getting used. The version that was built into the C-shell interpreter, or the executable sitting under /usr/bin/nohup. This brought up an interesting point, how do you know what commands are builtins to the shell itself?

Enter [...]]]> Background

I recently worked a problem in a previous post where it wasn’t clear which version of the command nohup was getting used. The version that was built into the C-shell interpreter, or the executable sitting under /usr/bin/nohup. This brought up an interesting point, how do you know what commands are builtins to the shell itself?

Enter the commands: builtins and enable

These are 2 commands that I’d never even heard of until I started doing research for this post. They both basically do the same thing. They show you all the builtin commands for a particular shell, builtins for C-shell, and enable for Bourne Shell.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# C-shell builtins
 
#NOTE: run this command within a C-shell (aka. csh or tcsh)
% csh
 
% builtins
:          @          alias      alloc      bg         bindkey    break
breaksw    builtins   case       cd         chdir      complete   continue
default    dirs       echo       echotc     else       end        endif
endsw      eval       exec       exit       fg         filetest   foreach
glob       goto       hashstat   history    hup        if         jobs
kill       limit      log        login      logout     ls-F       nice
nohup      notify     onintr     popd       printenv   pushd      rehash
repeat     sched      set        setenv     settc      setty      shift
source     stop       suspend    switch     telltc     termname   time
umask      unalias    uncomplete unhash     unlimit    unset      unsetenv
wait       where      which      while
1
2
3
4
5
6
7

# Bourne Again Shell builtins
 
#NOTE: run this command within Bourne Again Shell (aka. bash)
% bash
 
% enable | cut -d" " -f2 | sed -e :a -e '$!N;s/\n/ /;ta'
. : [ alias bg bind break builtin caller cd command compgen complete continue declare dirs disown echo enable eval exec exit export false fc fg getopts hash help history jobs kill let local logout popd printf pushd pwd read readonly return set shift shopt source suspend test times trap true type typeset ulimit umask unalias unset wait

NOTE: The command is enable. I’m simply running the output of enable through this cut & sed filter so that the output of enable is more concise for this example.

nohup

As you can see, by looking at the output from builtins, the command nohup is listed there, which means that C-shell has it’s own implementation of nohup. Looking at the output from enable, you can see that Bourne Shell, doesn’t include it’s own builtin for nohup, so it would use /usr/bin/nohup. Pretty simple, eh?

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2362/one-liner-shell-builtins/feed/ 0 [one-liner]: How to Properly Use nohup http://www.lamolabs.org/blog/2349/one-liner-how-to-properly-use-nohup/ http://www.lamolabs.org/blog/2349/one-liner-how-to-properly-use-nohup/#comments Tue, 25 Aug 2009 02:55:32 +0000 slmingol http://www.lamolabs.org/blog/?p=2349 Today a co-worker, let’s call him Evan, was trying to retrofit the UNIX command nohup into a script that I originally developed, and was not having much luck. He was trying to debug a unstable server daemon that’s called by my script. The server daemon had been crashing for the better part of 2 weeks, [...]]]> Background

Today a co-worker, let’s call him Evan, was trying to retrofit the UNIX command nohup into a script that I originally developed, and was not having much luck. He was trying to debug a unstable server daemon that’s called by my script. The server daemon had been crashing for the better part of 2 weeks, and the vendor of this particular daemon was directing Evan to prefix the launching of the daemon with nohup. Seemed like a simple request, but nohup wasn’t behaving as he expected. Both Evan and the vendor support specialist, let’s call him Sal, were looking for a nohup.out file to be deposited in the directory where they were running my script. I initially thought that my script might be changing directories behind the scenes, causing the files to show up some where else. After a quick check I could find no evidence of any nohup.out files anywhere. I really didn’t have much experience with nohup either so I figured this was the perfect excuse to better understand how nohup works.

nohup

nohup stands for “no hangup”, and is a way to start a process up that is immune from being hungup. It’s typically invoked like this:

1

nohup sleep 100 &

This means that within UNIX, when a parent process spawns a child process, and the parent process is then sent the signal to hangup, the child process has been instructed via nohup to ignore that signal. At which point the child process then becomes a child process of the init process, i.e. the process with the PID of 1. The init process is the originating process of every process on a UNIX system. As always, an example can explain it better than I:

example #1 (simple nohup, without killing the parent process)
1
2
3
4
5
6
7
8
9
10
11
12
13
14

# start the command "sleep 5" nohup'd and backgrounded
 
% nohup sleep 5 &
[1] 32257
 
% ps -eaf | egrep "[s]leep|[U]ID|[/]sbin/init"
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun20 ?        00:00:01 /sbin/init
root     32257 31977  0 21:36 pts/8    00:00:00 sleep 5
 
% ps -eaf | egrep "[s]leep|[U]ID|[/]sbin/init"
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun20 ?        00:00:01 /sbin/init
[1]+  Done                    nohup sleep 5
example #2 (parent is killed, child remains)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# initially start 2 terminal windows up (termA & termB)
 
# termA
% nohup sleep 15 &
[1] 32229
% ps -eaf | egrep "[s]leep|[U]ID|[/]sbin/init"
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun20 ?        00:00:01 /sbin/init
root     32229 32195  0 21:30 pts/9    00:00:00 sleep 15
% exit
 
# at this point termA has been destroyed (its PID was 32195)
 
# termB
% ps -eaf | egrep "[s]leep|[U]ID|[/]sbin/init"
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun20 ?        00:00:01 /sbin/init
root     32229     1  0 21:30 ?        00:00:00 sleep 15
 
# notice how the sleep process is now owned by the parent process init (PPID 1)
 
% ps -eaf | egrep "[s]leep|[U]ID|[/]sbin/init"
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun20 ?        00:00:01 /sbin/init
root     32229     1  0 21:30 ?        00:00:00 sleep 15
 
% ps -eaf | egrep "[s]leep|[U]ID|[/]sbin/init"
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun20 ?        00:00:01 /sbin/init
%

So Where was the nohup.out File?

After debugging this problem for a bit I realized that my co-worker Evan was attempting to use nohup in a C-shell (csh). Turns out that made all the difference. See if you can see it:

1
2
3
4
5
6
7
8
9
10
11
12

# scenario 1
% nohup sleep 1 &
[1] 32583
% 
[1]    Done                          sleep 1
 
# scenario 2
% nohup sleep 1 &
nohup: ignoring input and appending output to `nohup.out'
[1] 32584
% 
[1]+  Done                    nohup sleep 1

Which of the above scenarios do you think is the C-shell? Which is the Bourne Shell? Scenario #1 is the C-shell & scenario #2 is the Bourne Shell. Why the difference? Using the commands which and whereis didn’t really help to answer why either, since they both turned up /usr/bin/nohup.

1
2
3
4
5
6
7
8
9
10
11

# csh
% which nohup
/usr/bin/nohup
% whereis nohup
nohup: /usr/bin/nohup /usr/share/man/man1p/nohup.1p.gz /usr/share/man/man1/nohup.1.gz
 
# sh
% which nohup
/usr/bin/nohup
% whereis nohup
nohup: /usr/bin/nohup /usr/share/man/man1p/nohup.1p.gz /usr/share/man/man1/nohup.1.gz

I knew it was shell dependent but I couldn’t see why. So I turned to the C-shell man page, and here was my first clue.

hup [command] (+)
With command, runs command such that it will exit on a hangup signal and arranges for the shell to send it a hangup signal when the
shell exits. Note that commands may set their own response to hangups, overriding hup. Without an argument (allowed in only a
shell script), causes the shell to exit on a hangup for the remainder of the script. See also Signal handling and the nohup
builtin command.

The bit at the end mentioned that C-shell had its own builtin version of nohup. A light bulb immediately went off and I now knew why nohup was behaving differently between C-shell and Bourne shell. Because there were in fact 2 different versions of nohup. One was built into C-shell, while the other was an executable on the system located here, /usr/bin/nohup. The version built into C-shell apparently doesn’t automatically create the nohup.out file, you have to do this yourself by redirecting STDOUT and STDERR. Bourne shell, on the other hand, doesn’t have its own version of nohup, and so it relies on the one located at /usr/bin/nohup. This version of nohup does automatically create the nohup.out file.

Additionally when I referred to the man page for /usr/bin/nohup I noticed this comment:

NOTE: your shell may have its own version of nohup, which usually supersedes the version described here. Please refer to your shell’s doc-
umentation for details about the options it supports.

So I told Evan that if he wants to get the nohup.out file he needs to call it explicitly with the full path like this:

1

/usr/bin/nohup sleep 10 &


The following URLs proved useful in helping me to round out my understanding of nohup.

  1. How to keep a process running when I log out?
  2. nohup Execute Commands After You Exit From a Shell Prompt
  3. How to make processes stay alive after a logout
  4. nohup that doesn’t generate nohup.out

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2349/one-liner-how-to-properly-use-nohup/feed/ 0 [one-liner]: Port Forwarding Using PuTTY http://www.lamolabs.org/blog/2234/one-liner-port-forwarding-using-putty/ http://www.lamolabs.org/blog/2234/one-liner-port-forwarding-using-putty/#comments Sat, 15 Aug 2009 06:02:56 +0000 slmingol http://www.lamolabs.org/blog/?p=2234 Here’s a little trick that I learned how to do the other day while at work. I’ve done this before using ssh in a linux terminal but hadn’t had the chance to actually do it from a windows box using PuTTY.

Objective

To access a web server (on a remote linux box) which is listening on port [...]]]> Here’s a little trick that I learned how to do the other day while at work. I’ve done this before using ssh in a linux terminal but hadn’t had the chance to actually do it from a windows box using PuTTY.

Objective

To access a web server (on a remote linux box) which is listening on port 80 through an ssh connection. We want to access the web server through port 10001 on the localhost. This can be accomplished by mapping localhost’s port 10001 to port 80 on the remote host.

Here’s a diagram that attempts to represent what’s going on with the ports, the hosts, and the ssh connection.

ports diagram

ports diagram

Setup

In this scenario we have 2 hosts, the local host (i.e. localhost) and the remote host (homer). Running PuTTY on the localhost, we specify that we want to login to homer as a user on that system, i.e. root, for example. It doesn’t have to be root, this just happens to be the user that I’m using in this example.

putty dialog #1

putty dialog #1

Next in the PuTTY Configuration dialog box, expand the category SSH, and select the category Tunnels. In Tunnels’ configuration, specify the Source port, 10001, and the Destination, homer:80. Then click the Add button.

putty dialog #2

putty dialog #2

At this point the PuTTY Configuration dialog should look like this, after clicking the Add button. Now go ahead and connect to the remote host by clicking the Open button, and login as you’d normally would to the remote host.

putty dialog #3

putty dialog #3

After successfully logging in, you should be sitting at your shells prompt.

putty dialog #4

putty dialog #4

Now in your web browser put in the URL http://localhost:10001/, and you should see the front page for the web server running on the remote host, homer.

firefox screenshot

firefox screenshot

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

]]> http://www.lamolabs.org/blog/2234/one-liner-port-forwarding-using-putty/feed/ 0 Managing OpenVZ Instance Descriptions http://www.lamolabs.org/blog/2094/managing-openvz-instance-descriptions/ http://www.lamolabs.org/blog/2094/managing-openvz-instance-descriptions/#comments Mon, 27 Jul 2009 03:31:01 +0000 slmingol http://www.lamolabs.org/blog/?p=2094 I recently discovered that you can assign descriptions to your OpenVZ instances. This isn’t really that surprising but I never really took the the time until now to scan through the vzctl and vzlist man pages. Prior to discovering these more esoteric features of vzctl and vzlist, I’d normally just run this command to see [...]]]> I recently discovered that you can assign descriptions to your OpenVZ instances. This isn’t really that surprising but I never really took the the time until now to scan through the vzctl and vzlist man pages. Prior to discovering these more esoteric features of vzctl and vzlist, I’d normally just run this command to see what’s what with my OpenVZ instances:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# list of all VE instances
% vzlist -a
      CTID      NPROC STATUS  IP_ADDR         HOSTNAME                        
       101         32 running 10.1.1.101      flanders.mydom.net              
       102         49 running 10.1.1.102      lisa.mydom.net                  
       103         36 running -               bart.mydom.net                  
       104         38 running 10.1.1.104      marge.mydom.net                 
       105         34 running 10.1.1.105      homer.mydom.net                 
       106         31 running 10.1.1.106      kang.mydom.net                  
       107         27 running 10.1.1.107      kodos.mydom.net                 
       108         32 running 10.1.1.108      maude.mydom.net                 
       109         30 running 10.1.1.109      nelson.mydom.net                
       110         32 running 10.1.1.110      ralphie.mydom.net               
       111         30 running 10.1.1.111      martin.mydom.net

This had worked fine, but sometimes I’d draw a blank about what’s running in each instance, hence my need for the description column. You can use the following command to see the description of each instance.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# initial description columns for VEs
% vzlist -o ctid,numproc,status,ip,hostname,description
      CTID      NPROC STATUS  IP_ADDR         HOSTNAME                         DESCRIPTION                     
       101         34 running 10.1.1.101      flanders.mydom.net               -                               
       102         49 running 10.1.1.102      lisa.mydom.net                   -                               
       103         35 running -               bart.mydom.net                   -                               
       104         38 running 10.1.1.104      marge.mydom.net                  -                               
       105         34 running 10.1.1.105      homer.mydom.net                  -                               
       106         31 running 10.1.1.106      kang.mydom.net                   -                               
       107         27 running 10.1.1.107      kodos.mydom.net                  -                               
       108         32 running 10.1.1.108      maude.mydom.net                  -                               
       109         30 running 10.1.1.109      nelson.mydom.net                 -                               
       110         32 running 10.1.1.110      ralphie.mydom.net                -                               
       111         30 running 10.1.1.111      martin.mydom.net                 -

I used the following commands to set the description for each VE.

1
2
3
4
5
6
7
8
9
10
11

vzctl set 101 --description "nis server" --save
vzctl set 102 --description "mail,imap,smtp server" --save
vzctl set 103 --description "samba server" --save
vzctl set 104 --description "mysql server" --save
vzctl set 105 --description "www,blog,webmail server" --save
vzctl set 106 --description "trac,git,svn server" --save
vzctl set 107 --description "trac,git,svn server (dev)" --save
vzctl set 108 --description "ldap server (dev)" --save
vzctl set 109 --description "cacti server" --save
vzctl set 110 --description "tracks server" --save
vzctl set 111 --description "wiki server" --save

Re-running the vzlist command from before now shows the newly added descriptions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# filled out description columns for VEs
% vzlist -o ctid,numproc,status,ip,hostname,description
      CTID      NPROC STATUS  IP_ADDR         HOSTNAME                         DESCRIPTION                     
       101         32 running 10.1.1.101      flanders.mydom.net               nis server                      
       102         49 running 10.1.1.102      lisa.mydom.net                   mail,imap,smtp server           
       103         36 running -               bart.mydom.net                   samba server                    
       104         38 running 10.1.1.104      marge.mydom.net                  mysql server                    
       105         34 running 10.1.1.105      homer.mydom.net                  www,blog,webmail server         
       106         31 running 10.1.1.106      kang.mydom.net                   trac,git,svn server             
       107         27 running 10.1.1.107      kodos.mydom.net                  trac,git,svn server (dev)       
       108         32 running 10.1.1.108      maude.mydom.net                  ldap server (dev)               
       109         30 running 10.1.1.109      nelson.mydom.net                 cacti server                    
       110         32 running 10.1.1.110      ralphie.mydom.net                tracks server                   
       111         30 running 10.1.1.111      martin.mydom.net                 wiki server
]]> http://www.lamolabs.org/blog/2094/managing-openvz-instance-descriptions/feed/ 1