From FreeBSDwiki
Revision as of 20:59, 27 June 2006 by Ninereasons (Talk | contribs)
Jump to: navigation, search

Redirection is sending the output of a program to somewhere other than where it would otherwise go - for example you can redirect the output of an ls command to a text file for later processing or to the grep command for filtering. Common operands include: >, >>, <, <<, and the ever popular |.

>  sends output to a file (may include special files such as /dev/null)
>> appends output to a file (without overwriting it)
<  read file to stdin
<< read to stdin from <<delimiter to delimiter (a HERE doc).
 |  sends output to a program (frequently, a system command like grep)

If you're using the bash or bourne shells, you also have some special options available to you: you can redirect standard input, standard output and standard error messages with far greater flexibility and reliability. Other shells such as csh are notably limited in redirection capability, making them better suited to interactive use than to shell programming or other complex uses.

Shell pipelines and redirection

There are three standard input/output file descriptors (fd) that are preconnected to the shell process running on your FreeBSD machine. Most commands that you would run from the command line expect these file descriptors to be open and accessible. The first (fd 0) provides a stream of input to your program, the second (fd 1) provides a stream of output, and the third (fd 2) provides a stream of diagnostic messages (usually to your terminal).

When you open a terminal, before running the shell the terminal device is opened three times to preconnect these file descriptors. The shell then inherits the file descriptors, and passes them on to each process run from the shell.

On FreeBSD systems it looks like this:

file descriptor Stream file descriptor file device path
0 Standard input /dev/stdin /dev/fd/0
1 Standard output /dev/stdout /dev/fd/1
2 Standard error /dev/stderr /dev/fd/2

The example below demonstrates that by default, all of these input/output streams are directed to your terminal (color is added).

$ for i in stdin stdout stderr; do echo $i stuff > /dev/$i; done
 stdin stuff
stdout stuff
stderr stuff

Redirection means that, the file descriptor is temporarily reassigned to somewhere other than the terminal device (a file, a pipe, another file descriptor). When the next process inherits the open fd from the shell the stream of data is passed along, for example, by writing to stdout, which has been temporarily reassigned to the stdin for the next process by means of a pipe (|). This handing off of open file descriptors by the shell, from one sub-process to another, is called a "shell pipeline".

There are fd n (3-9) descriptors potentially available, but only 0-2 are preconnected to the shell; the others must be created from the shell. In some shells, a standard file descriptor can be detached and reassigned to another file descriptor for as long as the terminal device is open, using the exec command (examples appear below). This capability is just one case among many of why sh is useful for scripting, and tcsh (which lacks it) is not (see the FAQ).

Redirection in sh compared to tcsh

The c-shells (tcsh or csh) and the Bourne shells (sh or bash) do not handle redirection or piping in quite the same way.

  • tcsh and sh
    1. Write output to a file
       % mycmd > out.txt
      stderr stuff
    2. Append output to a file
       % mycmd >> out.txt
      stderr stuff
    3. Redirect the output of a remote command to local.txt.
       % localcmd "remotecmd" > local.txt
      stderr stuff
    4. Same command as above, showing only the changes compared to local.txt.
      Note: Many programs recognize '-' as a shortcut for '/dev/stdin'. These two commands are equivalent.
       % localcmd "remotecmd" | diff /dev/stdin local.txt
       % localcmd "remotecmd" | diff - local.txt
      stderr stuff
    5. Direct stdout+stderr to file
       % localcmd >& out.txt
    6. Sort lines of jumble.txt into sorted.txt
      Note: the sequence in which redirection appears is not important. All of the following are exactly equivalent.
       % <jumble.txt sort >sorted.txt
       % >sorted.txt sort <jumble.txt
       % sort < jumble.txt >sorted.txt
       % <jumble.txt>sorted.txt sort
      stderr stuff
    7. Sort unique lines of jumble.txt into sorted.txt
       % <jumble.txt sort | uniq >sorted.txt
      stderr stuff
    8. Sort HERE doc delimited by "lines"
       % <<lines sort
      ? a second line
      ? a first line
      a first line
      a second line
      stderr stuff
  • tcsh only
    1. Discard errors, watch output (probably evil)
      Note: There is no reliable way to do this in tcsh. Here we exploit the fact that terminal reads from stdin.
       % ( mycmd > /dev/stdin ) > & /dev/null
      stdout stuff
       % ( mycmd > /dev/tty ) > & /dev/null
      stdout stuff
    2. Append output to out.txt; discard messages
       % (mycmd >> out.txt) >& /dev/null
    3. Write output to out.txt; store and watch errors
      Note: this happens to be easier in tcsh - a rare event. Compare the same task in sh.
       % ( mycmd > out.txt ) | & tee err.txt
      stderr stuff
  • sh only
    1. Discard errors, watch output
      $ mycmd 2> /dev/null
      stdout stuff
    2. Append output to out.txt; discard messages
      $ mycmd 2> /dev/null >> out.txt
    3. Write output to out.txt; store and watch errors
      Note: this happens to be harder in sh - a rare event. Compare the same task in tcsh.
      $ exec 3>&1 ; mycmd 2>&1 >&3 1>out.txt | tee err.txt ; exec 1<&3 3<&-;
      stderr stuff
    4. Write messages to err.txt; write output to out.txt and copy output to terminal
      $ mycmd 2> err.txt | tee out.txt
      stdout stuff
    5. Assign a variable from stored.txt
      $ <stored.txt read var; mycmd $var
      stderr stuff
      stdout stuff
    6. Assign first three lines of stored.txt to three different variables
      $ exec 3<&0; exec <test; read v1; read v2; read v3; exec 0<&3 3<&-; echo $v1 $v2 $v3
      stderr stuff
      stdout stuff
    7. Use all unique lines in stored.txt as variable input, appending to result.txt
      $ exec 3<&0; exec <test; sort | uniq | while read line; do mycmd $line >> result.txt ; done; exec 0<&3 3<&-;
      stderr stuff
    Note: By default, redirection pointed right represents stdout, so that these two commands are exactly equivalent:
    $ mycmd 1> out.txt
    $ mycmd > out.txt
    stderr stuff
    Note: By default, redirection pointed left represents stdin, so that these two commands are exactly equivalent:
    $ mycmd 0< in.txt
    $ mycmd < in.txt
    stderr stuff
    stdout stuff
    Note: To close a file descriptor, say n<&-
    $ mycmd >out.txt 1<&-;
    -sh: fails without messages
    -bash: mycmd: write error: Bad file descriptor

A little more about fd n in sh

Let's say you want to send output to your screen and errors to a file. You can't just do

samizdata# myprogram 1>&2 2>&1 > errors.txt

because when you do the first switch, it's done right away and when the second >& comes around, it's getting the switched data. This is where the other, normally unused, file descriptors 3-9 come in. You can use them as place-holders, such as:

samizdata# myprogram 3>&2 2>&1 1>&3 | command

will make the output of myprogram do this: 3 point to the same place as 2, 2 point to 1, and finally, 1 point to 3 and then pipe all of it to command

Also, see the man page for mkfifo, a utility for creating arbitrary file descriptor files.

Personal tools