This page is provided as a convenient way to download. It should also be helpful for people on dial-up to have one big page to read. A list of the other “AllOfs” can be found at http://linuxbasics.org/course/book/allallofs.

Please take into considerations that once you save this page, it cannot keep up with the dynamics of the wiki. If you find something to be corrected or improved, you need to check or edit the actual current wiki-page.

The current version of this document can be found at http://linuxbasics.org/course/book/index.

This book is licensed and may be distributed under the terms of the GNU Free Documentation License. A copy of the license can be found at http://linuxbasics.org/course/book/allof_appendix

Usually, these snapshots are updated daily between 05:00 and 05:15 UTC

This snapshot was created: Sun Jul 20 06:08:07 CEST 2008

• 5.1. Simple redirections
  • 5.1.1. What are standard input and standard output?
  • 5.1.2. The redirection operators
• 5.2. Advanced redirection features
  • 5.2.1. Use of file descriptors
  • 5.2.2. Examples
• 5.3. Filters
  • 5.3.1. More about grep
  • 5.3.2. Filtering output
• 5.4. Summary
• 5.5. Exercises

This chapter describes more about the powerful UNIX mechanism of redirecting input, output and errors. Topics include:

• Standard input, output and errors
• Redirection operators
• How to use output of one command as input for another
• How to put output of a command in a file for later referrence
• How to append output of multiple commands to a file
• Input redirection
• Handling standard error messages
• Combining redirection of input, output and error streams
• Output filters

Prev: Exercises
Home
Next: Simple redirections

Most Linux commands read input, such as a file or another attribute for the command, and write output. By default, input is being given with the keyboard, and output is displayed on your screen. Your keyboard is your standard input (stdin) device, and the screen or a particular terminal window is the standard output (stdout) device.

However, since Linux is a flexible system, these default settings don’t necessarily have to be applied. The standard output, for example, on a heavily monitored server in a large environment may be a printer.

Sometimes you will want to put output of a command in a file, or you may want to issue another command on the output of one command. This is known as redirecting output. Redirection is done using either the > (greater-than symbol), or using the | (pipe) operator which sends the standard output of one command to another command as standard input.

As we saw before, the cat command concatenates files and puts them all together to the standard output. By redirecting this output to a file, this file name will be created - or overwritten if it already exists, so take care.

First we will make two files called test1 and test2. The command cat > test1 is typed and then some words are typed. You can use the ENTER key and make several lines of text. When finished use either Ctrl-C or Ctrl-D. Then do the same to make test2:

nancy:~> cat > test1
some words

nancy:~> cat > test2
some other words

Now look at the files. Notice that when you use cat to look at the files, there is no redirection sign. The words you have put into the files appear on the screen as output. Then put two files together by redirecting them into test3 and look at test 3 using cat.:

nancy:~> cat test1
some words

nancy:~> cat test2
some other words

nancy:~> cat test1 test2 > test3

nancy:~> cat test3
some words
some other words

• Don’t overwrite!
  • Be careful not to overwrite existing (important) files when redirecting output. Many shells, including Bash, have a built-in feature to protect you from that risk: noclobber. See the Info pages for more information. In Bash, you would want to add the set -o noclobber command to your .bashrc configuration file in order to prevent accidental overwriting of files.

Redirecting “nothing” to an existing file is equal to emptying the file:

nancy:~> ls -l list
-rw-rw-r--    1 nancy   nancy     117 Apr  2 18:09 list

nancy:~> > list

nancy:~> ls -l list
-rw-rw-r--    1 nancy   nancy       0 Apr  4 12:01 list

This process is called truncating.

The same redirection to an nonexistent file will create a new empty file with the given name:

nancy:~> ls -l newlist
ls: newlist: No such file or directory

nancy:~> > newlist

nancy:~> ls -l newlist
-rw-rw-r--  1 nancy   nancy	    0 Apr  4 12:05 newlist

Chapter 7 gives some more examples on the use of this sort of redirection.

Some examples using piping of commands:

To find a word within some text, display all lines matching pattern1, and exclude lines also matching pattern2 from being displayed:

grep pattern1 file | grep -v pattern2

To display output of a directory listing one page at a time:

ls -la | less

To find a file in a directory:

ls -l | grep part_of_file_name

In another case, you may want a file to be the input for a command that normally wouldn’t accept a file as an option. This redirecting of input is done using the < (less-than symbol) operator.

Below is an example of sending a file to somebody, using input redirection.

andy:~> mail mike@somewhere.org < to_do

If the user mike exists on the system, you don’t need to type the full address. If you want to reach somebody on the Internet, enter the fully qualified address as an argument to mail.

This reads a bit more difficult than the beginner’s cat file | mail someone, but it is of course a much more elegant way of using the available tools.

The following example combines input and output redirection. The file text.txt is first checked for spelling mistakes, and the output is redirected to an error log file:

spell < text.txt > error.log

The following command lists all commands that you can issue to examine another file when using less:

mike:~> less --help | grep -i examine
  :e [file]      Examine a new file.
  :n          *  Examine the (N-th) next file from the command line.
  :p          *  Examine the (N-th) previous file from the command line.
  :x          *  Examine the first (or N-th) file from the command line.

The -i option is used for case-insensitive searches - remember that UNIX systems are very case-sensitive.

If you want to save output of this command for future reference, redirect the output to a file:

mike:~> less --help | grep -i examine > examine-files-in-less

mike:~> cat examine-files-in-less
  :e [file]      Examine a new file.
  :n          *  Examine the (N-th) next file from the command line.
  :p          *  Examine the (N-th) previous file from the command line.
  :x          *  Examine the first (or N-th) file from the command line.

Output of one command can be piped into another command virtually as many times as you want, just as long as these commands would normally read input from standard input and write output to the standard output. Sometimes they don’t, but then there may be special options that instruct these commands to behave according to the standard definitions; so read the documentation (man and info pages) of the commands you use if you should encounter errors.

Again, make sure you don’t use names of existing files that you still need. Redirecting output to existing files will replace the content of those files.

Instead of overwriting file data, you can also append text to an existing file using two subsequent greater-than signs:

Example. First we look at what is already on Mike’s wishlist using cat. Then we append the date. Looking again, we now see the date has been added at the end.:

mike:~> cat  wishlist
more money
less work

mike:~> date >> wishlist

mike:~> cat wishlist
more money
less work
Thu Feb 28 20:23:07 CET 2002

The date command would normally put the last line on the screen; now it is appended to the file wishlist.

Prev: I/O redirection
Home
Next: Advanced redirection features

There are three types of I/O, which each have their own identifier, called a file descriptor:

• standard input: 0
• standard output: 1
• standard error: 2

Also note that:

• > is the same as 1> meaning redirect to standard output
• < is the same as 0< meaning redirect to standard input

Some practical examples will make this more clear. Remember that the $ is not part of the command, but indicates the bash prompt:

First make a directory and a file:

$ mkdir testdir
$ cd testdir
$ touch file1
$ ls

Now try a command that will produce output with an error:

$ ls file1 file2 
ls: file2: No such file or directory
file1

Now redirect the output and the error to two different files:

$ ls file1 file2 1> dirlist 2> errorlist
$ cat dirlist
file1
$ cat errorlist
ls: file2: No such file or directory

Notice that there is no output to the terminal with the ls command. The output has gone to the two files. The file listing went to dirlist being directed there as standard ouput (1>) and the error message went to errorlist being directed there as standard error (2>)

Now try redirecting both standard output and error to the same file:

$ ls file1 file2 > dirlist 2>&1
$ cat dirlist
ls: file2: No such file or directory
file1

Here we put the redirection symbols at the end, because bash will check this from the right to the left. Following is a shortcut way to do the same thing:

$ ls file1 file2 &> dirlist

Things are getting quite complicated here, don’t confuse the use of the ampersand here with the use of it in Section 4.1.2.1, where the ampersand is used to run a process in the background. Here, it merely serves as an indication that the number that follows is not a file name, but rather a location that the data stream is pointed to. Also note that the bigger-than sign should not be separated by spaces from the number of the file descriptor. If it would be separated, we would be pointing the output to a file again. The example below demonstrates this:

[nancy@asus /var/tmp]$ ls 2> tmp

[nancy@asus /var/tmp]$ ls -l tmp
-rw-rw-r--  1 nancy nancy 0 Sept  7 12:58 tmp

[nancy@asus /var/tmp]$ ls 2 > tmp
ls: 2: No such file or directory

The first command that nancy executes is correct (even though no errors are generated and thus the file to which standard error is redirected is empty). The second command expects that 2 is a file name, which does not exist in this case, so an error is displayed.

All these features are explained in detail in the Bash Info pages.

If your process generates a lot of errors, this is a way to thoroughly examine them:

command 2>&1 | less

This is often used when creating new software using the make command, such as in:

andy:~/newsoft> make all 2>&1 | less
--output ommitted--

Note that the shortcut &> will not substitute here. That shortcut only works when redirecting to a file.

Constructs like these are often used by programmers, so that output is displayed in one terminal window, and errors in another. Find out which pseudo terminal you are using issuing the tty command first:

andy:~/newsoft> make all 2> /dev/pts/7

You can use the tee command to copy input to standard output and one or more output files in one move. Using the -a option to tee results in appending input to the file(s). This command is useful if you want to both see and save output. The > and » operators do not allow you to perform both actions simultaneously.

This tool is usually called on through a pipe (|), as demonstrated in the example below:

mireille ~/test> date | tee file1 file2
Thu Jun 10 11:10:34 CEST 2004

mireille ~/test> cat file1
Thu Jun 10 11:10:34 CEST 2004

mireille ~/test> cat file2
Thu Jun 10 11:10:34 CEST 2004

mireille ~/test> uptime | tee -a file2
 11:10:51 up 21 days, 21:21, 57 users,  load average: 0.04, 0.16, 0.26

mireille ~/test> cat file2
Thu Jun 10 11:10:34 CEST 2004
 11:10:51 up 21 days, 21:21, 57 users,  load average: 0.04, 0.16, 0.26

Prev: Simple redirections
Home
Next: Filters
Old version for comparison====== 5.3. Filters ======

When a program performs operations on input and writes the result to the standard output, it is called a filter. One of the most common uses of filters is to restructure output. We’ll discuss a couple of the most important filters below.

As we saw in Section 3.3.3.4, grep scans the output line per line, searching for matching patterns. All lines containing the pattern will be printed to standard output. This behavior can be reversed using the -v option.

Some examples: suppose we want to know which files in a certain directory have been modified in February:

jenny:~> ls -la | grep Feb

The grep command, like most commands, is case sensitive. Use the -i option to make no difference between upper and lower case. A lot of GNU extensions are available as well, such as --colour, which is helpful to highlight searchterms in long lines, and --after-context, which prints the number of lines after the last matching line. You can issue a recursive grep that searches all subdirectories of encountered directories using the -r option. As usual, options can be combined.

Regular expressions can be used to further detail the exact character matches you want to select out of all the input lines. The best way to start with regular expressions is indeed to read the grep documentation. An excellent chapter is included in the info grep page. If you do not have the info page for grep, or if it shows manpage on the top line, search for “REGULAR EXPRESSION” in the manpage. Another source is man regex. Since it would lead us too far discussing the ins and outs of regular expressions, it is strongly advised to start here if you want to know more about them.

Play around a bit with grep, it will be worth the trouble putting some time in this most basic but very powerful filtering command. The exercises at the end of this chapter will help you to get started, see Section 5.5.

The command sort arranges lines in alphabetical order by default:

thomas:~> cat people-I-like | sort
Auntie Emmy
Boyfriend
Dad
Grandma
Mum
My boss

But there are many more things sort can do. Looking at the file size, for instance. With this command, directory content is sorted smallest files first, biggest files last:

ls -la | sort -nk 5

• Old sort syntax
  • You might obtain the same result with ls -la | sort +4n, but this is an old form which does not comply with the current standards.

The sort command is also used in combination with the uniq program (or sort -u) to sort output and filter out double entries:

thomas:~> cat itemlist
1
4
2
5
34
567
432
567
34
555

thomas:~> sort itemlist | uniq
1
2
34
4
432
5
555
567

Prev: Advanced redirection features
Home
Next: Summary

In this chapter we learned how commands can be linked to each other, and how input from one command can be used as output for another command.

Input/output redirection is a common task on UNIX and Linux machines. This powerful mechanism allows flexible use of the building blocks UNIX is made of.

The most commonly used redirections are > and |.

Prev: Filters
Home
Next: Exercises

These exercises give more examples on how to combine commands. The main goal is to try and use the Enter key as little as possible.

All exercises are done using a normal user ID, so as to generate some errors. While you’re at it, don’t forget to read those man pages!

1. Use the cut command on the output of a long directory listing in order to display only the file permissions. Then pipe this output to sort and uniq to filter out any double lines. Then use the wc to count the different permission types in this directory.
2. Put the output of date in a file. Append the output of ls to this file. Send this file to your local mailbox (don’t specify anything <@domain>, just the user name will do). When using Bash, you will see a new mail notice upon success.
3. List the devices in /dev which are currently used by your username or UID. Pipe through less to view them properly.
   (Hints: man lsof and man id)
4. Issue the following commands as a non-privileged user. Determine standard input, output and error for each command.
   • cat nonexistentfile
   • file /sbin/ifconfig
   • grep root /etc/passwd /etc/nofiles > grepresults
   • /etc/init.d/sshd start > /var/tmp/output
   • /etc/init.d/crond start > /var/tmp/output 2>&1
   • Now check your results by issuing the commands again, now redirecting standardoutput to the file /var/tmp/output and standard error to the file /var/tmp/error.
5. How many processes are you currently running?
6. How many invisible files are in your home directory?
7. Use locate to find documentation about the kernel.
8. Find out which file contains the following entry:

   • root:x:0:0:root:/root:/bin/bash
       

9. And this one:

   •   
     system:		root
       

10. See what happens upon issuing this command:

    •   
        > time; date >> time; cat < time
      

11. What command would you use to check which script in /etc/init.d starts a given process?

Answer Key
Prev: Summary
Home
Next: Text editors