sed

, the Unix Stream EDitor

Editing standard input

sed has various commands to manipulate text input. substitute command is most commonly used, which will be briefly discussed in this chapter. It is used to replace matching text with something else. The syntax is s/REGEXP/REPLACEMENT/FLAGS where

• s stands for substitute command
• / is a delimiter character to separate various portions of the command
• REGEXP stands for regular expression
• REPLACEMENT specifies the replacement string
• FLAGS are options to change default behavior of the command

For now, it is enough to know that the s command is used for search and replace operations.

Some Simple Examples

For each input line, change only the first ',' to '-'

$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/'

Change all matches by adding the 'g' flag

$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/g'

Here sample input is created using printf command to showcase stream editing. By default, sed processes input line by line. To determine a line, sed uses the \n newline character.

Escaping Characters

There are two levels of interpretation here: the shell, and sed.

In the shell, everything between single quotes is interpreted literally, except for single quotes themselves. You can effectively have a single quote between single quotes by writing '\'' (close single quote, one literal single quote, open single quote).

Sed uses basic regular expressions. In a BRE, in order to have them treated literally, the characters $.*[\^ need to be quoted by preceding them by a backslash, except inside character sets ([…]). Letters, digits and (){}+?| must not be quoted (you can get away with quoting some of these in some implementations). The sequences \(, \), \n, and in some implementations \{, \}, \+, \?, \| and other backslash+alphanumerics have special meanings. You can get away with not quoting $^ in some positions in some implementations.

Furthermore, you need a backslash before / if it is to appear in the regex outside of bracket expressions. You can choose an alternative character as the delimiter by writing, e.g., s~/dir~/replacement~ or \~/dir~p; you'll need a backslash before the delimiter if you want to include it in the BRE. If you choose a character that has a special meaning in a BRE and you want to include it literally, you'll need three backslashes; I do not recommend this, as it may behave differently in some implementations.

In a nutshell, for sed 's/…/…/':

• Write the regex between single quotes.
• Use '\'' to end up with a single quote in the regex.
• Put a backslash before $.*/[\]^ and only those characters (but not inside bracket expressions). (Technically you shouldn't put a backslash before ] but I don't know of an implementation that treats ] and \] differently outside of bracket expressions.)
• Inside a bracket expression, for - to be treated literally, make sure it is first or last ([abc-] or [-abc], not [a-bc]).
• Inside a bracket expression, for ^ to be treated literally, make sure it is not first (use [abc^], not [^abc]).
• To include ] in the list of characters matched by a bracket expression, make it the first character (or first after ^ for a negated set): []abc] or [^]abc] (not [abc]] nor [abc\]]).

In the replacement text:

• & and \ need to be quoted by preceding them by a backslash, as do the delimiter (usually /) and newlines.
• \ followed by a digit has a special meaning. \ followed by a letter has a special meaning (special characters) in some implementations, and \ followed by some other character means \c or c depending on the implementation.
• With single quotes around the argument (sed 's/…/…/'), use '\'' to put a single quote in the replacement text.

If the regex or replacement text comes from a shell variable, remember that

• The regex is a BRE, not a literal string.
• In the regex, a newline needs to be expressed as \n (which will never match unless you have other sed code adding newline characters to the pattern space). But note that it won't work inside bracket expressions with some sed implementations.
• In the replacement text, &, \ and newlines need to be quoted.
• The delimiter needs to be quoted (but not inside bracket expressions).
• Use double quotes for interpolation: sed -e "s/$BRE/$REPL/".

$ echo 'a^c' | sed 's/^/b/'
ba^c

$ echo 'a^c' | sed 's/\x5e/b/'
ba^c

$ echo abc | sed 's/[a]/x/'
Xbc
$ echo abc | sed 's/\x5ba\x5d/x/'
Xbc

$ echo 'a^c' | sed 's/\^/b/'
abc

$ echo 'a^c' | sed 's/\\\x5e/b/'
a^c

Editing File Input

Although sed derives its name from stream editing, it is common to use sed for file editing. To do so, append one or more input filenames to the command. You can also specify stdin as a source by using - as the filename. By default, output will go to stdout and the input files will not be modified. In-place File Editing section will discuss how to apply the changes back to the source file(s).

Say file greeting.txt contains Hi there\nHave a nice day, then:

sed 's/day/weekend/' greeting.txt

produces

Hi there
Have a nice weekend

Mapping from Character Set to Character Set

When you need to map one or more characters with another set of corresponding characters, you can use the y command. Quoting from the manual:

y/src/dst/ Transliterate any characters in the pattern space which match any of the source-chars with the corresponding character in dest-chars.

Back-references and Subexpressions

back-references are regular expression commands which refer to a previous part of the matched regular expression. Back-references are specified with backslash and a single digit (e.g. '\1'). The part of the regular expression they refer to is called a subexpression, and is designated with parentheses.

Back-references and subexpressions are used in two cases: in the regular expression search pattern, and in the replacement part of the s command.

In a regular expression pattern, back-references are used to match the same content as a previously matched subexpression. In the following example, the subexpression is '.' - any single character (being surrounded by parentheses makes it a subexpression). The back-reference '\1' asks to match the same content (same character) as the sub-expression.

The command below matches words starting with any character, followed by the letter 'o', followed by the same character as the first.

$ sed -E -n '/^(.)o\1$/p' /usr/share/dict/words
bob
mom
non
pop
sos
tot
wow

Multiple subexpressions are automatically numbered from left-to-right. This command searches for 6-letter palindromes (the first three letters are 3 subexpressions, followed by 3 back-references in reverse order):

$ sed -E -n '/^(.)(.)(.)\3\2\1$/p' /usr/share/dict/words
redder

In the s command, back-references can be used in the replacement part to refer back to subexpressions in the regexp part.

The following example uses two subexpressions in the regular expression to match two space-separated words. The back-references in the replacement part prints the words in a different order:

$ echo "James Bond" | sed -E 's/(.*) (.*)/The name is \2, \1 \2./'
The name is Bond, James Bond.

When used with alternation, if the group does not participate in the match then the back-reference makes the whole match fail. For example, ‘a(.)|b\1’ will not match ‘ba’. When multiple regular expressions are given with -e or from a file (‘-f file’), back-references are local to each expression.

In-place File Editing*

This section will discuss how to write back the changes to the input file(s) itself using the -i command line option. This option can be configured to make changes to the input file(s) with or without creating a backup of original contents. When backups are needed, the original filename can get a prefix or a suffix or both. And the backups can be placed in the same directory or some other directory as needed.

sed -i.bkp 's/blue/green/' colors.txt

When an extension is provided as an argument to -i option, the original contents of the input file gets preserved as per the extension given. For example, if the input file is ip.txt and -i.orig is used, the backup file will be named as ip.txt.orig

sed -i.bkp 's/bad/good/' f1.txt f2.txt

sed -i'bkp.*' 's/green/yellow/' colors.txt

sed -i'backups/*' 's/good/nice/' f1.txt f2.txt

Selective Editing

By default, sed acts on the entire input content. Many a times, you only want to act upon specific portions of the input. To that end, sed has features to filter lines, similar to tools like grep, head and tail. sed can replicate most of grep's filtering features without too much fuss. And has features like line number based filtering, selecting lines between two patterns, relative addressing, etc which isn't possible with grep. If you are familiar with functional programming, you would have come across map, filter, reduce paradigm. A typical task with sed involves filtering a subset of input and then modifying (mapping) them. Sometimes, the subset is the entire input, as seen in the examples of previous subsections.

printf '1,2,3,4\na,b,c,d\n'
| sed '/2/ s/,/-/g'

printf 'sea\neat\ndrop\n'
| sed '/at/d'

$ printf 'sea\neat\ndrop\n'
| sed '/at/!d'

$ sed -n '/twice/p' programming_quotes.txt

sed -n 's/1/one/gp' programming_quotes.txt

printf 'sea\neat\ndrop\n'
| sed -n '/at/!p'

sed '/if/q' programming_quotes.txt

sed '/if/Q' programming_quotes.txt

tac programming_quotes.txt
| sed '/not/q' | tac

printf 'sea\neat\ndrop\n' | sed '/at/q2'

printf 'sea\neat\ndrop\n'
| sed -n -e 'p' -e 's/at/AT/p'

printf 'sea\neat\ndrop\n'
| sed -n 'p; s/at/AT/p'

$ sed -n '
> /not/ s/in/**/gp
> s/1/one/gp
> s/2/two/gp
> ' programming_quotes.txt

# same as: sed -n 'p; s/at/AT/p'
printf 'sea\neat\ndrop\n' |
sed '/at/{p; s/at/AT/}'

# spaces around {} is optional
printf 'gates\nnot\nused\n'
| sed '/e/{s/s/*/g; s/t/*/g}'

# same as: grep 'in' programming_quotes.txt | grep 'not'
sed -n '/in/{/not/p}' programming_quotes.txt

# same as: grep 'in' programming_quotes.txt
| grep 'not' | grep 'you'
$ sed -n '/in/{/not/{/you/p}}' programming_quotes.txt

# same as: grep 'not' programming_quotes.txt
| grep -v 'you'
sed -n '/not/{/you/!p}' programming_quotes.txt

xyz

xyz

xyz

xyz

# same as: sed -n '2~2p'
# n will replace pattern space with the next line of input
# as -n option is used, the replaced line won't be printed
# then the new line is printed as p command is used
seq 10 | sed -n 'n; p'

# if line contains 't', replace pattern space with the next line
# substitute all 't' with 'TTT' for the new line thus fetched
# note that 't' wasn't substituted in the line that got replaced
# replaced pattern space gets printed as -n option is NOT used here
printf 'gates\nnot\nused\n'
| sed '/t/{n; s/t/TTT/g}'

# append the next line to the pattern space
# and then replace newline character with colon character
seq 7 | sed 'N; s/\n/:/'

# then the substitution is performed
# on the two lines in the buffer
printf 'gates\nnot\nused\n'
| sed '/at/{N; s/s\nnot/d/}'

BRE/ERE Regular Expressions*

This section will cover Basic and Extended Regular Expressions as implemented in GNU sed. Though not strictly conforming to POSIX specifications, most of it is applicable to other sed implementations as well. Unless otherwise indicated, examples and descriptions will assume ASCII input.

By default, sed treats the search pattern as Basic Regular Expression (BRE). Using -E option will enable Extended Regular Expression (ERE). Older versions used -r for ERE, which can still be used, but -E is more portable. In GNU sed, BRE and ERE only differ in how metacharacters are applied, there's no difference in features.

Flags*

Just like options change the default behavior of shell commands, flags are used to change aspects of regular expressions. Some of the flags like g and p have been already discussed. For completeness, they will be discussed again in this section. In regular expression parlance, flags are also known as modifiers.

Shell substitutions*

So far, the sed commands have been constructed statically. All the details were known. For example, which line numbers to act upon, the search REGEXP, the replacement string and so on. When it comes to automation and scripting, you'd often need to construct commands dynamically based on user input, file contents, etc. And sometimes, output of a shell command is needed as part of the replacement string. This section will discuss how to incorporate shell variables and command output to compose a sed command dynamically. As mentioned before, bash is assumed to be the shell being used.

z, s and f Command Line Options*

This section covers the -z, -s and -f command line options. These come in handy for specific use cases. For example, the -z option helps to process input separated by ASCII NUL character and the -f option allows you to pass sed commands from a file.

append, change, insert*

These three commands come in handy for specific operations as suggested by their names. The substitute command could handle most of the features offered by these commands. But where applicable, these commands would be easier to use.

Adding content from file*

The previous section discussed how to use a, c and i commands to append, change or insert the given string for matching address. Any \ in the string argument is treated according to sed escape sequence rules and it cannot contain literal newline character.

The r and R commands allow to use file contents as the source string which is always treated literally and can contain newline characters. Thus, these two commands provide a robust way to add multiline text literally.

However, r and R provide only append functionality for matching address. Other sed features will be used to show examples for c and i variations.

Control structures*

sed supports two types of branching commands that helps to construct control structures. These commands (and other advanced features not discussed here) allow you to emulate a wide range of features that are common in programming languages. This section will show basic examples and you'll find some more use cases in a later section.

Gotchas and Tricks*