Contents

Substitute Command
– Search & Replace
– Line Ranges & Addressing
Pattern Description

– Anchors
– "Escaped" characters
or metacharacters
– Quantifiers, Greedy and Non-Greedy
– Character ranges
– Grouping and Backreferences
– Alternations
– Operator Precedence
Global Command
– Global search and execution
– Examples
Examples
– Tips & Techniques
– Creating Outline
– Working with Tables
Other Regexp Flavors
Links

Substitute Command

– Search & Replace

So, what can you do with regular expressions? The most common task is to make replacements in a text following some certain rules. For this tutorial you need to know VIM search and replace command (S&R)

. Here is an excerpt from VIM help:

Part of the command word enclosed in the "[" & "]" can be omitted.

– Range of Operation, Line Addressing and Marks

Some Vim commands can accept a line range in front of them. By specifying the line range you restrict the command execution to this particular part of text only. Line range consists of one or more line specifiers, separated with a comma or semicolon. You can also mark your current position in the text typing

, where "l" can be any letter, and use it later defining the line address.

If no line range is specified the command will operate on the current
line only.

Here are a few examples:

– from 10 to 20 line.

Each may be followed (several times) by "+" or "-"
and an optional number. This number is added or subtracted from the preceding
line number. If the number is omitted, 1 is used.

– all lines between Section 1 and Section
2, non-inclusively, i.e. the lines containing Section
1 and Section 2 will not be affected.

The

and

may
be followed by another address separated by a semicolon. A semicolon between
two search patterns tells Vim to find the location of the first pattern,
then start searching from that location for the second pattern.

– first find Section 1, then the first line
with Subsection, step one line down (beginning
of the range) and find the next line with Subsection,
step one line up (end of the range).

The next example shows how you can reuse you search pattern:

– this will search for the Section line
and yank (copy) one line after into the memory.

– and that will search for the next Section
line and put (paste) the saved text on the next line.

IV. Pattern Description

4.1 Anchors

Suppose you want to replace all occurrences of vi
with VIM. This can be easily done with

If you’ve tried this example then you, no doubt, noticed that VIM
replaced all occurrences of vi even if it’s
a part of the word (e.g. navigator). If we
want to be more specific and replace only whole words vi
then we need to correct our pattern. We may rewrite it by putting spaces
around vi:

But it will still miss vi followed by the
punctuation or at the end of the line/file. The right way is to put special
word boundary symbols "

" and "

"
around vi.

The beginning and the end of the line have their own special anchors
– "

" and "

", respectively.
So, for all vi only at the start of the line:

To match the lines where vi is the only
word:

Now suppose you want to replace not only all vi
but also Vi and VI.
There are several ways to do this:

• probably the simplest way is to put "i" – ignore case in
  a pattern

  1	%s:vi:VIM:gi

• define a class of characters. This is a sequence of characters enclosed
  by square brackets "[" and "]". It matches any character
  from this set. So

  1	:%s:[Vv]i:VIM:

  will match vi
  and Vi. More on character ranges in the
  following section.

4.2 "Escaped" characters or metacharacters

So far our pattern strings were constructed from normal or literal
text characters. The power of regexps is in the use of metacharacters.
These are types of characters which have special meaning inside the
search pattern. With a few exceptions these metacharacters are distinguished
by a "magic" backslash in front of them. The table below lists
some common VIM metacharacters.

So, to match a date like 09/01/2000 you can use (assuming you don’t use
"/" as a separator in the S&R)

To match 6 letter word starting with a capital letter

Obviously, it is not very convenient to write

for any
character in the pattern – what if you don’t know how many letters in
your word? This can be helped by introducing so-called quantifiers.

4.3 Quantifiers, Greedy and Non-Greedy

Using quantifiers you can set how many times certain part of you pattern
should repeat by putting the following after your pattern:

Now it’s much easier to define a pattern that matches a word of any
length

.

These quantifiers are greedy – that is your pattern will try to
match as much text as possible. Sometimes it presents a
problem. Let’s consider a typical example – define a pattern to match
delimited text, i.e. text enclosed in quotes, brackets, etc. Since we
don’t know what kind of text is inside the quotes we’ll use

But this pattern will match everything between the first "
and the last " in the following line:

This problem can be resolved by using non-greedy quantifiers:

Let’s use

in place of

in our pattern.
So, now

will match the first quoted text:

pattern is not without surprises. Look what will happen
to the following text after we apply:

Before:

After:

"As few as possible" applied here means zero character replacements.
However match does occur between characters! To explain this behavior
I quote Bram himself:

Matching zero characters is still a match. Thus it will replace
zero characters with a “_”. And then go on to the next position, where
it will match again.

It’s true that using “\{-}” is mostly useless. It works this way
to be consistent with “*”, which also matches zero characters. There are
more useless ones: “x\{-1,}” always matches one x. You could just use
“x”. More useful is something like “x\{70}”. The others are just consistent
behavior: …, “x\{-3,}”, “x\{-2,}”, “x\{-1,}.

– Bram

But what if we want to match only the second occurrence of quoted text?
Or we want to replace only a part of the quoted text keeping the rest
untouched? We will need grouping and backreferences. But
before let’s talk more about character ranges.

4.4 Character ranges

Typical character ranges:

will match any of the numbers inside the brackets.
The same range can be written as

, where dash indicates
a range of characters in ASCII order. Likewise, we can define the range
for all lowercase letters:

, for all letters:

,
letters and digits:

etc. Depending on your system
locale you can define range which will include characters like à,
Ö, ß and other non ASCII characters.

Note that the range represents just one character in the search
pattern, that is

and

are not the
same. Likewise the order (with a few exceptions) is not important:

and

are the same character ranges, while

and

are two different patterns. Watch what happens when
we apply

to the following text:

Before:

After:

and now:

Before:

After:

Sometimes it’s easier to define the characters you don’t want to match.
This is done by putting a negation sign

(caret)
as a first character of the range

/

/

– will match any character except capital letters. We can now
rewrite our pattern for quoted text using

/

Note: inside the [ ] all metacharacters behave like ordinary characters.
If you want to include "-" (dash) in your range put it first

– will match all digits and -. "^" will lose its special
meaning if it’s not the first character in the range.

Now, let’s have some real life example. Suppose you want to run a grammar
check on your file and find all places where new sentence does not start
with a capital letter. The pattern that will catch this:

– a period followed by one or more blanks and a lowercase word. We know
how to find an error, now let’s see how we can correct it. To do this
we need some ways to remember our matched pattern and recall it later.
That is exactly what backreferences are for.

4.5 Grouping and Backreferences

You can group parts of the pattern expression enclosing them with "

"
and "

" and refer to them inside the replacement
pattern by their special number

. Typical example
is swapping first two words of the line:

where

holds the first word,

– any number
of spaces or tabs in between and

– the second word. How
to decide what number holds what pair of

? – count opening
"

" from the left.

Replacement Part of :substitute

Replacement part of the S&R has its own special characters which
we are going to use to fix grammar:

Now the full S&R to correct non-capital words at the beginning of
the sentences looks like

We have corrected our grammar and as an extra job we replaced variable
number of spaces between punctuation and the first letter of the next
sentence with exactly two spaces.

4.6 Alternations

Using "

" you can combine several expressions
into one which matches any of its components. The first one matched will
be used.

will parse various mail headings and their contents into \1 and \2, respectively.
The thing to remember about VIM alternation that it is not greedy.
It won’t search for the longest possible match, it will use the first that
matched. That means that the order of the items in the alternation is important!

4.7 Regexp Operator Precedence

As in arithmetic expressions, regular expressions are executed in a certain
order of precedence. Here the table of precedence, from highest to lowest:

V. Global Command

5.1 Global search and execution

I want to introduce another quite useful and powerful Vim command which
we’re going to use later

The global commands work by first scanning through the [range]
of of the lines and marking each line where a match occurs. In a second
scan the [cmd] is executed for each marked line with its line number
prepended. If a line is changed or deleted its mark disappears. The default
for the [range] is the whole file.

Note: Ex commands are all commands you are entering on the Vim command
line like

etc.
Non-Ex commands (normal mode commands) can be also executed via

mechanism.

5.2 Examples

Some examples of

usage:

– delete all empty lines in a file

– reduce multiple blank lines to a single blank

– reverse the order of the lines starting from the line 10 up to the
line 20.

Here is a modified example from Walter
Zintz vi tutorial:

– in the text block marked by

and

find
all the lines starting with Error and copy
(append) them to "errors.txt" file. Note: . (current
line address) in front of the

is very important, omitting
it will cause

to write the whole file to "errors.txt"
for every Error line found.

You can give multiple commands after

using "|"
as a separator. If you want to use "|’ in an argument, precede it
with "\’. Another example from Zintz tutorial:

– will copy all Error line to the end of
the file and then make a substitution in the copied line. Without giving
the line address

will operate on the current line, which
is the newly copied line.

– here the order is reversed: first modify the string then copy to the
end.

VI. Examples

6.1 Tips and Techniques

A collection of some useful S&R tips:

(1) sent by Antonio Colombo:

"a simple regexp I use quite often to clean up a text: it
drops the blanks at the end of the line:"

or (to avoid acting on all lines):

6.2 Creating outline

For this example you need to know a bit of HTML. We want to make a table
of contents out of

and

headings, which
I will call majors and minors. HTML heading

is a text
enclosed by

tags as in

.

(1) First let’s make named anchors in all headings, i.e. put

around
all headings. The

" is a unique identifier
of this particular place in HTML document. The following S&R does
exactly this:

Explanation: the first pair of

saves the opening
tag (

or

) to the

, the second
pair saves all heading text before the closing tag, the third pair saves
the last word in the heading which we will later use for "anchor"
and the last pair saves the closing tag. The replacement is quite obvious
– we just reconstruct a new "named" heading using

and link tag

(2) Now let’s copy all headings to one place:

This command searches our file for the lines starting with

or

and copies them to the end of the file. Now
we have a bunch of lines like:

First, we want to convert all

to

in order to link table entries to their respective places in the text:

Second, we want our

entries look different from

.
Let’s define CSS classes "majorhead" and "minorhead"
and do the following:

Now our entries look like:

We no longer need

and

tags:

and replace closing tags with breaklines

6.3 Working with Tables

Quite often you have to work with a text organized in tables/columns.
Consider, for example, the following text

Suppose we want to change all "Europe" cells in the third column
to "Asia":

To swap the first and the last columns:

To be continued…

VII. Other Regexp Flavors

Here I would like to compare Vim’s regexp implementation with others,
in particular, Perl’s. You can’t talk about regular expressions without
mentioning Perl.

(with a help from Steve Kirkendall)
The main differences between Perl and Vim are:

• Perl doesn’t require backslashes before most of its operators. Personally,
  I think it makes regexps more readable – the less backlashes are there
  the better.
• Perl allows you to convert any quantifier into a non-greedy version
  by adding an extra ? after it. So *? is a non-greedy *.
• Perl supports a lots of weird options that can be appended to the
  regexp, or even embedded in it.
• You can also embed variable names in a Perl regular expression. Perl
  replaces the name with its value; this is called "variable interpolation".

VIII. Links

Read VIM documentation about pattern and searching. To get this type
":help pattern" in VIM normal mode.

There are currently two books on the market that deal with VIM regular
expressions:

• "Learning the
  vi Editor" by Linda Lamb and Arnold Robbins.
• "vi Improved – VIM" by Steve Oualline

Definitive reference on regular expressions is Jeffrey Friedl’s "Mastering
Regular Expressions" published by O’Reilly & Associates,
but it mostly deals with Perl regular expressions. O’Reilly has one of
the book chapters available online.