Developing Linguistic Corpora: a Guide to Good Practice

Adding Linguistic Annotation

Geoffrey Leech, Lancaster University

© Geoffrey Leech 2004

1. What is corpus annotation?

Corpus annotation is the practice of adding interpretative linguistic information to a

corpus. For example, one common type of annotation is the addition of tags, or labels,
indicating the word class to which words in a text belong. This is so-called part-of-
speech tagging (or POS tagging), and can be useful, for example, in distinguishing
words which have the same spelling, but different meanings or pronunciation. If a word
in a text is spelt present, it may be a noun (= 'gift'), a verb (= 'give someone a present')
or an adjective (= 'not absent'). The meanings of these same-looking words are very
different, and also there is a difference of pronunciation, since the verb present has
stress on the final syllable. Using one simple method of representing the POS tags —
attaching tags to words by an underscore symbol — these three words may be annotated
as follows:

present_NN1 (singular common noun)

present_VVB (base form of a lexical verb)
present_JJ (general adjective)

Some people (notably John Sinclair — see chapter 1) prefer not to engage in corpus
annotation: for them, the unannotated corpus is the 'pure' corpus they want to investigate
— the corpus without adulteration with information which is suspect, possibly
reflecting the predilections, or even the errors, of the annotator. For others, annotation is
a means to make a corpus much more useful — an enrichment of the original raw
corpus. From this perspective, probably a majority view, adding annotation to a corpus
is giving 'added value', which can be used for research by the individual or team that
carried out the annotation, but which can also be passed on to others who may find it
useful for their own purposes. For example, POS-tagged versions of major English
language corpora such as the Brown Corpus, the LOB Corpus and the British National
Corpus have been distributed widely throughout the world for those who would like to
make use of the tagging, as well as of the original 'raw' corpus. In this chapter, I will
assume that such annotation is a benefit, so long as it is done well, with an eye to the
standards that ought to apply to such work.

2. What different kinds of annotation are there?

Apart from part-of-speech (POS) tagging, there are other types of annotation,
corresponding to different levels of linguistic analysis of a corpus or text — for
example:

phonetic annotation

e.g. adding information about how a word in a spoken corpus was pronounced. prosodic
annotation — again in a spoken corpus — adding information about prosodic features
such as stress, intonation and pauses. syntactic annotation — e.g. adding information
about how a given sentence is parsed, in terms of syntactic analysis into such units such
phrases and clauses

semantic annotation

e.g. adding information about the semantic category of words — the noun cricket as a
term for a sport and as a term for an insect belong to different semantic categories,
although there is no difference in spelling or pronunciation.

pragmatic annotation

e.g. adding information about the kinds of speech act (or dialogue act) that occur in a
spoken dialogue — thus the utterance okay on different occasions may be an
acknowledgement, a request for feedback, an acceptance, or a pragmatic marker
initiating a new phase of discussion.

discourse annotation

e.g. adding information about anaphoric links in a text, for example connecting the
pronoun them and its antecedent the horses in: I'll saddle the horses and bring them
round. [an example from the Brown corpus]

stylistic annotation

e.g. adding information about speech and thought presentation (direct speech, indirect
speech, free indirect thought, etc.)

lexical annotation

adding the identity of the lemma of each word form in a text — i.e. the base form of the
word, such as would occur as its headword in a dictionary (e.g. lying has the lemma
LIE).

(For further information on such kinds of annotation, see Garside et al. 1997.) In fact, it
is possible to think up untold kinds of annotation that might be useful for specific kinds
of research. One example is dysfluency annotation: those working on spoken data may
wish to annotate a corpus of spontaneous speech for dysfluencies such as false starts,
repeats, hesitations, etc. — see Lickley, no date). Another illustration comes from an
area of corpus research which has flourished in the last ten years: the creation and study
of learner corpora (Granger 1998). Such corpora, consisting of writing (or speech)
produced by learners of a second language, may be annotated with 'error tags' indicating
where the learner has produced errors, and what kinds of errors these are (Granger et al
2002).

3. Why annotate?

As I have already indicated, annotation is undertaken to give 'added value' to the corpus.
A glance at some of the advantages of an annotated corpus will help us to think about
the standards of good practice these corpora require.
Manual examination of a corpus

What has been built into the corpus in the form of annotations can also be extracted
from the corpus again, and used in various ways. For example, one of the main uses of
POS tagging is to enhance the use of a corpus in making dictionaries. Thus
lexicographers, searching through a corpus by means of a concordancer, will want to be
able to distinguish separate (verb) from separate (adjective), and if this distinction is
already signalled in the corpus by tags, the separation can be automatic, without the
painstaking search through hundreds or thousands of examples that might otherwise be
necessary. Equally, a grammarian wanting to examine the use of progressive aspect in
English (is working, has been eating, etc) can simply search, using appropriate search
software, for sequences of BE (any form of the lemma) followed — allowing for certain
possibilities of intervening words — by the ing-form of a verb.

Automatic analysis of a corpus

Similarly, if a corpus has been annotated in advance, this will help in many kinds of
automatic processing or analysis. For example, corpora which have been POS-tagged
can automatically yield frequency lists or frequency dictionaries with grammatical
classification. Such listings will treat leaves (verb) and leaves (noun) as different words,
to be listed and counted separately, as for most purposes they should be. Another
important case is automatic parsing, i.e. the automatic syntactic analysis of a text or a
corpus: the prior tagging of a text can be seen as a first stage of syntactic analysis from
which parsing can proceed with greater success. Thirdly, consider the case of speech
synthesis: if a text is to be read aloud by a speech synthesiser, as in the case of the
'talking books' service provided for the blind, the synthesiser needs to have the
information that a particular instance of sow is a noun (= female pig) rather than a verb
(as in to sow seeds), because this make a difference to the word's pronunciation.

Re-usability of annotations

Some people may say that the annotation of a corpus for the above cases is not needed,
automatic processing could include the analysis of such features as part of speech: it is
unnecessary thereafter to preserve a copy of the corpus with the built-in information
about word class. This argument may work for some cases, but generally the annotation
is far more useful if it is preserved for future use. The fact is that linguistic annotation
cannot be done accurately and automatically: because of the complex and ambiguous
nature of language, even a relatively simple annotation task such as POS-tagging can
only be done automatically with up to 95% to 98% accuracy. This is far from ideal, and
to obtain an optimally tagged corpus, it is necessary to undertake manual work, often on
a large scale. The automatically tagged corpus afterwards has to be post-edited by a
team of human beings, who may spend thousands of hours on it. The result of such
work, if it makes the corpus more useful, should be built into a tagged version of the
corpus, which can then be made available to any people who want to use the tagging as
a springboard for their own research. In practice, such corpora as the LOB Corpus and
the BNC Sampler Corpus have been manually post-edited and the tagging has been used
by thousands of people. The BNC itself — all 100 million words of it — has been
automatically tagged but has not been manually post-edited, as the expense of
undertaking this task would be prohibitive. But the percentage of error — 2% — is
small enough to be discounted for many purposes. So my conclusion is that — as long
as the annotation provided is a kind useful to many users — an annotated corpus gives
'value added' because it can be readily shared by others, apart from those who originally
added the annotation. In short, an annotated corpus is a sharable resource, an example of
the electronic resources increasingly relied on for research and study in the humanities
and social sciences.

Multi-functionality

If we take the re-usability argument one step further, we note that annotation often has
many different purposes or applications: it is multi-functional. This has already been
illustrated in the case of POS tagging: the same information about the grammatical class
of words can be used for lexicography, for parsing, for frequency lists, for speech
synthesis, and for many other applications. People who build corpora are familiar with
the idea that no one in their right mind would offer to predict the future uses of a corpus
— future uses are always more variable than the originator of the corpus could have
imagined! The same is true of an annotated corpus: the annotations themselves spark off
a whole new range of uses which would not have been practicable unless the corpus had
been annotated.

However, this multi-functionality argument does not always score points for annotated
corpora. There is a contrary argument that the annotations are more useful, the more
they are designed to be specific to a particular application.

4. Useful standards for corpus annotation

What I have said above about the usefulness of annotated corpora, of course, depends
crucially on whether the annotation has been well planned and well carried out. It is
important, then, to recommend a set of standards of good practice to be observed by
annotators wherever possible.

Annotations should be separable

The annotations are added as an 'optional extra' to the corpus. It should always be easy
to separate the annotations from the raw corpus, so that the raw corpus can be retrieved
exactly in the form it had before the annotations were added. This is common sense: not
all users will find the annotations useful, and annotation should never result in any loss
of information about the original corpus data.

Detailed and explicit documentation should be provided

Lou Burnard (in chapter 3) emphasises the need to provide adequate documentation
about the corpus and its constituent texts. For similar reasons, it is important to provide
explicit and detailed documentation about the annotations in an annotated corpus.
Documentation to be provided about annotations should include the following, so that
users will know precisely what they're getting:

How, where, when and by whom were the annotations applied?

Mention any computer tools used, and any phases of revision resulting in new releases,
etc.
What annotation scheme was applied?

An annotation scheme is an explanatory system supplying information about the

annotation practices followed, and the explicit interpretation, in terms of linguistic
terminology and analysis, for the annotation. This is very important — Section 6 below
will deal with annotation schemes.

What coding scheme was used for the annotations?

By coding scheme, I mean the set of symbolic conventions employed to represent the
annotations themselves, as distinct from the original corpus. Again, I will devote a
separate section to this (Section 5).

How good is the annotation?

It might be thought that annotators will always proclaim the excellence of their
annotations. However, although some aspects of 'goodness' or quality elude judgement,
others can be measured with a degree of objectivity: accuracy and consistency are two
such measures. Annotators should supply what information they can on the quality of
the annotation. (see further Section 8 below.)

Arguably, the annotation practices should be linguistically consensual

This and the following maxims are more open to debate. Any type of annotation
presupposes a typology — a system of classification — for the phenomena being
represented. But linguistics, like most academic disciplines, is sadly lacking in
agreement about the categories to be used in such description. Different terminologies
abound, and even the use of a single term, such as verb phrase, is notoriously a prey to
competing theories. Even an apparently simple matter, such as defining word classes
(POS), is open to considerable disagreement. Against this background, it might be
suggested that corpus annotation cannot be usefully attempted: there is no absolute
'God's truth' view of language or 'gold standard' annotation against which the decision to
call word x as noun and word y a verb can be measured.

However, looking at linguistics more carefully, we can usually observe a certain

consensus: examining a text, people can more or less agree which words are nouns,
verbs, and so on, although they may disagree on less clear cases. If this is reasonable,
then an annotation scheme can be based on a 'consensual' set of categories on which
people tend to agree. This is likely to be useful for other users and therefore to fit in
with the re-usability goal for annotated corpora. An annotation scheme can additionally
make explicit how the annotations apply to the 10% or so of less clear cases, so that
users will know how borderline phenomena are handled. Significantly, this consensual
approach to categories is found not only in annotated corpora, but also in another key
kind of linguistic resource — dictionaries. If, on the other hand, an annotator were to
use categories specific to a particular theory and out of line with other theories, the
annotated corpus would suffer in being less useful as a sharable resource.

Annotation practices should respect emergent de facto standards

This principle of good practice may be seen as complementary to the preceding one. By
de facto standards, I mean some kind of standardisation that has already begun to take
place, due to influential precedents or practical initiatives in the research community.
These contrast with de iure or 'God's truth' standards, which I have just argued do not
exist. 'God's truth' standards, if they existed, would be imposed from on high. De facto
standards, on the other hand, emerge (often gradually) from the research community in
a bottom-up manner.

De facto standards encapsulate what people have found to work in the past, which
argues that they should be adopted by people undertaking a new research project, to
support a growing consensus in the community. However, often a new project breaks
new ground, for example with a different kind of data, a different language, a different
purpose those of previous projects. It would clearly be a recipe for stagnation if we were
to coerce new projects into the following exactly the practices of earlier ones.
Nevertheless it makes sense for new projects to respect the outcomes of earlier projects,
and only to depart from their practices where this can be justified. In 8 below, I will
refer to some of the incipient standards for different kinds of annotation and mark-up.
These can only be presented tentatively, however, as the practice of corpus annotation is
continually evolving.

In the early 1990s, the European Union launched an initiative under the name of
EAGLES (Expert Advisory Groups on Language Engineering Standards) with the goal
of encouraging standardisation of practices for natural language processing in academia
and industry, particularly but not exclusively in the EU. One group of 'experts' set to
work on corpora, and from this and later initiatives there emerged various documents
specifying guidelines (or provisional standards) for corpus annotation. In the following
sections, I will refer to the EAGLES documents where appropriate.

5. The encoding of annotations

But before focussing on annotation schemes and the linguistic categories they
incorporate, it will be helpful to touch briefly on the encoding of annotations — that is,
the actual symbolic representations used. This means we are for the moment
concentrating on how annotations are outwardly manifested — for example, what you
see when you inspect a corpus file on your computer screen — rather than what their
meaning is, in linguistic terms.

As an example, I have already mentioned one very simple device, the underscore
symbol, to signal the attachment of a POS tag to a word, as in Paula_NP1. The
presentation of the tag itself may be complex or simple. Here, for convenience, the
category of 'singular proper noun' is represented by a sequence of three characters, N for
noun, P for proper (noun), and 1 for singular.

One basic requirement is that the POS tag (or any other annotation device) should be
unambiguous in representing what it stands for. Another requirement, useful for
everyday purposes such as reading a concordance on a screen, is brevity: the three
characters, in this case, concisely signal the three distinguishing grammatical features of
the NP1 category. A third requirement, more useful in some contexts than in others, is
that the annotation device should be transparent to the human reader rather than opaque.
The example NP1 is at least to some degree intelligible, and is less mystifying than it
would be if some arbitrary sequence of symbols, say Q!@, had been chosen.

The type of tag illustrated above originated with the earliest corpus to be POS-tagged
(in 1971), the Brown Corpus. More recently, since the early 1990s, there has been a far-
reaching trend to standardize the representation of all phenomena of a corpus, including
annotations, by the use of a standard mark-up language — normally one of the series of
related languages SGML, HTML, and XML (see Lou Burnard, chapter 3). One
advantage of using these languages for encoding features in a text is that they provide a
general means of interchange of documents, including corpora, between one user or
research site and another. In this sense, SGML/HTML/XML have developed into a
world-wide standard which can be applied to any language, to spoken as well as to
written language, and to languages of different historical periods. Furthermore, the use
of the mark-up language itself can be efficiently parsed or validated, enabling the
annotator to check whether there are any ill-formed traits in the markup, which would
signal errors or omissions. Yet another advantage is that, as time progresses, tools of
various kinds are being developed to facilitate the processing of texts encoded in these
languages. One example is the set of tools developed at the Human Communication
Research Centre, Edinburgh, for supporting linguistic annotation using XML (Carletta
et al. 2002).

However, one drawback of these mark-up languages is that they tend to be more
'verbose' than the earlier symbolic conventions used, for example, for the Brown and
LOB corpora. In this connection we can compare the LOB representation Paula_NP1
(Johansson 1986) with the SGML representation to be found in the BNC (first released
in 1995): <w NP1>Paula, or the even more verbose version if a closing tag is added, as
required by XML: <w type="NP1">Paula</w>. In practice, this verbosity can be
avoided by a conversion routine which could produce an output, if required, as simple
as the LOB one Paula_NP1. This, however, would require a further step of processing
which may not be easy to manage for the technically less adept user.

Another possible drawback of the SGML/XML type of encoding is that it requires a

high-resolution standard of validation which sorts ill with the immensely unpredictable
nature of a real-world corpus. This is a particular problem if that corpus contains
spontaneous spoken data and data from less 'orderly' varieties of written language —
e.g. mediaeval manuscripts, old printed editions, advertisements, handwritten personal
letters, collections of children's writing. Attempts have been made to make this type of
logical encoding more accessible, by relaxing standards of conformance. Hence there
has grown up a practice of encoding corpora using a so-called 'pseudo-SGML', which
has the outward characteristics of SGML, but is not subjected to the same rigorous
process of validation (so that errors of well-formedness may remain undetected).

Within the overall framework SGML, different co-existing encoding standards have
been proposed or implemented: notably, the CDIF standard used for the mark-up of the
BNC (see Burnard 1995) and the CES recommended as an EAGLES standard (Ide
1996). One further drawback of the SGML/XML approach to encoding is that it
assumes, by default, that annotation has a 'parsable' hierarchical tree structure, which
does not allow cross-cutting brackets as in <x ...> ... <y...> ... <x/> ... <y/>. Any corpus
of spoken data, in particular, is likely to contain such cross-bracketing, for example in
the cross-cutting of stretches of speech which need to be marked for different levels of
linguistic information — such phenomena as non-fluencies, interruptions, turn overlaps,
and grammatical structure are prone to cut across one another in complex ways.

This difficulty can be overcome within SGML/XML, although not without adding
considerably to the complexity of the mark-up — for example, by copious use of
pointer devices (in the BNC) or by the use of so-called stand-off annotation (Carletta et
al. 2002).

It is fair to say, in conclusion, that the triumph of the more advanced

SGML/HTML/XML style of encoding is in the long run assured. But because of the
difficulties I have mentioned, many people will find it easier meanwhile to follow the
lead of other well-known encoding schemes — such as the simpler styles of mark-up
associated with the Brown and ICE families of corpora, or with the CHILDES database
of child language data.

CHILDES ('child language data exchange system') is likely to be the first choice not
only for those working on child language corpora, but on related fields such as second
language acquisition and code-switching. As the name suggests, CHILDES is neither a
corpus nor a coding scheme in itself, but it provides both, operating as a service which
pools together the data of many researchers all over the world, using a common coding
and annotation schemes, and common software including annotation software.

6. Annotation manual

Why do we need an annotation manual? This document is needed to explain the

annotation scheme to the users of an annotated corpus. Typically such manuals originate
from sets of guidelines which evolve in the process of annotating a corpus — especially
if hand editing of the corpus has been undertaken. A most carefully worked-out
annotation scheme was published as a weighty book by Geoffrey Sampson (1995). This
explained in detail the parsing scheme of the SUSANNE corpus (a syntactically-
annotated part of the Brown corpus). Sampson made an interesting analogy between
developing an annotation scheme and laying down a legal system by the tradition of
common law — the 'case law' of annotation evolves, rather as the law evolves over
time, through the precedent of earlier cases and the setting of new precedents as need
arises.

Although annotation manuals often build up piecemeal in this way, for the present
purpose we should see them as completed documents intended for corpus users. They
can be thought of as consisting of two sections — (a) a list of annotation devices and (b)
a specification of annotation practices — which I will illustrate, as before, using the
familiar case of a POS tagging scheme (for an example, see Johansson, 1986, for the
LOB Corpus, or Sampson, (1995, Ch.3 for the SUSANNE Corpus).

A list of annotation devices with brief explanations

This list acts as a glossary — a convenient first port of call for people trying to make
sense of the annotations. For POS tagging, the first thing to list is the tagset — i.e., the
list of symbols used for representing different POS categories. Such tagsets vary in size,
from about 30 tags to about 270 tags. The tagset can be listed together with a simple
definition and exemplification of what the tag means:
NN1 singular common noun (e.g. book, girl)
NN2 plural common noun (e.g. books, girls)
NP1 singular proper noun (e.g. Susan, Cairo)
etc.

A specification of annotation practices

This gives an account of the various annotation decisions made in:

1. segmentation: e.g. assignment of POS tags assumes a prior segmentation of the

corpus into words. This may involve 'grey areas' such as how to deal with
hyphenated words, acronyms, enclitic forms such as the n't of don't.
2. embedding: e.g. in parsing, some units, such as words and phrases, may be
included in other units, such as clauses and sentences; certain embeddings,
however, may be disallowed. In effect, a grammar of the parsing scheme has to
be supplied. Even POS tagging has to involve some embedding when we come
to segment examples such as the New York-Los Angeles flight.
3. the rules or guidelines for assigning particular annotation devices to particular
stretches of text.

The last of these, (c), is the most important: the guidelines on how to annotate particular
pieces of text can be elaborated almost ad infinitum. Taking again the example of POS
tagging, consider what this means with a particular tag such as NP1 (singular proper
noun). In the automatic tagging process, a dictionary that matches words to tags can
make a large majority of such decisions without human intervention. But problems
arise, as always, with 'grey areas' that the manual must attempt to specify. For example,
should New York be tagged as one example of NP1 or two? Should the tag NP1 apply
to [the] Pope, [the] Renaissance, Auntie, Gold (in Gold Coast), Fifth (in Fifth Avenue),
T and S (in T S Eliot), Microsoft and Word in Microsoft Word? If not, what alternative
tags should be applied to these cases? The manual should if possible answer such
questions in a principled way, so that consistency of annotation practices between
different texts and different annotators can be ensured and verified. But inevitably some
purely arbitrary distinctions have to be made. Languages suffer to varying extents from
ambiguity of word classifications, and in a language like English, a considerable
percentage of words have to be tagged variably according to their context of occurrence.

Other languages have different problems: for example, in German the initial capital is
used for common nouns as well as a proper nouns, and cannot be used as a criterion for
NP1. In Chinese, there is no signal of proper noun status such as capital letters in
alphabetic languages. Indeed, more broadly considered, the whole classification of parts
of speech in the Western tradition is of doubtful validity for languages like Chinese.

7. Some 'provisional standards' of best practice for different linguistics levels

In this section I will briefly list and comment on some previous work in developing
provisional de facto standards (see 4 above) of good practice for different levels of
linguistic annotation. The main message here is that anyone starting to undertake
annotation of a corpus at a particular level should take notice of previous work which
might provide a model for new work. There are two caveats, however: (a) these are only
a few of the references that might be chased up, and (b) most of these references are for
English. If you are thinking of annotating a corpus of another language, especially one
which corpus linguistics has neglected up to now, it makes sense to hunt down any
work going forward on that language, or on a closely related language. For this purpose,
grammars, dictionaries and other linguistic publications on the language should not be
neglected, even if they belong to the pre-corpus age.

Part-of-speech (POS) tagging

The 'Brown Family' of corpora (consisting of the Brown Corpus, the LOB
Corpus, the Frown Corpus and the FLOB Corpus) makes use of a family of
similar tagging practices, originated at Brown University and further developed
at Lancaster. The two tagsets (C5 and C7) used for the tagging of the British
National Corpus are well known (see Garside et al. 1997: 254-260).
An EAGLES document which recommends flexible 'standard' guidelines for EU
languages is to be found in Leech and Wilson (1994), revised and abbreviated in
Leech and Wilson (1999).
Note that POS tagging schemes are often part of parsing schemes, to be
considered under the next heading.

Syntactic annotation

A well-developed parsing scheme already mentioned is that of the SUSANNE

Corpus, Sampson (1995).
The Penn Treebank and its accompanying parsing scheme has been the most
influential of constituent structure schemes for syntax. (see Marcus et al 1993)
Other schemes have adopted a dependency model rather than a constituent
structure model — particularly the Constraint Grammar model of Karlsson et al.
(1995).
Leech, Barnett and Kahrel (1995) is another EAGLES 'standards-setting'
document, this time focussing on guidelines for syntactic annotation. Because
there can be fundamentally different models of syntactic analysis, this document
is more tentative (even) than the Leech and Wilson one for POS tagging.

Prosodic annotation

The standard system for annotating prosody (stress, intonation, etc.) is ToBI (=
Tones and Break Indices), which comes with its own speech-processing
platform. Its phonological model originated with Pierrehumbert (1980). The
system is partially automated, but needs to be substantially adapted for fresh
languages and dialects.
ToBI is well supported by dedicated software and a committed research
community. On the other hand, it has met with criticism, and two alternative
annotation systems worth examining are INTSINT (see Hirst 1991) and TSM —
tonetic stress marks (see Knowles et al. 1996).
For a survey of prosodic annotation of dialogue, see Grice et al. (2000: 39-54).

Pragmatic/Discourse annotation

For corpus annotation, it is difficult to draw a line between pragmatics and discourse
analysis.
 An international Discourse Resource Initiative (DRI) came up with some
recommendations for the analysis of spoken discourse at the level of dialogue
acts (= speech acts) and at higher levels such as dialogue transactions,
constituting a kind of 'grammar' of discourse. These were set out in the DAMSL
manual (= Dialog Act Markup in Several Layers) (Allen and Core 1997).
Other influential schemes are those of TRAINS, VERBMOBIL, the Edinburgh
Map Task Corpus, SPAAC (Leech and Weisser 2003). These all focus on
practical task-oriented dialogue. One exceptional case is the Switchboard
DAMSL annotation project (Stolcke et al. 2000), applied to telephone
conversational data.
Discourse can also be analysed at the level of anaphoric relations (e.g. pronouns
and their antecedents — see Garside et al 1997:66-84).
A survey of pragmatic annotation is provided in Grice et al. (2000: 54-67).
A European project MATE (= Multi-level annotation, tools engineering) has
tackled the issue of standardization in developing tools for corpus annotation,
and more specifically for dialogue annotation, developing a workbench and an
evaluation of various schemes, investigating their applicability across languages
(http://mate.nis.sdu.dk/).

Other levels of annotation

There is less to say about other levels of annotation mentioned in 2 above, either
because they are less challenging or have been less subject to efforts of standardization.
Examples particularly worth notice are:

phonetic annotation

SAMPA (devised by Wells et al 1992) is a convenient way of representing phonetic

(IPA) symbols in 7-bit ASCII characters. It can be useful for any parts of spoken
transcriptions where pronunciation has to be represented — but it is now giving way to
Unicode.

stylistic annotation

Semino and Short (2003) have developed a detailed annotation scheme for modes of
speech and thought representation — one area of considerable interest in stylistics. This
has been applied to a varied corpus of literary and non-literary texts.

8. Evaluation of annotation: realism, accuracy and consistency

In section 4 I mentioned that the quality or 'goodness' of annotation was one important
— though rather unclear — criterion to be sought for in annotation. Reverting to the
POS-tagging example once again, we may distinguish two quite different ideas of
quality. The first refers to the linguistic realism of the categories. It would be possible
to invent tags which were easy to apply automatically with 100% accuracy — e.g. by
arbitrarily dividing a dictionary into 100 parts and assigning a set of 100 tags to words
in the dictionary according to their alphabetical order — but these tags would be useless
for any serious linguistic analysis. Hence we have to make sure that our tagset is well
designed to bring together in one category words which are likely to have psychological
and linguistic affinity, i.e. are similar in terms of the syntactic distribution, their
morphological form, and/or their semantic interpretation.

A second, less abstract, notion of quality refers not to the tagset, but to the accuracy and
consistency with which it is applied.

Accuracy refers to the percentage of words (i.e. word tokens) in a corpus which are
correctly tagged. Allowing for ambiguity in tag assignment, this is sometimes divided
into two categories — precision and recall — see van Halteren (1999: 81-86).

Recall is the extent to which all correct annotations are found in the output of the
tagger.
Precision is the extent to which incorrect annotations are rejected from the output.

The obvious question to ask here is: what is meant by 'correct'? The answer is:
'correctness' is defined by what the annotation scheme allows or disallows — and this is
an added reason why the annotation scheme has to be specific in detail, and has to
correspond as closely as possible with linguistic realities recognized as such..

For example, automatic taggers can achieve tagging as high as 98% correct. However,
this is not as good as it could be, so the automatic tagging is often followed by a post-
editing stage in which human analysts correct any mistakes in the automatic tagging, or
resolve any ambiguities.

The first question here is: is it possible for hand-editors to achieve 100% accuracy?
Most people will find this unlikely, because of the unpredictable peculiarities of
language that crop up in a corpus, and because of the failure of even the most detailed
annotation schemes to deal with all eventualities. Perhaps between 99% and 99.5%
accuracy might be the best that can be achieved, given that unclear and unprecedented
cases are bound to arise. Nevertheless, 99.5% accuracy achieved with the help of a
human post-editor would still be preferable to 96% or 97% as the result of just
automatic tagging. Accuracy is therefore one criterion of quality in POS-tagging, and
indeed in any annotation task.

A second question that may be asked is: how consistently has the annotation task been
performed? One way to test this in POS tagging is to have two human annotators post-
edit the same piece of automatically-tagged text, and to determine in what percentage of
cases they agree with one another. The more this consistency measure (called inter-
rater agreement) approaches 100%, the higher the quality of the annotation. (Accuracy
and consistency are obviously related: if both raters achieve 100% accuracy, it is
inevitable that they achieve 100% consistency.)

In the early days of POS-tagging evaluation, it was feared that up to 5% of words would
be so uncertain in their word class that a high degree of accuracy and of consistency
could not be achieved. However, this is too pessimistic: Baker (1997) and Voutilainen
and Järvinen (1995) have shown how scores not far short of 100% can be attained for
both measures.

A more sophisticated measure of inter-rater consistency is the so-called kappa

coefficient (K). Strictly speaking, it is not enough to compare the output of two manual
annotators by counting the percentage of cases where they agree or do not agree. This
ignores the fact the even if the raters assigned the tags totally by chance, in a certain
proportion of cases would be expected to be in agreement. This factor is built into the
kappa coefficient, which is defined as follows:

K = P(A) - P(E)
1 - P(E)

"where P(A) is the proportion of time that the coders agree and P(E) is the proportion of
times that we would expect them to agree by chance." (Carletta 1996: 4).

There is no doubt that annotation tends to be highly labour-intensive and time-

consuming to carry out well. This is why it is appropriate to admit, as a final
observation, that 'best practice' in corpus annotation is something we should all strive
for — but which perhaps few of us will achieve.

9. Getting down to the practical task of annotation

To conclude, it is useful to say something about the practicalities of corpus annotation.

Assume, say, that you have a text or a corpus you want to work on, and want to 'get the
tags into the text'.

It is not necessary to have special software. You can annotate the text using a
general-purpose text editor or word processor. But this means the job has to be
done by hand, which risks being slow and prone to error.
For some purposes, particularly if the corpus is large and is to be made available
for general use, it is important to have the annotation validated. That is, the
vocabulary of annotation is controlled and is allowed to occur only in
syntactically valid ways. A validating tool can be written from scratch, or can
use macros for word processors or editors.
If you decide to use XML-compliant annotation, this means that you have the
option to make use of the increasingly available XML editors. An XML editor,
in conjunction with a DTD or schema, can do the job of enforcing well-
formedness or validity without any programming of the software, although a
high degree of expertise with XML will come in useful.
Special tagging software has been developed for large projects — for example
the CLAWS tagger and Template Tagger used for the Brown Family or corpora
and the BNC. Such programs or packages can be licensed for your own
annotation work. (For CLAWS, see the UCREL website
http://www.comp.lancs.ac.uk/ucrel/.)
There are tagsets which come with specific software — e.g. the C5, C7 and C8
tagsets for CLAWS, and CHAT for the CHILDES system, which is the de facto
standard for language acquisition data.
There are more general architectures for handling texts, language data and
software systems for building and annotation corpora. The most prominent
example of this is GATE ('general architecture for text engineering'
http://gate.ac.uk) developed at the University of Sheffield.

Continue to Chapter Three: Metadata for corpus work

Return to the table of contents

© Geoffrey Leech 2004. The right of Geoffrey Leech to be identified as the Author of
this Work has been asserted by him in accordance with the Copyright, Designs and
Patents Act 1988.

All material supplied via the Arts and Humanities Data Service is protected by
copyright, and duplication or sale of all or any part of it is not permitted, except that
material may be duplicated by you for your personal research use or educational
purposes in electronic or print form. Permission for any other use must be obtained from
the Arts and Humanities Data Service.

Electronic or print copies may not be offered, whether for sale or otherwise, to any third
party.