How to tag MBE files

This page gives a step-by-step overview over the training and tagging process for Modern British English. Details concerning training, running, and evaluating the tagger are available at How to train, test and run the fnTBL tagger.

$PPCMBE = /home/migration/other/MIDENG/PPCMBE

Constructing the training data
Training the tagger
Tagging new text

Constructing the training data

$PPCMBE/ut/sanity-check-lex *.lex
cat files into TRAINING.lex
$PPCMBE/ut/make-pos TRAINING.lex
$PPCMBE/ut/sanity-check-pos TRAINING.pos
search for illegal tags
$PPCMBE/ut/add-blanklines-after-period

The training data is ordinarily constructed from corrected .lex files in $PPCMBE/lex/DONE.

Check for formatting errors in the .lex files with


$PPCMBE/ut/sanity-check-lex

cat .lex files into a single .lex file (TRAINING.lex), reserving data for testing if desired.

Remove superfluous tags and comments with


$PPCMBE/ut/make-pos TRAINING.lex

The resulting training data are in $PPCMBE/pos/TRAINING.pos

Perform sanity checks on training data

Tags should be separated from the word by exactly one space; one word-tag pair per line.


$PPCMBE/ut/sanity-check-pos
egrep .  | egrep -v ' ' 
egrep .  | egrep '[ ].*[ ]'

Correct any formatting errors in the .lex files and reconstruct the training data until no more errors are found.

Check for illegal tags in training data.


cut -d ' ' -f 2 <training-data> | sort | uniq > TMP
diff TMP $PPCMBE/training-tagger/LEGAL-TAGS-ALL | grep '<'

Check that components of complex tags are themselves legal tags:


cut -d ' ' -f 2 <training-data> | tr '+' '\012' | sort | uniq > TMP
diff TMP $PPCMBE/training-tagger/LEGAL-TAGS-SIMPLEX | grep '<'

Be sure to perform both sanity checks, since the simplex check finds errors that the complex one doesn't (e.g. VBP+PRO).

Once again, correct any errors in the .lex files and reconstruct the training data until no errors are found.

If the training data is being constructed from parsed rather than POS-tagged files, filter out any instances of ID.

Finally, separate sentences in the training data by blanklines with


$PPCMBE/ut/add-blanklines-after-period

Otherwise, the training will take forever.

Training the tagger

See the xwiki under Fntbl and Results for further details.

TRAINING_DIR = /home/beatrice/fnTBL/test-cases/pos-tagging
ORIG = /home/beatrice/fnTBL/test-cases/pos-tagging-orig
adjust to your local case

For convenience, make a backup copy of $TRAINING_DIR ($ORIG).
Clear $TRAINING_DIR.
Copy contents of $ORIG into $TRAINING_DIR.
Copy the training data to $TRAINING_DIR.

Train the tagger, using the following alias:

train-fntbl-eng <training-data>

this unpacks to:


cd /home/beatrice/fnTBL/test-cases/pos-tagging
../../exec/pos-train.prl -v -F tbl.lexical.train.params,tbl.context.pos.params -t N,NPR -r 0.7 -T 1,100 <training data>

Tagging new text

confirm that text encoding is UTF-8
empty out tok directory
run $PPCMBE/ut/sanity-check-text on the texts
in text directory, $PPCMBE/ut/make-tok *.txt
run $PPCMBE/ut/sanity-check-tok on the tokenized texts
if necessary, add blanklines after periods and such
if necessary, clear and update tagger directory (~/fnTBL/test-cases/pos-tagging)
in tok directory, $PPCMBE/ut/make-lex *.tok
mv tagged .lex files to appropriate directory