How to tag text files - Middle French

This page gives a step-by-step overview over the training and tagging process for Middle French.

Constructing training data
Training the tagger
Tagging text files
Reconciling tagged and TEI-encoded files

Constructing training data

$MF = /home/migration/other/MIDENG/MIDDLE-FRENCH

$MF/ut/sanity-check-lex *.lex
review comments
cat files into TRAINING.lex
$MF/ut/make-pos TRAINING.lex
$MF/ut/sanity-check-pos TRAINING.pos
search for illegal tags
$MF/ut/add-blanklines-after-ponfp

The training data is ordinarily constructed from .lex files that have been corrected in Ottawa.

(In the exceptional case of commynes, the training data is constructed from a parsed file. sanity-check-lex can be adapted to this case; after the sanity check, the file needs to be converted from parsed to pos-tagged format by using some form of ref-to-pos.)

Check for formatting errors in the .lex files with

$MF/ut/sanity-check-lex

Review comments in the .lex files, making appropriate changes to them.

Add appropriate TEI comments to the .tok files corresponding to the .lex files.

Important: The updated .tok file will eventually have to be glued back together with its .header and .footer files to make an updated .xml file. Editing the .tok file is easier and safer in several respects than editing the .xml file. Because of its format, the .tok file is easier to read. Also, editing the .tok file saves having to retokenize the .xml file after every update. The updated .tok file will be the second argument of the script that reinserts TEI codes into pos-tagged files ($MF/reinsert-tei-codes).

Revise $MF/ut/make-tok if necessary.

cat .lex files into a single .lex file (TRAINING.lex), reserving data for testing if desired.

Remove superfluous tags and comments with

$MF/ut/make-pos TRAINING.lex

The resulting training data are in $MF/pos/TRAINING.pos

Perform sanity checks on the training data

Tags should be separated from the word by exactly one space; one word-tag pair per line.

$MF/ut/sanity-check-pos

egrep . <training-data> | egrep -v ' ' 
egrep . <training-data> | egrep '[ ].*[ ]'

Correct any formatting errors in the .lex files and reconstruct the training data until no more errors are found.
Check for illegal tags in the training data. For the moment, it looks like there are no complex tags in the GTRC tagging scheme. So the presence of the tr command should have no effect.
```
cut -d ' ' -f 2 <training-data> | tr '+' '\012' | sort | uniq > TMP
diff TMP $MF/training-tagger/LEGAL-TAGS | grep '<'
```
Once again, correct any errors in the .lex files and reconstruct the training data until no errors are found.
If the training data is being constructed from parsed rather than POS-tagged files, filter out any instances of ID.
Finally, separate sentences in the training data by blanklines with
```
$MF/ut/add-blanklines-after-ponfp
```
Otherwise, the training will take forever.
There needs to be a reference directory for the completed .lex and .xml files. Probably best in Ottawa.

Training the tagger

See the xwiki under Fntbl and Results for further details.

Clear the training directory ($HOME/fnTBL/test-cases/pos-tagging), copying the contents to the appropriate sister directory.
Copy the contents of ../pos-tagging-orig into the training directory.
Copy the training data to the training directory.

The tagger can be trained with the following alias:

train-fntbl-mf <training-data>

This unpacks to:

cd /home/beatrice/fnTBL/test-cases/pos-tagging

../../exec/pos-train.prl -F tbl.lexical.train.params,tbl.context.pos.params -r 0.3 -f 2 -t NCS,NPRS -T 2,2 <training-data>

Tagging new text

Retrieve text files (.xml) containing TEI comments.

Put a backup copy in

$MF/xml-ottawa-originals

Work with a copy in

$MF/xml-penn

If necessary, replace bad line breaks from Word.

cat $file | tr '\015' '\012'

The .xml files may contain corrections and notes that need to be revised since otherwise the TEI material would be treated as part of the text and annotated. Find such material using

$MF/ut/sanity-check-xml

sanity-check-xml also finds parenthesized words that are likely to be editorial emendations rather than part of the original text. Report these back to Ottawa.

Review make-tok for consistency with documentation.

If necessary (say, in texts with old punctuation), protect blanklines by replacing them with something like BLANKLINE

Split off TEI headers and footers from the .xml file with

$MF/ut/make-tok

The headers and footers are saved to .header and .footer files, which eventually will be glued back together with the corrected POS-tagged file. The main body of the text is written to a tokenized (.tok) file.

Delete ^M in the .tok file.

Run a sanity check on the .tok file.

$MF/ut/sanity-check-tok

Run the tagger and postprocess the output with

$MF/ut/make-lex

The postprocessing might be improved.

Do there need to be sanity checks?

Reintegrate xml tags using basically the same procedure as under Reconciling tagged and TEI-encoded files.
Tag reintegrated TEI tags as CODE (updated version of script should do this automatically).
Add blurb.
Send the resulting .lex file to Ottawa, keeping a backup copy.

Reconciling tagged and TEI-encoded files

check tagged file for comments, remaining portmanteau processing (e.g. l'on), etc.
process EOS?

prepare LHS

in golden directory, check for slash errors in pos-tagged file:

grep '//' FILE-golden.lex 

cat FILE-golden.lex | $MF/ut/clean-up-lex | tr ' ' '\012' | grep . | grep -v '/'

copy .lex file from golden directory

create a copy of the lex file without tags (= LHS)

cat FILE-golden.lex | $MF/ut/clean-up-lex | tr ' ' '\012' | grep . | cut -d '/' -f1 > LHS

prepare RHS
- copy .tok file
- check .tok file for extraneous stuff in comments with something like:
```
grep '>.+$' FILE.tok
```
- correct any errors in .tok file
- create a copy of the tok file without blank lines or xml tags (= RHS)
```
grep . FILE.tok | grep -v '<' > RHS
```
compare LHS and RHS and correct discrepancies until there are none
```
diffils LHS RHS
```

integrate lex and xml

make a clean copy of the lex file

cat FILE-golden.lex | $MF/ut/clean-up-lex | tr ' ' '\012' | grep . > FILE.1

make a clean copy of the tok file
```
grep . FILE.tok > FILE.2
```

in lex-with-xml directory

$MF/ut/reinsert-tei-codes.py FILE.1 ../FILE.2 > FILE.lex.xml

add back header and footer from pre and post directories