How to parse

COPY APPROPRIATE PARTS FROM HOW-TO-PARSE-MIDDLE-FRENCH

How to parse

How to train the parser
How to parse tagged files

How to train the parser

$PARSING = /home1/b/beatrice/parsing (on cluster) $UT = /home/migration/other/MIDENG/PPCEME/ut (EME, not MBE!)

construct training data
- strip comments using $UT/clean-up-mjc
- strip ID nodes using $UT/rm-id.prl
- check for illegal tags, either with CorpusSearch or by hand
  cat <training-data> | tr -s '\011' '\012 | tr -s ' ' '\012' | grep '(' | sort | uniq > TMP
copy training data to appropriate directory on cluster
- Schematically: $PARSING/DOMAIN/MODEL#
- Example: $PARSING/french/model3
add appropriate entry in $PARSING/Makefile
show queue with qstat (-f shows all jobs)
train parser
- cd $PARSING
- Schematically: submittrain2.sh DOMAIN MODEL#
- Example: submittrain2.sh classicalport 4
output of training is written to $PARSING/DOMAIN/MODEL#

How to parse tagged files

This section outlines the steps required to turn .lex files (or equivalent POS-tagged files) into .psd files compatible with CorpusDraw or the older Emacs-based correction interface.

$PARSING = /home1/b/beatrice/parsing (on cluster) $PPCEME = /home/migration/other/MIDENG/PPCEME $PPCMBE = /home/migration/other/MIDENG/PPCMBE $UT = /home/migration/other/MIDENG/PPCMBE/ut

deal with font encoding (use utf8-html if necessary)

check for illegal tags

cat <training-data> | tr ' ' '\012' | grep . | cut -d '/' -f 2 | tr '+' '\012' | sort | uniq > TMP
diff TMP $MF/training-tagger/LEGAL-TAGS | grep '<'

generate preparse files
- for PPCMBE, use $UT/make-preparse (assumes .lex extension; writes output to $PPCMBE/preparse)
- otherwise (DO NOT DELETE WITHOUT CHECKING THAT SOME STUFF IS HELPFUL)
  - resolve text queries and related comments
  - remove xml headers and footers
  - tag any other xml annotation as CODE
  - convert slashes in end tags (</, />) to double dollar signs
  - convert any parens to <paren> / <$$paren>
  - remove any incorrect tags with $UT/clean-up-lex
  - remove any trailing line-final spaces
  - separate tokens by blanklines, adding blank line at end of file if necessary
  - $UT/delimit-tokens-for-parser (surround tokens by parens)
  - if necessary, convert to word-per-line format with tr -s ' ' '\012'
  - each line should contain a slash: grep . $file | grep -v '/'
  - add last character of file if necessary (i.e. "COD" becomes "CODE", etc.)
  - $UT/change-tag-format-for-parser
sanity checks for .preparse files
- check for unbalanced parens with forward-sexp
- run $UT/sanity-check-preparse
  - grep -v ' '
  - grep -v '^([^ ]* ([^ ]*))$'
  - grep '<[^>]*[()]'
  - grep '[ ].*[ ]'
  - grep '[{}]' | grep -v 'CODE'
  - grep '[<>]' | grep -v 'CODE'
copy .preparse file(s) to appropriate run#/files directory on cluster (may involve moving files to subdirectory so as to make use of scp -r)
do not edit files on cluster with emacs - make changes on babel
add appropriate entry in $PARSING/Makefile
show queue with qstat
parse file(s)
- cd $PARSING
- Schematically: submittest.sh DOMAIN RUN#
- Example: submittest.sh classicalport 4
if necessary to delete job from queue, use qdel {queue-number}
correct null parses
- check whether null parses are represented by the preparse token (grep '^((') or by null (grep '^null$')
- if the latter, restore the text, either by hand (for the moment) or with the appropriate script (waiting for Seth to send it to me 7/3/08)
- otherwise, go on to next step
.../parsing/ut/fix-postparse-problems FILE.preparse.parsed > FILE.preparse.parsed.nn
- delete (CODE +unknown+
- add root node to fixed null parses
- switch order of words and tags in manually fixed null parses
if necessary, restore subtags
- the restore script currently handles only ASCII text. Therefore, preparse and parsed files containing accented characters need to be run through utf8-html.
- in parsing directory, restore subtags (note quotes and path information in arguments)
  - Schematically:
    make DOMAIN="dir" RUN="n" PREPARSE="path1" PARSED="path2" OUT="path3" restore
  - Example:
    make DOMAIN="cordial" RUN="1" PREPARSE="cordial/run1/TEST.preparse" PARSED="cordial/run1/TEST.preparse.parsed.nn" OUT="cordial/run1/TEST.preparse.parsed.nn.with.subtags" restore
- restore accented characters in output with html-utf8
copy parsed files (with null parses fixed and with subtags) to postparse directory on babel. The scripts there assume the extension .preparse.parsed.nn.
check for mismatched parens
$UT/make-psd-unfmt *.parsed.nn
pretty print according to our conventions
- cd $PPCEME/queries
- CS reformat.q ../../PPCMBE/postparse/*.psd.unfmt
in postparse directory, write .psd file to appropriate $PARSED directory and archive copy to appropriate $PARSED-ORIG directory
$UT/unfmt.fmt-to-psd *.fmt
check permissions on *.psd files in $PPCMBE/psd
if necessary, move .psd files to appropriate directory for further postprocessing (FIX-XXX or NP-PASS)
add dash tags with script (e.g. with $UT/postparse.prl)
if necesssary, regenerate IDs with re-ref