Instructions for running the Bikel parser on the cluster

Preparing the data
Training the parser
Testing
Evaluating
Miscellaneous

Preparing the data (back to top)

Construct training, testing, and evaluation data
- If necessary, concatenate individual text samples into a single file
- If necessary, deal with font encoding
- Check for comments and blurb and remove if necessary
```
        grep '<\+' {training-data}
        grep '\+>' {training-data}
        
```
- Remove IDs and various other minor replacements (see Preprocessing for parser 1 for details)
```
        preprocessForParser2 {training-data}
        
```
- Run a trivial query to check for problems like (X (CODE {foo})).
- Check for illegal tags. Use either CorpusSearch's make_tag_list utility (see get-tags.q) or a pipe like the following (\011 = tab; \012 = newline)
```
        cat {training-data} | tr -s '\011' '\012' | tr -s ' ' '\012' | grep '(' | sort | uniq > TMP
        
```
Run input through 15-20 iterations of raisePunc.q and several iterations of flattenConj1.q. (Use ls -l to tell when to stop.)
Divide data into training, testing, and evaluation data (80%, 10%, 10%)
First line(s) of file should contain description and properties of data

Copy the training, testing, and evaluation data from babel to the appropriate directory on nlpgrid. For example (assuming $TO as defined):

    $TO = /home1/b/beatrice/parser-results/better/corpus-data/1

    cd $TO

    scp beatrice@babel.ling.upenn.edu:/home/migration/other/MIDENG/PPCMBE/parsingExperiments/training-plain.psd .
    scp beatrice@babel.ling.upenn.edu:/home/migration/other/MIDENG/PPCMBE/parsingExperiments/testing-plain.psd .
    scp beatrice@babel.ling.upenn.edu:/home/migration/other/MIDENG/PPCMBE/parsingExperiments/eval-plain.psd .

In $TO, create three directories
```
    mkdir train test eval
    
```
The following commands are all invoked from the parser-config directory.
```
    cd /home1/b/beatrice/parser-config
    
```
Prepare training data with a command modeled on the following, yielding $OUTDIR/trees.txt (and other assorted files)
The command expects input with wrapper parens.
The file Makefile09 referenced below is analogous to Makefile, but using the old 0.9.9c-modified version of the parser.
```
    make -f Makefile09 \
    DOMAIN=better \
    MODEL=1 \
    INFILE=/home1/b/beatrice/parser-results/better/corpus-data/1/trainingData-0324.psd \
    OUTDIR=/home1/b/beatrice/parser-results/better/corpus-data/1/train/ \
    preparedata
    
```

Repeat for testing data

    make -f Makefile09 \
    DOMAIN=better \
    MODEL=1 \
    INFILE=/home1/b/beatrice/parser-results/better/corpus-data/1/testingData-0324.psd \
    OUTDIR=/home1/b/beatrice/parser-results/better/corpus-data/1/test/ \
    preparedata

Create parser input from test data, yielding $OUTDIR/trees.txt.preparse.

    make -f Makefile09 \
    DOMAIN=better \
    MODEL=1 \
    FROM=/home1/b/beatrice/parser-results/better/corpus-data/1/test/trees.txt \
    maketest

Splitting is deprecated since it is known to eat sentences in midfile
Split up test file (here, into 10 parts) for parallel parsing on cluster, yielding trees.txt.preparse.part1, ..., trees.txt.preparse.parse10
```
    python ./parser-config/split-file.py \
    /home1/b/beatrice/parser-results/better/corpus-data/1/test/ \
    trees.txt.preparse \
    10
    
```

Training the parser (back to top)

Include the following in .cshrc file on cluster (DONE)

    source ~sgeadmin/sge/nlp/common/settings.csh

Edit directory variables in Makefile09 if necessary.
Add appropriate line to Makefile09, modeled on the following.
DOMAIN is the directory where the files are set up (here, ~/parser-results/better)
MODEL is the subdirectory of DOMAIN indicating the model number (here, 1).
Be sure that the MODEL directory has a subdirectory model (e.g. parser-results/better/1/model)
BETTER_CORPUS_LOC is the subdirectory of DOMAIN with the training and testing data (here, .../better/corpus-data).
```
    ifeq ($(DOMAIN)$(MODEL),better1)
    TRAINING_FILE=${BETTER_CORPUS_LOC}/1/train/trees.txt
    TRAINING_SETTINGS_FILE=${DBPARSER_MYEXT_LOC}/settings/ppceme3R1.properties
    endif
    
```

Check training command (using -n option). Note the final 2 on train2.

    make -n -f Makefile09 DOMAIN=better MODEL=1 train2

Train on cluster with the following command, which runs exactly the above command on a cluster node. The two arguments are the DOMAIN and the MODEL, respectively.
```
    ./submittrain2better.sh better 1
    
```

Checking with qstat -f should give something like this:

    /mnt/castor/seas_home/b/beatrice/parser-config> qstat
    job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
    -----------------------------------------------------------------------------------------------------------------
       6227 0.55500 run_job.sh yugu         r     03/30/2011 11:28:51 default@nlpgrid01.seas.upenn.e     1        
       6230 0.00000 model1-bet beatrice     qw    04/01/2011 11:26:12  1

Delete jobs with qdel.
Ignore the fact that model-written.txt is empty.

Testing (back to top)

Add appropriate line to Makefile09, modeled on the following. Note line specifying GOLD_FILE.

   ifeq ($(DOMAIN)$(RUN),better1)
   MODEL=1
   TEST_FILE_ORIG=${BETTER_CORPUS_LOC}/1/test/trees.txt.preparse
   TESTING_SETTINGS_FILE=${DBPARSER_MYEXT_LOC}/settings/ppceme3R2.properties
   GOLD_FILE=$(BETTER_CORPUS_LOC)/1/test/trees.txt
   endif

Copy the input file to a run directory. For example:

    cd $BETTER/0
    mkdir runm0-4R2
    cd runm0-4R2
    cp $BETTER/corpus-data/0/test/trees.txt.preparse

or (with split input files)

    cp $BETTER/corpus-data/0/test/*preparse.part* .

cd ~/parser-config

Before running the next command, run the -n variant of something like this (see also below)

    make -n -f Makefile09 \
    DOMAIN=better \
    MODEL=1 \
    RUN=2 \
    TEST_FILE_BASE=trees.txt.preparse.part1 \
    testnocopy

Parsing with the cluster (temperamental)
- Edit the following command (the arguments are DOMAIN, MODEL, RUN, and INPUT_FILENAME, respectively).
```
        submit-all-better.sh
       
```
- Run submit-all-better.sh.
- If necessary, parse each part separately with the relevant line in submit-all-better.sh. For example (again, the arguments are DOMAIN, MODEL, RUN, and INPUT_FILENAME, respectively):
```
        ./submittestbetter.sh better 1 2 trees.txt.preparse.part1
        
```

Parsing without the cluster

    make -f Makefile09 DOMAIN=better MODEL=1 RUN=2 \
    TEST_FILE_BASE=trees.txt.preparse.part1 testnocopy

    make -f Makefile09 DOMAIN=better MODEL=1 RUN=2 \
    TEST_FILE_BASE=trees.txt.preparse.part2 testnocopy

    ...

    make -f Makefile09 DOMAIN=better MODEL=1 RUN=2 \
    TEST_FILE_BASE=trees.txt.preparse.part10 testnocopy

Commands like those above can of course be collected in a shell script.

Checking with qstat -f should give something like this:

    job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
    -----------------------------------------------------------------------------------------------------------------
       6219 0.55500 run2       skulick      r     03/27/2011 16:38:36 default@nlpgrid05.seas.upenn.e     1

If the file has been split, reassemble
- Edit ~/parser-config/reassemble.sh
- From the appropriate run directory, run reassemble.sh to yield trees.txt.preparse.parsed

The following command is useful for output with many null parses (it runs on an entire directory)

    make -f Makefile09 DOMAIN=better RUN=1 \
    INDIR=/mnt/castor/seas_home/b/beatrice/parser-results/better/1/run1 \
    conversion4

Various other postparse processing
- correct various postparse problems with .../parsing/ut/fix-postparse-problems FILE.preparse.parsed > FILE.preparse.parsed.nn
  - delete (CODE +unknown+)
  - fix (CODE (CODE ))
  - add root node to fixed null parses
  - switch order of words and tags in any manually fixed null parses
- copy parsed files (with null parses fixed and with subtags) to postparse directory on babel. The scripts there assume the extension .preparse.parsed.nn.
- check for unbalanced parens with forward-sexp
- ../ut/make-psd-unfmt *.parsed.nn
- pretty print using CS ../queries/reformat.q *.psd.unfmt
- if necessary, add-blurb
- $UT/fmt-to-psd *.psd.unfmt.fmt writes .psd file to appropriate $PARSED directory (currently, in-progress-ottawa) and archives copy to appropriate $PARSED-ORIG directory
- if necesssary, regenerate IDs with re-ref
- run appropriate queries to fix clitics and so on

Evaluating (back to top)

The directory structure and files are assumed to be

PARSER_RESULTS_LOC
- DOMAIN
  - MODEL
    - runXXX ("run" is omitted in the Makefile entry)
      - The eval file is assumed to have the extension .parsed, which is omitted in the Makefile09 entry.
        So: if the actual file for evaluation is
        DOMAIN/MODEL/RUN/anotherName.parsed
        the Makefile09 entry will be
        DOMAIN/MODEL/test/anotherName
        Example:
        berkeley/0/runm0/enhancedtest.parsed
        is referenced as
        berkeley/0/test/enhancedtest
    - test
      - The gold file and the eval file need not share the same name.
        gold file = DOMAIN/MODEL/test/someName
        Example: ~/berkeley/0/test/trees.txt
    - train

Check matching parens

Check for stray tags

    grep ' [A-Z]\+[ )]' {parsed-file} | grep -v ' [AIO])'

Fix any nulls

    Bikel:    null → ( (CODE NULL))
    Berkeley: (()) → ( (CODE NULL))

Use CorpusSearch queries to normalize eval file (w.r.t. location of punctuation and difficult conjunction strutures).
- CorpusSearch expects wrapper parens around each token.
```
    ~/ut/add-wrapper-parens {parsed-file} > {userRedirectsOutput}
    
```
- Run output through raisePunc.q and flattenConj1.q enough times so that input and output match (i.e. until diff reports no difference).
```
    CS3 ~/queries/raisePunc.q {parsed-file}
    CS3 ~/queries/raisePunc.q {parsed-file}.out
    CS3 ~/queries/raisePunc.q {parsed-file}.out.out
    and so on
    
```
- The remaining programs don't expect wrapper parens (or ID nodes), and they expect one-token-per-line format.
```
     ~/ut/rm-wrapper-parens {parsed-file} > {userRedirectsOutput}
    
```
In Makefile09, check directory variable (PARSER_RESULTS_LOC)
From the parser-config directory, run eval (no need to specify "gold" file from command line, as it is specified in the test entry in Makefile09)
```
    make -f Makefile09 DOMAIN=better RUN=2 eval
    
```
The above command will crash, but it creates the file trees.txt.preparse.parsed.scorable
Find ill-formed tokens like (LS XV (LS XV)) with the following command
```
    make -f Makefile09 DOMAIN=better RUN=2 parsedebug
    
```
Evaluation results have the extension .scorable.pd.results

Miscellaneous (back to top)

To pretty-print parser out, say something like
```
    CS3 ~/queries/reformat.q *.psd
    
```
CS3 expects its input files to have wrapper parens, which you may have to add.
```
    ~/ut/add-wrapper-parens {parsed-file} > {userRedirectsOutput}
    
```
The output of the reformatting query has the same name as its input, but with an .fmt extension.
To remove ID notes, say something like
```
    ~/ut/rm-id.prl {parsed-file} > {userRedirectsOutput}
    
```
rm-id.prl expects wrapper parens around tokens and blank lines between tokens.

To add or revise ID notes, say something like

    ~/ut/re-ref *.psd

re-ref expects wrapper parens around tokens and blank lines between tokens.

The output of the script has the same name as its input, but with an .with.id extension.