How to generate tagged files from parsed files

General notes:

1. reformat.q belongs in the directory where you store your CorpusSearch queries.

2. The remaining shell scripts mentioned below belong in the directory where you store shell scripts for manipulating corpus files. The path to this directory is referred to in the scripts by the path variable UT. Edit the path variable in the scripts as required.

3. The instructions below assume that you are processing files on a file-by-file basis. It is also possible to use the commands below on more than one file by replacing the variable FILE.psd. For instance, using a*.psd would process all files beginning with "a"; *.psd*, all files; etc.

4. Further, the instructions assume certain extensions. For instance, the tagged files that are generated from the parsed files with psd-to-pos have the extension .pos. The extensions can easily be modified if desired.

• Make a backup of the parsed file.

• Normalize the format of the parsed file. CS stands for whatever alias you invoke to run CorpusSearch.

  CS reformat.q FILE.psd

• Substitute the reformatted file for the old one.

  mv FILE.psd.fmt FILE.psd

• The last line of the file must be a blank line. Otherwise, the re-ref script in the next step won't generate an ID for the last token.

• Update the ID numbers of the tokens (in case tokens have been split or joined in the course of correcting the parsed file).

  re-ref FILE.psd

  The script keeps a backup copy (FILE.psd.old).

  The re-ref script assumes that the text contains page breaks of the following form..

  ( (CODE <pb%%%n="1"$$>))

  Some texts (for instance, Roland) contain no page breaks, but rather laisse breaks.

  ( (CODE <laisse="1">))

  For such texts, comment out the line for page breaks in the re-ref script and uncomment the appropriate line. Restore the script to the original form when done.

  Of course, you can also create several different variants of the re-ref script (re-ref-pb, re-ref-laisse, etc.) for the various sorts of files.

• Sanity check for ID's: The last ID number in the file should match the output of the following command:

  grep -c '^( (' FILE.psd

  More precisely, the last running number in the file should match the output of the grep command. All tokens are assigned a running number, but tokens that are exhaustively dominated by CODE don't get an ID node (since they would never be cited as examples). So if the last token in the file is exhaustively dominated by CODE, then the last ID number in the file won't match the output of the grep command. This isn't a problem, though, since it is easy to figure out what the last running number would be by looking at the last few tokens. For instance, in the following case, the grep command should return 7260.

  ( (IP-MAT (PP (P devant)
  	      (NP (DZ Nostre)
  		  (NCS Dame)
  		  (PP (P de@)
  		      (NP (D @l) (NPRS Val)))))
  	  (CODE )
  	  (VJ est)
  	  (NP-SBJ (DZ ses) (NCS ostex))
  	  (PP (Q tot)
  	      (P a)
  	      (NP (NCS estal)))
  	  (PONFP .))
    (ID 1170-YVAIN,207.7257))
  
  ( (CODE <$$p>))
  
  ( (CODE <$$div>))
  
  ( (CODE <$$back>))
  

• The parsed file is now ready to be posted.

• In order to generate a tagged file from the parsed file, run

  psd-to-pos FILE.psd

  The output is FILE.pos. As its name indicates, the psd-to-pos script deletes syntactic information, leaving only terminals and preterminals. In addition, the (tag word) format of the parsed file is switched to word/tag.