Prepare text for aligner

• convert Word file to UTF8-encoded text file
  • There are various file formats - notably, for our purposes, Word and plain text. Documents in Word format contain weird invisible control characters and can't be the input to the kinds of programs that we use, so we need to convert them to plain text format.
  • Piain text files in turn can be encoded in various ways - in particular, two Unicode encodings: UTF8 (the one that we use for the historical corpora) and the more recent UTF16. I'm not sure how well UTF16 encoding interacts with the various pieces of software that we use, so the instructions below are for converting Word format files to UTF8-encoded text files.
  • The only option that Word provides to save files in Unicode encoding is UTF16, so conversion is a two-step process.
  • In Word, open file (e.g. interview22.doc) and save as a UTF16-encoded text file (e.g. interview22-utf16.txt).
  • In TextEdit, set "Open and save" preferences to UTF16 and UTF8, respectively.
  • In TextEdit, open the UTF16-encoded file and save with UTF8 encoding (e.g. interview22-utf8.txt).
  • It's important to understand that there are two notions of text file. In general usage, a text file consists of running text without linguistic annotation. But as just mentioned, the term "text file" can also refer to a data format. In this second sense, POS-tagged and parsed files are plain text files (if neither the basic text nor the annotation contains weird control characters).

    By convention, I use .txt for unannotated text files, and other extensions like .pos, .lex, .psd and the like for annotated files. But the different extensions and the different linguistic content of these various files don't affect or change their data format.

• From here on in, use emacs
• Split file into appropriate parts (say, by tape and side)
• Save header information in a separate file (e.g. interview22Tape1Side1.header).
• Some files have footer information. It looks to me like it's fine to save the information along with the header information.
• Normalize formatting of interview text
  • In the commands below, the sequence "| ..." is intended to be replaced by further commands in a Linux/Unix pipe. For example, the full form of (1) might be any of the commands in (2). The first two write to the terminal; the third writes to a file.

    (1)		... | tr -s '\015' '\012' | ...
    (2)	a.	cat interview22Tape1Side1.txt | tr -s '\015' '\012' | tr -s '\011' ' '
    	b.	cat interview22Tape1Side1.txt | tr -s '\015' '\012' | tr -s '\011' ' ' | sort | uniq
    	c.	cat interview22Tape1Side1.txt | tr -s '\015' '\012' | tr -s '\011' ' ' > FOO

    For further examples, see the commands in the remainder of this section that don't contain "| ...".

  • Word's end-of-line character (ASCII character 015) causes problems and needs to be replaced by the end-of-line character used by Linux/Unix (ASCII character 012). The -s switch "squeezes" consecutive instances of the first argument of tr into one before transforming them into the second argument.

    cat interview22Tape1Side1-utf8.txt | tr -s '\015' '\012' | ...

  • Unwanted wrapping lines can be eliminated using fmt, adjusting the number of characters per line (here, 60) as desired. Note the absence of the -s switch in this case.

    cat interview.txt | tr '\015' '\012' | fmt -w 60 | ...

  • Unwanted blanklines can be eliminated using the "squeeze" capability of tr without specifying a replacement string.

    ... | tr -s '\012' | ...

  • Tabs (ASCII character 011) should generally be replaced by space.

    ... | tr -s '\011' ' ' | ...

  • It's useful if each orthographic word is on its own line.

    ... | tr -s ' ' '\012' | ...

  • The text needs to be reviewed for comments like GRAM, LEX, PRON, etc. and meta-information that is not part of the transcript. In order to find extraneous material, you can run the various commands listed till now and sort and uniq the result into a file for review.

    ... | sort | uniq > FOO

  • Use caution when using a pipe containing the following commands so that you don't delete punctuation in the reference version of your file.

    If it is useful in connection with the last step, it is possible to ignore punctuation (by converting it to newline). Period does not need to be "escaped" as long as it is in enclosed in brackets and/or quotes.

    ... | tr -s '[.,;:?!]' '\012' | ...

    It's possible to convert space and punctuation to newline in one fell swoop by adding space to the alternatives in brackets.

    ... | tr -s '[ .,;:?!]' '\012' | ...

  • When using grep -v, remember that grep works on lines. So make sure that the input to it is in word-per-line format. Otherwise, you'll end up deleting not, say, GRAM, but the entire line containing GRAM.

    Clean up the file manually and/or with "grep -v" commands modeled on the following. For safety, the caret and dollar sign "anchor" the expressions to the beginning and end of line, respectively.

    cat interview22-utf8.txt | tr -s '\015' '\012' | tr -s '\011' ' ' | tr -s ' ' '\012' | grep -v '^GRAM$' | grep -v '^LEX$' | grep -v '^PHON$' | grep -v '^[IB]:$' > interview22.txt

  • It's probably worth repeating the last two steps (changing filenames as required).

Align transcript with tape

• Run through Jiahong's aligner

Prepare text for tagger

If you want to change all instances in one session (you don't have to, though), you can grep for "UT" and edit the path in all the files that come up as a result of grepping. You can also edit the path on a one-by-one basis, though, for just the scripts that you are using at the moment.

The path variable is near the top of the scripts, and the relevant line looks something like this.

UT="/home/beatrice/appalachian/ut"

UT="/Users/christina_tortora/appalachian/trail/ut"

• DONE - Preliminaries
  • Set up lex mode, etc. (see Dropbox/appalachian/install-lex-mode/README)
  • Edit fix-contractions to use french conventions for contractions
  • Include A prefix in fix-contractions

• Pre-process text grid (= corrected output of jiahong's aligner, which is missing punctuation)

  • To run commands on more than one file, replace "FILE" by an appropriate regular expression, such as asterisk

  • Extract text grids

    • Is the text in all caps?

      grep '[A-Z][A-Z][A-Z]' FILE.TextGrid

    • If yes, use

      $UT/grid-to-text-with-case-conversion FILE.TextGrid

    • If no, use

      $UT/grid-to-text FILE.TextGrid

  • Tokenize resulting text

    $UT/make-tok FILE.txt

    The make-tok script assumes that the input file has a .txt extension and generates output with the same filename, but with a .tok extension.

    The forward slash in end tags is replaced by a double dollar sign. This is to avoid confusion with the forward slash that delimits POS tags in the tagged version.

    The pipe symbol that indicates variants (as in there | they) used to be replaced in the .tok files by "XOR". This is no longer necessary, since the text grids are no longer being compared to the original transcripts and there is no need to generate a diff file, where the pipe character has a meaning of its own.

  • Proofread .tok file

    • Add punctuation

    • Add blanklines between sentence tokens

    • Fix uppercase/lowercase if necessary

  • Run sanity check on output

    $UT/sanity-check-tok FILE.tok

    You can zero in on or ignore particular error types with commands like the following:

    $UT/sanity-check-tok *.tok | grep -v '-'
    $UT/sanity-check-tok *.tok | grep -v '-' | grep -v 'Mr.' | sort | uniq

  • Fix any errors that show up in the output of sanity-check-tok in the .tok file
    • Abbreviations like Mr. are safe to disregard if you recognize them as standard
    • Hyphens are probably safe to disregard.
    • Single quotes may or may not reflect tokenization errors. They are used for quotations inside quotations (in which case they are fine), but they may also reflect improperly handled contractions.

POS tag .tok file

• if necessary, add blanklines after periods and other token-final punctuation (this should have already been done by hand, though)

• if necessary, clear and update tagger directory (~/fnTBL/test-cases/pos-tagging)

• in tok directory, $UT/make-lex *.tok

• mv tagged .lex files to appropriate directory