Scribd Logo
Upload

Welcome to Scribd!

  • 98 views

    Unix For Poets

    Uploaded by

    Casey J. Murphy
    The document discusses how to perform simple natural language processing tasks using Unix command line tools. It introduces tools like grep, sort, uniq, tr, wc, sed, cat and echo. It provides exercises on word counting, sorting words, extracting bigrams and working with parts of speech tagged text. The key idea is that simple Unix tools can be combined through piping to perform more complex tasks like tokenization, sorting, counting and extracting n-grams from text.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Unix For Poets

    Uploaded by

    Casey J. Murphy
    98 views27 pages
    The document discusses how to perform simple natural language processing tasks using Unix command line tools. It introduces tools like grep, sort, uniq, tr, wc, sed, cat and echo. It provides exercises on word counting, sorting words, extracting bigrams and working with parts of speech tagged text. The key idea is that simple Unix tools can be combined through piping to perform more complex tasks like tokenization, sorting, counting and extracting n-grams from text.

    Original Description:

    Unix for poets

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document discusses how to perform simple natural language processing tasks using Unix command line tools. It introduces tools like grep, sort, uniq, tr, wc, sed, cat and echo. It provides exercises on word counting, sorting words, extracting bigrams and working with parts of speech tagged text. The key idea is that simple Unix tools can be combined through piping to perform more complex tasks like tokenization, sorting, counting and extracting n-grams from text.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    98 views27 pages

    Unix For Poets

    Uploaded by

    Casey J. Murphy
    The document discusses how to perform simple natural language processing tasks using Unix command line tools. It introduces tools like grep, sort, uniq, tr, wc, sed, cat and echo. It provides exercises on word counting, sorting words, extracting bigrams and working with parts of speech tagged text. The key idea is that simple Unix tools can be combined through piping to perform more complex tasks like tokenization, sorting, counting and extracting n-grams from text.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 27

    CS

    124/LINGUIST 180
    From Languages to
    Informa<on

    Unix for Poets (in 2013)


    Christopher Manning
    Stanford University

    Christopher Manning

    Unix for Poets


    (based on Ken Churchs presenta<on)

    Text is available like never before


    The Web
    Dic=onaries, corpora, email, etc.
    Billions and billions of words

    What can we do with it all?


    It is beEer to do something simple, than nothing at all.
    You can do simple things from a Unix command-line
    DIY is more sa=sfying than begging for help

    Christopher Manning

    Exercises to be addressed
    1. Count words in a text
    2. Sort a list of words in various ways
    1. ascii order
    2. rhyming order

    3. Extract useful info from a dic=onary


    4. Compute ngram sta=s=cs
    5. Work with parts of speech in tagged text
    3

    Christopher Manning

    Tools
    grep: search for a paEern
    (regular expression)
    sort
    uniq c (count duplicates)
    tr (translate characters)
    wc (word or line count)
    sed (edit string -- replacement)
    cat (send le(s) in stream)
    echo (send text in stream)
    4

    cut (columns in tab-separated


    les)
    paste (paste columns)
    head
    tail
    rev (reverse lines)
    comm
    join
    shuf (shue lines of text)

    Christopher Manning

    Prerequisites

    ssh into a corn


    cp /afs/ir/class/cs124/nyt_200811.txt .
    man, e.g., man tr (shows command op=ons; not friendly)
    Input/output redirec=on:
    >
    <
    |

    CTRL-C
    5

    Christopher Manning

    Exercise 1: Count words in a text


    Input: text le (nyt_200811.txt)
    Output: list of words in the le with freq counts
    Algorithm
    1. Tokenize(tr)
    2. Sort(sort)
    3. Count duplicates (uniq c)

    Christopher Manning

    Solu<on to Exercise 1
    tr -sc A-Za-z \n < nyt_200811.txt | sort | uniq -c

    25476 a
    1271 A
    3 AA
    3 AAA
    1 Aalborg
    1 Aaliyah
    1 Aalto
    2 aardvark

    Christopher Manning

    Some of the output


    tr -sc A-Za-z \n <
    nyt_200811.txt | sort |
    uniq -c | head n 5
    25476 a
    1271 A
    3 AA
    3 AAA
    1 Aalborg
    8

    tr -sc A-Za-z \n <


    nyt_200811.txt | sort |
    uniq -c | head
    Gives you the rst 10 lines
    tail does the same with
    the end of the input
    (You can omit the -n but
    its discouraged.)

    Christopher Manning

    Extended Coun<ng Exercises


    1. Merge upper and lower case by downcasing everything

    Hint: Put in a second tr command

    2. How common are dierent sequences of vowels (e.g., ieu)

    Hint: Put in a second tr command

    Christopher Manning

    Sor<ng and reversing lines of text

    sort
    sort f
    sort n
    sort r
    sort nr

    Ignore case
    Numeric order
    Reverse sort
    Reverse numeric sort

    echo Hello | rev



    10

    Christopher Manning

    Coun<ng and sor<ng exercises


    Find the 50 most common words in the NYT
    Hint: Use sort a second =me, then head

    Find the words in the NYT that end in zz


    Hint: Look at the end of a list of reversed words

    11

    Christopher Manning

    Lesson
    Piping commands together can be simple yet powerful in Unix
    It gives exibility.

    Tradi=onal Unix philosophy: small tools that can be composed

    12

    Christopher Manning

    Bigrams = word pairs counts


    Algorithm
    1. tokenize by word
    2. print wordi and wordi +1 on the same line
    3. count

    13

    Christopher Manning

    Bigrams

    14

    tr -sc A-Za-z \n < nyt_200811.txt > nyt.words


    tail n +2 nyt.words > nyt.nextwords
    paste nyt.words nyt.nextwords > nyt.bigrams
    head n 5 nyt.bigrams
    KBR said
    said Friday
    Friday the
    the global
    global economic

    Christopher Manning

    Exercises
    Find the 10 most common bigrams
    (For you to look at:) What part-of-speech paEern are most of them?

    Find the 10 most common trigrams

    15

    Christopher Manning

    grep
    Grep nds paEerns specied as regular expressions
    grep rebuilt nyt_200811.txt
    Conn and Johnson, has been rebuilt, among the rst of the 222
    move into their rebuilt home, sleeping under the same roof for the
    the part of town that was wiped away and is being rebuilt. That is
    to laser trace what was there and rebuilt it with accuracy," she
    home - is expected to be rebuilt by spring. Braasch promises that a
    the anonymous places where the country will have to be rebuilt,
    "The party will not be rebuilt without moderates being a part of
    16

    Christopher Manning

    grep
    Grep nds paEerns specied as regular expressions
    globally search for regular expression and print

    Finding words ending in ing:


    grep ing$ nyt.words| sort | uniq -c

    17

    Christopher Manning

    grep

    grep is a lter you keep only some lines of the input


    grep gh
    keep lines containing gh
    grep con
    keep lines beginning with con
    grep ing$
    keep lines ending with ing
    grep v gh
    keep lines NOT containing gh

    grep -P
    Perl regular expressions (extended syntax)
    grep -P '^[A-Z]+$' nyt.words | sort | uniq c
    ALL UPPERCASE
    18

    Christopher Manning

    Coun<ng lines, words, characters


    wc nyt_200811.txt
    140000 1007597 6070784 nyt_200811.txt

    wc -l nyt.words
    1017618 nyt.words

    19

    Christopher Manning

    grep & wc exercises


    How many all uppercase words are there in this NYT le?
    How many 4-leEer words?
    How many dierent words are there with no vowels
    What subtypes do they belong to?

    How many 1 syllable words are there


    That is, ones with exactly one vowel
    Type/token dis=nc=on: dierent words (types) vs. instances (tokens)
    20

    Christopher Manning

    sed
    sed is a simple string (i.e., lines of a le) editor
    You can match lines of a le by regex or line numbers and make
    changes
    Not much used in 2013, but
    The general regex replace func=on s=ll comes in handy
    sed 's/George Bush/Dubya/' nyt_200811.txt | less
    21

    Christopher Manning

    sed exercises
    Count frequency of word ini=al consonant sequences
    Take tokenized words
    Delete the rst vowel through the end of the word
    Sort and count

    Count word nal consonant sequences

    22

    Christopher Manning

    awk
    Ken Churchs slides then describe awk, a simple programming
    language for short programs on data usually in elds
    I honestly dont think its worth learning awk in 2013
    BeEer to write liEle programs in your favorite scrip=ng
    language, be that Python, or Perl, or groovy, or

    23

    Christopher Manning

    shuf
    Randomly permutes (shues) the lines of a le
    Exercises
    Print 10 random word tokens from the NYT excerpt
    10 instances of words that appear, each word instance equally likely
    Print 10 random word types from the NYT excerpt
    10 dierent words that appear, each dierent word equally likely
    24

    Christopher Manning

    cut tab separated les


    cp /afs/ir/class/cs124/parses.conll .
    head n 5 parses.conll
    1 Inuen=al _ JJ JJ _ 2 amod _ _
    2 members _ NNS NNS _ 10 nsubj _ _
    3 of _ IN IN _ 2 prep _ _
    4 the _ DT DT _ 6 det _ _
    5 House _ NNP NNP _ 6 nn _ _

    25

    Christopher Manning

    cut tab separated les


    Frequency of dierent parts of speech:
    cut -f 4 parses.conll | sort | uniq -c | sort nr
    Get just words and their parts of speech:
    cut -f 2,4 parses.conll
    You can deal with comma separate les with: cut d,
    26

    Christopher Manning

    cut exercises
    How oen is that used as a determiner (DT) that man versus
    a complemen=zer (IN) I know that he is rich versus a rela=ve
    (WDT) The class that I love
    Hint: With grep P, you can use \t for a tab character

    What determiners occur in the data? What are the 5 most


    common?
    27

    You might also like