Dist Sem Fin

Distributional Semantic Models
Niloy Ganguly
CSE, IIT Kharagpur
October 9, 2017
Niloy Ganguly (IIT Kharagpur) Distributional Semantics - Part I October 9, 2017 1 / 40

Introduction
1, 3, 4, . . .
Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 2 / 24

Introduction
1, 3, 4, . . .
I, III, IV, . . .

Introduction
1, 3, 4, . . .
I, III, IV, . . .
What is Semantics?

Introduction
1, 3, 4, . . .
I, III, IV, . . .
What is Semantics?
The study of meaning: Relation between symbols and their denotata.

Introduction
1, 3, 4, . . .
I, III, IV, . . .
What is Semantics?
The study of meaning: Relation between symbols and their denotata.
John told Mary that the train moved out of the station at 3 oclock.

Conceptual Graph Representation
Finding the underlying functional relations among various entities and events
Sentence: John told Mary that the

train moved out of the station at 3
oclock.

Computational Semantics

The study of how to automate the process of constructing and reasoning with
meaning representations of natural language expressions.

Methods in Computational Semantics generally fall in two categories:

Formal Semantics: Construction of precise mathematical models of the
relations between expressions in a natural language and the world.


John chases a bat x[bat(x) chase(john, x)]


John chases a bat x[bat(x) chase(john, x)]
Distributional Semantics: The study of statistical patterns of human
word usage to extract semantics.

Distributional Hypothesis

Distributional Hypothesis: Basic Intuition

The meaning of a word is its use in language. (Wittgenstein,
1953)
You know a word by the company it keeps. (Firth, 1957)


1953)
Word meaning (whatever it might be) is reflected in linguistic distributions.


1953)

Words that occur in the same contexts tend to have similar
meanings. (Zellig Harris, 1968)


1953)

Words that occur in the same contexts tend to have similar
meanings. (Zellig Harris, 1968)
Semantically similar words tend to have similar distributional patterns.

Distributional Semantics: a linguistic perspective
If linguistics is to deal with meaning, it can only do so through

distributional analysis. (Zellig Harris)


If we consider words or morphemes A and B to be more

different in meaning than A and C, then we will often find that the
distributions of A and B are more different than the distributions of A
and C. In other words, difference in meaning correlates with
difference of distribution. (Zellig Harris, Distributional Structure)


If we consider words or morphemes A and B to be more

different in meaning than A and C, then we will often find that the
distributions of A and B are more different than the distributions of A
and C. In other words, difference in meaning correlates with
difference of distribution. (Zellig Harris, Distributional Structure)
Differential and not referential

Distributional Semantics: a cognitive perspective
Contextual representation
A words contextual representation is an abstract cognitive structure that
accumulates from encounters with the word in various linguistic contexts.

We learn new words based on contextual cues


He filled the wampimuk with the substance, passed it around and we all drunk
some.


He filled the wampimuk with the substance, passed it around and we all drunk
some.
We found a little wampimuk sleeping behind the tree.

Distributional Semantic Models (DSMs)
Computational models that build contextual semantic repesentations from

corpus data


corpus data
DSMs are models for semantic representations
I The semantic content is represented by a vector
I Vectors are obtained through the statistical analysis of the linguistic
contexts of a word


corpus data
DSMs are models for semantic representations
I The semantic content is represented by a vector
I Vectors are obtained through the statistical analysis of the linguistic
contexts of a word
Alternative names
I corpus-based semantics
I statistical semantics
I geometrical models of meaning
I vector semantics
I word space models

Distributional Semantics: The general intuition
Distributions are vectors in a multidimensional semantic space, that is,

objects with a magnitude and a direction.
The semantic space has dimensions which correspond to possible
contexts, as gathered from a given corpus.

Vector Space

Vector Space

Vector Space
In practice, many more dimensions are used.

cat = [...dog 0.8, eat 0.7, joke 0.01, mansion 0.2,...]

Word Space
Small Dataset
An automobile is a wheeled motor vehicle used for transporting passengers .
A car is a form of transport , usually with four wheels and the capacity to carry around
five passengers .
Transport for the London games is limited , with spectators strongly advised to avoid
the use of cars .
The London 2012 soccer tournament began yesterday , with plenty of goals in the
opening matches .
Giggs scored the first goal of the football tournament at Wembley , North London .
Bellamy was largely a passenger in the football match , playing no part in either goal .
Target words: hautomobile, car, soccer, footballi

Term vocabulary : hwheel, transport, passenger, tournament, London, goal,
matchi

Constructing Word spaces
Informal algorithm for constructing word spaces

Pick the words you are interested in: target words


Define a context window, number of words surrounding target word


I The context can in general be defined in terms of documents, paragraphs
or sentences.


or sentences.
Count number of times the target word co-occurs with the context words:
co-occurrence matrix


or sentences.
Count number of times the target word co-occurs with the context words:
co-occurrence matrix
Build vectors out of (a function of) these co-occurrence counts

Constructing Word spaces: distributional vectors
distributional matrix = targets X contexts
wheel transport passenger tournament London goal match

automobile 1 1 1 0 0 0 0
car 1 2 1 0 1 0 0
soccer 0 0 0 1 1 1 1
football 0 0 1 1 1 2 1

2.5
2 football (0,2)
1.5
goal
1 soccer (0,1)
0.5
automobile (1,0) car (2,0)

0
0 0.5 1 1.5 2 2.5
transport

Computing similarity
wheel transport passenger tournament London goal match

automobile 1 1 1 0 0 0 0
car 1 2 1 0 1 0 0
soccer 0 0 0 1 1 1 1
football 0 0 1 1 1 2 1
Using simple vector product

automobile . car = 4 car . soccer = 1
automobile . soccer = 0 car . football = 2
automobile . football = 1 soccer . football = 5

Vector Space Model without distributional similarity
Words are treated as atomic symbols

Vector Space Model without distributional similarity
Words are treated as atomic symbols
One-hot representation

Distributional Similarity Based Representations
You know a word by the company it keeps


These words will represent banking

Building a DSM step-by-step
The linguistic steps

Pre-process a corpus (to define targets and contexts)

Select the targets and the contexts

Building a DSM step-by-step
The linguistic steps

Pre-process a corpus (to define targets and contexts)

Select the targets and the contexts
The mathematical steps

Count the target-context co-occurrences

Weight the contexts (optional)

Build the distributional matrix

Reduce the matrix dimensions (optional)

Compute the vector distances on the (reduced) matrix

Many design choices

Many design choices
General Questions
How do the rows (words, ...) relate to each other?
How do the columns (contexts, documents, ...) relate to each other?

The parameter space
A number of parameters to be fixed

Which type of context?
Which weighting scheme?
Which similarity measure?
...
A specific parameter setting determines a particular type of DSM (e.g. LSA,
HAL, etc.)

Documents as context: Word document

Words as context: Word Word

Words as contexts
Parameters
Window size
Window shape - rectangular/triangular/other

Words as contexts
Parameters
Window size
Consider the following passage

Suspected communist rebels on 4 July 1989 killed Col. Herminio Taylo, police
chief of Makati, the Philippines major financial center, in an escalation of street
violence sweeping the Capitol area. The gunmen shouted references to the
rebel New Peoples Army. They fled in a commandeered passenger jeep. The
military says communist rebels have killed up to 65 soldiers and police in the
Capitol region since January.

Words as contexts
Parameters
Window size
5 words window (unfiltered): 2 words either side of the target word


Words as contexts
Parameters
Window size
5 words window (filtered): 2 words either side of the target word


Context weighting: documents as context

Indexing function F: Essential factors

Word frequency (fij ): How many times a word appears in the document?
F fij
Document length (|Di |): How many words appear in the document?
1
F |Di |
Document frequency (Nj ): Number of documents in which a word
appears. F N1
j

Indexing function F: Essential factors

Word frequency (fij ): How many times a word appears in the document?
F fij
Document length (|Di |): How many words appear in the document?
1
F |Di |
Document frequency (Nj ): Number of documents in which a word
appears. F N1
j
Indexing Weight: tf-Idf

fij log( NNj ) for each term, normalize the weight in a document with
respect to L2 -norm.

Dist Sem Fin

Uploaded by

Copyright:

Available Formats

Dist Sem Fin

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Dist Sem Fin

Uploaded by

Copyright:

Available Formats

Distributional Semantic Models

CSE, IIT Kharagpur

Niloy Ganguly (IIT Kharagpur) Distributional Semantics - Part I October 9, 2017 1 / 40

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 2 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 2 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 2 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 2 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 2 / 24

Sentence: John told Mary that the

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 3 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 4 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 4 / 24

Methods in Computational Semantics generally fall in two categories:

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 4 / 24

Methods in Computational Semantics generally fall in two categories:

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 4 / 24

Methods in Computational Semantics generally fall in two categories:

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 4 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 5 / 24

Distributional Hypothesis: Basic Intuition

You know a word by the company it keeps. (Firth, 1957)

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 5 / 24

Distributional Hypothesis: Basic Intuition

You know a word by the company it keeps. (Firth, 1957)

Word meaning (whatever it might be) is reflected in linguistic distributions.

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 5 / 24

Distributional Hypothesis: Basic Intuition

You know a word by the company it keeps. (Firth, 1957)

Word meaning (whatever it might be) is reflected in linguistic distributions.

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 5 / 24

Distributional Hypothesis: Basic Intuition

You know a word by the company it keeps. (Firth, 1957)

Word meaning (whatever it might be) is reflected in linguistic distributions.

Semantically similar words tend to have similar distributional patterns.

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 5 / 24

If linguistics is to deal with meaning, it can only do so through

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 6 / 24

If linguistics is to deal with meaning, it can only do so through

If we consider words or morphemes A and B to be more

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 6 / 24

If linguistics is to deal with meaning, it can only do so through

If we consider words or morphemes A and B to be more

Differential and not referential

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 6 / 24

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 7 / 24

We learn new words based on contextual cues

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 7 / 24

We learn new words based on contextual cues

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 7 / 24

We learn new words based on contextual cues

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 7 / 24

Computational models that build contextual semantic repesentations from

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 8 / 24

Computational models that build contextual semantic repesentations from

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 8 / 24

Computational models that build contextual semantic repesentations from

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 8 / 24

Distributions are vectors in a multidimensional semantic space, that is,

Pawan Goyal (IIT Kharagpur) Distributional Semantics October 4th, 2016 9 / 24