1

Some fundamental aspects

of information storage and retrieval

[email protected]
Vrije Universiteit Brussel
Pleinlaan 2, B-1050 Brussels, Belgium

Prepared for a tutorial at IIVO in Brussels, in September 2005,

organised by IIVO
for delegates from Shangai, China,
who are interested in the management of municipal archives

These slides should be available from the WWW site

http://www.vub.ac.be/BIBLIO/nieuwenhuysen/presentations/
note: BIBLIO and not biblio
 3
Text information
storage and retrieval
systems
- contents Databases and
computerized information
- summary retrieval !
- structure Knowledge organisation:
classifications and
- overview thesaurus systems
Pictures on computers
of this tutorial Disks for storage of
information
Compact Discs:
CD and DVD for storage of
information

4
-Interruptions
-Questions
-Remarks
-Discussions
are welcome
 5

Text information
storage and retrieval systems

Database systems:
definition

A database (management) system is a program or set of

programs, providing a means by which a user can easily
store and retrieve data in the form of databases.
 7

Information retrieval software:

related terms

Software for
information storage and retrieval
(ISR software)
Text(-oriented) database management systems
(Text-DBMS)
Text information management systems
(TIMS)
Document retrieval systems
Document management systems

Information retrieval software:

applications (Part 1)

Archives Archived documents

Documentation centres Documents
Libraries Books / Documents
Musea Objects / Books / ...
Medical files Patients histories
Marketing departments Clients / Potential clients
Schools Courses / Teachers
Bibliographic databases Publications / ...
 9

Information retrieval software:

applications (Part 2)

Meeting calendars Meetings = conferences

Product information Product descriptions
Laboratories Recipes
Personal documentation Documents
Patent office Patents
Co-operating Documents / Persons /
information networks Institutes / Events / ...
...

10

Text-information management
systems: characteristics and definition

The information in the database is text oriented.

Therefore, several features are required:
ability to store relatively long blocks of texts
ability to retrieve items in which specific words or terms
occur anywhere
 11

Text-information management:
from free-form to structure

Free form text information without structure

Text database with information structured

in files, records, fields, sub-fields,
with links/relations among records,...

12

Functions of
database management software

Input / edit using keyboard or batch input

Indexing of the database(s)
Browse / Search data from database
Output
(Sort / Display / Print to file / Print to paper)

+
Export / Import
 13

The various formats of records

in a database
Format to
Input format(s) Internal format facilitate
= for retrieval
Edit format(s) long term storage =
inverted file

Display formats
Display formats Format for
Display
for formats
output toto
Display
for formats
output
for output
display,for
printer, toto exchange/export purposes
display, outputfile
printer, file
display,
display,printer,
printer,file
file

14

Indexes in books and databases:

a comparison

Book
Index_term_1 page x1, y1, z1,...
Index_term_2 page x2, y2, z2,... Printed
...
Invisible
Database
Index_term_1 record nr. x1 / field type nr. x1 / field occurrence x1 / position x1
record nr. y1 / field type nr. y1 / field occurrence y1 / position y1
...
Index_term_2 record nr. x2 / field type nr. x2 / field occurrence x2 / position x2
record nr. y2 / field type nr. y2 / field occurrence y2 / position y2
...
...
 15

Output from a database

to various devices

to video display

to printer

to computer file
(printing to a file)
! #

16

Hierarchy
in the use of a database

Database
structure

Input / Editing

Searching / Output
 17

Security / privacy / protection

of databases

Password for searching

specific database(s) and / or fields and / or record
Password for editing
specific database(s) and / or fields and / or records
Password for changing
database structure
input and modification work sheets
sort and print formats of data in records
sort and print formats of records in a selection

18

Databases and computerized

information retrieval
 19

Information retrieval:
via a database to the user

Information
Information Linear file Inverted file
content
content
Database

Search engine

Search interface User

User

20

Information retrieval:
the basic processes in search systems

Information Text
problem documents

Representation Representation

Query Indexed documents

Evaluation
and Comparison
feedback
Retrieved, sorted documents
 21

Information retrieval systems:

many components make up a system

Any retrieval system is built up of many, more or less

independent components.

To increase the quality of the results,

these components can be modified
more or less independent of each other.

22

Information retrieval systems:

important components

the information content

system to describe formal aspects of information items
system to describe the subjects of information items
concrete descriptions of information items
= application of the used information description systems
information storage and retrieval computer program(s)
computer system used for retrieval
type of medium or information carrier used for distribution
 23

What determines the results of a

search in a retrieval system?

1. the information retrieval system

( = contents + computer system)

Result
Resultof
ofaasearch
search

2. the user of the retrieval system

and the search strategy applied to the system

24

A simple database architecture:

all records together form a database

The sliced bread architecture

the bread is a database
each slice of bread is a database record
there are no relations between slices / records
the retrieval system tries to offer the appropriate slices /
records to the user
 25

Layered structure
of a database

Database
(File)
+
Records in many systems:
relations / links
Fields between
records
Characters

26

Text retrieval and language:

an overview

Problems/difficulties related to language / terminology

occur

$
in the case of multi-linguality:
cross-language information retrieval;
that is when more than 1 language is used
in the contents of the searched database(s)
and/or in the subject descriptors of the searched
database(s) OR
in the search terms used in a query
even when only 1 language is applied
throughout the system
 27

Text retrieval and language:

a word is not a concept

Problem:
A word or phrase or term is not the same as a concept or
subject or topic.

Word
Concept
Word

28

Text retrieval and language:

a word is not a concept

So, to cover a concept in a search,

to increase the recall of a search,
the user of a retrieval system should consider an
expansion of the query;
that is:
the user should also include other words in the query to
cover the concept.

$
 29

Text retrieval and language:

a word is not a concept

synonyms!
(such as :
Latin names of species in biology besides the common
names,
scientific names besides common names of substances in
chemistry)

30

Text retrieval and language:

a word is not a concept

narrower terms, more specific terms

(such as particular brand names);
including terms with prefixes
(for instance: viruses, retroviruses, rotaviruses...)

$
 31

Text retrieval and language:

a word is not a concept: example

Example: Searching for the concept sea can or should

involve for instance the following words in a
Boolean OR-combination:
baltic OR bay OR bays OR coast OR coastal OR coastline
OR coasts OR cove OR coves OR gulf OR mangrove OR
mangroves OR marine OR mediterranean OR noordzee OR
noordzeekust OR noordzeekusten OR ocean OR oceanic OR
oceans OR pacific OR reef OR reefs OR saline-freshwater
interface OR sea OR seas OR seashore OR seawater OR
seawaters OR shore OR shores

32

?? Question ??

Which
Whichproblems
problemsin intext
textretrieval
retrieval
are
are illustrated by the followingsentences?
illustrated by the following sentences?

$
 33

Time flies like an arrow.

Fruit flies like a banana.

34

Time flies like an arrow.

Fruit flies like a banana.
 35

Time flies like an arrow.

Fruit flies like a banana.

OK!

36

Text retrieval and language:

ambiguity of meaning

Problem:
A word or phrase can have more than 1 meaning.
Ambiguity of the meaning of a word is a problem for
retrieval.
This decreases the precision of many searches.
The meaning can depend on the context.
The meaning may depend on the region where the term is
used.

$
Example 37

Text retrieval and language:

ambiguity of meaning

Example of a word:
Pascal the philosopher
Pascal the computer language

38

Text retrieval and language:

ambiguity of meaning

Example of sentences:
The banks of New Zealand flooded our mailboxes with
free account proposals.
The banks of New Zealand flooded with heavy rains
account for the economic loss.

$
 39

Text retrieval and language:

ambiguity of meaning

Problem:
Ambiguity of meaning
may be the cause of low precision.

Relevant concept
Word
Irrelevant concept
NOT wanted

40

Text retrieval and language:

ambiguity of meaning

Method to solve the problem

at the time of database production:
adding to each database record codes from a classification
system or terms from a thesaurus system,
and providing the user with knowledge about the system
used;
in some cases, this process is computerized
(completely automatic or with intellectual intervention);
 41

Text retrieval and language:

ambiguity of meaning

Method to solve the problem,

provided by the computerized retrieval system:
offering to the user a partly computerized access to the
subject description system and then linking to the database
for searching

42

Text retrieval and language:

ambiguity of meaning

searching normally (without added value), but adding

value by categorizing the retrieved items in the
presentation phase to assist in the disambiguation;
this feature is offered for instance by
the public access module of the book catalogue of the
library automation system VUBIS at VUB, Belgium,
when a searching items that were assigned a particular
keyword
 43

Text retrieval and language:

ambiguity of meaning

Natural language processing of the queries:

linguistic analysis to determine possible meanings of the
query, which includes disambiguation of words in their
context:
lexical analysis = at the level of the word
semantic analysis = at the level of the sentence
However, most queries are short and therefore it is difficult
to apply semantic analysis for disambiguation.

44

Text retrieval and language:

ambiguity of meaning

Natural language processing of the documents:

linguistic analysis to determine possible meanings of a
sentence, which includes disambiguation of words in their
context:
lexical analysis = at the level of the word
semantic analysis = at the level of the sentence
However, most retrieval systems do not apply this
complicated method.
 45

A word is not a concept

A concept is not a word

Word1 Concept1

Word2 Concept2

Word3 Concept3

The most simple relation

between words and concepts is NOT valid.

46

A word is not a concept

A concept is not a word
Word1 Relevant concept 1

Word2 Irrelevant concept 2

Word3 Irrelevant concept3

A concept cannot be covered by only 1 word or term;

this may be the cause of low recall of a search.
The meaning of many words is ambiguous;
this may be the cause of low precision of a search.
 47

Text retrieval and language:

relation with recall and precision

Recapitulating the two problems discussed, we can say that

Expansion of the query allows

to increase the
recall.
Disambiguation of the query allows
to increase the

$
precision.

48

A good text retrieval system solves

some problems due to language

accepts words / terms / phrases in the query of the user

maps the words to corresponding concepts
presents these concepts to the user
who can then select the appropriate, relevant concept
(disambiguation)
searches for this concept,
even in documents written in another language
presents the resulting, retrieved documents
in the language preferred by the user
 49

Enhanced text retrieval

using natural language processing
Information Text
problem documents

Representation Representation

Query Indexed documents

Evaluation Natural language processing of

and the documents AND of the query
feedback Comparison and matching of both

Retrieved, sorted documents

50

Text retrieval and language:

conclusions

The use of terms and language to retrieve information

from databases/collections/corpora causes many
problems.
These problems are not recognized or underestimated by
many users of search/retrieval systems
= The power of retrieval systems is overestimated by
many users.
Much research and development is still needed to enhance
text retrieval.
 51

Hints on how to use information

sources: Boolean combinations

Most text search systems understand the basic

Boolean operators:
OR
= obtain records that contain one or both
search terms
AND
= obtain records that contain both search
terms
NOT or ANDNOT or AND NOT
= exclude records that contain a search term

52

Hints on how to use information

sources: Boolean combinations

In the case of computer-based information sources, use

Boolean combinations of search terms when appropriate
and when possible.

term x1 term y1 term z1

OR OR OR
term x2 AND term y2 AND term z2 AND ...
OR OR OR
term x3 term y3 term z3
 53

Hints on how to use information

sources: Boolean queries

Most text search systems understand the basic Boolean

operators typed in capital characters:
OR
AND

So this leads us to queries like for instance

(word1 OR word2 OR word3 OR word4) AND (wordA
OR wordB OR wordC)

54

Knowledge organisation:
classifications and thesaurus systems
 55

Knowledge organisation:
introduction
To organise knowledge / documents / books / reports /
information / data / records / things / items / materials
for more efficient storage and retrieval, some related,
similar tools / systems / methods / approaches are used.
Often but not yet always, this process is assisted by a
computer system.
Good systems are expanded and updated when the need
arises.
The organization system applied should ideally be clearly
and immediately visible or even searchable on computer,
by the user of the materials.

56

Knowledge organisation:
relations between tools

Controlled
vocabularies

Thesauri

Ontologies / Topic maps

Examples 57

Classification systems:
introduction

Classification systems
present the subjects in a
logical order, usually going
from the more general to the
more specific.

Examples 58

Classification systems:
examples of universal systems
Universal means here: covering all subjects
Not just one but several competing systems exist.
Examples
Universal Decimal Classification = UDC
used mainly outside U.S.A.
Dewey Decimal Classification = DDC
used mainly in U.S.A.
Library of Congress Classification
used mainly in U.S.A.
...
 59

Thesaurus:
description

Thesaurus (contents) =
system to control a vocabulary
(= words and phrases + their relations)
+ the contents of this vocabulary

Thesaurus program =
program to create, manage, modify and/or search a
thesaurus using a computer

60

Thesaurus
relations
Term(s) with broader meaning

BT (= Broader Term)

RT (= Related Term) UF (= Use(d) For)

Other term(s) Term Synonym(s)

NT (= Narrower Term)

Term(s) with narrower meaning

 61

Thesaurus applications
related to information searching (1)

For producers of a database:

To find/choose index terms to add these to items in a
database, when terms are taken from a controlled
vocabulary to increase precision and recall in the searches
by users of the database.

62

Thesaurus applications
related to information searching (2)

For users of a database:

When the database to be searched is produced with added
descriptors (words and terms) that are taken from a
controlled list of approved, selected words and terms,
then the searcher can use some printed or computer-
based system first, to find more and correct suitable
words and terms that belong to that controlled list of
descriptors;
then, the searcher can use these descriptors (and only
these words or terms) in a database query.
 63

Thesaurus applications
related to information searching (3)

For users of a database:

When the database to be searched is NOT produced with
added descriptors (words and terms) that are taken from
a controlled list of words and terms, then the searcher can
use one or several thesaurus systems first, to find more
words and terms and more suitable words and terms;
then the searcher can use these found words and terms to
formulate a query for that database (to increase recall
and precision).

64

Thesaurus systems
that cover all subjects

General systems
Universal systems
Covering all subjects
Broad and shallow systems
Horizontal systems
Examples 65

Thesaurus systems
that cover all subjects: examples (1)

Library of Congress Subject Headings (LCSH)

thesaurus system built into word processing software

thesaurus system that runs on a pc

(independent of Internet)
see for instance http://www.wordweb.co.uk/free/

Examples 66

Thesaurus systems
that cover all subjects: examples (2)

thesaurus systems that can be used through the WWW

http://education.yahoo.com/reference/thesaurus/index.html
http://thesaurus.plumbdesign.com/
Example 67

General thesaurus system through the

WWW: screenshot sea

Example 68

General thesaurus system through the

WWW: screenshot ocean
 69

Thesaurus systems covering all

subjects: comments

An ideal, complete thesaurus that covers all subjects does

not exist.

70

Thesaurus systems focused on a

particular subject

Focused on a particular subject domain =

narrow and deep, vertical systems
Examples 71

Thesaurus systems focused on a

particular subject: examples

ERIC: education, information science,...

Psychological Abstracts / PsycInfo
Sociological Abstracts / SocioFile
INSPEC: physics, electronics, information technology
the Aquatic Sciences and Fisheries Information System
Medline (the Medical Subject Headings = MeSH)
Various thesaurus systems for art and architecture can be
found online:
http://www.getty.edu/research/tools/vocabulary/

Examples 72

Thesaurus systems focused on a

particular subject: examples

A database of thesaurus systems is accessible online

through http://www.taxonomywarehouse.com/
 73

Knowledge organization:
classifications versus thesauri
Classification
Good for placement of documents (because documents on
many related subjects can be kept together)
Not well suited for computer searching (too complicated)
Thesaurus
Not suited for placement of documents
(because documents with related subjects would NOT be
kept together)
Well suited for computer searching
(relatively simple alphabetic listing of keywords)

74

Pictures on computers

Raster graphics versus vector graphics

 75

Graphics formats:
bitmaps and vector graphics

Bitmap/raster graphics
used in programs for painting

Vector-based graphics
used in programs for drawing

76

Graphics formats: bitmaps

compared with vector graphics

Property Bitmap/Raster Vector graphics

Suited for painting + -

and retouching

Resolution adapted - +
to output medium

Size of files large small

Examples 77

Graphics formats
for bitmaps only
File name extension Origin

.BMP Microsoft bitmap

.GIF CompuServe / Unisys
.JPG JPEG
.PCX, .PCC Z-Soft Paintbrush
.PNG (PNG8, PNG24) W3C
.RLE Microsoft compressed bitmap
.TIF Aldus PageMaker
... ...

Examples 78

Graphics formats
for vector graphics (+ bitmaps)

File name extension Name / Origin

CDR Corel Draw!

CGM Graphics Metafile
EPS Encapsulated PostScript
IMG GEM
PPT or PPS Microsoft PowerPoint
SVG Scalable Vector Graphics
WPG WordPerfect Graphics
... ...
 79

Pictures on computers

Bitmap pictures

80

Graphics file formats for bitmap

pictures: a comparison
BMP GIF JPG PNG24 TIF
Maximum number RLE
of colours in one image 256*256*256 = 256 256*256*256 = more 256*256*256 =
16 777 216 16 777 216 16 777 216
& of bits per pixel 24 8 24 24 24
Compression of file size RLE LZW JPEG + +
without loss of quality + + (but colours - - +
can be lost)
Maximum no. of pixels per 65535 x 65535 65535 x 65535 65535 x 65535 more more
image
Multiple images in 1 file - + - - +
Animation is possible - + - - +
 81

Maximum number of colours

displayed by computers: examples

Examples of limitations / maximum number

of colours on a computer display:
8-bit = 2**8 = 256 typical for older systems
16-bit = 2**16 = 65536 = thousands of colors
= High Color
24-bit = 2**24 = (2**8)*(2**8)*(2**8) = 256*256*256
=True Color
32-bit = 2**32

82

Disks
 83

Disks:
overview of various types

Magnetic disks Optical disks

Floppy disks Laserdisc

WORM
CD-ROM
DVD
...

Hard disks

84

Disks:
comparison of types

'!
Floppy
#
Hard Optical Other
Failure rate $ High Low Low
Disk capacity $ Low High High High
Storage cost per bit $ High $ High Low Low
Speed of data access and
transfer $ Lowest High $ Low
Exchangeability + $- + $-
Transportability + $- + +
Risk of disk crashes Low $ High Low
 85

Disks:
decreasing prices: 1970-1995

Source
Sourcecited
citedinin1997:
1997:
http://community.bellcore.com/lesk/ages/ages.html
http://community.bellcore.com/lesk/ages/ages.html

86

Disks:
decreasing prices: 1995-

#
 87

Disks:
formats

'! #
Physical format Hardware

Logical format Operating system

Applications formats Application programs

88

Disks:
data transfer rate
Transfer rate
= the speed at which the computer reads data from a disk

#
once the data is found (kB/s)

'!
Floppy CD-ROM Hard
disk disk

Transfer rate about 150 (digital audio, 1x) 3 000 up to

30 kB/s < up to 8 000 kB/s (40x) 30 000 kB/s
 89

Evolution of disk capacities:

advantages and disadvantages

Floppy disks + Hard disks

Fast evolution: increasing capacity
$ Incompatibilities between floppy disk and disk hardware

Compact disc
$ Stable from 1985: no increase of capacity
High compatibility with CD and DVD drives

90

Disk backup technologies:

various systems

Tape drives
Removable drives
Recordable CD (not rewritable)
Recordable and rewritable CD
Extra hard disk
DVD-RAM
Network drives
 91

Disk backup technologies:

tape drives

+Pros:
inexpensive hardware
low media cost
large capacity

-Cons:
slow
serial storage; no fast random access

92

Disk backup technologies:

recordable CD (not rewritable)

Named CD-R
The files are not erasable; not rewritable
+Pros:
inexpensive; low media cost
random access storage
disks can be read by most CD-ROM drives
-Cons:
limited to 700 MB
Not erasable / rewritable / reusable
 93

Disk backup technologies:

recordable and rewritable CD

Named CD-RW
The files are erasable; rewritable
+Pros:
inexpensive; low media cost
(but more expensive than CD-R)
random access storage
-Cons:
limited to 700 MB
disks can NOT be read directly by most CD-ROM drives

94

Disk backup technologies:

extra hard disk

+Pros:
fast
random access storage

-Cons:
expensive but prices come down
not removable / cannot be kept off-site
 95

Disk backup technologies:

DVD-RAM

+Pros:
random access storage
large capacity

-Cons:
drives are expensive
not many drives are available

96

Disk backup technologies:

DVD-RW and DVD+RW

+Pros:
random access storage
large capacity
inexpensive

-Cons:
fragile disks
low reliability
 97

Compact Discs

98

Media based on
optical discs

WORM (not following the CD standards)

CD-DA(digital audio)
CD-ROM for PC, CD-ROM for Apple, ...
[CD-i], [CDTV]
Photo-CD
CD-ROM - based WORM
= CD-R = CD-Recordable
or CD-RW = CD-Rewritable
DVD
 99

Compact Disc = CD:

properties

The dimensions are standardised:

the diameter is 120 mm,
and the hole in the centre is 15 mm
Applied in CD-DA, CD-ROM, CD-XA, CD-i, Photo-CD,...
Constant Linear Velocity (= CLV) while reading data
Data stored on 1 side in a spiral
of valleys (named pits)
and plains (named lands)

100

Compact Discs:
reading the data

Data are read in a drive with a laser beam which converts

transitions pit-land to bits.
The reader head does not touch the surface of the disc,
so that there is no friction and thus no wear or risk of disc
crashes.
This medium offers random access to data like other
disks for computers.
Thus access is fast in comparison with media that have to
be read sequentially like tapes.
 101

Compact Discs:
storage capacity (Part 1)

1 CD can store about 600 to 700 MB

= 600 000 to 700 000 KB
For comparison, we should realise that a common A4
sheet of paper can store an amount of information in the
form of printed characters that would require about 2 kB
of space on a computer.
So one CD can store about the same amount of text
information equivalent as 300 000 of such A4 sheets.

102

Compact Discs:
storage capacity (Part 2)

1000 paper sheets together make a pile of about 10 cm.

So one CD corresponds in this view to a pile of about 30

m of paper sheets, which is a pile of paper as high as a 10-
floor building.
 103

CD-ROM: advantages in comparison

with other information carriers (1)

The formats are well standardised and the technology is

stable; this ensures a high degree of compatibility.

The information density is high.

The cost of information storage per information unit is

low.

The disks are easy to store, to transport and to mail .

104

CD-ROM: advantages in comparison

with other information carriers (2)

Crashes with the reader head do not occur in normal use,

and the disks resist well to wear.

Random access to information is possible.

CD-ROM systems are easy to use.

 105

DVD:
description

DVD is the official name ;

stood for Digital Video (or Versatile) Disc

Standard for storing data on optical disc

with a higher capacity than in the case of the older CD.

First DVD discs and drives/players available in 1997.

DVD drives/players read CD-ROM also.

106

DVD-ROM:
comparison of design with CD-ROM

Feature CD-ROM DVD-ROM

Disc diameter 120 mm 120 (or 80) mm

Data sides 1 1 or 2
Data layers 1 1 or 2
Tracks per inch 16 000 34 000
Laser 780 nm 635 to 650 nm
wavelength (infrared) (red)
 107

DVD-ROM: comparison of
performance with CD-ROM

Feature CD-ROM DVD-ROM

Capacity 0.68 Gbyte 4.5 to 17 Gbyte

Data transfer rate 1.2 (up to 12) Mbit/s > 11 Mbit/s

108

Questions?
Suggestions?
Topics for discussion?