Information Retrieval

Question 1

indexing

Answer 1

task of finding terms that describe the documents well

Question 2

manual indexing

Answer 2

done by using a predefined set of index terms and fixed vocabularies
the indexing is done by humans
labour intensive
they are high precision searches and work well for closed collections, however, searches need to know terms to achieve precision and labellers need to be trained in order to achieve consistency

Question 3

text retrieval

Answer 3

find documents that are relevant to a user query given a large static document collection and information needed

Question 4

automatic indexing

Answer 4

uses natural language as indexing language, implementation of indices done via inverted files. It also consists of term manipulation and term weighting

Question 5

inverted file index

Answer 5

it can be used to record in which document a term occurs, how many occurrences, and the position of those occurrences

Question 6

bag-of-words approach

Answer 6

only records what terms are present and their occurrence. ignores the relationship between words i.e. ordering

Question 7

boolean model

Answer 7

the boolean query constructs complex search commands by combining basic search terms with Boolean operators.
precise simple, logical basis for deciding whether any document should be returned based on whether the basic terms of query do not appear in the document and the meaning of the logical operators.

Question 8

vector space model

Answer 8

uses the bag-of-words approach where documents = points in high dimensional vector space
dimension = term in an index
frequencies of terms in documents = values
queries are represented as vectors

Question 9

method to perform vector space model

Answer 9

1. select documents(s) with the highest document-query similarity (model for relevance => ranking_
2. number of documents returns => less relevant thus uses start at the top of ranking stop when satisfied

Question 10

normalized correlation coefficient

Answer 10

the cosine of the angles between the vectors

• vector pointing in the same direction: 1
• orthogonal vector: 0
• vectors pointing in opposite directions: -1

This computes how well occurrences of each term i correlate in query and document, then scales for the magnitude of the overall vectors

Question 11

term manipulation

Answer 11

the pre-process of terms for generalization

Question 12

tokenization

Answer 12

process of removing punctuation

Question 13

capitalisation

Answer 13

normalise all words to lower/upper case

Question 14

lemmatisation

Answer 14

conflate different inflected forms of a word to their basic form (singular, present tense, 1st tense)

Question 15

stemming

Answer 15

conflate morphological variants by chopping their affix (connected, connection -> connect)

Question 16

normalisation

Answer 16

heuristics to conflate variants due to spelling, hyphenation, spaces,

Question 17

stop list

Answer 17

removes non-content words (most frequency and least useful for retrieval

Question 18

bigram indexing

Answer 18

store each bigram as a term in index

i..e pease porridge in the pot => pease porridge, porridge in, in the, the pot

Question 19

position indexing

Answer 19

identifies multi word phrase during retrieval by storing position terms in documents

Question 20

term weighting

Answer 20

1. document collection
2. size of collection
3. term frequency
4. collection frequency
5. document frequency

Question 21

inverse document frequency

Answer 21

log(size of collection/documents containing the term)

Question 22

tf.idf

Answer 22

common weighting method which is the some of the term frequency and idf

Question 23

PageRank Algorithm

Answer 23

exploits the link structure of the web
- link from page A to page B confers authority on B depending on the PageRank score of A and its number of outgoing links (recursively defined)

Question 24

recall

Answer 24

the proportion of relevant documents returned

relevant & retrieved documents / all relevant

Question 25

precision

Answer 25

the proportion of retrieved documents that are relevant | relevant & retrieved / all retrieved

Question 26

f measure (f1)

Answer 26

combines precision & recall into a single figure to give equal weight to both

Question 27

precision at cut off

Answer 27

measures how well a method ranks relevant documents before non relevant documents

Question 28

average precision

Answer 28

precision computed for each point a relevant documents is found and figures averaged