10 - Text Analytics

Question 1

Describe the three main ways of data?

Answer 1

• Unstructured
  
  • PDF, JPEG, MP3, Movies
• Semi-structured
  
  • CSV, JSON, XML
• Structured
  
  • Oracle, MSSQL, MySQL

Question 2

What is Text Analytics?

Answer 2

Text mining, also referred to as text data mining, roughly equivalent to text analytics, is the process of deriving high-quality information from text. Used for:

• Recognition of texts with a case-related criminalistic relevance
• Recognition of relations in these texts in order to reveal whole relationship networks and planned activities
• Identification and/or tracking of fragmented texts
• Identification or tracking of hidden semantics

Question 3

What are the challenges of Text Analytics?

Answer 3

• Ambiguity (Mehrdeutigkeit)
• Non-Standard Language
• More Complex Languages Than English

Question 4

Which two parts has the general process of text analytics?

Answer 4

• Data Preparation
  
  • Text PreProcessing, Text Transformation
• Data Learning
  
  • Feature Selection / Mining / Evalute Results

Question 5

What are the five main steps of the Text Analytics Process?

Answer 5

1. Text Pre-Processing
2. Transformation
3. Feature Selection
4. Mining
5. Evaluation

Question 6

What is done in the Pre-Processing phase?

Answer 6

1. Tokenization to words
2. Stemming & Lemmatization
3. Remove Punctuations
4. Remove stop words

Question 7

What are Stemming & Lemmatization?

Answer 7

Stemming is a technique used to find out the root / stem of a word.

Lemmatization is a technique used to find out the lemma of a word.

Question 8

What does the abbreviation TF-IDF stands for and where is it used?

Answer 8

• TF - Term Frequency
• IDF - Inverse Document Frequency

Used in Text Data Analysis.

Question 9

Explain TF?

Answer 9

TF = Term Frequency

Gives us the frequency of the word in each document in the corpus. It increases as the number of occurrences of that word within the document increases. Each term has its own TF in each document.

Question 10

Explain IDF?

Answer 10

IDF = Inverse Document Frequency

Used to calculate the weight of rare words across all documents in the corpus. The words that occur rarely in the corpus have a high IDF score.

Question 11

What can be done with vectors?

Answer 11

• Document Summarization
• Document Ranking

Question 12

What are the three basic ways of text measurement in Text Mining?

Answer 12

• Euclidean Distance
• Cosine Similarity
• Jaccard Similarity

Limitations: The techniques do not cover the synonym scenario (dog / puppy) etc.

Question 13

What are common techniques for Text Data Analysis?

Answer 13

• TF-IDF
• Ranking documents
• Distance & Similarity
• Vectorization