Big Data Projects

Question 1

Steps in Big Data Analysis/Projects: Traditional with strucutred data.

Answer 1

**Conceptualize the task **-> Collect data -> Data Preperation & processing -> Data Exploration -> Model traning.

Question 2

Steps in Big Data Analysis/Projects: Textual Bid Data.

Answer 2

Text probelm formulation -> Data Curation ->** Text preperation and processing** -> Text exploration -> Classifier output.

Question 3

Preperation in strucutred data: Extraction

Answer 3

Creating a new variable from an already existing one for easing the analysis.

Example: Date of birth -> Age

Question 4

Preperation in strucutred data: Aggregation

Answer 4

2 or more variables aggregated into one signle variable.

Question 5

Preperation in strucutred data: Filtration

Answer 5

Eliminate data rows which are not needed.

[We filter out the information that is not relevant]

CFA Lv 2 Candidates only

Question 6

Preperation in strucutred data: Selection

Answer 6

Columns that can be eliminated

Question 7

Preperation in strucutred data: Conversion

Answer 7

Nominal, ordinal, integer, ratio, categorical.

Question 8

Cleansing strucutred data: Incomplete

Answer 8

Missing entries

Question 9

Cleansing strucutred data: Invalid

Answer 9

Outside a meaningful range

Question 10

Cleansing strucutred data: Inconsistent

Answer 10

Some data conflicts with other data.

Question 11

Cleansing strucutred data: Inaccurate

Answer 11

Not a true value

Question 12

Cleansing strucutred data: non-uniform

Answer 12

Non identical data format

American date (M/D/Y) vs European (D/M/Y)

Question 13

Cleansing strucutred data: Duplication

Answer 13

Multiple identical observation

Question 14

Adjusting the range of a feature: Normalization

Answer 14

Rescales in the rage 0-1

Sensitive to outliers.

Xi- Xmin /(Range)
Xi- Xmin /(Xmax -Xmin)

Question 15

Adjusting the range of a feature: Standardization

Answer 15

Centers and Rescales

Requiers normal distribution

(Xi - u) / Standard deviation

Question 16

Performance evaluation graph: Precision formula

Answer 16

P= TP / (TP + FP)

Remeber: Demoninator ( Positive)
Useful when type 1 error is high

is the ratio of correctly predictive positive classes to all predictive positive classes.
Precision is useful in situations where the cost of FP or Type I Error is high.

For example, when an expensive product fails quality inspection (predicted class 1) and is
scrapped, but it is actually perfectly good (actual class 0).

Question 17

Performance evaluation graph: Recall formula

Answer 17

TP / (TP + FN)

Remember: ( Recall we have the opposite in the denominator)

Sensitivity: useful when type 2 error is high.

also known as sensitivity i.e. is the ratio of correctly predicted positive classes to all actual
positive classes. Recall is useful in situations where the cost of FN or Type II Error is high.

For example, when an expensive product passes quality inspection (predicted class 0) and
is sent to the valued customer, but it is actually quite defective (actual class 1)

Question 18

Performance evaluation graph: Accuracy formula

Answer 18

(TP + TN) / (TP + FN + TN + FP)

Is the percentage of correctly predicted classes out of total predictions.

Question 19

Receiver operating characterisitcs: False Positive Rate Formula

Answer 19

FP / (FP + TN)

Statement / (Statement + Opposite)

Question 20

Receiver operating characterisitcs: True Positive Rate Formula

Answer 20

TP / (TP + FN)

Statement / (Statement + Opposite)

Question 21

In big data projects, which measure is the most appropriate for regression method

Answer 21

RMSE

(Root Mean Square Error)

Question 22

What is “trimming” in big data projects?

Answer 22

Removing the bottom and top 1% of observation on a feature in a data set.

Question 23

What is “Winsorization” in big data projects?

Answer 23

Replacing the extreme values in a data set with the same maximum or minumimum value

Question 24

Confusion Matrix: F1 Score Formula

Answer 24

(2 x P x R) / (P + R)

is the harmonic mean of precision and recall.

F1 Score is more appropriate than Accuracy when unequal class distribution is in the dataset andit is necessary to measure the equilibrium of Precision and Recall.

High scores on both of these metrices suggest good model performance.

Question 25

Confusion Matrix Display

Answer 25

TP FP FN TN

Question 26

What is **Mutual Information** in big data projects?

Answer 26

**How much info a token contributes to a class** Mutual Information (MI) Measures how much information is contributed by a token to a class of text. **MI = 0** The token’s distribution in all text classes is the same. **MI = 1** The token in any one class tends to occure more often in only that particular class of text.

Question 27

Feature Engineering

Answer 27

Final stage in Data Exploration **Numbers**: Differentitate among types of numbers **N-Grams**: Multi-Word patterns kept intact **Name entity recognition (NER)**: Class: Money, Time, Organization.

Question 28

How to deal with **Class Imbalance?**

Answer 28

The majority class can be under-sampled and the minority class can be over-sampled.

Question 29

Tokenization is the process of

Answer 29

Splitting a givien text into seperate words or characters. Token is equvulant to a word, and tokenization is the process of splitting the word into seperate tokens.

Question 30

the sequence of steps for text preprocessing is to produce

Answer 30

Tokens -> N-grams which to build a bag -> Input to a document term matrix.

Question 31

Big Data differs from traditional data sources based on the presence of a set of characteristics commonly referred to as the 4 V’s.. **What are thr 4 V’s?**

Answer 31

**Volume**: refers to the quantity of data. **Variety**: pertains to the array of available data sources. **Velocity**: is the speed at which data is created (data in motion is hard to analyze compared to data at rest). **Veracity**: related to the credibility and reliability of different data sources.

Question 32

What is **Exploratory Data Analysis (EDA)**, and in which stage is it in?

Answer 32

**Stage 4 and first stage in Data Exploration** is the preliminary step in data exploration. Exploratory graphs, charts and other visualizations such as heat maps and word clouds are designed to summarize and observe data.

Question 33

What is **Feature Selection**, and in which stage is it in?

Answer 33

**Stage 4 and Second stage in Data Exploration** is a process whereby only pertinent features from the dataset are selected for ML model training. Feature

Question 34

What is **Feature Engineering**, and in which stage is it in

Answer 34

**Stage 4 and Third and final stage in Data Exploration** is a process of creating new features by changing or transforming existing features. Feature Engineering techniques systematically alter, decompose or combine existing features to produce more meaningful features.