Data Pre processing

Question 1

What is Data Preprocessing?

Answer 1

A step in data science that transforms raw data into a format that can be understood and analyzed by computers and machine learning.

Question 2

Why is Data Preprocessing important?

Answer 2

Real-world data is often dirty, incomplete, noisy, or inconsistent, and preprocessing helps clean and prepare it for analysis.

Question 3

What are the major tasks involved in Data Preprocessing?

Answer 3

Data cleaning, data integration, data reduction, and data transformation.

Question 4

What is Data Cleaning?

Answer 4

The process of handling missing values, smoothing noisy data, identifying/removing outliers, and resolving inconsistencies.

Question 5

What are the common issues in real-world data?

Answer 5

Missing values, noise, errors, duplicate records, and inconsistent data formats.

Question 6

What are some methods to handle missing data?

Answer 6

Removal, interpolation, replacement with mean or most common value.

Question 7

What is noisy data?

Answer 7

Random errors or variance in a dataset that can affect model performance.

Question 8

What are common techniques for handling noisy data?

Answer 8

Binning, clustering, and combined computer-human inspection.

Question 9

What is Data Integration?

Answer 9

The process of combining data from multiple sources to create a unified dataset.

Question 10

What challenges arise in Data Integration?

Answer 10

Schema mismatches, redundancy, and inconsistencies across different sources.

Question 11

What is Data Reduction?

Answer 11

Reducing the amount of data while maintaining its integrity to improve efficiency.

Question 12

What are the types of Data Reduction?

Answer 12

Numerosity reduction and dimensionality reduction.

Question 13

What is Numerosity Reduction?

Answer 13

Reducing the number of data objects (rows) in a dataset.

Question 14

What is Dimensionality Reduction?

Answer 14

Reducing the number of features (columns) while preserving data structure.

Question 15

What are common Numerosity Reduction methods?

Answer 15

Random sampling, stratified sampling, and random over/undersampling.

Question 16

What is Random Sampling?

Answer 16

Randomly selecting a subset of data points to reduce computational cost.

Question 17

What is Stratified Sampling?

Answer 17

Selecting a sample that maintains the original proportions of different groups in the dataset.

Question 18

What is Random Over/Undersampling?

Answer 18

Altering the sample proportions to balance class distributions for classification tasks.

Question 19

What are common Dimensionality Reduction methods?

Answer 19

Linear Regression, Decision Trees, Random Forest, PCA, Functional Data Analysis (FDA).

Question 20

How does Linear Regression help in Dimensionality Reduction?

Answer 20

By identifying independent variables with weak predictive power and eliminating them.

Question 21

What is PCA (Principal Component Analysis)?

Answer 21

A technique that transforms data into new components, capturing the most variance while reducing dimensionality.

Question 22

What is the difference between Supervised and Unsupervised Dimensionality Reduction?

Answer 22

Supervised focuses on improving predictions, while Unsupervised reduces data size without prediction considerations.

Question 23

What is Data Transformation?

Answer 23

Modifying the dataset to ensure it is suitable for analysis and improves model performance.

Question 24

What are the key types of Data Transformation?

Answer 24

Normalization and Standardization.

Question 25

What is Normalization?

Answer 25

Scaling data so that all values fall within a specific range, usually [0,1].

Question 26

What is Standardization?

Answer 26

Transforming data to have a mean of 0 and standard deviation of 1.

Question 27

When should Normalization be used?

Answer 27

When distance-based algorithms like K-Means and KNN are used.

Question 28

When should Standardization be used?

Answer 28

When maintaining data variance is important for analysis.

Question 29

What is Clustering in Data Preprocessing?

Answer 29

A method to identify and group similar data points to detect outliers or patterns.

Question 30

What are the types of Clustering?

Answer 30

Partitioning methods (K-Means), Hierarchical methods, and Density-based methods (DBSCAN).

Question 31

What is K-Means Clustering?

Answer 31

An iterative clustering method that partitions data into K groups based on distance to centroids.

Question 32

What is the Elbow Method?

Answer 32

A technique to determine the optimal number of clusters in K-Means by analyzing inertia.

Question 33

What is DBSCAN Clustering?

Answer 33

A density-based clustering method that identifies clusters based on regions of high point density.

Question 34

What is the Silhouette Score?

Answer 34

A measure of how well a data point fits within its assigned cluster, ranging from -1 to 1.

Question 35

What is Inertia in Clustering?

Answer 35

The sum of squared distances between data points and their respective cluster centroids.

Question 36

What is Hierarchical Clustering?

Answer 36

A method that builds a hierarchy of clusters, merging them step by step.

Question 37

What is the difference between Agglomerative and Divisive Clustering?

Answer 37

Agglomerative starts with individual points and merges clusters, while Divisive starts with one cluster and splits it.

Question 38

What is an Outlier?

Answer 38

A data point significantly different from the rest of the dataset.

Question 39

What are common Outlier Detection methods?

Answer 39

Statistical methods, clustering-based methods, and density-based methods.

Question 40

What is an example of Outlier Handling using Clustering?

Answer 40

Using DBSCAN to separate noise points from valid clusters.

Question 41

What is Feature Engineering?

Answer 41

Creating new features from raw data to improve predictive performance.

Question 42

What is Feature Selection?

Answer 42

Choosing the most relevant features to improve model accuracy and reduce complexity.

Question 43

Why is Data Preprocessing crucial for machine learning?

Answer 43

Ensures high-quality input data, improving model accuracy and efficiency.

Question 44

What happens if data preprocessing is not done correctly?

Answer 44

Models may learn misleading patterns, perform poorly, or be biased.