4 - Exploratory Data Analysis

Question 1

What is the primary purpose of exploratory data analysis (EDA)?

Answer 1

To explore data without a priori hypotheses and uncover relationships.

Question 2

What is a hypothesis test (HT)?

Answer 2

A method to test specific hypotheses about data using statistical methods.

Question 3

What does EDA allow the user to do?

Answer 3

Explore relationships, derive new variables, and use binning to increase predictive value.

Question 4

What is the relationship explored in the bar graphs discussed?

Answer 4

The relationship between a categorical predictor and the target variable.

Question 5

What does ‘previous_outcome’ refer to?

Answer 5

The result of a previous marketing campaign with the same customer.

Question 6

What is the advantage of normalized bar graphs?

Answer 6

They allow easier comparison of response proportions between categories.

Question 7

What are two best practices when working with bar graphs?

Answer 7

• Supplement unclear bar graphs with normalized versions
• Provide non-normalized graphs to indicate original distribution.

Question 8

How do you create a contingency table in Python?

Answer 8

Using the crosstab() command.

Question 9

What should the response variable represent in a contingency table?

Answer 9

The rows.

Question 10

How do you calculate column percentages in Python?

Answer 10

Using the sum() and div() commands.

Question 11

What is a histogram?

Answer 11

A graphical representation of a frequency distribution for a numerical variable.

Question 12

What is the benefit of using a normalized histogram?

Answer 12

It helps distinguish response patterns more clearly.

Question 13

What is the best practice for histograms?

Answer 13

• Use non-normalized histograms for original distributions
• Use normalized histograms for response patterns.

Question 14

What command is used in R to create a contingency table?

Answer 14

The table() command.

Question 15

What is the purpose of the addmargins() command in R?

Answer 15

To add row and column totals to a contingency table.

Question 16

What is the significance of the geom_bar() function in ggplot2?

Answer 16

It specifies that a bar chart should be created.

Question 17

Fill in the blank: EDA is often preferred when clients have _______ about the data.

Answer 17

no salient a priori notions

Question 18

True or False: A normalized bar graph shows the original distribution of data.

Answer 18

False

Question 19

What is the main drawback of a normalized histogram?

Answer 19

It does not indicate the original distribution of the data.

Question 20

What is the purpose of using a non-normalized histogram?

Answer 20

To obtain the original distribution of the data values.

Question 21

What is the benefit of using a normalized histogram?

Answer 21

To help better distinguish the response patterns.

Question 22

What Python package is used for constructing histograms?

Answer 22

matplotlib

Question 23

What command in Python creates a stacked histogram?

Answer 23

plt.hist()

Question 24

In the plt.hist() command, what does the parameter ‘stacked = True’ do?

Answer 24

It stacks the two variables in the histogram.

Question 25

What does the 'bins' parameter specify in a histogram?

Answer 25

The number of bins in the histogram.

Question 26

What is the purpose of the column_stack() function in Python?

Answer 26

To combine the heights of the two variables’ bars into one array.

Question 27

How do you calculate the normalized proportions in a histogram?

Answer 27

By dividing each row by the sum across that row.

Question 28

What is the command to create a contingency table in Python?

Answer 28

pd.crosstab()

Question 29

Fill in the blank: The command 'cut()' is used in Python to _______.

Answer 29

bin the values into categories.

Question 30

What is the significance of using 'right = False' in the cut() command?

Answer 30

It excludes the right-hand cutpoint from the category.

Question 31

What R command is used to create a histogram with an overlay?

Answer 31

ggplot() + geom_histogram()

Question 32

What does the 'position = "fill"' input do in R's geom_histogram()?

Answer 32

It normalizes the histogram.

Question 33

True or False: The age group 27 to 60 has a high response proportion.

Answer 33

False

Question 34

What is the recommended best practice for binning?

Answer 34

Use binning based on predictive value.

Question 35

What is the main advantage of creating categorical variables through binning?

Answer 35

Some algorithms work better with categorical rather than numeric variables.

Question 36

What does the 'aes(fill = response)' command do in R?

Answer 36

It adds an overlay to the histogram based on the response variable.

Question 37

What visual representation is generated by the command 'crosstab_02.plot(kind="bar", stacked = True)'?

Answer 37

A stacked bar graph of age binned with response overlay.

Question 38

What does the command 'prop.table()' do in R?

Answer 38

It calculates the proportions of the contingency table.

Question 39

What is a key observation made about the age groups in relation to response rates?

Answer 39

Both the older and the younger groups have a much higher response rate than the middle group.

Question 40

What is exploratory data analysis (EDA)?

Answer 40

EDA is an approach to analyzing data sets to summarize their main characteristics, often using visual methods.

Question 41

When should analysts use exploratory data analysis (EDA) rather than hypothesis testing?

Answer 41

EDA should be used when the objective is to explore data and find patterns without predefined hypotheses.

Question 42

What are some examples of what EDA allows the user to do?

Answer 42

* Identify trends * Discover patterns * Detect anomalies * Generate hypotheses

Question 43

Which graph do we use to explore the relationship between a categorical predictor and the target variable?

Answer 43

Bar graph

Question 44

What are (non-normalized) bar graphs useful for?

Answer 44

They are useful for displaying the frequency of categorical data.

Question 45

State one advantage and one disadvantage of using a normalized bar graph.

Answer 45

Advantage: Easier comparison across categories. Disadvantage: Can obscure the actual counts.

Question 46

What are the two best practices when working with bar graphs for EDA?

Answer 46

* Clearly label axes * Use appropriate scales

Question 47

What does a contingency table help us to do?

Answer 47

It helps to summarize the relationship between two categorical variables.

Question 48

Explain the two best practices when working with contingency tables in EDA.

Answer 48

* Ensure proper variable representation * Report counts and percentages

Question 49

What is a histogram?

Answer 49

A histogram is a graphical representation of the distribution of numerical data.

Question 50

Describe one advantage and one disadvantage of using a normalized histogram.

Answer 50

Advantage: Facilitates comparison between different data sets. Disadvantage: Can mislead if the total counts differ significantly.

Question 51

What are the best practices for working with histograms in EDA?

Answer 51

* Choose appropriate bin sizes * Clearly label axes

Question 52

Why might it be useful for the analyst to bin a numeric variable?

Answer 52

Binning can simplify the analysis and help highlight trends.

Question 53

Why do we use the binning method shown in this chapter rather than automatic binning methods?

Answer 53

Manual binning allows for more control and better understanding of the data distribution.

Question 54

True or False: Data scientists should use automatic methods of data analysis without caution.

Answer 54

False

Question 55

What is the purpose of creating a bar graph of the previous_outcome variable?

Answer 55

To visualize the distribution of previous outcomes.

Question 56

What is the purpose of creating a normalized bar graph of the previous_outcome variable?

Answer 56

To compare the proportions of responses across different previous outcomes.

Question 57

What should be included when comparing a contingency table with bar graphs?

Answer 57

Counts and percentages for each category.

Question 58

What is the relationship between age and response demonstrated in a histogram?

Answer 58

It shows how age distribution correlates with the response variable.

Question 59

What is the purpose of binning the age variable?

Answer 59

To group ages into categories for easier analysis.

Question 60

What should be included in a contingency table of job with response?

Answer 60

Counts and column percentages.

Question 61

What is the significance of combining job categories based on response percentages?

Answer 61

It simplifies analysis and highlights significant trends.

Question 62

How do you define a new categorical variable from the duration variable?

Answer 62

By identifying cutoff points that separate low and high response values.

Question 63

What should be done after identifying outliers in the capital-loss variable?

Answer 63

Construct a bar graph for the outlier records.

Question 64

What is the effect of deleting outliers at the EDA stage?

Answer 64

It changes the character of the data set and can lead to misleading conclusions.

Question 65

What does the capital-loss-flag variable represent?

Answer 65

It equals 0 when capital-loss equals 0, and 1 otherwise.

Question 66

What is the rationale for combining certain categories in a contingency table?

Answer 66

To reduce complexity and improve interpretability.

Question 67

What is the purpose of renaming variables before further analysis?

Answer 67

To maintain clarity and track changes in variable representations.

Question 68

What should be analyzed when creating a histogram of the education variable?

Answer 68

The relationship between education levels and income.

Question 69

What is the significance of binning the age variable into specific ranges?

Answer 69

It helps to analyze trends and patterns specific to age groups.

Question 70

What type of data visualization is suggested for the sex predictor?

Answer 70

Both non-normalized and normalized bar graphs.

Question 71

What does the normalized bar graph of occupation with a sex overlay illustrate?

Answer 71

The distribution of sex across different occupations.

Question 72

What is a contingency table with sex for the rows and occupation for the columns used for?

Answer 72

To compare the distribution of sex across different occupations.