Exam 1

Question 1

Where is the most effort put in data mining?

Answer 1

Data preparation and cleaning

Question 2

What are the data-related steps in the CRISP-DM guide?

Answer 2

Select/find data, Clean the Data, Prepare the data, Integrate the data, and Format the data

Question 3

How is data represented?

Answer 3

Numeric: Continuous attributes. Measurements, int, float data types

Nominal - values are symbolic - labels: sunny, old, yellow: can perform equality checks. Categorical coding; may code “1” but has no arithmetic meaning.

Ratio - The measurement scheme defines a zero point a distance, temperature differential but not the temperature itself; math operations are valid.

Ordinal - rank order: “cold, cool, warm, hot” or “good, better, best”. No distance between them, cna beform equality checks.

Interval - ordered and measured in fixed units.

Question 4

When is numeric data easy to interpret?

Answer 4

When defined ranges exist.

Question 5

How do you measure something good, bad, healthy?

Answer 5

Need domain expert.

Question 6

What are some cautions on data cleaning?

Answer 6

Document what you do, work carefully, don’t make assumptions, be aware of bias.

Question 7

What are some was to introduce bias?

Answer 7

Language - different terms or grammars to describe the domain, data attributes, or the problem.

Search - look at other search options

Overfitting - results provide a solution based on bad assumptions/patters, or search stops too soon.

Actions already perform on the data.

How the data was gathered (how questions were asked, how responses were interpreted, who asked the questions, how samples were selected).

Synonym for “error”.

Question 8

What are some examples of data cleaning?

Answer 8

Handling invalid values, duplicates, missing data, data entry errors, converting data to specific values in order to perform correct measurements.

Question 9

What is meant by dirty data?

Answer 9

Data that is incorrect, inaccurate, irrelevant or incomplete.

Data needing to be converted (nominal to numeric)

Data with different formats or coding schemes (such as dates)

Data from >1 file with different field delimiters

Data that is coded

Data that must be summarized “rolled up”

Question 10

How does data get “dirty”?

Answer 10

Inconsistent definitions, meanings (especially when combining different sources)

Data entry mistakes

Collection errors

Corrupted data transmissions

Conversion errors.

Question 11

What are some data issues?

Answer 11

Out of range entries

Unknown, unrecorded or irrelevant data

Missing values

Language translation issues

Unavailable readings

Inapplicable data (asking a male if pregnant)

Customer provided incorrect data

Duplicate data

Stale data

Unavailable data

Data may be available but not in electronic form.

Data associated with the wrong person

User provided wrong data.

Question 12

Consider representing dates as YYYYMM or YYYYMMDD. What’s good about this formatting? What is the limitation?

Answer 12

Good: You can sort the data.
Limitation: Does not perserve intervals. (ie. 20040201 - 20040131

Question 13

What are some legacy issues when it comes to dates?

Answer 13

Y2K - 2 digit year. - Year 02 is it 1902 or 2002? Depends on context (child birthday or a year a house was built). Typical approach is to set a cutoff year. If YY < cutoff, then 20YY else 19YY

Question 14

What are some reasons values may be missing?

Answer 14

• They are unknown, unrecorded, irrelevant data
• Malfunctioning equipment
• Changes in the design
• Collation/merge of different datasets.
• Unavailable data
• Removals because of security or privacy issues.
• Translation issues (especially languages)
• That adata being used for a different purposes than originally planned (ethical/ legal issues)
• self-reporting - people may omit if the input mechanism does not require an input.

Question 15

How should one deal with missing values?

Answer 15

• Ignore the attribute or entire instances. (May throw out the needle in the haystack!)
• Try to estimate or predict: use mean, mode, or median values. Relatively easy and not bad on average.
• Treat as seperate value
• Look for , 0, “.”, 999, N/A Decide on a standard and create a new value.
• Does missing imply a default value?
• Compute the value based on previous values.
• If inserting zeros for missing values, think about what it has done to the mean and standard deviation.
• Be careful when using tools (some have default operations to handle missing data)
• Randomly select values from current distribution (pro: won’t change overall shape of the curve - little impact on the mean).

Question 16

Again, what are some sources of inaccurate data? :)

Answer 16

• Data entry mistakes
• Measurement errors
• Outliers previously removed
• Duplicates
• stale data
• Different values. New York, NY, N.Y

Question 17

How can you find inaccurate data?

Answer 17

Look for the obvious (run statistical tools), Look for nonsensical data (negative grade or age).

Question 18

What is discretization?

Answer 18

• Binning
• Useful for generating summary data
• produces discrete values

Question 19

What is one issue that can come from binning with equal-width?

Answer 19

It could result in clumping. For example, if 99% of employees earn 0-200,000 and the owner makes 2,000,000. With a width of 200,000 only one person in the upper bin.

Question 20

How can we even out the distribution?

Answer 20

By binning with equal-height. Instead of defining bin sizes of range N, assign N values to each bin.

Question 21

When binning…

Answer 21

• Do not split repeated values across bins

- Create a separate bin for special values.

Question 22

Talk about the considerations with equal-width and equal-height binning.

Answer 22

Equal-height is usually preferred because it avoids clumping. Equal-width is simplest and good in many situations. However, equal-height usually gives better results.

Question 23

Is by bins okay?

Answer 23

AFter you create bins, create a histogram of values and look at the general shape of it. Jagged shape may indicate a weakness in the way the bins were formed so try different number of bins and different boundaries (shit ranges).

Question 24

Why use rollup?

Answer 24

Can help reduce the complexity of your model.

Question 25

What is an outlier in data mining?

Answer 25

Any value that doesn't really look like most of the others in the data set. May just be a unique data point, or it could really be an outlier (an noise, error).

Question 26

How do you handle outliers?

Answer 26

- Do nothing - Enforce upper and lower bounds - Let binning handle the problem.

Question 27

Where can outlier exist?

Answer 27

In a normal distribution, they could be >3 standard deviations from your mean. With a bimodal distribution, they could be at the middle or at the ends. Some are easy to find such as negative age or an age > 120, negative number of children, gender that's not M/F.

Question 28

How can you tell an outlier fron an error?

Answer 28

Usually you can't. Don't discard outliers unless you are sure they are really outliers.

Question 29

What is the nest approach to dealing with outliers?

Answer 29

- Work with your domain expert. - Try to help identify why the data values are extreme. - Do remove the outliers if you think they will negatively impact your analysis. - Check the source and quality of the raw data.

Question 30

What is the scare with data warehouses?

Answer 30

If they clean the data and remove the outliers, the needle in the haystack you were looking for may have been removed.

Question 31

Why use transformations?

Answer 31

Such as house prices with a skewed tail due to extremely high house prices. May need to transform the data to make neater bins or to scale for visualizations.

Question 32

What is one approach to transform data?

Answer 32

``` Apply log10 function to numeric values. Log(10)1 = 0 log(10)10 = 1 log(10)100 = 2 log(10)1000 = 3 ``` To care for log0 being undefined, we can bump all values by 1 and take the absolute value to handle negative values. This scales the data making it easier to visualize and handle. In general form: Log(10)|X + 1|. Add sin to regain the sign on negative side: SinXLog(10)|X+ 1|

Question 33

What is regression modeling?

Answer 33

Try to fit the data to a line, calculate the error, sometimes easy to identify extreme values.

Question 34

What is the Curse of Dimensionality?

Answer 34

As the dimensionality increases (the number of attributes), the data becomes increasingly sparse in the space it occupies. Some clustering algorithms become less accurate.

Question 35

Why prune data?

Answer 35

- Lots of attributes are hard to work with - Many attributes many not even matter - Many algorithms perform better if we reduce number of attributes being considered.

Question 36

How do we prune data?

Answer 36

Remove fields with little or no variability and select the N most important.

Question 37

What are some pruning techniques?

Answer 37

Look for redundant data. | Look for irrelevant features.

Question 38

What are some data cleaning problems?

Answer 38

- Error correction and missing information is a challenging problem. - Incorrect or sloppy data cleaning can result in removing a large amount of data or incorrect data. - There is not enough information or application knowledge to know what to do: consult domain expert - Data cleaning is an iterative, time consuming. - Arguably most time consuming, painful, and expensive part of data mining.

Question 39

Why do we need a data management policy?

Answer 39

- After cleaning and removing errors, you don't want to repeat it if the collection changes or when new values are added. - When you clean the data, you need to document what you did to the data. - Need a way to audit and measure the resulting data. Backups, deletes, Changes, Access, Last Updated.

Question 40

How can we check for data integrity?

Answer 40

- Data quality: "noise" - modification of original values, outliers, missing values, duplicates. Cost- Continuing pressure for larger and faster systems. Driven by the quantity of data and the need for esults in a limited amount of time.

Question 41

What is a quality issues?

Answer 41

Some classes have a very unequal frequency. For example, 99.99% of Americans are not terrorists and saying that you are correct 97% of the time doesn't say much.

Question 42

What is metadata?

Answer 42

It encodes background knowledge. Can be used to restrict search space, may need to expand metadata attributes into multiple attributes.

Question 43

What are some questions to ask?

Answer 43

What data is available? Where will it come from? Is the data current/relevant? Is other data available? Is historic data available? Who is the domain expert? Which attributes are really important?

Question 44

How do you know if attributes are irrelevant?

Answer 44

- They have no influence on the outcome (e.g customer ID). - May slow things down. - May be illegal to use. - Now always easy (for example increased sales during phases of the Moon).

Question 45

Why do we not use the whole data set?

Answer 45

- Run the risk of overfitting the data. | - WHen model is built to fit training set, it loses the ability to generalize to new data.

Question 46

How is data split between the training set and the test set?

Answer 46

Theory: 70% of data for training, 30% for test set. | In practice: 50/50

Question 47

How do you build testing, training, and validation data sets?

Answer 47

- Random without replacement. - Systematic: Every nth instance. Don't select the first N - order may influence the data.

Question 48

What is the stratified random sample?

Answer 48

Select equal number of instances from each attribute value. Why? If looking for a small #, you are training on biased data.

Question 49

Important considerations for sampling?

Answer 49

Think counts, not percentages. Need enough data to saftely generalize. Otherwise, no general rules.

Question 50

What is the CRISP cycle?

Answer 50

Business Understanding Data Understanding -> Data preparation Modeling -> Evaluation -> business understanding OR deployment

Question 51

What are some reasons to cluster data?

Answer 51

Exploratory data analysis. In marketing discover customer groupings, in astronomy discover groups of similar objects, in earthquakes discover epicenters clusters along faults, in gene research group genes. Identify patterns, document classification, targeting marketing, insurance, taxing.

Question 52

What is clustering?

Answer 52

Form of unsupervised learning as there are no predefined classes. The goal is to find 'natural' grouping of instances. Objects in a cluster are similar to those in the same cluster but dissimilar to objects in a different cluster.

Question 53

What are some methods to determining similarity?

Answer 53

- visual inspection - mathematical measurements such as euclidian distance and manhattan distance. May need to weight more important attributes.

Question 54

What is a centroid?

Answer 54

Using euclidean distance or something similar, it is the average of the values corresponding attribute for all points in the cluster.

Question 55

What is K-Means clustering?

Answer 55

An exclusive clustering algorithm where each object is assigned to precisely one set of clusters.

Question 56

How can we measure the quality of a set of clusters?

Answer 56

By using an objective function and making it as small as possible.

Question 57

What is an objective function

Answer 57

An objective function takes the sum of the squares of the distances of each point from the centroid of the cluster which it is assigned.

Question 58

What is the major assumption of K-Means?

Answer 58

That the distance variance is an appropriate way to cluster. Other concenrs: Clusters may be disjoint or overlap. Exclusive clustering - all instances must be clustered.

Question 59

What is the requirement to perform K-Means clustering?

Answer 59

Need a distance measurement.

Question 60

What is the K-Means algorithm?

Answer 60

1. Choose a value K (number of clusters) 2. Select k objects in an arbitrary fashion and use these as the initial set of k. 3. Assign each object to the nearest centroid. 4. Recalculate the centroids of the k clusters. 5. Repeat step 3 and 4 until the centroids no longer move (convergence).

Question 61

What is convergence?

Answer 61

- Data points no longer move into different clusters or the change in cluster assignments is less than some threshold.

Question 62

What are some of the k-means questions to ask?

Answer 62

- Does it make sense? - Initial assignment has to be numeric and sensible - Does it end?

Question 63

What are some ways to determine the initial centroids?

Answer 63

- Randomly select the initial K centroids from the data instances - Pick centroids that are the furthest distance - Make a quick pass of the data and use some basic stats to select the initial centroids.

Question 64

Instead of assigning initial centroids, what method can be used?

Answer 64

Initially assign the values then calculate the centroids.

Question 65

What are some ways to initially assign the values then calculate the centroids?

Answer 65

- Randomly assign N/K instances to each cluster. - Assign sequentially N/K instances to cluster 1, then next N/K instances to cluster 2, .. - Assign instances to cluster 1.. K using a round robin .. then calculate the centroids.

Question 66

What is the pillar algorithm for K-Means?

Answer 66

It address initial centroid selection. The method esentially visualizing pillars which are placed the furthest apart from each other.

Question 67

How do you find the pillars?

Answer 67

Calculate the mean of all points Calculate the distance from each point to the mean. The point with the maximum distance is the 1st centroid. Calculate the distance from each point to the 1st centroid. Find the point the furthest distance from 1st centroid and this becomes the 2nd centroid. Find the distance from all points to the 2nd centroid. Add this distance to the accumulated distance metric. 3rd centroid is the point with the largest distance metric. Continue until all K centroids have been designated.

Question 68

What are some drawbacks of the pillars algorithm?

Answer 68

- Must be able to identify outliers | - Time complexity is on the order of O(N+K) but increases significantly when you add in outlier identification.

Question 69

Why should you consider using median instead of mean for k-means?

Answer 69

K-means is sensitive to outliers and it's not as affected by extreme values.

Question 70

What are some issues and problems with k-means?

Answer 70

- Algorithm has a difficult time handling noisy data and outliers. - Since it uses the means, think about what happens to a mean when the data contains an outlier.

Question 71

Summarize K-Means

Answer 71

- It always terminates but may not find the best clustering - Much depends on the initial selection of centroids - Difficult to visualize high dimension data - Outliers are a problem - Can be sensitive to data order.

Question 72

What is incremental clustering and why do we need it?

Answer 72

When we need to handle very large amounts of data. - Partition the data into clustered subsets - Incremental clustering - parallel implementation of an algorithm.

Question 73

What is an incremental clustering method?

Answer 73

Instance based clustering

Question 74

How does instance-based clustering work?

Answer 74

With instance-based incremental clustering, we build the clusters as instances are read in. After adding an instance, look to see if its in the best place; if not, restructure the clusters to make a better fit.

Question 75

What are some limitations of instance-based clustering?

Answer 75

May not build optimal clusters Ordering of data will impact the clusters Restructuring algorithm is often not enough to reverse the impact of a bad initial cluster assignment.

Question 76

What is the faster nearest neighbor calculation?

Answer 76

Represent the data in a tree such as a kD-tree. Easy to add new instances. More difficult to understand. Skewed dataset can result in imbalanced trees.

Question 77

What is hierarchical clustering?

Answer 77

Agglomerative (bottom-up) hierarchical clustering. Start each instance in its own cluster than repeatedly merge the closet pairs. Stop the merging with all instances in a single clusters. Produces a single hierarchical cluster. For n instances, iterate N-1 times.

Question 78

What is a dendrogram?

Answer 78

- Produces a binary tree.

Question 79

Why use density-based clustering (DB scan)?

Answer 79

Clusters not clean and neat, may not have specific shapes.

Question 80

What is density based clustering?

Answer 80

Points are labeled as core, border (edge) or noise. Noise points are ignored initially. Edges are drawn between core points within a certain range (density reachable). Connected core points are assigned to a cluster. Assigns clusters based on the density of the instances.

Question 81

What is fuzzy clusters?

Answer 81

Sometimes instances could belong to more than one clusters. Assign objects a probability that it belongs in the cluster.

Question 82

What is EM?

Answer 82

Expectation Maximization. It is a probability-based clustering algorithms. Classifies using mean and standard deviation of values. Uses values if attribute is continuous or categorical if it is nominal attributes.

Question 83

What is cobweb?

Answer 83

WUses probablistic clustering, representing the probability of a value being present. Handles categorical attributes and continous attributes. Incremental clustering so sensitive to data order. BUilds a tree representation of the clusters (hierarchical).

Question 84

When to terminate the clustering process?

Answer 84

End the merger process when we have converted the original N objects to a small enough set of clusters. - Merge clusters only until some predefined number remain - Stop when newly created cluster fails to meet some criterion for its compactness such as average distance between objects is too high.