Machine Learning - Supervised

Question 1

Classification

Answer 1

We are trying to predict results in a discrete output. In other words, we are trying to map input variables into discrete categories.

Question 2

Classification: Linear Binary

Answer 2

Binary classification problems arise when we seek to separate two sets of data points in R^n, each corresponding to a given class. We seek to separate the two data sets using simple ‘‘boundaries’’, typically hyperplanes. Once the boundary is found, we can use it to predict the class a new point belongs to, by simply checking on which side of the boundary it falls.

Question 3

Decision Trees

Answer 3

A tree of questions to guide an end user to a conclusion based on values from a single vector of data. The classic example is a medical diagnosis based on a set of symptoms for a particular patient. A common problem in data science is to automatically or semi-automatically generate decision trees based on large sets of data coupled to known conclusions. Example algorithms are CART and ID3. (Submitted by Michael Malak)

Question 4

Decision Trees

Answer 4

each node in the tree tests an attribute, decisions are represented by the edges leading away from that node with leaf nodes representing the final decision for all instances that reach that leaf node

Question 5

Decision Trees: Best Uses

Answer 5

…

Question 6

Decision Trees: Cons

Answer 6

1: Complex trees are hard to interpret; 2: Duplication within the same sub-tree is possible

Question 7

Decision Trees: Dealing with missing values

Answer 7

1) have a specific edge for no value; 2) track the number of instances that follow each path and assign that instance to the most popular edge; 3) give instances weights and split the instance evenly down each possible edge. Once the value has reached the leaf nodes, recombine it using the weights

Question 8

Decision Trees: Definition

Answer 8

Each node in the tree tests a single attribute, decisions are represented by the edges leading away from that node with leaf nodes representing the final decision.

Question 9

Decision Trees: Example Applications

Answer 9

1: Star classification; 2: Medical diagnosis; 3: Credit risk analysis

Question 10

Decision Trees: Flavors

Answer 10

CART, ID3

Question 11

Decision Trees: Pros

Answer 11

1: Fast; 2: Robust to noise and missing values; 3: Accurate

Question 12

Decision Trees: Pruining

Answer 12

since decision trees are often created by continuously splitting the instances until there is no more information gain, it is possible that some splits were done with too little information gain and result in overfitting of the model to the training data. Basic idea is to check child nodes to see if combining them with their parent would result in an increase in entropy below some threshold

Question 13

Decision Trees: Replicated subtree problem

Answer 13

because only a single attribute is tested at a node, it is possible that two copies of the same subtree will need to be placed in a tree if the attributes from that subtree were not tested in the root node

Question 14

Decision Trees: Restriction Bias

Answer 14

…

Question 15

Decision Trees: Testing nominal values

Answer 15

if a node has edges for each possible value of that nominal value, then that value will not be tested again further down the tree. If a node groups the values into subsets with an edge per subset, then that attribute may be tested again

Question 16

Decision Trees: Testing numeric values

Answer 16

can be tested for less than, greater than, equal to, within some range. Can result in just two edges or multiple edges. Can also have an edge that represents no value. May be tested multiple times in a single path

Question 17

Deep Learning

Answer 17

Refers to a class of methods that includes neural networks and deep belief nets. Useful for finding a hierarchy of the most significant features, characteristics, and explanatory variables in complex data sets. Particularly useful in unsupervised machine learning of large unlabeled datasets. The goal is to learn multiple layers of abstraction for some data. For example, to recognize images, we might want to first examine the pixels and recognize edges; then examine the edges and recognize contours; examine the contours to find shapes; the shapes to find objects; and so on.

Question 18

Ensemble Learning

Answer 18

…

Question 19

Hidden Layer

Answer 19

The second layer of a three-layer network where the input layer sends its signals, performs intermediary processing

Question 20

K-Nearest Neighbors: Cons

Answer 20

1: Performs poorly on high-dimensionality datasets; 2: Expensive and slow to predict new instances; 3: Must define a meaningful distance function;

Question 21

K-Nearest Neighbors: Definition

Answer 21

K-NN is an algorithm that can be used when you have a objects that have been classified or labeled and other similar objects that haven’t been classified or labeled yet, and you want a way to automatically label them.

Question 22

K-Nearest Neighbors: Example Applications

Answer 22

1: Computer security: intrusion detection; 2: Fault detection in semiconducter manufacturing; 3: Video content retrieval; 4: Gene expression

Question 23

K-Nearest Neighbors: Preference Bias

Answer 23

Good for measuring distance based approximations, good for outlier detection

Question 24

K-Nearest Neighbors: Pros

Answer 24

1: Simple; 2: Powerful; 3: Lazy, no training involved; 4: Naturally handles multiclass classification and regression

Question 25

K-Nearest Neighbors: Restriction Bias

Answer 25

Low-dimensional datasets

Question 26

K-Nearest Neighbors: Type

Answer 26

Supervised learning, instance based

Question 27

KNN

Answer 27

K-NN is an algorithm that can be used when you have a bunch of objects that have been classified or labeled in some way, and other similar objects that haven’t gotten classified or labeled yet, and you want a way to automatically label them.

Question 28

Linear Regression

Answer 28

Trying to fit a linear continuous function to the data. Univariate or Multivariate.

Question 29

Linear Regression: Cons

Answer 29

1: Unable to model complex relationships, 2: Unable to capture nonlinear relationships without first transforming the inputs

Question 30

Linear Regression: Definition

Answer 30

Trying to fit a linear continuous function to the data to predict results. Can be univariate or multivariate.

Question 31

Linear Regression: Example Applications

Answer 31

1: Fitting a line

Question 32

Linear Regression: Preference Bias

Answer 32

1: Prefers continuous variables; 2: A first look at a dataset; 3: Numerical data with lots of features

Question 33

Linear Regression: Pros

Answer 33

1: Very fast - runs in constant time, 2: Easy to understand the model, 3: Less prone to overfitting

Question 34

Linear Regression: Restriction Bias

Answer 34

Low restriction on problems it can solve

Question 35

Linear Regression: Type

Answer 35

Supervised learning, regression class

Question 36

Logistic Regression

Answer 36

A kind of regression analysis often used when the dependent variable is dichotomous and scored 0 or 1. It is usually used for predicting whether something will happen or not, such as graduation, business failure, or heart attack-anything that can be expressed as event/non-event. Independent variables may be categorical or continuous in logistic regression analysis.

Question 37

Multiclass-Classification: One-vs-all

Answer 37

Multiclass classification. Reducing a classification problem with multiple features that have to be predicted to a simple classification problem by looking at one feature at a time. Then to determin the final prediction we take the max of all the predicted values.

Question 38

Naive Bayes

Answer 38

…

Question 39

Naive Bayes: Cons

Answer 39

…

Question 40

Naive Bayes: Definition

Answer 40

Given its simplicity and the assumption that the independent variables are statistically independent, Naive Bayes models are effective classification tools that are easy to use and interpret. Naive Bayes is particularly appropriate when the dimensionality of the independent space is high. For the reasons given above, Naive Bayes can often outperform other more sophisticated classification methods. A variety of methods exist for modeling the conditional distributions of the inputs including normal, lognormal, gamma, and Poisson.

Question 41

Naive Bayes: Example Applications

Answer 41

…

Question 42

Naive Bayes: Flavors

Answer 42

A variety of methods exist for modeling the conditional distributions of the inputs including normal, lognormal, gamma, and Poisson.

Question 43

Naive Bayes: Preference Bias

Answer 43

Works on problems where the inputs are independent from each other

Question 44

Naive Bayes: Pros

Answer 44

1: Easy to use and interpret; 2: Works well with high dimensional problems

Question 45

Naive Bayes: Restriction Bias

Answer 45

Prefers problems where the probability will always be greater than zero for each class

Question 46

Naive Bayes: Type

Answer 46

Supervised learning; used for classification; probabalistic approach

Question 47

Neural Networks

Answer 47

In neuronal networks the process of calculating the subsequent layers of the network. Each layer depends on the calculations done on the layer before it.

Question 48

Neural Networks

Answer 48

Interconnected neural cells. With experience, networks can learn, as feedback strengthens or inhibits connections that produce certain results. Computer simulations of neural networks show analogous learning.

Question 49

Neural Networks: Cons

Answer 49

1: Prone to overfitting; 2: Long training time; 3: Requires significant computing power for large datasets; 4: Model is essentially unreadable; 5: Work best with “homogenous” data where features all have similar meanings

Question 50

Neural Networks: Definition

Answer 50

With experience, networks can learn, as feedback strengthens or inhibits connections that produce certain results. Each layer depends on the calculations done on the layer before it.

Question 51

Neural Networks: Example Applications

Answer 51

1: Images; 2: Video; 3: “Human-intelligence” type tasks like driving or flying; 4: Robotics

Question 52

Neural Networks: Flavors

Answer 52

Deep learning

Question 53

Neural Networks: Preference Bias

Answer 53

Prefers binary inputs

Question 54

Neural Networks: Pros

Answer 54

1: Extremely powerful, can model even very complex relationships; 2: No need to understand the underlying data; 3: Almost works by “magic”

Question 55

Neural Networks: Random Initialization

Answer 55

Symmetry breaking for neural networks is achieved by:

Question 56

Neural Networks: Restriction Bias

Answer 56

Little restriction bias

Question 57

Neural Networks: Type

Answer 57

Supervised learning; nonlinear functional approximation

Question 58

Overview of algorithms

Answer 58

…

Question 59

Probabilistic Graphical Model (a.k.a. Graphical Model)

Answer 59

Ways of encoding the structure (independencies) of a probability distribution into a picture. The two main types of graphical models are directed graphical models and undirected graphical models, probability distributions represented by directed and undirected graphs respectively. Each node in the graph represents a random variable, and a connection between two nodes indicates a possible dependence between the random variables. So, for example, a fully disconnected graph would represent a fully independent set of random variables, meaning the distribution could be fully factored as P(x,y,z,…)=P(x)P(y)P(z)… Note that the graphs represent structures, not probabilities themselves.

Question 60

Random Forest

Answer 60

…

Question 61

Random Forests

Answer 61

A decision tree classifier that produces a “forest of trees”, yielding highly accurate models, essentially by iteratively randomizing one input variable at a time in order to learn if this randomization process actually produces a less accurate classifier. If it doesn’t, then that variable is ousted from the model.

Question 62

Recommendation Systems: Collaborative filtering

Answer 62

Based on past user behavior. Each user’s history of behaviors (ratings, purchases, or viewing history) is used to make associations between users with similar behavior and between items of interest to the same users. Example: Netflix. Methods: 1. Neighborhood-based methods, based on user-user or item-item distances; 2. Latent factor or reduced- dimension models, which automatically discover a small number of descriptive factors for users and items; 3. Low-rank matrix factorization is the best-known example of reduced-dimension models and is among the most flexible and successful methods underlying recommendation systems. There are many variants of matrix factorization, including probabilistic and Bayesian versions. Restricted Boltzmann machines, a type of deep learning neural network, are another state-of-the-art approach.

Question 63

Recommendation Systems: Collaborative filtering: Matrix Factorization

Answer 63

….Probabilistic and Bayesian versions, Restricted Boltzmann machines, a type of deep learning neural network, are another state-of-the-art approach.

Question 64

Recommendation Systems: Companies using them

Answer 64

Retailers: Amazon, Target; Movies + Music Sites: Netflix, last.fm, Pandora; Social networks: Facebook, Twitter; Grocery stores: Tesco; Content publishers: Ad networks: Yahoo!, Google; CRM: Next-best offer in marketing decision making

Question 65

Recommendation Systems: Content-based filtering

Answer 65

Gathers information (e.g., demographics, genre, keywords, preferences, survey responses) to generate a profile for each user or item. Users are matched to items based on their profiles. Example: Pandora’s Music Genome Project.

Question 66

Regression Analysis

Answer 66

We are trying to predict results within a continuous output, meaning that we are trying to map input variables to some continuous function.

Question 67

Regression Trees

Answer 67

a single regression equation is much smaller and less complex than a regression tree, but tends to also be much less accurate.

Question 68

Regression Trees

Answer 68

decision trees which predict numeric quantities. The leaf nodes of these trees have a numeric quantity instead of a class. This numeric quantity is often decided by taking the average of all training set values to which the leaf node applies

Question 69

Sigmoid Function

Answer 69

an S-shaped mathamatical curve is often used to describe the activation function of a neuron over time

Question 70

Stepwise Regression

Answer 70

Variable selection process for multivariate regression. In forward stepwise selection, a seed variable is selected and each additional variable is inputed into the model, but only kept if it significantly improves goodness of fit (as measured by increases in R^2). Backwards selection starts with all variables, and removes them one by one until removing an additional one decreases R^2 by a non-trivial amount. Two deficiencies of this method are that the seed chosen disproportionately impacts which variables are kept, and that the decision is made using R^2, not Adjusted R^2. (submitted by Santiago Perez)

Question 71

Supervised Learning

Answer 71

We are given a data set and already know what our correct output should look like, having the idea that there is a relationship between the input and the output. Categorized into “regression” and “classification” problems.

Question 72

Support Vector Machine

Answer 72

can extrapolate information from one dimensional data (input space) and some information about weights & correlative relationships to another dimension (feature space)

Question 73

Support Vector Machine

Answer 73

divide an instance space by finding the line that is as far as possible from both classes. This line is called the “maximum-margin hyperplane”

Question 74

Support Vector Machine

Answer 74

Powerful Jedi machine learning classifier. Among classification algorithms used in supervised machine learning, SVM usually produces the most accurate classifications. Read more about SVM in this article “The Importance of Location in Real Estate, Weather, and Machine Learning.”

Question 75

Support Vector Machine

Answer 75

when determining the maximum-margin hyperplane for a support vector machine, only the points near the hyperplane are important. These points near the boundary are called the support vectors

Question 76

Support Vector Machines: Cons

Answer 76

1: Need to select a good kernel function; 2: Model parameters are difficult to interpret; 3: Sometimes numerical stability problems; 4: Requires significant memory and processing power

Question 77

Support Vector Machines: Definition

Answer 77

Divides an instance space by finding the line that is as far as possible from both classes. This line is called the “maximum-margin hyperplane”. Only the points near the hyperplane are important. These points near the boundary are called the support vectors.

Question 78

Support Vector Machines: Example Applications

Answer 78

1: Text classification; 2: Image classification; 3: Handwriting recognition

Question 79

Support Vector Machine: Kernels

Answer 79

since support vector machines use dot-products (just like linear classifiers) when determining the hyperplane, they can be turned into a non-linear classifier by replacing the dot-product with a kernel such as the radial-basis function

Question 80

Support Vector Machine: libsvm

Answer 80

open source library for SVMs written in C++ (w/a Java version as well). Trains an SVM model, makes predictions, and tests predictions w/in a dataset with support for kernel methods such as the radial-basis function

Question 81

Support Vector Machines: Preference Bias

Answer 81

Works where there is a definite distinction between two classifications

Question 82

Support Vector Machines: Pros

Answer 82

1: Can model complex, nonlinear relationships; 2: Robust to noise (because they maximize margins)

Question 83

Support Vector Machines: Restriction Bias

Answer 83

Prefers binary classification problems

Question 84

Support Vector Machines: Type

Answer 84

Supervised learning for defining a decision boundary