Classification

We are trying to predict results in a discrete output. In other words, we are trying to map input variables into discrete categories.

Classification: Linear Binary

Binary classification problems arise when we seek to separate two sets of data points in R^n, each corresponding to a given class. We seek to separate the two data sets using simple ‘‘boundaries’’, typically hyperplanes. Once the boundary is found, we can use it to predict the class a new point belongs to, by simply checking on which side of the boundary it falls.

Decision Trees

A tree of questions to guide an end user to a conclusion based on values from a single vector of data. The classic example is a medical diagnosis based on a set of symptoms for a particular patient. A common problem in data science is to automatically or semi-automatically generate decision trees based on large sets of data coupled to known conclusions. Example algorithms are CART and ID3. (Submitted by Michael Malak)

Decision Trees

each node in the tree tests an attribute, decisions are represented by the edges leading away from that node with leaf nodes representing the final decision for all instances that reach that leaf node

Decision Trees: Best Uses

…

Decision Trees: Cons

1: Complex trees are hard to interpret; 2: Duplication within the same sub-tree is possible

Decision Trees: Dealing with missing values

1) have a specific edge for no value; 2) track the number of instances that follow each path and assign that instance to the most popular edge; 3) give instances weights and split the instance evenly down each possible edge. Once the value has reached the leaf nodes, recombine it using the weights

Decision Trees: Definition

Each node in the tree tests a single attribute, decisions are represented by the edges leading away from that node with leaf nodes representing the final decision.

Decision Trees: Example Applications

1: Star classification; 2: Medical diagnosis; 3: Credit risk analysis

Decision Trees: Flavors

CART, ID3

Decision Trees: Pros

1: Fast; 2: Robust to noise and missing values; 3: Accurate

Decision Trees: Pruining

since decision trees are often created by continuously splitting the instances until there is no more information gain, it is possible that some splits were done with too little information gain and result in overfitting of the model to the training data. Basic idea is to check child nodes to see if combining them with their parent would result in an increase in entropy below some threshold

Decision Trees: Replicated subtree problem

because only a single attribute is tested at a node, it is possible that two copies of the same subtree will need to be placed in a tree if the attributes from that subtree were not tested in the root node

Decision Trees: Restriction Bias

…

Decision Trees: Testing nominal values

if a node has edges for each possible value of that nominal value, then that value will not be tested again further down the tree. If a node groups the values into subsets with an edge per subset, then that attribute may be tested again

Decision Trees: Testing numeric values

can be tested for less than, greater than, equal to, within some range. Can result in just two edges or multiple edges. Can also have an edge that represents no value. May be tested multiple times in a single path

Deep Learning

Refers to a class of methods that includes neural networks and deep belief nets. Useful for finding a hierarchy of the most significant features, characteristics, and explanatory variables in complex data sets. Particularly useful in unsupervised machine learning of large unlabeled datasets. The goal is to learn multiple layers of abstraction for some data. For example, to recognize images, we might want to first examine the pixels and recognize edges; then examine the edges and recognize contours; examine the contours to find shapes; the shapes to find objects; and so on.

Ensemble Learning

…

Hidden Layer

The second layer of a three-layer network where the input layer sends its signals, performs intermediary processing

K-Nearest Neighbors: Cons

1: Performs poorly on high-dimensionality datasets; 2: Expensive and slow to predict new instances; 3: Must define a meaningful distance function;

K-Nearest Neighbors: Definition

K-NN is an algorithm that can be used when you have a objects that have been classified or labeled and other similar objects that haven’t been classified or labeled yet, and you want a way to automatically label them.

K-Nearest Neighbors: Example Applications

1: Computer security: intrusion detection; 2: Fault detection in semiconducter manufacturing; 3: Video content retrieval; 4: Gene expression

K-Nearest Neighbors: Preference Bias

Good for measuring distance based approximations, good for outlier detection

K-Nearest Neighbors: Pros

1: Simple; 2: Powerful; 3: Lazy, no training involved; 4: Naturally handles multiclass classification and regression

K-Nearest Neighbors: Restriction Bias

Low-dimensional datasets

K-Nearest Neighbors: Type

Supervised learning, instance based

KNN

K-NN is an algorithm that can be used when you have a bunch of objects that have been classified or labeled in some way, and other similar objects that haven’t gotten classified or labeled yet, and you want a way to automatically label them.

Linear Regression

Trying to fit a linear continuous function to the data. Univariate or Multivariate.

Linear Regression: Cons

1: Unable to model complex relationships, 2: Unable to capture nonlinear relationships without first transforming the inputs

Linear Regression: Definition

Trying to fit a linear continuous function to the data to predict results. Can be univariate or multivariate.

Linear Regression: Example Applications

1: Fitting a line

Linear Regression: Preference Bias

1: Prefers continuous variables; 2: A first look at a dataset; 3: Numerical data with lots of features

Linear Regression: Pros

1: Very fast - runs in constant time, 2: Easy to understand the model, 3: Less prone to overfitting

Linear Regression: Restriction Bias

Low restriction on problems it can solve

Linear Regression: Type

Supervised learning, regression class

Logistic Regression

A kind of regression analysis often used when the dependent variable is dichotomous and scored 0 or 1. It is usually used for predicting whether something will happen or not, such as graduation, business failure, or heart attack-anything that can be expressed as event/non-event. Independent variables may be categorical or continuous in logistic regression analysis.

Multiclass-Classification: One-vs-all

Multiclass classification. Reducing a classification problem with multiple features that have to be predicted to a simple classification problem by looking at one feature at a time. Then to determin the final prediction we take the max of all the predicted values.

Naive Bayes

…

Naive Bayes: Cons

…

Naive Bayes: Definition

Given its simplicity and the assumption that the independent variables are statistically independent, Naive Bayes models are effective classification tools that are easy to use and interpret. Naive Bayes is particularly appropriate when the dimensionality of the independent space is high. For the reasons given above, Naive Bayes can often outperform other more sophisticated classification methods. A variety of methods exist for modeling the conditional distributions of the inputs including normal, lognormal, gamma, and Poisson.

Naive Bayes: Example Applications

…

Naive Bayes: Flavors

A variety of methods exist for modeling the conditional distributions of the inputs including normal, lognormal, gamma, and Poisson.

Naive Bayes: Preference Bias

Works on problems where the inputs are independent from each other

Naive Bayes: Pros

1: Easy to use and interpret; 2: Works well with high dimensional problems

Naive Bayes: Restriction Bias

Prefers problems where the probability will always be greater than zero for each class

Naive Bayes: Type

Supervised learning; used for classification; probabalistic approach

Neural Networks

In neuronal networks the process of calculating the subsequent layers of the network. Each layer depends on the calculations done on the layer before it.

Neural Networks

Interconnected neural cells. With experience, networks can learn, as feedback strengthens or inhibits connections that produce certain results. Computer simulations of neural networks show analogous learning.

Neural Networks: Cons

1: Prone to overfitting; 2: Long training time; 3: Requires significant computing power for large datasets; 4: Model is essentially unreadable; 5: Work best with “homogenous” data where features all have similar meanings

Neural Networks: Definition

With experience, networks can learn, as feedback strengthens or inhibits connections that produce certain results. Each layer depends on the calculations done on the layer before it.

Neural Networks: Example Applications

1: Images; 2: Video; 3: “Human-intelligence” type tasks like driving or flying; 4: Robotics

Neural Networks: Flavors

Deep learning

Neural Networks: Preference Bias

Prefers binary inputs

Neural Networks: Pros

1: Extremely powerful, can model even very complex relationships; 2: No need to understand the underlying data; 3: Almost works by “magic”

Neural Networks: Random Initialization

Symmetry breaking for neural networks is achieved by:

Neural Networks: Restriction Bias

Little restriction bias

Neural Networks: Type

Supervised learning; nonlinear functional approximation

Overview of algorithms

…

Probabilistic Graphical Model (a.k.a. Graphical Model)

Ways of encoding the structure (independencies) of a probability distribution into a picture. The two main types of graphical models are directed graphical models and undirected graphical models, probability distributions represented by directed and undirected graphs respectively. Each node in the graph represents a random variable, and a connection between two nodes indicates a possible dependence between the random variables. So, for example, a fully disconnected graph would represent a fully independent set of random variables, meaning the distribution could be fully factored as P(x,y,z,…)=P(x)P(y)P(z)… Note that the graphs represent structures, not probabilities themselves.

Random Forest

…

Random Forests

A decision tree classifier that produces a “forest of trees”, yielding highly accurate models, essentially by iteratively randomizing one input variable at a time in order to learn if this randomization process actually produces a less accurate classifier. If it doesn’t, then that variable is ousted from the model.

Recommendation Systems: Collaborative filtering

Based on past user behavior. Each user’s history of behaviors (ratings, purchases, or viewing history) is used to make associations between users with similar behavior and between items of interest to the same users. Example: Netflix. Methods: 1. Neighborhood-based methods, based on user-user or item-item distances; 2. Latent factor or reduced- dimension models, which automatically discover a small number of descriptive factors for users and items; 3. Low-rank matrix factorization is the best-known example of reduced-dimension models and is among the most flexible and successful methods underlying recommendation systems. There are many variants of matrix factorization, including probabilistic and Bayesian versions. Restricted Boltzmann machines, a type of deep learning neural network, are another state-of-the-art approach.

Recommendation Systems: Collaborative filtering: Matrix Factorization

….Probabilistic and Bayesian versions, Restricted Boltzmann machines, a type of deep learning neural network, are another state-of-the-art approach.

Recommendation Systems: Companies using them

Retailers: Amazon, Target; Movies + Music Sites: Netflix, last.fm, Pandora; Social networks: Facebook, Twitter; Grocery stores: Tesco; Content publishers: Ad networks: Yahoo!, Google; CRM: Next-best offer in marketing decision making

Recommendation Systems: Content-based filtering

Gathers information (e.g., demographics, genre, keywords, preferences, survey responses) to generate a profile for each user or item. Users are matched to items based on their profiles. Example: Pandora’s Music Genome Project.

Regression Analysis

We are trying to predict results within a continuous output, meaning that we are trying to map input variables to some continuous function.

Regression Trees

a single regression equation is much smaller and less complex than a regression tree, but tends to also be much less accurate.

Regression Trees

decision trees which predict numeric quantities. The leaf nodes of these trees have a numeric quantity instead of a class. This numeric quantity is often decided by taking the average of all training set values to which the leaf node applies

Sigmoid Function

an S-shaped mathamatical curve is often used to describe the activation function of a neuron over time

Stepwise Regression

Variable selection process for multivariate regression. In forward stepwise selection, a seed variable is selected and each additional variable is inputed into the model, but only kept if it significantly improves goodness of fit (as measured by increases in R^2). Backwards selection starts with all variables, and removes them one by one until removing an additional one decreases R^2 by a non-trivial amount. Two deficiencies of this method are that the seed chosen disproportionately impacts which variables are kept, and that the decision is made using R^2, not Adjusted R^2. (submitted by Santiago Perez)

Supervised Learning

We are given a data set and already know what our correct output should look like, having the idea that there is a relationship between the input and the output. Categorized into “regression” and “classification” problems.

Support Vector Machine

can extrapolate information from one dimensional data (input space) and some information about weights & correlative relationships to another dimension (feature space)

Support Vector Machine

divide an instance space by finding the line that is as far as possible from both classes. This line is called the “maximum-margin hyperplane”

Support Vector Machine

Powerful Jedi machine learning classifier. Among classification algorithms used in supervised machine learning, SVM usually produces the most accurate classifications. Read more about SVM in this article “The Importance of Location in Real Estate, Weather, and Machine Learning.”

Support Vector Machine

when determining the maximum-margin hyperplane for a support vector machine, only the points near the hyperplane are important. These points near the boundary are called the support vectors

Support Vector Machines: Cons

1: Need to select a good kernel function; 2: Model parameters are difficult to interpret; 3: Sometimes numerical stability problems; 4: Requires significant memory and processing power

Support Vector Machines: Definition

Divides an instance space by finding the line that is as far as possible from both classes. This line is called the “maximum-margin hyperplane”. Only the points near the hyperplane are important. These points near the boundary are called the support vectors.

Support Vector Machines: Example Applications

1: Text classification; 2: Image classification; 3: Handwriting recognition

Support Vector Machine: Kernels

since support vector machines use dot-products (just like linear classifiers) when determining the hyperplane, they can be turned into a non-linear classifier by replacing the dot-product with a kernel such as the radial-basis function

Support Vector Machine: libsvm

open source library for SVMs written in C++ (w/a Java version as well). Trains an SVM model, makes predictions, and tests predictions w/in a dataset with support for kernel methods such as the radial-basis function

Support Vector Machines: Preference Bias

Works where there is a definite distinction between two classifications

Support Vector Machines: Pros

1: Can model complex, nonlinear relationships; 2: Robust to noise (because they maximize margins)

Support Vector Machines: Restriction Bias

Prefers binary classification problems

Support Vector Machines: Type

Supervised learning for defining a decision boundary