Data_to_Insights

Question 1

Rand Index

Answer 1

The Rand index[1] or Rand measure (named after William M. Rand) in statistics, and in particular in data clustering, is a measure of the similarity between two data clusterings. A form of the Rand index may be defined that is adjusted for the chance grouping of elements, this is the adjusted Rand index. From a mathematical standpoint, Rand index is related to the accuracy, but is applicable even when class labels are not used.

Question 2

Rand Index

Answer 2

Measures like the Rand Index are called external evaluations,
because they require outside information about a ground truth clustering.

Question 3

Ggobi

Answer 3

GGobi is an open source visualization program for exploring high-dimensional data. It provides highly dynamic and interactive graphics such as tours, as well as familiar graphics such as the scatterplot, barchart and parallel coordinates plots. Plots are interactive and linked with brushing and identification.

Question 4

Amazon Mechanical turk

Answer 4

https://www.mturk.com/media/intro/mainbanner.gif

Question 5

Clustering

Answer 5

Grouping data according to similarity

Question 6

k-means

Answer 6

K-means clustering is typically very sensitive to outliers.

Question 7

k-medoids

Answer 7

s based on centroids (or medoids) calculating by minimizing the absolute distance between the points and the selected centroid, rather than minimizing the square distance. As a result, it’s more robust to noise and outliers than k-means.

Question 8

PCA

Answer 8

Principal Component Analysis
It is most often used when each data point contains a lot of measurements and
not all of those may be meaningful.
Or there’s a lot of covariance in the measurements.

Question 9

Clustering: ground truth

Answer 9

we believe that the ground truth is

that there’s exactly one true group that the data point belongs to.

Question 10

Feature vs Cluster

Answer 10

the underlying structure is a feature allocation instead of a clustering.
Note that this is a different use of the word feature than we saw on
previous videos.
A similar idea is to say that the data points exhibit mixed membership.

Question 11

feature allocation, admixture, mixed membership

Answer 11

capture the idea that data points can belong to multiple groups simultaneously.

Question 12

Eigenvector

Answer 12

It is most often used when each data point contains a lot of measurements and
not all of those may be meaningful.
Or there’s a lot of covariance in the measurements.

Question 13

Eigenvector

Answer 13

It is most often used when each data point contains a lot of measurements and
not all of those may be meaningful.
Or there’s a lot of covariance in the measurements.
The eigenvectors with the largest eigenvalues are the principle components.

Question 14

Volume

Answer 14

number of edges in a cluster

Question 15

LaPlacian

Answer 15

If there’s an edge, the entry is -1, otherwise it’s 0.
The entries on the diagonal are the degrees of the nodes.
The degree of a node is the number of edges that it meets.

Question 16

TF IDF format.

Answer 16

This stands for term frequency, inverse document frequency.

Question 17

Quantile regressions are great for

Answer 17

Quantile regressions are great for
determining the factors that affect the outcomes in the tails.
For example, in risk management,
we might predict the extremal conditional percentiles of Y using the information X.
This type of prediction is called the conditional value-at-risk analysis.
In medicine, we could be interested in how smoking and
other controllable factors X affect very low percentiles of infant birth weights Y.
In supply chain management, we could try to predict the inventory level for
a product that is able to meet the 90-th percentile of demand Y
given the economic conditions described by X.

Question 18

p value

Answer 18

strength of test

the larger p value, the stronger support for null hypothesis

Question 19

p - value

Answer 19

The p-value is naturally a random variable, as it is the probability of
a observing a value smaller than the observed sample average.
Since the simple average is a random variable so
is the probability of seeing the value below it.
What is the distribution of this random variable?
Well, there is a magic answer.
The distribution of the p-value is uniform in the interval between 0 & 1 when
the distribution corresponding to the measurement is continuous.

Question 20

Type 1 error

Answer 20

The error of rejecting a true hypothesis is called type 1 error,

Question 21

Type 2 error

Answer 21

Type 2 error corresponds to the case of accepting by mistake a hypothesis

Question 22

Perceptron

Answer 22

A perceptron is a function that has several inputs and one output.

Question 23

Belief Matching

Answer 23

A popular approach to do so is the belief propagation algorithm.

Question 24

spectral clustering

Answer 24

Goal is to find a projection of the data before clustering.

Question 25

Sparse methods

Answer 25

Goal is simply

keep only the most relevant side information.

Question 26

Locality-sensitive hashing

Answer 26

Part of the broader family of approximate

nearest neighbors.

Question 27

Bipartite graph

Answer 27

A graph is called bipartite if the nodes can be partitioned into two sets, so
that there are only edges between the two sets and not within a set.