probability / gradient descent

Question 1

what is gradient descent

Answer 1

numerical method for finding the input to a function f that minimizes the function

Question 2

most common use for gradient descent

Answer 2

minimizing empirical risk

Question 3

when is gradient descent guaranteed to work

Answer 3

convex function (happy face)

Question 4

convex function

Answer 4

a function f is convex if, for every a, b in the domain of f, the line segment between (a, f(a)) and (b, f(b))

Question 5

if f(t) is a function of a single variable and twice differentiable

Answer 5

f(t) is convex if and only if (d^2f)/(dt^2) (t) >= 0 for all t

Question 6

if f(t) is convex and differentiable

Answer 6

then gradient descent converges to a global minimum of f as long as the step size is small enough

Question 7

nonconvex functions and gradient descent

Answer 7

gradient descent might work, but not guaranteed

Question 8

choosing a step size

Answer 8

constant step size, alpha
t(i+1) = ti - alpha(df/dt)(ti)

Question 9

experiment

Answer 9

some process whose outcome is random

Question 10

set

Answer 10

an unordered collection of items, usually denoted with curly brackets

Question 10

sample space

Answer 10

set of all possible outcomes of an experiment

Question 11

event

Answer 11

subset of the sample space or a set of outcomes

Question 11

what do probabilities mean

Answer 11

if probability(E) = p, then if we repeat our experiment infinitely many times, the proportion of repetitions in which event E occurs is p

Question 12

the sum of the probabilities of each outcome

Answer 12

must be exactly 1

Question 12

probability distribution

Answer 12

describes the probability of each outcome s in a sample space S

Question 12

probability of each outcome

Answer 12

must be between 0 and 1

Question 12

the probability of an event

Answer 12

sum of the probabilities of the outcomes in the event

Question 13

if S (sample space) w/ n possible outcomes and all outcomes are equally likely

Answer 13

then the probability of any one outcome occurring is 1/n

Question 14

probability of an event

Answer 14

of outcomes in event/ # of outcomes in sample space

Question 15

complement rule

Answer 15

it is a complement if it contains the set of all outcome in the space that are not in the circle or A

Question 16

mutually exclusive

Answer 16

no overlap, or they both can’t happen at the same time

Question 17

if A and B are mutually exclusive, then the probability that A or B happens is:

Answer 17

P(A U B) = P(A) + P(B)
U = union or symbol for or

Question 18

principle of inclusion-exclusion

Answer 18

if A and B are any two events, then P(A U B) = P(A) + P(B) - P(A intersect B)

Question 19

the probability that events A and B both happen is

Answer 19

P(A intersect B ) = P(A)P(B|A)

<– given that

Question 20

P(B|A)

Answer 20

the probability that B happens given that A happened

Question 21

if P(B|A) = P(B)

Answer 21

then A and B are independent

Question 22

probability that events A and B both happen

Answer 22

P(A intersect B) = P(A)P(B|A)

Question 23

if P(B|A) = P(B)

Answer 23

A and B are independent

Question 24

conditional probability

Answer 24

the probability of an event may change if we have additional information about outcomes

Question 25

population

Answer 25

choosing k elements randomly from a group of n possible elements

Question 26

sequence of length k

Answer 26

obtained by selecting k elements from a group of n possible elements with replacement, ORDER MATTERS

Question 27

number of ways k elements from a group of n possible elements with replacement

Answer 27

n^k

Question 28

permutation

Answer 28

selecting k elements from a group of n possible elements without replacement

Question 29

number of ways to select k elements from a group of n possible elements without replacement and order matters

Answer 29

P(n,k) = (n)(n-1) ... (n - k + 1) = n! / (n - k)!

Question 30

combination

Answer 30

set of k elements selected from a group of n possible elements without replacement , order doesn't matter

Question 31

law of total probability

Answer 31

if A is an event and E1, E2, and Ek is a partition of S, then Probability(A intersect E1) + P(A intersect E2) + .... + P(A intersect Ek)

Question 32

Bayes Theorem

Answer 32

P(B|A) = ((P(B)P(A|B))/(P(A))