Classification and regression

Question 1

classification

Answer 1

predicts discrete class labels

Question 2

example of classification

Answer 2

labelling emails spam or ham

Question 3

decision tree classifier

Answer 3

flowchart-like structure in which each node represents a “test” on an attribute (e.g. whether a coin flip comes up heads or tails), each branch represents the outcome of the test, and each leaf node represents a class label tree like model that makes decisions by splitting data into subsets based on feature values creating branches that lead to outcomes (class labels)

Question 4

decision tree makes a sequence

Answer 4

of partitions of training data one attribute at a time

Question 5

probability in classification

Answer 5

probability helps determines likelihood of each class level given a set of features
relates to confidence in predictions

Question 6

ordering in classification

Answer 6

attributes are selected and split based on a measure like information gain creating an order of importance for features

Question 7

entropy

Answer 7

entropy is a measure of uncertainty or disorder in a system

Question 8

info entropy in classification

Answer 8

entropy measures how hard it is to guess the label of a randomly taken sample from dataset

Question 9

choose level with ___ entropy as ___

Answer 9

lowest
as the data labels are more uniform so its easy to guess

Question 10

how is entropy used in data splits for decision trees?

Answer 10

decision trees use information gain based on entropy to decide best feature to split the data at each node
entropy is calculated before and after split to determine how well a feature divides that data into pure sets

Question 11

3 steps of entropy and data splits

Answer 11

1) partition example recursively by choosing one attribute at a time
2) choose attribute based on which attribute can separate classes of training examples best
3) choose goodness function (info gain, gain ratio, gini ratio)

Question 12

3 attribute types

Answer 12

nominal (categorical values with no order like animal, food)
ordinal (categorical values that have order like hot, warm, cold)
numerical

Question 13

how do you handle numerical attribute in decision tree? and 3 ways you can?

Answer 13

convert to a nominal attribute
1) assign category to numerical and keep trying until you find a good split
2) use entropy value till you find the best split
3) frequency bining

Question 14

attribute resulting in ____ info gain is selected for split

Answer 14

highest

Question 15

process of splitting decision tree by attribtiues is continued recursively ____

Answer 15

building tree by splitting data using features that minimise uncertainty at each step

Question 16

Th is the

Answer 16

entropy threshold

Question 17

What is the purpose of Th

Answer 17

criterion for deciding when to stop splitting the data at a node or to continue

Question 18

When entropy of a node is below Th?

Answer 18

If the entropy of a node is below a certain threshold, it means that the data at that node is sufficiently pure (i.e., it mostly contains examples of one class). As a result, the decision tree can stop splitting further at that node, and the node is labeled with the majority class

Question 19

When entropy of a node is above Th?

Answer 19

If the entropy is above the threshold, it indicates that the data at the node is still impure, meaning there’s a mix of different class labels. In this case, the decision tree continues splitting by choosing the attribute that reduces entropy the most (maximizing information gain)

Question 20

only use Th=0 when

Answer 20

example is really simple

Question 21

Th=0, Th>0

Answer 21

=0 perfect order
>1 can tolerate some mixed levels

Question 22

avoid overfitting by using 1) and 2) and 3)

Answer 22

entropy threshold
pruning
limit depth of tree

Question 23

gain ratio formula

Answer 23

information gain A/ (#A x A entropy)

Question 24

want big or small gain ratio and why?

Answer 24

small as prevents selecting attributes that overfit the model by using many small, specific splits

Question 25

gini index doesn't rely on

Answer 25

entropy only on class proportion

Question 26

when you would use info gain as goodness function?

Answer 26

imbalanced dataset

Question 27

when you would use gain ratio as goodness function?

Answer 27

imbalanced dataset high brand attribute

Question 28

when you would use gini index as goodness function?

Answer 28

binary classification

Question 29

rank fastest to slowest for goodness function evaluation?

Answer 29

fastest gini, middle if IG, slowest is gain ratio

Question 30

perceptron is an

Answer 30

artificial neuron fundamental unit in neural networks modelled after biological neuron activity is weighted sum of its inputs + bias term passed through an activation function to produce output adjusting weights allows neuron to learn choice of activation function determines type of computation the neuron performs

Question 31

single neuron vs multiple computation ability wise

Answer 31

single neuron can only do simple computations but many connected in a large network can deliver any function mapping

Question 32

what is the activation function symbol

Answer 32

like a hook

Question 33

activation function

Answer 33

determines output of neuron based on weighted sumo inputs introduces non-linearity to make the model network capable of learning more complex patterns e.g sigmoid, relu, softmax

Question 34

weights

Answer 34

coefficients that adjust the influence of certain input attributes on the output

Question 35

bias

Answer 35

threshold value added to the sum of weighted inputs to shift the activation function's output

Question 36

why is the bias helpful?

Answer 36

shifts decision boundary away from the origin making the model more flexible as without the decision boundary would always pass through the origin

Question 37

half space

Answer 37

space divided by hyperplane which classifies the data points based on which side they fall on

Question 38

one hot encoding

Answer 38

converts categorical data variables into a numerical format that machine learning models can use

Question 39

binary classification vs multi class classification

Answer 39

binary classifies data into 1 of 2 classes multi classifies data into 1 of many classes

Question 40

how does multi class classification work?

Answer 40

uses K neuron's and trains each to separate one class from all others

Question 41

how is training done on a perceptron?

Answer 41

iteratively updating weights in a way that minimises error function (difference between actual and predicted output)

Question 42

how is a perceptron trained?

Answer 42

the goal is to learn a set of weights that allows the perceptron to correctly classify input data trained through supervised learning

Question 43

a perceptron model is guaranteed to converge if

Answer 43

1) learning parameters= small enough 2) linearly separable classification

Question 44

learning rate

Answer 44

controls magnitude of weight updates during training

Question 45

a too high learning rate can mean

Answer 45

computation is faster but can lead to potentially unstable training

Question 46

epoch is a

Answer 46

entire passing of training data through the algorithm

Question 47

online training

Answer 47

one update per training sample N updates per epoch

Question 48

batch learning

Answer 48

average updates from all training samples one update per epoch

Question 49

mini batch learning

Answer 49

dividing data into fixed size batches and taking the average over mini batches and shuffling data assigned to mini batches between epochs

Question 50

limit of perceptron model and how is it overcome?

Answer 50

linear separation simple single perceptron can only solve linearly separable problems so can't handle XOR problems in more complex neural networks MLP

Question 51

MLP is

Answer 51

forward feed artificial neural network that generates a set of outputs from a set of inputs with multiple hidden laters to allow model more complex problems and capture non linear patterns

Question 52

MLP structure

Answer 52

input layer each neuron in the input layer corresponds to one feature in the input data hidden layers can have 1 or more Each neuron in a hidden layer is connected to every node in the previous and next layer (fully connected) Neurons in the hidden layers apply non-linear activation functions to the weighted sum of inputs and produce an output learn abstract features from input data output layer provides the final prediction of the mode

Question 53

number of input layers is the number of

Answer 53

features

Question 54

how does a MLP learn

Answer 54

adjusting the weights between neurons to minimize error and is able to capture complex patterns in data due to its layered architecture

Question 55

SLP

Answer 55

one layer of neutrons can only solve linearly separable problems

Question 56

3 advantages to MLP

Answer 56

can solve non-linearly separable problems can model more complex decision boundaries and patterns in the data by stacking ML of neutrons can learn hierarchical features where each hidden layer captures different levels of abstraction from the data

Question 57

universal function approximation

Answer 57

technically single layer perceptron is model that can approximate any continuous function given enough neutrons and layers

Question 58

ufa in classification and regression

Answer 58

can produce desired class labelling on any data hypothesis that fits any day with arbitrarily small MSE

Question 59

3 non linear activation functions

Answer 59

relu sigmoid tanh

Question 60

relu (what, computation expense, pros and cons)

Answer 60

outputs input directly if its positive and gives it value of 0 if not no exponent so computationally efficient pros: most widely used, no vanishing/exploding gradient, not 0 centred output cons: incorrect mapping for negative values, dead relu

Question 61

what is dead relu and what's an attempted solution?

Answer 61

if input is bounded between 0 and 1 large gradient updates can cause bias towards large negative values difficult to recover as gradient of 0 is 0 leaky relu is an attempt to fix it and it says that if value is +ve just gives it its own value otherwise gives a* value

Question 62

sigmoid (what, computation expense, pros and cons) and what formula and graph roughly looks like

Answer 62

predicts probability as squashes real value between 0 and 1 exponent so expensive pros: guarantees gradient cannot grow pass certain bound cons: gradient is bounded so can get vanishing gradient, outputs are not 0 centred so all neutrons have same sing in training 1/ 1+ e^-v s from bottom to top and bottom in line with 0

Question 63

tanh (what, computation expense, pros and cons) and what formula roughly looks like

Answer 63

squashes values between -1 and 1 exponent is computationally expensive gradient is steeper than sigmoid pros: outputs are 0 centres so means faster learning cons: gradient is bounded so vanishing gradient got the most e's ion the formula and looks like s from bottom to top but bottom in line with -1 not 0

Question 64

which activation function to use?

Answer 64

use reLu binary classification use sigmoid for output layer multi class classification use softmax for output layer

Question 65

3 weight initialisation techniques

Answer 65

set =0 neural network acts as linear model choose randomly can lead to vanishing/exploding gradient but OK with reLu heuristic but x random w to some value to avoid grad stuff

Question 66

softmax

Answer 66

converts vector of K real numbers into a probability distribution of K possible outcomes

Question 67

backpropogation

Answer 67

an algorithm used to train MLP by computing gradient of loss function with respect to each weights in network then systematically propagates error backwards from output to all preceding layers

Question 68

forward pass back propagation

Answer 68

input passes through the network layer by layer and output is computed

Question 69

loss calculation in back propagation

Answer 69

computes difference between y hat with y and computes error

Question 70

backward pass back propagation

Answer 70

error flows backward through network layer by layer computing gradient (partial derivative of loss function with respect to weight) using chain rule Update the weights by moving in the opposite direction of the gradient

Question 71

back propagation allows

Answer 71

network to learn from its mistakes by adjusting weights and biases based on error

Question 72

steepest gradient decent

Answer 72

optimization method that uses the gradients computed by backpropagation to update the weights, aiming to minimize the loss function over time

Question 73

choice of output layer act func, activation and loss function for MLP for classification

Answer 73

output layer (multi class classification= softmax, binary= sigmoid (or tanh)) softmax takes too long for binary loss function: cross entropy activation function: reLu

Question 74

choice of output layer act func, activation and loss function for MLP for regression

Answer 74

output later: linear loss function: MSE (more sensitive to outliers) or MAE activation function: reLu or tanh in hidden layers

Question 75

impact of architecture on complexity and capability of MLP

Answer 75

more hidden layers allows MLP to model more complex relationships but also increase risk of overfitting

Question 76

5 steps of supervised learning and MLP

Answer 76

1) examine - how many inputs - how many outputs - what type is desired output (classification or regression) 2) # hidden layers - usually number attributes 3) activation functions of hidden layers decide 4) for each hidden and output layer - initialise w (not all 0) - initialise bais matrices (all 0 is ok) 5) train network on training data test performance on test data

Question 77

example of MLP and requirements

Answer 77

MLP with one hidden layer is classified as a universal function approxiamtor with enough neurons in the hidden layer and a sensible non linear activation function

Question 78

filtering and expanding

Answer 78

filtering is finding only key features and expanding is feature representation

Question 79

convert pixels to

Answer 79

grey scale as don't want model to learn the colours

Question 80

logistic regression

Answer 80

has 2 modifications to normal regression model to make it suitable for binary classification problem where y E {0,1} output of regression model passed through a sigmoid function to convert it to a continuous value between 0 and 1 with that being the probability of y hat being 1 class

Question 81

how does the sigmoid function work with logistic regression

Answer 81

the output is passed through a sigmoid function which converts the number into a probability between 0-1 with it being the probability of belonging to class 1 that number is checked with a hard limiting function P>0.5 then class 1 and P<0.5 class 0

Question 82

cross entropy loss

Answer 82

measures difference between predicted probability distribution and actual class labels penalises incorrect classifications more heavily when model is confident in wrong prediction

Question 83

feature space

Answer 83

set of all possible values for a chosen set of features from chosen data decision boundaries drawn in this feature space to separate classes based on the features

Question 84

degree of polynomial and model complexity

Answer 84

increase degree of polynomial increases flexibility of the model however could overfit

Question 85

steepest gradient descent

Answer 85

optimization technique that iteratively adjusts model parameters by following the steepest descent direction of the loss surface, aiming to minimize the error

Question 86

steepest gradient descent only works with

Answer 86

continuous loss function

Question 87

optimisation in ML

Answer 87

goal is to find set of parameters that minimises loss function similar to parameter search through a space of possible parameter values optimisation allows algorithm to learn and adapt

Question 88

loss surface

Answer 88

given constant dataset loss J evaluated on some h(x,w) for some choice of parameters w as a function J(w) in the space of all possible parameters. manifold J(w) makes in parameter space = loss surface

Question 89

linear in parameters and not

Answer 89

independent variables have linear relationship with output y hat= w1g1+w2g1 and also y hat= w1sinx etc but not for y hat= sin(x1w1) as w1 has sin relationship to y hat

Question 90

convex loss function

Answer 90

has one minimum (global) and no local minima guarantees gradient based optimization methods will always find best solution

Question 91

global minima

Answer 91

post on loss surface where loss is at its absolute minimum

Question 92

local minima

Answer 92

point of loss surface where loss is lower than neighbouring but not necessary lowest overall

Question 93

limitations to SGD

Answer 93

can get stuck in local minima for none-convex functions if gradient is near 0 the descent slows down significantly finding right learning rate is critical to performance

Question 94

GA for learning parameters

Answer 94

genetic algorithm is a search heuristic reflects process of natural selcetion can be used to learn parameters of a model especially when loss surface is non-convex or sgd struggles with local min

Question 95

regression

Answer 95

predicting numerical values

Question 96

linear regression

Answer 96

model relationship between dependant (output) and 1 or more indepdant variables (inputs) using a straight line uses model hypothesis that is weighted sum of inputs

Question 97

goal of regression

Answer 97

goal is to find optimal weights that minimise differences between predicted values and actual values using loss function

Question 98

MSE and formula

Answer 98

average squared difference between predicted and actual values JMSE= 1/N sum of y-y hat

Question 99

why do we care about MSE when using it for our model?

Answer 99

minimising MSE increases accuracy of regression model by decreasing prediction error

Question 100

least squares fit

Answer 100

maths procedure for finding best fitting curve to given set of points of minimising sum of squared residuals weights are adjusted iteratively using steepest gradient descent to find optimal values to minimise loss