Test 1 W1-4

Question 1

What is Deep Learning?

Answer 1

A set of machine learning algorithms that model high-level abstractions in data by using model architectures (often neural networks)

Question 2

What is Machine Learning?

Answer 2

Let machines learn from data instead of writing programs from hand.

Provide examples that specify correct outputs for a given input and a machine learning algorithm takes the data and produces a program/ model that does the job (if done right, the program works for new cases as well, can change the program by training it on new data)

Question 3

Given examples of inputs and corresponding desired outputs, predict outputs on future inputs

Answer 3

Supervised Learning

Question 4

Given only inputs, automatically discover representations, features, structures, etc

Answer 4

Unsupervised Learning

Question 5

Given sequences of inputs, actions from a fixed set, and scalar rewards/ punishments, learn to select sequences in a way that maximizes expected reward

Answer 5

Reinforcement Learning

Question 6

Outputs are categorical; inputs are anything. Goal is to select correct class for new inputs.

Answer 6

Classification (Supervised Learning)

Question 7

Outputs are continuous; inputs are anything (but usually continuous values). Goal is to predict outputs accurately for new inputs.

Answer 7

Regression (Supervised Learning)

Question 8

-?- is used to scale the range of a feature

Answer 8

Feature Normalization

(need to normalize features so that the distance between two points is not governed by one particular feature with a broad range of value)

Question 9

Many classifiers calculate the distance between two points by -?-

Answer 9

The Euclidean distance

Question 10

-?- is often used as the method to find optimal solution, which converges much faster with feature scaling than without it

Answer 10

gradient descent

Question 11

What are the three phases of classification tasks.

Answer 11

Data is split accordingly to be

1. Training set
2. Validation (aha development or held-out) data set (used for model selection/ averaging)
3. Test Set

Ration ofter 60:20:20

Question 12

What is a solution for when data is limited?

Answer 12

n-fold cross validation

Question 13

What is supervised learning?

Answer 13

Given examples of inputs and corresponding desired outputs, the system predicts outputs on future inputs.

Question 14

List 3 examples of supervised learning.

Answer 14

Classification, Regression, Time Series Prediction

Question 15

What is unsupervised learning?

Answer 15

Given only inputs, automatically discover representations, features, structure, etc.

Question 16

What is classification?

Answer 16

Outputs are categorical, inputs are anything. Goal is to select correct class for new inputs.

Question 17

What is Regression?

Answer 17

Outputs are continuous, inputs are anything (but usually continuous. Goal is to predict outputs accurately for new inputs.

Question 18

What is an example of unsupervised learning?

Answer 18

Clustering.

Question 19

What type of learning is KNN?

Answer 19

Supervised learning.

Question 20

In KNN what if k is too small?

Answer 20

Over fitting occurs. A learning algorithm corresponds too closely to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably. (Overfit to noise)

Question 21

In KNN what if k is too big?

Answer 21

Underfitting occurs.
Think about if k is equal to the total number of training data points, KNN does not learn much from the data it just becomes a majority classifier

Question 22

What is leave-one-out validation?

Answer 22

When data scarce, may be appropriate to consider making the number of folds (cross validation) to be the number of data points

Question 23

What is the K-Nearest Neighbot (KNN) ‘algorithm’?

Answer 23

A generative nonparametric classification model

Training: store all data in some way
Given a test point x, find a sphere around x enclosing k points
Classify x according to the majority of the k neighbors

Question 24

What is the nearest neighbor classifier?

Answer 24

in KNN of K=1

Question 25

How to solve overfitting? How to pick k?

Answer 25

We can use validation or cross validation to select model

Question 26

Adv/ Dis of KNN?

Answer 26

Adv: Learning is cheap (just need to remember all data points)
Dis: - Prediction expensive (need to retrieve k nearest neighbors from a large set of N points, for each prediction made)
- in high dimensions, points are far away from each other (poor performance)

Question 27

Nonparametric classification methods

Answer 27

aka instance-based/ case-based/ memory-based method. Number of model parameters grows with number of training cases/data points
(example is KNN)

Question 28

Parametric classification methods

Answer 28

aka model-based methods

Number of parameters is fixed

Question 29

What are two basic ideas for parametric models?

Answer 29

1. Linear classifiers

2. Decision Trees

Question 30

Linear classifiers goal

Answer 30

To find the line (or hyperplane) which can “best” (under some criterion/ objective) separate two classes

Question 31

Goal of Perceptron?

Answer 31

minimize an error function known as perceptron criterion (which associates a zero error with any data point correctly classified)

(Seeks a weight vector such that a pattern X in class C1 will have wTφ(xn) > 0 and in C2 will have wTφ(xn) < 0. When a data point is misclassified it’s feature vector is added to the current weight vector giving a new decision boundary. and the true class label ‘t’ takes the value +1 for C1 and -1 for C2)

Question 32

Optimization algorithm, -?- is often applied to minimize the errors, which updates model parameters at each training data point.

Answer 32

stochastic gradient descent (SGD) (aka online algorithm)

Question 33

What does the perceptron convergence theorem state?

Answer 33

if there exists an exact solution (if the training data is linearly separable) then the perceptron learning algorithm is guaranteed to find an exact solution in a finite number of steps.

If the data is not linearly seperable, perceptron will not converge

Question 34

What is the Decision Trees Alg

Answer 34

Parametric Model
- a classification tree, where (1) each internal node represents a "test" on variable/ feature; (2) each branch represents the outcome of the test, and (3) each leaf node has a class label

Question 35

What is Decision Tree Training?

Answer 35

(Greedily) search features, choose one that splits data in order to reduce impurity most, which is measure with:

• GINI
• Entropy
• Misclassification Errors

Question 36

Generative vs. Discriminative models

Answer 36

Generative: Use Bayes' theorem to find the posterior class probability p(Ck|x).
(by sampling from the joint distribution possible to generate synthetic/ unseen data points in the input space)

Discriminative: Learn p(Ck|x) directly by just using a discriminant function to map each input x directly onto a class label, where probabilities play no role

Question 37

Decision THeory

Answer 37

Used for both Generative/ Discriminative models to choose a class label.

Tells us how to make optimal decisions given the appropriate probabilities (minimize the chance of assigning x to the wrong class)

Question 38

What is leave-one-out validation?

Answer 38

When data is scarce, consider making the number of folds to be the number of data points

Question 39

What is an implementation issue of KNN?

Answer 39

Prediction is expensive. You need to retrieve k nearest neighbours from a large set of N points.

Question 40

What is the goal of a decision tree?

Answer 40

Minimize expected sum of impurity at leaves.

Question 41

Gaussian Classifier

Answer 41

A generative model (with continuous input)

Closed-form solution in which you obtain parameters using maximal likelihood estimation.
Need to manipulate mean vectors and co-variance matrixes when extending from univariate to multivariate gaussians

Question 42

Naive Bayes Classifier

Answer 42

A generative model (with discrete input)

Question 43

Naive Bayes Classifier Assumption?

Answer 43

conditioned on class, features/ variables are independent

Question 44

How do we perform Naive Bayes on continuous input e.g.., in a Gaussian classifier?

Answer 44

Gaussian Bayes Classifier

Has a separate, diagonal covariance matrix for each class

Question 45

logistic regression models are:

Answer 45

Discriminative

- have the same number of adjustable parameters as dimensions of the features space

Question 46

For a large value of M, (in an M-dimensional feature space) what is the best model to use?

Answer 46

Logistic regression (over Gaussian classifiers)

Question 47

What do we use to allow for direct calculations in an extended feature space?

Answer 47

use the “kernel trick”

Question 48

What is the key idea of Kernel methods?

Answer 48

reduce an algorithm to one which depends only on dot products between data vectors. Then replace the dot product with a kernel function k(x, z)

Question 49

a kernel machine

Answer 49

contains (two modules): the algorithm and the kernel function

take a standard algorithm and massage it so that all references to the original data vectors x appear only in dot products x(^T)z

Question 50

an N by N (N = # of training data points) symmetric matrix of all pairwise kernel evaluations

Answer 50

The “Gram Matrix”

can build and throw away original data if “kernelized” an algorithm successfully

Question 51

a support vectore machine (SVM) is nothing more than a:

Answer 51

kernelized maximum-margin hyperplane classifier

Question 52

For linearly seperable data, if all the hyperplaces that separate the data without misclassifying any data points, pick the hyperplane that maximizes the margin

Answer 52

maximum margin principle

Question 53

when multi dimensional vectors have have a linear boundary they share the..

Answer 53

same coverence matrix (Gaussian classifier)

Question 54

A supervised task with continuous output

Answer 54

regression

Question 55

Regression: what should we use for alpha?

Answer 55

Depends on your error function

• For squarred error e, the mean is best
• For abstract error, we get the median (absolute error)

Question 56

constant model

Answer 56

(regression) says y = t, independent of x -> biggest problem for regression

Question 57

What models out o the following is/are discriminate model(s)?

a. Naive Bayes
b. Logistic Regression
c. Perceptron
d. Gaussian Classifiers

Answer 57

Logistic regression, Perceptron

Question 58

We know that SVM learns parameters W and b so that:

argmax{(1/ ||w||)min[(t(subscript n))(W^T θ(x(subscript n)) + b)]}

(The notations here are same as those we used in the lecture)
To convert the above optimization problem to an easier form, we need to enforce all support vectors to satisfy:
t(subscript n)(w^T θ(x(subscript n)) + b) = 1

True or False

Answer 58

True

Question 59

Which statement(s) about validation and cross-validation is/are true?

a. A validation set can be used to prevent overfitting
b. In some situation you should use validation but not cross validation e.g., when training takes too long (month)
c. To obtain reliable models, you should use the entire training data to train your model and then use the entire training data to perform validation

Answer 59

a. A validation set can be used to prevent overfitting

b. In some situation you should use validation but not cross validation e.g., when training takes too long (month)

Question 60

To run Naive Bayes on continuous features, ——–

a. You must transform the continuous features to categorical features
b. You must transform the features to the range [0, 1] using scaling
c. You just need to round the features to the nearest integers
d. Naive Bayes can work on continuous features directly

Answer 60

d. Naive Bayes can work on continuous features directly

Question 61

If you have trained a Logistic Regression model and got the following hyperplane:
(w^T)x = (0.3, 0.5, -1.5)(1, x1, x2)^T
During test, this can be used to compute p(C1|x) with a logistic sigmoid function acting on it.
If you are given a data point x = (x1 = 5, x2 = 2), which of the following prediction is correct.
(For reference the sigmoid function is:
sigmoid(a) = 1/(1+ exp(-a))

a. p(c1 | x) > 0, so predict x to be c1
b. p(c1 | x) > 0.5, so predict x to be c1
c. p(c1 | x) < 0, so predict x to be c2
d. p(c1 | x) < 0.5, so predict x to be c2

Answer 61

d. p(c1 | x) < 0.5, so predict x to be c2

Question 62

Which one of the following statements is NOT true about support vector machines?

a. The kernel trick allows SVM to classify non linearly separable data
b. SVM tries to maximize the margin between two classes
c. During test phase, SVM needs to compute the dot product between a test data point and all training data points
d. SVM is a discriminative model

Answer 62

c. During test phase, SVM needs to compute the dot product between a test data point and all training data points