General guidelines and submission

Basic instructions are the same as in MP 1. To summarize:

Problem Statement

This dataset consists of 10000 32x32 colored images total. We have split this data set for you into 2500 development examples and 7500 training examples. The images have their RGB values scaled to range 0-1. This is a subset of the CIFAR-10 dataset, provided by Alex Krizhevsky.

The perceptron model is a linear function that tries to separate data into two or more classes. It does this by learning a set of weight coefficients \(w_i\) and then adding a bias \(b\). Suppose you have features \(x_1,\dots,x_n\) then this can be expressed in the following fashion: \[ f_{w,b} (x) = \sum_{i=1}^n w_i x_i + b \] You will use the perceptron learning algorithm to find good weight parameters \(w_i\) and \(b\) such that \(\mathrm{sign}(f_{w,b}(x)) > 0\) when there is an animal in the image and \(\mathrm{sign}(f_{w,b}(x)) \leq 0\) when there is a no animal in the image. Note that in our case, we have 3072 features because each image is 32x32 and they each have RGB color channels yielding 32*32*3 = 3072.

Training and Development

Please see the textbook and lecture notes for the perceptron algorithm. You will be using a single classical perceptron whose output is either +1 or -1 (i.e. sign/step activation function).

• Training: To train the perceptron you are going to need to implement the perceptron learning algorithm on the training set. Each pixel of the image is a feature in this case. Be sure to initialize weights and biases to zero.
  Note: To get full points on the autograder, use a constant learning rate (no decay) and do not shuffle the training data.

• Development: After you have trained your perceptron classifier, you will have your model decide whether or not images in the development set contain animals in them or not. In order to do this take the sign of the function \(f_{w,b}(x)\). If it is negative then classify as \(0\). If it is positive then classify as \(1\).

Logistic regression is another linear model that tries to separate data into two or more classes. It does this by learning a set of weight coefficients \(w_i\) and then adding a bias \(b\). Suppose you have features \(x_1,\dots,x_n\) then this can be expressed in the following fashion: \[ f_{w,b} (x) = \sigma \Big(\sum_{i=1}^n w_i x_i + b \Big) , \quad \sigma(x) = \frac{1}{1+e^{-x}} \] The sigmoid function \(\sigma\) is applied to the linear function to give an output between 0 and 1. You will use the gradient descent learning algorithm to find good weight parameters \(w_i\) and \(b\) such that \(round(f_{w,b}(x)) = 1\) when there is an animal in the image and \(round(f_{w,b}(x)) = 0\) when there is a no animal in the image.

Training and Development

Please see the lecture notes for training a logistic regression model.

• Training: To train the model you are going to need to implement gradient descent on the training set. The loss used in this MP is the binary cross entropy loss: \[ L = - \frac{1}{N} \sum_{i=1}^{N} y^{(i)}\ln(f_{w,b}(x^{(i)})) + (1-y^{(i)}) \ln(1-f_{w,b}(x^{(i)})) \] Where \(x^{(i)}\) and \(y^{(i)}\) are \(i^{th}\) data-label pair in the training set. Be sure to initialize weights and biases to zero.
  Note: To get full points on the autograder, use a constant learning rate (no decay) and do not shuffle the training data.

• Development: After you have trained your logistic regression classifier, you will have your model decide whether or not images in the development set contain animals in them or not. In order to do this round the function \(f_{w,b}(x)\) to the nearest integer. If it is \(< 0.5\) then classify as \(0\). If it is \(\geq 0.5\) then classify as \(1\).