Dataset

The dataset consists of 10000 32x32 colored images total. We have split this data set for you into 2500 development examples and 7500 training examples. There are 2999 negative examples and 4501 positive examples in the training set. This is a subset of the CIFAR-10 dataset, provided by Alex Krizhevsky.

To make things more precise, in MP5 you learned a function $f_{w}(x) = \sum_{i=1}^n w_i x_i + b$. In this assignment, given weight matrices $W_1,W_2$ with $W_1 \in \mathbb{R}^{h \times d}$, $W_2 \in \mathbb{R}^{h \times 2}$ and bias vectors $b_1 \in \mathbb{R}^{h}$ and $b_2 \in \mathbb{R}^{2}$, you will learn a function $F_{W}$ defined: $$F_{W} (x) = W_2\sigma(W_1 x + b_1) + b_2$$ where $\sigma$ is your activation function. In part 1, you should use either sigmoid or ReLU activation functions. You will use 32 hidden units ($h=32$) and 3072 input units, one for each of the image's pixels ($d=(32)(32)(3) = 3072$).

Training and Development

• Training: To train the neural network you are going to need to minimize the empirical risk $\mathcal{R}(W)$ which is defined as the mean loss determined by some loss function. For this assignment you can use cross entropy for that loss function. In the case of binary classification, the empirical risk is given by: $$\mathcal{R}(W) = \frac{1}{n}\sum_{i=1}^n y_i \log \hat y_i + (1-y_i) \log (1 - \hat y_i) .$$ Where the $y_i$ are the labels and the $\hat y_i$ are determined by $\hat y_i = \sigma(F_{W}(x_i))$, where $\sigma$ is the sigmoid function $\sigma(x) = \frac{1}{1+e^{-x}}$. For this assignment, you won't have to implement these functions yourself. You can use the built-in PyTorch functions, such as torch.nn.CrossEntropyLoss().

  Notice that torch.nn.CrossEntropyLoss() incorporates a sigmoid function. So you do not need to explicitly include an activation function in the last layer of your network.

• Development: After you have trained your neural network model, you will have your model decide whether or not images in the development set contain animals in them or not. This is done by evaluating your network $F_{W}$ on each example in the development set, and then taking the index of the maximum of the two outputs (i.e. argmax).
• Data Standardization: Convergence speed and accuracies can be improved greatly by simply centralizing your input data by subtracting the sample mean and dividing by the sample standard deviation. More precisely, you can alter your data matrix $X$ by simply doing $X:=(X-\mu)/\sigma$.

Part 2: Modern Network

1. Choice of activation function: Some possible candidates are (Tanh, ReLU, ELU, softplus, and LeakyReLU). You may find that choosing the right activation function will lead to significantly faster convergence, and or improved performance overall.
2. L2 Regularization: Regularization is when you try to improve your model's ability to generalize to unseen examples. One commonly used form of regularization is L2 regularization. Let $\mathcal{R}(W)$ be the empirical risk (mean loss), then you can implement L2 regularization by adding on an additional term that penalizes the norm of the weights. More precisely, your new empirical risk becomes $\mathcal{R}(W):= \mathcal{R}(W) + \lambda \sum_{W \in P} \Vert W \Vert_2 ^2$ where $P$ is the set of all your parameters and $\lambda$ (usually small) is some hyperparameter chosen by you. Additionally, there are several other techniques besides L2 regularization for improving the generalization of your model, like dropout, batch normalization etc.
3. Network Depth and Width: The sort of network you implemented in part 1 is called a two-layer network because it uses two weight matrices. Sometimes it helps performance to add more hidden units and or add more weight matrices to obtain greater representation power and make training easier.
4. Using Convolutional Neural Nets: While it is possible to obtain nice results with traditional multilayer perceptrons, when doing image classification tasks it is often best to use convolutional neural networks, which are tailored specifically to signal processing tasks such as image recognition. See if you can improve your results using convolutional layers in your network.

Some things to look for:

1. The autograder runs the training process for 500 batches (max_iter=500). This is done so that we have a consistent training process for each student's evaluation and comparison with benchmarks/threshold accuracies.


2. But, you still have one thing in your full control, the learning rate. In case you are confident about a model you implemented but are not able to pass the accuracy thresholds on gradescope, try increasing the learning rate. It could be possible that your model could do better with more training. However, be mindful that using a very high learning rate might deteriorate the performance as well since the model might just oscillate around the optima.

Provided Code Skeleton