CS440/ECE448 Fall 2023 Assignment 9: Neural Nets and PyTorch Due date: Monday November 6th, 11:59pm

The goal of this assignment is to employ neural networks, nonlinear and multi-layer extensions of the linear perceptron, to classify images into four categories: ship, automobile, dog, or frog. That is, your ultimate goal is to create a classifier that can tell what each picture depicts.

In MP9, you will create an 1980s-style shallow neural network. In MP10, you will improve this network using more modern techniques such as changing the activation function, changing the network architecture, or changing other initialization details.

You will be using the PyTorch and NumPy libraries to implement these models. The PyTorch library will do most of the heavy lifting for you, but it is still up to you to implement the right high-level instructions to train the model.

You will need to consult the PyTorch documentation, linked multiple times on this page, to help you with implementation details. We strongly recommend this PyTorch Tutorial, specifically the Training a Classifier section since it walks you thorugh building a classifier very similar to the one in this assignment. There are also other guides out there such as this one.

Template package and submission

The template package contains your starter code and also your training and development datasets (packed into a single file).

In the code package, you should see these files:

• reader.py - This file is responsible for reading in the data set. It creates a giant NumPy array of feature vectors corresponding to each image.
• mp9.py - This is the main file that starts the program, and computes the accuracy, and confusion matrix using your implementation.
• neuralnet.py - This is the file where you will be doing all of your work.

Run python3 mp9.py -h in your terminal to find more about how to run the program.

You will need to import torch and numpy. Otherwise, you should use only modules from the standard python library. Do not use torchvision.

The autograder doesn't have a GPU, so it will fail if you attempt to use CUDA.

Dataset

The dataset consists of 3000 31x31 colored (RGB) images (a modified subset of the CIFAR-10 dataset, provided by Alex Krizhevsky). This set is split for you into 2250 training examples (which are a mostly balanced sample of cars, boats, frogs and dogs) and 750 development examples.

The function load_dataset() in reader.py will unpack the dataset file, returning images and labels for the training and development sets. Each of these items is a Tensor.

Training and dataloaders

Two of the inputs to your main training function (fit) are the number of epochs and the batch size. You should run your training process for that many epochs on the training set. In each epoch your code needs to process ALL the training data, batch by batch, each batch having the specified size (failing to include every batch in each epoch will seriously harm your performance as there will be too few gradient updates to get decent performance). If you try to do this by hand, you'll run into issues such as what to do when the dataset size isn't a multiple of the batch size.

To make this simple, you will use a pytorch dataloader. A dataloader is a way for you to handle loading and transforming data before it enters your network for training or prediction. It will let you write code that looks like you're just looping through the dataset, with the division into batches happening automatically. Details on how to use a dataloader can be found in this tutorial by Shervine Amidi. We have provided an auxiliary function (get_dataset_from_arrays(X,Y)) in utils.py that converts tensors of features (X) and labels (Y) into a simple torch dataset class that can be loaded into their dataloaders for your convenience. The dataloader will return a dictionary, not a tuple. You have to get the labels and features out of that dictionary.

To get consistent output from the autograder, make sure to set "shuffle = False" on your dataloader.

Confusion Matrix

The top-level program mp9.py returns three types of feedback about your model.

• The accuracy on the dev set
• A confusion matrix for the dev set
• The total number of parameters in your network

A confusion matrix is a very useful tool for evaluating multi-class classification problems, as it helps in identifying possible sources of imbalance in your dataset - and can offer precious insights into possible biases in your training procedure.

Specifically, in a classification problem with k classes, a confusion matrix will have k rows and k columns. Each row corresponds to the ground truth label of the data points - and each column refers to the predicted label by your classifier. Each entry on the matrix contains a count of the corresponding tuple (ground_truth label, predicted_label). In other words, all elements in the diagonal of this square matrix have been correctly classified - and all other elements count as mistakes. For instance, if your matrix has many entries in [0,1]. This will mean that your classifier tends to mistake points belonging to class 0 for points belonging to class 1. Further details can be found on this Wikipedia page.

Classical Shallow Network

To make things more precise, in lecture you learned of a function \( f_{w}(x) = \sum_{i=1}^n w_i x_i + b\). In this assignment, given weight matrices \(W_1,W_2\) with \(W_1 \in \mathbb{R}^{h \times d}\), \(W_2 \in \mathbb{R}^{4 \times h}\) and bias vectors \(b_1 \in \mathbb{R}^{h}\) and \(b_2 \in \mathbb{R}^{4}\), you will learn a function \( F_{W} \) defined as \[ F_{W} (x) = W_2\sigma(W_1 x + b_1) + b_2 \] where \(\sigma\) is your activation function. In MP9, you should use either of the sigmoid or ReLU activation functions.

For this dataset, you have 2883 input values, one for each channel of each pixel in an image. That is \(d=(31)^2(3) = 2883\). You should be able to pass the Part 1 tests with no more than 200 hidden units. That is you should have \(h \le 200\).

Training and Development

Training: To train the neural network you are going to need to minimize the empirical risk \(\mathcal{R}(W)\) which is defined as the mean loss determined by some loss function. For this assignment you should use cross entropy for that loss function. In the case of binary classification, the empirical risk is given by \[ \mathcal{R}(W) = \frac{1}{n}\sum_{i=1}^n y_i \log \hat y_i + (1-y_i) \log (1 - \hat y_i) . \] where \(y_i\) are the labels and \(\hat y_i \) are determined by \(\hat y_i = \sigma(F_{W}(x_i))\) where \(\sigma(x) = \frac{1}{1+e^{-x}}\) is the sigmoid function. For this assignment, you won't have to implement these functions yourself; you can use the built-in PyTorch functions. Notice that because PyTorch's CrossEntropyLoss incorporates a sigmoid function, you do not need to explicitly include an activation function in the last layer of your network.

Development: After you have trained your neural network model, you will have your model decide whether or not images in the development set decide what is the class of each image. This is done by evaluating your network \(F_{W}\) on each example in the development set, and then taking the index of the maximum of the four outputs (argmax).

Data Standardization: Convergence speed and accuracies can be improved greatly by simply centralizing your input data by subtracting the sample mean and dividing by the sample standard deviation. More precisely, you can alter your data matrix \(X\) by simply setting \(X:=(X-\mu)/\sigma\). Notice that you are standarizing a feature value across all images, not standardizing a feature value relative to the other features in the same image. Be sure to standardize the dev data just exactly like you standardized the training data. This standardization should be done in the fit() function, not the forward() function. Do this right at the start!

Notice that the autograder will pass in the number of training epochs and the batch size. You don't control those. However, you do control the neural net's learning rate. If you are confident about a model you have implemented but are not able to pass the accuracy thresholds on gradescope, try increasing the learning rate. Be aware, however, that using a very high learning rate may worse performance since the model may begin to oscillate around the optimal parameter settings.

With the above model design and tips, you should expect around 0.62 dev-set accuracy.

Digging into the Skeleton Code

The file neuralnet.py gives you a NeuralNet class which implements a torch.nn.module. This class consists of __init__(), forward(), and step() functions. The main function fit() will use these to train the network and then classify the images from the test/development set.

__init__()

• Use the Linear and Sequential objects. Keep in mind that Linear uses a Kaiming He uniform initialization to initialize the weight matrices and sets the bias terms to all zeros.
• Alternatively, you can explicitly define weight matrices W1, W2, ... and bias terms b1, b2, ... by defining them as Tensors. This approach is more hands on and will allow you to choose your own initialization. For this assignment, however, Kaiming He uniform initialization should suffice and should be a good choice.

Look at the examples in the PyTorch Tutorial.

forward()

step()

When you return the loss_value from this function, make sure to convert it to a plain number. This allows proper garbage collection to take place, so that your program won't consume excessive amounts of memory. Two options:

• Return loss_value.item(). This works if it is just a single number.
• Or use loss_value.detach().cpu().numpy(). This which separates the loss value from the computations that led up to it, moves it to the CPU (e.g. if you are using a GPU locally), and then converts it to a NumPy array.

Remember that Gradescope won't have a GPU.

fit()

Some tips

Apparently it's still possible to be using a 32-bit environment. This may be ok. However, be aware that recent versions of PyTorch are optimized for a 64-bit environment and that is what Gradescope is using.