CS440/ECE448 Spring 2023¶

MP04: Neural Networks¶

The first thing you need to do is to download this file: mp04.zip. It has the following content:

  • submitted.py: Your homework. Edit, and then submit to Gradescope.
  • mp04_notebook.ipynb: This is a Jupyter notebook to help you debug. You can completely ignore it if you want, although you might find that it gives you useful instructions.
  • pytorch_tutorial.ipynb: A general introduction to PyTorch. You can ignore it if you want.
  • data: This directory contains the data.
  • tests: This directory contains visible test.
  • reader.py: This is an auxiliary program that you can use to read the data.

You will need to import torch and numpy. Otherwise, you should use only modules from the standard python library. Do not use torchvision.

The autograder doesn't have a GPU, so it will fail if you attempt to use CUDA.

This file (mp04_notebook.ipynb) will walk you through the whole MP, giving you instructions and debugging tips as you go.

Goal¶

The goal of this assignment is to employ neural networks, nonlinear and multi-layer extensions of the linear perceptron, to classify images into 5 categories: ship (0), automobile (1), dog (2), frog (3), or horse (4). That is, your ultimate goal is to create a classifier that can tell what each picture depicts.

You will be using PyTorch (one of the most commonly used machine learning framworks) and NumPy libraries to implement these models. The PyTorch library will do most of the heavy lifting for you, but it is still up to you to implement the right high-level instructions to train the model.

You will need to consult the PyTorch documentation, to help you with implementation details. You may want to take a look at the provided PyTorch tutorial.

Table of Contents¶

  1. Dataset
  2. Dataloader
  3. Confusion Matrix
  4. Neural Network
  5. Grade Your Homework
  6. Extra credit

Dataset¶

This section aims to help you get some ideas of the datasets we are using. Note that in this MP we have provided dataloaders for you, so you do not need to load any datasets by yourself.

The dataset consists of 3750 31x31 colored (RGB) images (a modified subset of the CIFAR-10 dataset, provided by Alex Krizhevsky). This set is split for you into 2813 training examples and 937 development examples.

The function Load_dataset() in reader.py will unpack the dataset file (you don't need to call this function in the MP), returning images and labels for the training and development sets. Note that the images have been flattened, therefore the dimension of one image sample is 2883 (31 x 31 x 3).

In [1]:
import reader
In [2]:
# filepath to data
filepath = "./data/mp_data"
# load datasets, you don't need to call this function in the MP
train_set, train_labels, test_set, test_labels = reader.Load_dataset(filepath)
print("Shape of train set:", train_set.shape)
print("Shape of test set:", test_set.shape)
Shape of train set: (2813, 2883)
Shape of test set: (937, 2883)
In [3]:
# helper functions to visualize images, you can ignore the implementations

import matplotlib.pyplot as plt
from PIL import Image
import numpy as np

names = {0: "ship", 1: "automobile", 2: "dog", 3: "frog", 4: "horse"}

def show_train_image(index):
    img_c1 = Image.fromarray(train_set[index][: 961].reshape(31, 31))
    img_c2 = Image.fromarray(train_set[index][961: 1922].reshape(31, 31))
    img_c3 = Image.fromarray(train_set[index][1922: ].reshape(31, 31))
    img_rgb = np.zeros((31, 31, 3), 'uint8')
    img_rgb[..., 0] = img_c1
    img_rgb[..., 1] = img_c2
    img_rgb[..., 2] = img_c3
    fig = plt.figure()
    plt.axis('off')
    plt.imshow(img_rgb)
    title = "Train[" + str(index) + "]  --  " + names[train_labels[index].item()] + " -- label " + str(train_labels[index].item())
    plt.title(title)

def show_test_image(index):
    img_c1 = Image.fromarray(test_set[index][: 961].reshape(31, 31))
    img_c2 = Image.fromarray(test_set[index][961: 1922].reshape(31, 31))
    img_c3 = Image.fromarray(test_set[index][1922: ].reshape(31, 31))
    img_rgb = np.zeros((31, 31, 3), 'uint8')
    img_rgb[..., 0] = img_c1
    img_rgb[..., 1] = img_c2
    img_rgb[..., 2] = img_c3
    fig = plt.figure()
    plt.axis('off')
    plt.imshow(img_rgb)
    title = "Test[" + str(index) + "]  --  " + names[test_labels[index].item()] + " -- label " + str(test_labels[index].item())
    plt.imshow(img_rgb)
    plt.title(title)
In [4]:
# You can use the helper function to visualize training set. The parameter is the index of image to visualize.
show_train_image(2812)      # feel free to change the number. note that the size of training set is 2813
In [5]:
# You can use the helper function to visualize testing set. The parameter is the index of image to visualize.
show_test_image(20)     # feel free to change the number. note that the size of testing size is 937

Dataloaders¶

If you are not sure what dataloaders are, please refer to the tutorial or PyTorch document. With dataloaders, you can iterate through the datasets easily.In this MP, we have provided dataloaders for training set and testing set, respectively.

In [6]:
# Generate dataloaders
# you don't need to call these functions in MP, train_loader and test_loader are passed to your function as arguments
train_set_p, test_set_p  = reader.Preprocess(train_set, test_set)
train_loader, test_loader = reader.Get_DataLoaders(train_set_p, train_labels, test_set_p, test_labels, batch_size=100)

Each iteration below returns a batch of train_features and train_labels (in this MP, we set up batch size equal to 100, so each batch contains 100 feature and label tensors respectively). You can pass the feature batch to your neural network, and then compare the label batch with your predictions. Let's iterate over the dataset and see what each batch looks like.

Labels:
ship: 0, automobile: 1, dog: 2, frog: 3, horse: 4

In [7]:
batch_index = 0
# Iterate over the dataloader
for features, labels in train_loader:
    # you can train your network with the feature and label batches here
    print("Batch #", batch_index)
    print("Feature shape:", features.shape)
    print(features, "\n")
    print("Label shape:", labels.shape)
    print(labels, "\n\n")
    batch_index += 1
Batch # 0
Feature shape: torch.Size([100, 2883])
tensor([[ 0.3525,  0.3150, -0.6970,  ...,  1.2380,  1.2221,  1.3589],
        [ 0.8684,  0.9081,  0.9093,  ...,  1.0154,  1.0167,  0.9519],
        [ 0.3932,  0.3839, -0.2262,  ..., -0.6230, -0.5951, -0.5977],
        ...,
        [-0.4756,  0.1494,  0.5631,  ..., -0.6230, -0.7689, -1.1299],
        [ 1.4793,  1.5012,  1.5324,  ...,  0.9199,  0.9219,  0.8737],
        [-0.5027, -0.5816, -0.5309,  ...,  1.0631,  1.0483,  0.9989]]) 

Label shape: torch.Size([100])
tensor([0, 0, 0, 3, 1, 1, 0, 1, 4, 4, 4, 2, 1, 0, 3, 4, 1, 1, 3, 1, 1, 4, 1, 1,
        4, 4, 3, 2, 0, 4, 2, 1, 0, 1, 0, 3, 0, 1, 0, 4, 4, 0, 4, 0, 3, 2, 2, 3,
        4, 2, 3, 3, 1, 2, 3, 3, 2, 2, 2, 2, 1, 0, 4, 0, 3, 3, 0, 3, 1, 0, 0, 2,
        1, 3, 0, 0, 2, 4, 3, 2, 3, 2, 2, 3, 3, 2, 0, 3, 4, 0, 1, 0, 3, 1, 0, 0,
        4, 2, 0, 1]) 


Batch # 1
Feature shape: torch.Size([100, 2883])
tensor([[-1.7381, -1.7541, -1.7771,  ..., -1.7523, -1.7487, -1.7403],
        [-0.3127, -0.3333, -0.3647,  ..., -0.2890, -0.3107, -0.3159],
        [-1.0594, -1.0368, -1.4863,  ..., -1.4024, -0.6899, -0.7229],
        ...,
        [ 0.3661,  0.4115,  0.4108,  ...,  0.6018,  0.4320,  0.4510],
        [ 1.7237,  1.7633,  1.7539,  ...,  2.3038,  2.2808,  2.2511],
        [-0.6249, -0.7609, -0.3232,  ..., -0.4003,  0.2898,  1.3276]]) 

Label shape: torch.Size([100])
tensor([3, 2, 3, 1, 3, 3, 0, 0, 0, 4, 1, 2, 3, 4, 2, 4, 3, 3, 3, 2, 3, 4, 2, 3,
        4, 4, 2, 1, 4, 3, 3, 2, 2, 1, 2, 2, 0, 2, 3, 3, 2, 0, 0, 3, 1, 0, 4, 1,
        3, 4, 1, 0, 3, 2, 3, 4, 2, 2, 3, 1, 4, 2, 2, 2, 4, 3, 1, 3, 3, 1, 4, 4,
        1, 3, 2, 3, 2, 4, 2, 2, 2, 4, 4, 3, 4, 0, 0, 0, 1, 3, 4, 1, 1, 4, 3, 4,
        0, 0, 3, 3]) 


Batch # 2
Feature shape: torch.Size([100, 2883])
tensor([[ 0.8412,  0.8943,  0.9093,  ...,  1.4766,  1.3643,  1.1084],
        [-0.8557, -0.1954, -0.1016,  ...,  1.1426,  1.1747,  1.1084],
        [ 0.1353,  0.1632,  0.4938,  ..., -0.0027,  0.0844, -0.0499],
        ...,
        [ 1.7237,  1.7081,  1.7262,  ..., -1.1956,  1.0641,  1.8911],
        [ 0.0403,  0.1494,  0.2584,  ..., -0.6389, -0.6741, -0.7542],
        [-0.9507, -1.0230, -1.0294,  ...,  0.7449,  0.2108, -0.1438]]) 

Label shape: torch.Size([100])
tensor([2, 4, 4, 2, 1, 4, 2, 3, 0, 3, 3, 1, 2, 0, 4, 3, 4, 2, 2, 0, 0, 3, 4, 3,
        4, 3, 2, 4, 3, 3, 2, 0, 2, 0, 3, 4, 0, 4, 3, 1, 2, 2, 3, 4, 1, 4, 1, 2,
        4, 1, 3, 1, 4, 0, 4, 2, 4, 3, 4, 0, 1, 3, 1, 4, 2, 3, 1, 2, 2, 1, 0, 2,
        3, 2, 2, 4, 3, 1, 0, 2, 2, 2, 4, 3, 3, 1, 1, 0, 2, 2, 4, 3, 4, 1, 0, 3,
        3, 0, 0, 3]) 


Batch # 3
Feature shape: torch.Size([100, 2883])
tensor([[ 1.1942,  1.1012,  1.2831,  ..., -0.6230, -0.5161, -0.3003],
        [ 1.5743,  1.4599,  1.4078,  ..., -1.3229, -1.3536, -1.3490],
        [ 1.6422,  1.6392,  1.6293,  ...,  0.2041,  0.2740,  0.3415],
        ...,
        [-0.6792, -0.7334, -0.4062,  ...,  0.4745,  0.4952,  0.5293],
        [-0.3263, -0.3471, -0.3508,  ..., -0.6230, -0.5951, -0.6916],
        [ 0.2982,  0.0805,  0.1615,  ..., -0.9570, -0.8954, -0.8481]]) 

Label shape: torch.Size([100])
tensor([1, 0, 4, 3, 2, 3, 1, 0, 4, 4, 1, 1, 3, 1, 3, 3, 3, 1, 1, 2, 0, 4, 3, 2,
        2, 2, 3, 2, 3, 2, 2, 3, 2, 4, 2, 0, 4, 3, 3, 2, 0, 4, 2, 0, 1, 4, 1, 1,
        1, 2, 3, 0, 1, 3, 0, 4, 1, 0, 4, 1, 4, 1, 0, 3, 2, 0, 2, 0, 3, 0, 2, 1,
        1, 2, 3, 4, 0, 2, 4, 1, 2, 1, 0, 1, 2, 4, 1, 2, 3, 1, 0, 4, 1, 2, 2, 0,
        4, 2, 0, 3]) 


Batch # 4
Feature shape: torch.Size([100, 2883])
tensor([[ 0.3118,  0.3150,  0.3277,  ...,  2.0811,  2.2176,  1.9380],
        [-1.3716, -1.3265, -1.3340,  ..., -1.1797, -1.2904, -1.1612],
        [-0.4077, -0.1126, -0.2539,  ..., -0.7980, -0.8005, -1.0203],
        ...,
        [-0.8286, -0.8023, -0.7801,  ..., -0.8616, -0.9112, -0.2690],
        [-0.1226, -0.1126, -0.1431,  ..., -1.2433, -1.2272, -1.1768],
        [-0.5978, -0.4851, -0.4201,  ..., -1.3229, -1.3536, -1.3803]]) 

Label shape: torch.Size([100])
tensor([0, 3, 1, 4, 3, 0, 3, 0, 0, 3, 4, 4, 0, 1, 2, 1, 4, 2, 1, 2, 4, 0, 3, 1,
        0, 1, 0, 3, 4, 0, 0, 3, 2, 2, 0, 2, 2, 1, 4, 0, 3, 1, 4, 4, 3, 0, 4, 2,
        3, 2, 2, 0, 2, 2, 1, 2, 4, 2, 2, 4, 2, 4, 2, 3, 0, 2, 1, 0, 2, 3, 2, 4,
        1, 0, 1, 4, 3, 4, 4, 3, 1, 2, 0, 2, 4, 4, 0, 3, 3, 3, 4, 0, 0, 0, 2, 2,
        1, 0, 4, 2]) 


Batch # 5
Feature shape: torch.Size([100, 2883])
tensor([[-0.6385, -0.5954, -0.6001,  ..., -0.1140, -0.1527, -0.1594],
        [ 0.0810,  0.0253, -0.1293,  ..., -0.4639, -0.4529, -0.1438],
        [-0.8693, -0.8713, -0.8632,  ...,  1.5880,  1.5697,  1.5311],
        ...,
        [-0.7607, -0.8161, -0.8217,  ...,  1.9061,  1.8226,  1.8128],
        [ 0.2711, -0.3057,  0.2723,  ...,  0.4109, -0.2791,  0.5763],
        [-0.8421, -0.8437, -1.0294,  ...,  0.8881,  0.9851,  0.8893]]) 

Label shape: torch.Size([100])
tensor([2, 3, 0, 3, 4, 3, 1, 2, 0, 2, 1, 3, 4, 1, 0, 2, 4, 1, 0, 3, 4, 1, 0, 0,
        4, 0, 1, 2, 2, 3, 0, 1, 2, 2, 0, 4, 4, 4, 4, 0, 1, 3, 2, 3, 0, 4, 3, 3,
        0, 0, 0, 1, 3, 0, 0, 4, 0, 3, 2, 1, 4, 3, 2, 4, 1, 0, 3, 4, 1, 3, 2, 2,
        4, 2, 0, 3, 1, 0, 4, 1, 1, 1, 0, 0, 2, 4, 0, 3, 1, 2, 1, 3, 1, 3, 1, 0,
        2, 0, 3, 2]) 


Batch # 6
Feature shape: torch.Size([100, 2883])
tensor([[ 0.3661,  0.4943,  0.3554,  ..., -0.1299, -0.1369, -0.1281],
        [ 1.7237,  1.7219,  1.6432,  ...,  2.1606,  1.2379,  1.1711],
        [ 1.4250,  1.4185,  1.5324,  ..., -0.9888, -0.9902, -0.9890],
        ...,
        [-0.1905, -0.1540, -0.1708,  ...,  0.3791,  0.3688,  0.3571],
        [ 1.2757,  1.2943,  1.2970,  ..., -1.5297, -1.5590, -1.5056],
        [-1.5345, -1.4782, -1.4586,  ..., -1.5774, -1.6380, -1.6777]]) 

Label shape: torch.Size([100])
tensor([1, 2, 1, 1, 2, 4, 4, 0, 4, 0, 4, 1, 3, 2, 2, 3, 4, 4, 4, 0, 0, 2, 1, 1,
        1, 2, 4, 2, 3, 1, 3, 0, 2, 1, 1, 0, 3, 0, 4, 2, 2, 4, 4, 4, 3, 4, 4, 4,
        1, 4, 4, 4, 4, 0, 1, 3, 4, 1, 2, 1, 3, 3, 3, 2, 0, 3, 0, 3, 3, 2, 2, 1,
        0, 4, 0, 1, 0, 2, 4, 2, 3, 2, 3, 3, 1, 2, 4, 2, 2, 4, 2, 4, 1, 3, 4, 2,
        4, 2, 1, 4]) 


Batch # 7
Feature shape: torch.Size([100, 2883])
tensor([[-1.4530, -1.4782, -1.5971,  ...,  2.0334,  1.9174,  1.8754],
        [ 0.0810,  0.1770,  0.2030,  ..., -1.1161, -0.6267,  0.0128],
        [ 0.9498,  0.7978,  0.7985,  ..., -0.4798, -0.5003, -0.5194],
        ...,
        [-0.6792, -0.5954, -0.6555,  ...,  1.0472,  0.5110, -0.0029],
        [-1.6295, -1.6162, -1.5971,  ..., -1.5138, -1.5116, -1.5212],
        [-0.1498, -0.1402, -0.1570,  ...,  1.1108,  1.0957,  0.9519]]) 

Label shape: torch.Size([100])
tensor([2, 0, 3, 2, 0, 4, 0, 1, 3, 1, 0, 4, 1, 4, 0, 1, 3, 4, 4, 2, 0, 2, 0, 3,
        0, 2, 0, 1, 2, 4, 0, 0, 4, 4, 2, 2, 3, 2, 4, 1, 3, 2, 4, 0, 0, 0, 4, 2,
        1, 2, 1, 1, 4, 4, 0, 3, 4, 0, 1, 1, 4, 0, 1, 3, 1, 4, 4, 4, 0, 4, 4, 1,
        2, 4, 1, 0, 0, 1, 0, 4, 3, 0, 2, 0, 0, 0, 4, 2, 0, 2, 0, 1, 1, 1, 1, 0,
        0, 3, 0, 0]) 


Batch # 8
Feature shape: torch.Size([100, 2883])
tensor([[-1.6431, -1.6437, -1.6663,  ..., -0.7662, -0.5951, -0.3629],
        [-0.5842, -0.4437, -0.3785,  ...,  0.2837,  0.3530,  0.3884],
        [-0.1498, -0.1264, -0.1847,  ..., -0.9411, -0.9586, -0.9890],
        ...,
        [ 0.3932,  0.3701,  0.3554,  ...,  0.1405,  0.2740,  0.3102],
        [-0.2991, -0.3609, -0.4201,  ...,  0.3632,  0.1792, -0.0655],
        [ 0.3389,  0.5219,  0.3692,  ...,  1.4289,  1.3485,  1.4372]]) 

Label shape: torch.Size([100])
tensor([1, 0, 0, 2, 4, 4, 4, 2, 2, 4, 0, 1, 1, 0, 2, 2, 1, 4, 2, 0, 4, 1, 2, 2,
        0, 2, 4, 2, 3, 2, 1, 0, 0, 3, 4, 0, 2, 2, 3, 0, 1, 1, 0, 0, 3, 3, 4, 1,
        3, 2, 3, 1, 0, 2, 0, 2, 4, 2, 3, 4, 0, 1, 1, 2, 0, 0, 4, 0, 4, 4, 4, 0,
        3, 2, 3, 0, 2, 2, 4, 2, 3, 3, 1, 4, 3, 2, 3, 3, 0, 2, 2, 1, 3, 4, 1, 0,
        1, 2, 3, 4]) 


Batch # 9
Feature shape: torch.Size([100, 2883])
tensor([[ 0.5969,  0.6184,  0.6046,  ..., -0.2731, -0.2949, -0.3003],
        [-1.7110, -1.7265, -1.7356,  ..., -1.2911, -1.4800, -1.6151],
        [ 0.7598,  0.7426,  0.7431,  ...,  1.3176,  1.3011,  1.2650],
        ...,
        [-0.8286, -0.7747, -1.0155,  ..., -0.0186, -0.0421,  0.0441],
        [ 0.9363,  1.1564,  0.8677,  ..., -1.4024, -1.4010, -1.4742],
        [-1.2901, -1.2713, -1.2925,  ...,  0.5541,  0.4478,  0.4197]]) 

Label shape: torch.Size([100])
tensor([4, 2, 0, 2, 3, 4, 3, 3, 0, 3, 4, 2, 3, 2, 4, 3, 2, 0, 3, 4, 4, 4, 4, 1,
        2, 4, 1, 0, 1, 4, 1, 3, 4, 4, 2, 3, 3, 4, 1, 3, 0, 3, 1, 2, 2, 4, 3, 2,
        3, 0, 4, 1, 0, 4, 1, 4, 4, 3, 4, 0, 4, 4, 1, 2, 1, 4, 4, 0, 2, 2, 4, 0,
        2, 2, 1, 1, 3, 4, 1, 2, 4, 2, 0, 4, 3, 1, 1, 0, 3, 2, 2, 3, 0, 3, 4, 1,
        1, 4, 1, 2]) 


Batch # 10
Feature shape: torch.Size([100, 2883])
tensor([[ 0.1489,  0.0115,  0.0923,  ..., -0.2731, -0.1843,  0.0910],
        [ 0.1353,  0.0253,  0.0230,  ..., -0.2412, -0.1685, -0.0655],
        [ 0.3932,  0.4529,  0.4523,  ..., -1.0207, -1.0850, -1.0516],
        ...,
        [ 1.1399,  1.1564,  1.1585,  ...,  1.2699,  1.1747,  1.1867],
        [ 0.5426,  0.5771,  0.5908,  ..., -0.7980, -0.7057, -0.4881],
        [-1.5209, -1.5196, -1.5971,  ..., -1.1956, -1.4010, -1.1455]]) 

Label shape: torch.Size([100])
tensor([2, 3, 4, 2, 1, 4, 3, 4, 3, 1, 0, 2, 1, 4, 0, 3, 1, 0, 3, 2, 1, 4, 3, 2,
        2, 0, 2, 4, 1, 3, 0, 0, 2, 1, 4, 2, 0, 1, 3, 4, 1, 3, 4, 4, 1, 1, 1, 3,
        4, 2, 4, 3, 4, 3, 1, 1, 4, 1, 4, 3, 2, 4, 2, 3, 0, 0, 4, 4, 3, 2, 2, 0,
        2, 4, 0, 0, 0, 2, 3, 2, 1, 0, 3, 4, 4, 3, 1, 4, 4, 4, 3, 4, 1, 4, 1, 4,
        3, 0, 4, 4]) 


Batch # 11
Feature shape: torch.Size([100, 2883])
tensor([[ 1.2621,  1.2529,  1.2416,  ..., -0.2890, -0.3423, -0.1125],
        [ 0.2846,  1.0874,  1.0893,  ...,  1.5562,  1.4117,  1.3589],
        [ 0.7462,  0.3977,  1.1308,  ..., -0.1140, -0.0263, -0.0655],
        ...,
        [ 1.5608,  1.4736,  1.5878,  ..., -1.7046, -1.6538, -1.6151],
        [-1.6974, -1.6713, -1.6940,  ..., -0.4957, -0.4529, -0.6916],
        [-0.5978, -0.1402, -0.1431,  ...,  1.2221,  1.1589,  1.1084]]) 

Label shape: torch.Size([100])
tensor([4, 0, 0, 3, 4, 4, 4, 1, 0, 4, 2, 3, 0, 2, 0, 0, 4, 2, 1, 1, 1, 2, 2, 4,
        3, 4, 0, 2, 0, 3, 4, 1, 0, 1, 3, 3, 3, 4, 4, 1, 4, 4, 2, 3, 4, 0, 1, 3,
        2, 4, 4, 0, 4, 2, 0, 0, 0, 0, 0, 3, 1, 2, 2, 3, 4, 2, 1, 1, 4, 0, 3, 4,
        3, 2, 4, 3, 3, 0, 0, 2, 2, 4, 4, 4, 3, 0, 0, 1, 0, 3, 1, 2, 2, 3, 1, 4,
        4, 4, 3, 1]) 


Batch # 12
Feature shape: torch.Size([100, 2883])
tensor([[-1.2358, -1.2437, -1.2371,  ..., -0.3526, -0.4687, -0.5664],
        [-0.4620, -0.4989, -0.5032,  ...,  0.1564,  0.3056,  0.4980],
        [ 0.2032,  0.2046,  0.1892,  ...,  0.7449,  0.7006,  0.4980],
        ...,
        [-0.2720, -0.1540, -0.0877,  ..., -0.5594, -0.4687, -0.5507],
        [-0.3806, -0.4023, -0.4062,  ..., -0.2890, -0.2791, -0.1751],
        [-1.7381, -1.7541, -1.7771,  ..., -1.7523, -1.7487, -1.7403]]) 

Label shape: torch.Size([100])
tensor([2, 0, 1, 0, 2, 0, 1, 2, 0, 3, 4, 3, 3, 3, 4, 2, 3, 4, 1, 2, 2, 0, 2, 1,
        0, 3, 1, 1, 2, 4, 2, 3, 1, 0, 2, 4, 0, 1, 2, 4, 2, 3, 2, 1, 4, 4, 4, 3,
        3, 0, 2, 0, 0, 4, 3, 1, 1, 0, 2, 4, 0, 2, 1, 3, 2, 1, 2, 1, 2, 4, 0, 2,
        2, 3, 2, 1, 2, 2, 2, 2, 4, 0, 2, 4, 0, 4, 3, 4, 3, 4, 0, 4, 3, 4, 0, 0,
        3, 0, 0, 2]) 


Batch # 13
Feature shape: torch.Size([100, 2883])
tensor([[ 1.6829,  1.6806,  1.6708,  ...,  2.3038,  2.2808,  2.2511],
        [ 0.6376,  0.9219,  0.9785,  ...,  1.6516,  1.5855,  1.4685],
        [-1.5617, -1.5748, -1.5833,  ..., -0.1776, -0.4529, -0.7073],
        ...,
        [-1.6974, -1.7127, -1.7356,  ..., -1.6251, -1.6064, -1.5682],
        [-0.7471, -0.7471, -0.7109,  ..., -0.9093, -1.1008, -1.2395],
        [-1.7381, -1.7541, -1.7771,  ...,  0.5859,  0.8270,  1.2024]]) 

Label shape: torch.Size([100])
tensor([1, 1, 0, 4, 3, 0, 4, 2, 1, 4, 0, 2, 0, 2, 2, 0, 4, 0, 0, 0, 1, 1, 3, 2,
        0, 4, 2, 3, 1, 4, 1, 3, 2, 4, 2, 1, 0, 0, 4, 0, 4, 3, 1, 3, 4, 3, 3, 0,
        0, 3, 4, 2, 0, 2, 0, 0, 2, 2, 0, 0, 2, 2, 3, 4, 1, 3, 1, 4, 1, 2, 4, 0,
        3, 4, 2, 2, 2, 0, 1, 2, 3, 1, 3, 4, 0, 1, 0, 0, 4, 3, 0, 1, 3, 2, 3, 3,
        0, 2, 4, 3]) 


Batch # 14
Feature shape: torch.Size([100, 2883])
tensor([[-0.8286, -0.8575, -0.3232,  ..., -1.2115, -1.0376, -0.9890],
        [-1.0322, -0.2230,  0.6185,  ..., -0.4162, -0.5003, -0.5038],
        [-0.8557, -0.4437, -0.9740,  ..., -1.3388, -1.4642, -1.5056],
        ...,
        [-0.4756, -0.1678, -0.5309,  ..., -1.0843, -1.1324, -1.1455],
        [ 0.7191,  0.7150,  0.7154,  ...,  0.2200,  0.1634,  0.1223],
        [ 0.7055,  0.7012,  0.7708,  ..., -0.7503, -0.5003, -0.6760]]) 

Label shape: torch.Size([100])
tensor([3, 4, 3, 2, 4, 3, 2, 0, 0, 1, 4, 0, 3, 0, 0, 4, 0, 1, 2, 3, 3, 1, 1, 1,
        2, 0, 0, 3, 3, 3, 2, 2, 3, 0, 1, 2, 2, 0, 4, 2, 4, 2, 4, 0, 4, 4, 2, 1,
        1, 0, 1, 0, 3, 2, 2, 3, 3, 1, 1, 2, 3, 2, 2, 1, 2, 1, 3, 4, 4, 1, 4, 4,
        2, 4, 3, 1, 1, 0, 2, 3, 3, 3, 1, 2, 1, 0, 4, 0, 1, 2, 3, 0, 3, 0, 0, 1,
        1, 4, 0, 2]) 


Batch # 15
Feature shape: torch.Size([100, 2883])
tensor([[ 0.1489, -0.0437,  0.0369,  ...,  0.7927,  0.7796,  0.7641],
        [-1.6295, -1.6575, -1.6802,  ...,  2.3038,  2.2808,  2.2511],
        [ 0.3661, -0.1126, -0.8355,  ...,  0.4268,  0.4004,  0.3102],
        ...,
        [-0.9643, -1.0506, -0.9740,  ..., -0.5753, -0.5951, -0.5664],
        [-0.7607, -1.0368, -1.4586,  ..., -0.5117, -0.7057, -0.9108],
        [ 1.1128,  1.1012,  1.1170,  ..., -0.3208, -0.2633, -0.3473]]) 

Label shape: torch.Size([100])
tensor([1, 4, 4, 2, 4, 0, 0, 2, 3, 3, 4, 4, 0, 0, 4, 1, 3, 2, 4, 3, 3, 0, 2, 1,
        3, 4, 4, 2, 0, 4, 0, 2, 1, 0, 1, 2, 3, 0, 2, 3, 0, 0, 2, 0, 0, 0, 0, 0,
        3, 1, 1, 0, 3, 3, 3, 2, 3, 3, 4, 3, 1, 1, 1, 0, 0, 0, 1, 3, 2, 1, 0, 3,
        3, 2, 0, 0, 3, 3, 4, 0, 2, 1, 3, 0, 1, 4, 3, 0, 1, 4, 2, 3, 4, 3, 1, 4,
        1, 4, 4, 4]) 


Batch # 16
Feature shape: torch.Size([100, 2883])
tensor([[ 1.6694,  1.6530,  1.6570,  ...,  0.2041,  0.2266,  0.7171],
        [ 0.9091,  0.8805,  0.8539,  ...,  0.3950,  0.3846,  0.3728],
        [-1.6974, -1.7127, -1.7079,  ...,  1.6357,  1.6329,  1.6093],
        ...,
        [-0.9100, -0.8161, -0.7663,  ..., -0.8616, -0.8480, -0.8638],
        [-0.2991,  0.0115,  0.0507,  ..., -0.1140, -0.1685, -0.1751],
        [ 1.6151,  1.6668,  1.6570,  ..., -0.0822, -0.0579, -0.0029]]) 

Label shape: torch.Size([100])
tensor([0, 1, 3, 4, 4, 2, 2, 1, 2, 1, 4, 0, 1, 0, 3, 0, 4, 3, 4, 2, 0, 1, 2, 2,
        0, 4, 4, 1, 0, 1, 1, 1, 4, 2, 1, 3, 0, 2, 2, 3, 2, 1, 2, 0, 1, 0, 2, 4,
        2, 4, 1, 0, 4, 2, 0, 1, 1, 2, 1, 1, 2, 1, 1, 4, 4, 3, 2, 3, 4, 3, 2, 0,
        4, 4, 1, 3, 0, 2, 2, 4, 4, 2, 2, 2, 0, 4, 2, 0, 4, 1, 3, 1, 3, 2, 2, 0,
        3, 0, 2, 4]) 


Batch # 17
Feature shape: torch.Size([100, 2883])
tensor([[-3.3984e-01, -3.4713e-01, -3.6469e-01,  ..., -6.5481e-01,
         -1.6855e+00, -1.7247e+00],
        [ 2.7106e-01,  2.8738e-01,  3.2767e-01,  ...,  3.4729e-01,
         -1.5267e-01, -4.0986e-01],
        [-1.3621e-01, -1.5402e-01, -1.7083e-01,  ..., -5.9119e-01,
         -6.1092e-01, -5.3509e-01],
        ...,
        [-1.2087e+00, -1.2299e+00, -1.2232e+00,  ...,  1.0869e-01,
         -3.8970e-01, -6.7596e-01],
        [-4.4889e-04, -2.2850e-03,  9.1806e-03,  ..., -6.7072e-01,
         -6.5833e-01, -6.1335e-01],
        [ 4.2040e-01,  3.7015e-01,  3.6921e-01,  ..., -3.6850e-01,
         -2.7909e-01,  1.0668e-01]]) 

Label shape: torch.Size([100])
tensor([1, 0, 0, 0, 2, 0, 2, 0, 4, 1, 3, 4, 0, 3, 4, 4, 1, 2, 1, 2, 2, 2, 2, 0,
        4, 3, 0, 3, 2, 4, 4, 3, 4, 0, 0, 2, 1, 0, 3, 4, 3, 2, 2, 3, 2, 0, 1, 3,
        4, 2, 3, 2, 1, 2, 1, 3, 2, 3, 3, 4, 3, 1, 2, 1, 3, 3, 3, 0, 0, 3, 1, 4,
        1, 1, 3, 1, 1, 4, 2, 1, 3, 3, 3, 4, 0, 0, 1, 0, 0, 1, 0, 2, 3, 4, 2, 0,
        4, 0, 0, 3]) 


Batch # 18
Feature shape: torch.Size([100, 2883])
tensor([[-1.6295, -1.5886, -1.5694,  ..., -0.7662, -0.7373, -0.9108],
        [ 0.7734,  0.6046,  0.6877,  ..., -1.0366, -1.2588, -1.2708],
        [-1.5481, -1.6162, -1.6802,  ...,  0.4586,  0.2740,  0.2476],
        ...,
        [ 0.8141,  0.8253,  0.8954,  ..., -0.5594, -0.4687, -0.4881],
        [-0.5299, -0.5816, -0.6139,  ..., -0.4162, -0.3739, -0.3786],
        [-0.2855,  1.1288,  1.5185,  ..., -0.9570, -1.5274, -1.4586]]) 

Label shape: torch.Size([100])
tensor([3, 3, 4, 4, 0, 4, 3, 4, 0, 4, 4, 0, 1, 2, 1, 4, 0, 1, 4, 1, 3, 2, 3, 2,
        1, 4, 4, 4, 2, 0, 3, 0, 3, 2, 0, 3, 2, 4, 4, 2, 2, 4, 3, 2, 2, 2, 3, 1,
        1, 1, 0, 0, 4, 3, 0, 1, 3, 3, 2, 1, 0, 3, 0, 4, 1, 3, 1, 1, 4, 1, 4, 2,
        2, 1, 1, 3, 1, 0, 1, 0, 4, 0, 0, 0, 2, 1, 3, 4, 0, 3, 0, 2, 1, 2, 1, 2,
        0, 4, 2, 1]) 


Batch # 19
Feature shape: torch.Size([100, 2883])
tensor([[-0.0819, -0.0850, -0.0739,  ..., -0.6071, -0.6425, -0.6290],
        [ 0.4340,  0.6184,  0.6323,  ..., -1.0207, -0.9586, -0.9734],
        [-1.3852, -1.4230, -1.3617,  ..., -1.5615, -1.2746, -0.8012],
        ...,
        [-0.1498, -0.1816, -0.2124,  ...,  0.5382,  0.6532,  0.6389],
        [ 0.2575,  0.2322,  0.2169,  ...,  0.3632,  0.3846,  0.3102],
        [ 0.0131,  0.1494,  0.1477,  ...,  1.1744,  0.9535, -0.3786]]) 

Label shape: torch.Size([100])
tensor([4, 4, 3, 3, 2, 2, 2, 0, 3, 4, 0, 1, 3, 4, 1, 3, 1, 1, 3, 1, 4, 0, 2, 1,
        2, 0, 2, 4, 0, 1, 3, 3, 3, 4, 1, 1, 1, 2, 2, 1, 3, 2, 3, 3, 4, 1, 0, 0,
        1, 0, 1, 4, 1, 2, 4, 1, 0, 2, 4, 3, 1, 1, 1, 3, 2, 0, 2, 2, 3, 1, 4, 3,
        4, 4, 2, 4, 4, 2, 2, 4, 2, 1, 2, 0, 4, 1, 4, 4, 1, 2, 2, 1, 3, 3, 0, 4,
        2, 0, 0, 2]) 


Batch # 20
Feature shape: torch.Size([100, 2883])
tensor([[ 0.4204,  0.4805,  0.5215,  ..., -0.3367, -0.3581, -0.3629],
        [ 0.4068,  0.4253,  0.4108,  ..., -1.7364, -1.7171, -1.6308],
        [ 0.6376,  0.7012,  0.1061,  ...,  1.9538,  1.9490,  1.8598],
        ...,
        [-0.2855, -0.4299, -0.3785,  ...,  1.4925,  1.2221,  1.0615],
        [ 1.5879,  1.6668,  1.6708,  ..., -0.0504, -0.1211, -0.1281],
        [-0.4484, -0.4713, -0.4893,  ..., -1.2752, -1.3220, -1.3334]]) 

Label shape: torch.Size([100])
tensor([0, 2, 4, 1, 4, 0, 4, 4, 3, 0, 4, 0, 4, 1, 2, 0, 4, 2, 4, 1, 3, 0, 3, 2,
        1, 0, 0, 2, 1, 3, 3, 3, 3, 3, 1, 1, 3, 2, 3, 3, 3, 4, 2, 3, 3, 3, 2, 1,
        2, 2, 0, 3, 3, 1, 4, 4, 0, 0, 1, 1, 1, 0, 3, 0, 1, 0, 3, 3, 1, 2, 2, 2,
        4, 0, 2, 1, 0, 1, 2, 3, 1, 0, 4, 4, 1, 4, 2, 0, 4, 0, 2, 0, 2, 4, 1, 1,
        1, 3, 0, 2]) 


Batch # 21
Feature shape: torch.Size([100, 2883])
tensor([[ 0.1760,  0.0805, -0.6693,  ...,  0.6177,  0.5268,  0.5136],
        [ 0.1489,  0.1219,  0.1061,  ..., -0.8298, -0.6267, -0.6290],
        [-0.0276, -0.6782, -0.1570,  ..., -1.7046, -1.3220, -0.5507],
        ...,
        [-0.9100, -0.9265, -0.9186,  ..., -0.4480, -0.5635, -0.5820],
        [ 1.6422,  1.6943,  1.6985,  ...,  0.5700,  0.9693,  0.9519],
        [-0.7743, -0.9265, -0.7524,  ..., -0.7503, -0.7215, -0.6916]]) 

Label shape: torch.Size([100])
tensor([4, 0, 4, 3, 3, 3, 0, 0, 4, 4, 1, 1, 1, 1, 1, 2, 2, 3, 0, 0, 4, 3, 2, 0,
        3, 3, 3, 3, 4, 2, 3, 3, 1, 1, 4, 4, 2, 0, 4, 0, 4, 4, 1, 3, 1, 3, 2, 1,
        1, 4, 0, 2, 1, 3, 3, 4, 0, 1, 4, 2, 2, 3, 4, 0, 1, 3, 4, 4, 3, 3, 4, 4,
        2, 0, 1, 1, 4, 2, 2, 4, 2, 2, 3, 3, 3, 4, 2, 3, 4, 2, 1, 1, 1, 2, 3, 4,
        0, 4, 4, 1]) 


Batch # 22
Feature shape: torch.Size([100, 2883])
tensor([[-0.8150, -0.4299, -1.0432,  ..., -1.4183, -1.3536, -1.4742],
        [-1.1951, -1.1058, -1.0294,  ...,  1.0790,  1.6803,  1.7972],
        [-1.4395, -1.4368, -1.4171,  ...,  1.6675,  1.6961,  1.7033],
        ...,
        [ 0.3254,  0.3012,  0.3554,  ...,  0.0292,  0.2108,  0.3884],
        [-0.4892, -0.4851, -0.5032,  ...,  1.2221,  1.4907,  1.0145],
        [ 1.7237,  1.7357,  1.7401,  ..., -0.6707, -0.4213, -0.4881]]) 

Label shape: torch.Size([100])
tensor([3, 2, 3, 2, 0, 1, 3, 0, 3, 3, 0, 2, 4, 4, 0, 2, 0, 4, 4, 1, 4, 3, 0, 3,
        2, 1, 3, 2, 1, 2, 4, 0, 0, 0, 2, 1, 0, 0, 3, 1, 2, 3, 2, 0, 4, 3, 3, 4,
        4, 3, 1, 0, 2, 3, 0, 4, 3, 4, 0, 3, 0, 4, 0, 3, 3, 3, 1, 3, 0, 1, 0, 2,
        2, 4, 4, 4, 4, 2, 1, 2, 1, 3, 1, 3, 3, 0, 2, 2, 2, 0, 4, 3, 0, 2, 0, 1,
        2, 0, 0, 4]) 


Batch # 23
Feature shape: torch.Size([100, 2883])
tensor([[-1.1408, -1.0092, -0.9740,  ..., -0.3844,  0.0370,  0.4354],
        [-1.0051, -1.0230, -1.0294,  ..., -1.0048, -1.0218, -1.0516],
        [ 0.2846, -0.2230,  0.2169,  ..., -0.0027, -0.0579, -0.1907],
        ...,
        [-0.2312, -0.5954,  0.2030,  ...,  0.2359,  0.2898,  0.3884],
        [ 1.7101,  1.7081,  1.6985,  ...,  2.3038,  2.2808,  2.2511],
        [-1.3444, -0.7058, -0.3093,  ..., -0.9570, -0.9744, -0.9734]]) 

Label shape: torch.Size([100])
tensor([1, 1, 2, 2, 1, 0, 2, 3, 4, 3, 2, 2, 1, 3, 4, 1, 4, 1, 3, 0, 3, 1, 0, 1,
        1, 3, 4, 2, 0, 3, 1, 1, 2, 4, 1, 2, 2, 4, 1, 3, 2, 1, 3, 1, 1, 4, 1, 2,
        1, 3, 2, 0, 2, 4, 4, 0, 1, 3, 2, 4, 0, 2, 1, 2, 3, 4, 0, 1, 0, 2, 3, 0,
        3, 0, 4, 1, 0, 3, 4, 4, 4, 4, 4, 3, 2, 4, 0, 2, 0, 0, 2, 3, 0, 0, 1, 3,
        1, 3, 1, 0]) 


Batch # 24
Feature shape: torch.Size([100, 2883])
tensor([[-0.1769, -0.1540, -0.2816,  ..., -0.5594, -0.2949,  0.0910],
        [-0.3398, -0.3333, -0.3370,  ..., -0.6866, -0.5951, -0.8794],
        [ 1.1263,  1.1426,  1.2000,  ...,  0.3155,  0.6374,  0.7797],
        ...,
        [ 0.4068,  0.4529,  0.4246,  ...,  1.5562,  1.5381,  1.5467],
        [-0.5842, -0.3747, -0.6139,  ..., -0.8616, -0.5793, -0.8951],
        [ 0.5019,  0.1494,  0.0646,  ..., -0.0822, -0.1843, -0.0185]]) 

Label shape: torch.Size([100])
tensor([1, 2, 0, 1, 1, 1, 3, 3, 4, 2, 0, 2, 3, 2, 0, 3, 2, 4, 3, 3, 0, 2, 4, 4,
        0, 1, 3, 4, 2, 4, 0, 4, 2, 2, 4, 2, 4, 3, 1, 3, 4, 0, 4, 0, 1, 0, 3, 2,
        1, 4, 2, 1, 3, 3, 1, 4, 0, 0, 1, 3, 3, 1, 0, 1, 3, 0, 0, 0, 4, 1, 3, 0,
        3, 4, 4, 2, 4, 2, 2, 4, 0, 2, 2, 1, 3, 0, 3, 4, 0, 2, 2, 3, 2, 2, 1, 4,
        3, 4, 3, 4]) 


Batch # 25
Feature shape: torch.Size([100, 2883])
tensor([[-1.0051, -1.0230, -0.9601,  ..., -0.3685, -0.4845, -0.5351],
        [-0.4756, -0.2368, -0.7940,  ...,  0.8086,  0.7322,  0.6076],
        [-0.2855, -0.2782, -0.2678,  ..., -0.3844, -0.4845, -0.5038],
        ...,
        [ 1.6151,  1.5288,  1.5878,  ...,  1.6198,  1.5065,  1.2650],
        [-1.5888, -1.3541, -1.1401,  ...,  1.1267,  1.2063,  1.1398],
        [ 0.7055,  0.3288,  0.1061,  ..., -1.0207, -0.8954, -1.1299]]) 

Label shape: torch.Size([100])
tensor([0, 1, 0, 4, 3, 4, 1, 3, 3, 2, 2, 3, 3, 4, 2, 0, 0, 0, 4, 3, 3, 4, 4, 0,
        4, 3, 2, 1, 2, 4, 0, 0, 0, 1, 4, 0, 3, 2, 1, 4, 4, 4, 2, 2, 2, 4, 1, 2,
        3, 1, 3, 3, 3, 0, 1, 2, 3, 0, 4, 3, 0, 1, 4, 1, 3, 0, 1, 4, 0, 2, 0, 2,
        0, 0, 1, 3, 0, 0, 1, 4, 1, 0, 4, 3, 3, 0, 0, 1, 3, 3, 1, 3, 4, 0, 3, 4,
        1, 0, 4, 2]) 


Batch # 26
Feature shape: torch.Size([100, 2883])
tensor([[ 1.7237,  1.7633,  1.7539,  ...,  2.2879,  2.2650,  2.2354],
        [-0.8964, -0.1264, -0.3232,  ..., -0.2412, -0.2159, -0.3629],
        [-0.6114, -0.6092, -0.6555,  ...,  0.3314,  0.3214,  0.3258],
        ...,
        [ 0.2168,  0.0115, -0.1847,  ..., -1.2433, -1.2746, -1.2864],
        [ 1.0720,  0.9633,  1.0893,  ...,  0.8086,  0.8112,  0.7954],
        [-0.2855, -0.2920, -0.3093,  ...,  0.1564,  0.1476,  0.0597]]) 

Label shape: torch.Size([100])
tensor([1, 4, 1, 0, 1, 0, 0, 3, 4, 1, 1, 0, 1, 3, 0, 2, 2, 3, 4, 0, 3, 1, 2, 2,
        0, 0, 3, 0, 1, 0, 0, 0, 3, 1, 4, 1, 3, 0, 2, 4, 2, 4, 1, 3, 0, 3, 1, 4,
        4, 0, 0, 2, 4, 1, 1, 1, 2, 0, 4, 3, 3, 0, 0, 0, 3, 2, 1, 4, 2, 2, 4, 4,
        4, 1, 4, 0, 1, 4, 3, 3, 0, 3, 0, 4, 0, 3, 2, 4, 2, 3, 3, 3, 0, 2, 3, 2,
        1, 3, 4, 0]) 


Batch # 27
Feature shape: torch.Size([100, 2883])
tensor([[-1.1951, -0.9678, -0.8355,  ..., -0.3844, -0.4687, -0.3786],
        [ 0.2711,  0.2460,  0.3138,  ...,  2.2720,  2.2334,  2.1885],
        [ 1.0449,  1.1840,  1.1862,  ..., -0.2731, -0.2949, -0.3003],
        ...,
        [ 0.5290,  0.7978,  0.7154,  ..., -0.2572, -0.4687, -0.6133],
        [-0.5842, -0.5954, -0.6139,  ..., -0.7025, -0.5793, -0.4412],
        [-0.1769, -1.2575, -0.7109,  ...,  0.4586,  0.4952,  0.3884]]) 

Label shape: torch.Size([100])
tensor([3, 1, 4, 0, 2, 0, 1, 4, 4, 1, 1, 4, 4, 3, 3, 4, 0, 4, 2, 4, 0, 0, 4, 1,
        3, 2, 1, 2, 4, 0, 3, 4, 0, 4, 2, 0, 3, 3, 2, 0, 2, 1, 3, 2, 2, 1, 3, 0,
        2, 0, 0, 3, 3, 4, 2, 2, 4, 2, 3, 3, 4, 0, 2, 0, 1, 1, 0, 4, 3, 4, 4, 4,
        2, 4, 4, 0, 2, 2, 1, 1, 0, 2, 1, 4, 2, 3, 1, 2, 1, 0, 0, 1, 3, 0, 1, 0,
        4, 1, 0, 2]) 


Batch # 28
Feature shape: torch.Size([13, 2883])
tensor([[-0.7064, -0.5678, -0.7386,  ...,  1.1267,  1.0957,  1.1241],
        [ 0.3932,  0.4115,  0.4108,  ..., -0.1140, -0.1527, -0.2846],
        [-0.2312, -0.1540, -0.0739,  ..., -0.2094, -0.3739, -0.9577],
        ...,
        [ 0.9091,  0.9771,  1.0339,  ..., -0.3526, -0.3107, -0.4099],
        [ 0.8820,  0.8805,  0.9369,  ..., -1.1797, -1.3062, -1.4429],
        [ 0.2303, -0.3333, -0.2678,  ...,  2.3038,  2.2808,  2.2511]]) 

Label shape: torch.Size([13])
tensor([1, 0, 3, 4, 1, 2, 4, 0, 4, 2, 0, 0, 1]) 


Confusion Matrix¶

The top-level program grade.py returns two types of feedback about your model.

  • The accuracy on the testing set.
  • A confusion matrix for the testing set.

A confusion matrix is a very useful tool for evaluating multi-class classification problems, as it helps in identifying possible sources of imbalance in your dataset - and can offer precious insights into possible biases in your training procedure.

Specifically, in a classification problem with k classes, a confusion matrix will have k rows and k columns. Each row corresponds to the ground truth label of the data points - and each column refers to the predicted label by your classifier. Each entry on the matrix contains a count of the corresponding tuple (ground_truth label, predicted_label). In other words, all elements in the diagonal of this square matrix have been correctly classified - and all other elements count as mistakes. For instance, if your matrix has many entries in [0,1]. This will mean that your classifier tends to mistake points belonging to class 0 for points belonging to class 1. Further details can be found on this Wikipedia page.

Neural Network¶

image.png

Now it's time for you to create a neural network. You will implement a neural network in Class NeuralNet, specify a loss function and optimizer in the function fit(), and write a training loop in function train().
In each training iteration, the input of your network is a batch of preprocessed image data of size (batch size, 2883). The number of neurons in the output layer should be equal to the number of categories.
For this assignment, you can use Cross Entropy Loss. Notice that because PyTorch's CrossEntropyLoss incorporates a softmax function, you do not need to explicitly include an normalization function in the last layer of your network
.

image.png

To get a full score, the accuracy of your network must be above 0.61 on the visible testing set, and above 0.59 on the hidden testing set. The structure of the neural network is completely up to you. You should be able to get around 0.62 testing-set accuracy with a two-layer network with no more than 200 hidden neurons.

Notice that the autograder will pass in the number of training epochs and the batch size. You don't control those. However, you do control the neural net's learning rate. If you are confident about a model you have implemented but are not able to pass the accuracy thresholds on gradescope, try increasing the learning rate. Be aware, however, that using a very high learning rate may worse performance since the model may begin to oscillate around the optimal parameter settings.

In [8]:
import submitted
import importlib

After you have implemented a network, you can use the code below to test your model.

In [9]:
# Helper functions to test your model, you can ignore the implementations

def ComputeAccuracy(model):
    pred_values = model(test_set_p)  # Predicted value of the testing set
    pred_values = pred_values.detach().numpy()  # Convert the tensor to a numpy array
    pred_labels = np.argmax(pred_values, axis=1)  # Predicted labels
    accuracy, conf_mat = reader.compute_accuracies(pred_labels, test_labels)
    print("Accuracy:", accuracy)
    print("Confusion matrix:")
    print(conf_mat, "\n")
    
def Predict(model, index):
    pred_values = model(test_set_p[index])
    pred_values = pred_values.detach().numpy()  # Convert the tensor to a numpy array
    pred_labels = np.argmax(pred_values)  # Predicted labels
    show_test_image(index)
    print("Your prediction:", names[pred_labels.item()],)
In [10]:
# Train your network
reader.init_seeds(42) # fix random seed
importlib.reload(submitted)
model, loss_fn, optimizer = submitted.fit(train_loader, test_loader, 50) # the last parameter is the number of epochs
Epoch # 0
Epoch # 1
Epoch # 2
Epoch # 3
Epoch # 4
Epoch # 5
Epoch # 6
Epoch # 7
Epoch # 8
Epoch # 9
Epoch # 10
Epoch # 11
Epoch # 12
Epoch # 13
Epoch # 14
Epoch # 15
Epoch # 16
Epoch # 17
Epoch # 18
Epoch # 19
Epoch # 20
Epoch # 21
Epoch # 22
Epoch # 23
Epoch # 24
Epoch # 25
Epoch # 26
Epoch # 27
Epoch # 28
Epoch # 29
Epoch # 30
Epoch # 31
Epoch # 32
Epoch # 33
Epoch # 34
Epoch # 35
Epoch # 36
Epoch # 37
Epoch # 38
Epoch # 39
Epoch # 40
Epoch # 41
Epoch # 42
Epoch # 43
Epoch # 44
Epoch # 45
Epoch # 46
Epoch # 47
Epoch # 48
Epoch # 49

Let's print the output for a random image in testing set, note that the output varies on a network-by-network basis, so your network's output will probably be different from the one below. Since we have five categories, the output is a 5-dimensional tensor. The label 0 is the one with the largest value in the output tensor, therefore this image is classified by our model into label 0, which is a ship.

In [11]:
# Let's print the output for a random image in testing set
index = 5
show_test_image(index)

output = model(test_set_p[index])
print(output)
tensor([ 7.6241, -3.2049,  4.7004, -1.1417, -5.9385], grad_fn=<AddBackward0>)
In [12]:
# You can use the helper function to compute the accuracy and confusion matrix
ComputeAccuracy(model)
Accuracy: 0.6264674493062967
Confusion matrix:
[[146.  24.   8.   5.   7.]
 [ 27. 123.  11.  22.  19.]
 [ 13.  13.  87.  45.  28.]
 [  9.   4.  34. 122.  17.]
 [ 14.  10.  23.  17. 109.]] 

You can use the helper function to compare your network's prediction with the true label.

Good prediction!

In [13]:
# make predictions
Predict(model, 24) # the number is the index of image in testing sets, feel free to change it
Your prediction: automobile

Bad prediction!

In [14]:
Predict(model, 23) # the number is the index of image in testing sets, feel free to change it
Your prediction: horse

Grade your homework¶

If you've reached this point, and all of the above sections work, then you're ready to try grading your homework! Before you submit it to Gradescope, try grading it on your own machine. This will run some visible test cases.

The exclamation point (!) tells python to run the following as a shell command. Obviously you don't need to run the code this way -- this usage is here just to remind you that you can also, if you wish, run this command in a terminal window.

In [15]:
!python grade.py
Epoch # 0
Epoch # 1
Epoch # 2
Epoch # 3
Epoch # 4
Epoch # 5
Epoch # 6
Epoch # 7
Epoch # 8
Epoch # 9
Epoch # 10
Epoch # 11
Epoch # 12
Epoch # 13
Epoch # 14
Epoch # 15
Epoch # 16
Epoch # 17
Epoch # 18
Epoch # 19
Epoch # 20
Epoch # 21
Epoch # 22
Epoch # 23
Epoch # 24
Epoch # 25
Epoch # 26
Epoch # 27
Epoch # 28
Epoch # 29
Epoch # 30
Epoch # 31
Epoch # 32
Epoch # 33
Epoch # 34
Epoch # 35
Epoch # 36
Epoch # 37
Epoch # 38
Epoch # 39
Epoch # 40
Epoch # 41
Epoch # 42
Epoch # 43
Epoch # 44
Epoch # 45
Epoch # 46
Epoch # 47
Epoch # 48
Epoch # 49
Total number of network parameters:  138677

 Accuracy: 0.6264674493062967

Confusion Matrix = 
 [[146.  24.   8.   5.   7.]
 [ 27. 123.  11.  22.  19.]
 [ 13.  13.  87.  45.  28.]
 [  9.   4.  34. 122.  17.]
 [ 14.  10.  23.  17. 109.]]
+5 points for accuracy above 0.15
+5 points for accuracy above 0.25
+5 points for accuracy above 0.48
+5 points for accuracy above 0.55
+10 points for accuracy above 0.57
+10 points for accuracy above 0.61
...
----------------------------------------------------------------------
Ran 3 tests in 1.599s

OK

If you got any 'E' marks, it means that your code generated some runtime errors, and you need to debug those.

If you got any 'F' marks, it means that your code ran without errors, but that it generated results that are different from the solutions.

If neither of those things happened, and your result was a series of dots, then your code works perfectly.

If you're not sure, you can try running grade.py with the -j option. This will produce a JSON results file, in which the best score you can get is 50.

Now you should try uploading submitted.py to Gradescope.

Gradescope will run the same visible tests that you just ran on your own machine, plus some additional hidden tests. It's possible that your code passes all the visible tests, but fails the hidden tests. If that happens, then it probably means that you hard-coded a number into your function definition, instead of using the input parameter that you were supposed to use. Debug by running your function with a variety of different input parameters, and see if you can get it to respond correctly in all cases.

Once your code works perfectly on Gradescope, with no errors, then you are done with the MP. Congratulations!

Extra credit¶

You can earn extra credits worth 10% of this MP if the accuracy of your network is above 0.66

Some tips:¶

  1. Choose a good activation function.
  2. L2 Regularization.
  3. Convolutional neural network.