{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q3sNohViD64Y"
   },
   "source": [
    "# PyTorch Introduction\n",
    "\n",
    "Welcome to the introduction of PyTorch. PyTorch is a scientific computing package targeted for two main purposes:\n",
    "\n",
    "1. A replacement for NumPy with the ability to use the power of GPUs.\n",
    "\n",
    "2. A deep learning framework that enables the flexible and swift building of neural network models.\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "\n",
    "### Goals of this tutorial\n",
    "\n",
    "- Understanding PyTorch's Tensor and neural networks libraries at an overview level.\n",
    "\n",
    "- Training a neural network using PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9JPpNm2qD64b"
   },
   "source": [
    "## Enable GPUs on Colab\n",
    "\n",
    "Having a library that has GPU support is one thing, the other is actually owning the hardware. Alternatively, you can use google colab though we have to manually enable it.\n",
    "\n",
    "To enable GPU support in Google Colab go to `Menu > Runtime > Change runtime type` and enable the GPU hardware accelerator to speed up your trainings considerably. However, this functionality might not be available at any time.\n",
    "\n",
    "<img src=\"https://i2dl.vc.in.tum.de/static/images/exercise_07/colab_gpu.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rYe6QoM5D64c"
   },
   "source": [
    "\n",
    "# Installing PyTorch\n",
    "\n",
    "Pytorch provides support for accelerating computation using CUDA enabled GPU's. If your workstation has an NVIDIA GPU, install PyTorch along with the CUDA component.\n",
    "\n",
    "#### Install [PyTorch](https://pytorch.org/) and [torchvision](https://github.com/pytorch/vision)\n",
    "\n",
    "For this class we will use the current Pytorch version 2.2.2. To install, please uncomment and run the proper line in the upcoming cell depending on your operating system (and CUDA setup). We won't go into details of the installation process.\n",
    "\n",
    "# **Note** (!!!!!!!!!!!):\n",
    "All packages should be installed on your i2dl conda enviroment. Otherwise, you'd start with mismatching versions loops of differnet libraries, which will make your life really hard later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IzUnH6stD64c",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Install a pip package in the current Jupyter kernel\n",
    "import sys\n",
    "\n",
    "# For google colab. Note: Colab now defaults to numpy 2.0, we downgrade to 1.x. IMPORTANT: Then RESTART the session! Runtime > Restart Session.\n",
    "# !python -m pip install \"numpy<2.0\" torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121\n",
    "\n",
    "# For Linux and probably Windows (CPU)\n",
    "!{sys.executable} -m pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cpu\n",
    "\n",
    "# For Linux and probably Windows (Prerequisites: Nvidia GPU + CUDA toolkit 11.8)\n",
    "# !{sys.executable} -m pip install torch==2.2.2+cu118 torchvision==0.17.2+cu118 torchaudio==2.2.2+cu118 --index-url https://download.pytorch.org/whl/cu118\n",
    "\n",
    "\n",
    "# For OS X/Mac\n",
    "# !{sys.executable} -m pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N7uWweV0D64d",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b>Nvidia GPU</b>\n",
    "    <p>If you have a rather recent Nvidia GPU, you can go ahead and install the CUDA toolkit together with a current version of cudnn (though it is possible to use other versions as long as you build it yourself). Afterwards, you can run the respective line in the cell above.</p>\n",
    "    <p>There are multiple setups on how to install those on both Linux and Windows, but it depends on your setup. If you want to utilize your GPU you have to go through those steps. Use the forum for help if you get stuck.\n",
    "    But, google or ChatGPT are your new best friends.</p>\n",
    "    <br>\n",
    "    <b>Google Colab Pytorch Installation Time</b>\n",
    "    <p>Google colab might use an older/newer version of pytorch. Since we are mostly using default functionality, you should be fine by using the default colab version to avoid the long installation time at your own risk.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ja1QJsjaD64e"
   },
   "source": [
    "#### Checking PyTorch Installation and Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zjRXFqhoD64e",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "print(f\"PyTorch version Installed: {torch.__version__}\\nTorchvision version Installed: {torchvision.__version__}\\n\")\n",
    "if not torch.__version__.startswith(\"2.2\"):\n",
    "    print(\"you are using an another version of PyTorch. We expect PyTorch 2.2. You may continue using your version but it\"\n",
    "          \" might cause dependency and compatibility issues.\")\n",
    "if not torchvision.__version__.startswith(\"0.17\"):\n",
    "    print(\"you are using an another version of torchvision. We expect torchvision 0.17. You can continue with your version but it\"\n",
    "          \" might cause dependency and compatibility issues.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yS_ICCMyD64f"
   },
   "source": [
    "That's the end of installation. Let's dive right into PyTorch!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Mount folder in Colab\n",
    "\n",
    "Uncomment the following cell to mount your gdrive if you are using the notebook in google colab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hPhYm2fjD64a"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "from google.colab import drive\n",
    "import os\n",
    "\n",
    "gdrive_path='/content/gdrive/MyDrive/i2dl/exercise_07'\n",
    "\n",
    "# This will mount your google drive under 'MyDrive'\n",
    "drive.mount('/content/gdrive', force_remount=True)\n",
    "# In order to access the files in this notebook we have to navigate to the correct folder\n",
    "os.chdir(gdrive_path)\n",
    "# Check manually if all files are present\n",
    "print(sorted(os.listdir()))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ySplvG38D64g",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Getting Started\n",
    "\n",
    "In this section you will learn the basic element Tensor and some simple operations in PyTorch.\n",
    "The following block imports the required packages for the rest of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3my6b4_WD64g",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torchvision.transforms as transforms\n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "pd.options.mode.chained_assignment = None  # default='warn'\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True' # To prevent the kernel from dying."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yy1213DcD64g"
   },
   "source": [
    "## 1. Tensors\n",
    "\n",
    "[`torch.Tensor`](https://pytorch.org/docs/stable/tensors.html) is the central class of PyTorch.\n",
    "Tensors are similar to NumPy’s ndarrays. The advantage of using Tensors is that one can easily transfer them from CPU to GPU and therefore computations on tensors can be accelerated with a GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "laG6DZ4aD64h",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 1.1 Initializing Tensor\n",
    "Let us construct a NumPy array and a tensor of shape (2,3) directly from data values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gJbeDFusD64h",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Initializing the Numpy Array\n",
    "array_np = np.array([[1,2,3],[5,6,7]]) # A NumPy array\n",
    "\n",
    "# Initializing the Tensor\n",
    "array_ts = torch.tensor([[1,2,3],[4,5,6]]) # A Tensor\n",
    "\n",
    "print(\"Variable array_np:\\nDatatype: {}\\nShape: {}\".format(type(array_np), array_np.shape))\n",
    "print(\"Values:\\n\", array_np)\n",
    "print(\"\\n\\nVariable array_ts:\\nDatatype {}\\nShape: {}\".format(type(array_ts), array_ts.shape))\n",
    "print(\"Values:\\n\", array_ts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eVhIrwcTD64h",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 1.2 Conversion between NumPy array and Tensor\n",
    "\n",
    "The conversion between NumPy ndarray and PyTorch tensor is quite easy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XcZeWG9rD64h",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Conversion\n",
    "array_np = np.array([1, 2, 3])\n",
    "\n",
    "# Conversion from  a numpy array to a Tensor\n",
    "array_ts_2 = torch.from_numpy(array_np)\n",
    "\n",
    "# Conversion from  Tensor to numpy array\n",
    "array_np_2 = array_ts_2.numpy()\n",
    "\n",
    "# Change a value of the np_array\n",
    "array_np_2[1] = -1\n",
    "\n",
    "# Changes in the numpy array will also change the values in the tensor\n",
    "assert(array_np[1] == array_np_2[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YZupByISD64i",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<div class=\"alert alert-block alert-info\"><b></b> During the conversion, both ndarray and Tensor share the same memory address. Changes in value of one will\n",
    "affect the other.</div>\n",
    "\n",
    "## 1.3 Operations on Tensor\n",
    "\n",
    "### 1.3.1 Indexing\n",
    "\n",
    "We can use the NumPy array-like indexing for Tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "csCarVKVD64i",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Let us take the first two columns from the original tensor array and save it in a new one\n",
    "b = array_ts[:2, :2]\n",
    "\n",
    "# Let's assign the value of first column of the new variable to be zero\n",
    "b[:, 0] = 0\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7M8HIChKD64i"
   },
   "source": [
    "We will now select elements which satisfy a particular condition. In this example, let's find those elements of tensor which are array greater than one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sxS8eRyTD64i",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Index of the elements with value greater than one\n",
    "mask = array_ts > 1\n",
    "new_array = array_ts[mask]\n",
    "print(new_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qoNudarzD64j"
   },
   "source": [
    "Let's try performing the same operation in a single line of code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oz0eehBUD64j",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "c = array_ts[array_ts>1]\n",
    "\n",
    "# Is the result same as the array from the previous cell?\n",
    "print(c == new_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-ZXv3bQbD64j",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 1.3.2 Mathematical operations on Tensor\n",
    "\n",
    "#### Element-wise operations on Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XD0B-aZDD64j",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "x = torch.tensor([[1,2],[3,4]])\n",
    "y = torch.tensor([[5,6],[7,8]])\n",
    "\n",
    "# Elementwise Addition of the tensors\n",
    "# [[ 6.0  8.0]\n",
    "#  [10.0 12.0]]\n",
    "\n",
    "# Addition - Syntax 1\n",
    "print(\"x + y: \\n{}\".format(x + y))\n",
    "\n",
    "# Addition - Syntax 2\n",
    "print(\"x + y: \\n{}\".format(torch.add(x, y)))\n",
    "\n",
    "# Addition - Syntax 3\n",
    "result_add = torch.empty(2, 2)\n",
    "torch.add(x, y, out=result_add)\n",
    "print(\"x + y: \\n{}\".format(result_add))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kcZhQ2HKD64k",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Similar syntax holds for other element-wise operations such as subtraction and multiplication.\n",
    "\n",
    "When dividing two integers in NumPy as well as in PyTorch, the result is always a **float**.   \n",
    "For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "44k0qZ99D64k",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "x_np = np.array([[1,2],[3,4]])\n",
    "y_np = np.array([[5,6],[7,8]])\n",
    "print(x_np / y_np)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xhvjVUNBD64k"
   },
   "source": [
    "## 1.4 Devices\n",
    "\n",
    "When training a neural network, it is important to make sure that all the required tensors as well as the model are on the same device. Tensors can be moved between the CPU and GPU using `.to` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wVw9KrXaD64k"
   },
   "source": [
    "Let us check if a GPU is available. If it is available, we will assign it to `device` and move the tensor `x` to the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GRO3dlaID64k",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)\n",
    "\n",
    "print(f\"Original device: {x.device}\") # \"cpu\"\n",
    "\n",
    "tensor = x.to(device)\n",
    "print(f\"Current device: {tensor.device}\") #\"cpu\" or \"cuda\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "itgM_d46D64l"
   },
   "source": [
    "So `x` has been moved on to a CUDA device for those who have a GPU; otherwise it's still on the CPU.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"><b>Tip:</b> Try including the <b>.to(device)</b> calls in your codes. It is then easier to port the code to run on a GPU.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xXLt5-NsD64l",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# 2. Training a classifier with PyTorch\n",
    "\n",
    "Now that we have introduced PyTorch tensors, we will look at how to use PyTorch to train neural networks. We will do the following steps:\n",
    "\n",
    "1. Load data\n",
    "2. Define a two-layer network\n",
    "3. Define a loss function and optimizer\n",
    "4. Train the network\n",
    "5. Test the network\n",
    "\n",
    "## 2.1 Loading Datasets\n",
    "\n",
    "The general procedure of loading data is :\n",
    "- Extract data from  source\n",
    "- Transform the data into a suitable form (for example, to a Tensor)\n",
    "- Put our data into an object to make it easy to access further on\n",
    "\n",
    "### 2.1.1 Loading the Housing Price dataset\n",
    "\n",
    "We'll use both our `DataLoader` class from the previous exercises and PyTorch's `DataLoader` to load the house price dataset that we used in Exercise 4 to classify the price of the houses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LyGMivV5D64l"
   },
   "source": [
    "Let us fetch the data and setup the `Dataset` class as in Exercise 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XS9b4o-WD64l",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from exercise_code.data.csv_dataset import CSVDataset, get_exercise5_transform\n",
    "from exercise_code.data.dataloader import DataLoader as our_DataLoader\n",
    "\n",
    "# dataloading and preprocessing steps as in ex04\n",
    "target_column = 'SalePrice'\n",
    "i2dl_exercises_path = os.path.dirname(os.path.abspath(os.getcwd()))\n",
    "root_path = os.path.join(i2dl_exercises_path, \"datasets\", 'housing')\n",
    "housing_file_path = os.path.join(root_path, \"housing_train.csv\")\n",
    "download_url = 'https://i2dl.vc.in.tum.de/static/data/housing_train.zip'\n",
    "\n",
    "# Set up the transform to get two prepared columns\n",
    "select_two_columns_transform = get_exercise5_transform()\n",
    "\n",
    "# Set up the dataset\n",
    "our_csv_dataset = CSVDataset(target_column=target_column, root=root_path, download_url=download_url, mode=\"train\",\n",
    "                             transform=select_two_columns_transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TdFrVlGBD64l"
   },
   "source": [
    "We will now set our `DataLoader` class to help us to load batches of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5Edf96RVD64l"
   },
   "outputs": [],
   "source": [
    "# Let's look at the first batch of the data\n",
    "batch_size = 4\n",
    "our_dataloader = our_DataLoader(our_csv_dataset, batch_size=batch_size)\n",
    "\n",
    "for i, item in enumerate(our_dataloader):\n",
    "    print('Batch {}'.format(i))\n",
    "    for key in item:\n",
    "        print(\"\\nDictionary Key:\",key)\n",
    "        print(\"Value Type\",type(item[key]))\n",
    "        print(\"Shape of the Value\",item[key].shape)\n",
    "\n",
    "    if i+1 >= 1:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7l0GbNuSD64l",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In PyTorch we can use the [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) class to accomplish the same objective. It provides more parameters than our `DataLoader` class, such as easy multiprocessing using `num_workers`. You can refer the documentation to learn those additional features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3Ho7JLsiD64l"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "pytorch_dataloader = DataLoader(our_csv_dataset, batch_size=batch_size)\n",
    "\n",
    "# We can use the exact same way to iterate over samples\n",
    "for i, item in enumerate(pytorch_dataloader):\n",
    "    print('Batch {}'.format(i))\n",
    "    for key in item:\n",
    "        print(\"\\nDictionary Key:\",key)\n",
    "        print(\"Value Type\",type(item[key]))\n",
    "        print(\"Shape of the Value\",item[key].shape)\n",
    "\n",
    "    if i+1 >= 1:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5njsf5QWD64l"
   },
   "source": [
    "<div class=\"alert alert-block alert-info\">As seen above, both the data loaders load the data with the same batch size and the data contains 2 features and 1 target. The only difference here is that PyTorch's <code>DataLoader</code> will automatically transform the dataset into <b>Tensor</b> data type.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YfbmjCs8D64m"
   },
   "source": [
    "### 2.1.2 Torchvision\n",
    "\n",
    "Specifically for computer vision, the `torchvision` packages has data loaders for many common datasets such\n",
    "as ImageNet, FashionMNIST, MNIST and additional data transformers for images in `torchvision.datasets` and `torch.utils.data.DataLoader` modules.\n",
    "\n",
    "This is highly convenient and is useful in avoiding  to write boilerplate code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AOpLpJwZD64m"
   },
   "source": [
    "Let's try loading the [`Fashion-MNIST`](https://github.com/zalandoresearch/fashion-mnist) dataset. It has  gray-scale images of size $28* 28$ belonging to 10 different classes of clothing accessories such as T-Shirt, Trousers, Sneakers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QbuPfmY_D64m"
   },
   "source": [
    "`transforms.Compose` creates a series of transformation to prepare the dataset.\n",
    "- `transforms.ToTensor` converts `PIL image` or numpy.ndarray $(H \\times W\\times C)$ in the range [0,255] to a `torch.FloatTensor` of shape $(C \\times H \\times W)$ in the range [0.0, 1.0].\n",
    "\n",
    "- `transforms.Normalize` normalizes a tensor image with the provided mean and standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jPFTBNe8D64m"
   },
   "outputs": [],
   "source": [
    "# Mean and standard deviations have to be sequences (e.g. tuples), hence we add a comma after the values\n",
    "transform = transforms.Compose([transforms.ToTensor(),\n",
    "                                transforms.Normalize((0.5,),(0.5,))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A6mxTFjLD64m"
   },
   "source": [
    "`datasets.FashionMNIST` downloads the Fashion MNIST dataset and transforms it using our previous cell definition.  \n",
    "By setting the value of `train`, we get the training and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KOftW5icD64m"
   },
   "outputs": [],
   "source": [
    "fashion_mnist_dataset = torchvision.datasets.FashionMNIST(root='../datasets', train=True,\n",
    "                                                          download=True, transform=transform)\n",
    "fashion_mnist_test_dataset = torchvision.datasets.FashionMNIST(root='../datasets', train=False,\n",
    "                                                          download=True, transform=transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sGzg1F4pD64m"
   },
   "source": [
    "\n",
    " `torch.utils.data.Dataloader` takes our training data or test data with parameter\n",
    "`batch_size` and `shuffle`. The variable `batch_size` defines how many samples per batch to load. The variable `shuffle=True` makes the data reshuffled at every epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lhfit6WND64n"
   },
   "outputs": [],
   "source": [
    "fashion_mnist_dataloader = DataLoader(fashion_mnist_dataset, batch_size=8)\n",
    "fashion_mnist_test_dataloader = DataLoader(fashion_mnist_test_dataset, batch_size=8)\n",
    "\n",
    "classes = ('T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
    "           'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nzR62hJaD64n"
   },
   "source": [
    "Let's look at the first batch of data from the `fashion_mnist_dataloader`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U3RsmlAxD64n",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# We can use the exact same way to iterate over samples\n",
    "for i, item in enumerate(fashion_mnist_dataloader):\n",
    "    print('Batch {}'.format(i))\n",
    "    image, label = item\n",
    "    print(f\"Datatype of Image: {type(image)}\")\n",
    "    print(f\"Shape of the Image: {image.shape}\")\n",
    "    print(f\"Label Values: {label}\")\n",
    "\n",
    "    if i+1 >= 1:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t19FOxvkD64n",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Since we loaded the data with `batch_size` 8, the shape of the input is (8, 1, 28, 28).\n",
    "\n",
    "Let's look at  some of the training images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3Kb5z_OaD64n",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def imshow(img):\n",
    "    img = img / 2 + 0.5 # unormalize\n",
    "    npimg = img.numpy()\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "    plt.show()\n",
    "\n",
    "# get some random training images\n",
    "dataiter = iter(fashion_mnist_dataloader)\n",
    "for images, labels in fashion_mnist_dataloader:\n",
    "    break\n",
    "# show images\n",
    "imshow(torchvision.utils.make_grid(images))\n",
    "# print labels\n",
    "print(' '.join('%5s' % classes[labels[j]] for j in range(8)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H8IkKibbD64o",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 2.2 Defining the Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SPflFQQYD64o",
    "pycharm": {
     "is_executing": true,
     "name": "#%% md\n"
    }
   },
   "source": [
    "We implemented the `ClassificationNet` class in Exercise 06. Let's use it here again for Fashion-MNIST.\n",
    "\n",
    "Have a look at our lengthy implementation first/again to appreciate the upcoming shortness ;)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Gr3tock1D64o",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from exercise_code.networks.classification_net import ClassificationNet\n",
    "hidden_size = 100\n",
    "std = 1.0\n",
    "model_ex06 = ClassificationNet(num_layer=2,input_size=1*28*28, hidden_size=hidden_size, std=std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pLVIp3qOD64o"
   },
   "source": [
    "PyTorch provides a `nn.Module` that builds neural networks. Now, we will use it to define our network class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yRGcOuWmD64o",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self, activation=nn.Sigmoid(),\n",
    "                 input_size=1*28*28, hidden_size=100, classes=10):\n",
    "\n",
    "        super().__init__()\n",
    "        self.input_size = input_size\n",
    "\n",
    "        # Here we initialize our activation and set up our two linear layers\n",
    "        self.activation = activation\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.fc2 = nn.Linear(hidden_size, classes)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, self.input_size) # flatten\n",
    "        x = self.fc1(x)\n",
    "        x = self.activation(x)\n",
    "        x = self.fc2(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JyYhuEoyD64o",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Looking at the constructor of `Net`, we have,\n",
    " - `super().__init__` creates a class that inherits attributes and behaviors from the parent class.\n",
    "\n",
    " - `self.fc1` creates an affine layer with `input_size` inputs and `hidden_size` outputs.\n",
    "\n",
    " - `self.fc2` is the second affine layer.\n",
    "\n",
    "The `Forward` function defines the forward pass of the mode.:\n",
    "\n",
    " - Input `x` is flattened with `x = x.view(-1, self.input_size)` to be able to use as input to the affine layer.\n",
    "\n",
    " - Apply `fc1`, `activation`, `fc2` sequentially to complete the network.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HE2G5lLVD64p"
   },
   "source": [
    "Central to all neural networks in PyTorch is the [`autograd`](https://pytorch.org/docs/stable/autograd.html) package. It provides automatic differentiation for all operations on Tensors.\n",
    "If we set the attribute `.requires_grad` of `torch.Tensor` as `True`, it tracks all operations applied on that tensor. Once all the computations are finished, the function `.backward()` computes the gradients into the `Tensor.grad` variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "18MLosnXD64p"
   },
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "Thanks to the <b>autograd</b> package, we just have to define the <b>forward()</b> function. We can use any of the Tensor operations in the <b>forward()</b>  function.\n",
    " The <b>backward()</b> function (where gradients are computed through back-propagation) is automatically defined by PyTorch.\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8mv0pMxtD64p"
   },
   "source": [
    "We can use `print()` to look at all the defined layers of the network (but it won't show the information of the forward pass).\n",
    "\n",
    "The learned parameters of a model are returned by `[model_name].parameters()`. We can also access the parameters of different layers by `[model_name].[layer_name].parameters()`.\n",
    "\n",
    "Let's create an instance of the `Net` model and look at the parameters matrix shape for each of the layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yza4j1BtD64p",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "net = Net()\n",
    "# Always remember to move the network to the GPU/CPU depending on device\n",
    "net = net.to(device)\n",
    "\n",
    "print(net)\n",
    "\n",
    "print(\"Shapes of the Parameter Matrix:\")\n",
    "for parameter in net.parameters():\n",
    "        print(parameter.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fHamNxXND64p",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 2.3 Defining the Loss function and optimizer\n",
    "\n",
    "Since it is a multi-class classification, we will use the Cross-Entropy loss and optimize it using SGD with momentum. We had implemented SGD with momentum in Exercise 05. Have a look at the implementations in `exercise_code/networks/optimizer.py` and `exercise_code/networks/loss.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MBXX6SMuD64p",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from exercise_code.networks.optimizer import sgd_momentum\n",
    "from exercise_code.networks.loss import CrossEntropyFromLogits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r9S7bZT3D64q",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The `torch.nn` and `torch.optim` modules include a variety of loss functions and optimizers. We will initialize an instance of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ds_v6zsBD64q",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ejtpLKSnD64q",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 2.4 Training the network\n",
    "\n",
    "We have completed setting up the dataloader, loss function as well as the optimizer. We are now all set for training the network.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "99BU25dKD64q",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Initializing the list for storing the loss and accuracy\n",
    "\n",
    "train_loss_history = [] # loss\n",
    "train_acc_history = [] # accuracy\n",
    "\n",
    "for epoch in range(2):\n",
    "\n",
    "\n",
    "    running_loss = 0.0\n",
    "    correct = 0.0\n",
    "    total = 0\n",
    "\n",
    "    # Iterating through the minibatches of the data\n",
    "\n",
    "    for i, data in enumerate(fashion_mnist_dataloader, 0):\n",
    "\n",
    "        # data is a tuple of (inputs, labels)\n",
    "        X, y = data\n",
    "\n",
    "        X = X.to(device)\n",
    "        y = y.to(device)\n",
    "\n",
    "        # Reset the parameter gradients  for the current  minibatch iteration\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "\n",
    "        y_pred = net(X)             # Perform a forward pass on the network with inputs\n",
    "        loss = criterion(y_pred, y) # calculate the loss with the network predictions and ground Truth\n",
    "        loss.backward()             # Perform a backward pass to calculate the gradients\n",
    "        optimizer.step()            # Optimize the network parameters with calculated gradients\n",
    "\n",
    "\n",
    "        # Accumulate the loss and calculate the accuracy of predictions\n",
    "        running_loss += loss.item()\n",
    "        _, preds = torch.max(y_pred, 1) #convert output probabilities of each class to a singular class prediction\n",
    "        correct += preds.eq(y).sum().item()\n",
    "        total += y.size(0)\n",
    "\n",
    "        # Print statistics to console\n",
    "        if i % 1000 == 999: # print every 1000 mini-batches\n",
    "            running_loss /= 1000\n",
    "            correct /= total\n",
    "            print(\"[Epoch %d, Iteration %5d] loss: %.3f acc: %.2f %%\" % (epoch+1, i+1, running_loss, 100*correct))\n",
    "            train_loss_history.append(running_loss)\n",
    "            train_acc_history.append(correct)\n",
    "            running_loss = 0.0\n",
    "            correct = 0.0\n",
    "            total = 0\n",
    "\n",
    "print('FINISH.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nKSNYQZED64q",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "So the general training pass is summarized below:\n",
    "\n",
    "- `zero_grad()`: Zero the gradient buffers of all the model parameters to start the current minibatch iteration.\n",
    "\n",
    "- `y_pred = net(X)`: Make a forward pass through the network by passing the images to the model to get the predictions, which are log probabilities of image belonging to each of the class.\n",
    "\n",
    "- `loss = criterion(y_pred, y)`: Calculate the loss from the generated predictions and the training data `y`.\n",
    "\n",
    "- `loss.backward()`: Perform a backward pass through the network to calculate the gradients for model parameters.\n",
    "\n",
    "- `optimizer.step()`: Do an optimization step to update the model parameters using the calculated gradients.\n",
    "\n",
    "We keep tracking the training loss and accuracy over time. The following plot shows average values for train loss and accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lkzK49Y6D64q",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(train_acc_history)\n",
    "plt.plot(train_loss_history)\n",
    "plt.title(\"FashionMNIST\")\n",
    "plt.xlabel('iteration')\n",
    "plt.ylabel('acc/loss')\n",
    "plt.legend(['acc', 'loss'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kJkgLIRqD64r"
   },
   "source": [
    "## 2.5 Testing the performance of the model\n",
    "\n",
    "We have trained the network for 2 passes over the entire training dataset. Let's check the model performance using the test data.\n",
    "We will pass the test data to the model to predict the class label and check it against the ground-truth.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "caugU33ED64r"
   },
   "outputs": [],
   "source": [
    "# obtain one batch of test images\n",
    "dataiter = iter(fashion_mnist_test_dataloader)\n",
    "images, labels = dataiter.__next__()\n",
    "images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "# get sample outputs\n",
    "outputs = net(images)\n",
    "# convert output probabilites to predicted class\n",
    "_, predicted = torch.max(outputs, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "md8FxDfqD64r",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "We will visualize the results to display the test images and their labels in the following format: `predicted (ground-truth)`. The text will be green for accurately classified examples and red for incorrect predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3M26Kt15D64r"
   },
   "outputs": [],
   "source": [
    "# prep images for display\n",
    "if not isinstance(images, np.ndarray):\n",
    "    images = images.cpu().numpy()\n",
    "\n",
    "# plot the images in the batch, along with predicted and true labels\n",
    "fig = plt.figure(figsize=(25,4))\n",
    "for idx in range(8):\n",
    "    ax = fig.add_subplot(2, 8//2, idx+1, xticks=[], yticks=[])\n",
    "    ax.imshow(np.squeeze(images[idx]), cmap='gray')\n",
    "    ax.set_title(f\"{classes[predicted[idx]]} ({classes[labels[idx]]})\",\n",
    "                color=\"green\" if predicted[idx]==labels[idx] else \"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FRB3ROO1D64r",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's find which classes of images performed well, and the classes that did not perform well!  \n",
    "`torch.no_grad()` makes sure that gradients are not calculated for the tensors since we only are performing a forward pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VAqXB1U2D64r",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class_correct = list(0. for i in range(10))\n",
    "class_total = list(0. for i in range(10))\n",
    "\n",
    "with torch.no_grad():\n",
    "    for data in fashion_mnist_test_dataloader:\n",
    "        images, labels = data\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        outputs = net(images)\n",
    "        _, predicted = torch.max(outputs, 1)\n",
    "        c = (predicted == labels).squeeze()\n",
    "        for i in range(8):\n",
    "            label = labels[i]\n",
    "            class_correct[label] += c[i].item()\n",
    "            class_total[label] += 1\n",
    "\n",
    "\n",
    "for i in range(10):\n",
    "    print('Accuracy of %11s: %2d %%' % (classes[i], 100 * class_correct[i] / class_total[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9fqItsKLD64r"
   },
   "source": [
    "That's the end of the `PyTorch` Tutorial. In the next notebook, we will look at [`TensorBoard`](https://www.tensorflow.org/tensorboard) which helps us visualize the results of the training process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pIPvH__vD64s",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## References\n",
    "\n",
    "1. [PyTorch Tutorial](https://pytorch.org/tutorials/)\n",
    "\n",
    "2. [Fashion MNIST dataset training using PyTorch](https://medium.com/@aaysbt/fashion-mnist-data-training-using-pytorch-7f6ad71e96f4)\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "i2dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
