{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "026e0255",
   "metadata": {},
   "source": [
    "# Embeddings\n",
    "\n",
    "Let's discuss the input to our model first: So far, with images, its quite clear that similar values on a pixel level correspond to somewhat similar images. Now, how about text?\n",
    "\n",
    "As discussed in the tokenizer notebook, our words or sub-words that make up a sentence are transformed into individual IDs.\n",
    "Imagine what would happen if we used these token IDs directly as an input to the neural network: Each input neuron would be activated proportionally to the ID of the token. This would mean that the model would assume that the tokens with similar IDs are somehow more related and that the distance between the token IDs is meaningful. This is, of course, not the case! \"House\" might have ID 627 and \"Bungalow\" might be ID 9384, even though they mean similar things and a classifier should be able to classify both words as buildings. Remember that during the training of the tokenizer, we didn't care much about the words semantics, only about the frequency!\n",
    "\n",
    "Ideally, we would like to represent our words in a way that similar meanings have similar values. This is where **embeddings** come in:\n",
    "Embeddings are vector representations of our tokens, so that tokens with similar meanings have similar vectors, i.e. closer together in the vector space (called **embedding space**)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3c01bc",
   "metadata": {},
   "source": [
    "## (Optional) Mount folder in Colab\n",
    "\n",
    "Uncomment the following cell to mount your gdrive if you are using the notebook in google colab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32e6fa9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the following lines if you want to use Google Colab\n",
    "# We presume you created a folder \"i2dl\" within your main drive folder, and put the exercise there.\n",
    "# NOTE: terminate all other colab sessions that use GPU!\n",
    "# NOTE 2: Make sure the correct exercise folder (e.g exercise_11) is given.\n",
    "\n",
    "\"\"\"\n",
    "from google.colab import drive\n",
    "import os\n",
    "\n",
    "gdrive_path='/content/gdrive/MyDrive/i2dl/exercise_11'\n",
    "\n",
    "# This will mount your google drive under 'MyDrive'\n",
    "drive.mount('/content/gdrive', force_remount=True)\n",
    "# In order to access the files in this notebook we have to navigate to the correct folder\n",
    "os.chdir(gdrive_path)\n",
    "# Check manually if all files are present\n",
    "print(sorted(os.listdir()))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "319a34da",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f568e01",
   "metadata": {},
   "outputs": [],
   "source": [
    "from exercise_code.util import notebook_util as util\n",
    "from exercise_code.network import *\n",
    "from exercise_code.tests import *\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d585523a",
   "metadata": {},
   "source": [
    "# Token Ids vs Embeddings\n",
    "\n",
    "Let's kick off where we stopped in the last notebook and have a look at the tokens with their raw IDs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0660fed",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels, embeddings = util.create_embeddings(1, token_id=True)\n",
    "util.plot_embeddings(labels, embeddings, integer=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "166e8774",
   "metadata": {},
   "source": [
    "So the first idea you might have is to reorder the IDs and to group similar words together, assuming we could somehow \"measure\" similarity between two words! Let's have a look at what this could look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6292766",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels, embeddings = util.create_embeddings(1)\n",
    "util.plot_embeddings(labels, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a392c301",
   "metadata": {},
   "source": [
    "Even with this relatively simple algorithm, we can already see certain groups form. On the right we have school related words, on the left we have animals. \n",
    "\n",
    "What we have actually just created are basically 1D Embeddings!\n",
    "\n",
    "However, with 1D Embeddings we can't really encode more complicated relationships, like for example a triangle relationship, where three words are all equally similar to each other. By adding dimensions to our embedding, we can see more complex connections between word pairs. In 2D this could look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ad1fc34",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels, embeddings = util.create_embeddings(2)\n",
    "util.plot_embeddings(labels, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09521599",
   "metadata": {},
   "source": [
    "In our transformer model, we will actually create an embedding with 512 dimensions! And we won't even have to define a similarity function between words, during training the model will decide on its own where to place words in the embedding space!\n",
    "\n",
    "At the end of the day, the embedding layer is a simple lookup table that maps each token ID to these embedding vectors.\n",
    "\n",
    "If you are more interested in Embeddings, go check out this awesome tool, that let's you play around with word embeddings: http://vectors.nlpl.eu/explore/embeddings/en/\n",
    "\n",
    "Also, Computerphile made a nice video on vector embeddings, you can find it here: https://www.youtube.com/watch?v=gQddtTdmG_8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4310bd92",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>WARNING: Task Implementation</h3>\n",
    "    <p>Throughout this notebook you will as always have to complete several tasks to complete the individual modules! Please be aware though, that some Modules may have several tasks that have to be completed, but please <strong>only</strong> concentrate on the <strong>current task</strong> and the <strong>corresponding hints</strong> (if there are any). In other words, if you are working on Task 1, and there is also Task 4 in the TODOs of that Module, you don't have to work on it at this moment! <br>\n",
    "    Also, if we mention any specific pytorch modules in the task description or hints, you <strong>are allowed to use them!</strong> With that said, let's work on your first task!\n",
    "    </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b214995c",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "    <h3>Task 1: Implement</h3>\n",
    "    <p>Implement the class <code>Embedding</code> in <code>exercise_code/network/embedding.py</code>.\n",
    "    </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0450c30e",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = test_task_1()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80c3908e",
   "metadata": {},
   "source": [
    "That was it on embeddings! We will add some additional functionality to this module down the line, but we first have to discuss the most important part of our future transformer: Attention! See you in notebook 3!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "i2dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
