{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "41b7ad3e4aeda15",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Tokenization: The Power of Byte Pair Encoding (BPE)\n",
    "\n",
    "Tokenization, a crucial step in language processing, involves breaking text into smaller units. Among various techniques, Byte Pair Encoding (BPE) stands out as a powerful method.\n",
    "\n",
    "## Why Use Tokenizers?\n",
    "\n",
    "Text is complex, but tokenizers simplify it by splitting it into smaller parts. They're crucial because they:\n",
    "\n",
    "1. **Prepare Text:** Take text like \"The cat jumps\" and turn it into tokens: [\"The\", \"cat\", \"jumps\"]. \n",
    "2. **Manage Words:** Handle words effectively, like breaking down \"unpredictability\" into smaller parts for easier understanding: [\"un\", \"p\", \"red\", \"ict\", \"ability\"].\n",
    "3. **Create Features:** Tokens become the features that machines use to understand text, like identifying common phrases or terms.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eae484d2b294ba1a",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## (Optional) Mount folder in Colab\n",
    "\n",
    "Uncomment the following cell to mount your gdrive if you are using the notebook in google colab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "886673dc0bd87ea9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:30:28.091273Z",
     "start_time": "2023-12-29T16:30:28.085072Z"
    }
   },
   "outputs": [],
   "source": [
    "# Use the following lines if you want to use Google Colab\n",
    "# We presume you created a folder \"i2dl\" within your main drive folder, and put the exercise there.\n",
    "# NOTE: terminate all other colab sessions that use GPU!\n",
    "# NOTE 2: Make sure the correct exercise folder (e.g exercise_11) is given.\n",
    "\n",
    "\"\"\"\n",
    "from google.colab import drive\n",
    "import os\n",
    "\n",
    "gdrive_path='/content/gdrive/MyDrive/i2dl/exercise_11'\n",
    "\n",
    "# This will mount your google drive under 'MyDrive'\n",
    "drive.mount('/content/gdrive', force_remount=True)\n",
    "# In order to access the files in this notebook we have to navigate to the correct folder\n",
    "os.chdir(gdrive_path)\n",
    "# Check manually if all files are present\n",
    "print(sorted(os.listdir()))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f297cabe2c755ffc",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Setup\n",
    "\n",
    "First, let's download the required datasets as well as the pretrained models!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1a4774d5934ac2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from exercise_code.util.download_util import download_pretrainedModels, download_datasets\n",
    "\n",
    "download_datasets(force_download=False)\n",
    "download_pretrainedModels(force_download=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "500ea43b",
   "metadata": {},
   "source": [
    "A couple of new packages to install ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3f17746",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install transformers\n",
    "%pip install tokenizer\n",
    "%pip install ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a53a0ffc17375470",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And now, we can import all of the required packages and get started on this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1d013bb2c83b8af",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:36:35.592056Z",
     "start_time": "2023-12-29T16:36:35.572542Z"
    }
   },
   "outputs": [],
   "source": [
    "from exercise_code.data.tokenizer import *\n",
    "from tokenizers import Tokenizer\n",
    "import os\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True'\n",
    "\n",
    "root_path = os.path.dirname(os.path.abspath(os.getcwd()))\n",
    "model_path = os.path.join(os.getcwd(), 'models')\n",
    "pretrained_model_path = os.path.join(model_path, 'pretrainedModels')\n",
    "dataset_path = os.path.join(root_path, 'datasets', 'transformerDatasets')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7b0b9537c6db95f",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Byte Pair Encoding (BPE)\n",
    "\n",
    "Byte Pair Encoding (BPE) initially served as a text compression algorithm and later found application in OpenAI's GPT model for tokenization. It remains a foundational technique employed across numerous Transformer models such as GPT, GPT-2, RoBERTa, BART, and DeBERTa. BPE intelligently breaks text into tokens by merging pairs of characters. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bf30fa844adeb99",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:41:47.756182Z",
     "start_time": "2023-12-29T16:41:47.721488Z"
    }
   },
   "outputs": [],
   "source": [
    "file_path = os.path.join(pretrained_model_path, 'pretrained_tokenizer')\n",
    "tokenizer = Tokenizer.from_file(file_path)\n",
    "\n",
    "sentence = \"Hi, Introduction to Deep Learning is class IN2346!\"\n",
    "encodings = tokenizer.encode(sentence)\n",
    "tokens = encodings.tokens\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2236276976dc3994",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Note: The character Ġ is used to mark the location of whitespaces.\n",
    "\n",
    "From here we can convert the individual tokens into a list of IDs, that we can feed into a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e04f6135a1baaae4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:41:49.886359Z",
     "start_time": "2023-12-29T16:41:49.864670Z"
    }
   },
   "outputs": [],
   "source": [
    "token_ids = encodings.ids\n",
    "print(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40a1fbe49eebae6a",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And we can also go back to the original sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90b2d6c334252b79",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:41:51.235500Z",
     "start_time": "2023-12-29T16:41:51.216864Z"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer.decode(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e38295cdc5f1e12",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Training Algorithm\n",
    "Let's create our own BPE Tokenizer from scratch! You can see the entire implementation in <code>exercise_code/data/BytePairTokenizer.py</code>! Note: While these algorithms are often called training algorithms, they usually do not perform training as we've seen it so far using some kind of loss function! It's really more of an algorithm that, step by step, creates the individual tokens! With that said, let's dive in!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2ceb2a9",
   "metadata": {},
   "source": [
    "BPE training starts by computing the unique set of words used in the corpus (this could be your entire dataset, or a long text file). It then builds the initial vocabulary by taking all the symbols used to write those words. As a very simple example, let’s say our corpus uses these five words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "858c25ca77bf9e13",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:41:52.577628Z",
     "start_time": "2023-12-29T16:41:52.557665Z"
    }
   },
   "outputs": [],
   "source": [
    "words = [\"hug\", \"pug\", \"pun\", \"bun\", \"hugs\"]\n",
    "\n",
    "base_vocabulary = create_alphabet_from_list(words)\n",
    "print(base_vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2541574d1ac48ae1",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "For real-world cases, that base vocabulary will contain all the ASCII characters, at the very least, and probably some Unicode characters as well. If a character that was not in the training corpus is passed on to the tokenizer, that character will be converted to the unknown token. That’s one reason why lots of NLP models are very bad at analyzing content with emojis, for instance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "795fdb6ecec99dec",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "The GPT-2 and RoBERTa tokenizers (which are pretty similar) have a clever way to deal with this: they don’t look at words as being written with Unicode characters, but with bytes. This way the base vocabulary has a small size (256), but every character you can think of will still be included and not end up being converted to the unknown token. This trick is called byte-level BPE."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "413a31bc5dc36507",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "After getting this base vocabulary, we add new tokens until the desired vocabulary size is reached by learning merges, which are rules to merge two elements of the existing vocabulary together into a new one. So, at the beginning these merges will create tokens with two characters, and then, as training progresses, longer subwords."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e09d43c687eca8",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "At any step during the tokenizer training, the BPE algorithm will search for the most frequent pair of existing tokens (by “pair,” we mean two consecutive tokens in a word). That most frequent pair is the one that will be merged, and we rinse and repeat for the next step.\n",
    "\n",
    "Going back to our previous example, let’s assume the words in our corpus had the following frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19aaf3336f990903",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:48.987532Z",
     "start_time": "2023-12-29T16:44:48.958823Z"
    }
   },
   "outputs": [],
   "source": [
    "word_freq = {\"hug\": 10, \"pug\": 5, \"pun\": 12, \"bun\": 4, \"hugs\": 5}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3f97e18dca5210a",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Meaning \"hug\" was present 10 times in the corpus, \"pug\" 5 times, \"pun\" 12 times, \"bun\" 4 times, and \"hugs\" 5 times. We start the training by splitting each word into characters (the ones that form our initial vocabulary) so we can see each word as a list of tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bdc5541e3cdcb7f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:50.391907Z",
     "start_time": "2023-12-29T16:44:49.708219Z"
    }
   },
   "outputs": [],
   "source": [
    "splits = create_splits(word_freq.keys())\n",
    "print('Words split into characters: {}'.format(splits))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c774bbf597bb1258",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Then we look at pairs. The pair (\"h\", \"u\") is present in the words \"hug\" and \"hugs\", so 15 times total in the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76ed2cc5cda64912",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:50.451896Z",
     "start_time": "2023-12-29T16:44:50.438459Z"
    }
   },
   "outputs": [],
   "source": [
    "pair_freq = compute_pair_freq(word_freq, splits)\n",
    "\n",
    "print('Pair frequencies in corpus: {}'.format(dict(pair_freq)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7bb96da",
   "metadata": {},
   "source": [
    "It’s not the most frequent pair, though: that honor belongs to (\"u\", \"g\"), which is present in \"hug\", \"pug\", and \"hugs\", for a grand total of 20 times in the vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ab863a0f6ef7778",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:50.470576Z",
     "start_time": "2023-12-29T16:44:50.451384Z"
    }
   },
   "outputs": [],
   "source": [
    "best_pair = compute_best_pair(pair_freq)\n",
    "\n",
    "print(\"The best pair is: {}\".format(best_pair))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf953e920111c71",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Thus, the first merge rule learned by the tokenizer is (\"u\", \"g\") -> \"ug\", which means that \"ug\" will be added to the vocabulary, and the pair should be merged in all the words of the corpus. At the end of this stage, the vocabulary and corpus look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4713c86aa566118e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:53.353911Z",
     "start_time": "2023-12-29T16:44:52.649376Z"
    }
   },
   "outputs": [],
   "source": [
    "merges = {}\n",
    "splits = merge_pair(word_freq, *best_pair, splits)\n",
    "merges[best_pair] = best_pair[0] + best_pair[1]\n",
    "base_vocabulary.append(best_pair[0] + best_pair[1])\n",
    "\n",
    "print('The new splits are: {}'.format(splits))\n",
    "print('New vocabulary: {}'.format(base_vocabulary))\n",
    "print('Dictionary with all merges: {}'.format(merges))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77ed8beb2d2b981b",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now we have some pairs that result in a token longer than two characters: the pair (\"h\", \"ug\"), for instance (present 15 times in the corpus). However, the most frequent pair at this stage is (\"u\", \"n\"), present 16 times in the corpus, so the second merge rule learned is (\"u\", \"n\") -> \"un\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "733d3545c031a5a4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:53.416007Z",
     "start_time": "2023-12-29T16:44:53.396986Z"
    }
   },
   "outputs": [],
   "source": [
    "pair_freq = compute_pair_freq(word_freq, splits)\n",
    "best_pair = compute_best_pair(pair_freq)\n",
    "\n",
    "print('Pair frequencies in corpus: {}'.format(dict(pair_freq)))\n",
    "print(\"The best pair is: {}\".format(best_pair))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ecb25d0",
   "metadata": {},
   "source": [
    " Adding that to the vocabulary and merging all existing occurrences leads us to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec0e38ae28fbaeef",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:55.559865Z",
     "start_time": "2023-12-29T16:44:54.180127Z"
    }
   },
   "outputs": [],
   "source": [
    "splits = merge_pair(word_freq, *best_pair, splits)\n",
    "merges[best_pair] = best_pair[0] + best_pair[1]\n",
    "base_vocabulary.append(best_pair[0] + best_pair[1])\n",
    "\n",
    "print('The new splits are: {}'.format(splits))\n",
    "print('New vocabulary: {}'.format(base_vocabulary))\n",
    "print('Dictionary with all merges: {}'.format(merges))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cf275def6324764",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Again, let's compute the most frequent pair:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3be3b9a2c609b46",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:56.302433Z",
     "start_time": "2023-12-29T16:44:56.283983Z"
    }
   },
   "outputs": [],
   "source": [
    "pair_freq = compute_pair_freq(word_freq, splits)\n",
    "best_pair = compute_best_pair(pair_freq)\n",
    "\n",
    "print('Pair frequencies in corpus: {}'.format(dict(pair_freq)))\n",
    "print(\"The best pair is: {}\".format(best_pair))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baeb742c",
   "metadata": {},
   "source": [
    "Now the most frequent pair is (\"h\", \"ug\"), so we learn the merge rule (\"h\", \"ug\") -> \"hug\", which gives us our first three-letter token. After the merge, the corpus looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9a1d3310fe392c0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:57.003413Z",
     "start_time": "2023-12-29T16:44:56.314324Z"
    }
   },
   "outputs": [],
   "source": [
    "splits = merge_pair(word_freq, *best_pair, splits)\n",
    "merges[best_pair] = best_pair[0] + best_pair[1]\n",
    "base_vocabulary.append(best_pair[0] + best_pair[1])\n",
    "\n",
    "print('The new splits are: {}'.format(splits))\n",
    "print('New vocabulary: {}'.format(base_vocabulary))\n",
    "print('Dictionary with all merges: {}'.format(merges))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3881aa41aeebe790",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And we continue like this until we reach the desired vocabulary size.\n",
    "\n",
    "Feel free to have a look at the Tokenizer Implementation in BytePairTokenizer! Note that we will be using a different implementation from Huggingfacce though for the following notebooks of this exercise. It is implemented in Rust and is a lot faster than this Python code, however, the algorithm remains the same! In fact, let's train one right now!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45da8e98005931cf",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "First we have to initialize our model as a Byte Pair Encoder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69e975a6eeaa45a9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:44:59.682509Z",
     "start_time": "2023-12-29T16:44:58.377068Z"
    }
   },
   "outputs": [],
   "source": [
    "from tokenizers import Tokenizer\n",
    "from tokenizers.models import BPE\n",
    "tokenizer = Tokenizer(BPE())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2fccb7b7a7d0dfc",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Next we have to initialize our Trainer. We will have one special character - '<[EOS]>' - which will mark the beginning and the end of a sentence and will also be used for padding, more on that later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b17484b70160e82",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:45:01.127636Z",
     "start_time": "2023-12-29T16:45:00.413561Z"
    }
   },
   "outputs": [],
   "source": [
    "from tokenizers import trainers\n",
    "vocab_size = 300\n",
    "eos_token = '<[EOS]>'\n",
    "trainer = trainers.BpeTrainer(vocab_size=vocab_size, special_tokens=[eos_token])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b72f8489a26f21b",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Next, we have to define our Pretokenizer, which splits the sentences into individual words. We will in fact be using a sequence of predefined models:\n",
    "\n",
    "1. **ByteLevel**: Replaces all whitespaces with a special character Ġ and splits the sentences.\n",
    "2. **Digits**: Splits all sequences of digits into individual digits. That way we don't waste any words on often occurring numbers.\n",
    "3. **Punctuation**: Splits sentences at punctuations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b38c1c26d4e0e7db",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:47:17.513983Z",
     "start_time": "2023-12-29T16:47:17.474741Z"
    }
   },
   "outputs": [],
   "source": [
    "from tokenizers.pre_tokenizers import ByteLevel, Digits, Sequence, Punctuation\n",
    "pre_tokenizer = Sequence([ByteLevel(add_prefix_space=False), Digits(individual_digits=True), Punctuation()]) \n",
    "output = pre_tokenizer.pre_tokenize_str(sentence)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0e6c252737e61c5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:47:19.409797Z",
     "start_time": "2023-12-29T16:47:18.575375Z"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer.pre_tokenizer = pre_tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c91fc24c2608a0f",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And finally we can declare the list of files to train on and start the actual training process. Depending on the size of vocabulary and your hardware, this might take a couple minutes.\n",
    "\n",
    "Note: If this doesn't work or takes way too long (>10 min), don't worry about it and stop the process! Just read through the following cells and try to understand what is happening! We have a pretrained version of this tokenizer that you can use in the following exercises!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2ec5aab2bd9a60",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:49:19.747606Z",
     "start_time": "2023-12-29T16:48:55.576553Z"
    }
   },
   "outputs": [],
   "source": [
    "files = [os.path.join(dataset_path, 'europarlOpusDatasets', 'corpus_english.txt'),\n",
    "         os.path.join(dataset_path, 'europarlOpusDatasets', 'corpus_german.txt')]\n",
    "tokenizer.train(files, trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c8d2ad83f98ec79",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Alright, training is done and last thing we have to do is define the template of our output. We want each token sequence to start and end with an end of sequence token. We will discuss why later in the actual transformer notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd18468b16b9bc45",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:49:24.021845Z",
     "start_time": "2023-12-29T16:49:23.999072Z"
    }
   },
   "outputs": [],
   "source": [
    "from tokenizers.processors import TemplateProcessing\n",
    "\n",
    "tokenizer.post_processor = TemplateProcessing(\n",
    "    single=eos_token + \" $0 \" + eos_token,\n",
    "    pair=None,\n",
    "    special_tokens=[\n",
    "        (eos_token, tokenizer.token_to_id(eos_token))\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2add4d53631e21e6",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now let's test it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d92b3ed157b0aac8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:49:26.160992Z",
     "start_time": "2023-12-29T16:49:26.140652Z"
    }
   },
   "outputs": [],
   "source": [
    "output = tokenizer.encode(sentence)\n",
    "print(output.tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e511ba2e408da253",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:49:28.050134Z",
     "start_time": "2023-12-29T16:49:27.199854Z"
    }
   },
   "outputs": [],
   "source": [
    "print(output.ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7475b66897bdf38",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Let's try to decode it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e622dba9f6ff42d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:49:29.799799Z",
     "start_time": "2023-12-29T16:49:29.782683Z"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer.decode(output.ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "343fa861bc331b90",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Ups, something doesn't look right... That's because we still have to configure the decoder! Otherwise it does't know what to do with the Ġ character!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53c7d05dcaf5e724",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:49:31.567469Z",
     "start_time": "2023-12-29T16:49:30.707624Z"
    }
   },
   "outputs": [],
   "source": [
    "from tokenizers.decoders import ByteLevel\n",
    "tokenizer.decoder = ByteLevel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480925e9d1e399c5",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Let's try that again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4665ffe5d60e7155",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-28T11:18:29.210128Z",
     "start_time": "2023-12-28T11:18:29.173871Z"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer.decode(output.ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ea86cb22be4645",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Perfect, everything seems to work! Let's save this model and reuse it later in the transformer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "184cddcce8a34b8a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:52:35.673155Z",
     "start_time": "2023-12-29T16:52:35.634465Z"
    }
   },
   "outputs": [],
   "source": [
    "file_path = os.path.join(model_path, \"custom_tokenizer\")\n",
    "tokenizer.save(file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e39f94ef2731a483",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "So far, the tokenizer is still implemented in python. If we want to use the faster Rust implementation we have to load it as a Fast Tokenizer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cbcd802af95dac3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:52:38.259604Z",
     "start_time": "2023-12-29T16:52:38.219935Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import PreTrainedTokenizerFast\n",
    "\n",
    "tokenizer_fast = PreTrainedTokenizerFast(\n",
    "    tokenizer_file=file_path,\n",
    "    # tokenizer_object=tokenizer, # This also works!\n",
    "    eos_token=eos_token,\n",
    "    pad_token=eos_token\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0941c72fa08350",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:52:39.984430Z",
     "start_time": "2023-12-29T16:52:39.950261Z"
    }
   },
   "outputs": [],
   "source": [
    "output = tokenizer_fast.encode(sentence)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3aed9eaab40a055",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-29T16:52:41.850311Z",
     "start_time": "2023-12-29T16:52:41.012880Z"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer_fast.decode(output, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cebfa92c",
   "metadata": {},
   "source": [
    "Perfect, thats all you have to know about tokenizers for now! Next stop - feeding tokens into our model using embeddings! See you in notebook 2 :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "i2dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
