tutorials/1_farm_building_blocks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "colab": {
      "name": "1_farm_building_blocks.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4U_l_eGYrWVE",
        "colab_type": "text"
      },
      "source": [
        "# FARM Building Blocks\n",
        "\n",
        "Welcome to the FARM building blocks tutorial! There are many different ways to make use of this repository, but in this notebook, we will be going through the most import building blocks that will help you harvest the rewards of a successfully trained NLP model.\n",
        "\n",
        "Happy FARMing!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nKnv3c03rWVH",
        "colab_type": "text"
      },
      "source": [
        "## 1) Text Classification\n",
        "\n",
        "GermEval 2018 (GermEval2018) (https://projects.fzai.h-da.de/iggsa/) is an open data set containing texts that need to be classified by whether they are offensive or not. There are a set of coarse and fine labels, but here we will only be looking at the coarse set which labels each example as either OFFENSE or OTHER. To tackle this task, we are going to build a classifier that is composed of Google's BERT language model and a feed forward neural network prediction head."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hULXmgZSrWVJ",
        "colab_type": "text"
      },
      "source": [
        "### Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ShqSNsNTrWVL",
        "colab_type": "code",
        "outputId": "3866129c-f0de-4f62-8f59-049bd74ec79f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# Let's start by adjust the working directory so that it is the root of the repository\n",
        "# This should be run just once.\n",
        "\n",
        "import os\n",
        "os.chdir('../')\n",
        "print(\"Current working directory is {}\".format(os.getcwd()))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Current working directory is /content/FARM\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kPHWNPNBrWVT",
        "colab_type": "code",
        "outputId": "9ef6693b-a47e-4b37-c801-30419fb9359d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "# Here are the imports we need\n",
        "import torch\n",
        "from farm.modeling.tokenization import Tokenizer\n",
        "from farm.data_handler.processor import TextClassificationProcessor\n",
        "from farm.data_handler.data_silo import DataSilo\n",
        "from farm.modeling.language_model import LanguageModel\n",
        "from farm.modeling.prediction_head import TextClassificationHead\n",
        "from farm.modeling.adaptive_model import AdaptiveModel\n",
        "from farm.modeling.optimization import initialize_optimizer\n",
        "from farm.train import Trainer\n",
        "from farm.utils import MLFlowLogger"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:19:03 - INFO - transformers.file_utils -   PyTorch version 1.4.0 available.\n",
            "05/05/2020 08:19:04 - INFO - transformers.file_utils -   TensorFlow version 2.2.0-rc3 available.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RYpdtDGLrWVb",
        "colab_type": "code",
        "outputId": "20a15b95-9f74-49ad-8e6c-aee3c3e8a3e7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "source": [
        "# Farm allows simple logging of many parameters & metrics. Let's use MLflow framework to track our experiment ...\n",
        "ml_logger = MLFlowLogger(tracking_uri=\"https://public-mlflow.deepset.ai/\")\n",
        "ml_logger.init_experiment(experiment_name=\"Public_FARM\", run_name=\"Tutorial1_Colab\")"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            " __          __  _                            _        \n",
            " \\ \\        / / | |                          | |       \n",
            "  \\ \\  /\\  / /__| | ___ ___  _ __ ___   ___  | |_ ___  \n",
            "   \\ \\/  \\/ / _ \\ |/ __/ _ \\| '_ ` _ \\ / _ \\ | __/ _ \\ \n",
            "    \\  /\\  /  __/ | (_| (_) | | | | | |  __/ | || (_) |\n",
            "     \\/  \\/ \\___|_|\\___\\___/|_| |_| |_|\\___|  \\__\\___/ \n",
            "  ______      _____  __  __  \n",
            " |  ____/\\   |  __ \\|  \\/  |              _.-^-._    .--.\n",
            " | |__ /  \\  | |__) | \\  / |           .-'   _   '-. |__|\n",
            " |  __/ /\\ \\ |  _  /| |\\/| |          /     |_|     \\|  |\n",
            " | | / ____ \\| | \\ \\| |  | |         /               \\  |\n",
            " |_|/_/    \\_\\_|  \\_\\_|  |_|        /|     _____     |\\ |\n",
            "                                     |    |==|==|    |  |\n",
            "|---||---|---|---|---|---|---|---|---|    |--|--|    |  |\n",
            "|---||---|---|---|---|---|---|---|---|    |==|==|    |  |\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            " \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-JxI84OErWVi",
        "colab_type": "code",
        "outputId": "78254dc0-d408-4e51-b06c-e5542b03a309",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# We need to fetch the right device to drive the growth of our model\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(\"Devices available: {}\".format(device))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Devices available: cuda\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyZMrssgrWVn",
        "colab_type": "text"
      },
      "source": [
        "### Data Handling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TUU-Gq-XrWVo",
        "colab_type": "code",
        "outputId": "74a4cabe-1fcf-4869-8583-d6cffaa80785",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "# Here we initialize a tokenizer that will be used for preprocessing text\n",
        "# This is the BERT Tokenizer which uses the byte pair encoding method.\n",
        "# It is currently loaded with a German model\n",
        "\n",
        "tokenizer = Tokenizer.load(\n",
        "    pretrained_model_name_or_path=\"bert-base-german-cased\",\n",
        "    do_lower_case=False)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:19:14 - INFO - farm.modeling.tokenization -   Loading tokenizer of type 'BertTokenizer'\n",
            "05/05/2020 08:19:15 - INFO - transformers.tokenization_utils -   loading file https://int-deepset-models-bert.s3.eu-central-1.amazonaws.com/pytorch/bert-base-german-cased-vocab.txt from cache at /root/.cache/torch/transformers/da299cdd121a3d71e1626f2908dda0d02658f42e925a3d6abd8273ec08cf41a6.31ccc255fc2bad3578089a3997f16b286498ba78c0adc43b5bb2a3f9a0d2c85c\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pAJ2QrnPrWVt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# In order to prepare the data for the model, we need a set of\n",
        "# functions to transform data files into PyTorch Datasets.\n",
        "# We group these together in Processor objects.\n",
        "# We will need a new Processor object for each new source of data.\n",
        "# The abstract class can be found in farm.data_handling.processor.Processor\n",
        "\n",
        "LABEL_LIST = [\"OTHER\", \"OFFENSE\"]\n",
        "processor = TextClassificationProcessor(tokenizer=tokenizer,\n",
        "                                        max_seq_len=128,\n",
        "                                        data_dir=\"data/germeval18\",\n",
        "                                        label_list=LABEL_LIST,\n",
        "                                        metric=\"f1_macro\",\n",
        "                                        label_column_name=\"coarse_label\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WR5jFJEErWV0",
        "colab_type": "code",
        "outputId": "f0a8b748-8097-4617-8fdc-156b3d420527",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# We need a DataSilo in order to keep our train, dev and test sets separate.\n",
        "# The DataSilo will call the functions in the Processor to generate these sets.\n",
        "# From the DataSilo, we can fetch a PyTorch DataLoader object which will\n",
        "# be passed on to the model.\n",
        "# Here is a good place to define a batch size for the model\n",
        "\n",
        "BATCH_SIZE = 32\n",
        "\n",
        "data_silo = DataSilo(\n",
        "    processor=processor,\n",
        "    batch_size=BATCH_SIZE)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:19:21 - INFO - farm.data_handler.data_silo -   \n",
            "Loading data into the data silo ... \n",
            "              ______\n",
            "               |o  |   !\n",
            "   __          |:`_|---'-.\n",
            "  |__|______.-/ _ \\-----.|       \n",
            " (o)(o)------'\\ _ /     ( )      \n",
            " \n",
            "05/05/2020 08:19:21 - INFO - farm.data_handler.data_silo -   Loading train set from: data/germeval18/train.tsv \n",
            "05/05/2020 08:19:21 - INFO - farm.data_handler.data_silo -   Got ya 1 parallel workers to convert 5009 dictionaries to pytorch datasets (chunksize = 1002)...\n",
            "05/05/2020 08:19:21 - INFO - farm.data_handler.data_silo -    0 \n",
            "05/05/2020 08:19:21 - INFO - farm.data_handler.data_silo -   /|\\\n",
            "05/05/2020 08:19:21 - INFO - farm.data_handler.data_silo -   /'\\\n",
            "05/05/2020 08:19:21 - INFO - farm.data_handler.data_silo -   \n",
            "Preprocessing Dataset data/germeval18/train.tsv:   0%|          | 0/5009 [00:00<?, ? Dicts/s]05/05/2020 08:19:22 - INFO - farm.data_handler.processor -   *** Show 2 random examples ***\n",
            "05/05/2020 08:19:22 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-873-0\n",
            "Clear Text: \n",
            " \ttext: @AfD_OF_Land Der deutsche Bürger ist mittlerweile ein Mensch zweiter Klasse.\n",
            " \ttext_classification_label: OTHER\n",
            "Tokenized: \n",
            " \ttokens: ['@', 'AfD', '_', 'O', '##F', '_', 'Land', 'Der', 'deutsche', 'Bürger', 'ist', 'mittlerweile', 'ein', 'Mensch', 'zweiter', 'Klasse', '.']\n",
            " \toffsets: [0, 1, 4, 5, 6, 7, 8, 13, 17, 26, 33, 37, 50, 54, 61, 69, 75]\n",
            " \tstart_of_word: [True, False, False, False, False, False, False, True, True, True, True, True, True, True, True, True, False]\n",
            "Features: \n",
            " \tinput_ids: [3, 26991, 10732, 26983, 169, 26934, 26983, 414, 233, 2006, 1507, 127, 5330, 39, 5099, 8266, 5546, 26914, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \ttext_classification_label_ids: [0]\n",
            "_____________________________________________________\n",
            "05/05/2020 08:19:22 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-780-0\n",
            "Clear Text: \n",
            " \ttext: ich bin so gerne primitiv heute!\n",
            " \ttext_classification_label: OTHER\n",
            "Tokenized: \n",
            " \ttokens: ['ich', 'bin', 'so', 'gerne', 'prim', '##itiv', 'heute', '!']\n",
            " \toffsets: [0, 4, 8, 11, 17, 21, 26, 31]\n",
            " \tstart_of_word: [True, True, True, True, True, False, True, False]\n",
            "Features: \n",
            " \tinput_ids: [3, 1169, 4058, 181, 6812, 14177, 18064, 1138, 26982, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \ttext_classification_label_ids: [0]\n",
            "_____________________________________________________\n",
            "Preprocessing Dataset data/germeval18/train.tsv: 100%|██████████| 5009/5009 [00:05<00:00, 935.67 Dicts/s]\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -   Loading dev set as a slice of train set\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -   Took 1001 samples out of train set to create dev set (dev split is roughly 0.1)\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -   Loading test set from: data/germeval18/test.tsv\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -   Got ya 1 parallel workers to convert 3532 dictionaries to pytorch datasets (chunksize = 707)...\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -    0 \n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -   /w\\\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -   /'\\\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.data_silo -   \n",
            "Preprocessing Dataset data/germeval18/test.tsv:   0%|          | 0/3532 [00:00<?, ? Dicts/s]05/05/2020 08:19:27 - INFO - farm.data_handler.processor -   *** Show 2 random examples ***\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-610-0\n",
            "Clear Text: \n",
            " \ttext: @Christoph_Hi In Wahrheit leben die Jesiden auch gar nicht so religiös.\n",
            " \ttext_classification_label: OTHER\n",
            "Tokenized: \n",
            " \ttokens: ['@', 'Christoph', '_', 'Hi', 'In', 'Wahrheit', 'leben', 'die', 'Jes', '##iden', 'auch', 'gar', 'nicht', 'so', 'religiös', '.']\n",
            " \toffsets: [0, 1, 10, 11, 14, 17, 26, 32, 36, 39, 44, 49, 53, 59, 62, 70]\n",
            " \tstart_of_word: [True, False, False, False, True, True, True, True, True, False, True, True, True, True, True, False]\n",
            "Features: \n",
            " \tinput_ids: [3, 26991, 7281, 26983, 26524, 173, 9848, 2564, 30, 6932, 882, 194, 2523, 149, 181, 26231, 26914, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \ttext_classification_label_ids: [0]\n",
            "_____________________________________________________\n",
            "05/05/2020 08:19:27 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-427-0\n",
            "Clear Text: \n",
            " \ttext: Gewalt d spanischen Polizei inakzeptabel! Bürgerkrieg in #Katalonien muss verhindert werden. Entsendung #OSZE-Beobachter nötig. #Referendum\n",
            " \ttext_classification_label: OTHER\n",
            "Tokenized: \n",
            " \ttokens: ['Gewalt', 'd', 'spanischen', 'Polizei', 'in', '##ak', '##zept', '##abel', '!', 'Bürgerkrieg', 'in', '#', 'Katal', '##onien', 'muss', 'verhindert', 'werden', '.', 'Ents', '##endung', '#', 'OS', '##Z', '##E', '-', 'Beobachter', 'nötig', '.', '#', 'Referendum']\n",
            " \toffsets: [0, 7, 9, 20, 28, 30, 32, 36, 40, 42, 54, 57, 58, 63, 69, 74, 85, 91, 93, 97, 104, 105, 107, 108, 109, 110, 121, 126, 128, 129]\n",
            " \tstart_of_word: [True, True, True, True, True, False, False, False, False, True, True, True, False, False, True, True, True, False, True, False, True, False, False, False, False, False, True, False, True, False]\n",
            "Features: \n",
            " \tinput_ids: [3, 4806, 9, 8467, 1908, 50, 464, 1871, 3549, 26982, 16716, 50, 26990, 17128, 10649, 1080, 10807, 266, 26914, 17623, 5776, 26990, 21628, 26950, 26931, 26935, 17012, 9475, 26914, 26990, 23327, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \ttext_classification_label_ids: [0]\n",
            "_____________________________________________________\n",
            "Preprocessing Dataset data/germeval18/test.tsv: 100%|██████████| 3532/3532 [00:03<00:00, 1050.88 Dicts/s]\n",
            "05/05/2020 08:19:30 - INFO - farm.data_handler.data_silo -   Examples in train: 4008\n",
            "05/05/2020 08:19:30 - INFO - farm.data_handler.data_silo -   Examples in dev  : 1001\n",
            "05/05/2020 08:19:30 - INFO - farm.data_handler.data_silo -   Examples in test : 3532\n",
            "05/05/2020 08:19:30 - INFO - farm.data_handler.data_silo -   \n",
            "05/05/2020 08:19:30 - INFO - farm.data_handler.data_silo -   Longest sequence length observed after clipping:     128\n",
            "05/05/2020 08:19:30 - INFO - farm.data_handler.data_silo -   Average sequence length after clipping: 40.47030938123753\n",
            "05/05/2020 08:19:30 - INFO - farm.data_handler.data_silo -   Proportion clipped:      0.0187125748502994\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HxLmbuzarWV6",
        "colab_type": "text"
      },
      "source": [
        "### Modeling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pg3GfYkbrWV6",
        "colab_type": "text"
      },
      "source": [
        "In FARM, we make a strong distinction between the language model and prediction head so that you can mix and match different building blocks for your needs.\n",
        "\n",
        "For example, in the transfer learning paradigm, you might have the one language model that you will be using for both document classification and NER. Or you perhaps you have a pretrained language model which you would like to adapt to your domain, then use for a downstream task such as question answering. \n",
        "\n",
        "All this is possible within FARM and requires only the replacement of a few modular components, as we shall see below.\n",
        "\n",
        "Let's first have a look at how we might set up a model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bonmw3uErWV7",
        "colab_type": "code",
        "outputId": "84fa93e5-cd2f-4e8e-f18d-ec44e3bf122a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "# The language model is the foundation on which modern NLP systems are built.\n",
        "# They encapsulate a general understanding of sentence semantics\n",
        "# and are not specific to any one task.\n",
        "\n",
        "# Here we are using Google's BERT model as implemented by HuggingFace. \n",
        "# The model being loaded is a German model that we trained. \n",
        "# You can also change the MODEL_NAME_OR_PATH to point to a BERT model that you\n",
        "# have saved or download one connected to the HuggingFace repository.\n",
        "# See https://huggingface.co/models for a list of available models\n",
        "\n",
        "MODEL_NAME_OR_PATH = \"bert-base-german-cased\"\n",
        "\n",
        "language_model = LanguageModel.load(MODEL_NAME_OR_PATH)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:19:39 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-cased-pytorch_model.bin from cache at /root/.cache/torch/transformers/e32f648561b03f77a129832928b7f16decdc5e0870f1e6558857e046169d4133.4e5eda3a0f09b32a0b7d1a9185034da1b3506d5c5b0c6880a7ca0122ab5eef2e\n",
            "05/05/2020 08:19:41 - INFO - farm.modeling.language_model -   Automatically detected language from language model name: german\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kWqLW5x2rWV_",
        "colab_type": "code",
        "outputId": "9c7398a7-9ee7-4805-ab95-3f3c8d6b832c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# A prediction head is a model that processes the output of the language model\n",
        "# for a specific task.\n",
        "# Prediction heads will look different depending on whether you're doing text classification\n",
        "# Named Entity Recognition (NER), question answering or some other task.\n",
        "# They should generate logits over the available prediction classes and contain methods\n",
        "# to convert these logits to losses or predictions \n",
        "\n",
        "# Here we use TextClassificationHead which receives a single fixed length sentence vector\n",
        "# and processes it using a feed forward neural network. layer_dims is a list of dimensions:\n",
        "# [input_dims, hidden_1_dims, hidden_2_dims ..., output_dims]\n",
        "\n",
        "# Here by default we have a single layer network.\n",
        "# It takes in a vector of length 768 (the default size of BERT's output).\n",
        "# It outputs a vector of length 2 (the number of classes in the GermEval18 (coarse) dataset)\n",
        "\n",
        "prediction_head = TextClassificationHead(num_labels=len(LABEL_LIST))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:19:45 - INFO - farm.modeling.prediction_head -   Prediction head initialized with size [768, 2]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fUeLxU8yrWWD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# The language model and prediction head are coupled together in the Adaptive Model.\n",
        "# This class takes care of model saving and loading and also coordinates\n",
        "# cases where there is more than one prediction head.\n",
        "\n",
        "# EMBEDS_DROPOUT_PROB is the probability that an element of the output vector from the\n",
        "# language model will be set to zero.\n",
        "EMBEDS_DROPOUT_PROB = 0.1\n",
        "\n",
        "model = AdaptiveModel(\n",
        "    language_model=language_model,\n",
        "    prediction_heads=[prediction_head],\n",
        "    embeds_dropout_prob=EMBEDS_DROPOUT_PROB,\n",
        "    lm_output_types=[\"per_sequence\"],\n",
        "    device=device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ITADnSE5rWWF",
        "colab_type": "text"
      },
      "source": [
        "### Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YrYk9jSkrWWG",
        "colab_type": "code",
        "outputId": "2c59dd46-a979-4f4a-a886-1956113c1195",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "# Here we initialize a Bert Adam optimizer that has a linear warmup and warmdown\n",
        "# Here you can set learning rate, the warmup proportion and number of epochs to train for\n",
        "\n",
        "LEARNING_RATE = 2e-5\n",
        "N_EPOCHS = 1\n",
        "\n",
        "model, optimizer, lr_schedule = initialize_optimizer(\n",
        "    model=model,\n",
        "    device=device,\n",
        "    learning_rate=LEARNING_RATE,\n",
        "    n_batches=len(data_silo.loaders[\"train\"]),\n",
        "    n_epochs=N_EPOCHS)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:20:00 - INFO - farm.modeling.optimization -   Loading optimizer `TransformersAdamW`: '{'correct_bias': False, 'weight_decay': 0.01, 'lr': 2e-05}'\n",
            "05/05/2020 08:20:01 - INFO - farm.modeling.optimization -   Using scheduler 'get_linear_schedule_with_warmup'\n",
            "05/05/2020 08:20:01 - INFO - farm.modeling.optimization -   Loading schedule `get_linear_schedule_with_warmup`: '{'num_warmup_steps': 12.600000000000001, 'num_training_steps': 126}'\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HO7h-g1jrWWL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Training loop handled by this\n",
        "# It will also trigger evaluation during training using the dev data\n",
        "# and after training using the test data.\n",
        "\n",
        "# Set N_GPU to a positive value if CUDA is available, to 0 if not\n",
        "N_GPU = 1\n",
        "\n",
        "trainer = Trainer(\n",
        "    model=model,\n",
        "    optimizer=optimizer,\n",
        "    data_silo=data_silo,\n",
        "    epochs=N_EPOCHS,\n",
        "    n_gpu=N_GPU,\n",
        "    lr_schedule=lr_schedule,\n",
        "    device=device,\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3-mYPF4frWWQ",
        "colab_type": "code",
        "outputId": "b3afa6b6-cf5b-4f64-8b91-9eadce5f13aa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "model = trainer.train()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:20:09 - INFO - farm.train -   \n",
            " \n",
            "\n",
            "          &&& &&  & &&             _____                   _             \n",
            "      && &\\/&\\|& ()|/ @, &&       / ____|                 (_)            \n",
            "      &\\/(/&/&||/& /_/)_&/_&     | |  __ _ __ _____      ___ _ __   __ _ \n",
            "   &() &\\/&|()|/&\\/ '%\" & ()     | | |_ | '__/ _ \\ \\ /\\ / / | '_ \\ / _` |\n",
            "  &_\\_&&_\\ |& |&&/&__%_/_& &&    | |__| | | | (_) \\ V  V /| | | | | (_| |\n",
            "&&   && & &| &| /& & % ()& /&&    \\_____|_|  \\___/ \\_/\\_/ |_|_| |_|\\__, |\n",
            " ()&_---()&\\&\\|&&-&&--%---()~                                       __/ |\n",
            "     &&     \\|||                                                   |___/\n",
            "             |||\n",
            "             |||\n",
            "             |||\n",
            "       , -=-~  .-^- _\n",
            "              `\n",
            "\n",
            "Train epoch 0/0 (Cur. train loss: 0.3120):  79%|███████▉  | 100/126 [01:18<00:19,  1.34it/s]\n",
            "Evaluating: 100%|██████████| 32/32 [00:07<00:00,  4.27it/s]\n",
            "05/05/2020 08:21:36 - INFO - farm.eval -   \n",
            "\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "***************************************************\n",
            "***** EVALUATION | DEV SET | AFTER 100 BATCHES *****\n",
            "***************************************************\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\n",
            "05/05/2020 08:21:36 - INFO - farm.eval -   \n",
            " _________ text_classification _________\n",
            "05/05/2020 08:21:37 - INFO - farm.eval -   loss: 0.43276190495753025\n",
            "05/05/2020 08:21:37 - INFO - farm.eval -   task_name: text_classification\n",
            "05/05/2020 08:21:37 - INFO - farm.eval -   f1_macro: 0.7674164243980119\n",
            "05/05/2020 08:21:37 - INFO - farm.eval -   report: \n",
            "               precision    recall  f1-score   support\n",
            "\n",
            "       OTHER     0.8348    0.8612    0.8478       663\n",
            "     OFFENSE     0.7098    0.6657    0.6870       338\n",
            "\n",
            "    accuracy                         0.7952      1001\n",
            "   macro avg     0.7723    0.7635    0.7674      1001\n",
            "weighted avg     0.7926    0.7952    0.7935      1001\n",
            "\n",
            "Train epoch 0/0 (Cur. train loss: 0.4050): 100%|██████████| 126/126 [01:47<00:00,  1.17it/s]\n",
            "Evaluating: 100%|██████████| 111/111 [00:26<00:00,  4.14it/s]\n",
            "05/05/2020 08:22:24 - INFO - farm.eval -   \n",
            "\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "***************************************************\n",
            "***** EVALUATION | TEST SET | AFTER 126 BATCHES *****\n",
            "***************************************************\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\n",
            "05/05/2020 08:22:24 - INFO - farm.eval -   \n",
            " _________ text_classification _________\n",
            "05/05/2020 08:22:24 - INFO - farm.eval -   loss: 0.46406682809373934\n",
            "05/05/2020 08:22:24 - INFO - farm.eval -   task_name: text_classification\n",
            "05/05/2020 08:22:25 - INFO - farm.eval -   f1_macro: 0.7533396714263575\n",
            "05/05/2020 08:22:25 - INFO - farm.eval -   report: \n",
            "               precision    recall  f1-score   support\n",
            "\n",
            "       OTHER     0.8150    0.8755    0.8442      2330\n",
            "     OFFENSE     0.7182    0.6148    0.6625      1202\n",
            "\n",
            "    accuracy                         0.7868      3532\n",
            "   macro avg     0.7666    0.7452    0.7533      3532\n",
            "weighted avg     0.7821    0.7868    0.7824      3532\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2UujbSqDrWWU",
        "colab_type": "code",
        "outputId": "71ac3449-4be5-49d3-ccc1-3a1a18d54014",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Test your model on a sample (Inference)\n",
        "from farm.infer import Inferencer\n",
        "from pprint import PrettyPrinter\n",
        "\n",
        "infer_model = Inferencer(processor=processor, model=model, task_type=\"text_classification\", gpu=True)\n",
        "\n",
        "basic_texts = [\n",
        "    {\"text\": \"Martin ist ein Idiot\"},\n",
        "    {\"text\": \"Martin Müller spielt Handball in Berlin\"},\n",
        "]\n",
        "result = infer_model.inference_from_dicts(dicts=basic_texts)\n",
        "PrettyPrinter().pprint(result)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:22:39 - INFO - farm.utils -   device: cuda n_gpu: 1, distributed training: False, automatic mixed precision training: None\n",
            "05/05/2020 08:22:39 - INFO - farm.infer -   Got ya 1 parallel workers to do inference ...\n",
            "05/05/2020 08:22:39 - INFO - farm.infer -    0 \n",
            "05/05/2020 08:22:39 - INFO - farm.infer -   /w\\\n",
            "05/05/2020 08:22:39 - INFO - farm.infer -   /'\\\n",
            "05/05/2020 08:22:39 - INFO - farm.infer -   \n",
            "05/05/2020 08:22:39 - INFO - farm.data_handler.processor -   *** Show 2 random examples ***\n",
            "05/05/2020 08:22:39 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-1-0\n",
            "Clear Text: \n",
            " \ttext: Martin Müller spielt Handball in Berlin\n",
            "Tokenized: \n",
            " \ttokens: ['Martin', 'Müller', 'spielt', 'Handball', 'in', 'Berlin']\n",
            " \toffsets: [0, 7, 14, 21, 30, 33]\n",
            " \tstart_of_word: [True, True, True, True, True, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 3810, 5477, 3727, 11065, 50, 715, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "05/05/2020 08:22:39 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-1-0\n",
            "Clear Text: \n",
            " \ttext: Martin Müller spielt Handball in Berlin\n",
            "Tokenized: \n",
            " \ttokens: ['Martin', 'Müller', 'spielt', 'Handball', 'in', 'Berlin']\n",
            " \toffsets: [0, 7, 14, 21, 30, 33]\n",
            " \tstart_of_word: [True, True, True, True, True, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 3810, 5477, 3727, 11065, 50, 715, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 31.84 Batches/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "[{'predictions': [{'context': 'Martin ist ein Idiot',\n",
            "                   'end': None,\n",
            "                   'label': 'OFFENSE',\n",
            "                   'probability': 0.907324,\n",
            "                   'start': None},\n",
            "                  {'context': 'Martin Müller spielt Handball in Berlin',\n",
            "                   'end': None,\n",
            "                   'label': 'OTHER',\n",
            "                   'probability': 0.9301861,\n",
            "                   'start': None}],\n",
            "  'task': 'text_classification'}]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBNdJEhqrWWW",
        "colab_type": "text"
      },
      "source": [
        "# Switch to NER"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ksJq_PRrWWX",
        "colab_type": "text"
      },
      "source": [
        "In a transfer learning paradigm, there is a core computation that is shared amongst all tasks. FARM's modular structure means that you can easily swap out different building blocks to make the same language model work for many different tasks.\n",
        "\n",
        "We can adapt the above text classification model to NER by simply switching out the processor and prediction head."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xESpWInRrWWX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import the new building blocks\n",
        "\n",
        "from farm.data_handler.processor import NERProcessor\n",
        "from farm.modeling.prediction_head import TokenClassificationHead\n",
        "ml_logger.init_experiment(experiment_name=\"Public_FARM\", run_name=\"Tutorial1_Colab_NER\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nN6BqqGcrWWa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# This processor will preprocess the data for the CoNLL03 NER task\n",
        "ner_labels = [\"[PAD]\", \"X\", \"O\", \"B-MISC\", \"I-MISC\", \"B-PER\", \"I-PER\", \"B-ORG\", \"I-ORG\", \"B-LOC\", \"I-LOC\", \"B-OTH\", \"I-OTH\"]\n",
        "\n",
        "ner_processor = NERProcessor(tokenizer=tokenizer, \n",
        "                             max_seq_len=128, \n",
        "                             data_dir=\"../data/conll03-de\",\n",
        "                             label_list=ner_labels,\n",
        "                             metric=\"seq_f1\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "92f-b_acrWWc",
        "colab_type": "code",
        "outputId": "9548938f-928e-4191-e643-2eda00c72181",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# This prediction head is also a feed forward neural network but expects one\n",
        "# vector per token in the input sequence and will generate a set of logits\n",
        "# for each input\n",
        "\n",
        "ner_prediction_head = TokenClassificationHead(num_labels=len(ner_labels))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:23:15 - INFO - farm.modeling.prediction_head -   Prediction head initialized with size [768, 13]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FtZAcpN2rWWf",
        "colab_type": "code",
        "outputId": "4fb7029d-87e8-4c51-a817-66804a66df45",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# We can integrate these new pieces with the rest using this code\n",
        "# It is pretty much the same structure as what we had above for text classification\n",
        "\n",
        "BATCH_SIZE = 32\n",
        "EMBEDS_DROPOUT_PROB = 0.1\n",
        "LEARNING_RATE = 2e-5\n",
        "N_EPOCHS = 1\n",
        "N_GPU = 1\n",
        "\n",
        "data_silo = DataSilo(\n",
        "    processor=ner_processor,\n",
        "    batch_size=BATCH_SIZE)\n",
        "\n",
        "model = AdaptiveModel(\n",
        "    language_model=language_model,\n",
        "    prediction_heads=[ner_prediction_head],\n",
        "    embeds_dropout_prob=EMBEDS_DROPOUT_PROB,\n",
        "    lm_output_types=[\"per_token\"],\n",
        "    device=device)\n",
        "\n",
        "model, optimizer, lr_schedule = initialize_optimizer(\n",
        "    model=model,\n",
        "    learning_rate=LEARNING_RATE,\n",
        "    n_batches=len(data_silo.loaders[\"train\"]),\n",
        "    n_epochs=N_EPOCHS,\n",
        "    device=device)\n",
        "\n",
        "trainer = Trainer(\n",
        "    model=model,\n",
        "    optimizer=optimizer,\n",
        "    data_silo=data_silo,\n",
        "    epochs=N_EPOCHS,\n",
        "    n_gpu=N_GPU,\n",
        "    lr_schedule=lr_schedule,\n",
        "    device=device,\n",
        ")"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:23:32 - INFO - farm.data_handler.data_silo -   \n",
            "Loading data into the data silo ... \n",
            "              ______\n",
            "               |o  |   !\n",
            "   __          |:`_|---'-.\n",
            "  |__|______.-/ _ \\-----.|       \n",
            " (o)(o)------'\\ _ /     ( )      \n",
            " \n",
            "05/05/2020 08:23:32 - INFO - farm.data_handler.data_silo -   Loading train set from: ../data/conll03-de/train.txt \n",
            "05/05/2020 08:23:32 - ERROR - farm.data_handler.utils -   Separator \t for dataset German CONLL03 does not match the requirements. Setting seperator to whitespace\n",
            "05/05/2020 08:23:32 - INFO - farm.data_handler.utils -    Couldn't find ../data/conll03-de/train.txt locally. Trying to download ...\n",
            "05/05/2020 08:23:32 - INFO - farm.data_handler.utils -   downloading and extracting file conll03-de to dir /content/data\n",
            "05/05/2020 08:23:34 - INFO - farm.data_handler.data_silo -   Got ya 1 parallel workers to convert 12152 dictionaries to pytorch datasets (chunksize = 2000)...\n",
            "05/05/2020 08:23:34 - INFO - farm.data_handler.data_silo -    0 \n",
            "05/05/2020 08:23:34 - INFO - farm.data_handler.data_silo -   /w\\\n",
            "05/05/2020 08:23:34 - INFO - farm.data_handler.data_silo -   / \\\n",
            "05/05/2020 08:23:34 - INFO - farm.data_handler.data_silo -   \n",
            "Preprocessing Dataset ../data/conll03-de/train.txt:   0%|          | 0/12152 [00:00<?, ? Dicts/s]05/05/2020 08:23:36 - INFO - farm.data_handler.processor -   *** Show 2 random examples ***\n",
            "05/05/2020 08:23:36 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-627-0\n",
            "Clear Text: \n",
            " \ttext: Noch nie habe sich ein Verein bei einem Club mit eigenem Gelände eingemietet .\n",
            " \tner_label: ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n",
            "Tokenized: \n",
            " \ttokens: ['Noch', 'nie', 'habe', 'sich', 'ein', 'Verein', 'bei', 'einem', 'Club', 'mit', 'eigenem', 'Gelände', 'einge', '##mie', '##tet', '.']\n",
            " \toffsets: [0, 5, 9, 14, 19, 23, 30, 34, 40, 45, 49, 57, 65, 70, 73, 77]\n",
            " \tstart_of_word: [True, True, True, True, True, True, True, True, True, True, True, True, True, False, False, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 4548, 1580, 555, 144, 39, 1057, 178, 297, 5341, 114, 14573, 5676, 888, 10839, 305, 26914, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tinitial_mask: [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tner_label_ids: [1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "05/05/2020 08:23:37 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-887-0\n",
            "Clear Text: \n",
            " \ttext: 650mal ließ man imaginäre zweistrahlige \" paper-jets \" , nur auf dem Papier existierende Flugzeuge , über den Nordatlantik fliegen .\n",
            " \tner_label: ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-MISC', 'O', 'O']\n",
            "Tokenized: \n",
            " \ttokens: ['650', '##mal', 'ließ', 'man', 'im', '##agin', '##äre', 'zwei', '##strahl', '##ige', '\"', 'pa', '##per', '-', 'je', '##ts', '\"', ',', 'nur', 'auf', 'dem', 'Papier', 'existieren', '##de', 'Flugzeuge', ',', 'über', 'den', 'Nord', '##atlant', '##ik', 'fliegen', '.']\n",
            " \toffsets: [0, 3, 7, 12, 16, 18, 22, 26, 30, 36, 40, 42, 44, 47, 48, 50, 53, 55, 57, 61, 65, 69, 76, 86, 89, 99, 101, 106, 110, 114, 120, 123, 131]\n",
            " \tstart_of_word: [True, False, True, True, True, False, False, True, False, False, True, True, False, False, False, False, True, True, True, True, True, True, True, False, True, True, True, True, True, False, False, True, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 24468, 446, 2660, 478, 106, 25648, 1170, 382, 5206, 288, 26944, 20000, 608, 26935, 543, 2643, 26944, 26918, 356, 115, 128, 7918, 10481, 57, 14419, 26918, 204, 86, 1188, 21955, 172, 15090, 26914, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tinitial_mask: [0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tner_label_ids: [1, 2, 1, 2, 2, 2, 1, 1, 2, 1, 1, 2, 2, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 3, 1, 1, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "Preprocessing Dataset ../data/conll03-de/train.txt: 100%|██████████| 12152/12152 [00:11<00:00, 1077.95 Dicts/s]\n",
            "05/05/2020 08:23:46 - INFO - farm.data_handler.data_silo -   Loading dev set from: ../data/conll03-de/dev.txt\n",
            "05/05/2020 08:23:46 - ERROR - farm.data_handler.utils -   Separator \t for dataset German CONLL03 does not match the requirements. Setting seperator to whitespace\n",
            "05/05/2020 08:23:46 - INFO - farm.data_handler.data_silo -   Got ya 1 parallel workers to convert 2867 dictionaries to pytorch datasets (chunksize = 574)...\n",
            "05/05/2020 08:23:46 - INFO - farm.data_handler.data_silo -    0 \n",
            "05/05/2020 08:23:46 - INFO - farm.data_handler.data_silo -   /w\\\n",
            "05/05/2020 08:23:46 - INFO - farm.data_handler.data_silo -   /'\\\n",
            "05/05/2020 08:23:46 - INFO - farm.data_handler.data_silo -   \n",
            "Preprocessing Dataset ../data/conll03-de/dev.txt:   0%|          | 0/2867 [00:00<?, ? Dicts/s]05/05/2020 08:23:47 - INFO - farm.data_handler.processor -   *** Show 2 random examples ***\n",
            "05/05/2020 08:23:47 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-54-0\n",
            "Clear Text: \n",
            " \ttext: Es ist ein Erfolg der UN-Konferenz , den der Süden auf sein Konto verbuchen kann , daß der moralische Druck auf den Norden gewachsen ist , sein Konsumniveau zur Disposition zu stellen .\n",
            " \tner_label: ['O', 'O', 'O', 'O', 'O', 'B-MISC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n",
            "Tokenized: \n",
            " \ttokens: ['Es', 'ist', 'ein', 'Erfolg', 'der', 'UN', '-', 'Konferenz', ',', 'den', 'der', 'Süden', 'auf', 'sein', 'Konto', 'verb', '##uchen', 'kann', ',', 'daß', 'der', 'moral', '##ische', 'Druck', 'auf', 'den', 'Norden', 'gewachsen', 'ist', ',', 'sein', 'Konsum', '##niveau', 'zur', 'Disposition', 'zu', 'stellen', '.']\n",
            " \toffsets: [0, 3, 7, 11, 18, 22, 24, 25, 35, 37, 41, 45, 51, 55, 60, 66, 70, 76, 81, 83, 87, 91, 96, 102, 108, 112, 116, 123, 133, 137, 139, 144, 150, 157, 161, 173, 176, 184]\n",
            " \tstart_of_word: [True, True, True, True, True, True, False, False, True, True, True, True, True, True, True, True, False, True, True, True, True, True, False, True, True, True, True, True, True, True, True, True, False, True, True, True, True, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 482, 127, 39, 2346, 21, 5931, 26935, 13854, 26918, 86, 21, 3236, 115, 167, 10909, 938, 2455, 479, 26918, 2751, 21, 14767, 262, 3548, 115, 86, 2964, 13963, 127, 26918, 167, 12780, 14466, 252, 23103, 81, 3392, 26914, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tinitial_mask: [0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tner_label_ids: [1, 2, 2, 2, 2, 2, 3, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "05/05/2020 08:23:47 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-163-0\n",
            "Clear Text: \n",
            " \ttext: Unter sowjetischer Herrschaft lebten in der autonomen Bergregion auf aserbaidschanischem Territorium einmal 180 000 Menschen .\n",
            " \tner_label: ['O', 'B-MISC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-MISC', 'O', 'O', 'O', 'O', 'O', 'O']\n",
            "Tokenized: \n",
            " \ttokens: ['Unter', 'sowjetische', '##r', 'Herrschaft', 'lebten', 'in', 'der', 'auton', '##omen', 'Berg', '##region', 'auf', 'as', '##erb', '##ai', '##ds', '##chan', '##ischem', 'Territorium', 'einmal', '180', '000', 'Menschen', '.']\n",
            " \toffsets: [0, 6, 17, 19, 30, 37, 40, 44, 49, 54, 58, 65, 69, 71, 74, 76, 78, 82, 89, 101, 108, 112, 116, 125]\n",
            " \tstart_of_word: [True, True, False, True, True, True, True, True, False, True, False, True, True, False, False, False, False, False, True, True, True, True, True, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 388, 18854, 26900, 5562, 6570, 50, 21, 18727, 3417, 2134, 8886, 115, 10205, 7564, 1446, 7108, 11930, 4871, 11497, 1757, 3796, 4847, 1075, 26914, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tinitial_mask: [0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tner_label_ids: [1, 2, 3, 1, 2, 2, 2, 2, 2, 1, 2, 1, 2, 3, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "Preprocessing Dataset ../data/conll03-de/dev.txt: 100%|██████████| 2867/2867 [00:02<00:00, 1079.97 Dicts/s]\n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.data_silo -   Loading test set from: ../data/conll03-de/test.txt\n",
            "05/05/2020 08:23:49 - ERROR - farm.data_handler.utils -   Separator \t for dataset German CONLL03 does not match the requirements. Setting seperator to whitespace\n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.data_silo -   Got ya 1 parallel workers to convert 3005 dictionaries to pytorch datasets (chunksize = 601)...\n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.data_silo -    0 \n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.data_silo -   /w\\\n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.data_silo -   /'\\\n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.data_silo -   \n",
            "Preprocessing Dataset ../data/conll03-de/test.txt:   0%|          | 0/3005 [00:00<?, ? Dicts/s]05/05/2020 08:23:49 - INFO - farm.data_handler.processor -   *** Show 2 random examples ***\n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-519-0\n",
            "Clear Text: \n",
            " \ttext: Keinen Erfolg hatte Bartsch beim Versuch , die Uniklinik Marburg in die bessere Verzahnung von ambulanter und stationärer Behandlung einzubeziehen .\n",
            " \tner_label: ['O', 'O', 'O', 'B-PER', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n",
            "Tokenized: \n",
            " \ttokens: ['Keine', '##n', 'Erfolg', 'hatte', 'Bart', '##sch', 'beim', 'Versuch', ',', 'die', 'Uni', '##klinik', 'Marburg', 'in', 'die', 'bessere', 'Verz', '##ahnung', 'von', 'ambul', '##anter', 'und', 'station', '##ärer', 'Behandlung', 'einzubeziehen', '.']\n",
            " \toffsets: [0, 5, 7, 14, 20, 24, 28, 33, 41, 43, 47, 50, 57, 65, 68, 72, 80, 84, 91, 95, 100, 106, 110, 117, 122, 133, 147]\n",
            " \tstart_of_word: [True, False, True, True, True, False, True, True, True, True, True, False, True, True, True, True, True, False, True, True, False, True, True, False, True, True, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 12918, 26898, 2346, 466, 7892, 28, 785, 6489, 26918, 30, 14890, 19502, 14692, 50, 30, 10245, 6070, 17109, 88, 19913, 15320, 42, 8142, 16588, 4902, 23163, 26914, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tinitial_mask: [0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tner_label_ids: [1, 2, 1, 2, 2, 5, 1, 2, 2, 2, 2, 9, 1, 10, 2, 2, 2, 2, 1, 2, 2, 1, 2, 2, 1, 2, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "05/05/2020 08:23:49 - INFO - farm.data_handler.processor -   \n",
            "\n",
            "      .--.        _____                       _      \n",
            "    .'_\\/_'.     / ____|                     | |     \n",
            "    '. /\\ .'    | (___   __ _ _ __ ___  _ __ | | ___ \n",
            "      \"||\"       \\___ \\ / _` | '_ ` _ \\| '_ \\| |/ _ \\ \n",
            "       || /\\     ____) | (_| | | | | | | |_) | |  __/\n",
            "    /\\ ||//\\)   |_____/ \\__,_|_| |_| |_| .__/|_|\\___|\n",
            "   (/\\||/                             |_|           \n",
            "______\\||/___________________________________________                     \n",
            "\n",
            "ID: train-170-0\n",
            "Clear Text: \n",
            " \ttext: Ich habe mit Hannes Löhr gesprochen , der will ihn . \"\n",
            " \tner_label: ['O', 'O', 'O', 'B-PER', 'I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n",
            "Tokenized: \n",
            " \ttokens: ['Ich', 'habe', 'mit', 'Hann', '##es', 'Lö', '##hr', 'gesprochen', ',', 'der', 'will', 'ihn', '.', '\"']\n",
            " \toffsets: [0, 4, 9, 13, 17, 20, 22, 25, 36, 38, 42, 47, 51, 53]\n",
            " \tstart_of_word: [True, True, True, True, False, True, False, True, True, True, True, True, True, True]\n",
            "Features: \n",
            " \tinput_ids: [3, 1671, 555, 114, 3592, 16, 3866, 48, 7665, 26918, 21, 1279, 716, 26914, 26944, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tpadding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tsegment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tinitial_mask: [0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            " \tner_label_ids: [1, 2, 2, 2, 5, 1, 6, 1, 2, 2, 2, 2, 2, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
            "_____________________________________________________\n",
            "Preprocessing Dataset ../data/conll03-de/test.txt: 100%|██████████| 3005/3005 [00:02<00:00, 1107.69 Dicts/s]\n",
            "05/05/2020 08:23:52 - INFO - farm.data_handler.data_silo -   Examples in train: 12152\n",
            "05/05/2020 08:23:52 - INFO - farm.data_handler.data_silo -   Examples in dev  : 2867\n",
            "05/05/2020 08:23:52 - INFO - farm.data_handler.data_silo -   Examples in test : 3005\n",
            "05/05/2020 08:23:52 - INFO - farm.data_handler.data_silo -   \n",
            "05/05/2020 08:23:52 - INFO - farm.data_handler.data_silo -   Longest sequence length observed after clipping:     128\n",
            "05/05/2020 08:23:52 - INFO - farm.data_handler.data_silo -   Average sequence length after clipping: 24.42947662936142\n",
            "05/05/2020 08:23:52 - INFO - farm.data_handler.data_silo -   Proportion clipped:      0.0002468729427254773\n",
            "05/05/2020 08:23:54 - INFO - farm.modeling.optimization -   Loading optimizer `TransformersAdamW`: '{'correct_bias': False, 'weight_decay': 0.01, 'lr': 2e-05}'\n",
            "05/05/2020 08:23:55 - INFO - farm.modeling.optimization -   Using scheduler 'get_linear_schedule_with_warmup'\n",
            "05/05/2020 08:23:55 - INFO - farm.modeling.optimization -   Loading schedule `get_linear_schedule_with_warmup`: '{'num_warmup_steps': 38.0, 'num_training_steps': 380}'\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JiRA8NrbrWWi",
        "colab_type": "code",
        "outputId": "a52098b2-d27d-4e8c-b319-b6f43a27d720",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "model = trainer.train()"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "05/05/2020 08:24:16 - INFO - farm.train -   \n",
            " \n",
            "\n",
            "          &&& &&  & &&             _____                   _             \n",
            "      && &\\/&\\|& ()|/ @, &&       / ____|                 (_)            \n",
            "      &\\/(/&/&||/& /_/)_&/_&     | |  __ _ __ _____      ___ _ __   __ _ \n",
            "   &() &\\/&|()|/&\\/ '%\" & ()     | | |_ | '__/ _ \\ \\ /\\ / / | '_ \\ / _` |\n",
            "  &_\\_&&_\\ |& |&&/&__%_/_& &&    | |__| | | | (_) \\ V  V /| | | | | (_| |\n",
            "&&   && & &| &| /& & % ()& /&&    \\_____|_|  \\___/ \\_/\\_/ |_|_| |_|\\__, |\n",
            " ()&_---()&\\&\\|&&-&&--%---()~                                       __/ |\n",
            "     &&     \\|||                                                   |___/\n",
            "             |||\n",
            "             |||\n",
            "             |||\n",
            "       , -=-~  .-^- _\n",
            "              `\n",
            "\n",
            "Train epoch 0/0 (Cur. train loss: 0.0358):  26%|██▋       | 100/380 [01:20<03:32,  1.32it/s]\n",
            "Evaluating:   0%|          | 0/90 [00:00<?, ?it/s]\u001b[A\n",
            "Evaluating:  37%|███▋      | 33/90 [00:10<00:17,  3.21it/s]\u001b[A\n",
            "Evaluating: 100%|██████████| 90/90 [00:27<00:00,  3.22it/s]\n",
            "05/05/2020 08:26:06 - INFO - farm.eval -   \n",
            "\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "***************************************************\n",
            "***** EVALUATION | DEV SET | AFTER 100 BATCHES *****\n",
            "***************************************************\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\n",
            "05/05/2020 08:26:06 - INFO - farm.eval -   \n",
            " _________ ner _________\n",
            "05/05/2020 08:26:06 - INFO - farm.eval -   loss: 2.9553135172348632\n",
            "05/05/2020 08:26:06 - INFO - farm.eval -   task_name: ner\n",
            "05/05/2020 08:26:07 - INFO - farm.eval -   seq_f1: 0.7203088489281723\n",
            "05/05/2020 08:26:07 - INFO - farm.eval -   report: \n",
            "            precision    recall  f1-score   support\n",
            "\n",
            "      PER       0.89      0.82      0.85      1401\n",
            "      ORG       0.64      0.65      0.64      1188\n",
            "      LOC       0.74      0.76      0.75      1181\n",
            "     MISC       0.72      0.72      0.72      1010\n",
            "\n",
            "micro avg       0.70      0.74      0.72      4780\n",
            "macro avg       0.75      0.74      0.75      4780\n",
            "\n",
            "Train epoch 0/0 (Cur. train loss: 0.0673):  53%|█████▎    | 200/380 [03:09<02:14,  1.34it/s]\n",
            "Evaluating:   0%|          | 0/90 [00:00<?, ?it/s]\u001b[A\n",
            "Evaluating:  37%|███▋      | 33/90 [00:10<00:17,  3.22it/s]\u001b[A\n",
            "Evaluating: 100%|██████████| 90/90 [00:28<00:00,  3.20it/s]\n",
            "05/05/2020 08:27:55 - INFO - farm.eval -   \n",
            "\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "***************************************************\n",
            "***** EVALUATION | DEV SET | AFTER 200 BATCHES *****\n",
            "***************************************************\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\n",
            "05/05/2020 08:27:55 - INFO - farm.eval -   \n",
            " _________ ner _________\n",
            "05/05/2020 08:27:56 - INFO - farm.eval -   loss: 2.018495182480428\n",
            "05/05/2020 08:27:56 - INFO - farm.eval -   task_name: ner\n",
            "05/05/2020 08:27:57 - INFO - farm.eval -   seq_f1: 0.7965030545607752\n",
            "05/05/2020 08:27:57 - INFO - farm.eval -   report: \n",
            "            precision    recall  f1-score   support\n",
            "\n",
            "      PER       0.91      0.89      0.90      1401\n",
            "      ORG       0.75      0.73      0.74      1188\n",
            "      LOC       0.83      0.79      0.81      1181\n",
            "     MISC       0.77      0.73      0.75      1010\n",
            "\n",
            "micro avg       0.80      0.79      0.80      4780\n",
            "macro avg       0.82      0.79      0.81      4780\n",
            "\n",
            "Train epoch 0/0 (Cur. train loss: 0.0370):  79%|███████▉  | 300/380 [04:59<00:59,  1.34it/s]\n",
            "Evaluating:   0%|          | 0/90 [00:00<?, ?it/s]\u001b[A\n",
            "Evaluating:  36%|███▌      | 32/90 [00:10<00:18,  3.18it/s]\u001b[A\n",
            "Evaluating:  36%|███▌      | 32/90 [00:20<00:18,  3.18it/s]\u001b[A\n",
            "Evaluating: 100%|██████████| 90/90 [00:28<00:00,  3.17it/s]\n",
            "05/05/2020 08:29:45 - INFO - farm.eval -   \n",
            "\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "***************************************************\n",
            "***** EVALUATION | DEV SET | AFTER 300 BATCHES *****\n",
            "***************************************************\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\n",
            "05/05/2020 08:29:45 - INFO - farm.eval -   \n",
            " _________ ner _________\n",
            "05/05/2020 08:29:46 - INFO - farm.eval -   loss: 2.173968605517842\n",
            "05/05/2020 08:29:46 - INFO - farm.eval -   task_name: ner\n",
            "05/05/2020 08:29:47 - INFO - farm.eval -   seq_f1: 0.8004199475065616\n",
            "05/05/2020 08:29:47 - INFO - farm.eval -   report: \n",
            "            precision    recall  f1-score   support\n",
            "\n",
            "      PER       0.93      0.88      0.90      1401\n",
            "      ORG       0.77      0.73      0.75      1188\n",
            "      LOC       0.81      0.85      0.83      1181\n",
            "     MISC       0.79      0.71      0.75      1010\n",
            "\n",
            "micro avg       0.80      0.80      0.80      4780\n",
            "macro avg       0.83      0.80      0.81      4780\n",
            "\n",
            "Train epoch 0/0 (Cur. train loss: 0.0248): 100%|██████████| 380/380 [06:33<00:00,  1.03s/it]\n",
            "Evaluating: 100%|██████████| 94/94 [00:29<00:00,  3.20it/s]\n",
            "05/05/2020 08:31:19 - INFO - farm.eval -   \n",
            "\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "***************************************************\n",
            "***** EVALUATION | TEST SET | AFTER 380 BATCHES *****\n",
            "***************************************************\n",
            "\\\\|//       \\\\|//      \\\\|//       \\\\|//     \\\\|//\n",
            "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\n",
            "05/05/2020 08:31:19 - INFO - farm.eval -   \n",
            " _________ ner _________\n",
            "05/05/2020 08:31:20 - INFO - farm.eval -   loss: 1.7611809622626535\n",
            "05/05/2020 08:31:20 - INFO - farm.eval -   task_name: ner\n",
            "05/05/2020 08:31:21 - INFO - farm.eval -   seq_f1: 0.7804878048780487\n",
            "05/05/2020 08:31:21 - INFO - farm.eval -   report: \n",
            "            precision    recall  f1-score   support\n",
            "\n",
            "      ORG       0.69      0.60      0.64       773\n",
            "      PER       0.94      0.93      0.93      1195\n",
            "      LOC       0.81      0.80      0.81      1031\n",
            "     MISC       0.68      0.67      0.68       670\n",
            "\n",
            "micro avg       0.78      0.78      0.78      3669\n",
            "macro avg       0.80      0.78      0.79      3669\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aiFk6UtWrWWl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}