ABLTagger is a bidirectonal LSTM Part-of-Speech Tagger with combined Word and Character embeddings, augmented with a morphological lexicon and a lexical category identification step. The work is described in the paper Augmenting a BiLSTM Tagger with a Morphological Lexicon and a Lexical Category Identification Step NOTE: This code has been updated to work with a revised fine-grained tagging schema for Icelandic. If you find this work useful in your research, please cite the paper:

@inproceedings{steingrimsson-etal-2019-augmenting,
    title = "Augmenting a {B}i{LSTM} Tagger with a Morphological Lexicon and a Lexical Category Identification Step",
    author = {Steingr{\'\i}msson, Stein{\th}{\'o}r  and
      K{\'a}rason, {\"O}rvar  and
      Loftsson, Hrafn},
    booktitle = "Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)",
    month = sep,
    year = "2019",
    address = "Varna, Bulgaria",
    url="/https://www.aclweb.org/anthology/R19-1133",
    doi = "10.26615/978-954-452-056-4_133",
    pages = "1161--1168",
}

The paper describes a method for achieving high accuracy in part-of-speech tagging a fine grained tagset. We show how the method is used to reach the highest accuracy reported for PoS-tagging Icelandic. The tagger is augmented by using a morphological lexicon, The Database of Icelandic Morphology (DIM), and by running a pre-tagging step using a very coarse grained tagset induced from the fine grained data.

Before training make sure the requirements in requirements.txt are set up.

The training data is a text file in the ./data/ folder. The file contains PoS-tagged sentences. The file has one token per line, as well as its corresponding tag. The sentences are separated by an empty line.

Við     fp1fn
höfum   sfg1fn
góða    lveosf
aðstöðu nveo
fyrir   af
barnavagna      nkfo
og      c
kerrur  nvfo
.       pl

Börnin  nhfng
geta    sfg3fn
sofið   sþghen
úti     aa
ef      c
vill    sfg3en
.       pl

In the paper we use the training sets from The Icelandic Frequency Dictionary and the MIM-GOLD. We download the training files and to make sure they are correctly formed (no spaces or extra symbols in the lines that should be empty), we run ./preprocess/generate_fine_training_set.py on the training file. In order to run the lexical category identification step we also generate a coarse grained training set from the data, by running ./preprocess/generate_coarse_training_set.py.

python3 ./preprocess/generate_fine_training_set.py 

The script can take two parameters:

python3 ./preprocess/generate_coarse_training_set.py 

The script can take two parameters:

When training with the coarse grained data, the file word_class_vectors.txt is needed. It is stored in the ./extra/ directory. The file contains one-hot vectors for all possible coarse-grained tags.

We represent the information contained in the morphological lexicon with n-hot vectors. To generate the n-hot vectors, different scripts will have to be written for different morphological lexicons. We use the DIM morphological lexicon for Icelandic. The ./preprocess/ folder contains a script, vectorize_dim.py, to create n-hot vectors from DIM. We first download the data in SHsnid format. After unpacking the SHsnid.csv file is copied into ./data/. To generate the n-hot vectors we run the script:

python3 ./preprocess/vectorize_dim.py 

The script can take two parameters:

A model can be trained with the script train.py. The program requires input corpora to be in the same format as the IFD-training/testing sets, as described above. Running ./train.py -h gives information on all possible parameters. The default parameters are the ones used in the paper. One parameter, model, is required. It is for the name of the model. In out example we call the model Full.

python3 ./train.py -m Full 

The program runs on a CPU and training with default settings and the two Icelandic corpora combined takes approximately 5 hours on an Intel i9-9900K CPU @ 3.60GHz. As some parts of the training process are memory hungry, 32GB of memory are recommended.

Texts can be tagged using the script train.py. The program loads a model stored in the ./models folder. It can be a model trained by the user or a pre-trained model. A model trained on the IFD and MIM-GOLD combined can be downloaded:

• Full
  • A model trained on all training data used in the paper cited above, taking advantage of the whole DMII morphological lexicon. This model needs at least 16GB RAM to load.
  • Download link: https://www.dropbox.com/s/59thds6lun2zki5/Full.tar.gz?dl=0 (374 MB download - 7.0 GB uncompressed)
  • The model should go into a folder called ./models/Full

The model needs the contents of https://www.dropbox.com/s/97s4zk4d9zk722x/extra.tar.gz?dl=0 to be in the ./extra folder.

Running ./tag.py -h gives information on all possible parameters. At minimum the input file(s) have to be specified, and normally the model is also specified.

python3 ./tag.py -m Full -i text_file.txt

Training/testing sets can be evaluated with the script evaluate.py. Before evaluation a script to minimize the DIM, minimize_dim_for_evaluation.py, can be run to reduce time spent in training and testing the model. The script finds all word forms in the training/testing data and removes n-hot vectors from the DIM file for words that are not in the training/testing data. Before evaluating the models the ./preprocess/generate_fine_training_set.py and ./preprocess/generate_coarse_training_set.py should be run as described in the previous section, on all train/test files. To evaluate the accuracy of the tagger on fold number 1 in a set of 10 folds from the mim_gold corpus, the following command does that with all the same settings as used in the paper.

python3 ./evaluate.py -c mim_gold -fold 1 -morphles dmii.vectors.mim_gold

Running ./evaluate.py -h gives information on all possible parameters.

The script writes results to files in the ./evaluate/ folder. ./preprocess/calc_accuracy.py reads these files and gives you the average accuracy over all the folds in a 10-fold validation. ./preprocess/quantify_errors.py gives you a list of the most common errors made by the tagger.