5  Word Tokenization

5.1 Word tokenization with LatinCy

# Imports & setup

import spacy
from pprint import pprint
nlp = spacy.load('la_core_web_sm')
text = "Haec narrantur a poetis de Perseo. Perseus filius erat Iovis, maximi deorum. Avus eius Acrisius appellabatur."
doc = nlp(text)
print(doc)

Haec narrantur a poetis de Perseo. Perseus filius erat Iouis, maximi deorum. Auus eius Acrisius appellabatur.

Word tokenization is the task of splitting a text into words (and wordlike units like punctuation, numbers, etc.). For the LatinCy models, tokenization is the fundamental pipeline component on which all other components depend. SpaCy uses non-destructive, “canonical” tokenization, i.e. non-destructive, in that the original text can be untokenized, so to speak, based on Token annotations and canonical in that indices are assigned to each token during this process and these indices are used to refer to the tokens in other annotations. (Tokens can be separated or merged, but this requires the user to actively undo and redefine the tokenization output.) LatinCy uses a modified version of the default spaCy tokenizer that recognizes and splits enlitic -que using a rules-based process. (NB: It is in the LatinCy development plan to move enclitic splitting to a separate post-tokenization component.)

The spaCy Doc object is an iterable and tokens are the iteration unit.

tokens = [item for item in doc]
print(tokens)

[Haec, narrantur, a, poetis, de, Perseo, ., Perseus, filius, erat, Iouis, ,, maximi, deorum, ., Auus, eius, Acrisius, appellabatur, .]

token = tokens[0]
print(type(token))

<class 'spacy.tokens.token.Token'>

The text content of a Token object can be retrieved with the text attribute.

for i, token in enumerate(tokens, 1):
    print(f'{i}: {token.text}')

1: Haec
2: narrantur
3: a
4: poetis
5: de
6: Perseo
7: .
8: Perseus
9: filius
10: erat
11: Iouis
12: ,
13: maximi
14: deorum
15: .
16: Auus
17: eius
18: Acrisius
19: appellabatur
20: .

Note again that the token itself is a spaCy Token object and that the text attribute returns a Python string even though their representations in the Jupyter Notebook look the same.

token = tokens[0]
print(f'{type(token)} -> {token}')
print(f'{type(token.text)} -> {token.text}')

<class 'spacy.tokens.token.Token'> -> Haec
<class 'str'> -> Haec

5.1.1 Token attributes and methods related to tokenization

Here are some atrributes/methods available for spaCy Token objects that are relevant to word tokenization.

SpaCy keeps track of both the token indices and the character offsets within a doc using either the i or idx attributes, respectively…

print(token.doc)

Haec narrantur a poetis de Perseo. Perseus filius erat Iouis, maximi deorum. Auus eius Acrisius appellabatur.

# token indices

for token in doc:
    print(f'{token.i}: {token.text}')

0: Haec
1: narrantur
2: a
3: poetis
4: de
5: Perseo
6: .
7: Perseus
8: filius
9: erat
10: Iouis
11: ,
12: maximi
13: deorum
14: .
15: Auus
16: eius
17: Acrisius
18: appellabatur
19: .

This is functionally equivalent to using enumerate…

# token indices, with enumerate

for i, token in enumerate(doc):
    print(f'{i}: {token.text}')

0: Haec
1: narrantur
2: a
3: poetis
4: de
5: Perseo
6: .
7: Perseus
8: filius
9: erat
10: Iouis
11: ,
12: maximi
13: deorum
14: .
15: Auus
16: eius
17: Acrisius
18: appellabatur
19: .

Another indexing option is the idx attribute which is the character offset of the token in the original Doc object.

# character offsets, 
for token in doc:
    print(f'{token.idx}: {token.text}')

0: Haec
5: narrantur
15: a
17: poetis
24: de
27: Perseo
33: .
35: Perseus
43: filius
50: erat
55: Iouis
60: ,
62: maximi
69: deorum
75: .
77: Auus
82: eius
87: Acrisius
96: appellabatur
108: .

Observe these idx attributes relate to the character offsets from the original Doc. To illustrate the point, we will replace spaces with an underscore in the output. We can see from the output above that narrantur begins at idx 5 and that the next word a begins at idx 15. Yet narrantur is only 9 characters long and the difference between these two numbers is 10! This is because we need to account for whitespace in the original Doc. This is handled by the attribute text_with_ws.

print(doc.text[5:15].replace(' ', '_'))

narrantur_

print(f'text -> {doc[1].text} (length {len(doc[1].text)})')
print()
print(f'text_with_ws -> {doc[1].text_with_ws} (length {len(doc[1].text_with_ws)})')

text -> narrantur (length 9)

text_with_ws -> narrantur  (length 10)

Accordingly, using the text_with_ws attribute (as opposed to simply the text attribute) we can reconstruct the original text. This is what was meant above by “non-destructive” tokenization. Look at the difference between a text joined using the text attribute and one joined using the text_with_ws attribute.

joined_tokens = ' '.join([token.text for token in doc])
print(joined_tokens)
print(joined_tokens == doc.text)

print()

reconstructed_text = ''.join([token.text_with_ws for token in doc])
print(reconstructed_text)
print(reconstructed_text == doc.text)

Haec narrantur a poetis de Perseo . Perseus filius erat Iouis , maximi deorum . Auus eius Acrisius appellabatur .
False

Haec narrantur a poetis de Perseo. Perseus filius erat Iouis, maximi deorum. Auus eius Acrisius appellabatur.
True

Because spaCy tokenization is set from the outset, you can traverse the tokens in a Doc objects from the tokens themselves using the nbor method. This method takes an integer argument that specifies the number of tokens to traverse. A positive integer traverses the tokens to the right, a negative integer traverses the tokens to the left.

print(doc[:6])
print('-----')
print(f'{doc[3]}, i.e. i = 3')
print(f'{doc[3].nbor(-1)}, i.e. i - 1 = 2')
print(f'{doc[3].nbor(-2)}, i.e. i - 2 = 1')
print(f'{doc[3].nbor(1)}, i.e. i + 1 = 4')
print(f'{doc[3].nbor(2)}, i.e. i + 2 = 5')

Haec narrantur a poetis de Perseo
-----
poetis, i.e. i = 3
a, i.e. i - 1 = 2
narrantur, i.e. i - 2 = 1
de, i.e. i + 1 = 4
Perseo, i.e. i + 2 = 5

5.1.2 Normalization at the tokenization stage

In order to start the pipeline processing with text data that is as close as possible to the data used for training all of the downstream components, the LatinCy models include a certain amount of normalization at the tokenization stage. These four normalization functions run at tokenization: 1. macron removal; 2. accent removal; 3. ligature separation; and 4. space adjustment (i.e. converting all whitespace to a single space and stripping whitespace from the left and right side of the input text).

text = " Hæc     peritè\n\nnarrāntur\t\ta poētīs dē Perseō Acrisiōque."
doc = nlp(text)

print("Before normalization:")
print(text, "\n")

print("After normalization:")
print(doc.text)

Before normalization:
 Hæc     peritè

narrāntur       a poētīs dē Perseō Acrisiōque. 

After normalization:
Haec perite narrantur a poetis de Perseo Acrisioque.

References

SLP Chapter 2, Section 2.8 “Rule-based tokenization” link