11  TRF Vectors

11.1 Accessing TRF Vectors with LatinCy

import warnings
warnings.filterwarnings("ignore", message="CUDA is not available")

# Imports & setup

import spacy
import numpy as np
from tabulate import tabulate
from pprint import pprint
nlp = spacy.load('la_core_web_trf')

# Workaround: _similarity in trf_vectors calls obj.vector, which is 0 for trf docs.
# The correct approach is to mean-pool trf_token_vecs directly.
from spacy.language import Language
def _trf_cosine(v1, v2):
    n1, n2 = np.linalg.norm(v1), np.linalg.norm(v2)
    return float(np.dot(v1, v2) / (n1 * n2)) if n1 and n2 else 0.0
def _trf_doc_similarity(obj, other):
    v1 = obj._.trf_token_vecs.mean(axis=0)
    v2 = other._.trf_token_vecs.mean(axis=0)
    return _trf_cosine(v1, v2)
def _trf_token_similarity(obj, other):
    return _trf_cosine(obj.vector, other.vector)
try:
    @Language.component('trf_similarity_fix')
    def _trf_similarity_fix(doc):
        doc.user_hooks['similarity'] = _trf_doc_similarity
        doc.user_span_hooks['similarity'] = _trf_doc_similarity
        doc.user_token_hooks['similarity'] = _trf_token_similarity
        return doc
except ValueError:
    pass  # already registered
if 'trf_similarity_fix' not in nlp.pipe_names:
    nlp.add_pipe('trf_similarity_fix', after='trf_vectors')

text = "Haec narrantur a poetis de Perseo. Perseus filius erat Iovis, maximi deorum; avus eius Acrisius appellabatur. Acrisius volebat Perseum nepotem suum necare; nam propter oraculum puerum timebat."
doc = nlp(text)
print(doc)

Haec narrantur a poetis de Perseo. Perseus filius erat Iouis, maximi deorum; auus eius Acrisius appellabatur. Acrisius uolebat Perseum nepotem suum necare; nam propter oraculum puerum timebat.

As noted in the previous notebook, the trf vectors work slightly differently than vectors in the other pipelines. This notebook covers the specifics of working with the trf contextual vectors.

Here is the vector for the first token ‘Haec’ in the text given above…

tokens = [token for token in doc]
for token in tokens:
    print(token.text)
    print(token.vector[:10], "etc.")
    break

Haec
[ 1.0053577  -0.6338967   0.28178048  0.3656186   1.6352     -0.96579456
  1.3706194   0.7118512   1.210259   -0.2383545 ] etc.

The LatinCy trf model uses MultilingualBERT as the basis of its ‘transformer’ component. The trf model has a custom Doc attribute that can give access to the per-token contextual vectors which is in turn mapped back to the vector attribute of the Token object via a user hook. Accordingly, you can just access the vector as you do with the other pipelines, but the output will be contextually informed.

# Here is an example using Acrisius, a word which appears twice in the given text...

print("Token.text values for two tokens...")
print(doc[17])
print(doc[20])

# Note that we have a Doc custom attribute `trf_token_vecs` which is a list of the vectors for each token in the document. We can access the vector for a specific token by using the index of the token in the document.

print("\nSlices correspondiong to Acrisius from the trf_token_vecs Doc custom attribute...")
print(doc._.trf_token_vecs[17][:5])
print(doc._.trf_token_vecs[20][:5])

# But we can access these contextual vectors directly via the `vector` Token attribute.
print("\nToken attribute for the Acrisius slices...")
print(doc[17].vector[:5])
print(doc[20].vector[:5])

print("\nVectors are the same?")
print(np.mean(doc[17].vector) == np.mean(doc._.trf_token_vecs[17]))

Token.text values for two tokens...
Acrisius
Acrisius

Slices correspondiong to Acrisius from the trf_token_vecs Doc custom attribute...
[ 0.15758377 -0.7168379   2.458179   -1.4819897   2.422638  ]
[ 0.14679147 -0.7611529   1.8628647  -0.72165     2.1575801 ]

Token attribute for the Acrisius slices...
[ 0.15758377 -0.7168379   2.458179   -1.4819897   2.422638  ]
[ 0.14679147 -0.7611529   1.8628647  -0.72165     2.1575801 ]

Vectors are the same?
True

Reminder that the shape of the (MultilingualBERT-derived) vectors is 768…

print(doc[0].vector.shape)

(768,)

For the trf pipelines, the similarity method is available via a user hook registered by the trf_vectors component. This means you can compare tokens, spans, or docs using .similarity() just as you would with the other pipelines — the comparison uses cosine similarity over the contextual vectors.

text_1 = "Omnia vincit amor."
text_2 = "Omnia vincit amicitia."

doc_1 = nlp(text_1)
doc_2 = nlp(text_2)

print("Similarity between the two sentences...")
print(doc_1.similarity(doc_2))

Similarity between the two sentences...
0.9511939883232117

References

SLP Chapter 5, “Embeddings” link
SLP Ch. 10 “Masked Language Models” link