One claim, through every stage of the pipeline

Contractions are expanded with the contractions package. Text is lowercased, and every ASCII punctuation mark becomes a space (so sentence.abc splits in two). NLTK tokenises, English stopwords and non-alphabetic tokens are dropped, and the Porter stemmer reduces what is left.

tokens

south australia has the most expensive electricity in the world

kept

south australia expensive electricity world

stems

south australia expens electr world

A claim’s tags are simply its stems, sorted: australia electr expens south world. The evidence was tagged once, offline, across all 1.2M passages: each passage keeps its ten highest-weighted features under a TF-IDF model with 20,000 uni- to tri-gram features, and only single words survive. Here is the first gold passage for this claim:

A quirk: numpy’s argsort pads short passages’ top ten with zero-weight features, in an order that depends on its unstable quicksort. The last vocabulary entry, محمد, is alphabetic, so it ended up in 167,592 passages’ tags (14.1%). For “Their number was increased to forty in 1855.” the tags are forti, increa, number, محمد (weight 0). The port reimplements numpy’s introsort so these tags come out identical.

Claim tags and passage tags are vectorised by a second, 1,000-term TF-IDF model (the tags are stemmed again on the way in). Each passage then gets three numbers:

• cosine

  cosine similarity of the two tag vectors

• overlap

  shared tags ÷ the smaller tag set

• score

  cosine + overlap (the notebook adds the cosine twice)

Keep passages with cosine > 0.55 and overlap > 0.5. Sort them by score and keep only those sharing the most tags with the claim, at most 6. For this claim 8 passages passed and the 4 sharing 3 tags were kept. They include 2 of the 2 gold passages (F = 0.67). When nothing passes the filter, the top six by score are returned instead. That happened for 47% of dev claims.

The claim’s stems and the stems of every retrieved passage are joined into one string. The string is wrapped in [CLS] … [SEP], mapped to a 5,886-word vocabulary built from the training claims and their gold evidence, and cut or padded to 128 tokens. This claim uses 37 real tokens and 91 [PAD]s.

model input

south australia expens electr world carl albert unbehaun februari electr engin south australia citat need south australia highest retail price electr countri south australia highest power price world class first electr train oper south australia

Embedding

256-d

Encoder

6 layers × 8 heads

Feed-forward

512

Training

10 epochs, Adam lr 1e-4

The quirk that makes it explainable. PyTorch’s nn.TransformerEncoder expects (sequence, batch, features) unless batch_first=True. The notebook passes (batch, sequence, features), so attention ran across the 16 claims of a mini-batch at each position, never across the words of a claim. The positional encoding likewise marks a claim’s place in the batch, not a word’s place in the sentence. Given one claim on its own, every position is transformed independently and the mean-pooled output becomes

logits = bias + (1/128) × Σ_t g[token_t]

Here g is a 5,886 × 4 table exported from the retrained weights. It matches PyTorch to within 7.7e-7, so the site needs a 94 KB table, not a 4.7M-parameter network, and can show each word’s exact share of the verdict. The notebook evaluated dev claims in batches of 16, so its predictions also depend on their neighbours in the file. The Explore page shows both protocols.