The site in three short walkthroughs

Step 1 (0:00). All 154 dev claims: the annotators' label, the model's verdict and the gold evidence it found

The Explore page opens on four headline figures with 95% intervals: mean evidence F of 0.043, 13 of 154 claims with any gold passage found, verdict accuracy of 38.3% and 47% of claims on the fallback path.

Step 2 (0:07). Filter to the claims where the 2024 retrieval found at least one gold passage

Choosing “Found gold” in the Evidence filter narrows the list to the 13 claims where retrieval found a gold passage.

Step 3 (0:12). Open a claim to compare gold and retrieved evidence side by side

Dev claim 752, “[South Australia] has the most expensive electricity in the world”, opens on its own page with its claim tags and the annotators' label, Supports.

Step 4 (0:17). The retrained Transformer's verdict three ways, with its class probabilities

Three cards show the verdict under the notebook protocol, one claim at a time, and with gold evidence. Each has bars for all four class probabilities and says whether it matches the label.

Step 5 (0:24). Retrieved passages (left) against the annotators' gold evidence (right); hits are marked gold

The 2024 submission retrieved four passages, two of them gold, so precision is 0.50, recall 1.00 and F 0.67. Each passage shows its tags, cosine similarity, tag overlap and score.

Step 6 (0:30). Switch between the submitted 2024 retrieval and the faithful re-runs, with P, R and F per run

The tabs switch between the saved 2024 output, the re-run of the submitted rule, the notebook's rule and the raw-claim variant, each with its own scores.

Step 1 (0:00). Check a claim: type any sentence about climate science

On the Try page the claim “Sea levels are rising faster than ever because glaciers are melting.” is typed into the claim box. The submitted 2024 scoring rule is selected.

Step 2 (0:08). Run the original 2024 pipeline: TF-IDF evidence retrieval, then the from-scratch Transformer

Pressing “Check this claim” sends the sentence to the server, which runs the TypeScript ports of the original preprocessing, retrieval and classifier.

Step 3 (0:14). The verdict with all four class probabilities, beside the model's dev accuracy and its 95% CI

The model answers Supports with 46.1% probability, ahead of not enough info at 27.3%. A note says the model is right on 37.7% of dev claims (95% CI 30.4% to 45.5%), no better than always answering “supports”.

Step 4 (0:22). The passages it retrieved, each with its cosine similarity and tag-overlap scores

Four retrieved passages about glacier and ice-sheet melt and sea-level rise are listed with their shared tags, cosine, overlap and combined score. A banner explains that this sentence was searched against a 39,666-passage subset.

Step 5 (0:28). Under the hood: the notebook's preprocessing, the TF-IDF tag vector and each word's exact contribution

The preprocessing panel shows the tokens kept and dropped, the tag TF-IDF weights, and a chart of how much each word pushes the verdict towards each label.

Step 6 (0:34). Or pick a one-click example, checked against the full 1.19M-passage search at build time

Clicking the example “Arctic sea ice has been shrinking for decades.” runs it straight away. A green note confirms that it returns the same passages as the full 2024 search over all 1.19M passages.

Step 1 (0:00). Original model vs LLM: the same dev claims and the same evidence, scored as a paired comparison

The LLM evaluation page sets out the protocol: the same seeded sample of dev claims, a structured JSON answer, retrieved and gold evidence conditions, and paired statistics.

Step 2 (0:05). Bring your own key: AI is optional, and the key stays in this browser and goes only to the provider

“Add key” opens the AI settings dialog: Anthropic (default) or OpenAI, the model, a key field, “Remember on this device” (off keeps the key in sessionStorage for this tab only) and a note that the key never reaches the site.

Step 3 (0:15). No key is entered in this demo. Cancel, then load a saved run file instead

The dialog is cancelled without entering a key.

Step 4 (0:19). A seeded mock run (no model was called) loaded through “Load a saved run” [Mocked AI response for illustration]

A run file generated by the tour script with a seeded random generator is loaded. Its model is named “mocked-llm-for-illustration” and every rationale says no model was called.

Step 5 (0:23). Accuracy and macro-F1 with 95% intervals, the paired difference, McNemar's exact test [Mocked AI response for illustration]

For each evidence condition the page shows accuracy (Wilson interval) and macro-F1 (bootstrap interval) for the mock and the classifier, the paired difference, McNemar's exact test, Cohen's h, citation validity and evidence F. The numbers describe the mock, not any real model.

Step 6 (0:29). Claim by claim, each LLM verdict is labelled AI-generated beside the classifier's [Mocked AI response for illustration]

The claim table lists each sampled claim with its gold label, the classifier's verdict and the mock's verdict for both conditions, marked right or wrong.