Skip to content
Climate Claim Checker

Dev claim · claim-316 · #74 of 154

“the models predicted seven times as much warming as has been observed”

Claim tags (sorted stems; highlighted when a passage shares them)

  • model
  • much
  • observ
  • predict
  • seven
  • time
  • warm

Annotators’ label

Disputed Run this claim live

The model’s verdict

The Transformer retrained from the notebook (the 2024 weights were never saved), shown three ways. The first is how the notebook evaluated it.

Notebook protocol

Dev claims predicted in batches of 16, in file order, from the 2024 retrieved evidence, as the notebook does.

Supports✗ differs from the label
Supports
59.2%
Refutes
35.9%
Not enough info
0.3%
Disputed
4.7%

One claim at a time

The same model run on a single claim, which is how the Try-it page runs it.

Supports✗ differs from the label
Supports
52.0%
Refutes
41.4%
Not enough info
3.8%
Disputed
2.8%

With gold evidence

Fed the human-annotated evidence instead of retrieved passages. This is the 'val accuracy' the training loop reports.

Supports✗ differs from the label
Supports
61.8%
Refutes
23.3%
Not enough info
2.9%
Disputed
11.9%

Retrieved vs gold evidence

Gold passages were picked by the dataset’s annotators. Retrieved passages are what the TF-IDF rule returned from all 1.19M passages. Switch between the submitted 2024 output and the re-runs.

2024 submission

The passages the team actually retrieved and submitted in 2024 (their saved output file). The file records only passage ids, so scores are recomputed with the submission rule.

Path
not recorded
Found
0 of 3 gold
P
0.00
R
0.00
F
0.00
  1. #1evidence-98712

    Much of this time has been on a houseboat.

    • much (shared with the claim)
    • محمد
    • time (shared with the claim)
    cos
    0.562
    overlap
    0.667
    score
    1.228
    shared
    2

Gold evidence (3)

  1. evidence-733847

    The model predicted <0.2 °C warming for upper air at 700 mb and 500 mb.

    • model (shared with the claim)
    • air
    • warm (shared with the claim)
    • محمد
    • predict (shared with the claim)
    • upper
  2. evidence-980959

    However, the report also observed that the rate of warming over the period 1998–2012 was lower than that predicted by 111 out of 114 Coupled Model Intercomparison Project climate models.

    • warm (shared with the claim)
    • observ (shared with the claim)
    • model (shared with the claim)
    • lower
    • report
    • rate
    • predict (shared with the claim)
    • coupl
  3. evidence-368241

    They found that the observed warming was similar to two of the three scenarios.

    • warm (shared with the claim)
    • observ (shared with the claim)
    • similar
    • found
    • two
    • three
    • scenario