Learn · Research integrity
A laptop-sized AI model can prescreen your citations for free, and never send a word off your machine. But there is one kind of claim it reliably gets wrong, and it is the one that matters most. We tested two local models on 120 real examples: a claim, the paper it cites, and the question of whether the source actually supports the claim. Here is where a local model helps, where it fails, and the rule that keeps it honest.
After this report you will be able to:
We wrote 120 short pairs. Each is a claim and a snippet from the source it cites, across three lung-cancer topics: nodule management, risk factors, and neoadjuvant therapy. Every pair got a reference label drawn from guidelines and trial reports, using four plain verdicts:
Two local models read each pair blind and gave a verdict: qwen3:14b (a 14-billion-parameter model) and hermes3:8b (8 billion). Both run on an ordinary laptop through Ollama, offline, at roughly two seconds per pair once loaded. For comparison we also had a large cloud model (Claude) rate the same pairs.
On clear cases, the local models did well. When a source plainly supports or plainly contradicts a claim, both models usually got it, and quickly. Their agreement with the reference:
| Topic (40 pairs each) | qwen3:14b | hermes3:8b |
|---|---|---|
| Nodule management | 88% | 80% |
| Risk factors | 95% | 82% |
| Neoadjuvant therapy | 85% | 90% |
For a tool that costs nothing, runs offline, and keeps every claim on your own machine, that is a genuinely useful first pass. Notice, too, that neither model was consistently the better one. The 14B won two topics and the 8B won the third. "Bigger" did not settle it.
The models had one consistent blind spot, and it is the important one. When a source neither confirms nor refutes a claim, when the honest answer is unclear, the small models rarely said so. They forced a verdict instead.
They forced it in opposite directions. The 14B model tended to read "the evidence has not established this" as a contradiction. The 8B model tended to wave the same claim through as support. Across all 120 pairs, every single case where both models broke from the reference was an "unclear" case. The clear claims they mostly handled; the honestly uncertain ones they could not sit with.
This matters because "unclear" is exactly where a citation most needs a careful human. A claim that overstates what its source shows, or leans on evidence that is still contested, is the kind of citation problem worth catching, and it is the kind the laptop model is least equipped to flag on its own.
An honest limit of this test. We wrote both the claims and the reference labels. That means the cloud model, which reasons the way we do, looked near-perfect partly because it was in effect agreeing with its own author, not because it had proven anything. Treat these numbers as indicative, not a validation. The honest version of this study needs the reference labels written by someone who did not write the claims. We say so here because a citation tool that hid its own weak evidence would be failing its own test.
It is tempting to trust a verdict more when two models land on it. Our data says do not. In the handful of cases where the two local models agreed with each other but against the reference, they were both wrong, agreeing on the same mistake. Agreement between models measures how alike they are, not how right they are. The only thing that tells you a citation holds up is an independent check, and a record of who checked it.
Some of the failure was instruction, not ability. When we spelled out the rule, that "not established" means unclear rather than contradicted, the 14B model jumped from 85% to 98% on the neoadjuvant topic, recovering almost all of its unclear cases. The 8B model improved on unclear too, but then started mislabelling genuine contradictions: fixing one branch broke another. A mid-size model could be taught the distinction; the smaller one hit a ceiling. So the practical lesson is twofold. Prompt these models carefully, and do not assume a smaller model will follow a more careful instruction.
A local model is a fast first pass, not a verdict. Use it to triage, to sort the obvious supports and obvious contradictions from the pile so your attention goes where it is needed. But the human rates the claim, especially any claim where the source hedges. And you record which verdict came from the model and which from you, so the reasoning is auditable later. Models suggest; you decide. On the "unclear" claims, that is not a nicety. It is the whole job.
This is exactly what CiteVahti does, locally.
CiteVahti runs the local model as a blinded prescreener over the claims in your manuscript, but has you rate each one first, before it reveals the model's opinion, so the model cannot anchor your judgement. Every rating is logged with its provenance. It never asserts a citation is true, only whether the source supports the claim, on the record.
See how CiteVahti checks your citationsSee also AI qualitative coding, human-reviewed, the same principle for coding interview data: the local model suggests, you review every one, and confidence is not accuracy. More local-first guides are on the Learn page.