How we turn evidence into a number

A fact only reaches a scoring rule if it carries at least one citation to a primary source, a privacy policy, an audit report, a court filing, source code, or a measurement we took ourselves. This is not a guideline we try to follow; a fact with an empty evidence list cannot be constructed in the first place, so it cannot reach a rule.

We never cite a competitor's review site. A domain blocklist fails our build if anyone tries. Ranking VPNs partly on what other ranking sites said about them would launder their conclusions into ours, and their conclusions are the thing we exist to replace.

Facts we have not sourced are represented by leaving the field out, never by a zero and never by an empty citation. The difference matters for the next rule.

The versioned kill-switch, DNS, WebRTC, and IPv6 specifications define the test matrix, evidence requirements, pass conditions, limitations, and blank raw-data format. Protocol pages are not results: no provider receives a pass until completed evidence and reviewer records are published.

If we have no evidence for a dimension, that dimension is excluded from the weighted mean, not scored as zero. Penalising a provider for a hole in our research would quietly reward whichever providers we happened to research least, which is the precise failure the whole model is built to avoid.

The price of this rule is that scores are not automatically comparable, and we pay it in the open. Every dimension carries a coverage fraction, how many of its rules had evidence to act on, and every score carries a scored-weight: an 8.4 scored on 72% of the methodology means 28% had no data, and the page says so beside the number. A dimension with no facts at all scores neither 0 nor 10 but nothing, and drops out.

We do not run a global speed matrix or a streaming unblock suite. Printing a vendor's own throughput figure in the same typeface as a privacy policy we read line by line would erase exactly the distinction that matters, so every score is stamped with how it was sourced. There are four kinds:

Provenance also sets a ceiling. Because we cite others' speed tests rather than run our own, speed tops out at 8.5. Because a streaming claim is a marketing page, not a test we ran, streaming tops out at 6.5. A number we did not earn the right to certify cannot reach the top of our scale, and the cap is shown, not hidden.

A provider is scored on eleven dimensions. The default weighting allocates the most weight to what we can independently verify and the least to what we can only read off a vendor's page. Speed and streaming, both vendor-territory here, carry roughly half the weight the industry standard gives them; that weight moves to privacy, transparency and server authenticity.

The Top10VPN parity column reproduces a competitor's published weighting over our evidence. It zeroes transparency and server authenticity, that methodology has no equivalent category, and it is shipped on purpose: it lets you separate “the data disagrees” from “the methodology disagrees.” If you suspect we tuned the weights to reach a conclusion, run the numbers under theirs and see what moves.

The same evidence is also published under three use-case weightings. They are not doorway pages: same data, a different question.

You cannot check a number without knowing the arithmetic behind it, so which of three models produced a dimension is published beside it.

Privacy is the one composite dimension, split into five sub-scores: what data is logged (60%), incident history (20%), jurisdiction (10%), trustworthiness (5%), and privacy-enhancing features (5%). Confining credits to those small parts is deliberate. Under a flat model, a credit for accepting cash could cancel a deduction for retaining payment records, and the clamp at 10 would hide the rest, a provider that logs and one that does not could print the same perfect score.

A weighted mean is still compensatory, and privacy is a safety property, so the dimension is capped at its own logging sub-score plus 1.5. No combination of favourable jurisdiction, transparent ownership and payment options can lift a privacy score more than 1.5 points above what the logging policy alone earns. A provider retaining browsing history indefinitely cannot buy its way out of that with a Panama address.

Renormalising a score over the dimensions we happened to measure is a defensible estimate, but it is not a bound, and the old kind of review site published the estimate as if it were one. So every score carries three numbers, not one.

The only ordering the evidence licenses is this: A ranks above B when A's guaranteed minimum clears B's best possible maximum over everything we have not measured, floor(A) > potential(B). It is a strict partial order, and crucially it cannot be moved by the order in which we happen to research providers. A renormalised average could not say that.

The evidence currently separates at least one pair of providers.

There is no tiebreak. An earlier version ordered ties by how much of each provider we had researched; it has been deleted, because that promoted the providers we knew least about.

Dominance is not withholding-proof, and we will not claim it is. Deleting a whole dimension's evidence lowers the hider's floor and can never win it a rank. But deleting a single citation inside a normalized dimension can raise that dimension's value, because the divisor counts only the rules that had evidence. This is invariant 2 applied to a ratio: “absent data never lowers a score” becomes “absent data raises a ratio” the moment the denominator shrinks with it.

This is live in the current corpus, not a hypothetical. One provider scores a perfect 10 on additional features at 33% coverage: we sourced the three features it has, and its site returns an error to our fetcher for the pages that would have shown the ones it lacks. A missing page bought a perfect score.

Closing this requires the schema to tell “we looked, and the provider does not disclose”, a scored negative, cited to the page that fails to mention it, apart from “we have not looked,” which is our gap. Until then, the interval and the dominance rule close the renormalisation channel; they leave the citation channel open. The completeness gate closes it only at publication: a provider is publishable only when its scored dimensions have no unassessed gaps, so deleting a citation to inflate a score reopens a gap and blocks the page. That is a gate on what we publish, not a property of the arithmetic, and we would rather say so than overstate the guarantee.

Everything above runs on one artifact. The same corpus that renders these pages is offered for download, so you can re-derive any headline, run it under a different weighting, or find the exact citation behind any deduction.

We take no affiliate commissions, sell no advertising, and have no commercial relationship with any VPN provider.