How we check

When an article covers a release we can install, we install that exact version and run it before the article is published. A green "Verified" block at the end of the article shows what we ran, what it printed and when.

What we run

We install the release from its official source only: the official Docker image of the project, or the package from PyPI or npm, by its exact version. Then we run fixed checks: the program's own --version, the package's version and import, and, where the article gives a self-contained Python or JavaScript example, that example as it is printed in the article.

Where it runs

Every check runs in a fresh, isolated container that is deleted afterwards. The container has no network while anything from the article runs, a read-only system, no administrator rights, no access to our files and strict limits on memory, processor time and running time. Packages are installed without running their build scripts. The images we start from are pinned by their digest, which the block shows for every check.

What it does and does not show

A "Verified" check shows that this exact version installs from its official source and does what the block says: prints its version, imports, or runs the example without an error. It is not a security audit, a benchmark or a test of every feature, and an example that is only a fragment of a larger program is left out rather than shown as passing. When a release cannot be installed this way, the article has no block; we never present a check we did not run.

The Sentinel’s Call

New articles close with a green card, "The Sentinel’s Call": two to four sentences on who should act first, what to check first and what can wait. It is our judgement, drawn only from the facts the article states and links. It adds no new facts, versions or numbers, and it ranks fixes only by the severity the projects themselves give. Every sentence of it is checked against the article and its sources before the editor approves the article.

If a check or a verdict in an article looks wrong, please tell us through the contact page; see also our corrections policy.