Methodology
How we measured it
Basis publishes three numbers: how often real models contain a hardcoded plug, how often our engine agrees with Excel, and how much cleaner published spreadsheets are than private ones. This page is how you check them: both corpora are public, the sample seed is fixed, the detectors are named, and the limitations are stated rather than buried.
The corpora
Two, chosen because they have nothing in common but the file format — one is a single company’s private working files, the other is whatever the open web was hosting. A finding that holds on both is unlikely to be an artifact of either.
| Corpus | Source | Files | What it is |
|---|---|---|---|
| Enron | figshare 1221767 | 15,929 | Private internal working files, recovered as legal evidence |
| FUSE | Zenodo 581678 | 249,376 | Deduplicated spreadsheets extracted from Common Crawl |
FUSE is described in Barik et al., MSR 2015. We reproduced its headline counts exactly before using it — 249,376 unique spreadsheets, 28,616 unreadable by the authors’ tooling, 205,978 with zero formula cells. If those had not matched, nothing downstream would be trustworthy.
Method
Enron was sampled: 1,000 files, uniform random, seed 1337, drawn before any file was read. 999 produced an audit. FUSE was not sampled — every one of the 10,703 .xlsx/.xltx files was processed, so its figures are population values rather than estimates.
Each workbook goes through the same command a user’s upload does. Median intake time is 2 ms; the full FUSE corpus audits in under a minute.
An independent check on the parser: Apache POI, used by the FUSE authors, and Basis agree exactly on the addressable set — both parse 10,699 of 10,703 files and both find formulas in 3,640 of them. Two independently written engines landing on identical counts is a stronger signal than either number alone.
What counts as a finding
Only high-precision detectors: a hardcoded plug (a constant sitting in an otherwise-uniform fill-down, identified by bytecode program identity — an overtyped cell is a structural discontinuity, not a heuristic match), a broken reference, and an external workbook link.
A fourth detector, formula_deviation, is excluded from every figure on this page. It flags a cell whose formula differs from its neighbours, and it over-fired on columns that were internally consistent but simply different from the column beside them — 6,571 flags on a single 110,997-formula workbook. That has been fixed, and it is still the noisier of the two, so it stays out of published statistics.
Results
Enron first, FUSE second, in each cell.
| Population | n | Plug | Broken ref | External | Any |
|---|---|---|---|---|---|
| All formula-bearing | 567 / 3,640 | 18.3% / 13.6% | 10.2% / 4.0% | 21.0% / 5.9% | 33.9% / 20.5% |
| Models (≥100 formulas) | 326 / 1,928 | 29.8% / 21.9% | 16.3% / 6.5% | 32.8% / 9.6% | 52.1% / 32.5% |
| Substantial (≥1,000) | 146 / 733 | 43.2% / 29.9% | 22.6% / 10.8% | 41.1% / 9.8% | 60.3% / 41.5% |
The rate rises monotonically with model size on both corpora. FUSE runs consistently about 1.4× lower than Enron — and that gap is itself a result, because the FUSE authors predicted it and could not test it:
“FUSE can only offer spreadsheets that are intentionally (or inadvertently) made publicly accessible, and as a result, may contain fewer errors than spreadsheets not for public dissemination.”
Measured: 29.9% vs 43.2% on substantial models. Spreadsheets people publish are cleaner than the ones they keep.
Verification against Excel
Every .xlsx stores the values Excel computed the last time it was saved, which makes each file its own test case. We recompute every formula and compare.
This figure uses FUSE only, and that restriction is the point. FUSE files were written by real Excel installations, so their cached values are Excel’s. Enron predates .xlsx — the company collapsed in 2001, the format arrived in 2007 — so its cached values were written by whatever converter the dataset publishers used. Enron cannot support a claim about matching Excel, and we do not use it for one.
Limitations
The ones that would change how you read the numbers:
- FUSE excludes large files. Common Crawl caps stored binaries at 1 MB, so the corpus structurally under-represents exactly the big models where the plug rate is highest. Its ≥1,000-formula tier is 733 files, but they are small ones.
- Enron is one company, in 2001. Its conventions are not the world’s, and the sample is 1,000 files rather than all 15,929.
- FUSE is 95.7% legacy
.xls, which Basis does not read at all. The 10,703 files measured are the.xlsxminority of a 2014 crawl. - “Formula-bearing” is doing real work as a denominator. 43% of the Enron sample and 66% of FUSE contain no formulas — they are exports and reports, not models. A file with no formulas cannot have a plug, and including them would halve every rate here.
Reproduce it
Both corpora are public and unencumbered — anyone can download them and check the file counts, the format breakdown and the formula-bearing share independently of us. The engine that produced the finding rates is not yet published, so those specific figures are ours to stand behind rather than yours to re-derive today; the audit the app runs on your own workbook is the same one, which is the nearest check available.
# FUSE — 9.4 GB, split 7z inside a zip curl -L -o fuse.zip https://zenodo.org/api/records/581678/files/fuse.zip/content bsdtar -xf fuse.zip && cat FUSE.7z.* > FUSE.7z bsdtar -xf FUSE.7z fuse-cc-binaries.tar.gz && tar -xzf fuse-cc-binaries.tar.gz # Enron — 993 MB curl -L -o spreadsheets.7z https://ndownloader.figshare.com/files/3242531 bsdtar -xf spreadsheets.7z -C enron/ # audit every workbook; audit.json carries plugs.by_kind and the # verification certificate for each file cd core && cargo build --release --bin intake ./target/release/intake <workbook.xlsx> <outdir>
Per-kind plug counts are emitted as plugs.by_kind, counted over every plug rather than the capped sample, so the split between detectors is readable from the output rather than inferred — added specifically so the split between detectors is a number you can read rather than one you have to take from us.
Want the same report for your own workbook? The audit that produced these figures is the one the app runs on any file you open.