Whitepaper

Machine-Readable Permissions

An Auditable Decision Record for Web Data Use

47-page research paper 85 min read 74 cited sources

Permission to mine web data comes from a license, a statutory exception, or a fair use defense – not from a general right to scrape. Yet ML and data engineering teams typically keep license files rather than dated, auditable records of what was checked, when, and under which rules.

This paper specifies a per-fetch decision record that integrates TDMRep signal tiers, ODRL policy resolution, SHA-256 content hashes, and a versioned rule set into a single auditable unit. An audit of 108 bibliography URLs found that 22 percent had moved, broken, or changed, and a live tool rated Elsevier's reserved TDM page 95 percent open – demonstrating that a license alone is insufficient evidence of permission.

Create a free account to read it The full essay and PDF unlock with a free, verified account.

Key ideas

The Decision, Not the License, Is the Unit of Assessment

A license attached to a work is one piece of evidence, not the answer. The dated, conditional decision a crawler makes – recording what was read, in what order, and which rule won – is the auditable unit of source-use assessment. Collapsing a conditional grant to a boolean discards exactly the information a reviewer needs.

Permission Evidence Decays and Must Be Timestamped

Sites can add TDMRep reservations, portals can move, and statutes can be amended. A verdict without a UTC timestamp and a hash of what was read cannot be checked against its evidence and therefore cannot be audited. The paper's own bibliography audit found 22 of 108 jurisdiction URLs had moved, broken, or changed by October 2026.

A Layered Signal Cascade With Explicit Read States

The record reads five signal tiers in order – robots.txt, TDMRep HTTP header, /.well-known/tdmrep.json, HTML meta tags, and asset metadata – assigning each a tri-state of read_and_found, read_and_none, or not_read. A tier read from partial input is read_incomplete, never read_and_none, and an incomplete read escalates the verdict to REVIEW.

Jurisdiction-Keyed Conflict Resolution Replaces Strictest-Rule

A single strictest-signal rule fails wherever statute overrides contract: UK section 29A(5), EU DSM Article 3 with its Article 7(1) bar, and Japan's Article 30-4 each resolve conflicts differently. Table 2 replaces the blanket rule with rows keyed on jurisdiction and purpose, and the verdict is the most restrictive result across all applicable rows.

A Categorical Gate Replaces an Ungrounded Numeric Threshold

The earlier 90 percent confidence gate had no derivation. The new gate is categorical: PASS requires every tier read in full, lawful access confirmed, and a grant covering the intended use in every applicable row; REVIEW applies when anything is unread or a conflict survives; FAIL applies when a binding prohibition holds. A reviewer signs any REVIEW verdict against the record's hash.

Test Fixtures Exposed Specification Defects Before Any Checker Ran

Encoding 88 cells from two earlier permission tables as test fixtures found four defects in the specification during authoring, including a public CC license beating a later TDMRep reservation because the record had no field distinguishing public from negotiated licenses. The suite drove successive schema and rule-set revisions from 0.2.1/1.0.1 through 0.2.3/1.0.3.