FDE BenchmarkPublic document reader
Available in sourcev0.1 · live public draft · official release pending

Canonical source view. This renders the bundled repository document as readable HTML. Source metadata remains in the repository, and relative document links remain inside this reader.

Aggregate research-claim substantiation rules

Version: 0.1.0 Status: proposed release gate; no count currently passes Applies to: interview counts, cohort descriptions, coded findings, and claims that the benchmark is “based on” or “informed by” commissioned research

1. Fail-closed rule

A number is publishable only when it is computed from the frozen manifest of eligible participant records and bound into a signed aggregate claim receipt. An invoice, spreadsheet total, slide, email, caller total, or requestor statement cannot bypass record-level checks.

The current claims of at least 500 practitioners and 50 executives are held because no underlying records are present.

2. Count equation

For a cohort c, the eligible public participant count is:

N_c = count_distinct(participant_key)
      where primary_cohort = c
        and session_completion = complete
        and eligibility_state = eligible
        and consent.analysis = granted
        and consent.aggregate_publication = granted
        and quality_state = passed
        and lifecycle_state = active
        and duplicate_state = unique
        and dispute_state = clear
        and artifact_receipt_state = complete

The eligible count is recomputed from records on every release. The register's displayed count must equal that computation. Separate claims about sessions may use session counts but must say sessions, declare repeated participants, and never substitute for people.

3. Record exclusions

The following records never enter a public participant count:

  • unknown or duplicate participant;
  • ineligible or unverifiable primary cohort;
  • partial contact incorrectly labeled complete;
  • missing/invalid consent or use outside the consent scope;
  • withdrawn, deleted, quarantined, disputed, or superseded record;
  • missing instrument/session metadata;
  • no controlled source artifact or contemporaneous substantive note;
  • failed chain of custody or unresolved fabrication/coercion concern; or
  • synthetic/test records.

Every excluded record remains in the private inclusion-flow ledger with a reason code unless deletion is required. Public methods report counts by exclusion reason after disclosure review.

4. Required cross-record checks

Before a count receipt can pass:

  1. participant_key is unique across the study and both headline cohorts.
  2. Practitioner and executive primary cohorts are mutually exclusive.
  3. The sum of eligible headline cohorts does not exceed distinct eligible people.
  4. Session and artifact digests resolve to the controlled manifest.
  5. Every interviewer in an eligible record appears in the trained collector roster for the session date and instrument version.
  6. Every consent version resolves and permits the stated analysis/publication.
  7. Study dates contain all eligible session dates.
  8. Counts reconcile across the record store, inclusion flow, collector disposition log, frozen analysis input, and proposed publication table.
  9. Caller-level anomaly and duplicate review is complete.
  10. No unresolved withdrawal, correction, complaint, security incident, or ethics hold affects the count.

5. Claim-specific gates

5.1 “Interviewed N people”

Requires record-level and cross-record checks, transparent cohort definitions, dates, collector/sponsor disclosure, recruitment/sampling description, and exclusions. It does not require thematic coding, but it cannot imply research findings or representativeness.

5.2 “N people said X” or “X% reported Y”

Additionally requires a traceable question/code, denominator, missingness, response mode, codebook, double-coding and agreement where interpretation is involved, weighting status, uncertainty appropriate to the design, and no population inference unsupported by the sample.

5.3 “Research shows X is important to FDE work”

Additionally requires operational definitions, context and counterexamples, importance/consequence evidence rather than mention frequency alone, practitioner/executive comparisons, coding reliability, sampling limits, and independent methods review.

5.4 “The benchmark is based on the study”

Additionally requires a versioned traceability matrix from study evidence to benchmark domains/capabilities, an expert-panel record, conflicts and dissent, content-coverage review, public change decision, and benchmark version binding.

5.5 Hiring, pass-rate, accuracy, or predictive claims

Not authorized by interview evidence. Claims such as “less than 5% pass,” “accurate score,” “the real deal,” or “validated for hiring” require separately preregistered measurement and intended-use validation studies, not a larger interview count.

6. Publication thresholds for FDE-JA-001

Claim Threshold Current eligible count Result
At least 500 AI practitioners interviewed 500 distinct practitioner-primary records 0 records received at cutoff Held
At least 50 executives interviewed 50 distinct executive-primary records 0 records received at cutoff Held
At least 550 unique people across both claims 550 distinct participant keys 0 records received at cutoff Held

These zeros are evidence-intake counts, not findings about what occurred outside the repository.

7. Receipt contents

The aggregate claim receipt records:

  • study, protocol, register, instrument, consent, codebook, analysis, and benchmark versions;
  • claim text and claim class;
  • target and computed eligible counts by cohort;
  • distinct-person total, received/partial/excluded/withdrawn/held counts;
  • frozen record-manifest and analysis-output digests;
  • eligibility and deduplication policy digests;
  • data-availability class;
  • disclosure, ethics, privacy/security, measurement, independent-methods, legal/use, and governance gate states;
  • exact permitted wording and prohibited inference;
  • signer roles and timestamps; and
  • expiration/correction monitoring.

The schema permits a held receipt with failed gates. It permits a publishable receipt only when semantic invariant checks confirm all required gates, computed counts, state, and digests. JSON Schema validation alone is necessary but not sufficient.

8. Evidence strength versus sample size

  • Larger N does not repair biased recruitment, an unsuitable instrument, absent consent, unverifiable eligibility, poor coding, or missing artifacts.
  • A sample can support a transparent description of the sampled participants without supporting population prevalence.
  • Executive and practitioner cohorts answer different questions; they are reported separately before synthesis.
  • Results are contextualized by industry, company size, deployment model, geography/language, seniority, opportunity/access, and other prespecified factors where privacy permits.
  • Null, contradictory, or inconvenient findings remain in the release package.

9. Corrections

If a record is later withdrawn, duplicated, found ineligible, corrected, or affected by an incident:

  1. append the record event;
  2. recompute the eligible manifest;
  3. assess whether any released count or finding changes materially;
  4. issue a superseding receipt and corrected publication when needed;
  5. preserve the prior release and explain the difference; and
  6. remediate participant rights and downstream benchmark decisions.

No claim remains “evergreen.” The release page shows its evidence cutoff, correction status, and monitored contact.