Benchmark publication standard

Standard ID: LL-FDE-PUB-001
Version: 0.1.0
Status: proposed release-control standard; it does not authorize publication, scoring, certification, or real-data intake
Applies to: benchmark specifications, benchmark cards, method papers, protocols, study reports, data releases, result releases, corrections, retractions, and public benchmark visualizations
Evidence cutoff: September 1, 2026

1. Purpose

This standard defines the minimum public record required for a LockedIn Forward Deployed Engineering (FDE) Benchmark release to be understandable, citable, reproducible within its disclosure boundary, and honest about its authority.

It operationalizes the publication constraints already present in the benchmark specification, benchmark card, scoring architecture, governance policy, independence and institutional firewall, data governance policy, benchmark formation protocol, and Live Versus Planned register.

Publication quality and measurement authority are separate. A polished site, a citable archive, a reproducible software build, or a peer-reviewed paper does not by itself establish that an assessment is valid, fair, predictive, certified, or suitable for employment decisions.

2. Authority and adoption states

This document uses three explicit control states.

The words MUST, MUST NOT, SHOULD, and MAY describe conformance to this publication standard. They do not change the status of the benchmark or override a stricter controlling document.

2.1 Current release boundary

The current v0.1 line is a Public Draft at independence stage I0. Official numeric scoring is disabled. The proposed FDE Benchmark Commission is not seated. No structural-entity decision has been approved. Only synthetic examples are permitted in public source. No real-person assessment, verified profile, capability interpretation, tier, credential, field-validated result, certification, population norm, pass rate, predictive-validity claim, or employment-use authorization exists.

The following claims remain blocked regardless of visual or software readiness:

• validated, predictive, unbiased, independently validated, certified, accredited, industry standard, official score, or field validated;
• “based on interviews with 500 practitioners and 50 executives,” or any equivalent research-volume claim, while the study register reports zero admitted and zero publication-eligible records;
• representative, generalizable, or population-level findings without a declared sampling frame and supporting study evidence;
• independently reproduced or peer reviewed without a named reviewer, method, scope, conflicts record, and signed outcome;
• official benchmark, official tier, official credential, independently governed, or organizationally independent without the exact issuer, governance, version, authority, and activation receipts required by the institutional firewall;
• compliant with a named law, standard, or certification scheme without the required authority and evidence.

3. Publication classes and status labels

Every public object MUST declare one publication class and one status. Labels MUST be visible in the object itself, not only in metadata.

3.1 Publication classes

3.2 Status labels

Permitted publication statuses are preregistration, working_draft, public_draft, reviewed_release, corrected, retracted, suspended, and retired.

Reviewed_release is permitted only when the named review scope, reviewer independence, conflicts, date, and disposition are public. It MUST NOT be shortened to “peer reviewed” unless the work underwent a documented scholarly peer-review process that supports that phrase.

3.3 Reserved “official” terminology

“Official” is a controlled authority term, not a quality adjective. Before independence stage I3, public materials MAY use exact negative state language such as “official numeric scoring is disabled,” but MUST NOT call the project, benchmark, score, tier, credential, Commission, or publication “official” without an authorized output class and named authority.

An authorized future use MUST name the issuer and accountable legal entity; benchmark, instrument, scoring-policy, task-form, and release versions; population, context, intended use, validity period, lifecycle, verification route, limitations, and appeal route; and the Commission decision. “Reviewed release” and “project-authorized publication” do not mean independent, validated, certified, accredited, or suitable for employment decisions.

4. Front-matter contract

Every citable publication MUST expose the following fields in human-readable and machine-readable form:

1. permanent title, publication ID, publication class, status, semantic version, and language;
2. benchmark, ontology, schema, rubric, evaluator, scoring-policy, instrument, and analysis-code versions, where applicable;
3. organizational and individual authors, affiliations, ORCID identifiers where available, accountable owner, and contributor roles;
4. evidence cutoff, protocol freeze, analysis freeze, first-publication time, latest-revision time, and applicable collection dates;
5. canonical URL, version-specific persistent identifier, archive record, predecessor, successor, and supersession state;
6. funding sources, in-kind support, sponsor role, conflicts, recusals, independent-review status, independence stage, governing-body status, and structural-entity status;
7. ethics or human-participant applicability determination, consent basis, privacy classification, and data/materials availability;
8. AI-use disclosure, software/environment identifiers, license, suggested citation, and correction channel.

Missing values MUST use not assigned, not applicable — [reason], not operational, or pending — [named blocker]. Empty fields and simulated identifiers are prohibited.

5. Intended-use and construct traceability

Every benchmark publication MUST begin from a declared interpretation-and-use proposition:

For [population] in [role context], evidence produced under [conditions] may support [interpretation] for [use], subject to [limits and decision authority].

The publication MUST then identify:

• target and sampled populations;
• decision owner and affected parties;
• intended and prohibited interpretations;
• unit of measurement;
• construct definition and excluded attributes;
• adjacent-role boundaries and contextual specializations;
• consequences of a false positive, false negative, insufficient-evidence result, or misuse;
• versioned evidence supporting each link in the claim chain.

5.1 Job-analysis and content-blueprint receipt

No task, rubric criterion, critical floor, or readiness interpretation may be called job-relevant merely because an expert proposed it. For each retained requirement, the public release MUST resolve this chain:

role context
  -> job activity or work behavior
    -> consequence and criticality
      -> observable behavior
        -> work product or response
          -> assessment event and conditions
            -> rubric criterion and anchor
              -> evidence source and maturity
                -> intended interpretation

The authoritative field definition and approval gates are in the Job Analysis and Content Blueprint Protocol. A release MUST publish or provide controlled-review access to the traceability matrix, source study IDs, panel dispositions, dissent, exclusions, coverage, accessibility analysis, and change history.

Current state: the ten-domain, 55-family ontology is an adopted public-draft artifact; empirical job-analysis and content-blueprint support are not established.

6. Validity evidence strands

A publication MUST report evidence by strand rather than compressing credibility into one badge or score. The structure follows the intended-interpretation emphasis of the Standards for Educational and Psychological Testing and the work-analysis and validation expectations in the SIOP Principles. These sources inform the design; no conformance is claimed.

A citation to external research MAY justify a hypothesis or design choice. It MUST NOT be represented as local validity evidence. An evidence strand is not established until the applicable LockedIn study and release artifacts exist.

7. Study, preregistration, and source receipts

7.1 Study identity

Every empirical claim MUST resolve to a permanent study ID in the study register. The publication MUST expose:

• sponsor, commissioner, collector, analysis owner, and review owner;
• prospective preregistration, retrospective reconstruction, or exploratory status;
• protocol and amendment versions with timestamps;
• sampling frame, recruitment, eligibility, consent, incentives, modes, languages, collection dates, and nonresponse;
• participant flow from contacted to eligible to admitted to analyzed;
• instruments, codebook, coder/rater preparation, double-review design, disagreement handling, and deviations;
• dataset or controlled-data manifest, analysis code, environment, output digests, and aggregate claim receipts.

Preregistration means a time-stamped, read-only study plan posted before the declared data collection or analysis phase. The Open Science Framework registration guidance is an acceptable archival pathway, subject to privacy and governance review. A retrospective record MUST never be relabeled as a preregistration.

7.2 Source receipt

Each external source used to support a design or interpretation MUST appear in the source ledger with:

• exact title, author or issuing body, stable URL or DOI, version/status, and reviewed date;
• authority type and relevance;
• the design decision or claim it influenced;
• transfer limits, non-adoption, and contrary evidence;
• re-review trigger and current link/status check.

Sources are cited at the point of use. A bibliography without claim-level traceability is insufficient.

8. Reproducible release package

Every citable source or result release MUST have an immutable package manifest. Where rights and privacy permit, the package includes:

publication/
  README.md
  benchmark-card.md
  method-or-paper.pdf
  method-or-paper.md
  CITATION.cff
  CHANGELOG.md
  LICENSES/
  manifests/
    release-manifest.json
    source-manifest.json
    data-manifest.json
    environment-manifest.json
  schemas/
  ontology/
  instruments/
  rubrics/
  analysis/
  results/
  accessible-data-tables/
  synthetic-reproduction/
  reviews/
  corrections/

The manifest MUST bind file digests, repository commit, release tag, build/runtime versions, dependency lockfiles, random seeds where relevant, source/data cutoffs, calculation inputs and exclusions, test results, approval records, and publication status.

Private or controlled inputs MAY be withheld, but the release MUST state why, who may request access, which reproduction claims remain possible, and provide a synthetic or metadata-only reproduction path when safe.

8.1 Benchmark-operation precedents

The packaging rule is informed by operating benchmark practices, not copied from them:

• MLCommons submission rules publish version-specific package, review, audit, objection, result-status, and post-publication correction rules.
• NIST AI 800-2, Initial Public Draft emphasizes explicit evaluation objectives, implementation conditions, analysis, reproducibility, and valid interpretation.
• METR RE-Bench binds a public task repository, versions, protected evaluation material, a paper, license, and citation record.
• Artificial Analysis methodology separates definitions, benchmark-specific methods, dimensions, and current results.

These precedents concern AI and system benchmarking. They do not validate a human occupational assessment, and their scores, weights, status labels, or visual trade dress are not adopted.

9. Statistical and result reporting

Every quantitative result MUST publish, beside the result:

• estimand or exact quantity being estimated;
• denominator, eligibility rules, exclusions, missingness, invalid runs, and censoring;
• measurement conditions, assistance, environment, rater/evaluator versions, and time window;
• central estimate and an uncertainty representation appropriate to the design;
• multiplicity, researcher degrees of freedom, model assumptions, and sensitivity analyses;
• subgroup or contextual variation only where rights, sample sufficiency, and disclosure controls permit;
• comparison and linking rules, including why cross-version comparison is or is not supported;
• limitations and plausible alternative explanations.

Official v0.1 numeric human scoring is disabled. Synthetic experimental values MUST be labeled EXPERIMENTAL SYNTHETIC OUTPUT — NOT AN OFFICIAL RESULT and may not be presented as evidence of a real person or population.

Missing evidence yields Insufficient Evidence, not zero. Evidence maturity, capability, support confidence, coverage, verification, integrity, and lifecycle remain separate.

10. Public metrics contract

The public site SHOULD expose a compact release-health panel. Each metric MUST resolve to a machine-readable receipt and exact as-of date.

Until empirical work exists, the release-health or transparency record MUST show zero, not established, not operational, or blocked rather than omit the metric or substitute a roadmap target. Absences MUST NOT be promoted as proof of rigor, framed as achievement metrics, or selected as a landing-page hero value proposition. A public-facing summary may lead with what the released method or artifact enables, provided the exact status remains one direct interaction or link away.

11. Authorship, funding, conflicts, ethics, and AI use

11.1 Author and contributor accountability

Authorship requires substantive intellectual accountability for the released work. Contributions SHOULD be reported using the CRediT taxonomy, including the applicable roles among conceptualization, methodology, software, validation, formal analysis, investigation, data curation, visualization, writing, supervision, funding acquisition, resources, and project administration.

The release MUST name:

• the guarantor accountable for the complete record;
• who had access to identifiable, controlled, or outcome data;
• who approved the final analysis and public wording;
• who is independent of the sponsor, operator, trainer, or evaluated party;
• who controlled application, training, staffing/recruiting, evidence custody, rating, calculation, issuance, appeal, and publication, where applicable;
• material contributor disagreements or recusals.

11.2 Funding and sponsor role

The funding statement MUST identify direct funding, in-kind contributions, infrastructure credits, commissioned collection, and uncompensated material support. It MUST explain the sponsor’s role in design, collection, analysis, interpretation, writing, publication, and access to data. “No external funding” and “funding information not yet finalized” are different statements and MUST not be interchanged.

11.3 Conflicts of interest

The publication MUST disclose which LockedIn Labs roles actually apply to the release or affected population, including founder, funder, steward, operator, employer, trainer, staffing or recruiting provider, services provider, platform provider, assessor, or issuer. It MUST distinguish current, prior, contemplated, and non-applicable roles. Training, employment, placement, payment, sponsorship, friendship, customer status, or platform use cannot supply benchmark credit. A conflicted author may contribute but may not provide the sole approval for the affected result.

For any future official human release, the publication MUST name the external-majority Commission, chair, eligible and recused voters, protected authority, role-separation receipt, funding and sponsor controls, appeals authority, structural-entity decision, and residual affiliate conflict. External review alone does not establish independent governance or validation.

11.4 Ethics, participant rights, and legal applicability

For human-participant or employment-adjacent work, the publication MUST identify the authority that determined review, exemption, non-applicability, or other governance status. It MUST NOT infer approval from silence. Consent, withdrawal, correction, accommodation, data rights, retention, and incident paths must be operational before intake.

11.5 AI-use disclosure

The publication MUST state which AI-assisted systems contributed to discovery, drafting, coding, analysis, translation, visualization, or review; which model/provider and material settings were used where reproducibility and confidentiality permit; what inputs were restricted; and how humans verified the output.

AI systems are tools, not authors, evidence sources, independent reviewers, or final decision authorities. The accountable human authors retain responsibility for citations, code, calculations, confidentiality, and claims.

12. Citation, DOI, and archive readiness

A citable release MUST include:

• a valid CITATION.cff and BibTeX entry;
• organization and individual creator metadata;
• title, abstract, keywords, version, date, language, license, repository URL, and related identifiers;
• a canonical landing page with current status and links to prior and later versions;
• a version-specific persistent identifier for exact reproducibility;
• an optional concept identifier for the evolving benchmark family;
• an archived immutable release package and checksums.

Zenodo DOI versioning distinguishes a version DOI from a concept DOI; citations to empirical results SHOULD use the exact version DOI. Zenodo’s GitHub integration and GitHub’s citation guidance describe one potential archive path. Selection of an archive requires license, privacy, security, longevity, and governance review.

Current v0.1 state: canonical publication decision, DOI, archival deposit, individual authorship record, and operative licenses remain blockers. No placeholder DOI may be displayed.

13. Corrections, suspension, retraction, and retirement

Every publication MUST provide a monitored correction channel and a materiality policy.

Retraction and correction practice SHOULD be reviewed against the COPE Retraction Guidelines, adapted to a benchmark and software release. Use of the guideline is not a COPE membership or conformance claim.

Material records are append-only. Search, citation, API, downloadable artifact, and public status surfaces MUST converge on the corrected status. Retraction removes authority, not history.

14. Release criteria

14.1 Public-draft source release

A v0.1 public-draft source release may proceed only when:

• status and prohibited-use language match the controlling documents;
• official numeric scoring remains disabled;
• every person, artifact, deployment, observation, and result example is synthetic and labeled;
• specification, benchmark card, ontology, schemas, tests, source manifest, changelog, and status register are version-consistent;
• all objective claims resolve to repository evidence or are labeled proposed/blocked;
• source links, Markdown structure, build, schema, and deterministic fixture checks pass;
• privacy, security, accessibility, license, trademark, conflict, and publication blockers are visible;
• no outside endorsement, review, study, or participant count is implied;
• an authorized release decision names the exact commit and artifact set.

14.2 Citable archived public-draft release

In addition to section 14.1:

• individual authors and contributor roles are approved;
• funding, conflicts, AI use, data/materials, and review statements are complete;
• operative licenses and third-party rights review are complete;
• canonical URL, CITATION.cff, archive deposit, version DOI, checksums, and preservation owner are operational;
• correction, security, and private conduct channels are monitored;
• the archive reproduces the approved source manifest.

14.3 Empirical findings release

In addition to sections 14.1 and 14.2:

• study state is publication_authorized in the study register;
• protocol timing, amendments, sample flow, consent, inclusion/exclusion, missingness, and limitations are disclosed;
• counts reproduce from an eligible-record manifest and signed aggregate receipt;
• code, environment, outputs, uncertainty, and deviations reproduce under the declared access boundary;
• privacy, disclosure, methods, measurement, accessibility, conflict, and independent-review decisions are signed;
• exact permitted interpretations and prohibited inferences are approved.

14.4 Official assessment or certification release

This class is blocked. It requires a later benchmark version and every activation gate in SCORING.md, plus:

• independence stage I3 under INDEPENDENCE_FIREWALL.md;
• a seated external-majority Commission with external-majority quorum for the exact decision;
• an operating separation of training, employment, staffing/recruiting, sales, evidence custody, rating, issuance, appeal, and publication roles;
• a published structural-entity decision with qualified legal, measurement, privacy/security, and credentialing review;
• operational identity, consent, correction, appeal, privacy, security, accessibility, measurement, impartiality, consequence, and legal/use controls; and
• the exact issuer, accountable legal entity, approved output class, version, scope, validity period, lifecycle, verification, and public claim receipt.

Software readiness, external review, an outside advisor, a separate entity, or an internal approval alone cannot satisfy this gate.

15. Publication review record

The final release decision MUST record:

• exact commit, tag, archive, manifest, and artifact digests;
• publication class, status, intended use, and prohibited use;
• completed reviews, reviewers, scope, independence, conflicts, findings, and unresolved dissent;
• independence stage, Commission composition and quorum, role-firewall receipt, funding control, and structural-entity decision where required;
• every waived non-blocking issue and rationale;
• all blocking issues and final disposition;
• correction and monitoring owners;
• decision authority, time, and signatures.

No single author may serve as sole methods reviewer, claims reviewer, conflict reviewer, and release authority for an empirical or high-consequence release.

16. Reference posture

The benchmark’s current reference base is a targeted design review, not a systematic review or proof of validity. The source ledger is the authoritative record of reviewed versions, influence, departures, and review triggers. The publication template governs individual research outputs. This standard governs the release system around them.