FDE BenchmarkPublic document reader
Available in sourcev0.1 · live public draft · official release pending

Canonical source view. This renders the bundled repository document as readable HTML. Source metadata remains in the repository, and relative document links remain inside this reader.

Benchmark evolution protocol

Protocol ID: LL-FDE-EVO-001
Version: 0.1.0
Status: proposed lifecycle protocol; no official scoring or operational assessment authority
Applies to: construct, ontology, content blueprint, tasks, evidence rules, rubrics, scoring policy, evaluator, public profiles, domain packs, publications, and release infrastructure
Evidence cutoff: September 1, 2026

1. Objective

The LockedIn FDE Benchmark must evolve as Forward Deployed Engineering work, AI systems, enterprise controls, evidence availability, and measurement knowledge change. Evolution must not silently change what an existing result means.

This protocol defines how the project discovers change, evaluates evidence, classifies compatibility, approves a new release, preserves old records, corrects defects, and retires unsupported interpretations.

It is subordinate to the specification, governance policy, scoring architecture, benchmark card, and Live Versus Planned register.

2. Evolution principles

  1. Purpose before metric. A release begins with a population, role context, intended interpretation, and use—not with a desired score or leaderboard.
  2. Empirical bridge before authority. New content, weights, thresholds, and result labels remain hypotheses until the required local evidence exists.
  3. Immutable releases. Published release bytes and their interpretation are never overwritten.
  4. Version-separated results. Results from different benchmark versions are not visually or analytically merged without a published linking study.
  5. No prestige shortcuts. Vendor, school, employer, sponsor, training, platform, title, fame, and social reach do not become capability inputs through a revision.
  6. Failures are evidence. Invalid tasks, held outputs, rater disagreement, accessibility barriers, appeals, contamination, and null findings influence evolution.
  7. Protection without obscurity. Controlled cases may remain secret, but release authority, categories of controls, limitations, and correction outcomes remain public.
  8. Reversibility and recourse. Material defects can suspend a task or result family, trigger replay, and provide participant correction and appeal.
  9. Contextual expansion. New industries, geographies, languages, seniority levels, or adjacent roles require explicit evidence; transfer is not assumed.
  10. Software completion is not validation. Passing code and schema tests supports implementation integrity only.

3. Evolution objects

Every change proposal MUST identify the object and current version it affects.

Object Examples Required authority
Construct and intended use FDE definition, target population, prohibited use Major benchmark decision plus measurement review
Ontology Domain, capability family, competency, atomic signal Ontology release decision and traceability evidence
Content blueprint Job activity, observable behavior, task coverage Job-analysis/content-panel decision
Instrument Scenario, prompt, environment, assistance, timing, accommodation Instrument release and security/accessibility review
Rubric and rater protocol Criterion, anchor, reviewer qualification, adjudication Measurement and evaluation-integrity review
Evidence and verification Maturity, provenance, reference, identity, contribution attribution Privacy, security, measurement, and governance review
Scoring and decision policy Eligibility, aggregation, uncertainty, floors, thresholds Later official-scoring release only
Evaluator or grader Provider, model, prompt policy, tool, release gate Evaluator release decision; no independent score authority
Publication or data product Chart, table, explorer, paper, dataset Publication standard and disclosure review
Domain or role pack Healthcare, security, modernization, adjacent AI role Separate construct, method, evidence, and release record
Credential scheme Issuer, eligibility, decision, status, renewal Separate impartial certification decision; blocked in v0.1

4. Change triggers

An evolution review opens when any of the following occurs:

  • a material FDE job activity, work product, technology, operating boundary, or failure mode changes;
  • new job-analysis, content, response-process, reliability, fairness, criterion, or consequence evidence is admitted;
  • a task, rubric, evaluator, scoring rule, or verification method shows instability, construct contamination, or accessibility burden;
  • contamination, leakage, gaming, or benchmark-specific preparation threatens interpretation;
  • appeal, correction, incident, rater-drift, or user-research patterns identify a recurring defect;
  • an external standard, regulator source, legal requirement, benchmark precedent, or provider condition changes;
  • a domain pack, geography, language, seniority, employment use, or adjacent role is proposed;
  • a source, license, dependency, archive, or provider is withdrawn or materially revised;
  • scheduled review reaches its due date.

Each trigger receives a stable issue or decision ID, discovery time, reporter, affected versions, provisional severity, and owner. Sensitive triggers use a protected record plus a public non-sensitive status.

5. Evidence intake for change

The proposal owner MUST distinguish the evidence class supporting the change:

  • professional standard or regulator guidance;
  • peer-reviewed research;
  • operating benchmark precedent;
  • job-analysis or participant research;
  • subject-matter-expert review;
  • steward/product experience;
  • public comment or external review;
  • synthetic engineering, accessibility, security, or red-team evidence;
  • operational monitoring, appeal, incident, or consequence evidence.

The source class determines what it can establish. A citation may motivate a design hypothesis; it does not supply local validity. Steward experience may identify a failure mode; it does not establish representativeness. Synthetic tests may establish deterministic software behavior; they do not establish human-measurement quality.

Every source MUST resolve to the source ledger, a registered study, an incident record, or a versioned design memo with conflicts and limitations.

6. Change classification and semantic versioning

6.1 Release identifiers

Benchmark releases use semantic identifiers MAJOR.MINOR.PATCH plus an optional prerelease label such as 0.2.0-draft.1.

The version applies to the benchmark release as a coherent bundle. Ontology, schemas, instrument, rubric, evaluator, and scoring policy retain their own versions and are bound in the release manifest.

6.2 Change classes

Class Typical change Result compatibility Required release
Editorial Typo, formatting, link repair, non-substantive clarification Meaning unchanged Patch or timestamped documentation correction
Corrective Defect repair that changes affected source bytes or calculation but not intended construct Old result may require impact review Patch with correction notice and replay decision
Additive compatible New optional metadata, synthetic fixture, proposed content, or non-authoritative view Existing interpretation unchanged Minor release
Measurement material Task, rubric, evidence eligibility, assistance, rater, uncertainty, weighting, floor, or threshold change Comparison unsupported until linking evidence Minor or major; version-separated results
Construct or use material Population, role definition, domain boundary, intended use, critical requirement, result meaning Incompatible Major release
Security/integrity emergency Exposure, contamination, unsafe provider, invalid task, unauthorized data Affected authority suspended Emergency patch or suspension followed by normal review
Retirement No longer supported or interpretable Historical only Retirement notice; no overwrite

A change is material if a reasonable user could make a different decision, interpret a capability differently, face a different burden, receive a different result state, or encounter a different privacy/security risk.

Patch numbering MUST NOT be used to disguise a material construct or measurement change.

6.3 Cross-version comparability

Result comparison across versions is prohibited by default. A linking study MUST declare:

  • populations, samples, task forms, conditions, raters, evaluator versions, and time period;
  • construct overlap and content changes;
  • missingness, order, practice, drift, and contamination effects;
  • linking/equating method, assumptions, uncertainty, sensitivity, and failure criteria;
  • ranges or groups for which comparison is unsupported;
  • governance decision and expiration/review trigger.

Absent this evidence, the interface MUST display versions separately and MUST NOT draw a continuous improvement trend.

7. Evolution workflow

E0 — Observe and register

Open a change record with the trigger, affected population/use, evidence class, conflicts, possible harm, urgency, and current workaround.

E1 — Triage and contain

Classify severity and materiality. If a defect could produce an invalid or harmful interpretation, suspend the affected task, evaluator, result, chart, or claim while impact is assessed. Suspension is not evidence of guilt or low capability.

E2 — Draft the change proposition

Write an RFC that states:

  • current behavior and problem;
  • proposed normative change and alternatives, including no change;
  • construct, measurement, fairness, accessibility, security, privacy, legal/use, gaming, and operational effects;
  • evidence and gaps;
  • versioning, migration, replay, and rollback plan;
  • validation plan and decision criteria;
  • contributors, funding, conflicts, and recusals.

E3 — Public and specialist review

Material normative proposals receive the public-review period defined in GOVERNANCE.md, plus technical, measurement, accessibility, fairness, security, privacy, legal/use, licensing, and conflict review as applicable. Controlled-case details may be reviewed privately by authorized, non-conflicted reviewers.

Review records MUST preserve substantive objections, response, disposition, and steward departures. Review volume is not a vote unless the governance policy says so.

E4 — Prototype and study

Implement the proposal behind a version boundary. Test schemas, invariants, deterministic recomputation, known-good/insufficient/invalid fixtures, accessibility, security, contamination, and migration behavior.

When the change affects human interpretation, execute the applicable job-analysis, response-process, rater, fairness, criterion, consequence, or linking study. A successful synthetic test cannot advance the human evidence state.

E5 — Freeze candidate

Freeze source, ontology, schemas, instruments, rubrics, evaluator, scoring policy, datasets, code, environment, and documentation. Generate manifests and digests. No result-sensitive tuning may occur after the analysis freeze without an amendment and a new candidate.

E6 — Reproduce and decide

Run the release package from a clean environment. Where a claim of independent reproduction is intended, an authorized party outside the implementation line reproduces the declared outputs and signs the scope, deviations, and outcome.

The release authority records accept, revise, defer, reject, suspend, or retire. A calendar date cannot waive a blocker.

E7 — Publish and migrate

Publish the immutable package, changelog, migration guide, compatibility statement, limitations, citation, archive, and correction route. Mark predecessor and successor states. Recalculate or re-review prior results only under a predeclared impact plan.

E8 — Monitor and revalidate

Monitor the release against its declared cadence and triggers. Record drift, task health, rater behavior, invalid runs, accessibility burden, contamination, appeals, consequences, use outside scope, and source changes.

8. Release evidence map

Every material release MUST include a change-to-evidence matrix.

Field Required content
change_id Stable identifier
affected_object_ids Exact benchmark, ontology, schema, task, rubric, evaluator, or policy versions
trigger Observation, study, incident, source revision, appeal, or scheduled review
intended_population_and_use Population, context, interpretation, decision
change_class Editorial, corrective, additive, measurement, construct/use, emergency, retirement
evidence_receipts Source, study, test, review, incident, and reproduction records
claims_enabled Exact newly permitted wording, if any
claims_withdrawn Exact wording no longer permitted
compatibility Compatible, conditionally linked, or version-separated
impact_scope People, results, publications, APIs, charts, credentials, and customers affected
accessibility_and_fairness Barriers, alternate demonstrations, subgroup/error evidence, unresolved limits
migration_and_replay Data, result, participant, and interface handling
decision Authority, date, vote/approval, recusals, dissent
monitoring Metrics, thresholds, owner, next review, suspension criteria

9. Cadence

Cadence creates a review opportunity, not automatic change.

Cadence Review scope
Continuous Security/privacy incidents, broken links, source withdrawals, contamination, material grader/task defects, unauthorized use
Monthly Public-site claims, publication status, source availability, dependency/provider changes, correction queue
Quarterly Source ledger, market landscape, open RFCs, study register, conflicts/funding, accessibility defects, task/evaluator health when operational
Semiannual Ontology coverage, job/technology drift, reviewer/rater drift, controlled-case rotation, accommodation evidence, domain-pack demand
Annual Benchmark card, intended population/use, validity argument, full limitations review, governance capacity, archive/license state, retirement candidates
Major release Complete construct, content, measurement, fairness/accessibility, consequence, security/privacy, legal/use, reproducibility, and governance review

The next review date and owner MUST appear in each active release manifest. A missed review changes the status to review_due; it does not silently validate the unchanged release.

10. Monitoring metrics and action rules

Before operational use, each metric and threshold remains proposed. Once authorized, release-specific action rules MUST be fixed before outcomes are examined.

Monitor at least:

  • construct and content: uncovered critical incidents, new work products, context gaps, task-to-blueprint coverage;
  • instrument: completion, invalid/underdetermined tasks, assistance deviations, burden, alternate-mode use, solver failures;
  • raters: agreement/reliability, drift, severity, disagreement classes, adjudication, conflict routing;
  • evaluators: version stability, schema failures, unsupported citations, automation bias, prompt injection, provider changes;
  • results: missingness, insufficient evidence, uncertainty, critical-gate patterns, corrections, reversals, appeals;
  • fairness/accessibility: mode and accommodation performance, construct-irrelevant barriers, subgroup/error differences, complaints;
  • integrity: leakage, contamination, anomalous behavior, hidden-case exposure, correlated evidence, reference fraud;
  • consequences: employer misuse, participant burden, privacy events, false-decision signals, strategic behavior;
  • operations: reproducibility, manifest drift, dependency/source availability, incident response, archive resolution.

Thresholds MUST specify whether they trigger investigation, hold, suspension, correction, revalidation, or retirement. A concealed canary cannot be the sole basis for an adverse capability or integrity conclusion.

11. Emergency procedure

A security, privacy, legal, integrity, or measurement emergency may bypass the normal review period only to contain harm.

Emergency action MUST:

  1. identify exact affected versions and surfaces;
  2. use the narrowest viable suspension or access restriction;
  3. preserve evidence and audit history without copying raw private data into logs;
  4. receive approval from at least two authorized, non-conflicted people where feasible;
  5. publish a non-sensitive notice when safe and lawful;
  6. begin impact analysis and participant notification where applicable;
  7. enter normal review within 30 days;
  8. issue a patch, correction, retraction, replay, or restoration decision.

Emergency authority cannot silently change a score, construct, threshold, or participant history.

12. Correction, replay, and participant impact

For a material defect, the decision record MUST answer:

  • When was the defect introduced, detected, contained, and corrected?
  • Which versions, tasks, raters, evaluators, results, profiles, publications, and users are affected?
  • Can the result be recomputed from frozen inputs, or is re-administration/re-review required?
  • Would replay introduce unfair practice, memory, availability, or opportunity effects?
  • Which participants need notice, correction, appeal, re-evaluation, deletion, or other remediation?
  • Which public claims, charts, citations, and downstream exports must be corrected?
  • Is the replacement comparable to the prior release?

The old record remains discoverable with a prominent corrected, suspended, retracted, or retired status.

13. Domain packs and adjacent roles

A domain pack cannot inherit validity merely by mapping to the core ontology. It requires:

  • defined population, work context, critical incidents, work products, and consequences;
  • expert and participant coverage appropriate to that context;
  • task, evidence, rubric, accommodation, and verification analysis;
  • local validity, fairness, uncertainty, and consequence evidence for its intended use;
  • version, release package, limitations, and monitoring owner.

An adjacent AI role—such as solutions architecture, applied AI engineering, implementation, technical program leadership, or AI operations—requires a separate construct decision and boundary study. It is blocked until the FDE core reaches its validation gates and governance authorizes expansion.

14. Retirement

A release, task, metric, domain pack, evaluator, or interpretation SHOULD be retired when:

  • the work no longer represents the declared role;
  • contamination or gaming cannot be adequately controlled;
  • reliability, fairness, accessibility, or consequence evidence no longer supports use;
  • a dependency, license, provider, or archive becomes unavailable;
  • maintenance or governance capacity is insufficient;
  • a successor provides a materially better supported interpretation.

Retirement MUST publish the effective date, reason, affected uses, last supported version, successor if any, archive, and whether historical results retain descriptive validity. Retirement does not delete correction, appeal, consent, retention, or legal obligations.

15. Current evolution state

As of the evidence cutoff:

  • the public draft contains a versioned ten-domain, 55-family ontology and initial schemas;
  • official numeric scoring, thresholds, pass rates, role-readiness bands, real-person results, and credentials are disabled or blocked;
  • no admitted job-analysis cohort, expert panel, reliability study, fairness study, criterion study, linking study, or longitudinal outcomes study exists;
  • the draft source manifest supports source-byte traceability but is not an authorized immutable publication;
  • external governing and measurement bodies are proposed, not seated;
  • changes may improve the public draft, but no change can activate an official human interpretation without the full evidence and governance path.

The Credibility Evidence Register records the evidence state that constrains the next release. The Benchmark Publication Standard defines the package and disclosure record for each release.