Benchmark formation protocol and evidence map
Protocol version: 0.1.0
Source-review cutoff: August 31, 2026
Status: public design protocol; no completed job analysis, validation study, official score, or certification claim
This protocol defines what the LockedIn Forward Deployed Engineering (FDE) Benchmark is, which inputs may shape it, how those inputs must be traced into design decisions, and which evidence is required before a public interpretation can become authoritative.
It exists to prevent four common failures:
- treating a skills list as a benchmark;
- treating citations as proof that this instrument is valid;
- treating product or founder experience as research data;
- treating polished software as authorization to score or certify people.
1. Formal definition
The LockedIn FDE Benchmark is a versioned, purpose-bound assessment system for collecting, reviewing, and interpreting evidence of demonstrated Forward Deployed Engineering capability in a defined role context.
A complete benchmark release binds all of the following:
- target population and context — who and which role conditions the interpretation covers;
- intended interpretations and uses — the claims a result may support and the decisions it may inform;
- construct model — the work, capabilities, boundaries, and excluded attributes being assessed;
- content blueprint — the relationship between job activities, observable behaviors, tasks, artifacts, and rubric criteria;
- administration and evidence protocol — conditions, tools, assistance, accommodations, identity, provenance, security, and missingness rules;
- measurement and decision rules — how reviewed observations become a profile, how uncertainty and coverage are represented, and how critical gates behave;
- quality evidence — content, response-process, reliability, fairness, criterion, consequence, and generalizability evidence for the stated use; and
- governance and release record — ownership, conflicts, review, versioning, corrections, appeals, retirement, and reproducible artifacts.
If one of these elements is absent, the project may still be a useful capability framework, research prototype, task library, or validation protocol. It is not yet a validated benchmark for that use.
This definition is informed by the Standards for Educational and Psychological Testing, the SIOP Principles for the Validation and Use of Personnel Selection Procedures, ISO/IEC 17024:2026, and current benchmark-documentation and evaluation practice. It is an adaptation for this project, not a conformance claim.
2. Keep five artifacts distinct
| Artifact | What it establishes | v0.1 public-draft state |
|---|---|---|
| Capability framework | Vocabulary and boundaries for FDE work | Released ontology: 10 domains and 55 capability families |
| Benchmark specification | Intended use, evidence rules, state model, and minimum implementation requirements | Available as a public draft |
| Assessment instrument | Standardized tasks, prompts, conditions, artifacts, and rubrics | Proposed and represented with synthetic prototypes only |
| Measurement model | Rules and evidence supporting interpretation, uncertainty, reliability, comparability, and decisions | Experimental software exists; no calibrated human model or official score |
| Certification scheme | Identity, eligibility, assessment decision, impartiality, certificate lifecycle, surveillance or renewal, complaints, and appeals | Not operational; no certification or accreditation claim |
Training is a sixth, separate product. Course completion, payment, employment by LockedIn Labs, or use of an affiliated platform cannot supply benchmark credit.
3. Intended interpretation and current authority
The current construct is demonstrated FDE capability: evidence that a practitioner can frame, engineer, deploy, operate, defend, and improve consequential technical change across a customer or enterprise boundary.
The current release may support:
- public critique of the construct and ontology;
- development of job-analysis, task, rubric, and validation protocols;
- synthetic evaluation and software-contract testing; and
- research planning.
It does not support a score, pass rate, readiness band, percentile, comparison population, employment decision, verified profile, field-validated designation, or certification.
4. Admissible formation inputs
Every input belongs to one class. Its class controls what it may establish.
| Class | Required record | What it may influence | What it cannot establish by itself | Current state |
|---|---|---|---|---|
| Professional standards and regulator guidance | Version, authority, reviewed date, applicable concept, departure, and review trigger | Intended-use boundaries, validation program, fairness, documentation, and governance requirements | Conformance, legal compliance, or validity of this benchmark | Dated sources recorded in the source ledger |
| Peer-reviewed research | Complete citation or DOI, population, method, result used, limitation, and non-transfer statement | Hypotheses about task formats, raters, references, measurement, and reporting | A validity, reliability, or fairness coefficient for this instrument | Selected literature recorded; no systematic-review claim |
| Operating benchmark precedents | First-party method/rules, version, observed practice, and adaptation boundary | Reproducibility, submission, audit, status, correction, and publication architecture | Validity of a human capability construct | Selected methods recorded; no endorsement claim |
| Practitioner and executive research | Frozen protocol, sampling frame, recruitment, consent, fieldwork, deidentified records, coding, quality controls, and aggregate receipt | Job activities, critical incidents, language, scope, content blueprint, and subgroup/context questions | Population prevalence, validation, or pass rates without the required study design | Zero admitted observations; the 500/50 figures remain an unverified target |
| Subject-matter-expert panel | Selection rules, qualifications, affiliations, conflicts, prompts, ratings, dissent, and decision record | Content relevance, representativeness, criticality, rubric anchors, and blueprint coverage | Criterion validity, fairness, or reliability without empirical evidence | Not convened |
| Steward and product experience | Named contributor, role, dated design memo, examples, conflicts, and explicit hypothesis label | Research questions, scenarios, failure modes, and candidate design proposals | Independent research evidence, representativeness, or validation | Not yet published as an auditable experience register |
| Public comment and external review | Versioned submission, contributor consent, conflict disclosure, disposition, and change record | Error correction, missing perspectives, construct challenge, and governance | Independent validation unless the reviewer and method meet a declared protocol | Not operational |
| Synthetic engineering and red-team evidence | Fixture version, expected behavior, environment, code, result, and defect record | Schema integrity, determinism, failure handling, privacy boundaries, and evaluator invariants | Human measurement validity or real-world capability | Available in source for selected components |
No input gains authority because it is numerous, expensive, commissioned, senior, familiar, or favorable. Minority and contradictory findings remain in the record.
5. Source-to-design traceability
The benchmark must maintain a claim-to-evidence map at the level where a design decision can be challenged.
| Design decision | Current basis | Required empirical bridge | Permitted v0.1 claim |
|---|---|---|---|
| FDE construct and 10-domain boundary | Steward synthesis, market observation, professional frameworks, and public literature | Job/practice analysis across declared FDE contexts; expert review; adjacent-role discriminant analysis | Proposed vendor-neutral construct |
| 55 capability families | Released ontology and internal design rationale | Trace each family to critical incidents, work products, importance/frequency/consequence ratings, and panel disposition | Released public-draft ontology, not a validated content domain |
| Hands-on task sequence | Work-sample literature, evidence-centered design, job-relevance guidance, and benchmark precedents | Blueprint review, task solvability, response-process study, accommodations, and representative pilot | Proposed applied-observation architecture |
| E0–E5 evidence maturity | Internal ordinal taxonomy informed by provenance and field-evidence needs | Comprehension, classification consistency, incremental-utility, and consequence studies | Descriptive ordinal evidence-link state; not a score |
| Human review and structured defense | Selection-interview and rater-reliability literature plus governance requirements | Rater training, double review, disagreement analysis, drift, decision consistency, and automation-bias study | Proposed review protocol |
| Identity, outcome, and reference corroboration | Identity guidance, reference research, privacy analysis, and fraud threat model | Burden, false-match, accessibility, privacy, contribution-attribution, and incremental-validity studies | Separate future verification inputs, not capability proxies |
| Multidimensional profile and critical gates | Intended-use analysis and experimental deterministic scorer | Calibration, uncertainty, missingness, floor/compensation, sensitivity, and standard-setting studies | Experimental synthetic projection only |
| Employment or certification use | Testing standards, EEOC guidance, ISO/IEC 17024 design input, and conflict policy | Role-specific validation, fairness/adverse-impact and accommodation evidence, operational impartiality, appeals, security, and independent review | Explicitly prohibited in v0.1 |
The detailed external-source record remains in the research and standards source ledger. The source ledger states influence and non-adoption; it does not silently convert a citation into proof.
6. Formation workflow
F0 — Declare the decision
Freeze the intended population, role context, interpretation, users, consequences, and prohibited uses before instrument design.
Required artifacts: benchmark card, use-case register, risk classification, conflict declaration.
F1 — Analyze the work
Collect critical incidents, work products, decisions, contexts, consequences, and adjacent-role boundaries using a predeclared job/practice-analysis protocol.
Required artifacts: protocol, sampling frame, instrument, participant ledger, coded corpus, negative cases, subgroup/context coverage, and aggregate receipts. The proposed operating record is Job Analysis and Content Blueprint Protocol.
F2 — Build the content blueprint
Trace construct claims to job activities, observable behaviors, task conditions, retained artifacts, rubric criteria, and coverage requirements. Record dissent and excluded content.
Required artifacts: traceability matrix, panel roster and conflicts, rating procedure, content-validity analysis, and approved blueprint version.
F3 — Engineer and pilot the instrument
Develop standardized tasks, administration rules, tool/assistance conditions, alternate forms, accommodations, security controls, contamination controls, and known-good/invalid cases.
Required artifacts: task packages, administration manual, solver record, accessibility review, threat model, pilot deviations, and item-change history.
F4 — Build the validity argument
Pre-register the propositions a result must support and the evidence needed for each inference: scoring, generalization, extrapolation to role performance, and decision use.
Required artifacts: analysis plan, rater study, response-process evidence, measurement-error and decision-consistency results, subgroup/fairness analysis, criterion evidence, and consequence monitoring plan.
F5 — Set decision rules
Only after the preceding evidence exists, use an explicit standard-setting method to define any thresholds, floors, compensation rules, insufficient-evidence behavior, or result classes. Publish sensitivity and uncertainty around the decision boundary.
Required artifacts: standard-setting protocol, panel record, data, calculation code, uncertainty, sensitivity, and approved interpretation language.
F6 — Review, reproduce, and release
Freeze the benchmark version and complete technical, measurement, fairness/accessibility, security/privacy, legal/use, conflict, and publication reviews. An independent party must be able to reproduce selected results from the release package before a result is labeled independently reproduced.
Required artifacts: signed decision record, immutable manifest, checksums, reference implementation, reproducibility report, corrections channel, archive, and retirement policy.
F7 — Monitor and revalidate
Monitor drift in the role, tasks, raters, tools, subgroup outcomes, consequences, contamination, appeals, and actual use. Material change or unsupported use triggers revalidation, version separation, suspension, or retirement.
7. Validity and quality evidence register
The project must report these dimensions separately. A green software build cannot satisfy them.
| Evidence family | Question | Minimum public evidence before a validated claim | v0.1 state |
|---|---|---|---|
| Content | Does the benchmark represent important FDE work for the declared role context? | Job analysis, content blueprint, panel method, ratings, dissent, coverage, and change record | Not established |
| Response process | Do participants and reviewers engage the intended capability rather than irrelevant strategy or interface friction? | Cognitive interviews, think-aloud or equivalent evidence, accessibility and accommodation evaluation, protocol-deviation analysis | Not established |
| Internal structure and reliability | Are observations and decisions sufficiently consistent for the claimed interpretation? | Rater agreement/reliability, drift, task/form effects, decision consistency, uncertainty, missingness, and adjudication analysis | Not established |
| Relations to external criteria | Does the result relate to later, independently measured work outcomes as proposed? | Pre-registered criteria, temporal separation, attribution controls, missingness, alternative predictors, confidence intervals, and replication | Not established |
| Fairness and accessibility | Are construct-irrelevant barriers identified and reduced, and are alternate demonstrations comparable? | Accessibility study, accommodation process, subgroup/error analysis, adverse-impact monitoring, qualitative feedback, and remediation record | Not established |
| Consequences and use | What happens when the benchmark is used, misused, appealed, or wrong? | Burden, false-decision, gaming, privacy, appeal, employer-use, and unintended-consequence monitoring | Not established |
| Reproducibility and security | Can a qualified reviewer reproduce the result under the same release while protecting controlled material? | Frozen inputs, environment, code, logs, hashes, contamination controls, independent replay, and correction history | Partially implemented with synthetic artifacts only |
8. Publication package and visible credibility signals
A release described as citable or official must expose, as applicable:
- title, organizational and individual authors, affiliations, ORCIDs, and contributor roles;
- version, release date, evidence cutoff, status, canonical URL, persistent identifier, and archived predecessor;
- abstract, construct, population, intended use, prohibited use, content blueprint, administration, scoring, and interpretation;
- protocol registrations, sampling and inclusion flow, data and materials availability, analysis code, environment, and reproducibility instructions;
- validity, reliability, uncertainty, fairness, accessibility, subgroup, burden, and consequence results;
- funding, sponsor role, conflicts, ethics/applicability determination, consent, privacy, AI-use disclosure, and independent-review status;
- benchmark card, schemas, reference implementation, task examples, rater guidance, result format, checksums, and licenses;
- correction, invalidation, appeal, change, compatibility, maintenance, retirement, and security records.
The interface must never simulate these signals. “Reviewed,” “independent,” “validated,” “certified,” “accredited,” “reproduced,” and “peer reviewed” are controlled claims with named evidence requirements.
9. Current publication disclosures
| Disclosure | Current record |
|---|---|
| Organizational author | LockedIn Labs Research |
| Individual authors and CRediT roles | Not yet published; required before an authored external release |
| Funding | No finalized public funding statement is on record; no outside sponsor is represented as endorsing this draft |
| Conflict of interest | LockedIn Labs is founder and operator and may also train, employ, or provide services to FDE practitioners and customers; these roles require separation, disclosure, and non-conflicted review |
| Human-participant research | No human-participant dataset is admitted to the benchmark; no Institutional Review Board or equivalent approval or exemption is claimed |
| Data and materials | Public specifications, schemas, synthetic fixtures, and source code are available in the repository; no real-person evidence is public or admitted |
| External or peer review | No external peer review, independent validation, or independent reproduction is recorded |
| AI assistance | AI-assisted tools have supported source discovery, drafting, software development, and testing; they are not evidence sources or decision authorities, and the human steward remains responsible for every public claim |
| License and DOI | Operative repository licenses, canonical publication URL, DOI, and archival deposit remain publication blockers |
10. Activation rule
The project may use a mature visual design and working software while remaining a public draft. It may not activate official human scores, result bands, pass/fail, certification, or employment use until the required formation and validity artifacts are present, reviewed, reproducible, and authorized in a versioned release decision.
The controlling state of current and future capabilities is Live Versus Planned. Research claims must also satisfy the publication template, study register, and aggregate substantiation rules.
11. Literature-review method
The current literature base is a targeted design review, not a systematic review, scoping review, or meta-analysis.
- Question: Which established occupational-assessment, validity, fairness, qualitative-research, identity, certification, benchmark-operation, and reproducibility principles should constrain the FDE benchmark’s formation?
- Source priority: official professional standards; regulators and government technical guidance; primary benchmark rules and repositories; peer-reviewed foundational, review, or meta-analytic literature; then first-party operating methods.
- Inclusion rule: a source must change a documented design decision, evidence requirement, use boundary, or research question. Prestige or citation count alone is insufficient.
- Exclusion rule: commentary without a traceable primary basis, marketing claims, copied taxonomies without rights review, and published coefficients proposed as direct weights for this instrument are excluded.
- Record: the source ledger retains the reviewed version/date, authority, material influence, non-adoption, and next review trigger.
REFERENCES.bibstores bibliographic metadata. - Limit: the project has not published database coverage, complete search strings, duplicate screening, dual review, quality/applicability ratings, or a flow diagram. It therefore makes no completeness or systematic-review claim.
Before a validity publication relies on a literature synthesis as evidence, the project must pre-register the questions, databases, dates, queries, inclusion/exclusion rules, screening procedure, extraction fields, risk-of-bias or applicability method, disagreements, and update schedule.
12. Principal design references
- American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. Open-access record.
- Society for Industrial and Organizational Psychology. (2018). Principles for the Validation and Use of Personnel Selection Procedures (5th ed.). PDF.
- Mislevy, R. J., Almond, R. G., and Lukas, J. F. (2003). A brief introduction to evidence-centered design. https://doi.org/10.1002/j.2333-8504.2003.tb01908.x.
- National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://doi.org/10.6028/NIST.AI.100-1.
- National Institute of Standards and Technology. (2026). Expanding the AI Evaluation Toolbox with Statistical Models (NIST AI 800-3). https://doi.org/10.6028/NIST.AI.800-3.
- National Institute of Standards and Technology. (2026). Practices for Automated Benchmark Evaluations of Language Models (NIST AI 800-2, Initial Public Draft). https://doi.org/10.6028/NIST.AI.800-2.ipd. This is an emerging AI-evaluation input, not a final standard and not direct evidence for human assessment.
- International Organization for Standardization and International Electrotechnical Commission. (2026). ISO/IEC 17024:2026 — Conformity assessment: General requirements for bodies operating certification of persons. Official record.
- U.S. Equal Employment Opportunity Commission. Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures. Official guidance.
- MLCommons. Submission rules, inference rules, and audit guidelines. Policy repository and inference policy repository. These are operating precedents, not human-assessment standards.