Benchmark Card: LockedIn FDE Benchmark v0.1

Status: Public Draft
Official evaluation service: Not operational
Official numeric scoring: Disabled
Validation status: Not validated
Examples and fixtures: Synthetic only
Founding steward: LockedIn Labs

This card describes the intended scope and present limitations of the v0.1 framework. It is a release constraint, not marketing copy.

1. Summary

The LockedIn FDE Benchmark proposes a continuous, evidence-backed way to describe demonstrated Forward Deployed Engineering capability. The v0.1 release context is AI-native Forward Deployed Engineering: practitioners applying the forward-deployed operating model to AI-enabled systems and workflows. It connects claims to provenance, engineering artifacts, controlled observations, contribution attribution, production operation, corroboration, human review, and versioned results.

The v0.1 release is a methodology and implementation foundation. It does not evaluate real candidates or establish that its proposed measures are reliable, fair, predictive, or suitable for employment selection.

2. Construct

The construct is demonstrated capability to move an ambiguous enterprise problem through discovery, architecture, hands-on engineering, AI/software evaluation, integration, security and governance, production operation, outcome substantiation, and capability transfer.

Forward Deployed Engineering is the broader operating model; AI-native deployment is the specialization measured first. The portable core covers customer-side discovery, implementation, production ownership, outcomes, and the field-to-product loop. This release adds explicit AI-system evaluation, data and model behavior, agentic workflow, safety, governance, and production requirements. It does not claim that all FDE roles are AI roles or that v0.1 transfers to non-AI deployment contexts.

The primary unit is an individual practitioner. The model does not treat role title, résumé language, course completion, repository volume, employer brand, or popularity as capability.

The full normative definition is in SPEC.md.

3. Intended population

The proposed v0.1 target population is adult practitioners performing or preparing for AI-native Forward Deployed Engineering work across a range of experience and responsibility levels.

The v0.1 construct has not yet been shown to transfer across industries, countries, languages, disability groups, employment models, non-AI forward-deployed contexts, or role variants. Population claims must remain narrow until calibration evidence supports expansion. Any later security, financial-infrastructure, telecommunications, defense, industrial, or other context pack requires its own practice analysis, content blueprint, validation plan, and release decision.

4. Intended uses in v0.1

• Review and improve the construct, ontology, evidence model, and governance.
• Exercise public schemas with synthetic data.
• Demonstrate deterministic evidence lineage and experimental calculation behavior.
• Prepare a controlled calibration protocol.
• Support clearly labeled self-reflection without verification or employment consequence.
• Develop a method-first research and publication program using privacy-protected aggregate findings.

5. Prohibited and unsupported uses

The v0.1 release is not suitable for:

• automated or sole-source hiring, rejection, termination, promotion, compensation, discipline, ranking, or staffing;
• official numeric scores, percentiles, score bands, or role-readiness labels;
• verified profiles, field-validated designations, or credentials;
• claims of certification, accreditation, predictive validity, independent validation, or absence of bias;
• comparison across benchmark versions without an approved linking study;
• public leaderboards or universal “best engineer” rankings;
• a single global number for a person;
• cost-per-person, compensation-efficiency, “value for money,” or speed-at-any-cost comparison;
• inference from missing private evidence;
• intake or analysis of real private repositories, customer artifacts, references, identity documents, or confidential deployment records.

6. Proposed evidence sources

Future controlled releases may consider:

• repository and engineering-artifact analysis;
• architecture decisions, runbooks, test/evaluation records, and deployment manifests;
• controlled labs, live defenses, and adaptive scenarios;
• production-operation, incident, reliability, and outcome evidence;
• contribution records and claim-specific corroboration;
• relevant external assessments or credentials with documented quality.

No single evidence type is mandatory when an equivalent, job-relevant, privacy-preserving demonstration is available. Public source-control activity is not required.

7. Proposed assessment flow

1. Inventory claims and evidence under explicit consent.
2. Map evidence to versioned capabilities.
3. identify provenance, attribution, maturity, lineage, contradictions, and uncertainty.
4. Generate targeted evaluation only where useful.
5. Record typed AI or human observations.
6. Route eligible observations through non-conflicted human review or adjudication.
7. Apply deterministic, versioned rules.
8. Publish only disclosure-authorized results with limitations and version context.

This workflow is planned; it is not an operational v0.1 service.

8. Output dimensions

The proposed system keeps these outputs independent:

• capability estimate;
• evidence-support confidence;
• benchmark coverage;
• evidence maturity;
• verification outcome and method;
• public result class;
• domain profile and specialization;
• integrity, dispute, and credential status.

An Insufficient Evidence result is distinct from weak demonstrated capability. Official numeric output is prohibited in v0.1. Synthetic reference calculations, if present, must be labeled experimental and nonofficial.

Three output authorities

The future product architecture separates what is often collapsed into an “FDE Score”:

The layers remain separately versioned and separately reviewable. A verified record cannot silently become a capability conclusion; a profile cannot silently become a credential; and a verifiable credential format cannot make its claims true. See the Outcome Architecture proposal and Independence Firewall.

Proposed research dimensions

The research-platform design uses a multidimensional capability profile rather than a global score:

Time to Production is vulnerable to role, scope, organizational delay, resources, risk tolerance, censoring, and definition choices. It cannot be used as a productivity or cost proxy without evidence that the comparison is job-relevant, fair, and valid. The benchmark permanently excludes a cost-per-person axis.

Any future public explorer is limited to privacy-protected distributions, cohort statistics, methods, and aggregate research. Named-person ranking is prohibited.

9. AI involvement

AI may support evidence normalization, capability mapping, contradiction detection, uncertainty analysis, question generation, and structured rubric observations.

AI cannot define the standard, edit a benchmark release, assign a final score, grant verification, issue a credential, decide an appeal, or make an employment decision. Model outputs are untrusted inputs that require schema validation and the applicable review path.

No Shared Work Challenge Battery has been administered; no human baseline, cross-condition comparison, AI-human agreement result, production evaluator, provider comparison, or calibrated model exists in v0.1.

10. Validation status

No v0.1 reliability, fairness, validity, or field-performance claims are supported.

The planned validation program includes:

• expert content review and job analysis;
• blinded review where practical;
• inter-rater reliability and adjudication analysis;
• test-retest and evaluator-version stability;
• AI-human agreement and error analysis;
• irrelevant-attribute and subgroup testing;
• accessibility and accommodation studies;
• contamination, gaming, and adversarial testing;
• longitudinal criterion validation against later field outcomes.
• grader invariants using known-good, known-insufficient, and known-invalid synthetic cases;
• scenario learnability and answerability review before use;
• routine adjudication-log review and grader-defect impact analysis;
• publication of system failures, held findings, invalid scenarios, failed hypotheses, and corrections without exposing participants or controlled cases.

Thresholds, cohort design, analysis plans, and permitted claims must be specified before data collection. A small feasibility cohort cannot support population norms, percentiles, predictive claims, or broad fairness conclusions.

11. Known limitations

• The role is heterogeneous and evolving.
• Production evidence is unevenly available and often confidential.
• Contribution attribution is probabilistic and team work can be misrepresented.
• Customer outcomes are affected by organizational and market conditions outside one engineer’s control.
• Communication and live evaluation formats can introduce disability, language, cultural, and socioeconomic barriers.
• AI evaluators can be unstable, biased, vulnerable to prompt injection, and sensitive to provider changes.
• Hidden cases can leak or become contaminated.
• Evidence maturity is ordinal; it is not a calibrated interval scale.
• Current domain weights, confidence rules, floors, and role profiles are research hypotheses.
• Time to Production has no validated event definition, normalization, or causal interpretation.
• Evidence Strength and support confidence can be confused with capability unless the interface preserves their separation.
• Dense comparisons can manufacture false precision or ranking even when the underlying method does not support it.
• LockedIn Labs’ role as founder, operator, employer, trainer, and services provider creates conflicts that require enforceable governance.

12. Fairness and accessibility

The project intends to test and reduce irrelevant influence from names, age proxies, gender-coded language, geography, résumé format, school or employer prestige, fame, writing style, disability, and tool access.

Future evaluations must provide an accommodation process and alternate ways to demonstrate the same job-relevant construct. The benchmark prohibits facial, emotion, and personality inference unrelated to the construct.

No claim of fairness or accessibility conformance is made for v0.1.

Future worker or candidate use requires jurisdiction- and use-specific review by qualified counsel. Potential review areas include employment, automated-decision, discrimination, accessibility, privacy, notice, audit, documentation, and human-review requirements. References to particular laws or regions in research materials are issue-spotting inputs, not legal advice or applicability conclusions.

13. Privacy and security

The public repository accepts synthetic examples only. Private-evidence intake remains blocked pending an approved threat model, consent and retention controls, tenant isolation, encryption, malware and secret scanning, sandboxed no-network execution, prompt-injection defenses, incident response, and verified deletion.

Candidate or customer artifacts are not permitted for model training by default in the proposed system. Future exceptions would require specific rights, consent, governance, and a separately disclosed purpose.

See SECURITY.md and DATA_GOVERNANCE.md.

14. Governance and conflicts

LockedIn Labs is the founding steward. Employment, training, use of any particular tool, payment, sponsorship, friendship, employer prestige, and founder status do not add score value.

No external governing body or independent validation panel is represented as operational unless listed as seated in LIVE_VS_PLANNED.md. Future verified results require non-conflicted review; founder and executive results require external review.

See CHARTER.md, GOVERNANCE.md, and CONFLICT_OF_INTEREST.md.

15. Release and change policy

Released versions are immutable. Material changes create a new version with a changelog, release manifest, migration notes, and renewed limitations review. Cross-version comparisons require a published linking study.

The v0.1 line remains nonofficial. Completion of software alone cannot activate official scoring; later releases require measurement, governance, security, privacy, accessibility, and appeal gates.

16. Contact and reporting

Use public issues or RFCs only for non-sensitive specification feedback once those channels are enabled. Never place private evidence, personal data, customer information, security vulnerabilities, or conduct reports in a public issue.

The project must publish operational private reporting channels before opening public participation or accepting sensitive reports. Security reporting guidance is in SECURITY.md.