FDE BenchmarkPublic document reader
Available in sourcev0.1 · live public draft · official release pending

Canonical source view. This renders the bundled repository document as readable HTML. Source metadata remains in the repository, and relative document links remain inside this reader.

Prospective outcome-linkage and criterion-evidence protocol

Protocol ID: LL-FDE-OUT-001
Version: 0.1.0
Status: proposed for a later study; no employer, participant, assessment, criterion, linked dataset, association, prediction, or validation result exists

1. Purpose

This protocol asks whether bounded benchmark evidence relates to independently defined field outcomes under declared contexts. It does not assume that a benchmark profile predicts job performance merely because tasks look realistic or experts approve the content.

The design is informed by the Standards, SIOP Principles, Kane’s interpretation/use argument, work-sample and selection-validity research, reference-checking research, NIST uncertainty guidance, and the legal/use sources in the Source Ledger. Published coefficients are not imported.

2. Preconditions

Outcome linkage is blocked until:

  • job-analysis and content evidence supports the tested capabilities;
  • pilot task/rubric/rater evidence supports the exact experimental interpretation;
  • outcome definitions and collection quality pass independent review;
  • assessment and outcome periods, populations, contexts, and intended use are frozen;
  • consent or other lawful authority, data sharing, privacy/security, retention, correction, withdrawal, accessibility, and participant-support controls are operational;
  • employers cannot use research participation or preliminary results for employment decisions;
  • analysis is preregistered before outcome access; and
  • a fieldwork receipt authorizes outcome_linkage for the exact sources and period.

3. Interpretation/use argument

Each study starts with one bounded claim candidate:

For [declared population/context], evidence from [assessment version and conditions]
is expected to relate to [independently defined outcome over period], subject to
[measurement, opportunity, attribution, missingness, and transportability limits].

Then specify which decisions the evidence might eventually inform and which remain prohibited. The first linkage study should not authorize automated hiring, ranking, termination, compensation, certification, or a universal cutoff.

4. Outcome taxonomy

Candidate outcomes must be observable, relevant, sufficiently reliable, and not circular with the benchmark. They may include:

  • accepted work products against predeclared criteria;
  • delivery/reliability/security outcomes with appropriate baselines and attribution;
  • stakeholder acceptance or decision quality using structured independent ratings;
  • incident prevention, recovery, escalation, or evidence quality;
  • adoption or operational outcomes that are not simply activity volume;
  • product feedback incorporated and verified;
  • structured multisource observations from people with direct opportunity to observe; and
  • retention/role progression only when opportunity, organizational policy, and confounding are explicitly addressed.

Employer rating, compensation, prestige, utilization, revenue, promotion, or “manager satisfaction” is not automatically a valid criterion. Every outcome needs a construct definition, observation window, opportunity denominator, source, collection method, rater evidence, missingness, manipulation risk, and harm analysis.

5. Temporal ordering and blinding

  • Freeze benchmark evidence before the outcome window or otherwise justify ordering.
  • Outcome raters should not see benchmark results; benchmark raters should not see outcome data.
  • Lock outcome definitions before inspecting associations.
  • Separate outcome custodians, assessors, analysts, and employer decision makers where feasible.
  • Record any prior knowledge, unblinding, participant re-administration, intervention, or role change.

Concurrent known-groups or retrospective studies can inform hypotheses but cannot be relabeled prospective prediction.

6. Sampling and selection effects

Document the eligible population and the path into assessment, consent, employment/assignment, observable opportunity, outcome availability, and analysis. Address:

  • self-selection into research and the role;
  • hiring/assignment decisions that restrict the observed range;
  • attrition, censoring, transfers, leave, termination, and role changes;
  • unequal opportunity, resources, manager support, customer constraints, and task mix;
  • employer and team clustering;
  • missing not at random and nonresponse;
  • prior training, experience, and alternative predictors; and
  • sample differences between development and replication.

Any correction for range restriction or selection requires justified assumptions and sensitivity analysis. An adjusted estimate must be reported beside observed evidence and uncertainty.

7. Analysis plan

Predeclare:

  1. primary benchmark variable/profile and outcome;
  2. estimand, time horizon, and unit of analysis;
  3. model, functional form, dependence/clustering, and diagnostics;
  4. covariates selected from a causal/measurement rationale rather than significance fishing;
  5. missingness, censoring, outlier, invalid-case, and withdrawal handling;
  6. uncertainty and multiplicity;
  7. incremental evidence over predeclared alternatives;
  8. calibration, prediction error, decision consequence, and sensitivity where prediction is studied;
  9. subgroup/context and accessibility/accommodation analyses with privacy protection;
  10. robustness to rater/task/form and outcome-measure uncertainty;
  11. preregistered replication rule; and
  12. exact supported and prohibited public wording under each result pattern.

Statistical significance is not the release gate. Evaluate magnitude, uncertainty, measurement quality, practical consequence, stability, alternative explanation, transportability, and harm.

8. Fairness and accessibility

Study whether benchmark and outcome measures reflect unequal access, opportunity, accommodation, or organizational assignment rather than capability. Protected attributes collected for approved fairness analysis remain separated from rating and employment decisions.

Report material differential missingness, task/rater behavior, criterion opportunity, prediction error, and decision simulation. Small cells are held or aggregated. Absence of detected disparity in an underpowered sample is not evidence of fairness.

9. Reference evidence

References are structured evidence sources, not reputation checks. The study must record relationship and observation opportunity, prompts, response scale, missing/not-observed options, conflicts, identity/authority verification, rater evidence, and consent. Do not solicit protected, medical, family, political, or unrelated personal information.

A reference’s favorable narrative is neither ground truth nor proof of identity. Candidate access, correction, and contestability rules require legal and participant-rights review.

10. Replication and release progression

A development study can support task/rubric revision and a bounded relation claim. Broader predictive or use claims require an independent or meaningfully separated replication across a declared new sample/context, with the same primary definitions or a prospectively justified change.

Release reports null, negative, contradictory, and subgroup/context findings alongside positive results. A failed replication triggers a claim and use review, not a search for a favorable subgroup.

11. Publication requirements

Publish protocol timing, population/sample flow, assessment and outcome versions, access/opportunity model, blinding, deviations, missingness, estimands, uncertainty, alternatives, fairness/accessibility evidence, privacy/ethics authority, data availability, analysis package, conflicts/funding, independent review, and exact claim receipts.

12. Current disposition

No criterion-related or predictive statement is permitted. The benchmark remains a construct/method draft with human scoring disabled.