Multi-cohort research program for the LockedIn FDE Benchmark

Program ID: FDE-MCRP-001
Protocol version: 0.1.0-draft
Protocol date: 2026-09-01
Evidence cutoff: 2026-09-01
Status: proposed; no study in this program has recruited participants, collected data, completed analysis, or authorized a public finding
Authorization: this document does not authorize participant contact, real-data intake, scoring, certification, hiring use, or publication of achieved sample counts
Organizational steward: LockedIn Labs Research
Required accountable roles before activation: Head of Measurement, FDE Research Director, Research Ethics and Participant Rights Lead, Fairness and Accessibility Research Lead, Privacy Statistician, and an independent methods reviewer

1. Purpose

This program defines the prospective research needed to turn the public FDE capability framework into a defensible occupational benchmark. It is deliberately broader than a single survey. It combines job analysis, structured surveys, qualitative inquiry, expert review, cognitive pretesting, assessment calibration, criterion validation, consequence research, and annual monitoring.

The program is designed to answer five different questions without collapsing them into one:

1. What is the work? Identify consequential FDE activities, decisions, artifacts, contexts, and outcomes.
2. How does the role differ? Distinguish FDE work from adjacent AI, platform, solutions, consulting, product, and delivery roles.
3. What can be observed fairly? Determine which capabilities can be elicited through realistic work samples, verified evidence, structured defense, and references without substituting prestige or opportunity for capability.
4. What does a result support? Estimate reliability, precision, validity, fairness, and limits for an exact population and intended use.
5. What happens when the benchmark is used? Monitor burden, exclusion, gaming, false decisions, employer behavior, appeals, and changing work.

Surveys provide one evidence stream. Survey majorities do not establish a construct, set a pass score, prove predictive validity, or authorize an employment decision. Every inference requires the evidence class appropriate to that inference.

2. Authority, relationship to existing records, and non-claims

This protocol sits beneath the benchmark formation protocol, job analysis and content blueprint protocol, Phase 2 research agenda, and research publication template.

The previously reported commissioned interviews remain governed by FDE-JA-001 and the study register. This program does not assume that those interviews occurred, met any protocol, or can be admitted. Its prospective job-analysis study uses a distinct ID, FDE-JA-002. Historical material may inform hypotheses only after its own intake and provenance gates pass.

At this protocol version:

• no sampling frame has been acquired;
• no ethics or human-participant applicability determination has been recorded;
• no recruitment vendor, research partner, or panel provider has been approved;
• no instrument has passed cognitive testing;
• no participant or employer outcome data exists in this program;
• every sample size below is a planned design target, not an achieved count;
• final sample sizes require a frozen precision, power, or simulation memo before recruitment;
• the ontology remains a public-draft hypothesis, not a validated occupational standard; and
• no score, pass band, certification, percentile, predictive claim, or hiring use is authorized.

3. Methodological foundation

The research design is informed by the following professional and primary sources:

• The Standards for Educational and Psychological Testing require validity evidence for the proposed interpretation and use, including evidence based on content, response processes, internal structure, relationships with other variables, and consequences; they also treat fairness and documentation as integral rather than optional (AERA, APA, and NCME, 2014).
• The SIOP Principles for the Validation and Use of Personnel Selection Procedures inform work analysis, representative sampling, criterion quality, reliability, subgroup analysis, documentation, and use-specific validation (SIOP, 2018).
• U.S. Office of Personnel Management job-analysis guidance links tasks and competencies to assessment content, while EEOC materials emphasize important work behaviors or work products and validation for the intended use (OPM, Job Analysis; EEOC, Uniform Guidelines Q&A). These are design inputs, not government approval or legal advice.
• AAPOR standards inform transparent disclosure of sampling, recruitment, disposition, participation, questionnaire wording, processing, weighting, and limitations. A response rate alone does not establish absence of nonresponse error (AAPOR, Disclosure Standards; AAPOR, Standard Definitions).
• Federal cognitive-interview guidance treats pretesting as a structured study of how respondents understand and answer questions, including design, conduct, analysis, and reporting (OMB/NCHS cognitive-interview standards; CDC CCQDER).
• COREQ and SRQR inform transparent reporting of interviews, reflexivity, sampling, analysis, and limitations; they are reporting guidance, not proof of study quality (Tong et al., 2007; O'Brien et al., 2014).
• CREDES informs the proposed Delphi process, especially a priori consensus rules, panel description, round-level reporting, and preservation of disagreement (Jünger et al., 2017). Its original clinical context does not automatically transfer to occupational assessment.
• STROBE informs complete reporting of the cross-sectional and longitudinal observational studies in this program; it is a reporting checklist, not a design-quality rating (STROBE Initiative).
• HHS/OHRP sources inform prospective consent, understandable disclosure, voluntariness, participant questions, and withdrawal where applicable. An authorized authority must determine applicability; this protocol does not self-award IRB approval or exemption (HHS Common Rule; OHRP informed-consent guidance).
• Work-sample and selection-method research supports studying direct, job-relevant demonstrations while cautioning against importing published coefficients into a new instrument (Roth, Bobko, and McFarland, 2005; Sackett et al., 2022).

The canonical influence, boundary, and review-trigger record for these and related sources remains the source ledger. Citation is not a conformance, endorsement, validity, or legal-compliance claim.

4. Program questions and inference ladder

5. Study portfolio and current status

These entries are prospective planning identifiers. They do not enter the public study register or advance its state machine until a separate governed registration event occurs.

FDE-SRV-EXEC-001 is the structured rating module embedded in the 50 FDE-JA-002 executive interviews. These are the same 50 planned people, not two additive cohorts. The nested practitioner, adjacent-role, and manager/reference interviews likewise remain inside their parent-cohort counts.

5.1 Meaning of target numbers

The initial public planning set is 50 executive interviews, 30 organizations, 250 practicing FDEs, 150 senior-AI/adjacent comparators, 200 linked manager/reference records, and 100 longitudinal cases. The protocol uses those same numbers. They are formation and feasibility targets, not promises, achieved counts, evidence of representativeness, or authorization for a predictive or fairness claim. The candidate cohort is an additional affected-party study and is labeled separately.

The initial targets are useful for coverage, instrument refinement, variance estimation, and deciding whether a larger confirmatory study is feasible. They are not methodologically sufficient by default for high-dimensional psychometric modeling, stable intersectional subgroup estimates, or a use-specific predictive-validation claim. In particular, the 100-case longitudinal cohort is a feasibility cohort; any later criterion-validation sample remains unset until outcome reliability, nesting, attrition, range restriction, and minimum practical effect are estimated.

Before each study opens, the Head of Measurement must approve a sample-size memo that states:

• primary estimand or qualitative coverage objective;
• minimum practically important difference or precision target;
• expected variance, event rate, intraclass correlation, attrition, and missingness assumptions;
• number and size of planned strata and clusters;
• design effect and weighting effect, where relevant;
• planned model degrees of freedom and shrinkage/regularization;
• simulation or analytic power method, code, seed, and sensitivity range;
• stopping rule and whether sequential looks are permitted; and
• how under-recruitment changes the claim rather than silently changing the threshold.

Any revised target is a prospective amendment. An achieved sample cannot be described as the planned sample if the plan was changed after outcome inspection.

6. Cross-program population and sampling rules

6.1 Units and role verification

The primary unit is defined before collection and is never switched to make a count larger:

• executives and practitioners: a distinct eligible person;
• firms: a distinct organization, with respondent-level records nested within it;
• references: a distinct referee-subject relationship, with both referee and subject clustering retained;
• employer outcomes: a distinct participant-period nested in participant and organization; and
• qualitative interviews: a distinct eligible person, with repeat sessions linked rather than counted as new people.

Title alone never establishes cohort eligibility. Eligibility uses recent responsibility, observable work, decision authority, and opportunity-to-observe. Each study freezes a role-verification rubric before recruitment. Ambiguous records are held for blinded adjudication.

6.2 Sampling frames and multiple-channel recruitment

There is no known complete census of FDEs. Most studies will therefore use explicitly labeled nonprobability or blended samples. Each cohort must use at least three materially independent channels when feasible, such as:

• employer or professional-association lists with documented coverage;
• verified professional communities and role-specific events;
• research panels with declared sourcing and quality controls;
• public role postings and employer directories for organization sampling;
• direct outreach to people whose qualifying work can be documented; and
• referrals used only as a tagged supplement, never as an unobserved chain.

Recruitment source, invitation version, contact dates, eligibility disposition, incentive, and duplicate resolution are retained. No single channel may supply more than 35% of the analytic cohort without a preregistered exception and sensitivity analysis. No single employer may supply more than 5% of a person-level analytic cohort. These are initial concentration controls, not claims of population representativeness.

6.3 Coverage quotas and strata

Quotas secure variation; they do not create a probability sample. Every study declares a small set of primary quota dimensions and treats the rest as coverage tags. The common coverage dimensions are:

• role variant and level of accountability;
• direct FDE title versus qualifying work under another title;
• employer size and maturity;
• end-user enterprise, AI product company, consultancy/services firm, public sector, or independent setting;
• deployment model, including embedded, platform-led, services-led, and hybrid work;
• regulated or safety-relevant exposure;
• industry/domain pack;
• geography and jurisdiction;
• production and customer access opportunity;
• language, device, disability/accommodation need, and paid-tool access; and
• recruitment channel.

Cross-classifying every variable would produce unusable cells. Before collection, the protocol names the interactions that matter to the decision. Unfilled cells are reported, not concealed through weighting.

6.4 Probability, nonprobability, weighting, and population claims

If a probability frame is available, inclusion probabilities and frame coverage are documented. If it is not, the report states that population prevalence and margin-of-sampling-error claims are unsupported. For nonprobability samples:

• report unweighted distributions first;
• use calibration or propensity weighting only against a defensible external frame and only under a preregistered model;
• show weighted and unweighted estimates, effective sample size, weight distribution, trimming rule, and sensitivity;
• never label quota shares as population shares; and
• avoid “X% of all FDEs” language unless the sampling design supports it.

6.5 Recruitment and participation protections

• Invitations separate research participation from hiring, employment management, training, customer delivery, and sales.
• A manager may distribute an invitation but cannot know who participated unless the participant separately authorizes it.
• Incentives are standardized by burden and locality, disclosed, and reviewed for coercion or undue influence.
• Reminder count and timing are fixed; high-status invitees do not receive outcome-shaping special treatment.
• A person can decline, skip nonessential items, stop, request correction, or withdraw within the declared withdrawal window without penalty.
• Bot, duplicate, speed, inconsistency, and identity-quality checks route to review; they do not silently erase people or infer capability.

7. Common measurement architecture

7.1 Core survey modules

Every cohort receives only modules relevant to its observation opportunity. The versioned item-bank proposal is in Survey Instrument Modules.

7.2 Rating scales

Ratings use behaviorally anchored categories rather than unlabeled numbers. Candidate anchors must be cognitively tested and may change before fielding. Illustrative constructs include:

• frequency: not performed; less than quarterly; monthly; weekly; most workdays;
• importance: little effect on acceptable work; useful; important; essential; failure makes the work unacceptable;
• consequence: readily reversible; local rework; material project impact; customer/production impact; severe safety, security, legal, or financial impact;
• entry criticality: can be learned after entry; expected with support; independently required at entry; must be demonstrated before assignment; and
• observation strength: hearsay; indirect observation; direct observation; reviewed artifact; repeated direct observation across contexts.

Ordinal categories remain ordinal unless a prespecified measurement model and diagnostics justify another treatment. “Not observed,” “not applicable,” and “prefer not to answer” are kept distinct from low ratings.

7.3 Criterion measures

External criteria are defined independently of benchmark scores and collected from sources not rewarded for confirming the benchmark. Candidate criteria include:

• delivery of an accepted production or pilot outcome under the governed scope;
• technical quality, reliability, security/control adherence, and reversibility;
• incident recognition, escalation, recovery, and learning;
• decision quality under changing constraints;
• customer or user adoption attributable to the work;
• rework, preventable defects, and sustained operation;
• documentation, evidence, and transfer of capability to the customer team; and
• manager/customer structured ratings anchored to observable behavior.

Title, compensation, employer prestige, educational pedigree, social visibility, sales volume, and promotion are not sole criteria. Time to production is contextualized by scope, risk, organizational dependency, quality, and delay outside the participant's control; it is not a personal speed score.

8. Foundational qualitative and consensus studies

8.1 FDE-JA-002: prospective critical-incident and work-product job analysis

Research questions. Which work activities, decisions, artifacts, failure modes, outcomes, and boundary-spanning behaviors characterize FDE work? Which are important, frequent, difficult, consequential, and required on entry? Which vary by setting? Which distinguish FDE work from adjacent roles?

Population and sample. The public planning target is 50 distinct eligible executive interviews. Qualitative critical-incident subsamples are also drawn from the practitioner, adjacent-role, and manager/reference cohorts; those people remain inside their parent-cohort targets and are not added to the public planning counts a second time:

Purposive maximum-variation sampling is used. The study does not estimate prevalence. Recruitment continues to the planned coverage matrix, not to an undefined claim of “saturation.” If thematic sufficiency is reported, it must use an ordered accrual analysis, stable codebook criteria, negative-case review, and explicit unsampled contexts.

Method. Semi-structured interviews use critical-incident probes for a recent success, failure or near miss, constraint change, and recovery. Participants identify observable behavior, work product, decision rights, collaborators, tools/assistance, customer boundary, outcome, and what a less-capable response would have looked like. No customer secret, credential, source code, or third-party personal data is requested.

Analysis and quality. Deductive ontology codes are separated from inductive field codes. At least 20% of eligible records in each perspective, spanning every interviewer and material stratum, are independently double coded. The frozen analysis plan declares the statistic, unit, uncertainty, prevalence handling, and action rule. The initial code-decision conventions in FDE-JA-001 may be reused only if justified prospectively. Supporting, contradicting, and boundary cases remain visible. Mention counts never become weights.

Outputs and gate. Outputs include a task inventory, work-product inventory, role-boundary map, negative cases, context variations, proposed ontology changes, and source-to-blueprint traceability records. No assessment task enters a human pilot until it traces to admitted job-analysis evidence and panel disposition.

Status. Proposed; no recruitment or data collection authorized.

8.2 FDE-DELPHI-001: independent SME and affected-party panel

Purpose. Review content relevance, representativeness, clarity, criticality, redundancy, context dependence, accessibility, and opportunity-to-demonstrate. The panel refines content; it does not validate scores or approve its own commercial use.

Panel. Plan 36–48 participants across three balanced constituencies:

• 16–20 experienced FDE/practice experts across operating contexts;
• 8–12 industrial-organizational psychology, psychometrics, assessment, accessibility, privacy, and security experts; and
• 12–16 affected parties, including candidates, hiring managers, adjacent-role practitioners, and customer-side leaders.

Panel selection, qualifications, affiliations, financial relationships, training relationships, benchmark authorship, recusals, and attrition by round are published. LockedIn employees and paid advisers are capped and reported; they cannot form a voting majority.

Rounds. A planned three-round modified Delphi begins with the admitted job-analysis inventory, not an empty brainstorm. Round 1 challenges scope and identifies missing or harmful content. Round 2 rates relevance, representativeness, criticality, clarity, and fairness/accessibility risk. Round 3 rerates unresolved items after controlled feedback showing distributions and anonymized rationales.

Prospective consensus convention. Before round 1, the independent methods reviewer freezes the final rule. A candidate starting rule is: retain when median relevance is at least 4 on a 5-point anchored scale, interquartile range is at most 1, at least 80% rate 4 or 5, and no unresolved substantiated safety, fairness, or construct objection remains. Reject when median is at most 2 and at least 80% rate 1 or 2. All other items are revised, rerated, contextualized, or recorded as no consensus. Consensus is not correctness. Minority rationales, recusals, abstentions, attrition, and steward departures are published.

Analysis. Report round-level denominators, medians, interquartile ranges, rating distributions, changes, stability, attrition, and cross-constituency differences. Do not pool away a conflict between field experts and affected participants. CREDES is used as a reporting prompt, with its domain-specific limits disclosed.

Status. Proposed; no panel appointed.

8.3 FDE-COG-001: multi-audience cognitive interviews

Research questions. Do respondents interpret role terms, task statements, response anchors, reference questions, consent/linkage choices, and fairness questions as intended? Can they retrieve an answer, map it to a response, and answer without disclosing confidential information? Does interpretation vary by audience, language, device, or accommodation?

Sample. Two planned rounds with approximately six participants from each of seven audiences per round: executives, firm leaders, practicing FDEs, adjacent practitioners, candidates, references/managers, and employer-outcome reporters. The design target is 84 sessions. Purposeful variation includes direct-title and untitled practitioners, regulated contexts, mobile/desktop, low-bandwidth participation, non-native English speakers, screen-reader/keyboard users, and people requesting common accommodations. This is qualitative pretesting, not a prevalence sample.

Method. Use concurrent or retrospective think-aloud where appropriate, scripted comprehension and paraphrase probes, retrieval probes, confidence probes, response-category probes, confidentiality probes, and usability/accessibility observation. Round 1 tests concepts and wording. The instrument is revised with an item-level decision log. Round 2 retests material changes and unresolved audience differences in the intended collection mode.

Analysis and gate. Each issue is linked to the item/version, audience, likely response-error mechanism, severity, evidence, revision, and retest state. Critical failures include incompatible interpretations of the construct, coerced disclosure, inaccessible routing, inability to distinguish “not observed” from “low,” or misunderstanding a research consent choice. A module cannot enter pilot while a critical failure remains open.

Status. Proposed; no sessions conducted.

9. Cohort protocols

9.1 FDE-SRV-EXEC-001: executive FDE hirers and accountable leaders

Research questions

1. Which business and technical outcomes cause organizations to create or procure FDE capacity?
2. Which observable activities and failure modes distinguish successful FDE work from generic implementation, sales engineering, consulting, or staff augmentation?
3. Which capabilities are required on entry, which can be developed, and which depend on team or organizational context?
4. What evidence and uncertainty would support a hiring or staffing decision, and what uses would be unacceptable?
5. Which outcome criteria can be collected independently and consistently in a longitudinal validation study?

Population and sampling frame. People with recent material accountability for FDE hiring, budget, deployment portfolio, customer delivery, operational risk, or outcome acceptance. Titles alone are insufficient. Frames may include sampled organizations with public AI deployment activity, professional executive networks, employer associations, and research panels with verified role/accountability data.

Planned quotas and target. Target N=50 eligible structured interviews. At least 25 should represent end-user/buyer organizations rather than AI vendors or consultancies. Primary coverage dimensions are organization size/stage, regulated versus unregulated exposure, end-user versus product/services setting, and geography. Seek at least 10 respondents with direct accountability in regulated or safety-relevant settings. Cap three respondents per employer in person-level summaries and retain employer clustering. This qualitative/structured target supports role formation and hypothesis generation, not executive-population prevalence estimates.

Recruitment. Organization-first sampling where possible, followed by role verification. Direct outreach, relevant associations, and a vetted executive panel form independent channels. Sales/customer lists may supplement but are tagged and cannot dominate. Incentives and sponsor wording are standardized.

Measures and questionnaire modules. Eligibility/accountability; operating model; role boundary; task importance/consequence/entry criticality; critical incidents; work products; hiring evidence; acceptable assistance and accommodations; outcome criteria; risk and governance; intended and prohibited use; burden and decision tradeoffs.

Analysis. Report distributions and uncertainty by context, with employer-clustered standard errors where relevant. Predeclare contrasts between end-user and supplier organizations, regulated and unregulated settings, and direct FDE versus adjacent hiring. Use multilevel or partial-pooling models only when assumptions and effective cell sizes support them. Do not convert executive preference into scoring weight. Compare executive ratings with practitioner task reports and manager observation; preserve disagreement.

Validity, bias, and nonresponse. Risks include status bias, aspirational job descriptions, vendor incentives, recall of exceptional projects, sponsor pleasing, and overrepresentation of visible AI adopters. Mitigations include recent behavior anchors, success and failure cases, source tagging, end-user minimums, anonymous response, frame comparisons, late/nonrespondent analysis where available, and sensitivity excluding sponsor/customer channels. A large convenience sample does not support “executives generally believe.”

Privacy and ethics. Do not request confidential strategy, customer names, employee identities, or active hiring decisions. Participation is separate from commercial relationships. Employer identity is controlled and not published without explicit authorization.

Publication receipts. In addition to the common receipt package in Section 11: organization frame/coverage memo; executive-accountability rubric; employer-deduplication report; end-user/supplier composition; channel-concentration analysis; clustered analysis specification; and a claim-by-claim denominator table.

Status. Proposed; no frame acquired or recruitment authorized.

9.2 FDE-SRV-FIRM-001: FDE and AI-services firms

Research questions

1. How do organizations define, staff, train, supervise, and quality-control FDE work?
2. Which delivery models and capability mixes vary by firm type, customer segment, platform ecosystem, and deployment risk?
3. Which firm-level systems enable or constrain individual performance?
4. Which operational metrics exist, who owns them, and can they be linked without contaminating individual assessment?
5. How can assessment and certification remain independent from training, placement, and services revenue?

Population and sampling frame. Organizations that employ, contract, or deploy teams performing qualifying FDE work. The primary unit is the organization. One primary respondent with operating authority is preferred; a second technical or people leader may provide a linked corroborating response, but the organization is not counted twice. Frames may combine public employer/vendor directories, relevant service-provider lists, professional networks, and sampled organizations from the executive study.

Planned quotas and target. Target N=30 unique organizations, with no parent group counted as independent subsidiaries unless the sampling rule justifies separate operating units. Primary strata are firm size, end-user versus product versus services business model, platform/vendor ecosystem, geography, regulated-industry exposure, and embedded versus centralized delivery. Seek at least eight end-user organizations, eight AI product/platform firms, and eight services/consulting firms; the remaining six fill documented coverage gaps. These are qualitative/feasibility coverage targets, not market shares, and the resulting cells will not support broad between-firm inference without a later study.

Recruitment. Organization-level invitations identify a primary operational respondent and optional corroborator. LockedIn training, applicant, placement, or customer relationships are disclosed and analyzed separately. Public directory outreach and association lists reduce customer-list dependence.

Measures and questionnaire modules. Organization eligibility; team and operating model; role architecture; selection and training separation; work distribution; supervision and review; project/incident portfolio; tooling and access; governance; performance and outcome data systems; accommodation and accessibility practices; reference policy; data-linkage feasibility; conflicts and sponsor relationships.

Analysis. Organization-level summaries use one governed record per unit. Multiple respondents are analyzed for within-organization agreement and adjudicated only for factual fields; disagreement on judgment remains data. Exploratory typologies require held-out confirmation or stability analysis and may not be marketed as universal firm categories. Relate firm systems to individual outcomes only in a later multilevel design.

Validity, bias, and nonresponse. Risks include marketing self-presentation, firm-size visibility, multiple responses from enthusiastic firms, and customer/vendor conflicts. Require evidence categories for factual claims, keep subjective ratings separate, deduplicate ownership groups, publish recruitment-source composition, compare responding and sampled firms on observable frame variables, and run analyses excluding commercial affiliates.

Privacy and ethics. No client lists, contract terms, employee-level performance, security incidents, or confidential delivery data enter the survey. Firm names remain controlled unless separately approved for attribution. A respondent cannot authorize release of another person's data.

Publication receipts. Organization roster digest; parent/subsidiary resolution; primary/corroborator linkage rules; respondent authority basis; firm-level missingness and agreement; affiliate/conflict analysis; and organization-weighting decision.

Status. Proposed; no organization contacted.

9.3 FDE-SRV-PRACT-001: practicing FDEs and qualifying untitled practitioners

Research questions

1. What work do practitioners actually perform, how often, under what constraints, and with what consequences?
2. Which work products and decisions provide observable evidence of capability?
3. Which capabilities require direct production/customer experience, and which can be demonstrated through a safe alternate task?
4. How do role scope and task mix vary by seniority, organization, domain, deployment model, and access opportunity?
5. Which proposed benchmark requirements introduce construct-irrelevant burden or reward privilege of access?

Population and sampling frame. Adults who have recently performed material work in the design, integration, deployment, operation, or adoption of an AI-enabled system across a customer or enterprise boundary. A title is neither necessary nor sufficient. A frozen eligibility rubric records recency, work setting, observable behavior/work product, and responsibility.

Planned quotas and target. Target N=250 eligible completes. Primary quotas cover career stage, direct FDE title versus qualifying untitled work, organization setting, deployment model, regulated exposure, and geography. Coverage tags include domain, production access, customer access, tool/model access, language, device, and accommodation. No employer may exceed 5% and no recruitment channel 35% without amendment. Seek at least 50 qualifying untitled practitioners to reduce title circularity and at least 50 practitioners outside North America, subject to language/privacy feasibility. The cohort supports task and context estimation; it is not assumed sufficient for every intersectional or latent-variable analysis.

Recruitment. Use sampled employer outreach, professional communities, direct work-based outreach, research panels, and limited tagged referral recruitment. Verification relies on role evidence and consistency review without demanding public portfolios or employer prestige. People with confidential work must have an equivalent privacy-preserving eligibility path.

Measures and questionnaire modules. Role verification; work context; task/work-product inventory; importance/frequency/difficulty/consequence/entry criticality; critical incidents; autonomy and decision rights; collaboration; assistance/tool conditions; evidence availability and attribution; role boundaries; learning versus assessment; accessibility/burden; optional fairness demographics; consent to recontact or outcome linkage.

Analysis. Produce task-level distributions with denominators and uncertainty, co-occurrence maps, context contrasts, and ontology coverage. Separate respondent-level task prevalence from excerpt or artifact frequency. Calibration and confirmatory subsets are separated where dimensional models are explored. Exploratory factor or item-response models do not determine the ontology by themselves; qualitative content evidence and context dependence remain visible.

Validity, bias, and nonresponse. Risks include self-selection by highly visible practitioners, title inflation, recall and self-enhancement, survivor bias, English-language coverage, exclusion of confidential work, employer-channel dependence, and unequal access to production. Mitigations include role-behavior eligibility, independent channels, confidential evidence paths, recent-event anchors, negative incidents, bounded recall periods, channel/employer sensitivity, nonresponse follow-up, and separate opportunity-to-demonstrate measures. Self-report never becomes a capability score.

Privacy and ethics. No employer verification contact occurs without separate consent. Protected attributes are optional, stored separately, and unavailable to raters or scoring systems. Participants are instructed not to upload customer data, source code, credentials, personal data, or proprietary artifacts. Recontact and outcome linkage are distinct opt-ins.

Publication receipts. Eligibility/adjudication rubric; employer/channel concentration; direct-title/untitled composition; coverage matrix; instrument translations; confidential-evidence pathway test; missingness and opportunity analysis; and item-level task inventory with no unsupported population prevalence language.

Status. Proposed; no participants recruited.

9.4 FDE-SRV-ADJ-001: senior AI practitioners and adjacent roles

Research questions

1. Which activities and artifacts overlap with FDE work, and which distinguish it?
2. Where can adjacent-role experience transfer directly, conditionally, or not at all?
3. Which ontology requirements are generic seniority or engineering behaviors rather than FDE-specific capabilities?
4. Which role boundary would create false positives or false negatives in hiring or certification?

Population and sampling frame. Senior practitioners with recent qualifying work in at least five prespecified clusters: AI/ML engineering; solutions architecture/engineering; platform, SRE, or DevOps; consulting/professional services; and technical product/program leadership. Participants with mixed histories are assigned a primary cluster prospectively and retain secondary tags.

Planned quotas and target. Target N=150, with at least 25 eligible completes in each of the five primary clusters and the remaining 25 allocated to coverage gaps or additional preregistered clusters. Match or calibrate recruitment coverage to the practicing-FDE cohort on career stage, organization setting, regulated exposure, and geography where feasible. These small cluster targets support role-boundary exploration and item challenge, not stable population estimates or a production classifier. Do not force equal quota shares into population estimates.

Recruitment. Use role-specific associations, employer sampling, research panels, and direct outreach independent of the FDE channels when possible. Invitations do not describe the adjacent cohort as a control group or as less capable.

Measures and questionnaire modules. Eligibility; work context; the common task/work-product inventory; customer-boundary and production responsibility; role differentiation; transferability; critical incidents; evidence access; assessment conditions; fairness and burden.

Analysis. Predeclare discriminant hypotheses at the task or behavior level, not a broad “FDEs are different” hypothesis. Use matched, weighted, or regression-adjusted comparisons only with overlap diagnostics and sensitivity to unmeasured selection. Report effect sizes and uncertainty, not only significance. Classification analyses, if attempted, use nested cross-validation and remain exploratory until independently confirmed; they cannot define a person by title.

Validity, bias, and nonresponse. Risks include denigrating adjacent work, role misclassification, unequal seniority, different employer contexts, and circular items written from the current ontology. Mitigations include neutral wording, mixed-history tags, common item administration, independent item challenge, covariate overlap checks, negative controls, and review by adjacent-role experts.

Privacy and ethics. Role comparison results are aggregate and do not rank professions or imply personal worth. Protected and employment data follow the same separation and minimization rules as the practitioner cohort.

Publication receipts. Cluster definitions; assignment/adjudication rules; matching/overlap diagnostics; common-item version map; discriminant-analysis preregistration; negative/null findings; and affected-party review.

Status. Proposed; no sampling frame approved.

9.5 FDE-SRV-CAND-001: candidates and potential candidates

Research questions

1. Do candidates understand the benchmark's construct, evidence states, uncertainty, result uses, and appeal rights?
2. What burden, access, privacy, language, disability, tool, time, or caregiving barriers could affect participation or results?
3. Which assistance and accommodation policies preserve the construct?
4. How do identity, evidence, reference, and employer-linkage requests affect trust and completion?
5. What harms follow from false positive, false negative, insufficient-evidence, held, or disputed outcomes?

Population and sampling frame. Adults who may seek FDE work, including current FDEs, adjacent practitioners considering transition, early-career candidates, career changers, and people who would decline assessment. Recruiting only from people who finish the product is prohibited because it removes the burden and exclusion signal.

Planned quotas and target. Target N=200; this affected-party cohort is additional to the six headline public planning cohorts. Primary strata include career stage, prior FDE title, current employment status, prior access to production/customer work, geography/language, and assessment participation intent. Intentionally include at least 40 people with limited production/tool access and 40 career changers or early-career candidates, plus purposive accessibility interviews. The target supports burden, comprehension, access, and consequence research; it does not support definitive subgroup fairness claims. Protected-attribute subgroup sizes are determined by privacy and precision review, not public quota promises.

Recruitment. Use professional communities, workforce organizations, accessible research panels, career-transition groups, and a separately consented subset of platform visitors. No active employer can condition an application on research participation. Decliners and abandoners receive a short, optional, non-coercive follow-up path.

Measures and questionnaire modules. Construct comprehension; expected benefit and risk; evidence access; privacy comfort; reference feasibility; tool/device/language access; assessment time and financial burden; accommodation needs; assistance policy; result/uncertainty comprehension; appeal/redress; willingness to participate or withdraw; scenario-based consequence judgments.

Analysis. Estimate completion, abandonment, item nonresponse, comprehension, burden, and perceived coercion with denominators. Model stage-specific dropout as an outcome rather than removing it. Compare wording and workflow variants only under randomized or otherwise justified designs. Report accommodation and language findings with privacy-protective aggregation and qualitative context.

Validity, bias, and nonresponse. Risks include survivor bias, platform-user bias, social desirability, fear that criticism harms employment, and undercoverage of people excluded by technology. Mitigations include independent recruitment, anonymous response, explicit non-impact notice, mobile/low-bandwidth and assisted modes, abandonment follow-up, and oversampling of access-constrained perspectives. Willingness to take the benchmark is not evidence of capability.

Privacy and ethics. Candidate research data is never visible to employers or used in scoring. Declining or withdrawing has no hiring consequence. Sensitive demographics, disability/accommodation data, and linkage consent are separated. Research participation cannot be bundled with training, placement, or assessment terms.

Publication receipts. Recruitment separation audit; participant notice and comprehension test; workflow exposure/version; completion funnel; burden and abandonment report; accommodation/accessibility review; privacy and coercion review; and affected-party disposition log.

Status. Proposed; no candidate data admitted.

9.6 FDE-SRV-REF-001: references and managers

Research questions

1. Can structured references provide reliable, claim-specific corroboration beyond self-report and work samples?
2. Which source relationships and observation opportunities support which claims?
3. How do nomination, conflict, recency, fear of retaliation, and organizational policy affect response and rating?
4. Do multiple sources add information without turning network prestige or popularity into capability?
5. What correction and contestability process is required for adverse or disputed reference evidence?

Population and sampling frame. Adults with direct, recent opportunity to observe a consenting subject's relevant work, such as a manager, peer, customer-side collaborator, or direct report. Recruiters, friends, and public endorsers do not qualify unless they meet the same observation rule. The unit is the referee-subject relationship; analyses cluster by referee and subject.

Planned quotas and target. Target N=200 eligible linked manager/reference records covering at least 120 unique consenting subjects. Seek at least 60 subjects with two independent source perspectives where feasible. Coverage targets include managers, peers, customer collaborators, and direct reports; relationship duration; recency; same-employer versus cross-boundary observation; and subject-nominated versus independently identified with consent. These targets support feasibility and preliminary source-variance estimation, not a universal reference requirement or definitive reliability coefficient.

Recruitment. Subjects provide separate consent for contact and identify the claims a source may address. Independent-source identification, if studied, requires explicit subject permission and a lawful contact basis. Invitations disclose sponsor, purpose, confidentiality limits, subject rights, and whether the subject may see a summary. Employer policy refusals are valid dispositions, not negative evidence.

Measures and questionnaire modules. Source relationship; duration/recency; direct observation opportunity; conflict and incentive; claim-specific behavior anchors; critical incident; artifact familiarity; confidence and basis; inability to observe; organizational constraints; consent for recontact. Do not ask for a global popularity or “would you rehire?” score as a capability proxy.

Analysis. Estimate source, subject, claim, and context variance using generalizability or multilevel models when the design supports it. Report inter-source agreement with confidence intervals and distinguish agreement from reliability. Compare nominated and other approved sources for missingness and distribution. Test incremental value over direct work evidence on held-out criterion data before references influence any result. Disputed statements remain flagged, not averaged away.

Validity, bias, and nonresponse. Risks include leniency, retaliation, fear of legal exposure, nomination bias, shared-employer halo, network inequality, policy-driven nonresponse, and limited observation. Standardized job-related prompts, anchored observation categories, source conflict disclosure, multi-source designs, explicit “not observed,” and independent criterion testing mitigate but do not eliminate these risks. Reference availability never becomes a low capability score.

Privacy and ethics. Both subject and referee receive understandable notice. The subject can contest attribution, identity, or process without editing the referee's original record. Raw narratives are controlled; employers do not receive research responses. Safety, harassment, or unlawful-conduct disclosures follow a prospectively approved escalation policy, not ad hoc product handling.

Publication receipts. Subject/referee consent versions; contact authority; relationship/observation rubric; source-composition and nonresponse; clustering map; conflict distribution; agreement/reliability plan and results; disputed-record flow; incremental-validity analysis; and data-access limits.

Status. Proposed; no references contacted.

9.7 FDE-LONG-EMP-001: longitudinal employer outcomes

Research questions

1. Do frozen benchmark observations predict independently measured, job-relevant outcomes over time for the exact role and context studied?
2. Does the benchmark add information beyond less burdensome alternatives such as structured experience records or direct work samples alone?
3. How do organizational opportunity, assignment difficulty, team quality, manager behavior, and access affect observed outcomes?
4. Are prediction, error, burden, and consequences materially different across groups, accommodations, role contexts, or employers?
5. Does providing benchmark information change employer decisions or outcome ratings, creating criterion contamination?

Population and sampling frame. Consenting adults entering or currently performing a prespecified FDE role in participating organizations, with an independently defined outcome source and adequate observation window. The intended-use population is frozen before recruitment. Incumbent and applicant designs are analyzed separately.

Planned quotas and target. Initial feasibility target N=100 linked cases across at least 10 organizations, with baseline benchmark observation and outcomes at approximately 6 and 12 months. This cohort estimates recruitment, linkage, criterion availability/reliability, clustering, range restriction, and attrition parameters. It does not authorize a predictive-validity or fairness claim. A later criterion-validation target must be set through multilevel simulation using the feasibility estimates and the smallest practically important incremental relationship. No organization should contribute more than 15% of the feasibility sample without a preregistered sensitivity plan.

Recruitment. Organizations are recruited independently of result favorability and sign a research firewall: no access to individual survey responses; no influence over analysis or suppression; no removal of unfavorable results; and no employment consequence for declining. Participants separately consent to assessment research and employer-data linkage. Employer criterion raters remain blind to benchmark results where feasible.

Measures and questionnaire modules. Baseline role/context and opportunity; benchmark task/evidence profile under a frozen release; independent structured manager/customer ratings; governed project outcomes; reliability, security, incident, adoption, rework, documentation, and capability-transfer indicators; assignment difficulty; team and organization covariates; exposure to benchmark results; accommodations; employment changes; attrition reasons.

Analysis. Freeze primary outcomes and temporal order. Use multilevel models with participants nested in managers/teams/organizations as supported. Report criterion reliability, missingness, attrition, censoring, range restriction, calibration, effect sizes, uncertainty, and prediction intervals. Compare prespecified baseline models with incrementally richer models on held-out organizations or nested cross-validation. Evaluate sensitivity to criterion contamination, common-source method, restricted range, and organizational opportunity. No causal claim follows from association alone.

Validity, bias, and nonresponse. Key threats are employer self-selection, selective hiring, range restriction, criterion deficiency/contamination, manager halo, visibility differences, turnover, informative attrition, and score disclosure. Mitigations include multiple organizations, independent criterion construction, blinded raters, multi-source outcomes, baseline covariates, attrition follow-up, inverse-probability and pattern-mixture sensitivity where justified, organization-held-out validation, and replication. A favorable result in one role or customer segment does not generalize automatically.

Privacy and ethics. Research participation and linkage are voluntary and separated from employment decisions. Employers receive only governed aggregate findings unless a later, separately authorized operational use exists. Linkage keys are held by a limited trusted function separate from analysts; research files use pseudonymous IDs. Termination, performance management, disability, health, and protected information require heightened minimization and access control. Withdrawal and deletion limits are clear before consent.

Publication receipts. Organization protocol/firewall; participant and employer consent; benchmark release freeze; criterion-development record; blinding check; baseline and follow-up flow; attrition and missingness; cluster structure; model/code/environment freeze; held-out validation receipt; fairness/consequence review; sponsor/COI statement; and explicit transportability boundary.

Status. Proposed; no employers enrolled and no outcome data exists.

10. Instrument pilot, calibration, confirmation, fairness, and monitoring

10.1 FDE-PILOT-001: technical and administration pilot

Plan N=60 participants across at least six material contexts after cognitive pretesting. Test routing, accessibility, secure artifact handling, task solvability, assistance declaration, instrumentation, evidence provenance, invalid-state behavior, time/burden, and withdrawal. Pilot outputs can establish feasibility and defects only. They cannot establish norms, validity, fairness, or a pass threshold.

10.2 FDE-RATER-001: rater calibration and generalizability pilot

Plan N=120 participants and 24 trained raters. Each scored response receives at least two independent ratings; a randomized 25% receives a third rating. Assignment is balanced across task, form, declared assistance, participant context, and rater. Training cases remain separate from analysis cases. Original ratings are retained after adjudication.

Estimate candidate, task, rater, candidate-by-task, candidate-by-rater, and residual variance where the design permits. Report agreement and reliability statistics matched to the scale, confidence intervals, decision consistency around any experimental boundary, drift, severity/leniency, and missingness. A many-facet model may be explored only if design connectivity and fit support it; the model must not erase substantive rubric disagreement. AI evaluator output, if studied, is another rater signal and cannot be its own ground truth.

10.3 FDE-CAL-001 and FDE-CONF-001: measurement calibration and confirmation

Calibration and independent confirmation sample sizes are not yet set. They require simulation after the rater pilot establishes task completion, missingness, variance components, alternate-form connectivity, and subgroup coverage. The 250-person practitioner formation cohort and 150-person adjacent comparator cohort are not silently relabeled as calibration or confirmation samples after data inspection. Calibration may examine dimensionality, task information, alternate forms, scoring invariants, missingness rules, and provisional aggregation. Confirmation tests frozen structure, reliability, decision consistency, subgroup behavior, and prespecified relationships without re-estimating every choice on the same people.

The calibration and confirmation samples must not overlap at the person level. Task exposure and training contamination are tracked. Exploratory factor analysis stays in calibration; confirmatory factor, item-response, or generalizability models use the held-out plan and report misfit. If the proposed model does not fit, the result is model failure, not participant failure.

10.4 FDE-CONSEQ-001: consequence and fairness program

Fairness is evaluated from entry into recruitment through interpretation and use. The study includes:

• coverage and opportunity-to-demonstrate by role context;
• accessibility and accommodation equivalence;
• language, device, bandwidth, time, tool, and paid-resource effects;
• differential item/task functioning and measurement invariance where sample/design supports them;
• differential prediction, calibration, false-positive and false-negative consequences;
• completion, abandonment, holds, appeals, reversals, and time to resolution;
• intersectional analysis subject to privacy and precision limits;
• qualitative evidence from people who decline, abandon, dispute, or are adversely affected; and
• comparison with less burdensome or less exclusionary alternatives.

Protected attributes are never scoring inputs. Statistical parity alone is not fairness, and a nonsignificant test is not proof of equivalence. The plan defines minimum detectable differences, multiplicity control, shrinkage or partial pooling where justified, small-cell suppression, and qualitative escalation. If a subgroup estimate is too imprecise to support a claim, the claim remains unresolved.

Operational study stop conditions include material unexplained subgroup error, inaccessible administration without an equivalent path, inability to explain or contest a decision, evidence of employer retaliation, material privacy breach, or a scoring model that uses opportunity/prestige as capability.

10.5 FDE-TRACK-001: annual state-of-field tracking

After the core role construct and instrument are stable enough for monitoring, plan an annual repeated cross-sectional sample plus an opt-in longitudinal panel. No annual target is authorized yet. The first-wave allocation must follow a documented frame-coverage, precision, design-effect, subgroup-privacy, and attrition simulation; later refreshment and panel targets use observed prior-wave parameters.

The tracker uses:

• a stable core task/work-context module;
• rotating modules for emerging tools, regulation, delivery models, and failure modes;
• a version bridge sample when wording, mode, taxonomy, or language changes;
• independent refreshment samples to separate cohort change from panel conditioning;
• role/context and recruitment-source composition reporting;
• preregistered trend estimands and discontinuity flags;
• instrument equivalence testing before trend claims; and
• public correction/version history.

Annual tracking is not a leaderboard and cannot support “the average FDE” claims unless the sampling frame warrants them. Tool adoption or title growth does not redefine capability automatically. Material role drift reopens job analysis and content review.

11. Cross-study analysis, bias, and quality controls

11.1 Analysis principles

1. Define the estimand, unit, denominator, missingness, exclusions, and uncertainty before analysis.
2. Report unweighted counts and distributions before justified weighting.
3. Separate exploratory, calibration, confirmation, validation, and monitoring data.
4. Use effect sizes and uncertainty, not a significance/no-significance binary.
5. Preserve ordinal meaning unless a validated model supports transformation.
6. Adjust for clustering and repeated measures.
7. Control researcher degrees of freedom through preregistration, frozen code, and amendment logs.
8. Correct or hierarchically model multiplicity for families of confirmatory tests; label unadjusted exploration.
9. Report nulls, reversals, unstable models, failed tasks, and adverse findings.
10. Do not infer causality from observational relationships without an appropriate design.

11.2 Nonresponse and selection

Every recruitment record receives an AAPOR-compatible disposition where applicable. Reports provide response rates for probability samples and participation rates for nonprobability samples, plus eligibility, completion, breakoff, and recontact rates. Because response rate alone does not measure nonresponse bias, analyses also consider:

• frame variables for respondents and nonrespondents where lawful;
• early/late and low/high-contact comparisons;
• recruitment-channel and invitation-version effects;
• follow-up with a small optional refusal/breakoff instrument;
• inverse-probability or calibration adjustments only when their assumptions and external targets are defensible; and
• worst-case, tipping-point, or pattern-mixture sensitivity for material missing outcomes.

Unknown eligibility is not silently treated as ineligible. Item missing, skip-by-design, not applicable, not observed, refusal, technical failure, invalid response, and withdrawn data remain distinct states.

11.3 Survey-quality and integrity controls

• Versioned questionnaires, routing, translations, screenshots, mode, dates, and duration.
• Duplicate and eligibility review using minimal, privacy-preserving signals.
• Attention and consistency checks designed not to punish disability, language, or thoughtful response.
• Speeding and straight-line patterns route to manual review rather than automatic exclusion.
• Collector/interviewer training, observation, feedback, and drift review.
• Random verification of recruitment and consent receipts.
• Predeclared exclusion reasons and blinded review where feasible.
• Detection of coordinated or synthetic responses without converting fraud signals into capability.
• Full inclusion flow and excluded-record counts by reason.

11.4 Qualitative rigor

Qualitative studies publish researcher roles, positionality/conflicts, sampling rationale, setting/mode, interviewer guide and training, recording/transcription method, coding framework, double-code selection, reliability/agreement evidence, adjudication, negative cases, reflexive decisions, quotations/permissions, and limits. At least 20% double coding is the program minimum for benchmark-content decisions, with higher review for safety-critical, novel, disputed, or removal decisions.

11.5 Triangulation rule

No ontology or assessment decision is “confirmed” because multiple respondents repeated it. A proposed decision must show:

• job-analysis support and contradiction;
• structured rating evidence with context;
• adjacent-role boundary evidence;
• SME panel disposition and dissent;
• candidate/accessibility challenge;
• traceability to an observable work product or response; and
• a governed decision with explicit uncertainty and excluded uses.

12. Privacy, ethics, accessibility, and participant rights

12.1 Applicability and review

Before any study opens, an authorized ethics/privacy authority documents jurisdiction, study purpose, human-participant applicability, review path, consent basis, recruitment authority, data transfer, compensation, vulnerable-population considerations, and adverse-event escalation. LockedIn does not claim IRB approval, exemption, or regulatory status until the responsible authority records it.

Initial studies enroll adults only. Research with minors, incarcerated people, people lacking consent capacity, or other populations requiring additional protections requires a separate protocol and authority.

12.2 Consent architecture

Consent is prospective, understandable, accessible, and specific. Separate choices cover:

• participation;
• audio/video recording;
• attributed or deidentified quotation;
• future contact;
• employer or reference contact;
• longitudinal linkage;
• controlled data sharing; and
• future materially different research use.

Participation cannot be bundled with employment, assessment, training, placement, customer access, or a credential. The withdrawal window and limits after aggregate freeze are stated before participation.

12.3 Data separation and minimization

• Contact/identity information is stored apart from survey, assessment, and outcome data.
• Pseudonymous keys do not encode name, email, employer, or social handle.
• Protected attributes and accommodation data are separated from scoring and rater views.
• Employer/customer identifiers are controlled and generalized for publication.
• No credentials, customer source code, confidential architecture, proprietary prompts, health data, or identifiable third-party information are requested.
• Raw recordings, narratives, and linkable work artifacts are controlled; the public repository receives instruments, schemas, synthetic fixtures, aggregate tables, and disclosure-approved excerpts only.
• Retention, correction, deletion, access logging, breach response, and data-transfer rules are defined by artifact class.
• Participant data is not sent to external generative-AI services or used to train a model by default. Any future AI-assisted coding/evaluation requires a separately approved data path, disclosed provider/version, human review, and no silent reuse.

12.4 Accessibility and language

Research modes support keyboard and screen-reader use, zoom/reflow, captions/transcripts, low-bandwidth access, pause/resume, flexible timing, and an assisted non-digital path where feasible. Translations use forward translation, expert review, cognitive testing, and version bridges rather than unreviewed machine translation. An accommodation is studied for construct preservation; it is not treated as a favor or an invalidation.

12.5 Participant remedy

Participants receive a contact path for questions, correction, withdrawal, privacy concerns, accessibility issues, and complaints. Adverse records can be disputed. Corrections append to the audit trail. Research findings about groups do not become unexplained person-level labels.

13. Publication receipts and claim control

Each study follows the existing state sequence:

claim_received -> evidence_requested -> evidence_received -> chain_of_custody_verified -> record_validation_complete -> coding_complete -> analysis_frozen -> independent_review_complete -> aggregate_substantiated -> publication_authorized -> published

Corrections, suspensions, withdrawals, and retractions remain append-only states. A draft document or completed data collection does not skip a state.

13.1 Common receipt package

Every public cohort result must link to:

1. permanent study ID, protocol version, amendment history, preregistration timestamp, and study-register state;
2. accountable owner, authors/affiliations/ORCIDs where available, CRediT roles, independent reviewer, funding, sponsor role, conflicts, recusals, and AI-use disclosure;
3. ethics/applicability determination, consent versions, participant-rights path, compensation, and recruitment authority;
4. target population, sampled population, frame coverage, sampling method, recruitment channels/messages, field dates, modes/languages, incentives, dispositions, response/participation/breakoff rates, and nonresponse analysis;
5. planned sample-size memo, achieved flow, exclusions, duplicates, missingness, attrition, cluster structure, and unweighted denominators;
6. questionnaire/interview/rubric versions, translations, screenshots or rendered archive, routing, cognitive-test report, and change log;
7. role-eligibility, interviewer/rater/coder qualifications, training, assignments, blinding, conflicts, reliability/agreement, drift, and adjudication;
8. frozen eligible-record manifest, data dictionary, lineage, input/output digests, analysis plan, code, environment, package lock, random seeds, and reproduction instructions;
9. weighting/model assumptions, effective sample size, uncertainty, sensitivity, multiplicity, subgroup/fairness/accessibility, privacy/disclosure, and consequence analyses;
10. nulls, negative cases, broken instruments/tasks, deviations, incidents, unresolved disagreements, and claim limitations;
11. data/materials availability class, controlled-access process or reason for nonrelease, retention/deletion, and synthetic reproduction fixture where needed;
12. signed aggregate claim receipt mapping exact public wording and numbers to frozen outputs;
13. independent methods, measurement, fairness/accessibility, privacy/security, ethics, legal/use, and disclosure review appropriate to the claim;
14. canonical release URL, persistent identifier/DOI where available, immutable archive, suggested citation, license, correction channel, and superseded-version links.

13.2 Cohort-specific receipts

The cohort sections above add receipts for their unique unit, frame, conflicts, and analyses. No public headline may combine cohorts unless the aggregation rule, overlap/deduplication, and distinct denominators are published. “Interviewed 500 practitioners and 50 executives,” for example, requires exactly reproducible eligible-person counts and cannot be inferred from invitations, calls, sessions, excerpts, invoices, or target numbers.

13.3 Publication language by evidence state

14. Governance and independence

14.1 Required separations

• Benchmark content authors do not solely approve their own content evidence.
• Training, placement, recruiting, and services revenue have no authority over participant results or research publication.
• Employer sponsors cannot suppress unfavorable results or access person-level research responses.
• The measurement lead cannot alone approve fairness, ethics, privacy, legal/use, or certification claims.
• AI systems may assist administration or analysis only under versioned, tested, disclosed human authority; they do not approve a benchmark release.

14.2 Decision bodies

14.3 Annual review and revalidation triggers

At least annually, review role drift, sample coverage, instrument performance, raters, tools/assistance, accommodations, subgroup error, employer use, appeals, security/privacy incidents, contamination, and public interpretation. Immediate revalidation or suspension triggers include:

• material change in construct, population, use, task, rubric, scorer, or administration;
• significant unexplained subgroup error or burden;
• evidence of criterion contamination or employer misuse;
• rater drift beyond the preregistered hold rule;
• security, privacy, identity, or data-lineage compromise;
• task leakage or widespread coaching that changes the response process;
• sustained appeal reversal indicating a system defect; or
• inability to reproduce a material public claim.

15. Activation sequence

1. Appoint accountable research, measurement, ethics, privacy, fairness/accessibility, and independent-review roles.
2. Record ethics/applicability, recruitment, data-protection, compensation, and participant-rights decisions.
3. Register FDE-JA-002 prospectively; conduct the job-analysis interviews and structured task inventory.
4. Convene FDE-DELPHI-001 after admissible job-analysis outputs exist.
5. Freeze candidate constructs, task inventory, and source-to-blueprint map.
6. Build audience-specific instruments from Survey Instrument Modules.
7. Complete two cognitive-interview rounds and resolve all critical item failures.
8. Field cohort surveys with independent sampling and published dispositions.
9. Run technical and rater pilots; hold defective tasks and rules.
10. Separate calibration from confirmation; publish model misfit and uncertainty.
11. Begin criterion validation only after criteria, blinding, employer firewalls, and outcome linkage pass review.
12. Complete consequence and fairness research before any high-stakes operational use.
13. Publish only through the study-register and aggregate-receipt gates.
14. Start annual tracking after a stable core exists; version-bridge every material change.

16. Current blockers

• No named accountable research or independent-review roster.
• No approved human-participant, consent, privacy, recruitment, compensation, or data-linkage protocol.
• No acquired or evaluated sampling frames.
• No instrument has been cognitively tested.
• No admitted prospective job-analysis evidence or SME panel.
• No real-person pilot, calibration, confirmation, criterion, fairness, or consequence dataset.
• No approved rater corps, training package, criterion system, employer firewall, or appeal operation.
• No authority for an official score, threshold, certification, hiring use, or participant-facing claim.

These are activation gates, not cosmetic disclosures. The program can be reviewed and improved now; fieldwork begins only after the relevant authority and receipts are recorded.

17. References

• American Association for Public Opinion Research. Disclosure Standards. Current page reviewed 2026-09-01. https://aapor.org/standards-and-ethics/disclosure-standards/
• American Association for Public Opinion Research. Standard Definitions. Current page reviewed 2026-09-01. https://aapor.org/standards-and-ethics/standard-definitions/
• American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. https://www.testingstandards.net/
• Centers for Disease Control and Prevention, National Center for Health Statistics. Collaborating Center for Questionnaire Design and Evaluation Research. Current page reviewed 2026-09-01. https://www.cdc.gov/nchs/ccqder/index.html
• HHS Office for Human Research Protections. Common Rule and Informed Consent FAQs. Current pages reviewed 2026-09-01. https://www.hhs.gov/ohrp/regulations-and-policy/regulations/common-rule/index.html and https://www.hhs.gov/ohrp/regulations-and-policy/guidance/faq/informed-consent/index.html
• Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017). Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine, 31(8), 684–706. https://doi.org/10.1177/0269216317690685
• O'Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for Reporting Qualitative Research: A Synthesis of Recommendations. Academic Medicine, 89(9), 1245–1251. https://doi.org/10.1097/ACM.0000000000000388
• Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A Meta-Analysis of Work Sample Test Validity: Updating and Integrating Some Classic Literature. Personnel Psychology, 58(4), 1009–1037. https://doi.org/10.1111/j.1744-6570.2005.00714.x
• Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range. Journal of Applied Psychology, 107(11), 2040–2068. https://doi.org/10.1037/apl0000994
• Society for Industrial and Organizational Psychology. (2018). Principles for the Validation and Use of Personnel Selection Procedures (5th ed.). https://doi.org/10.1017/iop.2018.195
• STROBE Initiative. STrengthening the Reporting of OBservational Studies in Epidemiology. Current page reviewed 2026-09-01. https://www.strobe-statement.org/
• Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated Criteria for Reporting Qualitative Research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6), 349–357. https://doi.org/10.1093/intqhc/mzm042
• U.S. Equal Employment Opportunity Commission. Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures. Current page reviewed 2026-09-01. https://www.eeoc.gov/laws/guidance/questions-and-answers-clarify-and-provide-common-interpretation-uniform-guidelines
• U.S. Office of Management and Budget / National Center for Health Statistics. Standards and Guidelines for Cognitive Interviews. Current page reviewed 2026-09-01. https://wwwn.cdc.gov/qbank/learn/CI-standards.aspx
• U.S. Office of Personnel Management. Job Analysis. Current page reviewed 2026-09-01. https://www.opm.gov/policy-data-oversight/assessment-and-selection/job-analysis/