Controlled assessment pilot sampling and analysis plan

Plan ID: LL-FDE-PILOT-001
Version: 0.1.0
Status: proposed design; no pilot sample, task form, participant, administration, rater, result, threshold, or official assessment is authorized

1. Pilot purpose

The first pilot should answer whether a bounded set of applied FDE tasks can be administered, understood, observed, rated, appealed, and reproduced as intended. It is not designed to certify people, rank candidates, estimate a population pass rate, or prove hiring validity.

The pilot begins only after the job analysis produces an approved content blueprint. The current 55 family dossiers are design candidates, not evidence that every family belongs in an assessment.

2. Staged pilot program

Each stage has a separate protocol, consent, fieldwork receipt, stopping rules, publication, and integrity monitoring. Later-stage authorization cannot be inferred from an earlier stage.

3. First task portfolio

Use approximately 10–15 capability families only if the approved blueprint and feasibility review support that range. The exact count remains unresolved. Selection should cover a coherent deployment episode rather than a checklist:

1. problem framing and acceptance criteria;
2. system/data/constraint discovery;
3. architecture and integration judgment;
4. hands-on implementation or technical diagnosis;
5. evaluation, security, or production-readiness evidence;
6. change, failure, or ambiguity response;
7. stakeholder decision and evidence communication; and
8. outcome ownership and product/field feedback.

For each candidate family, record job-evidence support, consequence, proposed task, observable behavior, artifact, rubric, alternate demonstration, access burden, contamination risk, security/privacy risk, and reasons to include or hold it.

Avoid a portfolio dominated by coding merely because it is easy to automate, or by presentation merely because it is easy to observe.

4. Forms and assignment

• Define task forms and content overlap before data collection.
• Randomize or counterbalance form/order only when the design and sample can support the inference.
• Record environment, model/tool access, assistance, time, interruptions, accommodation, and change injection.
• Use common anchor material where needed to study form/rater comparability.
• Do not equate forms by label or intended difficulty.
• Protect sequestered cases; monitor exposure and retire compromised material.
• Preserve attempts and failures that pass quality rules; do not rerun selectively for favorable output.

5. Pilot population and sampling

The target population is defined by observable experience and intended future use, not job title. Candidate strata include:

• current hands-on FDE work;
• adjacent engineering/product/solutions roles with overlapping work;
• responsibility/autonomy level;
• domain and regulated/high-consequence context;
• delivery model and organization stage;
• geography/language where the instrument is authorized;
• accessibility/accommodation path; and
• varied access to expensive tools, models, cloud, and public portfolios.

The sampling plan must distinguish:

• design strata required for a primary comparison;
• monitoring variables used to inspect coverage;
• convenience channels and resulting limitations;
• repeat/known participants versus fresh participants;
• trained/affiliated participants versus unaffiliated participants; and
• completed, partial, invalid, withdrawn, and held administrations.

No sample-size target is supplied here. Before registration, justify it against the primary estimands, rater/task design, desired uncertainty, subgroup/context coverage, expected missingness, and burden. The stopping rule cannot depend on a desirable score distribution.

6. Administration and participant protections

Pilot consent states prominently:

• experimental status and no credential/hiring consequence;
• tasks, duration, tools, recording/telemetry, data retention, and AI-assisted processing;
• confidentiality limits and rules for employer/customer/private material;
• acceptable assistance and security restrictions;
• compensation;
• accessibility/accommodation and alternate-demonstration paths;
• feedback, withdrawal, correction, incident, and appeal routes; and
• who will see individual evidence and results.

Participants must use synthetic or authorized sandbox material. The platform must block or warn against secrets, production credentials, identity documents, protected health information, personal data, confidential customer artifacts, and proprietary code not authorized for the study.

7. Preregistered outcomes

Primary outcomes should be limited and stage-specific. Candidate classes include:

• completion/invalid/withdrawal rate and reason;
• task and item missingness;
• instruction comprehension and response-process evidence;
• time/burden and environment failure;
• rater agreement/reliability/decision consistency with uncertainty;
• rater severity, drift, task/form, and criterion effects;
• evidence coverage and insufficient-evidence frequency;
• rubric category use, boundary confusion, and adjudication;
• task security, contamination, and assistance deviations;
• accessibility/accommodation performance and reported barriers; and
• context/subgroup variation appropriate to the approved sample and privacy plan.

Do not predeclare “top 5%,” a pass rate, universal readiness, or a cutoff as a pilot outcome.

8. Analysis plan

Freeze:

1. unit of analysis and repeated-attempt treatment;
2. primary/secondary/exploratory outcomes;
3. task/rater/form assignment model;
4. missing, invalid, withdrawn, interrupted, accommodated, and contaminated attempt treatment;
5. reliability/agreement/precision estimands and uncertainty;
6. rater/task/form/context model assumptions and diagnostics;
7. fairness and accessibility questions, minimum cell protection, and non-inferential small-sample reporting;
8. sensitivity and robustness analyses;
9. decision-simulation rules, only if authorized;
10. multiplicity and researcher-degree-of-freedom controls; and
11. software/environment, seeds, manifests, and reproduction instructions.

Separate a missing opportunity to demonstrate from observed weak performance. A one-dimensional score is not the default; publish capability/evidence profiles and uncertainty before considering aggregation.

9. Pause and stop rules

Immediately hold affected activity for:

• consent/rights failure, coercion, retaliation, confidentiality breach, or unauthorized data;
• task defect that makes the intended capability unobservable;
• inaccessible path without an approved alternative;
• security compromise, case leakage, or prohibited assistance;
• scoring/rater defect capable of changing material interpretations;
• platform failure causing unequal conditions; or
• unexpected participant risk.

The methods lead also predeclares statistical or evidence-quality stop/revise rules. Paused data remain quarantined until an authorized disposition; operators cannot delete inconvenient attempts.

10. Rater and adjudication system

Apply the Rater Qualification, Calibration, Agreement, and Drift Protocol. Every criterion-level rating cites evidence and permits insufficient evidence, not applicable, and invalid task separately from performance levels.

Adjudication preserves original ratings. Participant appeals are handled by a different process and cannot be decided solely by the original rater.

11. Release and progression decision

Publish method, sampling flow, task coverage, participant protections, deviations, failures, burden, rater evidence, accessibility/fairness analysis, uncertainty, limitations, and claim receipts. Controlled task details may be withheld with a concrete reason and an independent-review path.

The stage decision is one of:

• stop;
• revise and repeat the same stage;
• continue to a specified later stage with conditions; or
• hold pending new evidence.

“Pilot complete” does not mean “benchmark validated.” Progression requires named methods, accessibility, privacy/security, participant-rights, and independent-review approvals for the exact next protocol.

12. Current disposition

No pilot activation gate has passed. This plan specifies what to build and learn next while preserving the current prohibition on official human scoring.