Research operations runbook

Record ID: LL-FDE-RESEARCH-OPS-001
Version: 0.1.0
Status: proposed operating system; no recruitment, data collection, completed independent review, archive deposit, DOI, or empirical result is authorized
Evidence cutoff: September 1, 2026

1. Purpose

This runbook converts the benchmark’s publication and research standards into work that can be assigned, inspected, and signed. It covers the path from an open public draft to a citable archive, then from a preregistered job analysis to a controlled assessment pilot and an outcome-linkage study.

The controlling rule is simple: an artifact is not a completed activity. A reviewer template is not a review, a consent draft is not consent, a repository upload is not a verified archive, and a passing schema test is not research evidence.

2. Current baseline

3. Workstream order

Stage A — seat accountable roles

1. Appoint the publication owner, publication guarantor, research director, measurement lead, data steward, participant-rights owner, privacy/security owner, accessibility reviewer, archive owner, and release authority.
2. Appoint each independent reviewer through the Reviewer Appointment and Review Protocol.
3. Publish only approved names, affiliations, CRediT roles, conflicts, recusals, and review scopes.

Exit receipt: completed review/appointment records plus an approved authors-and-contributors record. No person may be inferred from a commit, job title, or repository access.

Stage B — make the public draft citable

1. Freeze operative code, content, and data licenses and third-party rights review.
2. Complete funding, sponsor-role, conflict, AI-use, materials, and review disclosures.
3. Activate and test a public corrections route and private security and participant/conduct routes.
4. Freeze the release manifest and complete the Archive and DOI Deposition Packet.
5. Verify the identifier, downloaded archive, checksums, metadata, canonical links, and preservation ownership.
6. Sign the publication decision for the exact frozen bytes.

Exit receipt: a verified archive-deposition receipt and signed publication decision. Search indexing can be considered only after this gate, not merely after a deployment.

Stage C — authorize job-analysis fieldwork

1. Freeze the target population, sampling frame, strata, sample-size rationale, replacement and stopping rules.
2. Freeze recruitment, consent, accessibility, instruments, codebook-development method, analysis plan, security, retention, incident, and participant-support plans.
3. Obtain the applicable ethics/human-participant and legal determinations from accountable authorities.
4. Complete independent methods, privacy/security, accessibility, and legal-use reviews.
5. Register the immutable protocol prospectively.
6. Issue a fieldwork-authorization-receipt for the exact protocol and approved activities.

Exit receipt: authorization_state: authorized, all required gates passed, named roles accepted, no blockers. Until then, do not recruit, screen, consent, contact, interview, survey, collect a reference, or collect an outcome.

Stage D — execute and publish the job analysis

1. Use the frozen recruitment and consent materials without silent variation.
2. Maintain pseudonymous disposition, session, artifact, withdrawal, and deviation records.
3. Train coders on the frozen unitization and codebook rules; double-code the preregistered sample independently.
4. Calculate the preregistered agreement evidence, retain uncertainty and disagreement, and follow the declared action rule.
5. Produce the evidence-to-blueprint traceability matrix, negative-case analysis, sampling flow, and reproducible analysis package.
6. Authorize each public count or finding through aggregate and publication-claim receipts.

Exit receipt: study state publication_authorized and signed claim receipts. This supports bounded job-analysis wording, not a score or hiring prediction.

Stage E — controlled assessment pilot

1. Select an initial subset of capability families from the approved content blueprint.
2. Freeze cases, forms, conditions, permitted assistance, accommodations, evidence capture, scoring rubrics, rater assignments, adjudication, and stop rules.
3. Qualify and calibrate raters. Keep scoring experimental and participant-visible.
4. Recruit a coverage-oriented pilot under its own protocol and consent.
5. Evaluate response processes, task functioning, reliability/agreement, rater drift, burden, accessibility, missingness, security, subgroup/context variation, and decision consistency.

Exit receipt: pilot findings release with no operational credential. Any score or profile remains experimental until its precise interpretation has sufficient evidence.

Stage F — prospective outcome linkage

1. Define independent outcomes before viewing benchmark results.
2. Establish temporal ordering, consent, data separation, access, attribution limits, employer non-retaliation, and withdrawal handling.
3. Predeclare models, missingness, selection effects, range restriction, subgroup/context analyses, sensitivity analyses, and replication conditions.
4. Keep assessment raters blind to outcomes and outcome raters blind to assessment results where feasible.
5. Replicate before making broad predictive or hiring-use claims.

Exit receipt: an independently reviewed relation-to-outcomes report. A statistically nonzero association alone does not authorize pass/fail, ranking, certification, or automated hiring.

4. Required operational records

5. Weekly operating review

The research director and publication owner should review this fixed agenda:

1. newly opened, changed, or expired authorities;
2. recruitment and participant-rights incidents;
3. deviations, exclusions, missingness, withdrawals, and data-quality holds;
4. coder/rater calibration and drift status;
5. unresolved reviewer findings and conflicts;
6. public claims approaching expiry or affected by new evidence;
7. archive, link, DOI, correction-route, and monitoring health; and
8. explicit stop, suspend, correct, or escalate decisions.

Minutes must record owners, deadlines, evidence references, recusals, dissent, and the exact decision. A meeting without an append-only decision record changes no authority state.

6. Source foundation and boundary

This operating system is informed by the Standards for Educational and Psychological Testing, the SIOP Principles, OPM job-analysis guidance, AAPOR disclosure standards, COREQ, SRQR, OHRP consent guidance where applicable, Census questionnaire-testing guidance, Krippendorff’s reliability recommendations, NIST measurement and identity guidance, MLCommons operating rules, and COPE retraction guidance. The Source Ledger controls the exact reviewed sources and non-adoption boundaries.

No cited organization has reviewed, endorsed, accredited, or validated the LockedIn FDE Benchmark.