Commissioned interview intake and job-analysis protocol

Protocol ID: FDE-JA-001 Version: 0.1.0 Status: evidence intake proposed; underlying records not received Study design timing: retrospective reconstruction of previously commissioned work, with a prospective supplemental study if the historical package is incomplete Authorization: this document does not authorize recruitment, contact with participants, real-data intake, publication of interview counts, or a hiring use

1. Why this protocol exists

LockedIn Labs has reported that paid callers conducted interviews intended to include at least 500 AI practitioners and 50 executives. At the evidence cutoff for this protocol, the repository contains no participant-level interview records, recruitment disposition log, consent record, instrument, collector roster, transcript or note manifest, codebook, analysis file, or independent review that substantiates those figures.

The figures are therefore requestor-reported claims, not findings. They must not be represented on a website, in a benchmark publication, in sales material, or in a hiring policy as completed research until the release gate in this protocol passes.

This protocol provides two honest paths:

1. Retrospective intake. Receive and validate the contemporaneous materials from the commissioned work without implying that this protocol governed work already completed.
2. Prospective supplement or replacement. If the historical package is incomplete, run a newly approved job-analysis study under a preregistered protocol. Historical interviews may remain contextual input but cannot be silently upgraded into compliant study records.

Payment records, invoices, call counts, calendar entries, contact lists, or a collector's attestation may corroborate that outreach occurred. None alone proves that an eligible person completed an interview or supports a research finding.

2. Decision and claim boundaries

2.1 Research questions

The study may address:

• which observable work behaviors, decisions, artifacts, constraints, and outcomes practitioners associate with forward-deployed engineering;
• how that work varies by deployment context, product maturity, company scale, regulated environment, and executive accountability;
• which work behaviors practitioners and accountable executives judge important, frequent, difficult, consequential, and required at entry;
• where practitioners and executives agree, disagree, or use different terminology; and
• which proposed benchmark capabilities require revision, addition, removal, or separate domain treatment.

2.2 What the interview study cannot establish

Interviews alone cannot establish:

• that the benchmark predicts job performance or hiring success;
• a pass score, percentile, universal readiness threshold, or scoring weight;
• inter-rater reliability of a future assessment;
• fairness, accessibility equivalence, criterion validity, or predictive validity;
• that a title holder is an FDE or that an untitled practitioner is not;
• that interview frequency equals capability importance; or
• that the sample represents all AI practitioners, executives, companies, industries, or geographies.

Statements such as “this benchmark was informed by a documented job-analysis study” require a traceable content map and methods release. Statements such as “this benchmark identifies who will perform well” require separate criterion evidence and are outside this protocol.

2.3 Claim classes

3. Study population and cohort rules

The public count claims use people, not calls, sessions, contacts, responses, transcript files, or coded excerpts, as the unit of analysis.

3.1 Practitioner cohort

An eligible practitioner must have directly performed material work in the design, integration, deployment, operation, or customer adoption of an AI-enabled system within the declared recency window. Title alone is insufficient. The record must document:

• the work setting and recency;
• at least one observable work behavior or work product;
• whether the experience was employee, contractor, founder, independent, academic/clinical, or another declared setting; and
• the role-verification method and its limitations.

The phrase FDE practitioner may be used only for the subset meeting a separately declared FDE role definition. Otherwise the correct cohort label is AI practitioner.

3.2 Executive cohort

An eligible executive must have held material accountability for at least one of: AI strategy, deployment portfolio, budget, delivery risk, workforce/hiring, product ownership, or operational outcomes. A senior-sounding title is insufficient. The record must state the accountability basis, organizational setting, and recency.

3.3 Deduplication and cohort assignment

• Each person receives one pseudonymous participant_key across the study.
• Repeated sessions with the same person are linked and count as one participant unless a publication explicitly reports sessions.
• For the headline “500 practitioners and 50 executives,” the cohorts are mutually exclusive: a dual-role participant is assigned one preregistered primary cohort and may receive secondary tags.
• The two target figures therefore require at least 550 distinct eligible participants.
• Contractors, recruiters, callers, researchers, and LockedIn personnel are not excluded automatically, but their relationships and conflicts must be recorded and reported.

4. Required historical intake package

The commissioned-work custodian must transfer the following through an approved controlled channel. Direct identifiers must not enter the public repository.

4.1 Commission and collector records

• signed scope of work, dates, sponsor, collector organization, and payment basis;
• collector/caller roster with opaque staff identifiers, qualifications, training dates, languages, conflicts, and supervisor;
• assignment and disposition log covering every sampled contact;
• interviewer instructions, quality-control instructions, escalation rules, and version history;
• invoices or payment records linked to completed-work batches without exposing compensation publicly; and
• a custodian attestation identifying systems of record and known gaps.

4.2 Recruitment and sampling records

• target population and sampling frame;
• source of contact information and authorization to use it;
• recruitment channels, messages, dates, incentives, and follow-up limits;
• contact-level dispositions, including unknown eligibility, refusal, ineligible, duplicate, partial, complete, and unreachable;
• inclusion/exclusion rules actually applied;
• nonresponse and attrition information; and
• any quotas, substitutions, snowball/referral paths, or sponsor-supplied lists.

Convenience, referral, panel, customer, employee, or vendor samples must be named as such. A large convenience sample does not become representative because of its size.

4.3 Instrument and session records

• every discussion guide or instrument version, including translations;
• mapping from each session to the version used;
• mode, language, start/end timestamps, duration, interviewer, and completion state;
• recording state and consent, if a recording exists;
• cryptographic digest and controlled location for each audio/video file, transcript, contemporaneous note, or structured response file;
• question coverage, deviations, interruptions, and assistance; and
• participant correction, withdrawal, or deletion events.

4.4 Analysis records

• analysis plan and whether it existed before or after collection;
• codebook versions and change log;
• coder roster, training, blinding, conflicts, and assignments;
• code-level decisions, double-coded sample, disagreements, adjudications, and excluded excerpts;
• analysis scripts, data dictionary, frozen input manifest, output tables, and software environment; and
• prior reports, slides, summaries, or marketing language derived from the work.

5. Consent, ethics, and participant rights

5.1 Applicability determination

Before real records are accepted, an authorized ethics/privacy owner must document whether the activity is research involving human participants, market research, internal job analysis, or another category for each jurisdiction and institution involved. The team must not self-award an exemption or imply Institutional Review Board approval that did not occur.

Where HHS human-subject regulations apply, the governing institution and review authority determine review and consent requirements. Regardless of legal category, LockedIn requires a rights baseline:

• voluntary participation without employment or customer retaliation;
• understandable notice of sponsor, purpose, duration, data use, recording, foreseeable risks, compensation, confidentiality limits, and contacts;
• separate choices for participation, recording, deidentified analysis, attributed quotation, future contact, and future unrelated use;
• a practical withdrawal/correction path and a stated point after which aggregate withdrawal may no longer be feasible;
• accessible and language-appropriate participation paths;
• no collection of customer secrets, credentials, protected source code, or identifiable third-party information unless specifically approved; and
• no public quotation unless quote permission and disclosure review are recorded.

Consent cannot be reconstructed from silence. If historical consent scope cannot be established, the affected record is held from analysis and public counts until a lawful, non-coercive remedy is approved.

5.2 Data separation and minimization

• Identity/contact data is stored separately from research records.
• participant_key is pseudonymous and cannot encode a name, email, phone number, employer, or social account.
• Demographic or protected-attribute data, if justified, is separated from benchmark content decisions and scoring.
• Raw recordings and transcripts are controlled; the public repository receives only schemas, synthetic fixtures, aggregate manifests, and disclosure-approved excerpts.
• Retention and deletion schedules are specified by artifact class, not left indefinite.

6. Instrument reconstruction and quality

Historical instruments are frozen exactly as recovered. Researchers may annotate defects but must not edit an old instrument and present it as the one used. If different callers used different guides, each version and its session count is disclosed.

The reconstructed instrument is reviewed for:

• alignment to the declared research questions;
• leading, loaded, double-barreled, prestige-coded, or title-dependent language;
• distinguishability of observed work from belief or aspiration;
• coverage of context, constraints, outcomes, failure, and counterexamples;
• treatment of training versus entry requirements;
• accessibility, language, and modality barriers; and
• interviewer discretion and unrecorded probes.

If the instrument cannot support a finding, a larger sample does not cure the defect. Findings are narrowed or withheld.

7. Record-level completeness gate

An interview can enter a public participant count only when all of the following are true:

1. a unique, pseudonymous participant key passes duplicate review;
2. recruitment source, disposition, and eligibility decision are present;
3. primary cohort eligibility is supported by the declared role-verification rule;
4. applicable consent/version/purpose fields permit analysis and aggregate publication;
5. session date, mode, duration, language, interviewer, instrument version, and completion state are present;
6. a transcript, recording, structured response, or sufficiently detailed contemporaneous note has a digest and controlled-location receipt;
7. required instrument coverage and deviations are recorded;
8. the record is not withdrawn, deleted, disputed, quarantined, duplicated, or out of scope;
9. quality review and collector chain of custody pass; and
10. the frozen record digest validates.

A record may be useful as contextual evidence while failing this count gate. Exclusions are reported with reason codes and never silently discarded.

8. Coding and synthesis protocol

8.1 Codebook

The codebook must define each code, inclusion/exclusion rules, positive and negative examples, unit of coding, hierarchy, allowable multiple coding, and relationship to benchmark capabilities. Code changes create a new version; prior coded records remain reproducible.

Inductive themes discovered after data review are labeled exploratory. Deductive codes derived from the existing ontology are separated from new field-derived codes to avoid circularly “validating” the ontology with itself.

8.2 Coder process

• Coders disclose employment, training, sponsor, candidate, and authorship conflicts.
• Training uses records outside the reliability sample.
• At least 20% of eligible records in each primary cohort, and records spanning every interviewer and material sampling stratum, are selected for independent double coding under a frozen random-selection receipt.
• All safety-critical, disputed, novel, and benchmark-removal decisions receive a second review.
• Adjudicators do not replace the original ratings; they append a resolution and rationale.

8.3 Agreement rule

The analysis plan must preregister the statistic appropriate to each code type, its uncertainty interval, the unit of analysis, prevalence handling, and a code-level action rule. As an initial governance rule—not an empirical truth—categorical codes intended to support a public benchmark decision require Krippendorff's alpha of at least 0.80. Values from 0.67 through 0.79 may support explicitly provisional exploratory reporting only after full adjudication; values below 0.67 block the code from supporting a benchmark decision. Prevalence, sample size, confidence intervals, and raw agreement are reported alongside alpha. These conventions are informed by Krippendorff's reliability recommendations; the study must justify whether they are appropriate for the consequences and coding design.

Changing the threshold after seeing results is a protocol deviation and requires a new analysis version.

8.4 From interviews to benchmark content

Every proposed benchmark change receives a traceability entry containing:

• source cohort and eligible denominator;
• interview and excerpt identifiers, never public identities;
• supporting, contradicting, and boundary cases;
• importance/frequency/difficulty/consequence evidence where collected;
• relevant domain/context differences;
• coder agreement and adjudication status;
• expert-panel decision, conflicts, dissent, and rationale; and
• ontology/RFC/version outcome.

Interview mention counts are not scoring weights. Executive statements do not override practitioner evidence by status. Vendor, employer, school, or title prestige is never treated as capability evidence.

9. Quantitative and qualitative analysis

The frozen plan must distinguish:

• eligible participants, completed sessions, partial sessions, and coded records;
• confirmatory questions from exploratory themes;
• participant prevalence from excerpt frequency;
• unweighted counts from any justified design weights;
• denominators and missingness for every percentage;
• within-person repeated references from independent people; and
• practitioner, executive, and contextual strata.

No probability-of-population inference is made from a nonprobability sample. Quotations illustrate a coded finding; they do not prove its prevalence. “Saturation” is reported only with a declared definition, ordered accrual analysis, codebook stability record, and limitations. It is not a substitute for sampling coverage.

10. Integrity and commissioned-work audit

The intake lead runs duplicate and anomaly checks before analysis:

• repeated contact or participant keys;
• identical or implausibly similar notes/transcripts;
• impossible or clustered timestamps and durations;
• caller-level completion, exclusion, and missing-artifact outliers;
• instrument-version/date inconsistencies;
• unverifiable role claims or sponsor-scripted responses;
• missing consent or recording mismatches; and
• inconsistencies between dispositions, invoices, session records, and artifact manifests.

If prior consent permits recontact, an independent reviewer may conduct a documented, privacy-limited confirmation sample. Recontact is not mandatory and must not be invented as completed. A failed or refused recontact does not automatically label a participant fraudulent; it opens a review under a prespecified rule.

Evidence of fabrication, coercion, unauthorized data acquisition, or material chain-of-custody failure suspends affected records and may suspend the study. The incident, scope, remediation, and publication effect are recorded without exposing participants or integrity controls.

11. Study-state machine

The study register is append-only. The current state is derived from ordered events; it is not a manually editable marketing flag.

claim_received
  -> evidence_requested
  -> evidence_received
  -> chain_of_custody_verified
  -> record_validation_complete
  -> coding_complete
  -> analysis_frozen
  -> independent_review_complete
  -> aggregate_substantiated
  -> publication_authorized
  -> published
  -> corrected | retracted

Any active state from evidence_received through publication_authorized may transition to suspended. A suspended study may return only to evidence_received after a recorded remediation decision, or terminate as withdrawn. published may transition only to corrected or retracted; a correction creates a new release and never overwrites the prior one.

Advancing state requires all gates named for that state in the machine-readable study register. A website or content-management system may display published only from a signed aggregate claim receipt whose study state and counts reproduce from the frozen eligible-record manifest.

12. Independent review and release

The following roles must sign separately before publication_authorized:

• accountable Research Director;
• Measurement Lead;
• Research Ethics/Participant Rights owner;
• Privacy and Security owner;
• independent methods reviewer not in the collection or primary analysis line;
• disclosure reviewer for any public data or quotation; and
• legal/employment review when the publication will inform or market a hiring use.

The independent reviewer receives the protocol, deviations, recruitment flow, collector audit, schemas, frozen manifest, codebook, agreement results, analysis code, output, limitations, and proposed public wording. Disagreement and management response are retained.

13. Public methods and data-availability package

An authorized release includes:

• protocol and explicit retrospective/prospective timing;
• study register and state history;
• commissioning/funding/conflict disclosure;
• target and sampled populations, recruitment, dispositions, and inclusion flow;
• collector/interviewer count, training, mode, language, dates, and instrument versions;
• eligible practitioner and executive counts reproduced from the frozen manifest;
• duration distribution, missingness, exclusions, duplicates, withdrawals, and quality holds;
• codebook, coder process, double-code sample, agreement, deviations, and adjudication;
• analysis plan/code/environment digest and table-level provenance;
• benchmark traceability matrix and dissent;
• privacy/disclosure review and data-availability statement;
• limitations, corrections channel, and suggested citation; and
• a machine-readable aggregate claim receipt.

Raw identifying data is never required for public reproducibility. Public availability classes are:

14. Exact publication wording

Current permitted wording

LockedIn Labs has recorded an unverified requestor assertion that previously commissioned outreach included at least 500 AI practitioners and 50 executives. Underlying interview records have not been received or validated. The figures are not published findings and did not authorize benchmark or hiring claims.

This wording belongs in the research register, not promotional copy.

Wording after count substantiation but before analytic release

A controlled evidence review reproduced N eligible practitioner participants and M eligible executive participants from deidentified, deduplicated interview records collected between dates. The study design, recruitment sources, cohort definitions, exclusions, consent limitations, and data-availability statement are published with the count. Analysis and benchmark-content claims remain under review.

Wording after full methods release

LockedIn FDE Benchmark release version was informed by the FDE-JA-001 job-analysis study of N eligible AI practitioners and M eligible accountable executives, collected between dates using the published recruitment and interview protocol. The sample was sampling description and does not represent all practitioners or executives. Methods, denominators, limitations, coding reliability, traceability, and data availability: permanent citation.

The number 500 or 50 may replace N or M only if the signed receipt reproduces that minimum. “Research-backed,” “validated,” “industry standard,” “scientifically proven,” and predictive hiring language remain prohibited unless separately supported.

15. Method sources and boundaries

• AAPOR Disclosure Standards inform public disclosure of sponsor, sampling, recruitment, instrument, mode, language, dates, duration, and analysis. This protocol does not claim AAPOR membership or conformance.
• COREQ and the Standards for Reporting Qualitative Research inform reporting prompts for the research team and reflexivity, design, sampling, setting, collection, coding/analysis, findings, and limitations. They are reporting guidance, not proof that the work was valid, representative, ethical, or conformant; no completed checklist or conformance claim exists.
• EEOC Employment Tests and Selection Procedures and the Uniform Guidelines Q&A inform the separation between job-analysis content evidence and validation for employment use. Applicability and legal sufficiency require qualified review.
• HHS OHRP informed-consent guidance informs the participant-rights baseline where human-subject requirements may apply. The protocol does not assert that HHS rules govern every interview or that any review/exemption exists.
• FTC advertising-substantiation policy informs the fail-closed rule for objective public claims about the existence or size of research support. This is not legal advice.

This protocol complements, and does not replace, the Phase 2 research agenda, publication template, source ledger, data governance policy, and Live Versus Planned claims register.