FDE BenchmarkPublic document reader
Available in sourcev0.1 · live public draft · official release pending

Canonical source view. This renders the bundled repository document as readable HTML. Source metadata remains in the repository, and relative document links remain inside this reader.

Job-analysis coding, agreement, and adjudication protocol

Protocol ID: LL-FDE-JA-CODE-001
Version: 0.1.0
Status: proposed; no codebook, coder roster, agreement result, coded corpus, theme, or benchmark-content decision is established

1. Purpose

This protocol governs the path from deidentified incidents, work descriptions, artifacts, surveys, and observer accounts to job activities, consequences, context conditions, capability mappings, and content-blueprint decisions.

It is informed by COREQ, SRQR, Krippendorff’s recommendations on reliability in content analysis, AAPOR disclosure principles, and the mixed-method design sources in the Source Ledger. Reporting completeness or agreement alone does not establish construct validity.

2. Analysis freeze

Before production coding, freeze:

  • research questions and confirmatory/exploratory boundary;
  • unit of analysis and unitization rules;
  • inclusion/exclusion and duplicate rules;
  • codebook-development process and pilot corpus;
  • coder qualifications, conflicts, training, assignments, blinding, and data access;
  • independently double-coded sampling rule;
  • agreement statistic(s), estimand, uncertainty method, calculation code, and action rule;
  • adjudication and dissent treatment;
  • negative-case, contradictory-evidence, missingness, and sensitivity plans;
  • ontology/content decision rules; and
  • software, environment, manifests, and random seed where used.

Statistics and thresholds remain blank until a qualified methods lead selects them for the actual data type, prevalence, unitization, sampling, and intended consequence. This protocol does not invent a universal alpha, kappa, correlation, or percentage threshold.

3. Unitization

The codebook defines how records become analyzable units. Candidate units include:

  • a consequential work episode;
  • a decision and rejected alternative;
  • an observable work behavior;
  • an artifact/work product and its use;
  • an operating constraint or change event;
  • a failure, refusal, rollback, or escalation;
  • an outcome and attribution boundary; and
  • a context qualifier or adjacent-role comparison.

Coders do not infer missing actions from job title, employer, reputation, outcome, or polished narrative. One passage may contain multiple units only under the frozen segmentation rule. Unitization disagreements are measured or explicitly treated, not hidden inside code agreement.

4. Codebook record

Every code includes:

CODE ID / VERSION:
LABEL:
DEFINITION:
UNIT TYPE:
INCLUSION RULES:
EXCLUSION RULES:
DECISION RULE / NEAREST-NEIGHBOR DISTINCTION:
POSITIVE EXAMPLE (synthetic or authorized deidentified):
NEGATIVE / NEAR-MISS EXAMPLE:
CONTEXT QUALIFIERS:
PARENT / CHILD / MUTUAL-EXCLUSION RELATIONSHIPS:
ONTOLOGY OR BLUEPRINT LINKS, IF ANY:
SOURCE / RATIONALE:
CHANGE HISTORY AND EFFECTIVE DATE:

Examples cannot expose a participant, customer, confidential implementation, or controlled assessment content.

5. Coder qualification and training

Coders need documented competence in the study method, FDE work/context, confidentiality, coding software, bias/conflict management, and escalation. A mixed team may intentionally combine methods and content expertise.

Training sequence:

  1. protocol, construct, intended-use, privacy, and non-inference orientation;
  2. unitization exercises;
  3. code-by-code instruction using synthetic/training material;
  4. independent practice set;
  5. blinded comparison and discrepancy conference;
  6. revision of ambiguous codebook language before production freeze;
  7. qualification set scored under the preregistered rule; and
  8. signed readiness and access receipt.

Training-set discussion cannot be counted as independent agreement evidence.

6. Independent double coding

  • Assignment is generated from the frozen sampling rule before either coder sees the other’s work.
  • Coders cannot discuss assigned units until both submissions are locked.
  • The system preserves coder, codebook version, timestamp, assignment, decisions, confidence/uncertainty where declared, and submission digest.
  • Replacement of a coder or record follows the frozen rule and remains visible.
  • Agreement is reported by code and relevant unit/context, with prevalence, missingness, and uncertainty needed to interpret it.
  • Overall agreement cannot conceal a critical code or unitization failure.

7. Agreement action rule

The preregistration must select an action for each range of evidence, such as:

  • proceed to adjudication and production use;
  • clarify definitions and test a new independent sample;
  • merge/split/narrow a code and version the codebook;
  • add training and requalify coders;
  • hold a theme or blueprint decision;
  • redesign unitization; or
  • reject the code as not reproducible enough for the claimed use.

Do not repeatedly tune the same qualification sample until a desired number appears. Revised codebooks use fresh or appropriately sequestered evidence.

8. Adjudication

Adjudication occurs after independent submissions lock. The adjudicator:

  1. sees both decisions and rationales;
  2. applies the frozen codebook rather than inventing a desired ontology mapping;
  3. records agreement type, resolution, rationale, codebook ambiguity, and affected units;
  4. distinguishes clerical mistakes from substantive disagreement;
  5. retains minority/dissenting interpretation when material; and
  6. opens an amendment if resolution changes definitions or decision rules.

Adjudicated consensus is not reported as pre-adjudication reliability. An unresolved disagreement can remain unresolved.

9. Negative cases and sponsor firewall

The analysis actively seeks work that contradicts the draft ontology, shows a capability is contextual rather than core, exposes inaccessible evidence expectations, or places responsibility in an adjacent role. Unsupported families can be narrowed, context-limited, split, merged, or rejected.

LockedIn Labs’ training, product, hiring, marketing, or services interests cannot require a code, theme, family, or favorable finding. Sponsor requests and analyst departures are logged.

10. Traceability output

Every proposed content decision links:

admitted record → unit → code/version → theme or work activity → consequence/context
→ ontology family disposition → observable behavior → work product/alternate demonstration
→ proposed assessment event → panel decision → public claim receipt

Public traceability may aggregate or withhold controlled identifiers, but an authorized reviewer must be able to inspect the complete controlled chain.

11. Release requirements

Publish or disclose under the approved access class:

  • codebook and version history;
  • coder qualifications at an appropriate level, conflicts, training, assignments, and blinding;
  • unitization and double-coding methods;
  • agreement estimand, calculation, uncertainty, missingness, code prevalence, and action rule;
  • adjudication volume and unresolved dissent;
  • negative-case and contradictory-evidence treatment;
  • software/environment and reproducibility package;
  • changes made after data inspection; and
  • what coding supports and does not support.

No claim is permitted until the study and publication claim receipts pass.