Assessment rater qualification, calibration, agreement, and drift protocol

Protocol ID: LL-FDE-RATE-001
Version: 0.1.0
Status: proposed for a future controlled pilot; no human rating, qualified rater, calibration result, reliability result, or official score exists

1. Scope

Coders characterize job-analysis material; raters evaluate assessment evidence. These roles may overlap only when conflicts, qualifications, blinding, and analysis effects are addressed.

This protocol governs human rating of future work samples, artifact reviews, reference evidence, and structured observations. The Standards, SIOP Principles, evidence-centered design, structured-interview research, interrater agreement literature, and many-facet measurement literature in REFERENCES.bib inform the design. No published coefficient is imported into this benchmark.

2. Rater role specification

For each criterion, predeclare:

• capability and observable behavior;
• evidence the rater may inspect;
• context needed and context deliberately hidden;
• anchor definitions and boundary examples;
• prohibited proxies and inferences;
• required FDE/domain/method expertise;
• conflicts and recusals;
• accessibility and alternate-demonstration competence;
• decision consequence and escalation authority; and
• whether adjudication, audit, or participant feedback is part of the role.

Raters never infer personal worth, honesty, intelligence, identity, protected status, compensation value, employer prestige, or universal readiness from missing or bounded evidence.

3. Selection and conflicts

Verify the minimum identity and qualifications needed. Record current/recent employment, supervision, customer/vendor, investment, training, personal, referral, and prior-participant relationships.

A rater must recuse from evidence involving a person or organization where impartiality is compromised. Raters cannot score their own training clients or use employment, LockedIn training, platform activity, or acquaintance as positive evidence.

4. Training sequence

1. intended interpretation, prohibited use, uncertainty, and missing-evidence orientation;
2. task, conditions, assistance, accommodation, and evidence-lineage orientation;
3. criterion and anchor instruction;
4. construct-irrelevant variance, bias, privacy, security, and accessibility scenarios;
5. independent practice on synthetic or sequestered training cases;
6. facilitated comparison focused on reasoning, not forced consensus;
7. qualification on a different sequestered case set; and
8. signed role, conflict, confidentiality, and qualification receipt.

Qualification criteria, resubmission limits, new-form rules, and expiry are frozen before raters see qualification outcomes.

5. Calibration package

Each calibration case includes:

• case/task/form/version and contamination status;
• conditions and permitted assistance;
• evidence/artifact bundle and provenance;
• accessibility/accommodation path;
• criterion-level reference rationale from a qualified panel;
• plausible alternate high-quality approaches;
• expected uncertainty and evidence gaps;
• critical errors and non-inferences; and
• protected answer/rationale handling.

Reference ratings are a governed expert judgment, not ground truth by declaration. Substantive panel dissent remains visible.

6. Rating operations

• Assign raters by a reproducible rule that balances workload and conflict without cherry-picking.
• Blind identity, prestige cues, training status, prior score, outcome, and other irrelevant context where feasible.
• Lock ratings and criterion rationales before another rater or adjudicator is visible.
• Separate insufficient evidence, not applicable, invalid task, and observed weak evidence.
• Preserve evidence citations for each criterion.
• Record tool failure, suspected leakage, accessibility issue, security concern, and protocol deviation.
• Do not use a language model as an independent rater or human replacement without a separately authorized evaluator study.

7. Agreement and severity analysis

The analysis plan chooses methods appropriate to scale type, design, rater sampling, missingness, and intended decision. It may examine agreement, reliability, consistency, decision consistency, rater severity/leniency, criterion interpretation, form/task effects, and interactions.

Report uncertainty and design limitations. A high overall coefficient can coexist with unacceptable disagreement at a critical floor. A low coefficient may reflect ambiguous criteria, sparse categories, restricted range, rater training, task defects, or heterogeneous evidence; it is not automatically “bad raters.”

No threshold is declared here. The measurement lead and independent reviewer must predeclare the estimand, minimum evidence, action rule, and consequence before pilot results.

8. Adjudication and appeals separation

Adjudication resolves a rating under the current evidence and rubric. Appeal reviews whether process, evidence, conflicts, accessibility, or policy warrant reconsideration. They are separate.

Adjudicators record the changed criterion, original ratings, evidence, rationale, rubric ambiguity, and whether the case/rubric/rater requires remediation. Adjudicated ratings do not replace original ratings in agreement analysis.

9. Drift monitoring

During a pilot, use scheduled sequestered anchors and live-data monitoring declared before rating. Track:

• criterion/category use;
• deviation from reference rationale or earlier calibrated behavior;
• disagreement by rater pair, task, form, criterion, context, and accommodation path;
• missing/insufficient-evidence patterns;
• adjudication and appeal rates;
• time/burden and tool errors; and
• possible differential behavior across approved fairness groups, with privacy protection.

Trigger responses may include review, targeted retraining, temporary hold, requalification on fresh material, double-rating, rerating affected work, rubric/task revision, or rater removal. The trigger and remedy are preregistered; operators cannot selectively intervene to change a participant outcome.

10. Pilot release packet

Publish rater counts and qualifications at an appropriate privacy level, assignment design, training and qualification method, conflicts/recusals, calibration material provenance, rating/adjudication flow, agreement/reliability/decision-consistency evidence with uncertainty, drift events, exclusions, reratings, appeals, limitations, and exact permitted interpretation.

Until this protocol is approved and executed with real evidence, all rater and score language remains proposed or synthetic.