Phase 2 research agenda

Status: Proposed research agenda
Version: 0.1.0-draft
Evidence cutoff: 2026-08-31
Authorization: No study, recruitment, real-person intake, scoring, or publication is authorized by this document.

Purpose

Phase 2 proposes a T0–T4 trust-progression ladder, a separate operational-readiness axis and result lifecycle, evidence verification, an adaptive assessment, behavioral integrity controls, and a privacy-preserving aggregate register. Each component makes empirical claims about people or operating systems. Those claims must be tested separately before they are combined.

This agenda defines the research required to decide whether those components are understandable, useful, reliable, valid, fair, accessible, privacy-preserving, and operationally supportable. It does not set passing values after seeing results.

Research rules

1. Register the question, population, measures, exclusions, missing-data treatment, subgroup plan, stopping rule, and decision rule before collecting outcome data.
2. Separate exploratory, calibration, confirmation, and monitoring datasets.
3. Treat the ontology, rubrics, scenarios, assistance policy, evaluator, reviewers, and calculation policy as versioned interventions.
4. Analyze process burden and exclusion, not only completion and score distributions.
5. Report null results, broken scenarios, grader failures, protocol deviations, and unanticipated harms.
6. Never use employer prestige, school prestige, fame, geography, compensation, or current title as ground-truth capability.
7. Keep research participation separate from employment management; require approved consent and withdrawal paths.
8. Publish only aggregates that pass the approved disclosure-control review.
9. Treat subgroup analysis as both a measurement and privacy problem; small cells are not published.
10. Require independent measurement, fairness, accessibility, privacy, security, and legal review appropriate to the study.

Workstream 1 — role and job analysis

Question: What observable work, decisions, artifacts, contexts, and outcomes constitute forward-deployed engineering across relevant settings?

Method candidates:

• critical-incident interviews;
• structured task and work-product inventory;
• practitioner and manager panels with conflicts recorded;
• importance, frequency, difficulty, consequence, and entry-versus-learned-on-job ratings;
• comparison across company size, product maturity, deployment model, regulated context, and domain pack;
• cognitive walkthroughs of representative work.

Outputs: role boundary; task inventory; work-product inventory; competency map; domain-pack differences; content blueprint; excluded constructs; terminology decisions.

Decision gate: No controlled assessment scenario is used until its criterion and probe chain trace to an approved content blueprint. The EEOC's published selection guidance emphasizes job relevance and validation for the intended use (selection procedures, Uniform Guidelines Q&A).

Workstream 2 — trust-status comprehension

Question: Do participants, employers, researchers, and the public correctly distinguish T0–T4 trust progression, operational readiness, E0–E5 evidence maturity, capability, result lifecycle, disclosure holds, and credential status?

Study design: randomized comprehension testing of labels, explanations, profile layouts, and paired cases. Include cases with equal capability but different corroboration, and equal corroboration but different capability.

Primary measures: correct interpretation; unsupported inference rate; perceived employability inference; confidence; time to answer; accessibility and language effects.

Decision gate: A public trust state does not launch if a material audience interprets it as capability, endorsement, certification, or hiring recommendation. Copy or visualization changes require a new test.

Workstream 3 — verification-path equivalence

Question: Can different identity and evidence-verification paths achieve comparable assurance without making fame, employer size, geography, disability, or technical visibility a de facto accelerator?

Study design: synthetic adversarial testing followed, only after approval, by a consented feasibility cohort across multiple legitimate paths.

Measures: completion; abandonment; time; false hold; manual-review burden; evidence availability; accessibility; correction; participant trust; error by path; privacy data collected; provider failure.

Decision gate: Every approved path documents assurance, errors, burden, privacy, fallback, and redress. Prestige is never an input. NIST SP 800-63A is a technical design reference for risk assessment, comparable proofing options, privacy, fraud management, and redress—not a conformance claim (SP 800-63A).

Workstream 4 — behavioral-integrity controls

Question: Which signals detect scaled abuse with acceptable privacy cost and false-positive burden?

Phases:

1. Threat model and synthetic attack corpus.
2. Offline signal evaluation with no person-level action.
3. Shadow-mode monitoring in an approved cohort.
4. Human-reviewed routing only.
5. Limited step-up or temporary hold after independent review.

Measures: precision/recall by attack class; false holds; differential error; feature stability; evasion cost; review time; appeal correction; data volume; retention; participant comprehension.

Stop conditions: unexplained subgroup disparity; unbounded retention; inability to state a reason; signal leakage into capability; reviewer overload; reconstruction or cross-context tracking risk.

Decision gate: No signal can independently reject, lower capability, or publish a fraud label. Device fingerprinting requires an approved privacy assessment and a less-intrusive-path analysis.

Workstream 5 — referral and corroboration graph

Question: Does scoped corroboration add reliable evidence without producing coercion, collusion, retaliation, or automatic guilt by association?

Study design: synthetic graph simulations, red-team collusion cases, conflict scenarios, and controlled reviewer exercises before any real referral pilot.

Measures: incremental verification value; collusion detection; false association; network concentration; conflict disclosure; reviewer consistency; participant burden; sanction and appeal outcomes.

Decision gate: Graph association alone never alters capability or credential status. Confirmed abuse may affect referral privileges only through a reasoned, human-reviewed, appealable process.

Workstream 6 — adaptive probe assessment

Question: Do probe chains elicit job-relevant judgment beyond polished first answers, across declared assistance modes?

Method: scenarios authored from the content blueprint; constraint changes; rationale and failure-mode probes; alternate demonstration; known-good, insufficient, invalid, and ambiguous fixtures; controlled assistance conditions.

Measures: criterion coverage; learnability; response-process evidence; reviewer agreement; evaluator-human agreement; model/provider drift; assistance sensitivity; test-retest stability; accessibility parity; burden and completion.

Decision gate: A scenario is held if its intended answer is under-specified, if competent reviewers disagree beyond the preregistered rule, if assistance changes the construct unintentionally, or if an accommodation changes what is measured.

Workstream 7 — multidimensional measurement

Question: Which proposed lenses are distinct, reliable, interpretable, and related to relevant work outcomes?

Required analyses:

• content representation;
• internal structure where appropriate;
• inter-rater and test-retest reliability;
• convergent and discriminant evidence;
• criterion relationships with independently defined work outcomes;
• differential item functioning and measurement invariance where sample design permits;
• missingness, coverage, and range restriction;
• sensitivity to weights, floors, and aggregation;
• incremental value of each lens;
• consequences and alternative procedures.

Special hypotheses:

• AI-native delivery remains separate from engineering capability.
• Evidence strength, coverage, and recency describe claim support and do not enter capability aggregation.
• Time to production remains a research candidate until organizational delay, risk, scope, and quality confounds are resolved.

Decision gate: No overall number is published unless independent review finds that its intended use and composition are supported beyond the component profiles. Higher scores cannot be marketed as better job performance without appropriate empirical evidence.

Workstream 8 — evidence freshness and decay

Question: When does older evidence cease to support a current claim, and does a refresh policy improve accuracy enough to justify its burden?

Method: longitudinal evidence study by evidence class, capability family, and role context; policy simulations before changing participant status.

Measures: outcome drift; evidence availability; refresh completion; false loss of current eligibility; burden; accessibility; domain differences; appeal and exception rates.

Decision gate: No universal decay interval. A policy must identify what becomes stale—verification confidence, current evidence eligibility, profile freshness, or capability support—and must never demote or retroactively rewrite T0–T4 event history. Any capability change requires a separately authorized recalculation.

Workstream 9 — aggregate register and geographic disclosure

Question: Can aggregate participation and evidence density be informative without exposing individuals or inviting national-aptitude claims?

Method: synthetic population and attack simulations, sampling analysis, comprehension testing, then privacy review of any proposed real release.

Candidate controls: metro-level aggregation; separate opt-in consent; unique-person deduplication; minimum subject and organization thresholds; organization-dominance limits; complementary suppression; coarse count and share bands; denominator-recovery tests; comparison against the exact retained prior-snapshot set; a shared cross-channel disclosure-budget ledger; temporal aggregation; filter-combination restrictions; downloadable-data review; visible sampling-bias labels.

Measures: reconstruction success; membership inference; correct chart interpretation; geographic coverage; sampling imbalance; suppression rate; utility loss; small-community risk.

Decision gate: A map is not published if users infer capability by geography, if the sampling frame cannot support the title, if a fine ratio recovers a denominator, or if protected cells can be reconstructed within one snapshot or by differencing retained snapshots. k >= 30 is an initial simulation candidate, not a privacy guarantee.

Workstream 10 — human review and appeals

Question: Can trained reviewers apply the versioned rubric consistently, identify system defects, and provide meaningful correction without adding hidden discretion?

Method: blinded double review, drift cases, conflict cases, accessibility cases, evaluator-failure cases, and end-to-end appeal drills.

Measures: agreement by criterion; severity and direction of disagreement; calibration drift; automation-bias indicators; review time; conflict routing; appeal correction; participant comprehension; remediated system-error rate.

Decision gate: No operating profile without staffed, trained, non-conflicted review and appeal coverage. An appeal can correct process, evidence, identity, rubric application, accessibility, or system error. Correction creates superseding records without replacing the audit history; a result-lifecycle event exists only under a later release that separately authorizes results, and Phase 2 issues none.

Workstream 11 — publication reproducibility

Question: Can an external reviewer understand and reproduce every aggregate claim from the frozen release package?

Required package: protocol; preregistration; data dictionary; source manifest; transformations; analysis code; environment; inclusion flow; missingness; uncertainty; disclosure-control record; conflicts/funding; limitations; corrections; data-access statement.

Decision gate: Every chart links to its method and release receipt. Exploratory findings are labeled exploratory. Changes after analysis freeze are versioned and explained. The operating reference is transparent benchmark methodology and version history, not another site's visual design (Artificial Analysis methodology).

Cohort sequence

Research operating roles

• Head of Measurement: construct, validation, decision rules, and release hold authority.
• FDE Research Director: job analysis, field methods, domain packs, and practitioner panels.
• Evaluation Research Engineer: scenarios, invariants, evaluator studies, and reproducibility.
• Fairness and Accessibility Research Lead: subgroup, language, accommodation, and consequence studies.
• Privacy Statistician: disclosure control, sampling, reconstruction testing, and aggregate release review.
• Verification Integrity Lead: assurance paths, attack studies, reviewer operations, and incident learning.
• Research Ethics and Participant Rights Lead: consent, coercion controls, withdrawal, correction, and participant communication.
• Methods Editor: preregistration, publication package, limitations, failures, and corrections.

The first three roles form the minimum research nucleus. Privacy, fairness/accessibility, security, legal, and ethics authority must be available before the study that needs them; they cannot be replaced by product review.

Publication series

1. FDE role and job-analysis report.
2. Ontology content map and expert-review record.
3. Evidence availability and verification-path feasibility report.
4. Trust-status comprehension report.
5. Adaptive-probe learnability and reviewer-invariant report.
6. Reliability and calibration report.
7. Accessibility, assistance-mode, and fairness report.
8. Integrity-control and false-hold report.
9. Aggregate-register privacy and sampling report.
10. Validation report scoped to an exact intended use.
11. Annual state-of-field report only after sample and disclosure gates.

Each publication must use the publication template and chart and data-product contract.

Current blockers

• No named research, measurement, fairness, accessibility, privacy, or ethics owners.
• No approved participant protocol or consent materials.
• No job-analysis study.
• No real-person dataset and no authorization to create one.
• No approved identity, verification, reviewer, accommodation, appeal, or disclosure-control operation.
• No independent validity, security, privacy, legal, or accessibility review.
• No approved numeric scoring release.

The research agenda may be reviewed now. Participant work begins only after the relevant governance and operating gates are recorded.