FDE BenchmarkPublic document reader
Available in sourcev0.1 · live public draft · official release pending

Canonical source view. This renders the bundled repository document as readable HTML. Source metadata remains in the repository, and relative document links remain inside this reader.

Shared Work Challenge Battery

Status: Proposed research design / not authorized
Administration: None
Human baseline: Not collected
Official or comparative scores: None

1. Research proposition

The Shared Work Challenge Battery is a differentiated research asset: the same versioned enterprise work problems can be attempted under three declared conditions:

  1. Human only — a practitioner works without generative-AI assistance under an explicit tools and references policy.
  2. Model only — an AI system or agent scaffold works under an explicit model, tool, memory, retrieval, authority, and resource envelope.
  3. Human with AI — a practitioner and declared AI system operate as a socio-technical team under an explicit assistance and attribution policy.

The battery studies how work gets done, where failures occur, and which condition is effective for a particular task and operating boundary. It is not a spectacle that asks whether “AI beats humans,” and it is not a named-person leaderboard.

2. The crucial measurement boundary

A common scenario substrate improves experimental control; it does not make the constructs identical.

Condition Primary object of evaluation Example interpretation
Human only Practitioner performance under the declared human condition What eligible observations support about demonstrated FDE capability in this task and context
Model only AI-system or agent performance under the declared model condition How the exact model/scaffold/tool configuration performed on the task
Human with AI Team performance under the declared assistance condition How the practitioner, AI, tools, handoffs, and control design performed as a socio-technical system

Direct score comparison requires more than using the same task. Interfaces, time, compute, information access, iteration opportunities, tool authority, accommodations, and interaction costs can differ materially. Results therefore remain condition-specific unless a later linking or invariance study supports a particular comparison.

3. Battery architecture

3.1 Common scenario substrate

Every scenario has one immutable manifest binding:

  • problem and enterprise context;
  • work objective and acceptance contract;
  • inputs, hidden state, and allowed disclosures;
  • deliverable and artifact schemas;
  • observable events and attempt logs;
  • rubric and metric definitions;
  • safety, privacy, and authority boundaries;
  • scenario version, known defects, and learnability review;
  • practice, controlled, or held-out availability class; and
  • contamination and rotation controls.

The three conditions reference that same scenario manifest. Condition-specific adapters may change interaction mechanics only where predeclared and justified; they may not silently change the problem, acceptance criteria, or available facts.

3.2 Condition contract

Every condition binds:

  • agent class and assistance policy;
  • resource envelope, including time, compute, calls, concurrency, personnel, references, and tool access as applicable;
  • environment and dependency versions;
  • interaction, retry, escalation, and stopping rules;
  • adaptation or prompting policy;
  • identity, model, scaffold, and tool versions as applicable;
  • artifact and trace capture; and
  • failure, invalid-run, censoring, and exclusion rules.

Equality of resources is not always meaningful. A CPU-minute and a human-minute are not interchangeable. The design should predeclare the resource envelope, report differences, and align each comparison to the research objective.

4. Initial task families

The first battery should cover a small number of deep, stateful scenarios rather than many shallow questions.

Family Scenario arc Observable outputs Failure pressure
Discover and contract Interview a synthetic stakeholder environment, resolve ambiguity, reject unsafe value, define the outcome and acceptance contract Assumption register, decision record, risk boundary, architecture brief, acceptance criteria Missing facts, conflicting goals, incentive to over-promise
Build, evaluate, and recover Implement or repair an AI-enabled workflow against tests, hidden failure cases, security constraints, and an evolving requirement Code/configuration, test and evaluation artifacts, threat controls, failure analysis, revised release candidate Tool injection, brittle evals, specification ambiguity, time pressure
Deploy, operate, and prove outcome Plan and execute a governed rollout in a controlled environment, respond to an incident, and determine whether to stop, iterate, or scale Release/rollback evidence, observability, incident actions, outcome analysis, handoff and operating model Partial outage, noisy metrics, ownership gaps, pressure to ship

Each task family must link to the released job-analysis/content blueprint before it can support a human capability interpretation.

5. Multi-metric result design

No single task-success number is sufficient. At minimum, the design should predeclare condition-applicable measures across:

  • task and artifact quality;
  • production outcome and acceptance-contract attainment;
  • safety, privacy, security, and authority control;
  • error discovery, recovery, and escalation;
  • robustness across scenario variants;
  • calibration of confidence and uncertainty;
  • collaboration and handoff quality for human/team conditions;
  • time and resource use under the declared envelope; and
  • invalid-run, abstention, refusal, and unresolved rates.

HELM's value here is the pattern of broad scenario coverage, multi-metric reporting, transparent omissions, and controlled adaptation—not its model scores or a model-to-human construct analogy. Every metric remains visible; this proposal prohibits aggregation into a shared human/model composite.

Each metric declares the condition or conditions to which it applies. Its definition is immutable and common only across those declared applicable conditions. A collaboration or handoff measure may apply to a human or human-with-AI workflow without being forced onto a model-only run. A metric being applicable in multiple conditions still does not authorize a direct score comparison.

6. Human baseline requirements

NIST AI 800-2, Initial Public Draft notes that evaluations comparing AI with alternatives such as humans-only or humans assisted by AI benefit from relevant, statistically robust baselines, including sufficient numbers and appropriate expertise of human test-takers. For this occupational research program, that principle means:

  • define the target practitioner populations and expertise strata before recruitment;
  • perform a power and precision analysis for the intended estimands rather than selecting a convenient round number;
  • disclose recruitment, inclusion, exclusions, attrition, missingness, and compensation;
  • avoid treating trainees, staff, referrals, or public volunteers as a representative FDE population without evidence;
  • predeclare repeated-measure, task-order, learning, fatigue, and carryover handling;
  • use accommodations and alternate paths that preserve the construct; and
  • report uncertainty and subgroup sample limitations rather than hiding them behind an average.

Target sample size in the machine contract is a design input. The contract sets observed sample size to zero because no baseline has been collected.

7. Experimental designs worth testing

7.1 Parallel-condition study

Randomize eligible scenario forms and task order across independent condition samples. Estimate condition-specific performance distributions and failure profiles. Use this for broad comparisons when carryover would compromise the task.

7.2 Within-practitioner assistance study

Where learning and leakage can be controlled with linked forms, compare the same practitioner's human-only and human-with-AI performance. Predeclare order, washout, counterbalancing, assistance versions, and learning-effect analysis. This design studies augmentation; it does not establish a model-only human equivalence scale.

7.3 Complementarity and handoff study

Manipulate which party performs discovery, implementation, evaluation, and operation. Measure error interception, escalation quality, recovery, outcome, and coordination cost. The unit of interpretation is the workflow or team, not an attempt to attribute all value to one actor.

8. Review and grading controls

  • Human reviewers are blinded to condition where artifacts can be safely and authentically normalized; when blinding is impossible, the limitation is reported.
  • Rubrics and metrics are frozen before the scored run and linked to known-good, known-insufficient, and invalid synthetic examples.
  • Raters complete qualification, calibration, agreement, and drift monitoring.
  • Model graders, if researched, produce typed proposals and never final human capability or credential decisions.
  • Invalid or underdetermined scenarios are suspended and impact-analyzed.
  • Attempts preserve versioned traces, exclusions, adjudication, and replay references without publishing controlled solutions or private data.

9. Publication design

The first publication should be a battery design and synthetic dry-run report, not a leaderboard. Later authorized studies should publish:

  • objective, estimands, hypotheses, protocol, and preregistration;
  • scenario and metric coverage, including material omissions;
  • exact condition, model/scaffold/tool, administration, and resource versions;
  • sampling, human expertise, attrition, missingness, exclusions, and invalid runs;
  • condition-specific results with uncertainty and attempt counts;
  • sensitivity, subgroup, rater, contamination, and robustness analyses;
  • artifacts and reproduction material where rights and integrity allow;
  • deviations, corrections, limitations, and negative findings; and
  • an explicit statement of comparisons the design does not support.

Named-person ranking, employment recommendations, universal “human versus AI” winners, and one blended score remain prohibited.

10. Activation sequence

  1. Link every task to the job-analysis and content blueprint.
  2. Complete scenario specification, accessibility, learnability, threat, and contamination review.
  3. Freeze condition, assistance, resource, logging, metric, rater, and analysis protocols.
  4. Complete synthetic dry runs and known-case invariants.
  5. Obtain approved human-research, privacy, consent, security, and data-rights decisions before any recruitment.
  6. Conduct a bounded pilot; publish defects and feasibility findings before making capability claims.
  7. Collect statistically adequate condition data under a preregistered plan.
  8. Separately evaluate human capability interpretations, AI-system interpretations, and team/workflow interpretations.
  9. Require independent review and a new release decision for any public score, comparison, tier, credential, or employment-adjacent use.

The proposal contract at shared-work-challenge-battery.schema.json captures the design boundary and structurally carries no result.