Scoring Architecture

Status: v0.1 Public Draft
Official numeric scoring: Disabled
Reference calculations: Synthetic and experimental only

This document defines the v0.1 scoring boundary. It explains what a future scoring system must do while preventing experimental calculations from being presented as official results.

1. Normative status

Sections 2 through 10 are normative for v0.1-aligned implementations. Section 11 is informative.

Nothing in this document establishes calibrated weights, score bands, role thresholds, percentiles, or a valid comparison population. Those are future measurement decisions.

2. Non-negotiable rules

Official numeric scoring MUST remain disabled for every v0.1 release.
No administrator, reviewer, evaluator, sponsor, employer, trainer, or API client may directly assign or mutate a capability result projection.
AI systems may propose typed observations but MUST NOT call a score-write interface.
A calculation MUST be a deterministic projection of an immutable benchmark version, eligible human-review records, and a versioned scoring policy.
Capability, support confidence, coverage, E0–E5 evidence maturity, verification, integrity, T0–T4 trust progression, operational readiness, result lifecycle, and disclosure holds MUST remain separate.
Missing evidence MUST yield Insufficient Evidence, not zero or assumed weakness.
Critical security, integrity, production, or forward-deployed requirements MUST NOT be erased by averaging.
Correlated evidence MUST NOT be counted as independent corroboration.
Employment, education prestige, fame, social following, LockedIn employment or training, use of any particular tool, sponsorship, payment, and friendship MUST NOT be inputs.
Every calculation MUST be reproducible from a signed or digested calculation manifest.
A public human result MUST be a multidimensional capability profile, not one global number or a named-person rank.
A cost-per-person, compensation-efficiency, or “value for money” score MUST NOT be designed or inferred.
A proposed trust-progression, readiness, result-lifecycle, disclosure-hold, verification, recency, referral, integrity, or aggregate-register state MUST NOT create score authority or enter capability aggregation unless a later released specification explicitly defines and validates that boundary.
Human-only, model-only, and human-with-AI results MUST NOT be collapsed into a shared scalar, rank, or league table unless a later release defines and validates cross-condition comparability. No such authorization exists in v0.1.

3. Numeric representation

Authoritative fractional quantities use integer basis points in the inclusive range 0 through 10,000. This avoids floating-point ambiguity across implementations.

This internal scoring representation does not authorize fine-grained public aggregate shares. The isolated Phase 2 aggregate proposal uses approved coarse share bands and separate disclosure controls to avoid denominator recovery and differencing.

Presentation layers MAY convert basis points to percentages. Qualitative labels MAY be displayed only when derived from a versioned display policy. Display conversion MUST NOT change the underlying result.

Evidence maturity E0–E5 is ordinal. It MUST NOT be treated as an interval scale merely by assigning evenly spaced numbers.

4. Eligible inputs

The scoring boundary may consume only:

a released benchmark and ontology version;
released rubric and scoring-policy versions;
evidence-capability links that pass lifecycle, consent, authorization, integrity, and retention checks;
reviewer-accepted structured observations with a bound human-review record;
explicit lineage groups and contradiction records;
applicable critical-gate decisions;
a calculation timestamp and versioned recency policy.

Raw artifacts, résumé text, model prose, repository counts, provider reputation, and unreviewed model outputs MUST NOT be direct score inputs.

A human reviewer may amend an observation’s rubric finding with a reason and audit event. That action changes an input observation; it does not directly set the resulting score.

The v0.1 machine boundary does not accept an adjudication outcome as numeric authority because no released adjudication artifact is bound into the scoring input and calculation manifest. Adjudication may route or resolve a human workflow, but a later contract version must define and verify the complete adjudication record before such an outcome can become a scoring input. Until then, only an eligible accepted human-review record may cross the first numeric boundary.

5. Deterministic pipeline

A conforming experimental calculation follows this order:

Bind versions. Resolve immutable benchmark, ontology, schema, rubric, evaluator, and scoring-policy identifiers.
Validate authorization. Confirm subject, tenant, consent, permitted use, and disclosure constraints.
Filter eligibility. Exclude rejected, withdrawn, deleted, superseded, out-of-scope, or unresolved-integrity evidence.
Resolve lineage. Group copied, derivative, or correlated evidence before evaluating diversity or corroboration.
Build capability support. Associate eligible human-reviewed findings, contradictions, maturity, attribution, complexity, responsibility, and recency with the relevant competency.
Evaluate sufficiency. Apply versioned minimum coverage and evidence-diversity requirements. Emit Insufficient Evidence where unmet.
Apply critical gates. Evaluate integrity and safety requirements separately from aggregation.
Aggregate within scope. Produce capability and domain projections under the named experimental policy.
Keep uncertainty separate. Produce support confidence and coverage without blending them into capability.
Emit a manifest. Record inputs, exclusions, policy versions, intermediate digests, gates, limitations, and output digest.
Append an event. Preserve the calculation event without overwriting prior history.

The same manifest and implementation version MUST reproduce the same output.

6. Evidence weighting constraints

The draft factors of maturity, verification, attribution, recency, complexity, ownership, integrity, and diversity are research hypotheses, not validated weights.

Any experimental policy MUST:

expose each factor separately in its manifest;
apply diminishing returns to redundant evidence;
apply diversity at an evidence-set or lineage-group level, not to a single artifact;
handle suspected fraud or material integrity failure through eligibility and gates rather than a soft discount;
preserve contradictory evidence and reviewer disagreement;
avoid equating artifact quantity with capability;
document capability-specific recency behavior;
avoid double-counting one deployment through multiple representations.

Experimental maturity constants MUST be labeled policy choices, not empirical facts.

7. Coverage and support confidence

Coverage describes how much required benchmark scope has adequate eligible evidence. It is not a reward for submitting more material.

Support confidence describes the strength of support for a capability estimate. It may consider evidence independence, verification, attribution, reviewer agreement, scope, recency, and known contradiction under a released policy.

Neither measure may silently raise or lower capability. Low confidence or coverage MUST be visible to the user. Where capability cannot be defensibly estimated, the system MUST return Insufficient Evidence regardless of apparent average strength elsewhere.

8. Critical gates and role profiles

A future release may define gates for integrity, security, production evidence, discovery behavior, or other consequential capabilities. Each gate MUST specify:

construct rationale and intended role scope;
eligible evidence and review requirements;
pass, fail, unresolved, and not-applicable handling;
accommodation and alternate-demonstration rules;
appeal behavior;
validation evidence and known limitations;
release owner and change history.

Role readiness is not identical to a score band. No role-readiness designation or critical-floor threshold is active in v0.1.

9. Publication and history

An experimental reference implementation MAY emit a numeric output only when:

every input is synthetic;
the policy is explicitly named experimental;
the output prominently states EXPERIMENTAL SYNTHETIC OUTPUT — NOT AN OFFICIAL RESULT;
no real person or organization can reasonably be inferred;
the output is not published as a credential, ranking, or comparison.

Public-facing v0.1 experiences MUST present methodology, evidence state, or an Insufficient Evidence example rather than an official numeric result.

Future numeric research, if authorized, must first operate at the domain or explicitly defined dimension level and remain accompanied by confidence, coverage, verification, recency, version, and limitations. A global human score is not authorized in v0.1 and MUST NOT be introduced as a feature flag or governance shortcut. A later release may study an optional, purpose-bound scalar only through the separate future scalar-score authorization gate; it may never replace the multidimensional profile, uncertainty, critical floors, or Insufficient Evidence. A named-person leaderboard remains outside the benchmark design.

History is append-only. Correction, new evidence, policy change, withdrawal, or revocation creates a new event. Results from different benchmark versions MUST remain visually separated unless a published linking study permits comparison.

10. Activation gates for a later official release

Official scoring requires a new benchmark release and all of the following:

defined construct, target population, role profiles, and intended uses;
externally reviewed ontology and rubrics;
pre-specified calibration design and acceptance thresholds;
at least two appropriately designed calibration cohorts;
measured inter-rater reliability and adjudication quality;
calibrated uncertainty, coverage, floors, and score interpretation;
fairness, accessibility, irrelevant-attribute, and accommodation evaluation;
secure controlled cases and contamination monitoring;
operational consent, correction, appeals, revocation, privacy, and security controls;
independent review for conflicts and high-consequence results;
tested deterministic recomputation and score-impact analysis;
a public benchmark card, limitations report, and governance decision authorizing the release.
independence stage I3 under INDEPENDENCE_FIREWALL.md, including a seated external-majority FDE Benchmark Commission with external-majority quorum for the exact release;
operating separation of training, employment, staffing or recruiting, sales, evidence custody, rating, calculation, issuance, appeal, and publication roles, with tested access and compensation controls;
a published structural-entity decision comparing hosted, affiliate, independent-entity, and external-certification paths, with qualified legal, measurement, privacy/security, and credentialing review; and
separate authority and lifecycle rules for any Verified Production Record, Capability Profile, and Governed Credential Tier. One output MUST NOT silently confer another.

Software completion, an external review, an outside advisor, a separate entity, or an internal vote alone is insufficient.

11. Informative synthetic example

A synthetic practitioner provides a detailed architecture diagram but no contribution record, implementation artifact, live defense, or production corroboration. The system may map the diagram to several architecture capabilities, but attribution and coverage remain unresolved.

The correct v0.1 public outcome is:

Result: Insufficient Evidence
Observed support: architecture artifact received
Uncertainty: authorship, implementation ownership, production behavior
Next useful evidence: contribution record or controlled architecture defense
Official numeric score: disabled

The example demonstrates uncertainty handling; it does not prescribe a scoring formula.