AnnouncementTurnkeeper Ward brings scattered safety signals into cases people can review.

Turnkeeper Product · Evals

Bind the claim to the evidence.

Turnkeeper Evals helps AI and safety teams bind a release claim to the exact synthetic evidence and thresholds reviewed by the human decision-maker.

Open the workspaceExplore the evaluator

Product available · Human release decision required

From research to human decisionSources · datasets · evaluation · review

Bounded public sources become a cited synthetic dataset. Coverage is refined, the artifact and protocol are frozen, deterministic evaluation gates run, and the evidence stops for a human decision.

01Research question
Evaluate refusal boundaries
  1. S01
    Official guidanceCitation retained
  2. S02
    Technical standardCitation retained

2 bounded public sources

02Synthetic dataset
SourceLabelCoverage
S01Policy boundaryCore
S02Failure modeCore
S03Edge caseAdded
Gap foundEdge case added
Artifact v1 savedManifest 097e9b52…
03Evaluation
SourceG1G2
S01
S02

Source-by-source checks

04 · Evidence handoffAwaiting human review

Interactive evidence

See how one claim changes when the evidence does.

The protocol stays fixed. Switch only the submitted synthetic results to inspect how a pass, a critical failure, or an unavailable source changes the advisory conclusion.

Claim under review

Within two declared synthetic WardBench cohorts, the candidate source improves exact-label agreement while keeping operational failures within the frozen limit.

Sources
2
Gates
2

Change the submitted evidence

Every declared gate passes for both synthetic cohorts.

Synthetic evidence sources evaluated against every frozen release gate
Evidence sourceQuality improvement>= 0.050 · criticalOperational failure<= 0.020 · criticalSource finding
Development cohortSynthetic cohort
0.080Threshold >= 0.050Pass
0.012Threshold <= 0.020Pass
All gates pass
Holdout cohortSynthetic cohort
0.060Threshold >= 0.050Pass
0.018Threshold <= 0.020Pass
All gates pass
Advisory conclusion
Supported within tested synthetic scope

Every declared source-specific gate passed. A human still owns release.

  • Human release decision required
  • Automatic promotion not authorized
  • Automatic enforcement not authorized
Protocol details
Protocol reference
evals.synthetic.wardbench-source-qualification.v1
Schema
turnkeeper-evals-protocol-v1
Manifest hash
097e9b52987a72d94f40d755188b765aed20f3a41de048129eec4862cd85c96a
Frozen at
2026-09-01T16:00:00.000Z
Claim reference
claim.synthetic.source-quality-delta · claim-version.v1
Claim definition hash
187927efb34a86e3646a5dc9f68ce41b2aabaed8e828adf13a2b21bcfdefb9f8
Comparison
source.candidate.synthetic-v2 → source.comparator.synthetic-v1

Exact source references

  • Development cohortwardbench.synthetic.development · cohort-version.v2wardbench-v2-development-cohortsha256:3fbf55946583124769db86221b87cc2861ae4ce0ecaf5e7cb27155df47b3c29c
  • Holdout cohortwardbench.synthetic.holdout · cohort-version.v2wardbench-v2-holdout-cohortsha256:2f9f54a05c393497161441b830549b042f6e78b5ba691bbe64f4f4d5d0054b39

Exact gate references

  • Quality improvementgate.exact-label-agreement-deltaexact_label_agreement_delta
  • Operational failuregate.operational-failure-rateoperational_failure_rate

Known exclusions

  • exclusion.no-customer-or-production-data
  • exclusion.no-demographic-fairness-conclusion
  • exclusion.no-organizational-release-authorization

From protocol to decision

Bind. Evaluate. Decide humanly.

Turnkeeper Evals keeps chronology and authority explicit: declare first, evaluate second, then hand the evidence to the people responsible for the release decision.

  1. 01

    Bind the protocol

    Declare the candidate, comparator, synthetic cohorts, release gates, exclusions, and freeze time as one exact manifest.

  2. 02

    Evaluate every source

    Check every declared source against every gate. Failed, unavailable, and untested evidence stays visible instead of disappearing into one pooled score.

  3. 03

    Hand evidence to a human decision

    Return a bounded conclusion for review. The evaluator does not promote, deploy, release, or enforce anything.

Capability status

An implemented workspace with explicit authority boundaries.

The synthetic customer workspace is implemented end to end. Qualified private-data ingestion and organizational release authority remain separate future scope.

Implemented

Synthetic evidence contract

  • Freeze and hash one exact protocol
  • Evaluate every declared source and gate
  • Return four bounded advisory conclusions

Implemented

Hosted agent workspace

  • Tenant-scoped durable Agent sessions
  • Cloudflare Workers AI reasoning
  • Bounded public-source discovery
  • Immutable synthetic artifacts and append-only review
  • Durable deterministic evaluation jobs

Roadmap

Qualified private data and release controls

  • Customer or private dataset ingestion
  • Organizational release approval
What the hash proves

A changed claim, source, gate, exclusion, or freeze time creates a different hash. Detecting substitution still requires comparison with a previously retained manifest.

What the evaluator cannot do

It cannot certify safety, prove production performance, or authorize model promotion, deployment, release, or enforcement.