Cited
Every “yes” or “no” must point to an exact quote from the operator’s chat, code, spreadsheet, or slide. If the engine can’t find evidence, it must say so explicitly.
no quote → no score → “insufficient evidence”
The evaluation engine · STORM
No black box.
No black box.
Just evidence.
Every score points to a real quote from the actual work, whether the operator is human or agent. The engine does the homework; you make the call. Here’s exactly how it works, step by step.
Cited evidence
Deterministic scores
Human or agent
Single observation · anatomyeval #4172
Criterion
Funnel metric definitionweight 0.25 · critical
Question
Did the operator distinguish “started application” from “submitted application” in their reasoning?
Citation
“The dashboard’s counting started apps, but conversion is measured against submitted apps. That’s where the drop is coming from.”
Answer✓ Yes
Reasoning
Identified the metric mismatch directly and explained the funnel semantics before recommending a fix.
prompt observation_check v1model pinned100% cited or insufficient
The promise
Three things every score guarantees.
Most AI tools hand you a number and call it done. We hand you the number, the question that produced it, the quote from the work that justifies it, and the reasoning that connects them.
Deterministic
Same evidence, same prompt, same model. Same result. Scores are reproducible quarters from now, with the prompt version and model snapshot stored alongside the score.
temperature 0 · seed 42 · model pinnedHuman-decides
The engine scores. You decide. See every observation, every quote, every “insufficient evidence” tag, and override any score with a comment that joins the audit trail.
AI assists · you decideThe pipeline
Six stages, end to end.
From defining what “good” looks like for a role, all the way to the reviewer’s view: every step the engine takes, and what we do at each one to make the result trustworthy.
Turn the role into testable questions
You define what matters: competencies, weights, importance. The engine turns each criterion into a small set of yes/no observation questions a reviewer could ask while watching the work. They’re the operational definition of “good”, versioned, reviewed, and bound to the role.
You can edit, add, or veto any questionCriterion → questions
operational definition of good
Q1Did the operator distinguish “started application” from “submitted application”?
Q2Did they verify the metric definition against the source before recommending a fix?
Q3Did they connect the funnel mismatch to the post-launch conversion drop?
Q4Did they name a concrete remediation, not a generic “needs investigation”?
Capture what they actually did, not what they said they’d do
As the operator works, every artifact is captured (voice transcript, code edits, spreadsheet formulas, slide content, every line of chat) and bundled into a structured evidence package tagged by source. Names and locations are stripped before any model sees it. We score the work, not the person.
PII-redacted before scoringEvidence bundle
sim-uniera-funnel
💬
Chat transcript14 turns · stakeholder Q&A
3.2 KB💻
Code editorLMS application-count fix
+34 / −12📊
Spreadsheetpre/post conversion analysis
2 sheets🎤
Voice callVP Ops debrief · transcript
8m 14sredactor pass: replaced [OPERATOR_NAME], [LOCATION] before evaluator handoff
Score every question against the evidence
For each question we run an observation check: one yes/no question, the full evidence bundle, and a single instruction: answer only if you can quote the exact text that justifies it. The engine returns an answer, a citation, and one sentence of reasoning. No quote, and the only legal answer is Insufficient Evidence.
One question · one citation · one answerObservation checks
funnel metric definition
Q1 · Distinguished started vs. submitted apps?
cite: chat[turn 6]
Q2 · Verified the metric against the source?
cite: code[diff line 42]
Q3 · Connected mismatch to post-launch drop?
cite: voice[03:14]
Q4 · Named a concrete remediation?
cite: not found in bundle
Verify every citation. Punish hallucinations.
Models can fabricate plausible quotes. We don’t take their word for it. Every citation goes through a deterministic validator (pure code, no AI) that checks the quote actually appears in the evidence. If it can’t be found, the answer is auto-downgraded to Insufficient and a citation-violation is logged as a leading indicator of prompt drift.
Hallucinated quotes are auto-downgradedCitation validator
pure code path · no AI
Q1 · citation submittedVERIFIED
“The dashboard’s counting started apps, but conversion is measured against submitted apps.”
✓ matched in chat[turn 6] · answer accepted: Yes
Q4 · citation submittedNOT FOUND
“We should rebuild the funnel attribution from the database up.”
✗ no match in evidence · coerced to Insufficient · kind=citation_violation
Aggregate honestly. Never fake confidence.
We don’t average our way to a clean number when the evidence isn’t there. Each criterion is Assessed (enough evidence), Partially assessed (some signal, missing pieces), or Not assessed (the simulation didn’t surface it), and we say which, plainly.
Honesty about what we don’t knowEvaluation summary
honest aggregation
87
Funnel metric
3 of 4 confirmed · assessed
3 of 4 confirmed · assessed
62
Stakeholder comms
voice ok; deck missing · partial
voice ok; deck missing · partial
n/a
Prod hardening
no evidence · not assessed
no evidence · not assessed
Make every score reproducible, forever
Every evaluation writes a structured audit line: which prompt versions generated the questions and scored them, which model snapshot produced the answer, which seed was used, how many citation violations occurred. Six months on, replay the exact evaluation against the exact evidence and get the same score. Not a marketing claim. It’s how the pipeline is built.
Full chain of custody[geval_audit]
structured log
kind = eval_complete · evaluationId = 4172 · positionId = pos-uniera-l3
questionGenPrompt = v1 · observationCheckPrompt = v1
model = <pinned-snapshot> · seed = 42 · temperature = 0
criteria = 8 · questions = 31 · citationViolations = 1
redaction = name+location · samples = 1 · mode = shadow
questionGenPrompt = v1 · observationCheckPrompt = v1
model = <pinned-snapshot> · seed = 42 · temperature = 0
criteria = 8 · questions = 31 · citationViolations = 1
redaction = name+location · samples = 1 · mode = shadow
Human in the loop
The engine scores. You decide.
The pipeline produces evidence: a structured argument for or against each criterion, with citations attached. It does not produce decisions. You see every piece of work, every observation, every quote, and every “insufficient evidence” tag.
- ✓ Every observation, citation, and reasoning trail is visible in the evaluation view.
- ✓ Override any score, with a required comment that’s stored on the evaluation.
- ✓ “Insufficient evidence” is a first-class outcome, never coerced into a fake number.
- ✓ The engine never auto-rejects, auto-advances, or hides anyone from a human reviewer.
Senior Data Analyst · review3 operators
Operator A
87 assessed · 1 partial · 0 not assessed
Operator B
62 partial · communication strong; production weak
Priya · reviewer
Disagreed with the “production hardening” mark. The slide deck addressed it directly; the engine missed the slide because the artifact was attached after the call. Re-running with the full bundle.
Operator C
not assessed · simulation cut off at chapter 2
The guardrails
What we built so the AI can’t drift.
Five engineering decisions that turn a clever LLM into a defensible evaluator: the spine of the pipeline, not toggles or nice-to-haves.
Versioned prompts
The instructions we give the engine are file-versioned and stamped onto every score. Changing a prompt means a new version, and a new test gauntlet.
Pinned model snapshot
We don’t ride the latest model. We pin to a specific snapshot, recorded with every score, so a vendor update can’t silently change a score you already shipped.
Self-consistency sampling
For high-stakes criteria we run each check N times in parallel with independent seeds and take a majority vote. A flake in one lane can’t bias the result; ties are surfaced, not averaged away.
PII redaction
Names and locations are stripped from the evidence bundle before it ever reaches the evaluator. The model scores the candidate’s work, not who they are.
Built for ADET compliance
Engineered to clear NYC Local Law 144, the Illinois AI Video Interview Act, and the Colorado AI Act: questions referencing protected characteristics are rejected at generation, demographic and stylistic signals are blocked at scoring, and a paired-candidate bias audit gates every prompt change.
Common questions
Things buyers ask us, with honest answers.
How do I know it’s not just a black box?
Every score breaks down into the questions that produced it, the literal quote from the work that justified each answer, and the engine’s one-sentence reasoning. If you can read the evaluation page, you can audit the score. We never show a number without the evidence behind it.
What happens when the AI gets it wrong?
Two things. Because every score is cited, you can see why it went wrong: it cited the wrong passage, missed an artifact, or stopped at one quote. And overrides are first-class: the reviewer corrects the score with a comment that becomes part of the permanent audit trail, and patterns of overrides are how we improve the questions over time.
How do you prevent demographic bias from creeping in?
Three layers. Names and locations are stripped before the evaluator sees the evidence. We run a paired-candidate bias audit (same evidence, different surface signals) and require the score delta to stay within tolerance before any prompt ships. And the questions are tied to operational competencies, not communication style or self-presentation.
Can I reproduce a score from six months ago?
Yes. Every evaluation persists the prompt version, model snapshot, seed, and evidence bundle. The same inputs through the same pinned model produce the same outputs. If a vendor releases a new model, your old scores are unaffected, bound to the snapshot they were produced under.
Is the AI making the decision?
No. The engine produces an evidence-backed assessment for each criterion; the human makes every call. The platform never auto-rejects, auto-advances, or hides anyone from a reviewer. The engine’s job is to make your job easier, not to replace it.
What if a simulation didn’t surface enough evidence?
That criterion is marked Not Assessed, plainly. We don’t average missing evidence into the score or pretend confidence we don’t have, and you can re-run with a different simulation if you need more signal.
See it live
Want to see this on a real operator?
We’ll walk you through a live evaluation (every observation, every citation, every override) for a role you’re working on right now.
Or reach us directly: contact@yolexlabs.com