Failure-mode categories
forgetting- A stale or lost fact is returned in place of the current one — the system "forgot" an update and surfaced an outdated value.
chronology- Facts are returned in the wrong time order. Even when every item is individually correct, a mis-ordered history makes downstream reasoning unsafe.
entity-confusion- A fact is attributed to the wrong entity — the right information is pinned to the wrong person, account, or object.
Checks
answerEquals- Asserts the system-under-test’s answer exactly equals an expected value (case- and whitespace-insensitive).
mustContain- Asserts every listed value appears in the answer — used to confirm a current or retained fact was surfaced.
mustNotContain- Asserts no listed value appears in the answer — used to confirm a known-stale value did not leak back in.
order- Asserts the actual sequence deep-equals the expected sequence — the chronology check. Any reordering fails.
attributedTo- Asserts the answer’s entity matches the expected entity (case- and whitespace-insensitive) — the entity-confusion check.
Verdicts
pass- At least one check passed and none were failed or unknown.
fail- At least one check failed. Fail takes precedence over every other verdict.
unknown- No check failed, but the evidence needed to assign a pass was missing (for example, no answer was supplied). A first-class verdict: missing evidence never fabricates a pass or a fail.
Fixture kinds
positive- A fixture whose correct system behavior should score a pass.
negative- A fixture that encodes a real regression — the system behaving wrongly should score a fail.
malformed- A valid fixture-run that exercises a malformed memory scenario (distinct from an invalid input envelope, which is a contract error).
missing-data- A fixture where required evidence is absent, exercising the unknown verdict path.
adversarial- A fixture built to trip a naive scorer — look-alike values, partial matches, or injection-shaped content — that the evaluator must still score honestly.
See also
The methodology page walks through how these checks combine
into a verdict. Contract version 1, evaluator
1.0.0.