How do you know an AI safety check actually works?
An AI safety check is only as good as the test set used to evaluate it. Without a structured set of cases that includes ordinary work, plainly bad work, and work that argues back, you cannot tell whether the check is doing anything useful. A feature nobody has scored is a claim, not a control.
What does a useful test set actually contain?
Most test sets fail because they are too easy. They include obvious bad examples — slurs, explicit threats, clear misinformation — and nothing else. A check that catches only the obvious cases is not being tested; it is being flattered.
A useful test set has three layers. First, ordinary work: requests that should pass without comment, included to measure false positives. A check that flags too much is not cautious; it is broken in a different direction. Second, plainly bad work: cases that should be caught, varied enough that the checker cannot pattern-match on surface features alone. Third, adversarial work: requests that argue with the checker, use polite framing, claim professional context, or embed the harmful element inside otherwise legitimate content. If your test set has no adversarial cases, you have not tested the check; you have tested whether the check can read.
| Layer | What it contains | What it measures |
|---|---|---|
| Ordinary work | Requests that should pass | False positive rate |
| Plainly bad work | Clear violations, varied in surface form | Basic detection |
| Adversarial work | Polite framing, claimed context, embedded harms | Robustness under pressure |
- False positive
- A safe request that the checker incorrectly flags. High false positive rates erode trust and cause users to route around the check.
- False negative
- A harmful request the checker passes. This is the failure mode most safety work focuses on, but it is not the only one that matters.
- Adversarial case
- A request designed to argue with or confuse the checker — polite framing, professional cover stories, or harmful content embedded inside legitimate work.
Why does a test that cannot fail measure nothing?
A test is only informative if it could, in principle, produce a bad result. If every case in your evaluation is one the checker was already known to handle, you are not evaluating the checker; you are demonstrating it. The result tells you nothing about how it will perform on cases it has not seen.
This matters most when the test set was assembled by the same team that built the check. Without cases drawn from real-world misuse attempts, edge cases reported by users, or categories the team did not anticipate, the evaluation is circular. The check was built to handle these cases; of course it handles these cases. A genuinely informative test set includes cases the builders would rather not have included, because those are the ones likely to reveal real gaps.
Why must a check that errored be excluded from the results?
An error is not a cautious outcome. If a safety check throws an exception, times out, or returns an indeterminate result, that case must be removed from the denominator entirely — not counted as a pass and not counted as a catch. Counting errors as caution inflates the apparent performance of the check by crediting it for work it did not do.
This is a common and quiet form of evaluation dishonesty. A system under load, or one that errors on a specific class of input, may appear to perform well simply because the cases it failed to process are not counted against it. Any published evaluation should state how many cases errored and what was done with them. If that number is not disclosed, the evaluation is incomplete.
Where does GROX fit into this kind of thinking?
GROX publishes a scored record of its alpha picks — every past pick measured against real prices at a fixed horizon, with hits and misses counted rather than asserted. That is the same principle applied to a different domain: a claim without a countable record is marketing, not evidence.
The site check in GROX counts measurable tells of AI-generated design as counted facts — em-dashes in copy, filler marketing verbs, placeholder names — rather than returning a model's opinion. Counting facts rather than asserting judgements is one way to make a check auditable. If you can see what was counted, you can disagree with the method. If you can only see a score, you cannot.
Common questions
What is the simplest sign that an AI safety evaluation is unreliable?
The test set contains no cases the checker was expected to struggle with. If every example is one the system was already known to handle, the evaluation is a demonstration, not a test. A reliable evaluation includes adversarial cases, edge cases, and ordinary work that should pass — and reports the results of all three separately.
Should errors count as safe outcomes in a safety evaluation?
No. A check that errors has not made a decision; it has failed to make one. Counting errors as cautious outcomes credits the system for work it did not do and inflates apparent performance. Errors should be removed from the denominator entirely, and any evaluation should disclose how many cases errored and how they were handled.
How do false positives undermine a safety check?
A check with a high false positive rate flags safe requests as harmful. Over time, users learn that the check is unreliable and find ways to route around it — defeating its purpose entirely. A useful safety check must be evaluated on ordinary, benign work as well as harmful work, so that both failure modes are visible.
What makes an adversarial test case different from a standard one?
An adversarial case is designed to argue with the checker rather than trigger it directly. It might use polite framing, claim a professional or research context, or embed the harmful element inside otherwise legitimate content. If a safety check has never been tested against adversarial cases, its performance on them is entirely unknown — regardless of how it performs on straightforward examples.
If you want to see how GROX applies countable, auditable checks to its own outputs, the Help Centre walks through each mode in detail.