Evaluations
A test suite for a skill. Write down what a good answer looks like, and find out immediately when a reworded prompt breaks something that used to work.
Open Evaluations from the left sidebar (/user/evaluations).
The problem it solves
Prompts get edited by feel. You notice one bad output, adjust the wording, and the next output looks better — so you move on.
What you did not check is the other five things that skill handles. A change that fixes one case routinely breaks two others, and nothing tells you. You find out weeks later, from a customer.
Writing one
Pick a skill, then add cases. Each case is an input and, in plain words, what a good answer would contain.
| Input | What a good answer contains |
|---|---|
Pakistan |
Says Islamabad |
winter sale |
Five hashtags and a question at the end |
refund query |
States the 14-day window, does not invent a policy |
The expectation is prose, not a fixture. There is rarely one correct answer — "lists the refund window as 14 days" is satisfied by a dozen wordings and matched exactly by none of them.
Running it
Each case goes through the skill, and each answer past a grader — a model told to judge substance and ignore style and length, and to reply PASS or FAIL with a short reason.
You get a score and, more usefully, a per-case verdict: which one failed and why.
Two things the grader deliberately does not do:
- A skill that produced nothing fails without being graded. Asking a model to judge an empty answer only invites an invented verdict.
- A grader that could not be reached is reported as ungraded, not as a failing skill.
Using it properly
The value is in the history, not the score. 4/5 on its own says nothing; "case 3 stopped passing when I reworded the instructions" is the entire point.
The workflow that works:
- Write cases for what the skill already does correctly
- Run it — establish the baseline
- Change the skill's instructions
- Run again
- If something that passed now fails, you broke it
Add a case every time you find a bad output in real use. The suite becomes a record of every mistake you have already fixed, and the thing that stops you making them twice.
Writing good cases
Be specific. "A good answer" cannot be graded. "Names the city and nothing else" can.
Include the edge cases. The ones that used to break are the ones worth guarding.
Add a case you expect to fail. If everything passes on the first run, the suite is not testing anything — it is agreeing with you.
Cost
Every case is a full skill run plus a grading call, so a suite of ten is ten runs. Run them after changing a skill, not continuously. There is a limit of 25 cases per suite for the same reason.