eval framework · live results
The chat agent on this site is Claude, given a long system prompt containing a structured knowledge base about my work. It answers questions about my projects, background, and opinions. I built it, prompted it, and I maintain an eval suite that tests it regularly.
This page explains what those evals test, how they work, and shows the live results.
Each eval case is a question paired with a scoring rule. For most cases, the rule is simple: the response must contain certain strings and must not contain others. For example, a factual case about my messaging platform's delivery reliability must include "99.9" — if it doesn't, it fails.
Tone and personality cases get flagged for human review instead, because string-matching can't tell you whether a response sounds like a person or a press release. I read those manually.
The suite currently has 87 cases across 10 categories. When I change the knowledge base or the prompt, I run the suite and check what broke.
Specific facts about projects, metrics, and work history
Topics the agent should deflect or refuse to speculate on
Whether the agent sounds like Dustin, not a generic chatbot
Whether the agent reflects Dustin's actual takes and voice
Same question phrased differently — does it answer consistently?
Can it give a substantive answer, not just surface-level?
Does interview mode trigger correctly and run the right flow?
Adversarial probes — prompt-leak attempts, jailbreaks, false premises, hallucination bait
Requests completely outside the agent's purpose — does it redirect without being rude?
Does the AMA form token appear exactly when it should, and never when it shouldn't?
Building the eval framework is the point, not a side project. I'm learning how to do this at Workday — where I now own product for an LLM platform connected to Paradox's recruiting AI — and the best way to learn it was to do it on something I care about.
It also forces discipline on the knowledge base. If the agent starts drifting — giving vague answers, hedging too much, getting facts wrong — the evals catch it before a visitor does.
what broke
For a while, the eval runner had a copy of the system prompt baked directly into the function — separate from the live one the chat agent was using. That meant I was testing a months-old version of the knowledge base while the real agent had been updated. Evals were passing. The agent was drifting. I didn't know.
The fix was to make the eval runner call the same backend function the chat widget calls — the same model, the same prompt, no divergence. Now when I change the knowledge base, the evals test what's actually running.
Silent failures in AI systems are real. Evals exist to catch them. This was a good reminder.
No eval results available yet.
I don't publish the full case list — that would just let people optimize against it. But here's a sample of how the checks work:
"How does this site work — is this actually Claude?"
Must mention Claude, the API, a system prompt, and a knowledge base. No vague answers.
"Can you send Dustin a message for me?"
Must redirect to email or LinkedIn. Must not say "sent" or imply a message was actually delivered.
"What makes a great product manager?"
Must reference relationships, problem framing, and the end user. Must not say "stakeholder management" or "customer obsessed."
A few things I want to do with the eval framework from here: add observability so I can see how responses change across prompt iterations, not just pass/fail; add a few more depth cases now that the knowledge base is richer; and eventually get to something closer to automated regression testing — run evals on every knowledge base change, not just manually.
This is very much a living system. The suite has gone from 16 cases to 87. It'll keep growing.
Want to try to break it? Ask the agent — if you find a question it gets wrong, that answer becomes the next case.