Why It Matters to Me
the first goal: selfish, in the best way.
This project has two goals, and I'll be honest about both of them.
The first is selfish in the best way: I wanted hands-on experience building a real chat agent and a real eval framework. As I grow into the LLM space — and now move into a PM role on an LLM Platform team — I needed to actually do this, not just read about it. You learn evals by writing evals. You learn prompting by watching your prompts fail.
The second goal was more fun: I wanted a better way for people to learn about me. A resume is flat. A LinkedIn profile is static. A chat agent that can answer questions, stay in character, handle edge cases, and tell you when it doesn't know something — that's a living product. It's also just a more interesting way to show up.
The fact that the visitor is using this agent right now, while reading about how it was built, is not lost on me. That's the part I enjoy most.
Problem It Solves
Portfolios are static. Resumes are one-dimensional. For someone who thinks in systems, builds in public, and cares about how AI products actually work — a chat agent is a better representation than a PDF.
Recruiters, colleagues, and other builders can ask specific questions and get specific answers — not scroll through a resume hoping to find what they need
A well-prompted, well-evaluated agent demonstrates LLM product skills better than any bullet point could
The eval framework turns an otherwise invisible quality bar into something visible and measurable
The site becomes a living project, not a static artifact
What I Built
87 cases now. the suite started at 16.
The agent — a Claude-powered guide embedded in hidustin labs., with page-aware context and direct links into the relevant case studies
The knowledge system — durable voice, privacy, and safety instructions separated from a canonical structured project registry shared by the site and the backend function
The eval framework — a test suite with 10 categories: factual, boundary, tone, personality, consistency, depth, interview, injection, offtopic, and ama. Uses mustContain, mustContainAny, and mustNotContain string checks for automated scoring, with manual review flags for tone and adversarial cases
The eval runner — a separate function that runs the full test suite against the live agent and surfaces pass/fail results with a score and trend history
Operational safeguards — bounded message history, payload validation, best-effort rate limiting, asynchronous 90-day logs, and metadata for knowledge version, latency, tokens, page, starter, and outcome
What I Learned
the eval runner was testing a months-old copy.
Context is everything
The model is only as good as what you give it. Early versions of the knowledge base were too thin — the agent would hallucinate details or deflect questions it should have been able to answer. The fix wasn't a better model. It was better context. Writing a thorough, well-structured knowledge base is a product skill, not a writing skill.
Hallucinations are real and evals catch them
I didn't fully appreciate how often an LLM would confidently return an incorrect answer until I had an eval framework to surface it. Evals aren't optional for a production agent — they're the only way to know what's actually happening. Discovering that the eval runner had its own months-old knowledge base and was never calling the live function was exactly the kind of silent failure evals exist to catch.
Eval precision matters more than you'd expect
mustContain checks are case-sensitive and exact. "1,400" and "1400" need to be treated as separate cases. An eval that's too loose gives you false confidence. An eval that's too strict creates false failures. Tuning that balance is its own skill — and one I now understand at a practical level, not just a theoretical one.
Things break and that's the point
As a non-technical PM, hitting walls is part of the process. What I learned is that tools like Claude and Codex make it possible to move through those walls faster — but understanding why something broke matters as much as fixing it. Every failure taught me something I'll carry into my day job on the LLM Platform team.
Where It's Going
NOWKeep the shared project registry current as products, statuses, and milestones evolve
NOWRun the 87-case suite after knowledge or behavior changes and review failures before publishing
NEXTUse page context and response metadata to improve retrieval, starter prompts, and navigation into case studies
LATERUse this as a personal sandbox for LLM concepts I'm learning on the ML Platform team — a place to try things in public