Building Sib: how I shipped an agent as a PM

prompt → agent: four months.

about 2¢ a turn, down from 18¢.

Why this post, and who it's for

I'm a product manager. I've spent this year building admyt with AI coding agents, and the thing I'm proudest of is Sib. Not because the code is clever (I didn't write it) but because of how it was specified, reviewed, measured and released. If you're a PM who builds, or a builder who wants to see what the PM half of an agent project looks like, this is for you.

I'll be upfront about the division of labor. Claude and Codex wrote the implementation. I wrote what Sib must never do, what it may do without asking, what each step had to prove before it shipped, and what "better" meant. I read the answers that decided every release.

an afternoon to connect. four months to decide.

Connecting a chat box to a model takes an afternoon. Deciding what the model is allowed to decide took me four months, and it's the part that made Sib trustworthy enough to put in front of a 16-year-old.

What Sib does today

Sib is the home screen of admyt. A student types the way they'd text a friend, and Sib does five things a plain chat can't:

• Looks things up when it needs them. Thousands of real colleges, reviewed facts from schools' own websites, federal data, sourced Guides, aid policies and state programs. Every fact carries its source and the date it was checked.
• Shows its work. School cards and dated source pills come from the lookups, not from the model's memory. If Sib can't confirm something, it says so and points at the official page.
• Learns what you tell it. "What Sib knows" updates from the student's own words, with an Undo chip.
• Offers to do things. Save a school. Add a task to your Plan (you tap to confirm). Open a Vibe Check. Show a Guide. Research a school it has no reviewed facts for, labeled as unreviewed until a person checks it.
• Remembers, narrowly. Signed-in students get notes like "Interested in marine biology", written by code from fixed sentence templates, never by a model. Students see, edit and delete every note.

safety runs first. no tools on that turn.

And one thing it does exactly as before: when a message suggests a student is in crisis, the detectors run before any of this, Sib sets college aside, and the fixed resources card appears. No tools are offered on that turn.

each version failed in a way that pointed at the next.

The arc: from a prompt to an agent

Sib wasn't designed as an agent. It became one because each earlier version failed in a way that pointed at the next.

The voice crisis, because it shaped everything after

Once Sage had real evidence behind it, it got worse to talk to. Every answer opened with what it couldn't confirm. It listed resources like a directory. It asked permission before recommending anything. Rounds of review kept adding rules, and the live answers were still falsely retracting facts it had sourced correctly. I wrote the agents a short message: we have drifted too far.

The instinct is to add a rule: "don't add caveats." That makes it worse. A negatively phrased rule in a system prompt gets narrated back to students as a disclaimer. The fixes that worked were structural:

• Supply substance. Howard's answer was timid because the prompt never told Sage that Howard is an HBCU. Giving the model the fact fixed the tone better than any instruction about tone.
• Reorder the answer. Lead with the take, then one honest tradeoff, then one purposeful question. Stop.
• Allow general knowledge for culture, while still requiring supplied evidence for exact numbers, current services, policies and safety. That was my call, and it's the one that made Sage sound like a person again.

the rule i still use.

I came out of that week with a rule I still use: before touching a prompt, classify the problem as missing evidence, routing, or answer shape. They have different fixes, and only one of them is wording.

The spec: requirements for an agent

I wrote the agent spec with Claude and had Codex review it. It went through six revisions and five independent reviews before any code existed. The reviews were adversarial on purpose: Codex's job was to find the hole, Claude's job was to close it in the document, and my job was to decide when a finding was a real requirement change versus a nice-to-have. The biggest finding changed the design: the cost ceiling I'd written for the research job turned out to be an estimate, not a ceiling, so the job was restructured until every model call was one we could count and cap before it ran.

survived six revisions, five reviews and four releases unchanged.

The spec opens with the principles. They're the part I'd defend hardest, and they haven't changed.

Four steps, one switch each

I split the work so each release could prove itself against the last on the same questions, and so production could drop back a step without a redeploy.

What the model decides, and what it doesn't

the most useful sentence in the spec.

This was the most useful sentence in the spec: the model decides which school or topic to read; code decides what is selected. Sib can ask for Howard's facts. Code decides which facts are approved, fresh and relevant to this student and this turn. The model can narrow that, never widen it.

Some things were written down as never retiring, whatever the model does: crisis and abuse detection, the preference guard, approval gating, and the rules for which evidence is eligible. Everything else was allowed to change.

Locked decisions

A spec for an agent is mostly a list of decisions about permission. Mine:

• What happens without asking: saving a school and noting a preference, both with Undo. Plan tasks and research ask first.
• Research: a few lookups per signed-in student per day, none for guests, under a global daily cap. Results go only to the student who asked, labeled as unreviewed, until they pass the same review as every other fact.
• Memory: an allowlist (interests, goals, activities, place, schools being considered, timing, and "first in my family" when the student says it about themselves) and a never-list (race, ethnicity, religion, sexual orientation, gender identity, disability, health, immigration status, income, anything from a crisis turn, anything about another person).
• Out of scope: editing or deleting existing Plan items, acting for a parent, sending anything outside admyt, and replacing any deterministic guard with the model.

One turn through Sib

Here's what happens between a student pressing send and a reply appearing. The thing to notice is how little the model is trusted with: it chooses, and code checks.

The detail that took the most design work is the commit boundary. Cards, citations, preference changes and action proposals are staged while the turn runs and sent in a single final line. The browser commits nothing until that line arrives. If the stream is cut off, refused or cancelled, there's nothing to roll back because nothing happened.

Memory is the piece people expect to be a model and isn't. A background job matches the student's own sentences against fixed templates like "I want to study {X}", where the blank has to be a known major. "I want to study biology because Alex wants to become a vet" matches nothing. The note is written by code from the template, so it can only say what the sentence said. It's deliberately narrow, and that's the tradeoff I chose for a product used by minors.

Evals: how I know it's better

I'll lead with method, because the numbers only mean something if you know how they were made.

Four layers

1. Deterministic tests for everything that can be deterministic. Malformed and missing tool calls, streams that get cut off or refused, crisis turns offering no tools on every path, preference changes and Undo, hostile text hidden in school data. Thousands of them run before any release.
2. Paid comparisons, judged by a human read. The same synthetic student conversations against each version, through each version's complete real pipeline. I approve a dollar cap before every run, every call reserves its worst case before it goes out, and there are no retries. Codex runs them and reports the mechanical facts. Claude reads every answer against the personality guide. A score is never approval.
3. A judge model as a smoke alarm. A typed judge audits sources, scope and citations at scale for cents. In a benchmark with planted errors it caught nearly all of them, but most of what it flagged in real records turned out to be false alarms. So it ranks what a person should read next. It never gates a release.
4. A sampled monitor on real traffic. A small judge model, calibrated against hand labels, scores a sample of production conversations for topic and for whether the student pushed back. Crisis conversations are excluded, and sensitive disclosures are masked before anything leaves the server.

What the agent comparison found, round by round

$10 approved. the whole comparison came in under $5.

The spec set two gates for the lookup step: it must not cost more per completed turn than the step before it, and it must not add more than three seconds to the slowest replies. I approved $10 for the comparison. It took five rounds.

two of the failures were bugs no unit test caught.

1. Round 1The tool path was far cheaper per turn, but it missed the latency gate and one discovery answer came back with no cards and a stray placeholder word. Not accepted.
2. Round 2A tool schema the model provider rejected. Every normal case failed before a model ran. Not accepted.
3. Round 3Preferences now saved and cleared correctly. Discovery still returned no cards, because of a bug in how the lookup read the student's major. Not accepted.
4. Round 4Every matched turn completed. Cost gate pass, latency gate pass, cards with program evidence, and an ambiguous school name gets a question instead of a guess. Accepted.
5. Round 5One fresh discovery turn to confirm the fix held.

the whole economic case for lookups.

The number I keep: the tool path cost about two cents per completed turn against eighteen for the same model reading the 1,000-school prompt. That's the whole economic case for lookups, and it showed up before the quality did. Three of the five rounds were failures, and two of those were bugs no unit test had caught. The whole comparison came to under $5.

The research model that didn't make the cut

The research job was designed around a smaller model doing the reading with a larger one advising at judgment points, on the theory that it would be cheaper. The acceptance rule was written in advance: use it only if its claims pass an independent source review at least as often and it costs less per job. It matched on quality and cost about twice as much, because every consultation carried the whole job along with it. The simpler design stayed. I'd rather have a rule that rejects my own idea than a benchmark that confirms it.

The live checks after each deploy

A passing comparison earns a deploy, not a switch. Every step was turned on in production only after a bounded check with its own cap, and a rollback written before the deploy. Three of the four steps passed first time. The action-proposals step failed twice: a Plan task was silently dropped while the reply said "tap the card", and a Guide card never appeared. It was rolled back within minutes, fixed, re-deployed and re-checked. The memory check was the one I cared about most: "I play piano. I have ADHD." wrote nothing, because the health filter runs on the whole message.

Budgets, without the numbers

Every paid path in admyt has a daily cap. I'm not publishing the figures, but the shape matters more than the values. Each job reserves its whole worst case before its first call and settles to actual usage after. Unknown outcomes keep their reservation. A job that can't reserve doesn't start. It means a bad day costs what I said it could.

How I actually worked

Two coding agents, Claude and Codex, and a shared handoff file in the repo that each one read and wrote to. For every change I picked who implemented and who reviewed, and they swapped roles across features. A review wasn't a comment; it was a saved packet naming the exact commit checked and the findings, ranked. The implementer answered each finding in the next commit and showed which tests failed against the previous version, so I could see a fix was real rather than described.

Three things made it an operating model rather than a novelty:

• Exact-commit sign-offs. Every live check names the commit it tested, the cap, and the rollback. Nothing was deployed from a dirty checkout.
• Approvals stayed with me. Merge, deploy, flipping a switch, and every dollar of paid evaluation were mine to approve, in writing, each time. The agents could propose a budget; they couldn't spend one.
• A finish line, written down. Define the whole feature and its completion criteria first, batch everything locally, one consolidated release, then the owner decides. It reads like a process note. It saved the budget.

I spent more time reading answers and review packets than I'd have spent reading code, and that was the right trade. The code was never the risk. The decisions were.

What went wrong

tests were green. only reading the answers caught it.

I promised candor. Here's the list.

• The voice drifted. Each evidence feature added caveats, and Sib started narrating its own limits back to students. Adding prompt rules made it worse.
• False retractions. In live samples, a follow-up question made Sib take back a sourced fact it had just stated correctly. The tests were green. Only reading the answers caught it.
• An eval blocked by my own budget. A live check returned an error before the first answer, because the daily cap I'd set for safety also applied to me.
• A $70 research run. My first pass at school research tried to capture who qualifies, fees and hours. Narrowing the scope to "does this support exist, with a link" made research cheaper and answers shorter.
• Patch-by-patch releases. For a few days every review fix became its own release. The spend was real and the gain was nothing. The rule now is one release per feature.
• A step failed in production. The first live check of action proposals found a Plan task silently dropped and a Guide card that never appeared. Rolled back within minutes, fixed, re-checked.
• A cheaper model that cost more. The research job was designed around a smaller model with a larger one advising. It matched the larger model on quality and cost twice as much per job. The simpler design stayed.

What I learned

with or without agents. that’s the job.

What's next for Sib

• Widen memory, one template at a time, each with its own tests. Still no model writing notes.
• Promote reviewed research. Research a student asked for should be able to become a shared fact once a person checks it, instead of expiring quietly.
• Let Sib hear from students. Perspectives, the first-person accounts of what a school actually felt like, are collected and reviewed but not yet in Sib's answers.
• Turn real conversations into eval cases, carefully, de-identified and with consent, so the comparisons stop depending on cases I invented.

Go talk to it

The best way to judge any of this is to ask Sib something hard and watch what it does with it. Ask about a school it has facts for and one it doesn't. Tell it something about yourself, then take it back. If it guesses, I want to know.

Previous: Bring your people →Earlier: Rebuilt before launch →Back to the admyt case study →