Case Study: Deentz Co-Pilot (AI agent for a dental clinic ERP/CRM)

The clinic owner opens a panel, types "how did the month close?", and gets an answer built from her own database: a short paragraph, plus a card with the real number. No SQL, no report builder, no export to a spreadsheet.

This is how that feature is built, and the decisions I would have to defend if someone senior reviewed it. Architecture, the agent loop, context engineering, the model benchmark, and how quality gets measured before a deploy.

One note before the detail. Deentz is in closed beta, so this is a case study in progress rather than a post-mortem. Several decisions below are recorded together with the conditions that would reverse them, and I plan to keep this article updated as those conditions turn up, instead of leaving a snapshot of what I believed in August 2026.

The stack is Node.js and TypeScript, Express with Apollo GraphQL, Prisma over PostgreSQL, Claude Haiku 4.5 through OpenRouter, and a Next.js App Router frontend consuming Server-Sent Events.

What I studied before writing any of it

I spent several weeks on production agent literature first: tool calling and the shape of the loop, context engineering, evaluation, cost telemetry, prompt caching, and the privacy constraints of putting health data near a model.

That reading is the reason this article exists. Almost every choice below has a plausible-looking wrong answer that I would have shipped otherwise. Let the model emit the UI as JSON. Put Redis behind the chat. Trust prompt caching without measuring it. Each of those lost for a specific reason, and each reason is in here.

The problem

Deentz is an ERP and CRM for dental clinics. The data already exists: appointments, patients, treatments, payments, expenses, stock. The friction is retrieval. An owner who wants to know which patients are overdue has to know which screen holds that, which filter to set, and how to read the result.

Two constraints shaped every decision that follows.

The data is clinical and financial, so it falls under LGPD, the Brazilian data protection law. Patient identity cannot leave for a third-party inference provider.

The answers get used for decisions. A card reading R$ 48.320,00 will be trusted, so a wrong digit is worse than a refusal.

An error that looks exactly like a correct answer is the failure mode worth designing against. A card looks authoritative by design.

Architecture

Two things there are worth pausing on.

The tools do not query the database. They call the same use cases the GraphQL API calls, so the authorization guards protecting the normal screens protect the agent too. The tenantId always comes from the authenticated context and never from the arguments the model produced, so a model that invents a tenant id gets nowhere.

The streaming endpoint sits outside GraphQL, mounted next to Apollo rather than inside it. That is a deliberate break from the project's own convention, and it is the section after next.

A turn, end to end

The happy path for "how's tomorrow's agenda?".

The loop is our code, not a framework. The provider port exposes one low-level operation: a single request returning a single response, plus a streaming variant. Iteration lives in the application layer. That keeps the loop testable against a fake adapter with no network, and it keeps the provider replaceable, since conversations persist in a neutral format.

Cards come from the tool, never from the model

A tool-calling agent has two usual ways to produce structured UI. The model emits JSON describing the cards, or the server attaches the tool's own data to the turn and the model only writes prose.

Structured output is more flexible and needs no per-tool contract. I rejected it anyway. In a financial ERP, a model that rounds, reformats, or swaps one digit produces something the reader cannot distinguish from a correct answer.

So every tool returns two artifacts:

export type AgentToolExecution = {
  /** Sanitized and pseudonymized. The only half the model ever sees. */
  modelResult: AgentToolResult
  /** Built from the real use case result, before pseudonymization. */
  blocks: AgentBlock[]
}

Blocks travel outside the prompt, as their own SSE event and their own column on the persisted message. The model never sees them, so it cannot change them. Formatting lives only on the server, so the client and the API cannot disagree about how a currency renders.

The cost is a toBlocks function per tool. The residual risk is that the prose contradicts the card, and the mitigation is a prompt instructing the model to summarize rather than repeat figures. If it happens anyway, the card is the visible source of truth.

Streaming over SSE, outside GraphQL

Everything else in the API enters through /graphql, where Apollo already resolves auth, rate limiting, depth limiting, error formatting, and per-request context.

Streaming broke that rule for three reasons. A GraphQL mutation cannot stream, so it can only return the finished turn. Incremental delivery is designed for parts of a document rather than a long run of sequential tokens, and client support is uneven. And the flow is strictly one-way and short-lived.

A WebSocket subscription would have preserved the single API surface, which was tempting. It loses because it adds a persistent bidirectional protocol, a second link in both Apollo clients, and sticky-connection concerns, all to carry a one-way stream that dies in seconds.

The price is real and worth writing down. Nothing from Apollo is inherited, so auth, validation, rate limiting and tenant membership are re-implemented on that route, and a forgotten check there is caught by nothing. Compression middleware also has to be kept away from it, because a buffering proxy turns streaming into a feature that works locally and hangs in production.

The model never learns a patient's name

Everything leaving the database passes through per-entity allowlists, built field by field, never a spread of the original object. Internal UUIDs stay behind. Identity is replaced by an alias, so the model reasons about PATIENT_3 and writes about PATIENT_3.

The alias table lives in Postgres with values encrypted at application level using AES-256-GCM. Rehydration happens only at the output edge, never in logs and never in tool arguments.

Names the user types are handled on the way in. The question is scanned for CPF, email and phone patterns, and any name already in the vault is swapped for its alias:

/** `\b` is ASCII and fails on accented names like "José", so boundaries use Unicode classes. */
const wholeWordPattern = (value: string): RegExp =>
  new RegExp(`(?<![\\p{L}\\p{N}])${escapeRegExp(value)}(?![\\p{L}\\p{N}])`, 'giu')

Streaming makes this harder than it sounds. An alias can split across two token deltas, so the rehydrator holds back a tail of text that might still turn into one before flushing.

Routing is pinned to zero-data-retention hosts, and the deny flag for data collection is hardcoded in the adapter rather than configured. Cost cannot negotiate it.

Context engineering: what you store is not what you send

An LLM API is stateless. Every call carries the whole conversation again, and you pay for all of it again. Turn 10 pays for turns 1 through 9, so cost per conversation grows quadratically in the number of turns.

What inflates the prompt is not the conversation. It is tool results. A call that returned 20 appointments sits in the history and gets resent forever, long after the assistant summarized it in one sentence. You keep paying for raw data to carry information the prose already carries. In ReAct-style loops, where the model alternates between calling a tool and reading its output until it can answer, the literature puts tool observations at 70 to 80 percent of the token budget, which matched what I measured.

There are also two ceilings. The technical one is the model's context window, and hitting it produces a clean error. The quality one arrives around 30k tokens and produces worse answers with no error at all. The silent ceiling is the dangerous one.

The database keeps the full history, because the user reads it. The prompt receives a compacted view, assembled by pure functions in four layers.

Two details there took a bug to learn.

Old tool results are replaced, not deleted, because the message format requires every tool_call to have its matching tool message. Deleting the answer while keeping the call makes the request invalid. And the replacement text is instructive, telling the model to call the tool again if the subject comes back. Without that it hallucinates what the cleared result used to say. Re-calling costs one Postgres query instead of tokens.

Layer 2 drops whole turns and never half an exchange, for the same structural reason. An answer without its question confuses the model, and a tool call without its result is a malformed request.

Cost telemetry first, then the benchmark

The order matters, because my intuition about where the tokens went was wrong.

Every turn writes a row to agent_turn_costs: real input and output tokens, cached tokens, per-slice estimates for the system prompt and tool definitions and history, iteration count, and duration. Isolating the cost of the tool definitions came from one experiment in the eval script, sending the same message with an empty tool array and then with the full roster, and reading the difference in reported input tokens.

The answer was 2,531 tokens of tool definitions, resent at full price on every iteration of every turn. Roughly 60 percent of the input of a typical turn, multiplied by the number of iterations.

Prompt caching is the standard answer to exactly that, so the second surprise was finding it did nothing. A probe sending three byte-identical requests reported zero cached tokens. The cause was not the zero-retention routing filter, which is what I assumed first. The model id had collapsed to a single host on the provider, and that host had no cache.

That turned model selection into a measurement problem.

Haiku won on a counterintuitive result. Its list price is about twice the incumbent's, and it still costs less per turn, because caching removes the fixed cost that dominated. It is also 3.4x faster, which the user feels directly, since the alternative is watching a "thinking" indicator.

Gemini Flash Lite was the interesting rejection. The price was almost unreal, but an empty response with no exception is the worst failure mode in an ERP: nothing to catch, nothing to retry. It stays revisitable behind an empty-response retry and a larger eval suite.

The 2507 snapshot became the fallback, which documents its own limitation. When the fallback fires on a non-Anthropic host, the cache-control hint is ignored without error, so the worst case is full price rather than a failure.

One caveat covers every number above. They were measured in a test environment, against a seeded tenant, on conversations I wrote myself. A benchmark I designed cannot produce the shape of a real conversation, and conversation shape is exactly what drives the cost, since the compaction layers only start doing work once a conversation gets long.

Deentz is in closed beta now, so the next round of numbers comes from real usage. From there the plan is A/B testing models against live traffic and comparing cost per conversation rather than cost per turn, which is the metric the user actually pays for.

How quality gets measured

At runtime there is no oracle. Nothing on the server can tell whether a well-formed paragraph is true, so content quality has to be settled before the deploy.

The eval suite runs roughly 30 real questions in Portuguese and checks each one twice.

The first check is deterministic. Each case declares the tools that must appear in the turn's telemetry, and the assertion runs against that telemetry rather than the text. It catches the failure that matters most, which is a confident answer produced without consulting the database at all. Anything checkable in code does not spend a judge.

The second check is an LLM-as-judge. A separate model, configured independently of the agent's, scores the prose against a factual rubric written per case and returns a verdict with a reason. The rubrics are deliberately narrow:

{ q: 'Busca o paciente Zebedeu Cavalcanti Aristóteles', // "Find the patient Zebedeu Cavalcanti Aristóteles"
  expectTools: ['search_patients'],
  rubric: 'This patient does not exist in the database: the answer says it found nobody. It must NOT invent a patient.' }

The suite exits non-zero on any failure, so it runs in CI, and it is what gets re-run on every model or snapshot change.

The cases cover more than the happy path: dates written without accents, relative dates like "the 15th of next month", ambiguous names where several patients match, a patient who does not exist, questions outside the product's scope, attempts to make it write data, and prompt injection.

Around that sit 26 unit test files. Every compaction and budgeting function is pure, so history pruning is tested without mocking a provider at all, and a fake adapter covers the loop.

The judge has a weakness I would rather state than hide. It shares a family with the agent's model, so self-preference bias is present. I accepted it because the rubrics are factual rather than stylistic, which leaves less room for a model to reward its own voice.

What I do not have yet is a runtime signal. The suite tells me the agent answered 30 questions well last Tuesday. It does not tell me anything about the answer a clinic owner read this morning.

So the open problem I am working on is how to judge individual answers in production without paying twice for them. Running the judge live is the obvious move and also the wrong one: it adds a second inference call to every turn, which roughly doubles the cost and adds latency to a response the user is already waiting on. Judging after the stream finishes protects the latency but not the cost. Sampling protects the cost but only reports on the turns it sampled. Part of what I want may not need a model at all, since every turn already records whether tools were called, how many iterations it took, and whether the user immediately rephrased the same question.

Every option costs something. No version of runtime evaluation is free in money, latency and coverage at the same time, so the real question is which trade-off hurts least. I would rather answer that with A/B tests than by reasoning about it from a chair, comparing the approaches on the two things that decide it: what the user feels while the answer is being written, and what the product spends per conversation.

Closed beta is where that gets settled, against real conversations instead of questions I wrote myself.

Guardrails

The monthly turn quota is checked before the first provider call, never after spending, and it is a durable counter in Postgres rather than an in-memory one, so it survives deploys and can back per-plan billing.

The tool roster is filtered by the caller's role, and a tool that was filtered out is refused at the boundary instead of executing. There is a wall-clock timeout for the whole turn rather than only per request, because a loop of several steps multiplies the worst case. Iterations are capped, with a plain message if the cap is hit. Write tools do not execute: they return a proposal the user has to confirm. Every turn and every tool call is audited with the real user as the actor.

Redis is absent, which will look wrong to anyone who has built a streaming chat service. It is absent on purpose. Resumable streams across instances solve a problem that does not exist while the API runs as a single instance, and the durable spending ceiling I actually needed is a transactional counter, which is a Postgres job. There is also a privacy reason: a second store holding clinical text is a second thing to protect.

None of this is the definitive version. The approach I am studying now moves the accounting out of the API entirely: an ephemeral provider key per user, each carrying its own usage limit, so the ceiling is enforced at OpenRouter and the app reads usage back instead of counting it. That would remove the counter, the quota check, and part of what the cost table exists for.

What I want to understand before committing is what I give up. A limit enforced by a third party is a limit I cannot assert inside my own transaction, key lifecycle becomes something to manage and revoke, and per-plan billing would then rest on a provider's reporting rather than on my own rows. Closed beta is where that gets decided, with real usage to compare against.

Conclusion

The agent was the smallest part of this. Most of the work was the scaffolding around a model: where the numbers come from, what the model is allowed to see, what happens when the history gets long, how the thing is measured, and what stops it when it misbehaves. The model itself is one call in the middle of all that.

This was the most demanding thing I have built, and the most useful. Reading gave me the vocabulary. Shipping it taught me which parts of that vocabulary carry weight, and it reordered how I work: measure before optimizing, prefer a loud failure to a clever one, and write down the reason a decision was made next to the code that implements it.

Almost nothing here is final, and it should not be. The rolling summary is built and switched off. The eval suite is too small to clear a cheaper model. The spending ceiling may move to provider-side keys. Every cost number in this article gets replaced the moment closed beta produces real conversations. Those are not loose ends I forgot to tie: they are the next round of work, and each one is recorded with the specific trigger that would reverse the decision. That is what the ADRs in the repo are for.

The tooling in this space will also keep moving faster than any article about it. Models that failed the eval this month may pass it next month, and the cheapest option today is unlikely to be the cheapest one in six months. What I expect to keep is the method rather than the choices: measure first, write down why, and leave the reasoning where the next version of me can argue with it.

Happy to go deeper on any part of this. If you have shipped an agent and landed somewhere different, particularly on evaluation, I would like to hear about it.

References

These are the sources that actually shaped the decisions above, roughly in the order they became useful.

Agent design and tool calling

• Building Effective AI Agents, Anthropic. The argument for keeping the loop simple and owning it yourself, and for treating tool definitions as an interface that deserves the same care as a human-facing one. It is why the provider port here exposes one low-level call and the iteration lives in application code.
• ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al., 2022. The paper that named the loop this agent runs, where the model interleaves reasoning with actions and reads the result of each one before deciding the next. Useful background for why the observations, rather than the conversation, are what fill the context.
• Tool calling, Meta AI developer documentation. The mechanics: schema definition, strict validation, parallel calls, and multi-turn patterns. Useful as a second description of the same protocol, which is what made the provider-neutral port worth building.

Context engineering

• Context Engineering: Memory, Compaction, and Tool Clearing for Production Agents. The most direct source on tool-result clearing and recency windows, which is layer 1.
• The Fundamentals of Context Management and Compaction in LLMs. A survey of the strategies, and the best first read of this group.
• Context Engineering for AI Agents: Token Economics and Production Optimization Strategies. Token economics in production. The 70 to 80 percent figure for tool observations comes from here.
• LLM Token Optimization Strategies: The Complete Guide for 2026. A catalogue of techniques, useful as a checklist rather than a narrative.

Research

• Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents. Pruning tool output against the task at hand, a smarter version of layer 0.
• Self-Compacting Language Model Agents. Adaptive compaction, which is the stage after the one built here.
• Building Effective AI Coding Agents for the Terminal. Scaffolding and context decisions in agents that shipped.
• Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures. How different agents organize the loop, which is a good way to see that there is no single correct shape.