It's 4:40 on a Thursday. Your processor comes over: the lock on the purchase closing Tuesday expires tomorrow, and the file still shows one open condition — a CPA letter that actually arrived Monday, went in as an uploaded document, and never got cleared. Now it's an extension, and somebody eats the cost.
Nothing about that was hard. The expiration date was in the LOS. The document was in the file. The condition list was one click away for three different people. And the AI your lender rolled out in the spring sat in its chat window all week, ready and willing, waiting for somebody to type the question nobody thought to ask.
That gap is not a gap in intelligence. It is the difference between software that answers and software that works.
The short version
- Grade AI on its write path — what it produces when nobody asked. Answering questions is table stakes.
- Three output shapes, one rule: briefs and internal flags run on their own; anything client-facing is a draft a human taps send on.
- Nothing can be flagged "stalled" until you write a day-count next to each pipeline stage. That job is yours, not the vendor's.
- Set an interrupt budget — a hard cap on items per day — then score the flags every Friday. Noise over a third means your thresholds are wrong, not your people.
- In the demo, ask what it suppressed. Anybody can show you a long list.
Grade the write path, not the answer path
Every AI demo you will sit through is a demo of the answer path. Somebody types a question, the model returns a paragraph. Impressive, and table stakes.
Your branch's expensive problems are not the questions you asked. They are the ones nobody knew to ask at six in the morning on the day it mattered. So grade the other side — the write path. What does this thing produce when nobody is logged in? A chief of staff, human or otherwise, is measured by what is already on your desk before you sit down.
There are three shapes worth paying for.
- The brief. One ranked list, once a day. What changed overnight, what is at risk this week, what deserves your first hour. Runs on its own.
- The flag. A mid-day interrupt on one file, because a clock crossed a line. Runs on its own, internally.
- The draft. A written artifact — the borrower status update, the note to the listing agent, the condition chase — sitting in the LO's queue with the facts already right. Never sends itself.
An AI that only answers is a search bar with manners.
The answer path still earns its keep — you will want the average loan amount on this month's closings in ten seconds instead of an export. It just is not the product.
Nothing is "stalled" until you write down normal
Here is where most AI pilots quietly die, and it has nothing to do with the model.
You want it to catch files that have gone quiet. Fine — quiet compared to what? Your LOS stamps a date on every stage change. Almost nobody sets a threshold against those dates. Without a threshold there is no such thing as a stalled file. There is just a file with a date on it.
That work is yours, and it takes about forty minutes. Sit down with your processing lead and write a day-count beside each stage — the point past which a file sitting there is abnormal for your branch, not for the industry.
Two rules keep it honest. Measure from the last real state change, not from application date: a file eight days into underwriting with three conditions added yesterday is healthy; three days in with nobody touching it is not. And a system note is not activity — if a nightly sync resets your clock, your clock measures the sync.
The second clock is a different animal. Dwell compares to a norm; a countdown compares to the work still left. Lock expiration, contract closing date, appraisal due back, disclosure timing, and on the funded side the EPO window on a recent payoff — none of them care what normal is.
If your stages are not clean enough to time — if "processing" means five different things depending on who moved the file — fix that before you buy anything. Clocks run on pipeline stages and handoffs that everyone defines the same way.
The interrupt budget, and the noise rate nobody scores
The real skill of a good chief of staff is not finding things. It is suppression. They read forty files and hand you four.
Every dead AI pilot dies the same way. Week one it surfaces thirty items a day, all technically true. Week two the team skims. Week three, muted. The model never got worse. The branch just learned to look away.
So give it a budget before you turn it on. Say seven items in the morning brief and no more than three mid-day interrupts. A cap is not a limitation. It is the only thing that forces ranking to exist.
A tool that surfaces thirty things a day has not prioritized anything. It moved the sorting job back onto you and called it intelligence.
Then score it, because nobody else will. Every Friday, drop the week's interrupts into three buckets: acted — I did something; noted — useful, no action; noise — I would not have wanted to know. Say eleven flags land six, three, two. Keep that system. Flip it to two acted and seven noise and you do not have a people problem, you have a threshold problem.
The money, illustratively. Say your branch funds twenty units a month at an average loan amount of $360,000, and two files a month need a lock extension a tighter clock would have caught. At a hypothetical 12.5 basis points, that is roughly $450 a file, $900 a month, about $11,000 a year nobody budgets. That is the small half. The large half is pull-through — the purchase that dies because disclosures went out late.
Client contact is a human tap, by architecture
Inside the branch, autonomy is the feature. An AI that pings a processor about an idle file, or tells you a lock dies Thursday, without asking permission — that is the point. Staff absorb a redundant nudge. It costs a shrug.
Client contact is a different animal, and it is not close.
Compliance first. A message to a borrower about rate, terms, or approval status is a regulated communication. "You're clear to close" sent to a file that is not clear is not a UX bug. It is something your compliance officer now owns.
Then the relationship. The borrower chose a loan officer, not a model. The LO's name is on that message, and the LO answers for it at the closing table and in every referral conversation after.
Then error asymmetry. Internal mistakes are cheap and recoverable. Client-facing mistakes compound — a spooked borrower, an annoyed listing agent, a referral partner who retells the story at their office meeting.
That is why drafting is the design, not a temporary limit waiting on a better model. It is how MAVIS works inside MAVYN: she composes the morning, watches every file for stalls, and everything client-facing lands as a draft for a human to review and send. Full autonomy inside the branch. Zero autonomy toward the client.
Five questions for the demo
- Show me this morning's brief — the one produced with no prompt. If the answer is a chat window, the product is reactive, and reactive means you are still the one watching.
- Live pipeline or an export? Ask how old the underlying data is. An AI reasoning over last night's snapshot will confidently flag a problem you fixed at nine.
- Show me what it suppressed. Anybody can produce a long list. Ask how many exceptions it found this morning and how many it decided not to show you. If that second number is zero, you are buying a firehose.
- Who sees what, and where is that enforced? Branch P&L, margins, and comp should render only for the seats entitled to them, enforced at the database row level. "It just doesn't show financials to an LO seat" is a curtain in the interface, not access control — and a model with unrestricted reads will eventually paraphrase something it shouldn't.
- Can it contact a client on its own, under any setting? The right answer is no by architecture, not no by default. A toggle gets toggled.
None of those are questions about the model. Vendors want to talk about intelligence. Talk about shape: what it writes unprompted, what it reads, what it declines to show you, and what it is structurally incapable of doing.
Monday morning, before the next demo, do the forty-minute version yourself. Write your stages down one column and a day-count beside each. Then list the last three things that slipped and cost you money, and check whether those thresholds would have caught them. You will learn which numbers you believe, and whether those misses were failures of information or of attention. Bring both lists into the room. Ask for the unprompted brief, the suppression count, and the draft-to-send flow on a live client message. If all they can show you is a chat window, you did not find a chief of staff. You found a place to type.