What does two years of RAG in production actually teach you?

Two years ago, at LoanOptions.ai, we shipped an assistant that let finance brokers ask questions about lender credit policies and get answers drawn straight from the policy documents themselves. AWS wrote it up with us at the time, in a piece I co-authored with two of their solutions architects: Boosting our fintech efficiency using generative AI and Amazon Bedrock.

Launch write-ups are honest about ambition and quiet about what comes after. This is the after. The system ran, brokers used it, and it taught me most of what I now believe about AI in production. Here is what held up and what did not.

The model was never the hard part

Everyone who has not built one of these systems worries about the model. Everyone who has built one worries about retrieval.

Lender policy documents are hostile terrain. The critical detail lives in a table. The exception that changes the answer lives in a footnote. A definition on page four quietly changes the meaning of everything on page forty. How you split those documents up, index them, and decide what to hand the model for a given question determines the quality of the answer far more than which model you picked. We could swap models and move the needle a little. Fixing retrieval moved it a lot.

If a vendor demos an AI product to you and every question they ask it comes back perfect, ask what the document set looked like. Clean, curated documents make any system look brilliant. Real documents are where the work is.

Freshness beats cleverness

Lenders change policy whenever they like, without ceremony, and sometimes without much announcement. An assistant answering from last quarter’s document sounds exactly as confident as one answering from today’s. That is the dangerous part. Confidence and correctness are completely unrelated in these systems.

So the real product was never the chat window. The real product was the unglamorous pipeline behind it that kept the document set current, so that when a broker asked a question, the answer came from the policy as it stood that morning. Nobody demos the pipeline. The pipeline is the thing.

A wrong answer in finance is not a bug ticket

In most software, a wrong answer is an annoyance. In lending, a wrong answer is a deal workshopped on bad information, and that has consequences for a real client with a real application.

The design answer was to make the system checkable rather than trying to make it infallible. Answers pointed back to the source, so a broker could open the policy and verify before acting on it. The assistant drafted, the human decided. Trust did not come from the system sounding sure of itself. It came from the fact that anyone could look behind the answer in one click.

I hold every AI proposal I review to that standard now. If a system’s answers cannot be traced back to something a human can verify, it does not belong anywhere near a regulated workflow.

The architecture did not survive contact unchanged

In the early period, the vector store that powered retrieval was its own piece of infrastructure, separate from the main database. Separate meant synchronisation jobs. Synchronisation meant drift, overhead, and one more thing to break at 2am. The platform later consolidated vector search into the core data platform, which removed a whole category of moving parts. MongoDB wrote about that architecture recently, and it is worth reading if you are weighing up a separate vector database against one built into your existing stack.

The lesson generalises. The first architecture of an AI system is a hypothesis. Expect to revise it once production teaches you which parts earn their complexity and which do not. Fewer moving parts won, every time it was contested.

Demos are priced like demos

The demo costs almost nothing to run. Production sends an invoice. Every question triggers retrieval. Every policy update means re-processing documents. Context is not free, and the habit of stuffing more of it into every request compounds quietly across thousands of queries.

None of this is a reason not to build. It is a reason to design for the bill on day one, and to treat any AI cost estimate that only covers the demo as fiction. When I model these systems for clients now, the running cost gets designed alongside the architecture, not discovered after it.

Tools get used where people already work

The last lesson is the least technical and it decided everything. Brokers used the assistant because it sat inside the flow of work they were already doing, at the moment they had a policy question, on a deal they were actually working. A tool that lives outside the workflow gets opened twice, admired once, and abandoned.

Adoption was never a training problem or an enthusiasm problem. It was a placement problem.

The distance between the two write-ups

The launch article and this one describe the same system. The distance between them is where the actual work happened, and it is the distance most AI projects never cross. Prototypes are cheap now. Anyone can build a demo in a weekend, and plenty of vendors have. Production systems that a regulated business can rely on every day are a different discipline entirely.

If someone is pitching you an AI system, the most useful question you can ask is what their two-year retrospective would say. If they cannot imagine it, that is worth knowing before you sign.