Do you actually need a separate vector database?

Some background. At LoanOptions.ai we built an assistant that let brokers ask questions about lender credit policies and get answers drawn from the policy documents themselves. I wrote about what two years of running that system taught me in an earlier post. Under the hood, a system like that depends on vector search: policy documents get converted into embeddings, and when a broker asks a question, the system finds the most relevant passages by searching those vectors. Where the vectors live is an architecture decision, and in the early period, ours lived in a dedicated vector database, separate from the main platform database.

At the time, that looked like the obvious choice. This was the era when every AI reference architecture had a standalone vector database drawn proudly in its own box, and a wave of specialised products had appeared to fill it. Specialised tool for a specialised job. Hard to argue with on a whiteboard.

What the second database actually cost

Production argued with it.

The moment your vectors live in a different system from the documents they describe, you own a synchronisation problem. Every policy update has to land in two places. The pipelines that keep them aligned have to run, be monitored, and be fixed when they fail. And when they drift, they drift silently: the assistant keeps answering, confidently, from embeddings that no longer match the source. In lending, where the whole point of the system was answers a broker could trust, silent drift is about the worst failure mode you can design in.

Then there is everything around the data. Access control lived in one system, vectors in another. Backups were two stories instead of one. Monitoring was two dashboards. Cost was two invoices. And every query that needed to combine meaning with metadata, which in a real business is most of them, had to reach across the seam. A question like “what does this lender’s policy say, but only the current version, for this product type” is trivial when everything lives together and awkward when it does not.

None of these problems is dramatic on its own. Together they are a tax on every change you make, paid by the engineers who could be building something instead.

The consolidation

So we consolidated. The platform ran on MongoDB Atlas, and when vector search became available inside it, we moved the vectors to live beside the documents they described. One query path. One security perimeter, which mattered a great deal for us, because the platform handled sensitive financial data inside a locked-down network posture where nothing touched the public internet. One backup story, one set of access controls, one bill, and one less system to break at two in the morning.

MongoDB later wrote about the platform’s architecture, including this consolidation, and it is worth reading if you are weighing the same decision. But the punchline from the inside was simple: nothing the dedicated vector database did better was worth what the second system cost to keep honest.

The rule underneath it

The general principle travels well beyond vector databases.

Specialised infrastructure has to earn its keep, and the bar is higher than “better at its one job.” The real question is whether it is better enough to justify a second system of record, with everything a second system drags in: synchronisation, drift, duplicated security, duplicated operations, and one more thing your team has to be good at. For most businesses, most of the time, the answer is no. The boring option, the capability inside the platform you already run and already trust, wins on the arithmetic even when it loses the benchmark.

There are exceptions. At extreme scale, or with latency requirements most businesses will never have, a dedicated store can earn its place. If you are genuinely there, you will know, because the numbers will tell you. If you are reaching for it because the architecture diagram in a vendor’s pitch deck had a box for it, you are about to pay rent on a box.

What this means if you are the one signing off

If you are a founder or CEO looking at a proposed AI architecture, here is the practical version. Every box on that diagram is a bill, a failure mode, and a thing someone has to operate. So for each one, ask what it would cost not to have it. If the answer is a benchmark number rather than a business consequence, the box is decoration.

I wrote in the RAG retrospective that fewer moving parts won every time the question was contested. This was the biggest single instance of that, and the platform was better for it in every way a business actually measures: fewer failures, less engineering time spent on plumbing, and a security story that fit on one page.

If someone is pitching you an AI build right now and the diagram is beautiful, that is usually the moment to have someone who has operated one look at it.