RAG Chatbot: Grounding a Website Assistant in Its Own Content

The Problem: A Bot That Was Confidently Out of Date

The old assistant worked the way most website chatbots do. A paragraph-long summary of the business sat in its instructions, and it answered from that. If it needed something specific, it could fetch a page and read it.

Two things were wrong with that, and only one of them is obvious. The obvious one: the summary was maintained by hand, so it drifted. On the day I replaced it, it was still offering services I had stopped selling, at prices that no longer existed. Nobody caught it because nothing was checking.

The subtler problem is what it did when it didn't know. Asked what an MVP costs with nothing to go on, a language model will not say "I don't know" — it will produce a confident, fluent, industry-average number. I tested exactly this: ungrounded, the models answered "$10,000 to $50,000" and "$15k–$100k." Both plausible. Both wrong about my business. A chatbot that invents a price in front of a prospect is worse than no chatbot.

What RAG Actually Is

Retrieval-Augmented Generation is simpler than the name suggests. Instead of hoping the model already knows your business, you look up the relevant passages from your own content first, hand them to the model, and tell it to answer only from those.

The lookup doesn't work by keyword. Every passage on the site is converted into a list of numbers — an embedding — that represents its meaning, so that passages about the same thing end up near each other. A visitor's question gets the same treatment, and the closest passages win. That's why the bot can answer "who keeps the code when we're done?" from a page that says "what do I own at the end?" — no shared words, same meaning.

Building It: Where the Quality Actually Came From

My first working version retrieved the right page 62% of the time. The finished one gets 83%. None of that gain came from a better model or a fancier search algorithm — it came from how the content was cut up.

The single biggest fix: the FAQ on my homepage was being stored as one block containing all eight questions and answers. An embedding is one point in meaning-space, so that block's "meaning" was the average of eight unrelated topics — ownership, timelines, pricing, technical background. It sat near none of them and lost every question it should have won. Splitting it into one passage per question moved accuracy from 71% to 83% on its own.

That failure is worth dwelling on because it is invisible. Nothing errors. No log line appears. The system just quietly returns the wrong passage and the model writes a fluent answer on top of it. You only find it if you are measuring.

Deciding What Not to Index

Not all content deserves equal weight. Service pages, pricing, and case studies are indexed in full, split at their section headings. Blog posts get a single entry each — title, description, link — so the bot knows a post exists and can point you to it, without two thousand words of an old article competing against the pricing page for one of five slots.

There is also a confidentiality decision built into the architecture rather than the instructions. The indexer reads only the published website. It has no path to client files or internal documents, so it cannot leak them — not because it is told not to, but because they are never within reach. A rule in a prompt holds until someone rephrases the question; this holds because there is nothing to reach.

The honest cost of that choice: gaps in the site become gaps in the bot. I treat that as a feature. It cannot invent a price that isn't published, and the questions it fails to answer are a reading list for what the site is missing.

The Hard Part: Teaching It to Say "I Don't Know"

My original design was going to decide this with a number. Every retrieved passage comes with a similarity score, so the plan was: below some threshold, decline to answer. Clean, cheap, and — the measurements showed — impossible.

Two questions from the test set. "Will you disappear after launch?" scores 0.370, and the site does answer it. "Who are your clients?" scores 0.501, and the site does not. The question that must be refused scores higher than the one that must be answered. No threshold separates them.

The reason is worth understanding, because it generalizes. Similarity measures how topically close a passage is, not whether it contains an answer. "Who are your clients" pulls up passages that are intensely about clients — client confidentiality, client poaching — which are topically excellent and completely useless as an answer. Closeness and usefulness are different things that correlate just often enough to fool you.

So the judgment moved to where the actual information is: the model reads the retrieved passages and decides. The instruction that makes it work is one sentence — that saying "the site doesn't cover this" is a correct answer, not a failure. Models are trained to be helpful, and without explicit permission to decline, they reach for something plausible.

The live bot: a grounded answer with its source, and — the harder behaviour — an honest "the site doesn't cover that."

How I Measured It

Before writing any retrieval code, I wrote 30 questions in the words a visitor would actually use, and recorded which page should answer each one. Six of them have no answer on the site at all; for those, the correct behaviour is to decline.

That file is the answer key, and writing it first is the step most teams skip. Without it, you can only tune by vibes — every change feels like an improvement because you go looking for the examples where it is. Two metrics come out of it: how often the right page appears in the top five, and how high up it lands.

The evaluator itself taught me the sharpest lesson. I built hybrid search — combining meaning-based retrieval with old-fashioned keyword matching — and the measurements came back byte-identical to the version without it. That's not a plausible result. The cause: my evaluation script contained its own copy of the search function, so the feature I was testing never ran. The harness was confidently measuring a system nobody was running.

Once it was pointed at the real code, hybrid search turned out to make ranking slightly worse. So it ships turned off. Building something standard, measuring it honestly, and then not shipping it is a better outcome than adding it because everyone else does.

The Part Most RAG Projects Get Wrong

The instinct, when a retrieval system gives bad answers, is to reach for a bigger model or a more sophisticated search algorithm. On this build, neither would have helped. The wins came from splitting content at the boundaries of what a reader would call one answer, from removing duplicate passages that my own page templates were generating, and from deciding which content deserved to be indexed at all.

And I only knew any of that because there was an answer key. The measurement is not paperwork you do after the interesting part — it is what tells you which changes were real. At one point a change I was confident in made the headline number go down, and the right response was to check whether the ruler still measured what I meant, not to quietly re-score until the number agreed with me.

The same evaluation framework I designed for a client's AI governance platform — the SORTIE taxonomy — is what I used to decide what to measure here. It applied unchanged to my own system, which is the strongest test a framework gets.

If You're Building Something Like This

A grounded assistant over your own content is a genuinely good first AI product: the scope is contained, the failure modes are visible, and it produces something you can show people. The parts that decide whether it's trustworthy are unglamorous — how you cut up your content, and whether you built a way to tell if it's working.

If you're weighing an AI feature and want to know what it would actually take, the fastest way to find out is a free working demo — a real sample of the hard part, built on your own material.