The demo always works. Someone loads forty documents into a vector store, asks three questions, gets three good answers, and the meeting ends with a decision to roll it out.
Then it meets ten thousand documents, a compliance team, and a user who phrases things differently from everyone in the demo. Here are the five questions worth answering first — in the order they will hurt you.
1. What is the actual task?
"Let staff ask questions about our documents" is not a task. It is a category. A task has a user, a trigger, an expected output and a definition of done.
Compare:
- Category: a chatbot over our policy documents
- Task: when a support agent is on a call, return the clause that governs a specific refund scenario, with a link to the source page, in under five seconds
The second is buildable, testable and has an obvious failure mode. The first will be declared a success by whoever proposed it and quietly abandoned in four months.
2. How will you know when it is wrong?
This is the question that separates a system from a demo, and almost nobody has an answer on day one.
You need an evaluation set: 50 to 200 real questions with known-correct answers, written by people who do the job. Not generated, not idealised — actual questions, including the badly worded ones. That set becomes your regression suite. Every prompt change, model upgrade and chunking tweak gets measured against it.
Without it you are not engineering; you are redecorating, and you have no way to tell whether last week's change made things better.
3. Is the retrieval good enough to be worth generating from?
Most bad answers are not generation failures. They are retrieval failures wearing a confident voice. If the right passage never reaches the model, no amount of prompt engineering saves the answer.
Measure retrieval on its own, before you look at a single generated sentence: for your evaluation questions, is the correct source document in the top k results? If that number is 60%, your ceiling is 60% and the model is only deciding how eloquently it fails.
Practical things that move retrieval more than model choice: chunking that respects document structure instead of character counts, hybrid keyword-plus-vector search, and metadata filters so a query about the 2026 policy cannot return the 2019 one.
4. Who is allowed to see what?
This is where a proof of concept becomes a liability. If your document set contains anything role-restricted — salaries, client contracts, medical or financial records — then permissions have to be enforced at retrieval time, per user, not by asking the model politely in a system prompt.
A prompt instruction is not an access control. It is a suggestion to a system that can be argued with.
5. What happens at the edges?
Three behaviours to specify before launch, not after the first incident:
- Refusal. What does the system do when the answer is not in the corpus? "I don't have that" is a feature. A plausible invention is the single fastest way to lose user trust permanently.
- Escalation. Where does a user go when the answer is wrong or missing? There should be a visible path to a human.
- Attribution. Every answer should cite its source, and the citation should be clickable. Users verify the first ten answers, then stop — but only if the first ten checked out.
The cost question, briefly
Model the token cost at realistic volume before you commit to an architecture, not after. Re-embedding a large corpus on every document change, retrieving twenty chunks when five would do, or running the largest available model on a classification task that a small one handles are the three most common ways a system that worked in the pilot becomes indefensible at scale.
None of this argues against putting a model in front of your data. It argues for treating it as production software — with an evaluation set, access control, a defined failure behaviour and a cost model — rather than as a demo that escaped.