Building Production LLM Applications: RAG, Evaluation and Observability That Actually Work

Almost every team I speak with has built an LLM prototype. Very few are confident about running one in production. The gap is not the model; it is everything around it. Users ask questions the prototype never saw, documents change, latency spikes, costs creep up and, worst of all, the system produces confident wrong answers that nobody notices until a customer complains.

Production LLM engineering is ordinary backend engineering with a probabilistic component in the middle. The principles I rely on when building Django services, such as clear contracts, tests, monitoring and graceful degradation, all still apply. They just need to be adapted for a component whose output varies from one call to the next.

This article covers the three pillars: retrieval-augmented generation (RAG) to ground answers in your data, evaluation to know whether changes help, and observability to see what happens in production.

The idea behind RAG, introduced in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, is to retrieve relevant documents at query time and include them in the model's context, rather than relying on knowledge baked into its weights. That gives you fresher answers, the ability to cite sources and control over what information the model can see.

A production RAG pipeline has two halves.

Indexing

1. Ingest and clean: extract text from PDFs, HTML and databases, removing boilerplate like headers and navigation.
2. Chunk: split documents into passages. Structure-aware chunking, by heading or section, usually beats fixed character windows. Keep some overlap and store the parent document reference.
3. Enrich metadata: attach source, date, access permissions, language and document type to every chunk. You will need these for filtering and for authorisation.
4. Embed and store: compute embeddings and store them in a vector index. If you already run PostgreSQL, the pgvector extension is often enough to start.

Query time

1. Rewrite the query if needed, for example to resolve "what about the second one?" using chat history.
2. Retrieve with hybrid search: combine dense vector similarity with keyword search such as BM25. Dense retrieval handles paraphrase; keyword search handles exact identifiers, drug names and error codes.
3. Rerank: a cross-encoder reranker rescores the top candidates for relevance. This is often the single biggest quality improvement.
4. Filter by permission: enforce access control before content reaches the model. Never rely on the prompt to keep secrets.
5. Generate with citations: instruct the model to answer only from provided context, cite chunk IDs and say when the answer is not present.

def answer(question, user):
    candidates = hybrid_search(question, k=40, filters={"acl": user.groups})
    context = reranker.top(question, candidates, n=6)
    prompt = render("answer_v3.txt", question=question, context=context)
    response = llm.generate(prompt, temperature=0.1)
    return validate_citations(response, context)

Ordering matters too. Research on how models use long contexts found that performance can degrade when relevant information sits in the middle of a long prompt. Fewer, better-ranked chunks usually beat stuffing the context window.

Without evaluation, every prompt tweak is a guess. The goal is an evaluation suite you can run in CI, just like unit tests.

Build a golden dataset. Start with a modest set of realistic questions, ideally drawn from real user logs, each with a reference answer and the documents that should support it. Include hard cases: ambiguous questions, questions with no answer in the corpus, multilingual or code-switched queries and adversarial inputs.

Evaluate retrieval and generation separately.

• Retrieval metrics: recall at k (did the right chunk appear in the top results?) and mean reciprocal rank. If retrieval fails, no prompt can rescue the answer.
• Generation metrics: faithfulness (is every claim supported by the retrieved context?), answer relevance and correctness against the reference. Frameworks such as RAGAS propose reference-free metrics for these properties.

Use LLM-as-a-judge carefully. Using a strong model to grade outputs scales far better than human review, and research on judging LLMs with LLMs found strong judges can agree with human preferences at rates comparable to agreement between humans. The same work documents biases, such as favouring longer answers or the first option presented. Mitigate by using clear rubrics, asking for a short justification before the score, randomising order in pairwise comparisons and periodically calibrating the judge against human labels.

JUDGE_RUBRIC = '''Score 1-5 for faithfulness.
5 = every claim is supported by CONTEXT.
1 = main claim contradicts or is absent from CONTEXT.
Return JSON: {"reason": str, "score": int}'''

Gate releases on regressions. Treat prompts, retrieval settings and model versions as code. A pull request that changes any of them should run the suite and show a diff of scores. Block merges that drop faithfulness below an agreed threshold.

Offline evaluation tells you how the system behaves on known questions. Observability tells you what is happening with real users.

• Trace every request end to end: log the query, rewritten query, retrieved chunk IDs with scores, prompt template version, model name, token counts, latency per stage and final response. OpenTelemetry provides a vendor-neutral way to emit traces and spans, and its community has been developing semantic conventions for generative AI calls.
• Track cost per request: token usage multiplied by price, broken down by feature and customer. Costs surprise teams more often than outages do.
• Capture user feedback: thumbs up or down, edits to drafted text and follow-up questions that suggest the first answer missed.
• Run online evaluators: sample a percentage of production traffic and score it with the same faithfulness judge used offline. Alert on drops.
• Watch for drift: new query topics, a rising rate of "I don't know" answers or falling retrieval scores signal that the corpus or user needs have shifted.

Add low-scoring production examples to the golden dataset. That feedback loop is what turns a static prototype into a system that improves over time.

The OWASP GenAI Security Project maintains a Top 10 for LLM applications that every team should read. The risks I emphasise most:

• Prompt injection: retrieved documents and user inputs can contain instructions that hijack the model. Treat all retrieved content as untrusted data, keep system instructions separate, restrict what tools the model can call and require confirmation for consequential actions.
• Data leakage: enforce document-level permissions at retrieval time, and redact personal data before logging. In health tech, I treat prompts and traces as sensitive records subject to the same controls as the underlying patient data.
• Hallucination: faithfulness checks, mandatory citations and a clear "not found" path reduce, but never eliminate, fabricated answers. For high-stakes domains, keep a human in the loop.
• Availability and cost: set timeouts, retries with backoff, per-user rate limits and a fallback model or cached answer when the primary provider is slow.

For teams in Lagos and other emerging tech hubs, budgets and GPU access are often limited. A sensible sequence:

1. Ship a narrow use case over a well-defined corpus, with citations from day one.
2. Build a small golden dataset and wire it into CI before adding features.
3. Add hybrid search and a reranker before reaching for bigger models.
4. Instrument tracing and cost tracking from the first deployment.
5. Cache aggressively; many questions repeat, and cached answers cost nothing.
6. Evaluate smaller or open-weight models against your suite; you may not need the most expensive option.

When I mentor developers through BuildWithMMO, the teams that progress fastest are the ones that treat evaluation as a first-class deliverable rather than an afterthought.

• Production LLM work is backend engineering with a probabilistic component; apply testing, monitoring and graceful degradation.
• Good RAG depends on structure-aware chunking, rich metadata, hybrid search, reranking and permission filtering.
• Evaluate retrieval and generation separately, and gate releases on a golden dataset run in CI.
• LLM-as-a-judge scales evaluation but must be calibrated against human labels and guarded against known biases.
• Trace every request, track cost and feed low-scoring production cases back into your test set.
• Treat retrieved content as untrusted, enforce access control before generation and plan for provider failures.

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv
2. Lost in the Middle: How Language Models Use Long Contexts — arXiv
3. RAGAS: Automated Evaluation of Retrieval Augmented Generation — arXiv
4. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — arXiv
5. OpenTelemetry — Cloud Native Computing Foundation
6. OWASP GenAI Security Project — OWASP

Cover image: original illustration created for this article. Cited works remain the property of their respective authors and publishers.