GenAI Course · Module 9 · Session 38

Production Challenges

⏱ 13 min read🎯 Advanced🖼 2 diagrams🧪 2 live demos🐍 Python code

A RAG demo takes a weekend. A RAG system that thousands of employees trust every day takes much more. In this final RAG session, let us tackle the five problems every team meets in production, with practical fixes.

In this session

Indexing and incremental updates (with runnable code)
Latency budgeting and caching
Hallucinations and mandatory citations
Cost optimisation, with a calculator
Privacy and security: access control, PII handling, audit logging

01Production architecture

A production RAG has guardrails around every step of the simple demo pipeline.

02Indexing and incremental updates

Documents change every day. Re-embedding everything nightly is slow and costly. Instead, fingerprint each document and only process what changed:

incremental_index.py

import hashlib

def fingerprint(text):
    return hashlib.sha256(text.encode()).hexdigest()[:12]

index = {"fd_policy.pdf": fingerprint("FD penalty is 1%"),          # what is already indexed
         "kyc_rules.pdf": fingerprint("Aadhaar or passport"),
         "old_offer.pdf": fingerprint("Festive offer 2025")}

today = {"fd_policy.pdf": "FD penalty is 1%",                       # files in the source today
         "kyc_rules.pdf": "Aadhaar, passport or voter ID",
         "card_faq.pdf":  "Block your card from the app"}

to_add    = [f for f in today if f not in index]
to_update = [f for f in today if f in index and fingerprint(today[f]) != index[f]]
to_delete = [f for f in index if f not in today]
print("add:", to_add, "| update:", to_update, "| delete:", to_delete, "| skip:", len(today) - len(to_add) - len(to_update))

add: ['card_faq.pdf'] | update: ['kyc_rules.pdf'] | delete: ['old_offer.pdf'] | skip: 1

Add new files, re-embed changed ones, delete chunks of removed files (very important: old policies must not keep answering!).
LangChain’s indexing API (RecordManager + index()) does this bookkeeping for you.
Keep an effective_date in metadata so you can prefer the latest version.

03Latency budgeting and caching

Stage	Typical p95	How to speed up
Query rewrite / routing	200–400 ms	small fast model, or skip for simple queries
Retrieval (hybrid)	50–150 ms	HNSW index, filters, keep the DB close to the app
Re-ranking	100–300 ms	fewer candidates, smaller re-ranker
LLM first token	400–1,000 ms	smaller model, shorter prompt, streaming
LLM full answer	1–4 s	stream tokens so users read while it writes

Cache answers to frequent questions. Clear the cache when the source documents change.

04Hallucinations and mandatory citations

Instruct the model to answer only from the context and to say “I don’t know” otherwise.
Make citations mandatory: every claim must point to a chunk id or page.
Automatically verify citations: does the cited chunk exist and contain the claim? If not, regenerate or refuse.
If retrieval finds nothing relevant, refuse politely and suggest who to contact.

Demo · Same question, three outcomes

05Cost optimisation

Most of the cost is LLM tokens: the prompt (question + retrieved chunks) and the answer. Move the slider to see how monthly cost scales (illustrative prices):

Demo · Monthly LLM cost

Send fewer, better chunks (re-ranking and compression pay for themselves).
Route easy questions to a small model; keep the large model for hard ones.
Cache frequent questions.
Batch embedding jobs and only re-embed changed documents.

06Privacy and security

Risk	Control
Employee sees documents they shouldn’t	filter chunks by the user’s access level at retrieval time (never rely on the prompt)
Personal data (PAN, Aadhaar, phone numbers) in chunks	detect and mask PII before indexing; keep raw files in a restricted store
Hidden instructions inside documents (indirect prompt injection)	treat retrieved text as data; sanitise; restrict tools after reading untrusted docs
No way to investigate incidents	audit log: user, question, retrieved chunk ids, answer, time
Data sent to external APIs	check data residency and contracts; consider self-hosted models for sensitive data

Go deeper: the OWASP LLM Top 10 lesson covers LLM02 (sensitive information disclosure) and LLM08 (vector and embedding weaknesses) with animated attacks and fixes.

07Summary

Use incremental indexing: add, update and delete only what changed.
Budget latency per stage; stream answers and cache frequent questions.
Make citations mandatory and verify them; refuse when nothing is found.
Cut cost with fewer chunks, smaller models and caching.
Enforce access control at retrieval, mask PII, and keep audit logs.

08Quick quiz

1. A policy PDF was deleted from the source. What must the index do?

2. Where should access control be enforced?

3. Best way to make long answers feel fast?

4. The model cites page 9, but page 9 doesn’t mention the claim. You should…

5. Biggest cost driver in RAG is usually…

09References

Created by Ajeetkumar