Stop letting your AI guess. Give it a library.#

• ✓ Call any LLM (OpenAI, Groq, Ollama)
• ✓ Snap a LangChain chain together
• ✓ Pass chat history with placeholders
• ✓ Build an agent that picks tools

An LLM is like a smart intern. They finished training a year ago and have never seen your office. RAG gives the intern fresh and private data. It hands them a relevant page exactly when you ask a question.

• 🗓️ Knowledge cutoffThe model's training ended months ago. It doesn't know yesterday's RBI rate cut or last week's Budget numbers. ✓ RAG: fetch today's article
• 🔒 Private / internal dataIt has never seen your HR policy, contracts, product wiki, or customer tickets. You cannot paste them all into one prompt. ✓ RAG: search your docs
• 🌀 HallucinationsWhen unsure, the model invents confident-sounding answers. This is bad in customer support, dangerous in legal, and fatal in healthcare. ✓ RAG: ground in real text
• 🔗 No citationsAsk "where did you get that?" and a raw LLM has nothing. Real products need "see source: page 14". ✓ RAG: return the source chunk

Forget the buzzwords. The whole pipeline is just a smart librarian sitting between your question and the model.

flowchart LR
  Q[❓ Question<br>user types it] --> S[🔎 Search<br>find relevant chunks]:::hl
  S --> F[📋 Stuff<br>add chunks to prompt]:::hl
  F --> L[🧠 LLM<br>answer using them]
  L --> A[💬 Answer<br>+ source citation]:::good
                

Two phases · don't confuse them.

Phase 1 — Indexing (offline, once): read your docs → break into chunks → turn each chunk into numbers (embedding) → store in a vector database. Slow, but you only do it when documents change.

Phase 2 — Querying (online, every question): turn the question into numbers → find nearest chunks in the database → paste them into the prompt → LLM answers. Milliseconds per query.

Before we can \"search by meaning\", we must turn meaning into numbers. This is called an embedding. Once you understand this, RAG becomes much easier to understand.

Definition · keep this in your head. An embedding is a list of numbers (a vector) that captures the meaning of a piece of text. Similar meanings ⟶ similar numbers ⟶ nearby points in space. Different meanings ⟶ far apart.

Forget AI for a second. If you plot people by height vs weight, people with a similar build are close together on the chart. Embeddings use the same idea, just scaled up. Real embeddings have 384, 768 or even 3072 dimensions. This is too many to draw, but the principle is identical: similar ⟶ close, different ⟶ far.

This example shows why engineers trust embeddings. After training on enough text, vectors carry relationships that you can use in arithmetic. Here are four examples:

This is a real word-vector plot reduced from 768 dimensions to 2 so we can draw it. The examples show that the direction between related words stays similar:

When two vectors are close together, the angle between them is small. The cosine of that angle is our score. It ranges from -1 (opposite) to +1 (identical). In practice, above 0.7 means \"very similar\", and below 0.3 means \"barely related\". You do not need to do the math. Just remember: bigger number = more similar.

This example compares pairs of sentences with cosine similarity. It shows the scores for preset pairs and the scoring rules. The scores come from a small built-in table of topics, so common topics look realistic.

You do not have to train anything. Somebody else already did it. Open-source models from Hugging Face (sentence-transformers) or APIs from OpenAI / Cohere give you embeddings in one line:

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")   # ① free, fast, 384 dims

vec = model.encode("A cat is sleeping on the couch")  # ② that's an embedding!
print(vec.shape)         # → (384,)  — 384 numbers
print(vec[:5])          # → [-0.05, 0.12, 0.41, -0.08, 0.22] (something like that)

# to compare two pieces of text → encode both → take cosine similarity
from numpy import dot
from numpy.linalg import norm

v1 = model.encode("A cat is sleeping on the couch")
v2 = model.encode("A kitten is napping on the sofa")
similarity = dot(v1, v2) / (norm(v1) * norm(v2))   # cosine
print(similarity)       # → 0.87 (very similar 🎉)

🔍 Decoder

• ① all-MiniLM-L6-v2 is a tiny but excellent open-source embedding model — runs on your laptop, no API key needed. For production, look at BAAI/bge-large-en-v1.5 (best English) or OpenAI's text-embedding-3-small (paid, very good multilingual).
• ② .encode("text") returns the vector. That's the whole API surface — one call, 384 numbers back. Now your text is searchable by meaning.

• 📍 Vectors = coordinatesEvery chunk of text gets a point in space.
• 📐 Cosine similarityScore from −1 to +1. Bigger = closer.
• 🪄 Math you can doking − man + woman ≈ queen. Really.
• ⚡ One function call.encode(text) — that's it.

Before we embed anything, we must cut it up. You cannot embed a 200-page PDF as one vector. The meaning will turn to mush. Instead, we split it into chunks. How you chunk decides how well your RAG works. Many engineers underestimate this, but the best ones obsess over it.

• 📏 Fixed sizeEvery N characters or tokens — done. Very simple, fast, your default. Downside: chops mid-sentence, mid-table, mid-thought. Best for: prototypes, blog posts
• 🔁 RecursiveTry to split on paragraphs first; if too big, split on sentences; if still too big, on words. A clear fallback ladder. The default in LangChain. Best for: most real apps
• 🧠 SemanticEmbed every sentence; group consecutive sentences that "talk about the same thing". Chunks follow meaning, not size. Best for: long flowing prose
• 📑 Structure-awareUse the document's own structure — markdown headings, code blocks, HTML sections. Each section becomes a chunk. Best for: docs, code, wikis

The same source passage is cut in four ways. The table shows where the cuts land and how chunk shape changes. The differences are the lesson.

Read this sequence: Fixed at size 80 creates mid-word cuts. Recursive at the same size removes those cuts. Semantic creates three chunks that match the three topics. Structure uses the markdown ## headers as ready-made boundaries.

The recall vs precision tradeoff · keep this in mind. Big chunks → high recall (the answer is probably in there), but low precision (lots of fluff around it). Small chunks → high precision (the chunk is exactly the answer), but low recall (the relevant bit might be split across two chunks and you only pulled one). Most teams sweep chunk size as their first RAG tuning knob.

from langchain_text_splitters import RecursiveCharacterTextSplitter

# ① choose chunk size and overlap so nearby context stays connected
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,        # aim for ~800 chars per chunk
    chunk_overlap=100,     # adjacent chunks share 100 chars (context glue)
)

# ② split the long document into chunks ready for embedding
chunks = splitter.split_text(your_long_document)
# ③ print how many chunks will go into the vector search index
print(len(chunks))    # → e.g. 47 chunks ready to embed

You have thousands of embeddings. For every question, you need the top-k nearest ones fast. This is exactly what a vector database does.

• 🟢 Chroma ★ todayOpen-source, runs in-process (no server!), one pip install. Perfect for prototypes & up to ~10M vectors. What we'll use today.
• 🔵 PineconeFully managed SaaS. Zero ops, scales to billions, but paid. The boring-and-reliable choice for production.
• 🟣 QdrantOpen-source and production-grade. Self-host or use their cloud. Great middle ground when you outgrow Chroma.

A vector DB is just these three calls:

1. db.add(documents, embeddings) — store chunks and their vectors. (Indexing.)
2. db.query(query_vector, k=3) — return the 3 chunks whose vectors are closest. (Retrieval.)
3. db.delete(ids) — remove chunks when a doc is deleted. That's it. Three calls.

We've talked about every piece. Now we wire them up. This is the whole pattern — every RAG system you'll see in production is just a fancier version of this.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma

# ① load your documents (any text — for now, hardcoded)
docs = [
    "Our return policy allows refunds within 30 days of purchase.",
    "Shipping is free for orders above ₹999 across India.",
    "For corporate orders above 50 units, contact sales@example.com.",
    "Our office is in Indiranagar, Bangalore. Open Mon-Fri 10am-7pm.",
]

# ② split into chunks (small docs here, but production = thousands of pages)
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.create_documents(docs)

# ③ pick an embedding model (free, runs locally, no API key)
embedder = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

# ④ build the vector store from chunks + embeddings (saves to disk)
db = Chroma.from_documents(chunks, embedder, persist_directory="./chroma_db")

print(f"Indexed {len(chunks)} chunks 🎉")

from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from dotenv import load_dotenv
load_dotenv()

# ⑤ open the same vector store we built in step 1
embedder = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
db       = Chroma(persist_directory="./chroma_db", embedding_function=embedder)
model    = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# ⑥ the prompt: instructions + chunks + question
prompt = ChatPromptTemplate.from_template("""
Answer the question using ONLY the context below. If the context doesn't contain
the answer, say "I don't know." Be concise and quote facts directly.

Context:
{context}

Question: {question}
""")

def rag_answer(question):
    chunks = db.similarity_search(question, k=3)         # ⑦ retrieve top 3
    context = "\n\n".join(c.page_content for c in chunks)
    chain = prompt | model                                # ⑧ same LangChain chain trick
    return chain.invoke({"context": context, "question": question}).content

print(rag_answer("How long do I have to return something?"))
# → "30 days from the purchase date."  ✅ from your data, not a guess

🔍 Decoder — the 8 numbered steps

• ① Your data. In real life this is PDFs, web pages, Notion exports — anything you can turn into text.
• ② Chunk it. Block 17 in action. chunk_size=500 is a sane default.
• ③ Pick an embedder. We're using a free local one — swap to OpenAIEmbeddings() for a paid, slightly better version.
• ④ Chroma.from_documents embeds every chunk and stores both the text and the vector. Done once per dataset.
• ⑤ Reopen the same database. The embeddings are already on disk — no re-encoding.
• ⑥ The important prompt template. Notice "ONLY the context below" — this single line is the most important hallucination reducer in RAG.
• ⑦ similarity_search = embed the question, find the 3 nearest chunks, return them. The librarian.
• ⑧ Same LangChain pipe (prompt | model) you used for LangChain chains. RAG didn't replace your previous knowledge — it slotted right in.

Here is what happens inside rag_answer("How long do I have to return something?"). Each item is one step in the RAG flow:

• 80% of RAG quality lives here

A basic RAG works. A good RAG works well. The two tricks that close that gap are reranking and hybrid search. Both are easy to add.

Embedding similarity is fast, but it sometimes ranks shallow word-matches above deep semantic matches. So we use a two-stage retrieval — a fast first pass to narrow down, then a slow accurate pass to pick the real winners.

• 📋 Bi-encoder · like a resume scanReads the query and the document separately, turns each into a vector, then compares the two vectors with cosine similarity. The encoder never sees them together — it judges each "card" alone.
  ⚡ Fast — sub-10ms over millions of docs
  💾 Pre-computable — encode all your docs once, store forever, only encode the query at runtime
  📉 Shallow — misses subtle relevance because the model never compares them side-by-side
  → use it to grab top 50–100 candidates
• 🎙️ Cross-encoder · like a 30-min interviewFeeds the query and the document into one transformer together, lets the model attend to both at once, then outputs a single relevance score. The model can compare them token by token, weigh trade-offs, spot nuance.
  🎯 Way more accurate — sees query↔doc word interactions directly
  🐢 Slow — full transformer run per (query, doc) pair
  ❌ Not pre-computable — each score is pair-specific, you can't cache anything
  → use it to rerank those 50 → top 3

flowchart LR
  A[1,000,000<br>all your chunks] -->|📋 bi-encoder · ~10 ms| B[50<br>candidates]:::hl
  B -->|🎙️ cross-encoder · ~200 ms| C[3<br>to the LLM]:::good
                

Remember this — the two-stage pattern is universal.

Stage 1 (bi-encoder): grab top 50-100 candidates from the vector DB. Fast.

Stage 2 (cross-encoder): re-score those candidates with a smarter model. Slow per item, but you're only re-scoring 50, not 50 million. Keep the top 3-5 for the LLM.

Same query, same candidates. Left: what cosine similarity returned. Right: what a cross-encoder reranker returned. The relevant answer rises to the top:

The code to add reranking is short:

from sentence_transformers import CrossEncoder

# ① load a cross-encoder that can score a question with one chunk
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")   # free, fast

def retrieve_with_rerank(question, top_k=3):
    # ① retrieve more cheap candidates than you finally need
    candidates = db.similarity_search(question, k=25)        # grab 25 cheap candidates
    # ② pair the same question with each candidate chunk for scoring
    pairs = [(question, c.page_content) for c in candidates]
    # ③ score every question chunk pair with the cross-encoder
    scores = reranker.predict(pairs)                          # cross-encoder scores each pair
    # ④ sort by score and return the best chunks
    return [c for _, c in sorted(zip(scores, candidates), reverse=True)[:top_k]]

Pure semantic search ignores keywords. If you ask \"what's the price of SKU-4429?\", the embedding does not care about the specific code. It just sees \"price\" and \"product\". For codes, names, IDs, and dates, keyword search is better. The fix is not to pick just one. The fix is to use both.

• 🔤 Librarian A · BM25 · the literalistObsessed with exact words. Type "SKU-4429" or "Article 21" and she'll find every single doc containing those exact characters. But ask for "running shoes" and she'll skip the doc titled "marathon footwear" — different words, even though same meaning.
  ✅ Brilliant at product codes, SKUs, model numbers, version strings
  ✅ Brilliant at proper nouns, dates, legal references, technical jargon
  ❌ Blind to synonyms, paraphrase, intent — words are just characters to her
  → catches what vector misses
• 🧠 Librarian B · Vector · the philosopherObsessed with meaning. Ask for "running shoes" and she surfaces "marathon footwear", "jogging trainers", even "sneakers for athletes". But ask for "SKU-4429" and she shrugs — random letters and numbers are just noise to her.
  ✅ Brilliant at natural-language questions, "what's this about?"
  ✅ Brilliant at unclear queries, paraphrase, multilingual matching
  ❌ Blind to exact codes, IDs, technical strings — drowns them in averages
  → catches what BM25 misses

Send the same query to both librarians at the same time. Each ranks the docs using their own logic. Then you merge their two ranked lists into one using a small formula called Reciprocal Rank Fusion (RRF):

Same query, three rankings. Librarian A and Librarian B return different top picks. Hybrid merges the rankings:

Time to ship. Point it at any PDF — your resume, a research paper, an annual report, your college notes — and chat with it. Every answer cites the page it came from. This is the project that gets people asking "how did you build this?"

flowchart LR
  P[📄 PDF in<br>user gives a path] --> C[✂️ Chunk<br>~800 chars]
  C --> E[📍 Embed + store<br>Chroma]:::hl
  E --> R[🔎 Retrieve<br>top-3 chunks]:::hl
  R --> T[💬 Chat<br>terminal loop]:::good
                

# pip install langchain langchain-openai langchain-chroma langchain-huggingface \
#             langchain-community pypdf sentence-transformers python-dotenv

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma

def build_index(pdf_path):
    pages    = PyPDFLoader(pdf_path).load()                    # ① read all pages
    splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
    chunks   = splitter.split_documents(pages)                # ② chunk them
    embedder = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
    db       = Chroma.from_documents(chunks, embedder)         # ③ in-memory store
    return db

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from dotenv import load_dotenv
# ① load the API key and create the PDF assistant model
load_dotenv()

model = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# ② define the grounded answer prompt and source-citation format
prompt = ChatPromptTemplate.from_template("""
You are a helpful PDF assistant. Answer the question using ONLY the context below.
If the context doesn't contain the answer, say "I couldn't find that in the document."
After your answer, list the page numbers you used as: Sources: page X, page Y.

Context:
{context}

Question: {question}
""")

def ask(db, question):
    # ① retrieve the most relevant PDF chunks for the question
    chunks  = db.similarity_search(question, k=4)
    # ② include page numbers beside each chunk before prompting the model
    context = "\n\n".join(
        f"[page {c.metadata['page']+1}] {c.page_content}" for c in chunks)
    # ③ connect the prompt to the model for this answer
    chain = prompt | model
    # ④ ask the model using only the retrieved PDF context
    return chain.invoke({"context": context, "question": question}).content

A tiny main.py indexes the PDF once, then answers questions in a loop:

from pdf_chat import build_index, ask

pdf_path = input("Path to your PDF: ")
db = build_index(pdf_path)                 # ① index once, before the chat loop
print("✅ PDF indexed! Ask me anything about it. (blank line to quit)")

while True:
    question = input("\nYou: ").strip()    # ② every question reuses the same index
    if not question:
        break
    print("Bot:", ask(db, question))

🔍 Decoder · the 3 small choices that matter

• ① build_index runs once, before the loop, and the result is kept in db. Without that, every question would re-index the PDF from scratch (slow!).
• ② Indexing only once means each question is instant. This is the same indexing/querying split from Block 15.
• ③ We pass page from the chunk metadata into the prompt (in ask) — that's how the model knows which page to cite. Metadata is RAG's strength.

Save pdf_chat.py (parts 1 + 2), main.py and your .env in one folder, then run:

$ python main.py

Path to your PDF: annual_report.pdf
✅ PDF indexed! Ask me anything about it. (blank line to quit)

You: What was the revenue this year?

1. Point it at any PDF. Your resume, the Indian Constitution, your company's HR policy, last quarter's earnings call transcript. Anything text-based.
2. Ask 3 questions. One factual ("when was X founded?"), one summary ("what's the main argument?"), one tricky ("compare X and Y"). Notice the citations.
3. Try a question that's NOT in the doc. The answer should be "I couldn't find that in the document." — that's RAG refusing to hallucinate. Watch for this moment. It's the main point.

This walkthrough uses three sample documents that are already indexed. The table shows the documents, example questions, retrieved answers, and the refusal message for questions not in the document:

The recipe is the same; the data makes it interesting. Try one of these on your own:

• 📜 Chat with the ConstitutionIndex the Indian Constitution PDF. Ask "what are the Fundamental Rights?" with citations. data: indiacode.nic.in
• 📊 Chat with an earnings callIndex TCS or Infosys' last quarterly transcript. Ask about margins, guidance, hiring. data: investor relations sites
• 📚 Chat with your textbookIndex a chapter. Ask exam-style questions. Suddenly: a personalised tutor. data: your bookshelf
• 🧾 Chat with company HR policyGenuinely useful at your workplace. "How many leaves do I have left?" "What's the WFH policy?" data: your HR portal

RAG works. But the frontier is making it smarter — and that's where this course leads next.

The RAG you just built always retrieves. But sometimes you don't need to (small talk), and sometimes one retrieval isn't enough (multi-hop questions). Agentic RAG = combine the LangChain agent loop with this RAG library. The agent decides: "do I need to look this up? Is what I found enough? Should I rewrite my query and try again?"

• ✍️ HyDEHypothetical Document Embeddings: have the LLM imagine what the answer would look like, then search for chunks that match the imagined answer. Counter-intuitive, works surprisingly well.
• 🪜 Step-back promptingBefore searching, ask the LLM to generalize the question. "What's the formula for compound interest in this case?" → "What is compound interest?" → broader, better retrieval.
• 🕸️ Graph RAGBuild a knowledge graph from your docs (entities + relationships). Now you can answer "who reports to X" by walking the graph, not just searching.
• 📊 RAG evaluationHow do you measure if your RAG is good? Three metrics: relevance, faithfulness, correctness. We'll wire up a real eval pipeline.

• 📍 EmbeddingsText → vectors. Similar meaning = nearby.
• ✂️ ChunkingThe unglamorous knob that decides quality.
• 🗂️ Vector DBThree calls: add, query, delete.
• 🎯 You shippedA real "Chat with your PDF". 🎉