📄 PDF Chat

Concept

PDF Chat is a complete end-to-end RAG pipeline applied to pasted PDF text. First, the text is chunked and embedded into an in-memory Chroma vector store (/pdf-index). Then, for each question, the most relevant chunks are retrieved and injected into a prompt that tells GPT-4o mini to answer only from the document.

The in-memory store persists for the browser session on the server (_pdf_state). This keeps the server from storing uploaded files while still showing the full RAG pipeline on real document text.

Theory & Concepts

End-to-End Document Intelligence

The PDF Chat demo combines chunking, vector storage, and grounded generation into a cohesive pipeline for querying real documents:

• Optimized Chunk Sizing: When you paste PDF text, build_pdf_text_index() splits it into 800-character chunks—larger than a standard text demo. This is intentional: PDF paragraphs often contain dense, continuous context, so a larger chunk size ensures that related concepts aren't prematurely cut off before being stored in the in-memory Chroma database.
• Ephemeral Vector Storage: For an interactive web demo, spinning up a persistent database is overkill. This demo leverages an in-memory Chroma instance (_pdf_state). The vectors exist only as long as the server session, allowing you to ask multiple questions (using ask_pdf() to retrieve the top 4 chunks) without permanently saving your private PDF data to disk.
• Reliability via Prompting: The PDF_PROMPT acts as a strict guardrail. By telling GPT-4o mini to explicitly state "I couldn't find that in the document" if the answer isn't in the context, we prioritize factual accuracy and trust over the model's natural inclination to invent plausible-sounding answers.

Request flow

Code flow

Backend

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate

from config import get_chat_model, get_embedder

# Strict RAG system prompt that forces the LLM to ground answers exclusively in the PDF text
PDF_PROMPT = ChatPromptTemplate.from_template("""
You are a helpful PDF assistant. Answer the question using ONLY the context below.
If the context doesn't contain the answer, say "I couldn't find that in the document."

Context:
{context}

Question: {question}
""")


def build_pdf_text_index(pdf_text: str) -> Chroma:
    """Chunk pasted PDF text and build an in-memory vector store.

    Args:
        pdf_text: Text copied from a PDF.

    Returns:
        A ``Chroma`` vector store containing the embedded text chunks.
    """
    # Step 1: Split PDF text with larger chunk size (800 chars / 100 overlap) to preserve multi-sentence paragraphs
    splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
    chunks = splitter.create_documents([pdf_text])

    # Step 2: Initialize embedding model
    embedder = get_embedder()

    # Step 3: Create temporary in-memory Chroma vector store (persist_directory=None)
    db = Chroma.from_documents(chunks, embedder)  # index is rebuilt each time new PDF text is submitted
    return db


def ask_pdf(db: Chroma, question: str) -> str:
    """Answer a question using the content of an indexed PDF.

    Args:
        db: A Chroma vector store built by ``build_pdf_text_index``.
        question: The user's question about the PDF.

    Returns:
        The model's answer.
    """
    # Step 1: Initialize LLM chat model
    model = get_chat_model()

    # Step 2: Retrieve top 4 most relevant text chunks from the vector database
    chunks = db.similarity_search(question, k=4)

    # Step 3: Assemble retrieved chunks into unified context block
    context = "\n\n".join(c.page_content for c in chunks)

    # Step 4: Run LCEL prompt-to-model chain and extract textual response
    chain = PDF_PROMPT | model
    return chain.invoke({"context": context, "question": question}).content

API routes

@bp.route("/pdf-index", methods=["POST"])
def pdf_index():
    """Index pasted PDF text in an in-memory vector store.

    Request body (JSON): ``{ "pdf_text": "<text copied from a PDF>" }``
    Response (JSON): ``{ "result": "PDF text indexed. Ask me anything about it." }``
    Error response:  ``{ "error": "<message>" }`` with HTTP 400 or 500
    """
    # Step 1: Parse request JSON
    data = request.get_json(force=True)
    pdf_text = (data.get("pdf_text") or "").strip()

    # Step 2: Validate that non-empty text was provided
    validation = validate_textarea(pdf_text, "PDF text")
    if validation:
        return validation

    # Step 3: Build Chroma vector index and cache in session state dictionary
    try:
        _pdf_state["db"] = build_pdf_text_index(pdf_text)
        return jsonify({"result": "PDF text indexed. Ask me anything about it."})
    except Exception as e:
        return jsonify({"error": str(e)}), 500


@bp.route("/pdf-chat", methods=["POST"])
def pdf_chat():
    """Answer a question about the previously indexed PDF text.

    Request body (JSON): ``{ "question": "<text>" }``
    Response (JSON):     ``{ "result": "<answer>" }``
    Error response:      ``{ "error": "<message>" }`` with HTTP 400 or 500
    """
    # Step 1: Parse request JSON and extract user query
    data = request.get_json(force=True)
    question = (data.get("question") or "").strip()

    # Step 2: Validate question and verify that a PDF index has been uploaded
    if not question:
        return jsonify({"error": "A question is required."}), 400
    if _pdf_state["db"] is None:
        return jsonify({"error": "Please add PDF text first."}), 400

    # Step 3: Perform vector search, prompt LLM, and return grounded answer
    try:
        answer = ask_pdf(_pdf_state["db"], question)
        return jsonify({"result": answer})
    except Exception as e:
        return jsonify({"error": str(e)}), 500