🔎 Website Summarizer

Concept

This demo uses a LangChain prompt | model | parser chain to summarize any web page. First, WebBaseLoader fetches and cleans the page text. Then the chain fills a prompt template with that text and calls GPT-4o mini with low temperature (0.3) for a factual, consistent summary. The | operator is LangChain's way of composing steps — each step's output becomes the next step's input.

Theory & Concepts

The Website Summarizer demo demonstrates how to build robust AI data pipelines using LangChain and its Expression Language (LCEL). Instead of manually concatenating strings and parsing complex API responses, this demo links specialized components together into a single, cohesive Chain where the output of one step flows naturally into the next using the pipe (|) operator.

When you submit a URL to be summarized, the application processes it through a sequence of well-defined stages:

• Document Loading: Before the LLM can summarize a webpage, the raw HTML must be fetched and cleaned. The demo uses LangChain's WebBaseLoader to download the page (using custom headers to avoid basic bot detection) and extract just the readable text, stripping away messy navigation menus and scripts.
• Prompt Templating: The scraped text is then passed to a ChatPromptTemplate. Instead of hardcoding the prompt, this template holds a static instruction ("give a short, friendly summary") and dynamically injects the website's text into a {website} variable.
• Model Execution: The formatted prompt flows into the LLM (GPT-4o-mini). By using a low temperature (0.3), the model is instructed to be highly factual and consistent, ensuring the summary closely reflects the actual webpage content rather than hallucinated details.
• Output Parsing: Finally, the LLM returns a complex object containing token counts and metadata. The StrOutputParser at the end of the chain extracts just the raw markdown summary string, which is then sent back to your browser to be rendered cleanly in the UI.

Request flow

Code flow

Scraper

from langchain_community.document_loaders import WebBaseLoader

# Many servers block requests that don't look like a real browser.
# Sending a realistic User-Agent avoids most simple bot-detection checks.
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
    "AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/120.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}


def fetch_website_contents(url: str) -> str:
    """Download a web page and return its title and cleaned text using LangChain."""
    if not url.startswith(("http://", "https://")):
        url = "https://" + url
    try:
        # WebBaseLoader uses requests and BeautifulSoup under the hood.
        loader = WebBaseLoader(url)
        # Pass headers and timeout via requests_kwargs
        loader.requests_kwargs = {"headers": HEADERS, "timeout": 15}
        docs = loader.load()
        if not docs:
            return "Could not fetch the website."
        title = docs[0].metadata.get("title", "No title found")
        # WebBaseLoader parses text with soup.get_text() by default.
        text = docs[0].page_content.strip()
        return f"Title: {title}\n\nPage contents:\n{text}"
    except Exception as e:
        return f"Could not fetch the website. Error: {e}"

Summarizer

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

from config import get_chat_model
from scraper import fetch_website_contents

# A reusable prompt template with a {website} blank the chain fills in.
# Low temperature = factual, consistent output. Markdown = renders nicely in UI.
PROMPT = ChatPromptTemplate.from_template(
    "You analyze the contents of a website and give a short, friendly "
    "summary. Ignore navigation menus. Respond in markdown.\n\n"
    "Summarize this website:\n\n{website}"
)


def summarize(url: str) -> str:
    """Fetch a web page and return a short markdown summary of it."""
    # Step 1 — get the page text (title + body, scripts/nav stripped).
    website = fetch_website_contents(url)
    # Step 2 — run the chain; the dict fills the {website} blank by name.
    chain = PROMPT | get_chat_model(temperature=0.3) | StrOutputParser()
    return chain.invoke({"website": website})

API route

@bp.route("/summarize", methods=["POST"])
def summarizer():
    """Scrape a URL and return a LangChain-generated markdown summary."""
    data = request.get_json(force=True)
    url = (data.get("url") or "").strip()
    # Validate at the boundary — return 400 immediately rather than letting
    # the scraper make a request with an empty URL.
    if not url:
        return jsonify({"error": "A website URL is required."}), 400
    try:
        text = summarize(url)
        return jsonify({"result": text})
    except Exception as e:
        return jsonify({"error": str(e)}), 500