🔎 Website Summarizer

Concept

This demo fetches a web page, strips out the HTML noise, and asks an LLM to summarize what's left. It's a two-step pipeline kept in separate modules so each step is independently testable and the scraper can be reused by future features.

1. Scrape — scraper.py downloads the page with a browser-like User-Agent and removes <script>, <style>, <nav>, <header>, <footer>, <img>, and <input> tags before extracting plain text.
2. Summarize — summarizer.py passes the cleaned text to GPT-4o mini, which is instructed to respond in markdown and ignore leftover navigation boilerplate.

Theory & Concepts

HTML Sanitization and Prompt Constraints

The Website Summarizer demo demonstrates how to effectively prepare messy real-world data for an LLM:

• Token Efficiency through Scraping: Web pages are filled with HTML tags, scripts, and navigation menus that are irrelevant to the actual content. In the fetch_website_contents() function, we use BeautifulSoup to proactively strip out <script>, <style>, and <nav> tags. This is crucial because passing raw HTML to GPT-4o mini would waste valuable context window tokens and increase API costs, while potentially confusing the model with structural noise.
• Constraining the Output Format: Once the cleaned text is passed to the LLM, the system prompt explicitly commands it to "Respond in markdown." This ensures the summarizer's output can be predictably rendered by our frontend. Furthermore, instructing the model to "Ignore navigation menus" provides a second layer of defense against any stray boilerplate that survived the BeautifulSoup cleanup.

Request flow

Code flow

Scraper

def fetch_website_contents(url: str) -> str:
    # Users often paste bare domains; add a scheme so requests doesn't error.
    if not url.startswith(("http://", "https://")):
        url = "https://" + url

    try:
        # timeout=15 prevents the endpoint from hanging indefinitely on slow
        # or unresponsive servers.
        response = requests.get(url, headers=HEADERS, timeout=15)
        # Raise an HTTPError for 4xx/5xx status codes so we can catch and
        # return a friendly error message below.
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        # Return an error string instead of raising, so it can reach the UI.
        return f"Could not fetch the website. Error: {e}"

    # Parse the raw HTML with BeautifulSoup's built-in html.parser (no extra
    # C libraries required, unlike lxml).
    soup = BeautifulSoup(response.text, "html.parser")
    title = soup.title.string if soup.title else "No title found"

    # Remove tags that add noise but no useful content for an LLM:
    #   script / style        — code and CSS, not human-readable text
    #   nav / header / footer — repeated site chrome that inflates token count
    #   img / input           — non-text elements whose attributes we don't need
    for tag in soup(["script", "style", "nav", "footer", "header", "img", "input"]):
        tag.decompose()

    text = soup.get_text(separator="\n", strip=True)
    return f"Title: {title}\n\nPage contents:\n{text}"

Summarizer

# Tell the model to ignore leftover nav boilerplate and reply in markdown.
SYSTEM_PROMPT = """You analyze the contents of a website and
give a short, friendly summary. Ignore navigation menus.
Respond in markdown."""


def summarize(url: str) -> str:
    # Step 1 — get the page text (title + body, scripts/nav stripped).
    website = fetch_website_contents(url)

    # Step 2 — ask the model to summarize what we scraped.
    client = get_openai_client()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                # We embed the full page text directly into the user message.
                # For very large pages this could exceed the context window;
                # a production version would truncate or chunk the text first.
                "role": "user",
                "content": f"Summarize this website:\n\n{website}",
            },
        ],
    )
    return response.choices[0].message.content

API route

@bp.route("/summarize", methods=["POST"])
def summarizer():
    data = request.get_json(force=True)
    url = (data.get("url") or "").strip()
    # Validate at the boundary instead of scraping an empty URL.
    if not url:
        return jsonify({"error": "A website URL is required."}), 400
    try:
        text = summarize(url)
        return jsonify({"result": text})
    except Exception as e:
        return jsonify({"error": str(e)}), 500