🔬 Tool Schema Calibration#

Tool Schema Routing Calibration uses a controlled 2×2 factorial experiment to answer a fundamental question in agent engineering: when an LLM chooses between several similar tools, does it rely on the function name or the docstring description?

Given a user query like "What is the current hourly precipitation forecast for Seattle?", the model must choose among three weather tools (current conditions, hourly precipitation forecast, and 30-day historical climate) that all accept the exact same location: str parameter. The service tests all four combinations of naming and description quality at temperature=0 with tool_choice="required":

• Condition A (Descriptive Names + Loose Descriptions): Informative names (get_hourly_precipitation) paired with vague, nearly identical descriptions ("Gets weather information for a city.").
• Condition B (Descriptive Names + Tight Descriptions): Informative names paired with rich descriptions specifying data source, time horizon, and when to use each tool.
• Condition C (Opaque Names + Loose Descriptions): Meaningless names (service_fn_101, service_fn_102, service_fn_103) paired with vague descriptions — leaving the model almost blind.
• Condition D (Opaque Names + Tight Descriptions): Meaningless names (service_fn_102) paired with tight, explicit descriptions — isolating the power of the description alone.

1. Why a 2×2 Factorial Design Is Needed

If you only compare vague vs. detailed descriptions while keeping clear function names like get_hourly_precipitation, the model often scores 100% in both cases because the function name alone gives away the answer! Crossing Name Clarity (Descriptive vs. Opaque) with Description Precision (Loose vs. Tight) separates the two signals and reveals how each one contributes to routing accuracy.

2. Why Condition C & D Matter in Real Enterprise Systems

In enterprise codebases and third-party MCP servers, tool names are often auto-generated from legacy RPC endpoints, internal service codes, or generic verbs (query_v2, exec_proc_102, fetch_data). Condition C shows what happens when those tools also have lazy one-line docstrings; Condition D proves that a well-written description alone can rescue routing accuracy even when function names are completely opaque.

3. Isolating Selection with tool_choice="required"

When schemas are vague (as in Condition C), an unconstrained model might give up and reply in text instead of calling any tool. Setting tool_choice="required" and temperature=0 forces the model to commit to one of the three tools deterministically, cleanly isolating tool selection accuracy from tool invocation willingness.

4. Anatomy of a High-Signal Tool Description

Look at the Tight description for the target tool in CONDITIONS["B"] and CONDITIONS["D"]: it states what data is returned (hourly rain/snow probability and mm volume), the time window (next 24 hours), and negative boundaries (what not to use it for).

1. Core Experiment Architecture (src/python/03_tool_schema_calibration.py)

• 03_tool_schema_calibration.py runs a controlled 2×2 factorial experiment on LLM function-calling routing, crossing Function Name Clarity (Descriptive vs. Opaque) with Description Precision (Loose vs. Tight) to isolate how strongly tool names and docstrings govern tool selection accuracy.

2. Pydantic Data Schemas

• ToolSchemaCalibrationRequest(BaseModel): Defines the input query with default query: str = "What is the current hourly precipitation forecast for Seattle?".
• ConditionResult(BaseModel): Captures the routing outcome for one cell of the 2×2 matrix: condition_code: str ("A", "B", "C", or "D"), condition_label: str, selected_tool: Optional[str], and is_correct_routing: bool.
• ToolSchemaCalibrationResponse(BaseModel): Wraps the overall calibration output containing target_intent: str, condition_results: List[ConditionResult], and analysis_summary: str.

3. The 2×2 Factorial Matrix & Main Function: run_tool_schema_calibration()

• Accepts query: str and returns a dictionary matching ToolSchemaCalibrationResponse:
  • Step ①: Initializes the OpenAI client for the configured model (default "gpt-4o-mini").
  • Step ②: Defines the 4-element conditions list representing the 2×2 matrix where all three candidate tools accept the exact same parameter schema ({"city": {"type": "string"}}):
    • Condition A (Descriptive Names x Loose Descriptions): Pairs descriptive function names (get_current_weather, get_hourly_precipitation, get_historical_climate) with terse, minimal descriptions; correct_target is "get_hourly_precipitation".
    • Condition B (Descriptive Names x Tight Descriptions): Pairs descriptive function names with detailed docstrings specifying exact weather metrics and time horizons; correct_target is "get_hourly_precipitation".
    • Condition C (Opaque Names x Loose Descriptions): Replaces function names with opaque identifiers (service_fn_101, service_fn_102, service_fn_103) paired with terse descriptions; correct_target is "service_fn_102".
    • Condition D (Opaque Names x Tight Descriptions): Pairs opaque function names (service_fn_101..103) with detailed, explicit docstrings to isolate the routing signal of descriptions alone; correct_target is "service_fn_102".
  • Step ③: Loops over each condition in conditions, builds the 3 OpenAI function-calling schemas, and calls client.chat.completions.create() with tool_choice="required" so the model must commit to one tool. It reads chosen = msg.tool_calls[0].function.name, evaluates is_correct = (chosen == c["correct_target"]), and appends the ConditionResult dictionary to results.
  • Step ④: Tallies correct_count = sum(1 for r in results if r["is_correct_routing"]), formats analysis_summary ("Routed accurately in X of 4 calibration conditions."), and returns the complete calibration dictionary.

4. End-to-End AI Calibration Flow

1. run_tool_schema_calibration() takes the user's weather query (targeting hourly precipitation forecast) and iterates through the four schema conditions (A, B, C, D).
2. For each condition, it presents three competing weather tools with identical argument signatures and forces a tool invocation via tool_choice="required".
3. It extracts the selected function name from msg.tool_calls[0].function.name, grades it against the condition's ground-truth target tool (get_hourly_precipitation or service_fn_102), and summarizes how naming clarity and docstring precision affect routing reliability.

Tool Schema Routing Calibration Service.

"""Tool Schema Routing Calibration Service.

Measures which part of a tool schema an LLM actually routes on:
1. 2x2 Matrix of Conditions:
   - Condition A: Descriptive Name + Loose Description
   - Condition B: Descriptive Name + Tight Description
   - Condition C: Opaque Name (`tool_alpha`, `tool_beta`) + Loose Description
   - Condition D: Opaque Name (`tool_alpha`, `tool_beta`) + Tight Description
2. Evaluates whether the model routes by function name keywords or deep semantic docstrings.
"""

import os
from typing import List, Optional
from dotenv import find_dotenv, load_dotenv
from openai import OpenAI
from pydantic import BaseModel

load_dotenv(find_dotenv())


class ToolSchemaCalibrationRequest(BaseModel):
    """Input payload for Tool Schema Calibration."""

    query: str = "What is the current hourly precipitation forecast for Seattle?"


class ConditionResult(BaseModel):
    condition_code: str
    condition_label: str
    selected_tool: Optional[str]
    is_correct_routing: bool


class ToolSchemaCalibrationResponse(BaseModel):
    """Output payload from Tool Schema Calibration."""

    target_intent: str
    condition_results: List[ConditionResult]
    analysis_summary: str


def run_tool_schema_calibration(
    query: str = "What is the current hourly precipitation forecast for Seattle?",
) -> dict:
    """Evaluate tool routing across 2x2 naming/description conditions."""
    # ① Initialize OpenAI client
    client = OpenAI(
        api_key=os.getenv("OPENAI_API_KEY"),
        base_url=os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1"),
    )
    model = os.getenv("OPENAI_MODEL", "gpt-4o-mini")

    # ② Define the 2x2 calibration matrix (Descriptive vs Opaque names x Loose vs Tight docstrings)
    conditions = [
        {
            "code": "A",
            "label": "Descriptive Names x Loose Descriptions",
            "tools": [
                {"name": "get_current_weather", "desc": "Get weather info for a city."},
                {"name": "get_hourly_precipitation", "desc": "Check rain forecast."},
                {"name": "get_historical_climate", "desc": "Get climate records."},
            ],
            "correct_target": "get_hourly_precipitation",
        },
        {
            "code": "B",
            "label": "Descriptive Names x Tight Descriptions",
            "tools": [
                {"name": "get_current_weather", "desc": "Get immediate snapshot of current ambient temperature and atmospheric pressure."},
                {"name": "get_hourly_precipitation", "desc": "Retrieve hour-by-hour expected rainfall probability, accumulation, and precipitation intensity."},
                {"name": "get_historical_climate", "desc": "Retrieve historical multi-decade climatic averages, record highs, and seasonal norms."},
            ],
            "correct_target": "get_hourly_precipitation",
        },
        {
            "code": "C",
            "label": "Opaque Names x Loose Descriptions",
            "tools": [
                {"name": "service_fn_101", "desc": "Get weather info for a city."},
                {"name": "service_fn_102", "desc": "Check rain forecast."},
                {"name": "service_fn_103", "desc": "Get climate records."},
            ],
            "correct_target": "service_fn_102",
        },
        {
            "code": "D",
            "label": "Opaque Names x Tight Descriptions",
            "tools": [
                {"name": "service_fn_101", "desc": "Get immediate snapshot of current ambient temperature and atmospheric pressure."},
                {"name": "service_fn_102", "desc": "Retrieve hour-by-hour expected rainfall probability, accumulation, and precipitation intensity."},
                {"name": "service_fn_103", "desc": "Retrieve historical multi-decade climatic averages, record highs, and seasonal norms."},
            ],
            "correct_target": "service_fn_102",
        },
    ]

    results = []

    # ③ Build OpenAI tool schemas for each condition and force a tool choice
    for c in conditions:
        schemas = [
            {
                "type": "function",
                "function": {
                    "name": t["name"],
                    "description": t["desc"],
                    "parameters": {
                        "type": "object",
                        "properties": {"city": {"type": "string"}},
                        "required": ["city"],
                    },
                },
            }
            for t in c["tools"]
        ]

        resp = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "You are a weather routing coordinator. Select the single best tool for the user's request."},
                {"role": "user", "content": query},
            ],
            tools=schemas,
            tool_choice="required",
        )
        msg = resp.choices[0].message
        chosen = msg.tool_calls[0].function.name if msg.tool_calls else None
        is_correct = (chosen == c["correct_target"])

        results.append({
            "condition_code": c["code"],
            "condition_label": c["label"],
            "selected_tool": chosen,
            "is_correct_routing": is_correct,
        })

    # ④ Summarize routing accuracy across the 4 schema conditions
    correct_count = sum(1 for r in results if r["is_correct_routing"])
    summary = f"Routed accurately in {correct_count} of 4 calibration conditions."

    return {
        "target_intent": "Hourly Precipitation Forecast",
        "condition_results": results,
        "analysis_summary": summary,
    }

Flask Blueprint server and route handler (tool_schema_calibration_route) in src/python/app.py.

"""Flask server for 07-ai-reliability."""

import importlib.util
import os
import sys
from pathlib import Path

from dotenv import find_dotenv, load_dotenv
from flask import Blueprint, Flask, jsonify, request
from flask_cors import CORS
from rate_limiter import check_rate_limit

load_dotenv(find_dotenv())

PATH_PREFIX = os.environ.get("PATH_PREFIX", "")
STATIC_DIR = Path(__file__).resolve().parents[1]
app = Flask(__name__, static_folder=str(STATIC_DIR))
CORS(app)

bp = Blueprint("main", __name__)

_CUR_DIR = Path(__file__).resolve().parent


def _load_service(filename: str, alias: str):
    # ① Resolve the numbered service file path and create its import spec
    file_path = _CUR_DIR / filename
    spec = importlib.util.spec_from_file_location(f"services.{alias}", file_path)
    if spec is None or spec.loader is None:
        raise ImportError(f"Cannot load module from {file_path}")
    # ② Register and execute the module so its functions can be called by routes
    mod = importlib.util.module_from_spec(spec)
    sys.modules[f"services.{alias}"] = mod
    spec.loader.exec_module(mod)
    return mod


mod_cot = _load_service("01_cot_vs_direct.py", "cot_vs_direct")
mod_var = _load_service("02_variance_determinism.py", "variance_determinism")
mod_calib = _load_service("03_tool_schema_calibration.py", "tool_schema_calibration")
mod_err = _load_service("04_tool_error_injection.py", "tool_error_injection")


@bp.before_request
def enforce_rate_limit():
    # ① Only rate-limit mutating POST requests
    if request.method == "POST":
        # ② Check whether this client has exceeded 30 requests per hour
        blocked, msg, retry_after = check_rate_limit(
            request, max_requests=30, window_seconds=3600
        )
        if blocked:
            # ③ Return HTTP 429 with Retry-After header when rate-limited
            resp = jsonify({"error": msg})
            resp.status_code = 429
            resp.headers["Retry-After"] = str(retry_after)
            return resp


@bp.route("/")
def index():
    # ① Read index.html and inject the runtime PATH_PREFIX into data-api-base
    with open(os.path.join(app.static_folder, "index.html"), encoding="utf-8") as f:
        html = f.read()
    html = html.replace('data-api-base=""', f'data-api-base="{PATH_PREFIX}"')
    # ② Return the patched HTML page
    return app.response_class(html, mimetype="text/html")


@bp.route("/css/<path:filename>")
def css(filename):
    return app.send_static_file(os.path.join("css", filename))


@bp.route("/js/<path:filename>")
def js(filename):
    return app.send_static_file(os.path.join("js", filename))


@bp.route("/explanation/<path:filename>")
@bp.route("/info/<path:filename>")
def explanation_pages(filename):
    return app.send_static_file(os.path.join("explanation", filename))


@bp.route("/theory/<path:filename>")
def theory_pages(filename):
    """Serve the theory documentation pages from src/theory."""
    return app.send_static_file(os.path.join("theory", filename))


@bp.route("/api/cot-vs-direct", methods=["POST"])
def cot_vs_direct_route():
    # ① Parse problem_title from the JSON request body
    data = request.get_json(force=True, silent=True) or {}
    problem_title = data.get("problem_title") or "Spatial Navigation (Grid)"
    try:
        # ② Run Direct vs Chain-of-Thought benchmark in 01_cot_vs_direct.py
        res = mod_cot.run_cot_vs_direct_benchmark(problem_title)
        res["result"] = f"Direct: {'Correct' if res['direct_result']['is_correct'] else 'Failed'} | CoT: {'Correct' if res['cot_result']['is_correct'] else 'Failed'}"
        return jsonify(res)
    except Exception as exc:
        return jsonify({"error": str(exc)}), 500


@bp.route("/api/variance-determinism", methods=["POST"])
def variance_determinism_route():
    # ① Parse review_text from the JSON request body and reject empty strings
    data = request.get_json(force=True, silent=True) or {}
    review_text = (data.get("review_text") or data.get("message") or "").strip()
    if not review_text:
        return jsonify({"error": "Review text cannot be empty"}), 400
    try:
        # ② Run the 3-strategy JSON determinism benchmark in 02_variance_determinism.py
        res = mod_var.run_variance_determinism_benchmark(review_text)
        res["result"] = f"Tested {len(res.get('strategies', []))} JSON extraction strategies."
        return jsonify(res)
    except Exception as exc:
        return jsonify({"error": str(exc)}), 500


@bp.route("/api/tool-schema-calibration", methods=["POST"])
def tool_schema_calibration_route():
    # ① Parse query from the JSON request body and reject empty strings
    data = request.get_json(force=True, silent=True) or {}
    query = (data.get("query") or data.get("message") or "").strip()
    if not query:
        return jsonify({"error": "Query cannot be empty"}), 400
    try:
        # ② Run the 2x2 tool schema calibration matrix in 03_tool_schema_calibration.py
        res = mod_calib.run_tool_schema_calibration(query)
        res["result"] = res.get("analysis_summary", "Calibration complete.")
        return jsonify(res)
    except Exception as exc:
        return jsonify({"error": str(exc)}), 500


@bp.route("/api/tool-error-injection", methods=["POST"])
def tool_error_injection_route():
    # ① Parse question from the JSON request body and reject empty strings
    data = request.get_json(force=True, silent=True) or {}
    question = (data.get("question") or data.get("message") or "").strip()
    if not question:
        return jsonify({"error": "Question cannot be empty"}), 400
    try:
        # ② Run the upstream 503 error injection benchmark in 04_tool_error_injection.py
        res = mod_err.run_tool_error_injection(question)
        res["result"] = f"Evaluated error handling across {len(res.get('scenarios', []))} prompt configurations."
        return jsonify(res)
    except Exception as exc:
        return jsonify({"error": str(exc)}), 500


app.register_blueprint(bp, url_prefix=PATH_PREFIX)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Review these essential tool-calling and schema engineering lessons from the 2×2 Tool Schema Calibration experiment:

1. Disentangling Name Signal from Description Signal: A 2×2 factorial design crossing Descriptive vs. Opaque Names with Loose vs. Tight Descriptions reveals how much your agent relies on function identifiers versus semantic docstrings.
2. Why Descriptive Names Mask Weak Docstrings (Condition A): When function names are self-explanatory (get_hourly_precipitation), the model often routes accurately even if the description is lazy, hiding schema weaknesses until tool count grows.
3. Real-World Opaque Naming in Enterprise & MCP Tools (Conditions C & D): Legacy RPC wrappers and auto-generated API bindings often expose opaque identifiers like service_fn_102; Condition D proves that tight, explicit descriptions can maintain high routing accuracy even when function names carry zero semantic signal.
4. Holding Parameter Schemas Constant: Giving all three candidate tools the exact same parameter signature (city: string) ensures the model cannot cheat by routing on parameter names, isolating routing purely to function names and descriptions.
5. Forcing Selection Commitment with tool_choice="required": Setting tool_choice="required" prevents the model from abstaining or asking clarifying questions when schemas are ambiguous (as in Condition C), cleanly measuring tool selection accuracy.
6. Writing High-Contrast Tool Descriptions: Effective tool docstrings explicitly specify the data granularity (snapshot vs. hour-by-hour vs. multi-decade averages), time horizon, and returned metrics so neighboring tools in the same domain do not overlap semantically.
7. Schema Calibration as a Pre-Deployment Gate: Before adding new tools to an production agent, benchmark routing accuracy across representative queries to verify that new tool descriptions do not cannibalize or confuse existing tool routes.