AI Reliability

🧠 CoT vs Direct Reasoning ⓘ

Direct final-answer prompting vs Chain-of-Thought step-by-step reasoning at temperature=0.

Compare accuracy and latency on multi-step spatial navigation, family-tree logic, and calendar scheduling puzzles.

Run Logic Benchmark

🎲 Structured Output Determinism ⓘ

Unconstrained Free-Text vs Prompt-Only JSON vs Strict JSON Schema (response_format) extraction.

Extract structured product review fields (sentiment, rating, pros, cons) and compare JSON parser reliability across all 3 modes.

The new smartphone is amazing, the camera quality is top-notch but the battery life is a bit disappointing. I love the sleek design though! Rating: 4 out of 5 stars. Compare Determinism

🎯 Tool Schema Routing Calibration ⓘ

2×2 tool routing ablation matrix: Descriptive vs Opaque tool names × Loose vs Tight docstrings.

Tests whether the LLM routes weather intents (precipitation vs general forecast) by function name keywords or docstring semantics across Conditions A, B, C, and D.

Calibrate Routing

🛡️ Tool Error Injection & Scoring ⓘ

Fault injection benchmark testing LLM honesty vs hallucination when a tool returns HTTP 503.

Compares Normal Execution (Tokyo + Paris) against an injected Paris HTTP 503 outage under Unguided and Strictly Guided system prompts.

Inject Outage & Score