Evidence
Does Qloo actually make the plan better?
The hackathon asks: if it would work the same without Qloo, it's the wrong thing. So we run every sample event twice, once through the full agent with Qloo and once through the same model with the same brief and quality bar but no tools, and measure the difference. Measured 2026-10-23 on 8 events.
Items backed by Qloo taste data
61%
With Qloo
0%
LLM only
Items tied to any data source
91%
With Qloo
0%
LLM only
Specific items per plan (avg)
32
With Qloo
12
LLM only
Claims citing a Qloo receipt
183
With Qloo
0
LLM only
Event by event
| Event | Qloo-backed | Data-backed | Generic phrases | Qloo calls | Time |
|---|---|---|---|---|---|
Phoebe Bridgers The Fillmore, San Francisco | 77% vs 0% | 100% vs 0% | 0 vs 0 | 8 | 1.5 min · gemini-3.5-flash-lite |
Hasan Minhaj Chicago Theatre, Chicago | 38% vs 0% | 100% vs 0% | 0 vs 0 | 6 | 2.8 min · gemini-3.5-flash-lite |
Indie college fest with Prateek Kuhad COEP Ground, Pune | 83% vs 0% | 97% vs 0% | 0 vs 1 | 7 | 1.4 min · gemini-3.5-flash-lite |
Khruangbin Red Rocks Amphitheatre, Morrison, Colorado | 52% vs 0% | 91% vs 0% | 1 vs 0 | 6 | 2.5 min · gemini-3.5-flash-lite |
Fred again.. O2 Academy Brixton, London | 49% vs 0% | 65% vs 0% | 0 vs 0 | 8 | 2.8 min · gemini-3.5-flash-lite |
Romesh Ranganathan Eventim Apollo, Hammersmith, London | 77% vs 0% | 100% vs 0% | 0 vs 0 | 8 | 1.7 min · gemini-3.5-flash-lite |
Anuv Jain NSCI Dome, Worli, Mumbai | 66% vs 0% | 78% vs 0% | 0 vs 0 | 11 | 1.8 min · gemini-3.5-flash-lite |
Zakir Khan Chowdiah Memorial Hall, Bengaluru | 45% vs 0% | 93% vs 0% | 0 vs 0 | 6 | 1.4 min · gemini-3.5-flash-lite |
With Qloo first, LLM only after "vs". "Data-backed" counts items tied to Qloo, web search, maps, weather or the staffing formula rather than the model's own say-so.
How honesty is enforced
- Source check in code. A plan is rejected if it labels a vendor, sponsor or artist "Qloo" that no Qloo call returned, or uses a source whose tool failed in that run.
- Receipts. Every Qloo call gets an id (Q1, Q2…) and Qloo-backed claims cite it, so anyone can check a claim against the exact request.
- Code computes, the model explains. Staffing headcounts, safety checks and promo hot spots are calculated, not generated.
- Audience patterns, not predictions. Plans say "fans over-index on X", never "attendees will buy X".
What these numbers don't show
- A small sample (8 events), on free-tier Gemini models that change hour to hour.
- "Qloo-backed" measures where a claim came from, not whether an organiser would act on it. We haven't yet measured outcomes like sales or queue times.
- The generic-phrase list is ours. The LLM-only plan may name real vendors, but nothing checks them.
- Qloo's coverage varies by act and city; comedy and smaller acts return less.
Reproduce it: npx tsx scripts/eval-agent.ts --baseline --summary src/lib/evidence.json. See a full plan with receipts on the sample page.