Overview
- Tests how reliably language models pull structured specifications out of real AI model-release announcements, scored against a hand-written answer key.
- Running a model and scoring it are kept apart, so a change to the scoring rules re-scores every past run without paying for new calls.
- Every prompt is versioned and recorded with each run, so a score can always be traced to the exact prompt that produced it.
- A response that fails validation is recorded as a failure, never quietly turned into an empty answer, so a crash can't pass for the model rightly declining to guess.
Built with
Next.jsReactTypeScriptZodTailwind CSSGitHubVercelPrompt EngineeringContext EngineeringClaude CodeCodex