Independent eval research. Open methodology, published datasets, and the tools to run your own studies.
Original, open-source studies
Model comparisons, agent setups, cost, and modalities, measured on real tasks. Read the methodology and results, or rerun a study on your own data.
Five models classified 2,000 aerial images across 30 categories. We compared their predictions with human labels and measured accuracy and API cost.
Agreement with human labelshigher is better
Per 2,000 imageslower is better
Jev, Mammoth-1, Clef, GPT-6 Luna, and other models classified prompts and responses as safe or unsafe across three public datasets. The eval compared their judgments with human labels and measured missed harm, false alarms, and API cost.
Answer-harm F2higher is better
Per 10,000 answer checkslower is betterMammoth-1 excluded because ongoing API pricing was unavailable.
Muse, Instinct, and Dots answered 25 shopping requests in a fixed WebShop catalog and on the open web. One blinded reviewer ranked their product recommendations and written responses.
Human reviewaverage rank1 is best
Human reviewaverage rank1 is best
Run evals like us
Each skill packages best practices from leading AI teams and researchers into a plain SKILL.md file your coding agent can read. Use them to get started running evals on your data.
Point your agent at a skill
Ask it to do what the skill describes, like validating a scorer against human labels or finding hidden failure modes.
- 1.Name the reference tier before computing anything.
- 2.Verify alignment: scorer outputs and reference labels must line up at the item and criterion level.
- 3.Report agreement (κ or α) with uncertainty, not raw accuracy.
- 4.Lead with the most decision-relevant false acceptance before any aggregate, and enumerate the dangerous cells case by case.
- 5.Break errors down by class and severity.
- 6.Test sensitivity and shortcuts: inject known regressions and improvements and confirm the scorer moves; probe whether length, confidence, or polish raise the score independent of quality; probe whether text addressed to the judge moves it; slice agreement by subgroup.