BenchMIRT: What are LLM benchmarks actually measuring?
BenchMIRT audits LLM benchmarks at the individual-prompt level by using multidimensional Item Response Theory to separate underlying capabilities, recovering two dominant dimensions—safety and general reasoning—across 16 benchmarks and 34K+ questions without being told which benchmark measures what.
Sep 1, 2026 ·Hugging Face















