Public test results

How accurate is Irminsul?

Every number here comes from automated tests that run without an AI model, so they're repeatable and anyone can rerun them. Nothing is hand-picked.

Finding the right source

Before an AI writes anything, search has to find the right page. How often the correct source is among the top results, over the eval set's answerable questions.

What an answer costs

By kind of question

Questions answered from game data or refused by a guard are scored in full. Questions that need an AI-written answer are counted but not scored here, since that needs a model.

How it's measured

Rerun it yourself:

uv run irminsul-report --build data/build --index data/index

Source: github.com/MukulRay1603/Irminsul · raw data: results.json