Finding the right source
Before an AI writes anything, search has to find the right page. How often the correct source is among the top results, over the eval set's answerable questions.
Public test results
Every number here comes from automated tests that run without an AI model, so they're repeatable and anyone can rerun them. Nothing is hand-picked.
Before an AI writes anything, search has to find the right page. How often the correct source is among the top results, over the eval set's answerable questions.
Questions answered from game data or refused by a guard are scored in full. Questions that need an AI-written answer are counted but not scored here, since that needs a model.
Anonymous 👍 and 👎 on answers over the last 7 days, by kind of answer. Counted in memory, so a server restart starts it over.
Rerun it yourself:
uv run irminsul-report --build data/build --index data/index
Source: github.com/MukulRay1603/Irminsul · raw data: results.json