4.97.95
answer accuracy / 10The bot interrogated students.
Evaluation showed 18.6 percent of questions got a counter-question instead of an answer; I rewrote the clarification policy and verified the gain with an A/B evaluation.
Case study · Masaryk University, CZSThe answers were on the site. Staff answered by hand anyway.
An assistant that answers study-abroad questions on Masaryk University's official site, grounded in the university's own pages, every answer cited.
15,362graded evaluation runs
~90%measured answer accuracy
~1%measured hallucination rate
778source pages monitored
01 / Problem
778 sources, one inbox.
The rules live in 778 constantly changing sources in Czech and English, and every student question used to be answered by hand, one email at a time.
Source corpus, CS + EN
530 web pages174 PDF60 docx3 video transcripts
Before: every question answered manually, by email02 / Product
Ask in Czech or English. Get an answer with receipts.
Answers study-abroad questions in Czech or English, grounded in retrieved CZS sources and cited, so every claim traces back to a page. Facts that must not be guessed (current date, deadlines, contacts) come from deterministic tool calls, not the model's memory.
Opens with an AI disclaimer and suggested questions, in the student's language.
A real answer citing 11 named source documents, with feedback that routes to human review.
03 / System
The pipeline behind a straight answer
Every question is classified, then answered from hybrid OpenSearch retrieval (BM25 plus Voyage dense vectors, fused with RRF, two rerankers, MMR) over a heading-aware parent-child index, with a CRAG-style answerability gate that re-retrieves on weak context. Verified facts come from deterministic tool calls, not the model, so answers stream over SSE from DeepSeek-v3.2 on CERIT.
The path of one question
- 01QuestionCzech or English
- 02Classifyintent, entities, language
- 03Hybrid searchBM25 plus vectors, fused
- 04Reranktwo models, diversified
- 05Answerability gatere-retrieve if weak
- 06Tool callverified deadlines, contacts, date
- 07GenerateDeepSeek on CERIT
- 08Cited answerstreamed with sources
Freshness loop:A change monitor watches all 778 sources. Any change fires a webhook that re-indexes just that page, and a daily job sweeps the rest, so the bot never goes stale.
StackPythonFastAPIOpenSearchVoyage AIDeepSeek via CERITPostHognginx
04 / Feedback loop
Every approved answer makes the next one instant.
CZS handed over their archive of real student questions with verified answers; the system turns every new answer back into that archive.
707/ 715Q&A pairs staff verified8 promoted from live chats
- 01New questionscope check + FAQ match
- 02Known & verified?serve it, or draft from sources
- 03Staff reviewedit, approve
- 04Golden pairjoins the verified FAQ
Every approval grows the verified set
707 of 715 pairs are already staff verified, and the loop only adds to that number.
05 / Evaluation
Measured, not vibes
A continuous LLM-as-judge harness grades in-domain questions from the CZS FAQ (real historical plus generated) against the sources for accuracy, groundedness, and hallucination. Across 37 cycles and 15,362 runs, 10,438 were gradeable answers: 84 percent scored 9 or 10, mean 9.0, groundedness 8.6, hallucinations near 1 percent.
How 10,438 graded answers scored
scored 9 or 10scored 0 to 8
Accuracy distribution across 10,438 gradeable answers from 15,362 judged runs. The correct source appears in the top retrieval results 92 percent of the time, up from 79 before the parent-child index.
06 / Iteration
What measurement actually catches
0.790.92
correct source in top 7 resultsSmall chunks to find, full sections to answer.
I built a parent-child index: match on small, precise passages, then expand to the full section so the model answers with complete context. The right page now reaches the top results 92 percent of the time, up from 79.
15.0s10.6s
median (p50) answer latencyFaster, at higher accuracy.
The same retrieval redesign cut median answer latency by 4.4 seconds while answer accuracy went up, not down. Better context reached the model in fewer, cleaner passages.
None of these showed up in casual testing.
07 / Operations
CZS staff run this. Not me.
Staff see every conversation, review anything questionable, and update the knowledge base without touching code.
The screens shown here are illustrative mockups with invented example data; the real dashboards live behind a staff login.
Show: StudentsProgram: AllDirection: OutgoingSearch questions...
| ID | Source | Conf. | FB | Time | Intent | Program | User query |
|---|---|---|---|---|---|---|---|
| 4821 | STUDENT | 0.94 | 8.1s | check_requirements | Erasmus+ | What documents do I need for the selection? | |
| 4820 | STUDENT | 0.88 | · | 11.4s | find_deadline | Erasmus | When is the application deadline this year? |
| 4819 | STUDENT | 0.72 | 9.7s | scholarship_query | Erasmus+ | How much is the grant for a semester in Norway? | |
| 4818 | STUDENT | 0.91 | 7.3s | find_contact | Bilateral | Who is the coordinator for my faculty? | |
| 4817 | DEV | 0.83 | · | 10.2s | explain_procedure | Internship | How do I get my traineeship certificate signed? |
| 4816 | STUDENT | 0.79 | · | 12.8s | clarify_document | BIP | Do I upload the Learning Agreement before or after? |
| 4815 | STUDENT | 0.96 | 6.9s | find_deadline | Erasmus+ | When do results of the selection come out? | |
| 4814 | STUDENT | 0.87 | · | 9.1s | check_requirements | CEEPUS | How many ECTS do I need to complete abroad? |
Conversation Database
Every question and answer with confidence, sources, feedback, and response time; searchable.
Needs reviewIs the 20 ECTS requirement an Erasmus rule or an MU policy?
VerifyEdit answerResolve
Low confidenceWhich currency is the stipend paid in?
Thumbs downIs there a list of internship host institutions?
Flagged & Resolved
Thumbs-down and low-confidence answers wait here for staff verification.
| Source URL | Status | Chunks |
|---|---|---|
| czs.muni.cz/.../erasmus-evropa | up to date | 35 |
| czs.muni.cz/.../prakticke-staze/erasmus | up to date | 18 |
| czs.muni.cz/.../financni-podminky | changed | 12 |
| czs.muni.cz/.../po-ukonceni-pobytu | up to date | 9 |
| czs.muni.cz/.../mezifakultni-dohody | up to date | 14 |
Manage Sources
All 778 monitored pages with change-detection status, plus manual upload.
Can I go on Erasmus if I am over 26?match 0.91ApproveEdit
Which currency is the stipend paid in?match 0.87ApproveEdit
Is there a list of internship host institutions?match 0.82ApproveEdit
FAQ Review
Candidate FAQs mined from real conversations; nothing enters without human approval.
The AI drafts, humans decide; CZS gets the controls, not a support contract.
08 / Privacy
Private by default
No account, no personal data collected. Conversations are logged for quality review.
Everything runs on a dedicated server for the university, not a shared third-party AI service, and the bot reads only CZS's public pages.
Have a site full of answers nobody can find?
If users keep asking questions your site already answers, I build the assistant that closes the gap and the measurement proving it works.
An assistant your team controls, with the numbers to prove it works.