What question did this study set out to answer?

This research aims to extend the Kazakh retrieval benchmark by evaluating new models and hybrid techniques for improved search performance.

June 23, 2026Open Access

Hybrid Retrieval and Out-of-Domain Robustness for Kazakh Search: Extended Benchmark with New Models, RRF Fusion, and Tokenizer Analysis

Key Points

This research aims to extend the Kazakh retrieval benchmark by evaluating new models and hybrid techniques for improved search performance.
Evaluated three embedding models on a 300-query Wikipedia benchmark;
Implemented Reciprocal Rank Fusion (RRF) to combine BM25+stemmer with each dense model;
Conducted an out-of-domain validation on 244 queries from presidential speeches.
The BM25+stemmer ⊕ kazakh-e5 hybrid achieved nDCG@10 = 0.808 on Wikipedia, outperforming the best single model (0.785);
Out-of-domain scores were stable (Spearman ρ = 0.89) but lower on Akorda (best hybrid 0.562);
Kazakh-fine-tuned e5 performed significantly worse than base e5 (Δ=−0.037, p=0.005) on Wikipedia.

Abstract

We extend the Kazakh retrieval benchmark from Paper 1 in three directions: (1) we evaluate three new embedding models — IBM Granite R2 (97M and 311M) and a Kazakh-fine-tuned E5 (shyngys-e5) — on the original 300-query Wikipedia benchmark; (2) we add Reciprocal Rank Fusion (RRF) hybrids combining BM25+stemmer with each dense model; (3) we replicate the full 7-system comparison on an entirely different corpus — official presidential speeches from akorda.kz (244 queries) — as an out-of-domain (OOD) validation. We also analyze sub-word tokenizer fertility as a candidate mechanism for observed performance gaps. Key results: the BM25+stemmer ⊕ kazakh-e5 hybrid is the best system on Wikipedia (nDCG@10 = 0.808, vs 0.785 for the best single model). OOD rankings on Akorda are largely stable (Spearman ρ = 0.89), confirming that Paper 1 conclusions generalize beyond Wikipedia; absolute scores are lower on Akorda (best hybrid 0.562 vs 0.808 on Wikipedia) because formal political text is harder for all systems uniformly. Granite R2 underperforms R1 on Kazakh on both domains; sub-word fertility analysis reveals that R2's tokenizer fragments Kazakh words 2.3× more than R1/e5 — a plausible tokenizer-level mechanism for its domain drop. kazakh-e5 (Kazakh-specific fine-tune of e5) is significantly worse than base e5 overall (Δ=−0.037, p=0.005 on Wikipedia), an honest negative result for domain-specific fine-tuning. The hybrid is the safest cross-domain default.

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper