When Question-Based and Hierarchical Retrieval Don't Beat Flat Dense Retrieval

Generating questions at ingestion time and retrieving against them—rather than against raw passages—is an appealing way to shift large-language-model cost away from the query path while adding a question hierarchy to handle complex, multi-hop queries. We evaluate this idea with a fair eight-method, five-dataset benchmark and a controlled head-to-head in which a hierarchical question decomposition (HQD) tree and a flat question pool share an identical dense retrieval engine across two embedding models spanning a fivefold capacity range. Three findings emerge. First, lexical retrieval over full-text-indexed questions underperforms dense baselines because conjunctive query semantics discard most candidates before ranking, producing empty result sets on 78–95% of queries. Second, the hierarchy adds nothing over flat dense retrieval: the accuracy difference is at most 0.0072 in Mean Reciprocal Rank and does not grow under a fivefold increase in model capacity. Third, multi-hop accuracy plateaus near 0.64 on both models, suggesting a structural ceiling that additional model capacity cannot lift. We argue the missing ingredient is cross-document structure—a graph, not a tree—and that question generation's durable value lies in the ingestion-time cost shift: zero query-time language-model cost and deterministic retrieval.