Silent Failure Modes in Free-Tier Graph Database Benchmarking: A Methodology and Four Case Studies

Comparative benchmarks of cloud database services are widely published and widely trusted, yet the conditions under which they are run—constrained free tiers, geographically distant clients, heterogeneous engine internals—create failure modes that do not announce themselves. A benchmark that is silently measuring the wrong thing does not crash or produce outliers; it produces clean, plausible, well-ordered tables that are ready to publish and entirely meaningless. This work presents a reproducible benchmark of four managed and self-hosted graph databases (CognoDB Cloud, Neo4j Aura, Memgraph, and FalkorDB) on an identical 204,109-relationship subgraph of the SNAP soc-Pokec social network, together with a fifth engine (NebulaGraph) that could not be run at resource parity and is reported as a documented negative result. The primary contribution is not a ranking of these systems but a methodology for detecting when a benchmark has stopped measuring what it claims to. We document four concrete failure modes encountered and corrected during this study, each of which initially produced credible-looking numbers: (1) disconnected graph sampling that renders traversal latency meaningless while appearing excellent; (2) a network-dominated latency regime in which round-trip time exceeds engine time by up to two orders of magnitude, defeating naive subtraction-based correction; (3) identical query text executing under non-equivalent access paths, yielding a 285x performance difference attributable entirely to an implicit versus explicit index; and (4) a nominally neutral resource cap that structurally excludes distributed architectures. For each we give the observation, the root cause, and the correction, and we argue that the machinery required to catch these failures is the transferable output of the work. We additionally report a reproducible query-correctness defect in one engine, discovered by a data-validation check written for an unrelated purpose.