Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game

Simulated DAG models may exhibit properties that, perhaps inadvertently, their structure identifiable and unexpectedly affect structure learning. Here, we show that marginal variance tends to increase along the order for generically sampled additive noise models. We introduce as a measure of the agreement between the order of increasing variance and the causal order. For commonly sampled graphs and model, we show that the remarkable performance of some continuous learning algorithms can be explained by high varsortability and by a simple baseline method. Yet, this performance may not transfer to-world data where varsortability may be moderate or dependent on the choice measurement scales. On standardized data, the same algorithms fail to the ground-truth DAG or its Markov equivalence class. While removes the pattern in marginal variance, we show that data processes that incur high varsortability also leave a distinct pattern that may be exploited even after standardization. Our challenge the significance of generic benchmarks with independently parameters. The code is available at://github.com/Scriddie/Varsortability.