Second-Order Zipf’s Law for Word Co-Occurrences

Many advances in NLP, starting with distributional semantics, are founded on the insight that the semantics of words can be revealed by their co-occurrences (the frequency for two words to be present in the same sentence). And the most sophisticated modern machines are built to reproduce the statistics of even larger sets of words. Yet, these systems do not provide a transparent description of these statistics. We reveal a second-order Zipf’s law: the frequencies of co-occurrences follow an inverse power law in all 17 languages we tested. To assess this, we use the most conservative methods to detect a power law (and even develop its extension in the natural case of a finite vocabulary, which reveals new insights). We show that this second-order Zipf’s law is not a consequence of random processes, it is not a consequence of the usual Zipf’s law on word frequencies (it is less robust in a shuffled corpus), and it is not a consequence of syntactic constraints (it is less robust in a shuffled corpus that respects syntactic categories). The second-order Zipf’s law may therefore be more foundational than Zipf’s original law, and able to provide (transparent) information about semantic aspects of language.