An Approach to Organizing Data into Homogeneous Groups

Many problems which involve organizing data into homogeneous groups can be approached by defining a function for measuring or evaluating the structure present in a given partition of the set of data, and then attempting to find that partition for which the function is optimized. In general, different functions are required for different types of problems. One possible measure of structure is presented here, together with a discussion of the types of problems to which it applies. Also described is a general hill-climbing algorithm which can be used with any measure of structure to attempt to climb to the optimum partition. The algorithm does not require a numerical measure, but only a decision as to which of two partitions is more highly structured, or more valuable. Areas of application include biological taxonomy; isolation of disease syndromes in medicine; information retrieval; business applications such as “types” of sales offices, TV audiences, etc.; anthropology (categorization of civilizations); and sociology (categorization of tribes).