Development of a two-stage string similarity approach for address data deduplication in information systems

This study investigates the process that determines similarity for the purpose of further deduplication of address data in information systems for logistics purposes. The task addressed is to enhance the accuracy and completeness in the automated detection of duplicate address records and to improve the balance between these values. This work has devised and experimentally substantiated a two-stage approach to determining similarity for eliminating duplicate address data, based on the sequential application of the Jaro-Winkler and Damerau-Levenshtein metrics. The proposed approach uses the Jaro-Winkler metric at the first stage for highly sensitive selection of potential duplicates and, at the second stage, applies the Damerau-Levenshtein metric to reduce false positives. The experimental study was performed on a data set containing 2273 pairs of address lines of the Ukrainian administrative-territorial division. The results of experimental studies showed that the proposed approach yields a precision of 0.902, a recall of 0.518, as well as the highest F1-measure value of 0.658, which exceeds the Jaro-Winkler metric value by 5.8% and the Damerau-Levenshtein metric value o by 36.0%. The advantage of the approach is achieved by combining the high sensitivity of Jaro-Winkler to variable forms of address representation with the capability of Damerau-Levenshtein to effectively detect character permutations and reduce the number of false positives. The results are attributed to the mutual compensation of the shortcomings of each applied metric within the framework of sequential two-stage processing of address data. The proposed approach could prove appropriate in logistics and geographic information systems and geocoding services