On Pinsker's and Vajda's Type Inequalities for Csiszár's f-Divergences

Let D and V denote respectively Information Divergence and Total Variation Distance. Pinsker's and Vajda's inequalities are respectively D ≥ [ 1/ 2] V 2 and D ≥ log[( 2+ V )/( 2- V )] - [( 2 V )/( 2+ V )]. In this paper, several generalizations and improvements of these inequalities are established for wide classes of <;i>f<;/i>-divergences. First, conditions on f are determined under which an f -divergence Df will satisfy Df ≥ cf V 2 or Df ≥ c 2, f V 2 + c 4, f V 4 , where the constants cf , c 2, f and c 4, f are best possible. As a consequence, lower bounds in terms of V are obtained for many well known distance and divergence measures, including the χ 2 and Hellinger's discrimination and the families of Tsallis' and Rényi's divergences. For instance, if D (α) ( P || Q ) = [α(α-1)] -1 [∫ p α q 1-α d μ-1] and ℑ α ( P || Q ) = (α-1) -1 log[∫ p α q 1-α d μ] are respectively the relative information of type α and the Rényi's information gain of order α, it is shown that D (α) ≥ [ 1/ 2] V 2 + [ 1/ 72] (α+1)(2-α) V 4 whenever -1 ≤ α ≤ 2, α ≠ 0,1 and that ℑ α ≥ [( α)/ 2] V 2 + [ 1/ 36] α(1 + 5 α- 5 α 2 ) V 4 for 0 <; α <; 1. In a somewhat different direction, and motivated by the fact that these Pinsker's type lower bounds are accurate only for small variation ( V close to zero), lower bounds for Df which are accurate for both small and large variation ( V close to two) are also obtained. In the special case of the information divergence they imply that D ≥ log[ 2/( 2- V )] - [( 2- V )/2] log[( 2+ V )/2], which uniformly improves Vajda's inequality.