Multiplicity and Small Representations for Some Gelfand Pairs

Let Formula: see text be the semidirect product associated to the Gelfand pair Formula: see text where Formula: see text is a connected and simply connected nilpotent Lie group, and Formula: see text is a compact subgroup of the automorphism group, Formula: see text of Formula: see text Under some assumptions on the pair Formula: see text we give a precise connection between the spherical representations and the small representations of Formula: see text which are the irreducible representations of Formula: see text that cannot be Hausdorff separated from the trivial one-dimensional representation Formula: see text of Formula: see text (called the cortex of Formula: see text and denoted by Formula: see text). More precisely, we show that the set of the small representations of Formula: see text (Formula: see text) is strictly contained in the closure of the set of the spherical representations of Formula: see text (with respect to the Fell topology). Furthermore, we prove that any spherical representation of Formula: see text cannot be a small representation. In the special case, (Formula: see text is abelian) we show that the converse of this assertion is not true.