On decidability of the product of subgroups membership problem for nilpotent groups

Abstract The product of subgroups membership problem for a finitely generated group 𝐺 is the decision problem, where, given two or more finitely generated subgroups K i K₈, i = 1, …, m i=1, , m, of 𝐺 and a group element 𝑔, it is asked whether g ∈ ∏ i = 1 m K i g₈=₁^mK₈. In this paper, we prove that, for any finitely generated nilpotent group 𝑁 of class two, the product of two subgroups membership problem is decidable. Note that this problem is equivalent to the problem of determining the non-emptiness of the intersection of two cosets of finitely generated subgroups. We also show that, for a sufficiently large direct power H n H^n of the Heisenberg group ℍ, there exists a product K = ∏ i = 1 4 K i K=₈=₁^4K₈ of four finitely generated subgroups for which the membership problem is algorithmically undecidable. The proof of the last assertion is based on the undecidability of Hilbert’s 10th problem and interpretation of Diophantine equations in nilpotent groups. The question of decidability of the membership problem for a product of three finitely generated subgroups in a nilpotent group of class two remains open.