A relative trace formula and counting geodesic arcs in the hyperbolic plane

We study a modification of the hyperbolic circle problem: instead of all elements of a Fuchsian group Γ, we consider the double cosets by two hyperbolic subgroups. This has a geometric interpretation in terms of the number of common perpendiculars between two closed geodesics for Γ \ ℍ {Γ} . We prove an explicit relative trace formula, which is flexible for the counting problem. Using a large sieve inequality developed by the first author and Voskou, we prove a new bound in mean square for the error term of order O ⁢ ( X 1 2 ⁢ log ⁡ X ) {O(X1/2log X)} . We conjecture that the correct order of growth is O ⁢ ( X 1 2 + ϵ ) {O(X1/2+ε)} . Along the way, we provide a new proof of the pointwise error bound O ⁢ ( X 2 3 ) {O(X2/3)} , originally proved by Good.