Riemannian barycentres and geodesic convexity

Consider a closed subset of a complete Riemannian manifold, such that all geodesics with end-points in the subset are contained in the subset and the subset has boundary of codimension one. Is it the case that Riemannian barycentres of probability measures supported by the subset must also lie in the subset? It is shown that this is the case for 2-manifolds but not the case in higher dimensions: a counterexample is constructed which is a conformally-Euclidean 3-manifold, for which geodesics never self-intersect and indeed cannot turn by too much (so small geodesic balls satisfy a geodesic convexity condition), but is such that a probability measure concentrated on a single point has a barycentre at another point.