Bifurcation of limit cycles is investigated for a cubic Hamiltonian system perturbed by fifth-order polynomials. Compared with the existing work, there exist two nested homoclinic orbits in this unperturbed system, in which a large-sized homoclinic orbit encloses a smaller one starting at the same saddle point. Using the method of detection function, the Abelian integrals over irregular regions are precisely computed. It is proven that the system exhibits at least Formula: see text–Formula: see text limit cycles with four distinct distributions. Precise detection curves are computed to validate the analytical results, and four groups of parameter conditions are presented by considering Hopf, heteroclinic, and homoclinic bifurcation values, respectively. These results help explore the role of nested homoclinic loops in influencing the number and distribution of limit cycles.
Wang et al. (Thu,) studied this question.