Automated testing of NFV orchestrators against carrier-grade multi-PoP scenarios using emulation-based smoke testing

Future large-scale network function virtualization (NFV) environments will be based on hundreds or even thousands of NFV infrastructure installations, the so called points of presence (PoP). All their resources and the services deployed on top of them will be controlled by management and orchestration (MANO) systems. Such large-scale scenarios need to be automatically tested during the development phase of a MANO system. This task becomes very challenging because large-scale NFV testbeds are hard to maintain, too expensive, or simply not available. In this paper, we introduce the concept of emulation-based smoke testing, which enables automated, large-scale testing of MANO systems. We show that our test platform prototype can easily emulate up to 1024 PoPs on a single physical machine and that it can reduce the setup time of a single test PoP by a factor of 232 × compared to a DevStack-based PoP installation. In a case study, we test and compare two versions of a state-of-the-art MANO solution, namely ETSI’s Open Source MANO (OSM), in large-scale scenarios using our prototype. The issues we found in this study would not have been discovered with existing, lab-scale test environments.