CLOUDLIGHTNING: A Framework for a Self-organising and Self-managing Heterogeneous Cloud

As clouds increase in size and as machines of different types are added to the infrastructure in order to performance and power efficiency, heterogeneous clouds are being created. However, exploiting architectures poses significant challenges. To efficiently access heterogeneous resources and, at the time, to exploit these resources to reduce application development effort, to make optimisations easier to simplify service deployment, requires a re-evaluation of our approach to service delivery. We a novel cloud management and delivery architecture based on the principles of self-organisation self-management that shifts the deployment and optimisation effort from the consumer to the software running on the cloud infrastructure. Our goal is to address inefficient use of resources and to deliver savings to the cloud provider and consumer in terms of reduced power consumption improved service delivery, with hyperscale systems particularly in mind. The framework is general but endeavours to enable cloud services for high performance computing. Infrastructure-as-a-Service is the primary use case, however, we posit that genomics, oil and gas exploration, and ray tracing three downstream use cases that will benefit from the proposed architecture.