Towards Accountable Autonomic Computing: Gated Multi-Agent Program Repair for Cloud-Native Incidents

Monitoring tools indicate when an issue has occurred within a service. Engineers are responsible for diagnosing, fixing and deploying the service. A model can be used to generate patches; however, these are not guaranteed to be correct, can be influenced by a text injection in a log line or can pass tests by removing them. We present GhostOps, our proof-of-concept AIOps platform that uses the MAPE-K loop. The incident is diagnosed by three agents and a patch is produced, which must pass several independent checks. These include a prompt injection protection, Open Policy Agent allows list, blast radius check and network isolated Docker sandbox with regression test that must fail and then pass. A risk score between 0-100 is used to decide which path the fix will go down. An audit log with chained hashes and signed evidence bundles are used to make sure all decisions can be verified. In an offline evaluation using a fake model with no cluster there were three scenarios with fixes requiring between 2-4 lines of code and took between 2.6-9.0s. Of the 530 tests run, 529 passed. One test failed which meant one regression proof could not be verified.