The paper Automated Infrastructure as Code Program Testing was published in IEEE Transactions on Software Engineering (TSE) in 2024 SSS24a. Infrastructure as Code (IaC) automates infrastructure management through code, replacing manual configuration. With Programming Lan-guages IaC (PL-IaC) like Pulumi and AWS CDK, developers express infrastructure in general-purpose languages like TypeScript or Python. Systematic testing of such IaC programs is rare according to our large-scale analysis of public Pulumi IaC programs SSS24b. Developers were limited to slow and costly integration tests, or unit-testing with brittle, hand-written mocks, lacking quick and reliable testing methods. To address this gap, we introduced Automated Configuration Testing (ACT), which unit-tests IaC programs across many configurations by (1) automatically mocking all resource definitions and (2) using pluggable generators and oracles for input synthesis and validation. We implemented ACT in ProTI for Pulumi TypeScript, using provider schemas for type-directed generation and checks. Evaluation on 6,081 real-world programs showed that ProTI exposes configuration errors quickly in about 40% of cases. ACT improves IaC testing and promotes reproducible, verifiable DevOps practices.
Sokolowski et al. (Thu,) studied this question.