infrastructure-as-code

While LLMs show potential in general code generation, their efficacy in IaC development remains unknown. To address this, we developed the first dataset and benchmark capable of evaluating IaC code generation. Our dataset comprises 458 human-curated scenarios spanning various AWS services, involving over 1,720 hours of human effort. Our results reveal significant performance gaps.

Zodiac automatically unearths complex cloud IaC semantic checks/rules that state-of-the-art IaC tools cannot easily capture, allowing us to reduce runtime error violations that can take very long to debug, into simple compile time checks.