SpiceFuzz: LLM-Based Fuzzing for Spice Circuit Simulator Tools Bug Detection

Analog circuits play a vital role in electronic design and are key in connecting digital circuits with the physical environment. Ensuring the accuracy and reliability of analog circuit simulation tools (i.e., Simulation Program with Integrated Circuit Emphasis (SPICE)) is crucial; any errors in analog circuit simulation tools may lead to defects in analog circuit design, which may cause major problems and losses in the design and production process. Due to the lack of complete and unified specifications for analog circuit design, existing simulation tool testing works face the challenge of how to generate more effective analog circuits and improve test efficiency. To overcome these challenges, we propose a new fuzz testing method called SpiceFuzz, which is the first to use training-free LLMs to generate effective and diverse SPICE netlists to test SPICE simulators. First, SpiceFuzz combines carefully constructed mutation prompts with a flow with feedback to automatically generate effective analog circuits. Then, SpiceFuzz guides mutation selection by memorizing the historical selections and their results, thereby selecting mutations that are more likely to trigger bugs for the specific analog circuits. Finally, the differential testing component compares the simulation results of generated SPICE netlists to find potential bugs in the simulators. Experimental results show that SpiceFuzz outperforms the state-of-the-art methods, in terms of the number and simulation success rate of SPICE netlists generated. Meanwhile, in one month, SpiceFuzz reports 12 discovered bugs in NgSPICE and LtSPICE, 7 of which are confirmed by the developers.