Characterization of linux kernel behavior under errors

This paper describes an experimental study of Linux kernel behavior in the presence of errors that impact the instruction stream of the kernel code. Extensive error injection experiments including over 35,000 errors are conducted targeting the most fre- quently used functions in the selected kernel subsystems. Three types of faults/errors injection campaigns are conducted: (1) ran- dom non-branch instruction, (2) random conditional branch, and (3) valid but incorrect branch. The analysis of the obtained data shows: (i) 95% of the crashes are due to four major causes, namely, unable to handle kernel NULL pointer, unable to handle kernel paging request, invalid opcode, and general protection fault, (ii) less than 10% of the crashes are associated with fault propagation and nearly 40% of crash latencies are within 10 cycles, (iii) errors in the kernel can result in crashes that require reformatting the file system to restore system operation; the process of bringing up the system can take nearly an hour. Subsequently, over 35,000 faults/errors are injected into the kernel functions within four subsystems: architecture- dependent code (arch), virtual file system interface (fs), cen- tral section of the kernel (kernel), and memory management (mm). Three types of fault/error injection campaigns are con- ducted: random non-branch, random conditional branch, and valid but incorrect conditional branch. The data is analyzed to quantify the response of the OS as a whole based on the sub- system and to determine which functions are responsible for error sensitivity. The analysis provides a detailed insight into the OS behavior under faults/errors. The major findings in- clude: • Most crashes (95%) are due to four major causes: unable to handle kernel NULL pointer, unable to handle kernel paging request, invalid opcode, and general protection fault. • Nine errors in the kernel result in crashes (most severe crash category), which require reformatting the file system. The process of bringing up the system can take nearly an hour. • Less than 10% of the crashes are associated with fault propagation, and nearly 40% of crash latencies are within 10 cycles. The closer analysis of the propagation patterns indicates that it is feasible to identify strategic locations for embedding additional assertions in the source code of a given subsystem to detect errors and, hence, to prevent er- ror propagation.