Operating system enhancements to prevent the misuse of system calls

We propose a cost-effective mechanism, to control the invocation of critical, from the security viewpoint, system calls. The integration into existing UNIX operating systems is carried out by instrumenting the code of the system calls so that the system call itself once invoked checks to see whether the invoking process and the argument values passed comply with the rules held in an access control database. This method provides simple interception of both system calls and their argument values and do not require changes in the kernel data structures and algorithms. All kernel modifications are transparent to the application processes that can continue to work correctly without needing changes of the source code or re-compilation. A working prototype has been implemented inside the kernel of the Linux operating system, the prototype is able to detect and block also buffer overflow based attacks.