Monday, September 10, 2007

Multi-Variant Execution Environment

After modifying gcc to generate executables for upward growing stack and porting a C library for this purpose, it is time to prepare the multi-variant execution environment. The purpose of this environment is to check that whenever one of the variants calls a system call, all others also call the same system call with the same arguments. Besides, the whole multi-variant environment should impersonate a single process. Therefore, some system calls should be intercepted by the multi-variant environment and emulated rather than executed by all the variants. For example, when the variants try to open a file for writing, the file should be opened only once by the environment and its pointer should be sent back to all the variants.

Since I would prefer not to touch the Linux kernel, I decided to use ptrace with PTRACE_SYSCALL as the request to intercept the system calls. But, using this request causes the traced process to return to the tracer after executing int instruction, which is not what we need. As mentioned before, some system-calls should not be executed by the variants, but PTRACE_SYSCALL doesn't give us the ability to prevent them from calling the system-calls. Reading more about ptrace, I found out that PTRACE_SYSEMU was what I was looking for. Unfortunately, this is platform specific and, apparently, works well only on i386. Furthermore, our investigations show that it is not implemented in all Linux distributions. Thus, I am reluctant to use it.
So, the question remains open: "How can we intercept the syscalls and prevent the variants from executing them, with minimal performance overhead and without modifying the OS kernel?".

No comments: