Saturday, September 15, 2007

getpid for syscall emulation

A solution to the problem that I talked about in my previous post is to use PTRACE_SYSCALL and call another system-call that doesn't change the state of the program, such as getpid, instead of the requested system-call.
Here is how it works: A child process calls a system-call that shouldn't be executed by the child, e.g. open file for writing. PTRACE_SYSCALL returns to the parent process after execution of the int instruction and before running the syscall. The parent process replaces the syscall number (eax register) with getpid syscall number and lets the child continue. The kernel runs getpid instead of the open system-call and returns to the parent at the end of the syscall. Now the parent process, replaces the child's registers by appropriate values (perhaps the values that the parent has obtained by running or emulating the system-call itself) and returns to the child. The child continues execution using the values that the parent has given to it.
Using this method we can use PTRACE_SYSCALL and emulate system-calls for a traced process.

Monday, September 10, 2007

Multi-Variant Execution Environment

After modifying gcc to generate executables for upward growing stack and porting a C library for this purpose, it is time to prepare the multi-variant execution environment. The purpose of this environment is to check that whenever one of the variants calls a system call, all others also call the same system call with the same arguments. Besides, the whole multi-variant environment should impersonate a single process. Therefore, some system calls should be intercepted by the multi-variant environment and emulated rather than executed by all the variants. For example, when the variants try to open a file for writing, the file should be opened only once by the environment and its pointer should be sent back to all the variants.

Since I would prefer not to touch the Linux kernel, I decided to use ptrace with PTRACE_SYSCALL as the request to intercept the system calls. But, using this request causes the traced process to return to the tracer after executing int instruction, which is not what we need. As mentioned before, some system-calls should not be executed by the variants, but PTRACE_SYSCALL doesn't give us the ability to prevent them from calling the system-calls. Reading more about ptrace, I found out that PTRACE_SYSEMU was what I was looking for. Unfortunately, this is platform specific and, apparently, works well only on i386. Furthermore, our investigations show that it is not implemented in all Linux distributions. Thus, I am reluctant to use it.
So, the question remains open: "How can we intercept the syscalls and prevent the variants from executing them, with minimal performance overhead and without modifying the OS kernel?".

Monday, September 3, 2007

Sibling Calls in GCC

When we generate code for upward growing stack on x86 target, the stack pointer (SP) is adjusted at the beginning of all functions to bypass the return address (refer to this article). However, when a sibling function is called the control is transfered to the function using a jump instruction rather than a "call", which means that no return address is inserted on the stack. In these cases, we should "anti_adjust" (decrement, in our case) the SP before the jump in order to compensate the value added to the SP at the beginning of the function.

I made gcc insert the code to decrement the SP when emitting a call to a sibling function, but it muddled the stack offset computation in restoring saved registers. I removed the code and added it to the epilogue expansion. This works flawlessly.