use alternate stack, or the heap for local function variables

[Keld Jørn Simonsen <keld at dkuug dot dk>]

Hi!

I would like to discuss an idea here that I got on how to avoid buffer
overflow exploits.  I am not sure it is the right forum, but it is
related to how functions are called, and glibc is all about functions.
It may be more related to the compiler, but then I know you folks better.

The idea is to separate the call stack, and the local function
variables, these are normally also put on the stack. But buffer overflow
exploits are then done by overflowing a string buffer into the return
address area of the stack, overwriting the return address to point to a
place with some malicious code. 

If the stack with the parameters and the function return code is
separated from the local function data, then there is no way for a
buffer overflow to overwrite the return address, and thus there is no
way for the intruder to get into the instruction execution flow.

The local function data could then be allocated on another stack, in
another data segment, or allocated on the heap. I am not sure of the
individual processors' capabilities wrt having more than one stack
pointer, but it should also be possible to allocate the data on the
heap. This would of cause need a little more stringent tidy-up procedure
when leaving the function, to free the heap memory, but that would IMHO
be a very small overhead for the improved security.

Is the idea viable?

best regards
Keld