Well, you cannot expect to be able to recover from out-of-memory
conditions anyways. As competing processes fight for memory, you may
lose memory you intended to use for a graceful exit, like saving state
or something. Indeed, that's vexing, but such is life with multi-tasking
operating systems.
I can't believe I'm reading this, I truly hope I'm mistaken. I think you
are talking about the situation where one has the overcommit bit enabled.
Or are you truly saying that it's OK for that function to return an invalid
pointer
and that we should live with it, just waiting for the next segfault to
happen? What's the point?
I don't know on which system fetchmail is developed, probably a system
where alloca works as expected, returning NULL when the requested
ammount of memory is not available (as the manpage says it should).
Read the GNU Info docs on alloca, including the disadvantages, since
they're more complete than the man pages (on my Redhat Linux 7.1
system anyway).
Essentially, alloca is designed so it can simply "ride" the
stack-based allocation mechanism that is employed by most systems
(most combinations of CPU, OS, and ABI in use today as general-purpose
computing systems, as far as I'm aware).
In this mechanism, memory is "allocated" simply by incrementing or
decrementing a stack pointer. There is no "error code" that can be
returned for allocating more than is available from the OS, since the
OS is not directly consulted on the matter. (I'm making a distinction
here between virtual memory actually being unavailable and the
allocation failing due to "syntactic" limitations, e.g. the stack
pointer becoming inherently invalid, such as mathematically
overflowing or underflowing into another memory region. I'd be
surprised if every implementation of alloca() out there even caught
*that* sort of error and returned NULL, since function calls and the
like generally don't.)
The OS must cooperate with this approach by being willing to
dynamically *reserve* memory allocated in this way when the memory is
actually *touched* by the program. That is, the memory is actually
reserved not "in line" with the allocation of it by the program, but
more in an "out of band", ad-hoc fashion, later on, if it wasn't
reserved much earlier by earlier allocations of the stack (which one
hopes is the case more than 99% of the time for any program that runs
more than a few milliseconds).
I'm trying to make a careful distinction between "allocate" and
"reserve" here, in that, by "allocate", I mean the program tells
*itself* that a given area of memory is to be used for a given data
set, whereas, by "reserve", I mean the program is assured that the
memory actually will be available to store data, a guarantee typically
offered by the OS under which it runs.
Three male friends choosing among themselves which of them gets to
marry a particular woman are, in this terminology, "allocating" her
among themselves, while the woman agreeing to actually marry any one
of them is "reserving" herself for the role. These are fairly
orthagonal to each other, in that reservation may precede, follow, or
happen at the exact same time as allocation, but if one happens
without the other ever happening, then there is no marriage between
the two parties (the woman and any of the three men). Down this road
is an inevitable discussion involving dangling pointers and Mormonism,
so I won't go any further.
(An old operating system, PRIMOS, reserved memory for *all* dynamic
allocation this way: there was no sbrk() equivalent, you simply
touched memory somewhere, and PRIMOS would extend the range of memory
reserved to your process through that point, or signal an error. I
believe most UNIXes don't do this for heap-based memory, so you have
to use sbrk() to reserve the memory before you touch it. That's part
of why heap-based allocation is slower than stack-based.)
It is this same stack-based allocation scheme that is used to handle
procedure ("function" in C) calls, argument passing, and so on, on
most systems. These actions are not, in most languages, capable of
returning "failure" -- one cannot, in C, write code that says "invoke
function foo(), return error code if there's not enough stack space to
hold the return address, arguments, and foo's local variables".
Therefore, the mechanism whereby a stack pointer is simply incremented
and decremented to allocate and free memory, and the OS reserves
additional space only when actually needed, works very well -- all
sorts of code that would otherwise be generated by the compiler to say
"see if we need to reserve more memory from OS, if so try to reserve
it, fail if we can't, otherwise: adjust the stack pointer to allocate
it" is replaced by code that simply adjusts the stack pointer,
assuming the memory is already reserved or will be reservable when
actually needed later on.
Similarly, no code need be produced to return to the OS any reserved
stack space that is not expected to be used for awhile. The stack in
actual use is simply reduced, the OS never notices or cares that
memory it previously reserved for it on an ad-hoc basis can be
returned to the system, and the program continues this way until it
ends. (Maybe some OSes garbage-collect these regions for
long-running, dormant programs, but, personally, I'd rather they
didn't -- "out of band" deallocation of memory has the effect of
randomly changing the values in memory, as far as the running program
can see, so unless the OS *proves* the program cannot, even in error,
reference the unallocated regions, it should not unreserve them, so
programming errors are easier to track down, instead of becoming
Heisenbugs.)
So not only does the code generated to handle procedure calls,
argument passing, and reserving of (automatic) local storage get
smaller, but the overall speed of the system gets faster. Only when
the stack grows initially beyond whatever the OS might have allocated
for it do segmentation/page faults trigger slow, new reservations of
memory by the OS -- from then on, allocating and freeing stack memory
mean simply adjusting the stack pointer.
alloca() is fundamentally designed to offer this mechanism directly to
the C programmer, given that it's already offered to him indirectly
via the linguistic constructs known as function calls, argument
passing, and automatic local variables.
That is, your query "it's OK for that function to return an invalid
pointer" is, strictly speaking, incorrectly formulated: the pointer
returned by alloca() (which, by the way, isn't a function in the
normal C sense, it's more like a macro or an inlined function) is, I
would think, *always* valid per se, or at least valid except when it
has mathematically overflowed/underflowed the syntactic boundary of
the stack on that system (in which case it'd be nice if alloca()
returned NULL, but I wouldn't bet on it doing so on all systems).
What isn't clear is whether the operating system will cooperate when
the program actually touches the memory pointed to by the pointer
returned by alloca(), unless a "special" version of alloca() is used
that does not simply mimic the way in which function calls, the
passing of arguments, and automatic local variables are allocated on
that system -- that goes to extra trouble to ensure that reservation
happens "on the spot" or returns NULL if it doesn't.
So while alloca() can *theoretically* ensure that any memory it
allocates is actually reserved to the program via the OS, that would
slow it down vis-a-vis other uses of stack-based allocation. I'm
guessing this means most "native" alloca() implementations (most
built-in to the compiler) don't do such checking. Therefore, even if
alloca() returns a non-NULL pointer, the memory might not actually be
available, though one is assured that, if available, it won't be
allocated for any other purpose until the calling function returns.
Similarly, when a function containing a 1GB automatic local variable
is invoked, depending on how the stack frame is arranged, the function
might start executing and fail (with a segment violation) only when it
begins accessing portions of that 1GB variable beyond what the OS is
willing to actually reserve for it at the time.
I suppose one could gin up a "failsafe" version of alloca() in the
form of a macro that first invokes alloca(), then passes the pointer
it returns, along with the size to allocate, to a function that sets
up a signal catcher for segment violations, then scribbles over the
newly allocated memory in an appropriate way to ensure the OS actually
reserves the memory, returning NULL as the final pointer if any
signals are caught.
Going to that degree of trouble suggests that one is dealing with
existing code that can't easily be changed to simply use the
heap-based allocation provided by malloc()/free(). Both approaches
would certainly slow down the program compared to permitting
out-of-band segment violations as a result of running out of memory.
Though, using heap-based allocation in place of stack-based allocation
can lead to even worse performance and other problems, mainly due to
memory fragmentation.
(I've just skimmed fetchmail's alloca.c replacement: indeed, it uses
heap-based allocation, and, "worse yet", doesn't free() as quickly as
one would prefer. Rather, allocations are freed only upon subsequent
invocations of alloca(), so a 1GB alloca() by a function that then
returns to a function that malloc()'s a handful of bytes will leave a
1GB sorta-allocated "hole", likely resulting in a fragment, in the
heap, unless one hand-codes "alloca(0);" prior to the subsequent
malloc().)
But, ideally, one wants the linguistic, or semantic, capabilities of
alloca() -- that is, "give me N bytes dynamically allocated that're
automatically freed when my containing procedure returns" -- without
the headaches of using whatever existing mechanism might be in place
for other activities like function calls (headaches like "can't be
sure the memory allocated is actually usable without raising a segment
violation"). Especially if one isn't interested so much in
performance as in predictability, one would prefer a means to write
"clean" code and direct the system (compiler/libraries/OS) to
implement it in a "defensive" fashion.
Anyone wanting such fine-grained control over the distinctions between
design and implementation, between semantics and actual run-time
behavior, is best advised to avoid programming languages like C and
FORTRAN. Those languages are not for linguistic purists!
Since the languages themselves don't always cooperate, when they're
used in a situation where predictability is more important than
performance, stripped-down versions, or subsets, of these languages
must be used instead, with all sorts of explicit defensive coding in
place.
ANSI FORTRAN 77, for example, can be fully implemented by a system
such that *no* run-time dynamic allocation of memory is ever needed,
so a program written in that language can, in and of itself, be said
to be incapable of running out of memory once it successfully starts
running in the first place, *assuming* the language processor
(compiler, run-time libraries, etc.) cooperate with this goal (by not
using arbitrary levels of dynamic allocation "underneath the hood" for
things like I/O, for example). In this sense, ANSI FORTRAN 77 offers
the programmer a sort of guarantee, though the programmer is still
responsible for getting the corresponding guarantee from whatever
*system* he uses to deploy his code.
There's no equivalent dialect (subset) of C of which I'm aware, but a
programmer can stick to shallow depth of function invocations, few
arguments to functions, and small dynamic local variable sets per
function, such that a static analysis of the stack space required by
the program shows that it never can exceed an amount guaranteed by a
specific implementation (compiler, library, and OS combination) once
the program starts running.
A more practical approach is to understand the memory-allocation
patterns of a program and specifically target the larger allocations
that might be performed by an alloca() and fix just *those* to use
either in-line forced reservation (catching signals, or at least
failing, before committing to a transaction that can't be completed)
or malloc()/free(). (More precisely than targeting "larger
allocations", one can target allocations that aren't provably
"pre-reserved" by earlier reservations of the stack.)
In short, while it may well be said that "to iterate is human; to
recurse, divine", there appear to be practical real-world limitations
on the resources required by divinity.
--
James Craig Burley, Software Craftsperson
<http://world.std.com/~burley/>