pluggable modules? (dev idea)

I'm beginning to think I really should investigate what is involved with 
adding plugin functionality to procmail.  The more external processes I 
call, the slower things eventually get.  It'd be really handy to be able to 
write a support module for something which might normally be called 
multiple times during a procmail session, and make it into a functional 
extension of procmail.  Sure, the average user isn't going to be writing 
anything to extend procmail, but a handful of extensions made available to 
the procmail userbase could markedly simplify and optimize a variety of tasks.
        INCLUDEMODULE=some_program_module

One could even do this from within a condition, so it's not included if it 
isn't really needed, though I've a different idea on this outlined below.
With this, db lookups (such as greenlist checks generally performed using 
grep), 'host', SPF, DNSBL, all could easily be integrated into procmail 
with minimal overhead (particularly as they may apply to walking headers 
and performing multiple lookups).  Math functions (such as bc) would be 
handy.  IP quad inversion without gobs of code, some simple sed operations, 
basic formail type actions from WITHIN procmail, it goes on and on.
Yea, the modules would need to be written, but once that is done, they 
become a standard extension to procmail itself.  Calling a grep type 
operation a bunch of times suddenly doesn't look so bad.
An alternate approach might be to declare module function names, and the 
name of the module file which would be loaded to provide those functions:
INCLUDEMODULE=some_program_or_library_file FUNCTIONNAME FUNCTIONNAME2 ...

This way, procmail could hold off from loading and initializing a library 
module if the functions are never actually used (negating the need to 
perform INCLUDEMODULE from within a condition), and once they are used, 
they'd be loaded and initialized, then flagged in a table as loaded so that 
the next time they're used, they'd be recognized as already 
initialized.  Yes, as per the above syntax, I think it should be reasonable 
that a single module could define multiple functions if someone were so 
industrious as to write them.  A basic module could be produced with a 
variety of standard functions.
Alternatly, the normal function load would include the function names and 
insert them into a plugin function table, but that means the file needs to 
be opened and executed just to get the list of functions, whether they are 
used or not.
As to backward compatibility, the INCLUDEMODULE line would be ignored and 
emitted as a syntax warning in the logfile on older procmails, which would 
continue parsing the rcfile, and since the invocation of the functions 
would be generally indistinguishable from program invocations, those would 
either correspond to progs the user had (and COULD have instead of a 
plugin), or they'd get an error because the program wasn't found.  Thus, 
we've got pretty stable backward compatibility with no significant issues.

The syntax to invoke these loaded functions would be very nearly the same 
as how programs are already run, but the current program exec process in 
procmail would need mods to handle invoking plugins through a new plugin 
manager (checking the plugin name table, loading/initializing and then 
calling the internal function instead of shelling), and would certainly 
need some tweakage to contend with piplelined execs.

INCLUDEMODULE=whitelist.mod whitelist_lookup

WHITELISTED=`whitelist_lookup ${USERNAME} ${DATAFILE}`


Basically, if the program to be invoked is found to be registered in the 
pluggable module table, then we deal with initializing it (if not already 
done), and then calling the loaded function with the parameters.  Some 
dancing about with file handles (STDIN, STDOUT, STDERR), and I think 
reasonably, it could be pretty transparent for the rest of the procmailrc.

Some issues that spring to mind include forking a copy of procmail and 
inheriting the plugged modules, as well as contending with modules which 
for one reason or another might opt to retain static data (how are these 
dealt with in the event of a fork?), such as preloaded index files, or a 
cache of previous lookups on something.
Having support for pluggable modules wouldn't dramatically increase the 
size of procmail, nor appreciably slow down it's default (non-plugin) 
operation - a few cycles for new table lookups, but not much more beyond that.

Besides being loaded into memory and having the ability to run multiple 
operations without involving multiple processes, a plugged module _could_ 
conceivably have access to the procmail variable table, where it could say, 
create multiple variables based on some action, rather than needing to 
return a long string which would need to be parsed into pieces longhand in 
the procmailrc.  This, and the potential for statefullness (static 
variables) between invocations during the same procmail session would make 
the pluggable modules dramatically different than what you get in the 
current program exec model - implementing them would of course mean that 
one couldn't merely have a functionally equivalent program stub for older 
procmails to invoke, because those wouldn't be able to tweak the procmail 
variables.  This I don't envision to be a big problem - the added 
variable-access and optional stateful functionality would be beneficial, 
and people really aught to upgrade their mail tools once in a while. <g>
MIME handling stands out as one potential thorn which could be clipped with 
a good procmail plugin, and yet it keeps the MIME code _OUT_ of procmail - 
if you don't need it, it isn't there bulking procmail up and slowing it down.

Thoughts anyone?


As for me having the time to do this very soon, it is rather unlikely, but 
I think opening a dialogue and sorting some technical details is the first 
stage.
---
 Sean B. Straw / Professional Software Engineering

 Procmail disclaimer: <http://www.professional.org/procmail/disclaimer.html>
 Please DO NOT carbon me on list replies.  I'll get my copy from the list.


_______________________________________________
procmail mailing list
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail