procmail
[Top] [All Lists]

Re: processing emails that bounce.

2002-05-21 14:31:17
At 13:07 2002-05-21 -0700, Teddy Campbell did say:
What would be the best way to catch emails that bounce? Heres something similar to my situation.

Why not just describe _your_ situation?

I have a mailing list of people who want to get a news letter. Some of the people no longer have their email addresses so the email bounces.

This is a common issue on any mailing list. However, the reasons for the bounces go far beyond the accounts merely being closed:

        * user has mail filtering (I hate it when AOL'ers decide this is how
                you unsubscribe - by "refusing email from")
        * mailbox is full (you might not want to remove someone from your
                mailing list just because you get one of these -- a bunch
                however, and they're outta here)
        * final delivery cannot be made - intermediate server is bouncing you
                notification of this
        * DNS and MX configuration errors ("we do not relay" is one)
        * connectivity issues (remote host disconnected from the internet).
        * local config problems (incl. disk full) at the recipient host.

        plus a variety of others.

Often, the actual cause isn't even given in the bounce - you're just told the message couldn't be delivered.

One problem you'll find out in a hurry if you have much email traffic is that there are a LOT of truely abysmal MTAs out there (many of which are cheezy solutions for MS Windows based systems - I'm not knocking MS Windows, just the plethora of crap people choose to use on it). Some of these MTAs will bounce a message to the FROM: address rather than to the sender. There's even a few that address bounces to the original TO: address!

Ensuring that your list rejects posts from non-subscribers is a good first step in blocking such idiots. Adding loop checking (to ensure that a message which has already passed through your list isn't being sent back into it, say by some braindead wannabe MTA that ignores the envelope data and tries to deliver the message to the original cleartexed recipients -- usually your list) is another thing which should be done -- when you experience your first mailing list (whether you're the admin of a list or not) loop because some yo-yo's mail server is resubmitting messages, you'll have some respect for the grief -- but when your the sysadm of the server with a LOT of subscribers, the grief is really compounded.

Now, even if you were to somehow deal with the above problem MTAs, you're still stuck with the many which don't bounce an actual address identifier for the user which they're refusing to deliver for. Oversimplified "your message could not be delivered" or "could not deliver to user joebob, bob" (without address). Add to these messages which do identify a user address, but it's a rewritten address: bob(_dot_)joebob(_at_)corp(_dot_)somecompany(_dot_)tld instead of "bob(_at_)somecompany(_dot_)tld" who subscribed to your list. Don't forget delivery failures on the last leg of a FORWARDED subscriber: user subscribes as user(_at_)onebox(_dot_)com, which forwards to differentuser(_at_)yahoo(_dot_)com(_dot_) When yahoo bounces, it isn't going to provide you with the email address that is actually subbed to your list.

Some of these, you might discern the original recipient from an included set of headers from the message which was being bounced. Or not (esp if there were multiple recipients at the domain in question).

Some DSNs include multiple email addresses - with DIFFERENT failure reasons.

I could go on and on here...

What some email list managers do is set the sender address to a unique hash representing the subscribed address - when a message bounces (properly, but not necessarily with useful info in the body), the address it was sent to can be used to automatically determine the actual subscriber for that bounce. Of course, this means that EVERY MESSAGE through your list will be delivered as a UNIQUE message out to EACH RECIPIENT. Normally, if you delivered a list message to 20,000 subscribers, 500 of which happened to have hotmail accounts, there'd be ONE connection to the hotmail MX to attempt delivery of the message to all of the subscribers there. That makes the email exchange efficient - one copy of the message and a buggerin' large list of envelope recipients. To uniquely hash each message means that you'd have 500 message transactions with hotmail instead. Icko, esp on large lists or those with lots of traffic.

A workaround (if the list has enough traffic to justify it), is to handle regular list messages normally (in batch), but to send out a monthly subscription probe which is handled with a per-user hash. This subscription probe could be disguised as a FAQ or other informational message, to make it useful to the recipients.

What would be the best recipe to have procmail make a list of the email addresses that have bounced?

First would be to ensure that the list sender is an address separate from the list. The accepted standard is an owner-listname alias -- you could direct this to your individual address, or through an invoked procmail script which in turn delivers to you after pre-processing the messages.

A few months ago, I wrote a collection of procmail scripts to work in conjunction with several majordomo lists I'm involved with administering. These scripts handle messages being sent into the lists (attachments, HTML mail, idiots reposting complete digests in reply, etc), as well as the owner aliases, to which bounces issued by the majordomo processor - non-subscribers, mismatching subscription auths, keyword bounces, etc. are sent, as well as the majority of delivery bounces.

Owing to the wide variety of bounces and the issues outlined above (and many more), the listowner filtering doesn't automatically take action on problem addresses -- it does however afford a central place to put filters to eliminat certain nuisance bounces - say, when @home closed up shop, and all those addresses started bouncing -- after they'd been in the queue for five days, which resulted in five day lags on the bounces AFTER you manually unsubbed all the users. Centrally filtering them allowed those specific bounces to be weeded out and tossed rather than driving the listadmin insane. Likewise for some ISPs which insist on sending DSNs for messages every four hours for a week because their customer's email is handled over what must be a dialup...

These are handled on a case-by basis.

Something you should look at doing is separating out the DSNs which are emitted by your local server, versus those which are emailed back to you. Note the different types of failures reported.

If you come up with THE solution, please do share it -- the world is waiting for it.

---
 Sean B. Straw / Professional Software Engineering

 Procmail disclaimer: <http://www.professional.org/procmail/disclaimer.html>
 Please DO NOT carbon me on list replies.  I'll get my copy from the list.

_______________________________________________
procmail mailing list
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail

<Prev in Thread] Current Thread [Next in Thread>