At 09:28 2008-07-31 -0700, George Crum wrote:
Thanks for your help here. These are notifications from a Nagios server
from Vendor_B. The subject will be exactly the same except for the
beginning which will be "PROBLEM" or "RECOVERY".
Communications clarity is already beginning to fail. "The subject will be
exactly the same except for the beginning". Is this subjects relating to
any given instance, or subjects across ALL notifications. For example,
would "PROBLEM account 1285 server failure" and "RECOVERY account 1285
server failure" be the ONLY subjects you'd ever see, regardless of how many
variations or instances of system failure you might experience? Or are the
subjects relating to any given failure instance unique, but otherwise the
same between the PROBLEM and RECOVERY phases (which is more consistent with
my own experiences).
If the latter (unique identifiers to track problem instances) then you need
to track the part that is after the PROBLEM or RECOVERY (i.e. a tracking
number for instance), and THAT indicates whether the specific issue should
be forwarded or not.
PROBLEM 8162315
PROBLEM 9127346
should BOTH be forwarded to the first contact, then be set to forward
further messages with those identifiers to the second contact.
RECOVERY 8162315
would then cause further messages with that identifier to be "reset", while
the 9127346 identifier would still be going to the second contact until
resolved.
Basically we have parts of our site outsourced to 2 vendors which are tied
to each other. If a service breaks with Vendor_A then Vendor_B will send
us notifications. These go to our oncall pager and Vendor_B refuses to
ack this Nagios alert when we ask them to, which will spit out a
notification every 30 minutes until the recovery of the service. Because
the fix is out of our control and Vendor_A has a terrible service record,
as in they will let this problem go on for days. I have been using
procmail to solve this but I am currently changing the filters manually.
Er, why not continue to send things to Vendor A because they SHOULD be
resolving it, but ALSO send it to Vendor B ?
Here's the flow I'd like to see.
1. All emails go to pager
that would be, "make copy; forward to pager"?
2. Vendor_B PROBLEM notification is recieved for the first time: to pager
3. Vendor_B PROBLEM notifications continue: to inbox
Q: is "inbox" the account where the procmail process is running, or is it
some other address?
PROBLEM notification received
extract problem ID (you know, knowing the FORMAT would help)
Check for existing problem ID file
if not found, create problem ID file; forward to pager; done
otherwise, forward (or fall through) to inbox; done
If the problem notifications have a unique ID, then you need to be able to
track those IDs. Probably the easiest method would be to extract the ID
and use it to touch a file incorporating that ID (so, nominal change to the
recipe I already posted). After a while though, I'd expect you might end
up with some accumulated files - though if every PROBLEM should have a
corresponding RECOVERY within a few days, this probably isn't a big
concern. You could append the IDs to a file, but then REMOVING them from
the file is a bother, and checking for them is a tiny bit more involved.
4. Vendor_B RECOVERY notification: to pager and reset Vendor_B PROBLEM
notification back to pager
RECOVERY notification received
extract problem ID
delete problem ID file (really doesn't matter if it is there or
not, though we should expect it should be)
forward to pager; done
5. All other emails go to pager
Er, please refer to your step 1, where ALL mails are supposed to be
going to the pager already.
---
Sean B. Straw / Professional Software Engineering
Procmail disclaimer: <http://www.professional.org/procmail/disclaimer.html>
Please DO NOT carbon me on list replies. I'll get my copy from the list.
____________________________________________________________
procmail mailing list Procmail homepage: http://www.procmail.org/
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail