ietf-asrg
[Top] [All Lists]

[Asrg] Email Address Verification Notes (fwd)

2003-04-02 11:55:06
I'm reforwarding my email verification notes. For some reason they do not 
appear in the archive for my original post (anybody know why this happens?
https://www1.ietf.org/mail-archive/working-groups/asrg/current/msg01246.html)
The notes were also used in the presentation at:
http://www.elan.net/~william/asrg-emailpathverification-presentation.pdf
To a degree I need this post, so it goes to archive (so I could reference 
it in the summary I'm working on).

---------- Forwarded message ----------
Date: Wed, 12 Mar 2003 01:59:32 -0800 (PST)
From: william(_at_)elan(_dot_)net
To: ASRG <asrg(_at_)ietf(_dot_)org>
Subject: Email Address Verification Notes

That simple code for email verification as was discussed in thread
"RE: [Asrg] Several Observations and a solution that addresses them all" 
will not work -  it breaks mailing lists and is otherwise easily abused by 
spammers. But I'v probably spent more time looking at various verification
and callback ideas then almost anybody else on the list, so I'v put my 
notes in format that can be sent by email (see below). But beware these 
solutions are also breakable by spammers and generally require modification
to SMTP prootocol (but same applies to most other proposals, I simply 
have not seen any one idea that can actually work in the long run). 

if you do not mind in a couple days I'll publish notes on callback email 
retrieval as this has not been widely discussed yet. I have notes 
for most of ideas/proposals presented so far to ASRG, some notes are in 
good form (like below), some are not well organized (but I can try to find 
time to do organize it as below and if people are interested, post this 
to the list). I planned to post all this to website as well, never got 
around to it so far. [ Note: this is already sent, see
https://www1.ietf.org/mail-archive/working-groups/asrg/current/msg02315.html ]

From what is listed below, read #3. It does same as MAIL-FROM even more, 
but does not break anything (i.e. mailing lists, forwarders). 

1. Simple callback to verify email address
 Technology/Idea:
    When email is received and MAIL FROM command has been issues, the 
    receiving email server looks up MX for the domain entered in MAIL FROM
    and then connects to that server to verify sender address exists
 Origin: Postfix mailing list
 Pros:
   a. Easy to do using existing SMTP commands (like VRFY, EXPN or using RCPT)
   b. Implementations already exist and works fairly well
   c. Does not require modifications of SMTP or breaks anything
 Cons:
   a. Nothing is stopping spammer from using valid email address in FROM 
      even if they are not authorized to do it. Futhermore they could 
      still use different "From" header which will be seen by users as 
      the actual sender email address
   b. Nothing is stopping spammer from having valid domain name and having 
      it answer yes to all verification requests (but valid domain name 
      makes it easier to track spammer if we know where his mailserver is 
      located permanently)
   c. Email server operators do not want to allow email address verification 
      because this is abused by spammer to gather list of valid email addresses
 Summary:
   Usable now with no downsides but offers little protection against spammers

2. Callback verification with messageid tracking
 Technology/Idea:
    MTAs keep track of all outgoing messages for certain period of time
    (say 5 days) and when email is received, the server looks up MX and 
    then tries to verify that email really came from that domain by check 
    with mail server through special verify command to see if messageid is
    real. Variations require either messageid to be synchronized among all 
    MX servers or that verification be done for each MX servers listed or 
   that only first (MX 0) keep messageids and all other syncronize with it.
 Origin: (not entirely certain, one of mailing lists, TBD)
 Pros:
 a. Unlike #1 it allows verification only if email was sent, so no chance 
    of email address harvesting
 Cons:
 a. Spammers can use valid email domains and verification - see 1.cons.b
 a. Requires modification to SMTP (new command of extension to VRFY)
 b. Requires mail server to keep track of messageids for all outgoing email, 
either
    with separate database or using log files
 c. Breaks ability to send email directly and set from address unless email is 
    relayed through proper domain's from address 
 d. Syncronization of messageids among all MXs is difficult to accomplish and 
    all MXs being checked for messageid create too much delay
 e. Message can not be rejected at mail transmission time as messageid is 
    not available then, so entire message HAS to be received and then go 
    through filter and if messageid is wrong its, then returned back. 
 f. This approach relies on that email messages are received within 
    certain period of time (no long delays during transmission), otherwise 
    the message tracking information may no longer be available. It should 
    be noted that because of this, all verification must be done by 
    receiving mail server and not by mail client, because mail clients
    may check email several weeks too late and if this filter is installed 
    on mail client it could say that message is not authorized because it 
    checked email after message tracking information has already been deleted.
 Summary:
   Because of cons c & d this is not likely to be accepted by entire community 

3. Complex callback verification requirying full message tracking server 
   functionality with dns extensions
 Technology/Idea:
   Either new protocol/system is used for tracking mail message or its 
   integrated into SMTP as new extension. Either way this is complete system
   where outside autorized agent program can add and possibly delete message 
   tracking information into special database. A new Message Tracking 
     Header is introduced that has at least the following information:
   - sender email, - unique message id, - name of message verification server
     Additional extension to DNS is also necessary to list all possible mail 
   verification servers for the domain, this maybe either new field (MC) 
   or special agreed DNS name (mail-verify with CNAME or A) or with 
   possibly in-addr representation of agreed subdomain with A 127.0.0.1
     When mail server has received an email and supports this functionality, 
   it should parse message tracking header. First it needs to verify that
   listed sender email address is either one of mail transmission FROM or 
   the one in "From" header. It should then check on verification server by 
   by finding ip for listed name and then checking in dns for sender email
   domain to see if its one of authorized verification servers. If it is, 
   then connection should be made to that server to verify that listed 
   message id and message from (and possibly date) are as then are in that 
   server, and if it is, tracking information should be deleted (special 
   command is issued from connecting party when its verified and n
   mail can then be delivered.
     When new message is being sent, it can be sent from arbitrary location
   and arbitrary email address can be intered into "From" if necessary, but
   email message must be entered into correct verification server for that
   From domain and its up to the verification server to provide functionality
   to verify that you're allowed to use that particular email account 
   - this can be based on ip (most simple way)
   - if integrated into SMTP it it can be made part of AUTH
   - Possibly certificates can be used (maybe with STARTTLS) with certificate
     being issued and verified by user's ISP (so no central authority is needed)
   Whenver possible verification server should not only check that user is 
allowed
   to send email with from address from that domain but that user is actually
   who he says he is (i.e. his email address).
     Possibly message tracking should include also details on how long message
   ids are kept on message tracking server (expiration time).

 Origin: Compilation of several ideas (spaml & other mailing lists) plus DNS
         part taken from Vixies mailfrom. Actual proposal this full form is 
mine.
 Pros:
  a. Provides functionality to verify that email was sent by particular 
     sender and he has right to use that email
  b. System is similar to MAIL-FROM proposals but does not break mailing 
     lists or forwarders and in fact can be implemented with full backward 
     compatability
 Cons:
  a. Spammers can use valid email domains and verification - see 1.cons.b
  b. Probably will require modification to SMTP protocol for verificaiton
  c. New verfication server must keep track of all messages which requires 
     serious additional resources (cpu & storage)
  d. Message can not be rejected at mail transmission time. See #2.Cons.e
  e. Messages must be transmitted quickly (within period of time, tracking 
     is done). See #2.Cons.f for details
 Summary:
   Generally pretty good at determinning if user is allowed to send email 
 from mail domain (like RMX proposals) and does not break existing system.
 It does require modification to all mail servers and to protocol to be 
 of any use. Since this does not provide global verification, spammer can 
 make new system in compliance with new protocol/extensions and self-verify
 his emails, but having ip address of verification server can allow to 
 keep track of origin of emails sent by spammers.
 Additional Notes (STSA):
   Idea should be explored futher to use this with certificates, possibly
 approach is to issue certificates to verification servers and they can 
 issue certificates to users. 
   Possible if emails are groups together on the verfication server by who 
 sent it, a special command can be established to check how many emails
 this particular user has sent to determine if emails are sent in bulk
 (but because sender is providing his own verification, he can easily lie...)
    It would usefull to be able to deleted message tracking when message
 has been received (i.e. destination mail server can delete it after 
 verifying the message), unfortunetly this does not work well with mailing
 lists where multiple receipients receive the same message and all need 
 to verify it. But possibly sender can specify if email is certainly for 
 one user or possibly after delete command is received, message would not 
 be deleted immediatly but faster then usual.

3b. Complex callback verification with dynamic dns used for verification
 Technology/Idea:
   This is not idea on its own but variation of #3, so read #3 first.
   Particular to this variation is the idea that dynamic dns can be used
 for verificaiton which would not require modification to SMTP. In particular
 the setup would be such that when some server/user sends out email, it contacts
 special one of specially listed dynamic dns servers (as listed with special
 record for a zone its using for FROM) and adds new zone to it being actual
 messageid. The zone would include, several records - EMAIL (TXT record with 
 actual email address), VRFY (either A or CNAME of verification server). 
 SOA of the zone would already include date and expiration info, which is
 usefull. DNSSec can be used for secure verification and to establish identity.
 Origin: 
    My unpublished idea to go along with complex verification.
    Late addition - around March 02, 2003 (most other notes are from Nov/Dec 
2002)
 Pros:
   a. Everything listed for #3.Pros applies here
   b. Particular to this is that this would not require modification to SMTP
      protocol
   c. Since DynDNS software is already written and widely used, this can 
      be implimented fairlyquickly
 Cons:
   a. Everything listed for #3.Cons applies here
   b. This will most likely require dedicated ip address for dyndn sserver 
      as such this can be a problem for those who currently run everything
      (dns, mail, etc) from one ip address
   c. I have serious doubts that DynDNS can handle such a high frequency 
      of updates/additions that would happen (though retrievals would be 
      no problem) and that it can actually handle millions of subdomain 
      zone files. STSA and preferably ask dns experts
   d. Even without modification to SMTP its still necessary to modify 
transmission
      servers to enter correct info about eash message as well as modify 
      receiving servers to check for special header, etc. So in the end 
      this still requires changes to how SMTP works
  Summary:
     While pretty neet idea, it probably will not work because of con c

3c. Complex callback verification with modifications to MAIL FROM
 Technology/Idea:
   This is something to go with #3, and is not an idea on its own. Read #3 
first.
  Main addition here is that MAIL FROM as used during transmission can be 
modified
  to indicate that message has tracking information. This can be done as 
extension
  (ESMTP extension to MAIL FROM). Or it can also be done by modifying actual 
email
  address, so instead of user(_at_)domain(_dot_)com, this becomes:
    
verification(_dot_)server(_dot_)ip(_dot_)address+messageid++user(_at_)domain(_dot_)com
  Hopefully everything after the + can be ignored. But because full FROM address
  is passed along all between all servers used in relaying the message, it'll 
end
  up going up to the very end (receiving server) even if servers in between
  do not support new extensions. The receiving server that does support the
  extensions can use this information to bounce messages during transmission
  instead after email is already received and email headers can be retrieved.
 Origin: 
   My idea to go along with complex verification, part of original notes 
separated
   because this particular extension can break mailing lists
 Pros:
  a. See #3.Pros
  b. Additionally bad email can be bounced during transmission in most cases
  c. This basicly automaticly cloaks sender's email address which makes it 
     more difficult for email harvesters. 
 Cons:
  a. See #3.Cons
  b. This would fail with many mailing lists and there everything would 
     still rely on special header. This can not be used as substititure 
     for this "tracking" header
  c. Very long MAIL FROM can be a problem with old email servers.
 Summary:
  Because of cons b & c, probably better to go with ESMTP extension but 
  still STSA

3d. Complex callback verification using message tracking server
 Technology/Idea:
   This is something to go with #3, and is not an idea on its own, read #3
  first. Here the implementation is to use message tracking protocol being
  developed by MSGTRK IETF working group to provide message tracking 
  functionality. The information on messag etracking can be found at:
  http://www.ietf.org/html.charters/msgtrk-charter.html, the protocol will
  need to be modified to allow "reverse tracking" only by messageid but 
  then server would only report if message actually came through the 
  system. Also header will need to be introduced so that if mail server
  that support mail tracking sends email to next server in mail path that 
  does not support it and then that one sends to another server that does 
  supports it, then the last server would see what message tracking server
  was the last one in the chain and could resume message tracking. For 
  purposes of finding if email needs needs to be rejected, the destination
  server must check if domain listed in "From:" header supports message 
  tracking (lookup MTQP SRV record) if it does, it should check on the 
  listed message tracking server if message came from there and if 
  necessary check on others servers in the tracking chain to confirm 
  message path. In order to get around problems with deployment an 
  additional SRV record may have to be added specifiying that all MUAs at 
  source domain support messagetracking (i.e. when otherwise some email maybe 
  rejected that are ok, but where MUA send mail directly but did not support
  contacting mail message tracking server to leave a trace), this SRV record 
  would be indication that if mail came from the specified domain and there
  was no message tracking informaiton for it, that email can be rejected
  as SPAM. 
   It can be noted that functionality of message tracking could/should be 
  extended to have message tracking server keep store of some main headers 
  for emails (such as from, to, sender, etc), this can be usefull to 
  confirm changes in email (if they happen due to mailing list) and would 
  be a solution for some minor holes in the above system that exist due 
  to the fact that email maybe pathing through servers in between that 
  did not support message tracking and this could be used by spammers to 
  confuse message tracking system.
 Origin: 
  Notes on verification when futher worked on and trying to find protocol
  that can be used for message tracking. About Jan 2002 with some later 
  changes.
 Pros:
   a. Everything listed for #3.Pros applies here
   b. The deployment of this can be tied to deployment of Message Tracking 
      Protocol (which has not happened yet but protocol is in final 
      stage of compleition).
 Cons:
   a. See #3.Cons
   b. Necessary changes to MSGTRK may not be accepted as protocol is already 
      in final stage of approval by IETF. If changes are accepted it'll 
      delay official release of protocol by IETF.
   c. Message Tracking protocol is not in implemented in software yet and
      and as this requires completely new funcationality to SMTP and new 
      service to be run this may take quite a bit of time before new 
      software is available and begins to be installed by mail system 
      administrators. Its possible that use of this for anti-spam system
      will accelerate the deployment.
 Summary:
  If the approach for #3 is to go with SMTP extensions, than this would
  be the best as most of the necessary extensions have already been 
  developed for this protocol and eventhough cons b & c are substantial
  it'd still be easear to extend MSGTRCK then to work out completely new 
  extensions that does the same. This is probably preferred solution.

----
William Leibzon
Elan Communications Inc
william(_at_)elan(_dot_)net

_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg



<Prev in Thread] Current Thread [Next in Thread>