ietf-smtp
[Top] [All Lists]

Re: Authentication/Reputation, TBR (was Everyone Greylists...)

2007-12-12 18:01:29


On Dec 12, 2007, at 11:42 AM, David MacQuigg wrote:

Hiding valid email addresses is a fundamentally insoluble problem. If legitimate senders can find out when an address is invalid (which they must if we are to preserve reliability), then spammers can do the same. Perhaps we can sacrifice some reliability for a little more security, e.g. by sending "nosuch recipient" rejects only when the sender is authenticated and reputable. The rest can be tempfailed until they give up.

This is solvable. TBR can both limit discovery of valid email- addresses AND greatly improve delivery integrity.

Section 3.2 of the TBR draft says '''
  If no valid RCPT TO address is
  supplied, the TBR command will simply fail.  If at least one valid
  RCPT TO address is supplied, then the TBR eXAM-URI argument will be
  accepted.'''

Doesn't this give the sender the same information as a "nosuch recipient" SMTP reject?

It is fairly common (although fairly problematic) for inbound MTAs to use different "valid recipient lists" as messages are forwarded to their destination. With TBR, an MTA may accept any local-part for a domain, where subsequent MTAs are then able to silently eliminate recipients subsequently considered invalid. Expunging recipient TBR references does not mandate DSNs to maintain delivery integrity, since fetching a message represents acceptance of an obligation to ensure delivery or report failure.

Providers of domain names, IP addresses, or certificates have a conflict of interest, and are unable to prevent access to spammers.

Unwilling, not unable.

Unwilling, and likely unwise. There remains an inherent conflict of interest. Using registration to limit access may also lead to forms of extortion. What type of non-repudiation protects innocent domains? Must all messages be signed by a CA? What limits the number of certificates issued? Should all certificates be aged 30 days before being accepted?

I'm no expert here, but it seems that all of these would be effective if we had a good reputation system in place.

A reputation system is reactive. Abuse tactics are able cycle within minutes where reputation would be much after this activity.

Then we wouldn't need high security, just enough to raise the cost of abusable IDs above the profit from one short spam run. I'm guessing that profit is somewhere around $100.

Should Gmail charge $100 for their email service? Why expect $100 charges against a stolen credit cards will limit abuse?

A corporate registration would do it, or maybe a valid credit card with no reported theft in 30 days, or even just a Paypal account.

Domain registrars are not willing to hold requests for even 24 hours. Any credit card or online account can be compromised. As it is now, bad actors are able to publish their domain 24 hours before even cooperative registries indicate which domains are new. The domain registry process demonstrates the inherent conflict of interest with providing access and preventing abuse.

We can "piggyback" on the ID checking done by the financial-services industry, which has a lot more to lose than a domain-name vouching service.

The many many millions of entities wanting a domain reside in almost every country using different currencies and communication systems. A consortium of postal services might be able to handle a centralized registration process. Perhaps by correlating domain names with a physical postal addresses. This would likely create a cottage industry aimed at generating domain names used by abusers.

All of this applies only to those who need to "jump start" a new legitimate ID. A good reputation can also be built by simply sending lots of good mail and very little spam to many receivers over a long time.

There remains an inherent conflict of interest, in addition to compromised systems infiltrating otherwise reputable domains, and of course more time is needed to accrue meaningful reputations. The TBR mechanism affords perhaps 20 minutes needed to resolve reputations from questionable sources.

Messages can be submitted as legitimate users, but likely originate from compromised systems. Correlating source patterns and overall volume determines the trend.

I think I understand what you are saying. Zombies are sending spam not directly, but through their ISP's outgoing mail servers. The ISP needs better authentication, rate-limits, etc. Most large ISPs (aol.com, yahoo.com, etc.) have this type of abuse under control.

Some are better than others. Bad actors seek out networks imposing fewer limitations and may see dial-ups or port 25 blocking as a limitation, for example.

Blocking port 25 is not a complete solution. Compromised systems can bypass such blocks. In the case of compromised systems, abuse is widely diffused such that rate limiting and outbound content filtering offers diminished relief.

I'm seeing very little spam from the authorized transmitters of aol.com, yahoo.com, and most other legitimate senders. Eliminating outgoing spam is not that difficult.

With an exponential trend, detection failures may be manageable now, but are not likely to remain that way. Reliance upon content filtering appears to be creating increasingly difficult to detect spam and malware.

Agreed, but then this also runs into conflicts of interest. An email-provider often does not want credit. TBR demands an identity, a matching MailFrom, and an MX record. In addition, TBR better identifies a problem source, and not just the last MTA "holding the bag" so to speak.

An email provider that "does not want credit", or more accurately, one that "is unwilling to assume responsibility" for what is sent in its name, gets the reputation it deserves.

When the domains are large, blocking just at the domain will cause a significant amount of collateral damage. When domain level blocking is used to triage limited resources, head of queue blocking will become increasing predominate problems.

The last MTA on the sending side (the "transmitter" in my terminology) is more than a "bag holder". Of all the parties involved in handling email, the operator of that transmitter is in the best position to stop spam. Google.com cannot deny that its transmitters are sending spam.

Google's problem is likely a conflict of interest. It wants new accounts so much that it is making it too easy for spammers to sign up for these accounts. The fix is not costly, however. They just need to segregate the new accounts, and until they earn some trust, use a different HELO name, and apply more rigorous rate-limits and filtering.

A widespread abuse problem has been seen by most large email providers, where even Google has improved their process. Eliminating organizations who attract large numbers of new users seems counter productive, as these domains also represent desired sources of email.


When spam must comply with grey-listing, transactions will dramatically increase. The vast majority of email is spam where a percentage can not be detected based upon content.

You may be right. I'm seeing a still small but growing percentage getting past SpamAssassin.

I can see this heading to either of two endpoints. Either the spammers will win, overwhelming the statistical filters and forcing everyone to sign up with a large ESP with the resources to work out individual arrangements with other large ESPs, or we find a way to hold senders responsible, and as you say "shift the burden to the transmitter", where it is 1000X easier to deal with.

If the spammers win, the anti-spam industry and a few large ESPs can look forward to a long and prosperous future.

There is no profit allowing spammers to win. Spam represents a growing waste of resources. Instead of one receiving server accessed over a single path, email now often travels over two or more paths through an array of servers needed to meet a resource demand. When a small and growing percentage of spam slips past filters, and there is a rapidly growing volume and diversity of sources, customers will perceive a problem with the service.

I hope TBR succeeds, but I suspect that the cost to both senders and receivers will outweigh the benefit of prompt delivery.

Deployment of TBR would be able to better ensure prompt delivery. The cost to the transmitter needs to be increased, while also reducing costs for the receiver.

The cost of publishing CSV records was very little, and it went nowhere. Nothing wrong technically, just a lack of sender motivation.

I do not agree as CSV can offer path registration as well. This path would be by name instead of a large IP address list. CSV can insure an upper limit of a couple of transactions instead of 10 x N transactions needed to construct possibly sizeable IP address lists. CSV could have provided an authorization scheme using a sha1 base32 label published withn the authorizing domain. This approach scales to any number of servers associated with a domain without increasing the transactions performed by receivers. Too many wanted fairly dangerous scripts risking DNS instead. : (

I think to get widespread adoption, the cost and risk must be not much more than publishing an SPF record ending in ?all, and less than the cost of publishing CSV records.

Ensuring messages are handled in a timely fashion will likely justify rather extensive changes. As the problem grows, moving nearly the entire burden to the transmitter will not be a difficult sell. It will likely become perhaps the only choice.

I think senders may need a kick in the pants. I'm thinking of an SMTP reject, with a message like "Sorry, your authentication records are insufficient, and we cannot assess the reputation of <domain>. Your message will be sent to our quarantine, but we cannot guarantee delivery. If it is important, call your recipient to make sure they got it."

There will not be any new notifications. No one wants to explain anything to users. Its too damn expensive. Things will just become increasingly broken.

-Doug