ietf-asrg
[Top] [All Lists]

Re: [Asrg] URL shorteners, spam and DNS

2011-08-22 11:55:58

On Aug 22, 2011, at 9:35 AM, Martijn Grooten wrote:

URL shorteners (bit.ly, goo.gl, tinyurl.com etc.) have become popular in 
recent years for rather obvious reasons. They are being used by spammers for 
equally obvious reasons - both in email and on other platforms (e.g. Twitter).

A filter that checks URLs/domains against a blacklist will either miss the 
bad domains hidden behind the shorteners or, if they blacklist the shortener, 
find itself blocking legitimate messages. Do Not Use (third-party) URL 
Shorteners is sound advice to those sending email, but it's not going to stop 
random users from copying shortened URLs from Twitter or Facebook and pasting 
them into emails and shortened URLs are unlikely to stop featuring on Twitter.

Tell those providing shorteners to check URLs against blacklists is also a 
good idea - and probably necessary for them to stop ending up on blacklists 
themselves - but if a filter happens to prefer a different blacklist it 
doesn't help much. (I also don't know if checks are made every time someone 
clicks on the link or just when the shortened URL is generated.)

So I was wondering if it would help if shorteners published the URLs in a DNS 
txt record. As the path of a shortened URL usually consists of lowercase, 
uppercase letters and numbers, the uppercase letters need to be encoded, e.g. 
by preceding them with an underscore. So for instance to look up the URL 
behind

 http://bit.ly/gkP0H

would require a lookup of the TXT record for

 gk_p0_h._short.bit.ly

Now I don't know if this is something that would actually help those 
developing spam-/content-filters. Doing a HTTP lookup to determine the URL 
isn't exactly rocket science - though intuitively, it seems more 'natural' to 
use DNS, especially if that's what is used for the URL blacklist lookup.

Nor do I know if this would be something that would interest those providing 
shortening services. As it would allow browsers to avoid making a HTTP 
request to their services, it would mean they would stop having reliable 
click through statistics which, I guess, are a source of revenue to them.

It's not going to happen.

To do this would require a fairly complex, database backed "stunt" DNS server, 
hooked fairly intimately into the URL shorteners core application.

URL shorteners are deployed by people who understand webservers and HTTP, not 
DNS servers. Getting them to deploy anything beyond the basic DNS required to 
serve a website is a non-starter.

The only thing I'd expect to get any level of traction for would be something 
that can be implemented in a webserver, in much the same way as the URL 
shortener itself is implented (and, probably, running in the same webapp as the 
shortener itself).

There are several obvious ways to do that - http queries to a related hostname 
(http://destination.wttw.me/q8Qzth), a related path (http://wttw.me/d/q8Qzth), 
a suffix to the URL, a different MIME type, etc, etc.

The cleanest way would be to use a different request type. Conveniently, that's 
already implemented. If you send a HEAD request for the URL, you'll get a 301 
Moved response, and a Location: header containing the target URL.

And, at least for the URL shortener I use (bit.ly), that HEAD request will not 
count as a click for statistics tracking.

    platter:steve$ telnet wttw.me 80
    Trying 168.143.174.97...
    Connected to wttw.me.
    Escape character is '^]'.
    HEAD /q8Qzth HTTP/1.1
    Host: wttw.me
    Connection: close

    HTTP/1.1 301 Moved
    Server: nginx
    Date: Mon, 22 Aug 2011 16:52:38 GMT
    Content-Type: text/html; charset=utf-8
    Connection: close
    Set-Cookie: _bit=4eXXXXXXXXXXXXXXa8;domain=.wttw.me;expires=Sat Feb 18 
11:52:38 2012;path=/; HttpOnly
    Cache-control: private; max-age=90
    Location: http://irtf.org/
    MIME-Version: 1.0
    Content-Length: 108

    Connection closed by foreign host.

Cheers,
  Steve

_______________________________________________
Asrg mailing list
Asrg(_at_)irtf(_dot_)org
http://www.irtf.org/mailman/listinfo/asrg

<Prev in Thread] Current Thread [Next in Thread>